Orbiter History

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 1.396767786s trace: https://jaeger.dev.storj.io/trace/3b628ab935afa966 FAIL (2025-12-26 11:39:00 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.396794587s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 1.382774651s trace: https://jaeger.dev.storj.io/trace/3b628ab935afa966 FAIL (2025-12-26 11:38:44 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.382923704s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [......X..X.X......XX......XXX.XX......X.X..XX.X.X...X....X..] duration: 5.785319366s trace: https://jaeger.dev.storj.io/trace/730d54fb15e42c9 FAIL (2025-12-26 11:38:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: BUPXRO3RW37VVAD7STCKDV7MEL5IQ4XFGT7LV2TPL5GMYMR6HLNQ) context deadline exceeded; piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: SKLLCYFUPOQONKRHMVWQN2A7LTGYISC7U2EEICCUAHATLW4FHQ4A) context deadline exceeded; piecestore: (Node ID: 12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, Piece ID: RIJV3ILEDXMB4LIGMV4ZLPIIYGSV33JWESZMQECHMQIKLOAKYOBQ) context deadline exceeded ======LOGS====== Dec 26 11:36:56.505515043 (500h7m56.376598088s) [upload=438945]: Starting upload Dec 26 11:36:56.505618961 (500h7m56.376701996s) [upload=438945]: Done starting upload Dec 26 11:36:56.505837156 (500h7m56.376920212s) [upload=438945]: Uploading segments... Dec 26 11:36:56.506075150 (500h7m56.377158185s) [upload=438945]: Got next segment. Inline: false Dec 26 11:36:56.506117180 (500h7m56.377200215s) [upload=438945]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 11:36:56.506123451 (500h7m56.377206496s) [upload=438945]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 11:36:56.750379779 (500h7m56.621462814s) [seg_pos={0 0}, upload=438945]: Begin upload segment... Dec 26 11:36:56.750422250 (500h7m56.621505285s) [seg_pos={0 0}, upload=438945]: Done begin upload segment. Dec 26 11:36:56.750448380 (500h7m56.621531415s) [noise=true , address=145.79.227.4:4187 , piece_id=B7M6DVR5MO6LTCFI7XR3RY5UWO4JCUSHPSU54SM6OBX5OMUYUHOQ, seg_pos={0 0}, upload=438945]: Uploading piece... Dec 26 11:36:56.750524154 (500h7m56.621607189s) [noise=true , address=145.79.227.29:4941 , piece_id=JEMZO23FYHC2XJNQIQM5N6L3Y755GAHYB3XEE4XAKR5QPDPGWAKQ, seg_pos={0 0}, upload=438945]: Uploading piece... Dec 26 11:36:56.751138414 (500h7m56.622221449s) [noise=true , address=145.79.227.36:4519 , piece_id=DA2BGTCL7OSWGONK6SYEET2MUYWVYBAXQSYBQ63AYUX47QEVBH4A, seg_pos={0 0}, upload=438945]: Uploading piece... Dec 26 11:36:56.751453734 (500h7m56.622536769s) [noise=true , address=77.90.31.14:4661 , piece_id=RIJV3ILEDXMB4LIGMV4ZLPIIYGSV33JWESZMQECHMQIKLOAKYOBQ, seg_pos={0 0}, upload=438945]: Uploading piece... Dec 26 11:36:56.751751742 (500h7m56.622834777s) [noise=true , address=145.79.227.35:4481 , piece_id=ZRLEPLBXXZ2MXQCUF7RDJHHFOZK2DQ3ONBD6APNBZZR6O5HI3RNA, seg_pos={0 0}, upload=438945]: Uploading piece... Dec 26 11:36:56.752042666 (500h7m56.623125701s) [noise=true , address=145.79.227.52:4445 , piece_id=2Z2EBGR6LDCPB6VLER4RYPVDRT6YZQCSJSFKSHSKRTUNHNKJ5QLA, seg_pos={0 0}, upload=438945]: Uploading piece... Dec 26 11:36:56.752457315 (500h7m56.62354035s) [noise=true , address=45.91.116.13:4153 , piece_id=MNJ3SOQQ3DKQMHEKMOX6CTP2RGABGH265LCWJPTOUK7EEFZUYVAQ, seg_pos={0 0}, upload=438945]: Uploading piece... Dec 26 11:36:56.752715697 (500h7m56.623798732s) [noise=true , address=145.79.227.44:4194 , piece_id=25EWP6UT3JJRC4KYRHFH562MMWL244DH67QF5PCGZSJ5DKSZQMPA, seg_pos={0 0}, upload=438945]: Uploading piece... Dec 26 11:36:56.753043872 (500h7m56.624126907s) [noise=true , address=45.91.116.14:4192 , piece_id=BUPXRO3RW37VVAD7STCKDV7MEL5IQ4XFGT7LV2TPL5GMYMR6HLNQ, seg_pos={0 0}, upload=438945]: Uploading piece... Dec 26 11:36:56.754630302 (500h7m56.625713338s) [noise=true , address=145.79.227.16:4910 , piece_id=RMOVMJY6ZBHXW5CSVAK4KVJPZS4ZSY7VVV5BTWPYXUGFX2L7XX7A, seg_pos={0 0}, upload=438945]: Uploading piece... Dec 26 11:36:56.755866918 (500h7m56.626949953s) [noise=true , address=45.91.116.11:4118 , piece_id=SKLLCYFUPOQONKRHMVWQN2A7LTGYISC7U2EEICCUAHATLW4FHQ4A, seg_pos={0 0}, upload=438945]: Uploading piece... Dec 26 11:36:56.757061353 (500h7m56.628144388s) [noise=true , address=45.91.116.5:4297 , piece_id=Y5YU32R6FWZPM5SROGRHG3NWE7GVO54MNULIUGDGELMSFC52P7QQ, seg_pos={0 0}, upload=438945]: Uploading piece... Dec 26 11:36:56.758350327 (500h7m56.629433362s) [noise=true , address=145.79.227.46:4602 , piece_id=TEWX3BY3YSEH5GZA3ML35GJUN42NV3L2YKYWVI7724PJCLGNHO5Q, seg_pos={0 0}, upload=438945]: Uploading piece... Dec 26 11:36:56.759633310 (500h7m56.630716345s) [noise=true , address=145.79.227.41:4085 , piece_id=WR46TBHJSSONTI6SEQLGNOPSU3SK2LFV4LFX6TYVQQRU7YYV6LPQ, seg_pos={0 0}, upload=438945]: Uploading piece... Dec 26 11:36:56.760920681 (500h7m56.632003726s) [noise=true , address=45.91.116.8:4046 , piece_id=NGGRF6S3F2UMQ2GVIEY6EHAWKKYV75PHGHGIG24AISFEKAJDGSOQ, seg_pos={0 0}, upload=438945]: Uploading piece... Dec 26 11:36:56.762417291 (500h7m56.633500336s) [noise=true , address=145.79.227.42:4121 , piece_id=EUQVW2HYMPLXFJR7MADFAKVSIWXC6ABCGNNM5FMCNWLZYDN6WLEQ, seg_pos={0 0}, upload=438945]: Uploading piece... Dec 26 11:36:58.481396575 (500h7m58.35247961s) [upload=438945]: Next returned nil segment Dec 26 11:36:58.481413537 (500h7m58.352496572s) [upload=438945]: Waiting for error group managing segments... Dec 26 11:36:58.762340483 (500h7m58.633423518s) [noise=true , address=145.79.227.36:4519 , piece_id=DA2BGTCL7OSWGONK6SYEET2MUYWVYBAXQSYBQ63AYUX47QEVBH4A, seg_pos={0 0}, upload=438945]: Done uploading piece. err: <nil> Dec 26 11:36:58.769558617 (500h7m58.640641663s) [noise=true , address=145.79.227.35:4481 , piece_id=ZRLEPLBXXZ2MXQCUF7RDJHHFOZK2DQ3ONBD6APNBZZR6O5HI3RNA, seg_pos={0 0}, upload=438945]: Done uploading piece. err: <nil> Dec 26 11:36:58.775582498 (500h7m58.646665533s) [noise=true , address=145.79.227.52:4445 , piece_id=2Z2EBGR6LDCPB6VLER4RYPVDRT6YZQCSJSFKSHSKRTUNHNKJ5QLA, seg_pos={0 0}, upload=438945]: Done uploading piece. err: <nil> Dec 26 11:36:58.780348644 (500h7m58.651431689s) [noise=true , address=145.79.227.4:4187 , piece_id=B7M6DVR5MO6LTCFI7XR3RY5UWO4JCUSHPSU54SM6OBX5OMUYUHOQ, seg_pos={0 0}, upload=438945]: Done uploading piece. err: <nil> Dec 26 11:36:58.785099239 (500h7m58.656182284s) [noise=true , address=145.79.227.29:4941 , piece_id=JEMZO23FYHC2XJNQIQM5N6L3Y755GAHYB3XEE4XAKR5QPDPGWAKQ, seg_pos={0 0}, upload=438945]: Done uploading piece. err: <nil> Dec 26 11:36:58.791439683 (500h7m58.662522718s) [noise=true , address=145.79.227.46:4602 , piece_id=TEWX3BY3YSEH5GZA3ML35GJUN42NV3L2YKYWVI7724PJCLGNHO5Q, seg_pos={0 0}, upload=438945]: Done uploading piece. err: <nil> Dec 26 11:36:58.792281556 (500h7m58.663364591s) [noise=true , address=145.79.227.16:4910 , piece_id=RMOVMJY6ZBHXW5CSVAK4KVJPZS4ZSY7VVV5BTWPYXUGFX2L7XX7A, seg_pos={0 0}, upload=438945]: Done uploading piece. err: <nil> Dec 26 11:36:58.794862450 (500h7m58.665945516s) [noise=true , address=145.79.227.42:4121 , piece_id=EUQVW2HYMPLXFJR7MADFAKVSIWXC6ABCGNNM5FMCNWLZYDN6WLEQ, seg_pos={0 0}, upload=438945]: Done uploading piece. err: <nil> Dec 26 11:36:58.795905156 (500h7m58.666988201s) [noise=true , address=145.79.227.41:4085 , piece_id=WR46TBHJSSONTI6SEQLGNOPSU3SK2LFV4LFX6TYVQQRU7YYV6LPQ, seg_pos={0 0}, upload=438945]: Done uploading piece. err: <nil> Dec 26 11:36:59.013511989 (500h7m58.884595024s) [noise=true , address=77.90.31.14:4661 , piece_id=RIJV3ILEDXMB4LIGMV4ZLPIIYGSV33JWESZMQECHMQIKLOAKYOBQ, seg_pos={0 0}, upload=438945]: Done uploading piece. err: <nil> Dec 26 11:36:59.101556480 (500h7m58.972639516s) [noise=true , address=45.91.116.8:4046 , piece_id=NGGRF6S3F2UMQ2GVIEY6EHAWKKYV75PHGHGIG24AISFEKAJDGSOQ, seg_pos={0 0}, upload=438945]: Done uploading piece. err: <nil> Dec 26 11:36:59.109052565 (500h7m58.9801356s) [noise=true , address=45.91.116.11:4118 , piece_id=SKLLCYFUPOQONKRHMVWQN2A7LTGYISC7U2EEICCUAHATLW4FHQ4A, seg_pos={0 0}, upload=438945]: Done uploading piece. err: <nil> Dec 26 11:36:59.126268548 (500h7m58.997351583s) [noise=true , address=45.91.116.14:4192 , piece_id=BUPXRO3RW37VVAD7STCKDV7MEL5IQ4XFGT7LV2TPL5GMYMR6HLNQ, seg_pos={0 0}, upload=438945]: Done uploading piece. err: <nil> Dec 26 11:36:59.126296330 (500h7m58.997379365s) [seg_pos={0 0}, upload=438945]: Segment reached optimal threshold of 13 pieces. Dec 26 11:36:59.126530205 (500h7m58.99761325s) [noise=true , address=45.91.116.13:4153 , piece_id=MNJ3SOQQ3DKQMHEKMOX6CTP2RGABGH265LCWJPTOUK7EEFZUYVAQ, seg_pos={0 0}, upload=438945]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Dec 26 11:36:59.126541096 (500h7m58.997624131s) [noise=true , address=45.91.116.13:4153 , piece_id=MNJ3SOQQ3DKQMHEKMOX6CTP2RGABGH265LCWJPTOUK7EEFZUYVAQ, seg_pos={0 0}, upload=438945]: Long tail cancellation due to optimal threshold Dec 26 11:36:59.126640455 (500h7m58.99772349s) [noise=true , address=145.79.227.44:4194 , piece_id=25EWP6UT3JJRC4KYRHFH562MMWL244DH67QF5PCGZSJ5DKSZQMPA, seg_pos={0 0}, upload=438945]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 26 11:36:59.126695530 (500h7m58.997778575s) [noise=true , address=145.79.227.44:4194 , piece_id=25EWP6UT3JJRC4KYRHFH562MMWL244DH67QF5PCGZSJ5DKSZQMPA, seg_pos={0 0}, upload=438945]: Long tail cancellation due to optimal threshold Dec 26 11:36:59.126897886 (500h7m58.997980921s) [noise=true , address=45.91.116.5:4297 , piece_id=Y5YU32R6FWZPM5SROGRHG3NWE7GVO54MNULIUGDGELMSFC52P7QQ, seg_pos={0 0}, upload=438945]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 26 11:36:59.126908406 (500h7m58.997991441s) [noise=true , address=45.91.116.5:4297 , piece_id=Y5YU32R6FWZPM5SROGRHG3NWE7GVO54MNULIUGDGELMSFC52P7QQ, seg_pos={0 0}, upload=438945]: Long tail cancellation due to optimal threshold Dec 26 11:36:59.126957128 (500h7m58.998040173s) [seg_pos={0 0}, upload=438945]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 11:36:59.127105982 (500h7m58.998189027s) [upload=438945]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 11:36:59.127115209 (500h7m58.998198255s) [upload=438945]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 11:36:59.423053990 (500h7m59.294137035s) [upload=438945]: Done uploading segments... Dec 26 11:36:59.423075740 (500h7m59.294158775s) [upload=438945]: Upload finished. err: <nil> Duration: 5.785345915s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 1.389257224s trace: https://jaeger.dev.storj.io/trace/3b628ab935afa966 FAIL (2025-12-26 11:38:24 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.389288404s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 1.392535638s trace: https://jaeger.dev.storj.io/trace/3b628ab935afa966 FAIL (2025-12-26 11:37:56 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.392564072s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [.......X..X.X......XX......XXX.XX......X.X..XX.X.X...X....X.] duration: 5.92628483s trace: https://jaeger.dev.storj.io/trace/730d54fb15e42c9 FAIL (2025-12-26 11:37:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: BUPXRO3RW37VVAD7STCKDV7MEL5IQ4XFGT7LV2TPL5GMYMR6HLNQ) context deadline exceeded; piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: SKLLCYFUPOQONKRHMVWQN2A7LTGYISC7U2EEICCUAHATLW4FHQ4A) context deadline exceeded; piecestore: (Node ID: 12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, Piece ID: RIJV3ILEDXMB4LIGMV4ZLPIIYGSV33JWESZMQECHMQIKLOAKYOBQ) context deadline exceeded ======LOGS====== Dec 26 11:36:56.505515043 (500h7m56.376598088s) [upload=438945]: Starting upload Dec 26 11:36:56.505618961 (500h7m56.376701996s) [upload=438945]: Done starting upload Dec 26 11:36:56.505837156 (500h7m56.376920212s) [upload=438945]: Uploading segments... Dec 26 11:36:56.506075150 (500h7m56.377158185s) [upload=438945]: Got next segment. Inline: false Dec 26 11:36:56.506117180 (500h7m56.377200215s) [upload=438945]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 11:36:56.506123451 (500h7m56.377206496s) [upload=438945]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 11:36:56.750379779 (500h7m56.621462814s) [seg_pos={0 0}, upload=438945]: Begin upload segment... Dec 26 11:36:56.750422250 (500h7m56.621505285s) [seg_pos={0 0}, upload=438945]: Done begin upload segment. Dec 26 11:36:56.750448380 (500h7m56.621531415s) [noise=true , address=145.79.227.4:4187 , piece_id=B7M6DVR5MO6LTCFI7XR3RY5UWO4JCUSHPSU54SM6OBX5OMUYUHOQ, seg_pos={0 0}, upload=438945]: Uploading piece... Dec 26 11:36:56.750524154 (500h7m56.621607189s) [noise=true , address=145.79.227.29:4941 , piece_id=JEMZO23FYHC2XJNQIQM5N6L3Y755GAHYB3XEE4XAKR5QPDPGWAKQ, seg_pos={0 0}, upload=438945]: Uploading piece... Dec 26 11:36:56.751138414 (500h7m56.622221449s) [noise=true , address=145.79.227.36:4519 , piece_id=DA2BGTCL7OSWGONK6SYEET2MUYWVYBAXQSYBQ63AYUX47QEVBH4A, seg_pos={0 0}, upload=438945]: Uploading piece... Dec 26 11:36:56.751453734 (500h7m56.622536769s) [noise=true , address=77.90.31.14:4661 , piece_id=RIJV3ILEDXMB4LIGMV4ZLPIIYGSV33JWESZMQECHMQIKLOAKYOBQ, seg_pos={0 0}, upload=438945]: Uploading piece... Dec 26 11:36:56.751751742 (500h7m56.622834777s) [noise=true , address=145.79.227.35:4481 , piece_id=ZRLEPLBXXZ2MXQCUF7RDJHHFOZK2DQ3ONBD6APNBZZR6O5HI3RNA, seg_pos={0 0}, upload=438945]: Uploading piece... Dec 26 11:36:56.752042666 (500h7m56.623125701s) [noise=true , address=145.79.227.52:4445 , piece_id=2Z2EBGR6LDCPB6VLER4RYPVDRT6YZQCSJSFKSHSKRTUNHNKJ5QLA, seg_pos={0 0}, upload=438945]: Uploading piece... Dec 26 11:36:56.752457315 (500h7m56.62354035s) [noise=true , address=45.91.116.13:4153 , piece_id=MNJ3SOQQ3DKQMHEKMOX6CTP2RGABGH265LCWJPTOUK7EEFZUYVAQ, seg_pos={0 0}, upload=438945]: Uploading piece... Dec 26 11:36:56.752715697 (500h7m56.623798732s) [noise=true , address=145.79.227.44:4194 , piece_id=25EWP6UT3JJRC4KYRHFH562MMWL244DH67QF5PCGZSJ5DKSZQMPA, seg_pos={0 0}, upload=438945]: Uploading piece... Dec 26 11:36:56.753043872 (500h7m56.624126907s) [noise=true , address=45.91.116.14:4192 , piece_id=BUPXRO3RW37VVAD7STCKDV7MEL5IQ4XFGT7LV2TPL5GMYMR6HLNQ, seg_pos={0 0}, upload=438945]: Uploading piece... Dec 26 11:36:56.754630302 (500h7m56.625713338s) [noise=true , address=145.79.227.16:4910 , piece_id=RMOVMJY6ZBHXW5CSVAK4KVJPZS4ZSY7VVV5BTWPYXUGFX2L7XX7A, seg_pos={0 0}, upload=438945]: Uploading piece... Dec 26 11:36:56.755866918 (500h7m56.626949953s) [noise=true , address=45.91.116.11:4118 , piece_id=SKLLCYFUPOQONKRHMVWQN2A7LTGYISC7U2EEICCUAHATLW4FHQ4A, seg_pos={0 0}, upload=438945]: Uploading piece... Dec 26 11:36:56.757061353 (500h7m56.628144388s) [noise=true , address=45.91.116.5:4297 , piece_id=Y5YU32R6FWZPM5SROGRHG3NWE7GVO54MNULIUGDGELMSFC52P7QQ, seg_pos={0 0}, upload=438945]: Uploading piece... Dec 26 11:36:56.758350327 (500h7m56.629433362s) [noise=true , address=145.79.227.46:4602 , piece_id=TEWX3BY3YSEH5GZA3ML35GJUN42NV3L2YKYWVI7724PJCLGNHO5Q, seg_pos={0 0}, upload=438945]: Uploading piece... Dec 26 11:36:56.759633310 (500h7m56.630716345s) [noise=true , address=145.79.227.41:4085 , piece_id=WR46TBHJSSONTI6SEQLGNOPSU3SK2LFV4LFX6TYVQQRU7YYV6LPQ, seg_pos={0 0}, upload=438945]: Uploading piece... Dec 26 11:36:56.760920681 (500h7m56.632003726s) [noise=true , address=45.91.116.8:4046 , piece_id=NGGRF6S3F2UMQ2GVIEY6EHAWKKYV75PHGHGIG24AISFEKAJDGSOQ, seg_pos={0 0}, upload=438945]: Uploading piece... Dec 26 11:36:56.762417291 (500h7m56.633500336s) [noise=true , address=145.79.227.42:4121 , piece_id=EUQVW2HYMPLXFJR7MADFAKVSIWXC6ABCGNNM5FMCNWLZYDN6WLEQ, seg_pos={0 0}, upload=438945]: Uploading piece... Dec 26 11:36:58.481396575 (500h7m58.35247961s) [upload=438945]: Next returned nil segment Dec 26 11:36:58.481413537 (500h7m58.352496572s) [upload=438945]: Waiting for error group managing segments... Dec 26 11:36:58.762340483 (500h7m58.633423518s) [noise=true , address=145.79.227.36:4519 , piece_id=DA2BGTCL7OSWGONK6SYEET2MUYWVYBAXQSYBQ63AYUX47QEVBH4A, seg_pos={0 0}, upload=438945]: Done uploading piece. err: <nil> Dec 26 11:36:58.769558617 (500h7m58.640641663s) [noise=true , address=145.79.227.35:4481 , piece_id=ZRLEPLBXXZ2MXQCUF7RDJHHFOZK2DQ3ONBD6APNBZZR6O5HI3RNA, seg_pos={0 0}, upload=438945]: Done uploading piece. err: <nil> Dec 26 11:36:58.775582498 (500h7m58.646665533s) [noise=true , address=145.79.227.52:4445 , piece_id=2Z2EBGR6LDCPB6VLER4RYPVDRT6YZQCSJSFKSHSKRTUNHNKJ5QLA, seg_pos={0 0}, upload=438945]: Done uploading piece. err: <nil> Dec 26 11:36:58.780348644 (500h7m58.651431689s) [noise=true , address=145.79.227.4:4187 , piece_id=B7M6DVR5MO6LTCFI7XR3RY5UWO4JCUSHPSU54SM6OBX5OMUYUHOQ, seg_pos={0 0}, upload=438945]: Done uploading piece. err: <nil> Dec 26 11:36:58.785099239 (500h7m58.656182284s) [noise=true , address=145.79.227.29:4941 , piece_id=JEMZO23FYHC2XJNQIQM5N6L3Y755GAHYB3XEE4XAKR5QPDPGWAKQ, seg_pos={0 0}, upload=438945]: Done uploading piece. err: <nil> Dec 26 11:36:58.791439683 (500h7m58.662522718s) [noise=true , address=145.79.227.46:4602 , piece_id=TEWX3BY3YSEH5GZA3ML35GJUN42NV3L2YKYWVI7724PJCLGNHO5Q, seg_pos={0 0}, upload=438945]: Done uploading piece. err: <nil> Dec 26 11:36:58.792281556 (500h7m58.663364591s) [noise=true , address=145.79.227.16:4910 , piece_id=RMOVMJY6ZBHXW5CSVAK4KVJPZS4ZSY7VVV5BTWPYXUGFX2L7XX7A, seg_pos={0 0}, upload=438945]: Done uploading piece. err: <nil> Dec 26 11:36:58.794862450 (500h7m58.665945516s) [noise=true , address=145.79.227.42:4121 , piece_id=EUQVW2HYMPLXFJR7MADFAKVSIWXC6ABCGNNM5FMCNWLZYDN6WLEQ, seg_pos={0 0}, upload=438945]: Done uploading piece. err: <nil> Dec 26 11:36:58.795905156 (500h7m58.666988201s) [noise=true , address=145.79.227.41:4085 , piece_id=WR46TBHJSSONTI6SEQLGNOPSU3SK2LFV4LFX6TYVQQRU7YYV6LPQ, seg_pos={0 0}, upload=438945]: Done uploading piece. err: <nil> Dec 26 11:36:59.013511989 (500h7m58.884595024s) [noise=true , address=77.90.31.14:4661 , piece_id=RIJV3ILEDXMB4LIGMV4ZLPIIYGSV33JWESZMQECHMQIKLOAKYOBQ, seg_pos={0 0}, upload=438945]: Done uploading piece. err: <nil> Dec 26 11:36:59.101556480 (500h7m58.972639516s) [noise=true , address=45.91.116.8:4046 , piece_id=NGGRF6S3F2UMQ2GVIEY6EHAWKKYV75PHGHGIG24AISFEKAJDGSOQ, seg_pos={0 0}, upload=438945]: Done uploading piece. err: <nil> Dec 26 11:36:59.109052565 (500h7m58.9801356s) [noise=true , address=45.91.116.11:4118 , piece_id=SKLLCYFUPOQONKRHMVWQN2A7LTGYISC7U2EEICCUAHATLW4FHQ4A, seg_pos={0 0}, upload=438945]: Done uploading piece. err: <nil> Dec 26 11:36:59.126268548 (500h7m58.997351583s) [noise=true , address=45.91.116.14:4192 , piece_id=BUPXRO3RW37VVAD7STCKDV7MEL5IQ4XFGT7LV2TPL5GMYMR6HLNQ, seg_pos={0 0}, upload=438945]: Done uploading piece. err: <nil> Dec 26 11:36:59.126296330 (500h7m58.997379365s) [seg_pos={0 0}, upload=438945]: Segment reached optimal threshold of 13 pieces. Dec 26 11:36:59.126530205 (500h7m58.99761325s) [noise=true , address=45.91.116.13:4153 , piece_id=MNJ3SOQQ3DKQMHEKMOX6CTP2RGABGH265LCWJPTOUK7EEFZUYVAQ, seg_pos={0 0}, upload=438945]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Dec 26 11:36:59.126541096 (500h7m58.997624131s) [noise=true , address=45.91.116.13:4153 , piece_id=MNJ3SOQQ3DKQMHEKMOX6CTP2RGABGH265LCWJPTOUK7EEFZUYVAQ, seg_pos={0 0}, upload=438945]: Long tail cancellation due to optimal threshold Dec 26 11:36:59.126640455 (500h7m58.99772349s) [noise=true , address=145.79.227.44:4194 , piece_id=25EWP6UT3JJRC4KYRHFH562MMWL244DH67QF5PCGZSJ5DKSZQMPA, seg_pos={0 0}, upload=438945]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 26 11:36:59.126695530 (500h7m58.997778575s) [noise=true , address=145.79.227.44:4194 , piece_id=25EWP6UT3JJRC4KYRHFH562MMWL244DH67QF5PCGZSJ5DKSZQMPA, seg_pos={0 0}, upload=438945]: Long tail cancellation due to optimal threshold Dec 26 11:36:59.126897886 (500h7m58.997980921s) [noise=true , address=45.91.116.5:4297 , piece_id=Y5YU32R6FWZPM5SROGRHG3NWE7GVO54MNULIUGDGELMSFC52P7QQ, seg_pos={0 0}, upload=438945]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 26 11:36:59.126908406 (500h7m58.997991441s) [noise=true , address=45.91.116.5:4297 , piece_id=Y5YU32R6FWZPM5SROGRHG3NWE7GVO54MNULIUGDGELMSFC52P7QQ, seg_pos={0 0}, upload=438945]: Long tail cancellation due to optimal threshold Dec 26 11:36:59.126957128 (500h7m58.998040173s) [seg_pos={0 0}, upload=438945]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 11:36:59.127105982 (500h7m58.998189027s) [upload=438945]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 11:36:59.127115209 (500h7m58.998198255s) [upload=438945]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 11:36:59.423053990 (500h7m59.294137035s) [upload=438945]: Done uploading segments... Dec 26 11:36:59.423075740 (500h7m59.294158775s) [upload=438945]: Upload finished. err: <nil> Duration: 5.926303696s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [X.......X..X.X......XX......XXX.XX......X.X..XX.X.X...X....X] duration: 30.003450985s trace: https://jaeger.dev.storj.io/trace/730d54fb15e42c9 FAIL (2025-12-26 11:37:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: BUPXRO3RW37VVAD7STCKDV7MEL5IQ4XFGT7LV2TPL5GMYMR6HLNQ) context deadline exceeded; piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: SKLLCYFUPOQONKRHMVWQN2A7LTGYISC7U2EEICCUAHATLW4FHQ4A) context deadline exceeded; piecestore: (Node ID: 12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, Piece ID: RIJV3ILEDXMB4LIGMV4ZLPIIYGSV33JWESZMQECHMQIKLOAKYOBQ) context deadline exceeded ======LOGS====== Dec 26 11:36:56.505515043 (500h7m56.376598088s) [upload=438945]: Starting upload Dec 26 11:36:56.505618961 (500h7m56.376701996s) [upload=438945]: Done starting upload Dec 26 11:36:56.505837156 (500h7m56.376920212s) [upload=438945]: Uploading segments... Dec 26 11:36:56.506075150 (500h7m56.377158185s) [upload=438945]: Got next segment. Inline: false Dec 26 11:36:56.506117180 (500h7m56.377200215s) [upload=438945]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 11:36:56.506123451 (500h7m56.377206496s) [upload=438945]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 11:36:56.750379779 (500h7m56.621462814s) [seg_pos={0 0}, upload=438945]: Begin upload segment... Dec 26 11:36:56.750422250 (500h7m56.621505285s) [seg_pos={0 0}, upload=438945]: Done begin upload segment. Dec 26 11:36:56.750448380 (500h7m56.621531415s) [noise=true , address=145.79.227.4:4187 , piece_id=B7M6DVR5MO6LTCFI7XR3RY5UWO4JCUSHPSU54SM6OBX5OMUYUHOQ, seg_pos={0 0}, upload=438945]: Uploading piece... Dec 26 11:36:56.750524154 (500h7m56.621607189s) [noise=true , address=145.79.227.29:4941 , piece_id=JEMZO23FYHC2XJNQIQM5N6L3Y755GAHYB3XEE4XAKR5QPDPGWAKQ, seg_pos={0 0}, upload=438945]: Uploading piece... Dec 26 11:36:56.751138414 (500h7m56.622221449s) [noise=true , address=145.79.227.36:4519 , piece_id=DA2BGTCL7OSWGONK6SYEET2MUYWVYBAXQSYBQ63AYUX47QEVBH4A, seg_pos={0 0}, upload=438945]: Uploading piece... Dec 26 11:36:56.751453734 (500h7m56.622536769s) [noise=true , address=77.90.31.14:4661 , piece_id=RIJV3ILEDXMB4LIGMV4ZLPIIYGSV33JWESZMQECHMQIKLOAKYOBQ, seg_pos={0 0}, upload=438945]: Uploading piece... Dec 26 11:36:56.751751742 (500h7m56.622834777s) [noise=true , address=145.79.227.35:4481 , piece_id=ZRLEPLBXXZ2MXQCUF7RDJHHFOZK2DQ3ONBD6APNBZZR6O5HI3RNA, seg_pos={0 0}, upload=438945]: Uploading piece... Dec 26 11:36:56.752042666 (500h7m56.623125701s) [noise=true , address=145.79.227.52:4445 , piece_id=2Z2EBGR6LDCPB6VLER4RYPVDRT6YZQCSJSFKSHSKRTUNHNKJ5QLA, seg_pos={0 0}, upload=438945]: Uploading piece... Dec 26 11:36:56.752457315 (500h7m56.62354035s) [noise=true , address=45.91.116.13:4153 , piece_id=MNJ3SOQQ3DKQMHEKMOX6CTP2RGABGH265LCWJPTOUK7EEFZUYVAQ, seg_pos={0 0}, upload=438945]: Uploading piece... Dec 26 11:36:56.752715697 (500h7m56.623798732s) [noise=true , address=145.79.227.44:4194 , piece_id=25EWP6UT3JJRC4KYRHFH562MMWL244DH67QF5PCGZSJ5DKSZQMPA, seg_pos={0 0}, upload=438945]: Uploading piece... Dec 26 11:36:56.753043872 (500h7m56.624126907s) [noise=true , address=45.91.116.14:4192 , piece_id=BUPXRO3RW37VVAD7STCKDV7MEL5IQ4XFGT7LV2TPL5GMYMR6HLNQ, seg_pos={0 0}, upload=438945]: Uploading piece... Dec 26 11:36:56.754630302 (500h7m56.625713338s) [noise=true , address=145.79.227.16:4910 , piece_id=RMOVMJY6ZBHXW5CSVAK4KVJPZS4ZSY7VVV5BTWPYXUGFX2L7XX7A, seg_pos={0 0}, upload=438945]: Uploading piece... Dec 26 11:36:56.755866918 (500h7m56.626949953s) [noise=true , address=45.91.116.11:4118 , piece_id=SKLLCYFUPOQONKRHMVWQN2A7LTGYISC7U2EEICCUAHATLW4FHQ4A, seg_pos={0 0}, upload=438945]: Uploading piece... Dec 26 11:36:56.757061353 (500h7m56.628144388s) [noise=true , address=45.91.116.5:4297 , piece_id=Y5YU32R6FWZPM5SROGRHG3NWE7GVO54MNULIUGDGELMSFC52P7QQ, seg_pos={0 0}, upload=438945]: Uploading piece... Dec 26 11:36:56.758350327 (500h7m56.629433362s) [noise=true , address=145.79.227.46:4602 , piece_id=TEWX3BY3YSEH5GZA3ML35GJUN42NV3L2YKYWVI7724PJCLGNHO5Q, seg_pos={0 0}, upload=438945]: Uploading piece... Dec 26 11:36:56.759633310 (500h7m56.630716345s) [noise=true , address=145.79.227.41:4085 , piece_id=WR46TBHJSSONTI6SEQLGNOPSU3SK2LFV4LFX6TYVQQRU7YYV6LPQ, seg_pos={0 0}, upload=438945]: Uploading piece... Dec 26 11:36:56.760920681 (500h7m56.632003726s) [noise=true , address=45.91.116.8:4046 , piece_id=NGGRF6S3F2UMQ2GVIEY6EHAWKKYV75PHGHGIG24AISFEKAJDGSOQ, seg_pos={0 0}, upload=438945]: Uploading piece... Dec 26 11:36:56.762417291 (500h7m56.633500336s) [noise=true , address=145.79.227.42:4121 , piece_id=EUQVW2HYMPLXFJR7MADFAKVSIWXC6ABCGNNM5FMCNWLZYDN6WLEQ, seg_pos={0 0}, upload=438945]: Uploading piece... Dec 26 11:36:58.481396575 (500h7m58.35247961s) [upload=438945]: Next returned nil segment Dec 26 11:36:58.481413537 (500h7m58.352496572s) [upload=438945]: Waiting for error group managing segments... Dec 26 11:36:58.762340483 (500h7m58.633423518s) [noise=true , address=145.79.227.36:4519 , piece_id=DA2BGTCL7OSWGONK6SYEET2MUYWVYBAXQSYBQ63AYUX47QEVBH4A, seg_pos={0 0}, upload=438945]: Done uploading piece. err: <nil> Dec 26 11:36:58.769558617 (500h7m58.640641663s) [noise=true , address=145.79.227.35:4481 , piece_id=ZRLEPLBXXZ2MXQCUF7RDJHHFOZK2DQ3ONBD6APNBZZR6O5HI3RNA, seg_pos={0 0}, upload=438945]: Done uploading piece. err: <nil> Dec 26 11:36:58.775582498 (500h7m58.646665533s) [noise=true , address=145.79.227.52:4445 , piece_id=2Z2EBGR6LDCPB6VLER4RYPVDRT6YZQCSJSFKSHSKRTUNHNKJ5QLA, seg_pos={0 0}, upload=438945]: Done uploading piece. err: <nil> Dec 26 11:36:58.780348644 (500h7m58.651431689s) [noise=true , address=145.79.227.4:4187 , piece_id=B7M6DVR5MO6LTCFI7XR3RY5UWO4JCUSHPSU54SM6OBX5OMUYUHOQ, seg_pos={0 0}, upload=438945]: Done uploading piece. err: <nil> Dec 26 11:36:58.785099239 (500h7m58.656182284s) [noise=true , address=145.79.227.29:4941 , piece_id=JEMZO23FYHC2XJNQIQM5N6L3Y755GAHYB3XEE4XAKR5QPDPGWAKQ, seg_pos={0 0}, upload=438945]: Done uploading piece. err: <nil> Dec 26 11:36:58.791439683 (500h7m58.662522718s) [noise=true , address=145.79.227.46:4602 , piece_id=TEWX3BY3YSEH5GZA3ML35GJUN42NV3L2YKYWVI7724PJCLGNHO5Q, seg_pos={0 0}, upload=438945]: Done uploading piece. err: <nil> Dec 26 11:36:58.792281556 (500h7m58.663364591s) [noise=true , address=145.79.227.16:4910 , piece_id=RMOVMJY6ZBHXW5CSVAK4KVJPZS4ZSY7VVV5BTWPYXUGFX2L7XX7A, seg_pos={0 0}, upload=438945]: Done uploading piece. err: <nil> Dec 26 11:36:58.794862450 (500h7m58.665945516s) [noise=true , address=145.79.227.42:4121 , piece_id=EUQVW2HYMPLXFJR7MADFAKVSIWXC6ABCGNNM5FMCNWLZYDN6WLEQ, seg_pos={0 0}, upload=438945]: Done uploading piece. err: <nil> Dec 26 11:36:58.795905156 (500h7m58.666988201s) [noise=true , address=145.79.227.41:4085 , piece_id=WR46TBHJSSONTI6SEQLGNOPSU3SK2LFV4LFX6TYVQQRU7YYV6LPQ, seg_pos={0 0}, upload=438945]: Done uploading piece. err: <nil> Dec 26 11:36:59.013511989 (500h7m58.884595024s) [noise=true , address=77.90.31.14:4661 , piece_id=RIJV3ILEDXMB4LIGMV4ZLPIIYGSV33JWESZMQECHMQIKLOAKYOBQ, seg_pos={0 0}, upload=438945]: Done uploading piece. err: <nil> Dec 26 11:36:59.101556480 (500h7m58.972639516s) [noise=true , address=45.91.116.8:4046 , piece_id=NGGRF6S3F2UMQ2GVIEY6EHAWKKYV75PHGHGIG24AISFEKAJDGSOQ, seg_pos={0 0}, upload=438945]: Done uploading piece. err: <nil> Dec 26 11:36:59.109052565 (500h7m58.9801356s) [noise=true , address=45.91.116.11:4118 , piece_id=SKLLCYFUPOQONKRHMVWQN2A7LTGYISC7U2EEICCUAHATLW4FHQ4A, seg_pos={0 0}, upload=438945]: Done uploading piece. err: <nil> Dec 26 11:36:59.126268548 (500h7m58.997351583s) [noise=true , address=45.91.116.14:4192 , piece_id=BUPXRO3RW37VVAD7STCKDV7MEL5IQ4XFGT7LV2TPL5GMYMR6HLNQ, seg_pos={0 0}, upload=438945]: Done uploading piece. err: <nil> Dec 26 11:36:59.126296330 (500h7m58.997379365s) [seg_pos={0 0}, upload=438945]: Segment reached optimal threshold of 13 pieces. Dec 26 11:36:59.126530205 (500h7m58.99761325s) [noise=true , address=45.91.116.13:4153 , piece_id=MNJ3SOQQ3DKQMHEKMOX6CTP2RGABGH265LCWJPTOUK7EEFZUYVAQ, seg_pos={0 0}, upload=438945]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Dec 26 11:36:59.126541096 (500h7m58.997624131s) [noise=true , address=45.91.116.13:4153 , piece_id=MNJ3SOQQ3DKQMHEKMOX6CTP2RGABGH265LCWJPTOUK7EEFZUYVAQ, seg_pos={0 0}, upload=438945]: Long tail cancellation due to optimal threshold Dec 26 11:36:59.126640455 (500h7m58.99772349s) [noise=true , address=145.79.227.44:4194 , piece_id=25EWP6UT3JJRC4KYRHFH562MMWL244DH67QF5PCGZSJ5DKSZQMPA, seg_pos={0 0}, upload=438945]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 26 11:36:59.126695530 (500h7m58.997778575s) [noise=true , address=145.79.227.44:4194 , piece_id=25EWP6UT3JJRC4KYRHFH562MMWL244DH67QF5PCGZSJ5DKSZQMPA, seg_pos={0 0}, upload=438945]: Long tail cancellation due to optimal threshold Dec 26 11:36:59.126897886 (500h7m58.997980921s) [noise=true , address=45.91.116.5:4297 , piece_id=Y5YU32R6FWZPM5SROGRHG3NWE7GVO54MNULIUGDGELMSFC52P7QQ, seg_pos={0 0}, upload=438945]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 26 11:36:59.126908406 (500h7m58.997991441s) [noise=true , address=45.91.116.5:4297 , piece_id=Y5YU32R6FWZPM5SROGRHG3NWE7GVO54MNULIUGDGELMSFC52P7QQ, seg_pos={0 0}, upload=438945]: Long tail cancellation due to optimal threshold Dec 26 11:36:59.126957128 (500h7m58.998040173s) [seg_pos={0 0}, upload=438945]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 11:36:59.127105982 (500h7m58.998189027s) [upload=438945]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 11:36:59.127115209 (500h7m58.998198255s) [upload=438945]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 11:36:59.423053990 (500h7m59.294137035s) [upload=438945]: Done uploading segments... Dec 26 11:36:59.423075740 (500h7m59.294158775s) [upload=438945]: Upload finished. err: <nil> Duration: 30.003503865s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 1.379574628s trace: https://jaeger.dev.storj.io/trace/3b628ab935afa966 FAIL (2025-12-26 11:37:10 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.379608933s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.001603627s trace: https://jaeger.dev.storj.io/trace/3b628ab935afa966 FAIL (2025-12-26 11:37:08 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 30.001630408s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 57.61802232s trace: https://jaeger.dev.storj.io/trace/785a90e8be84a595 FAIL (2025-12-26 11:36:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== KXVRQQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.834123964 (499h15m56.705207009s) [noise=true , address=45.91.116.4:4963 , piece_id=U4T64KQ67C3BDKG5TJSTQWUS4WWYCSVPUZOWJ6YANR2IQ2JD4EFQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.834585043 (499h15m56.705668078s) [noise=true , address=145.79.227.9:4390 , piece_id=JDQNHNGABBGANIHKSKP4VHUWIPQBNNEOPC4ZQCKR3K5TTR4N4SGQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.834961800 (499h15m56.706044845s) [noise=true , address=145.79.227.30:4778 , piece_id=BMEAUWANP7FP6CFRULMIAZUV4JNQEXOUN2EAXJY5XOWLDZINWOIA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.835331334 (499h15m56.706414379s) [noise=true , address=77.90.31.18:4838 , piece_id=VYBIKY3MLEE2CXGS2MT23HX5HUFK25FKYKI4XLXPDRNWCC55ANXQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.835738399 (499h15m56.706821434s) [noise=true , address=145.79.227.20:4459 , piece_id=2EUW2AH7FBVZNJ27XVLRXKMXA737DWKAYBL7CR7PRXXRTO2M644A, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.836118653 (499h15m56.707201698s) [noise=true , address=77.90.31.14:4678 , piece_id=XQHQZJUPB24G3BWGFTFDD42NM6K5HJVHUXBXWH5TW7BU25QGHTMA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.836501923 (499h15m56.707584968s) [noise=true , address=45.91.116.6:4988 , piece_id=VPEKJF74D7SQWKR47MAIY5UJQPBJIEU5AQDKLFGATYIRHFREAQ5A, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.836904940 (499h15m56.707987985s) [noise=true , address=64.49.13.169:4711 , piece_id=PVBVY4TVJLPKYBP6A7LBNZADLN25XBDWZFEK2QVN45YHJATIR7TA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.837388641 (499h15m56.708471676s) [noise=true , address=145.79.227.40:4076 , piece_id=Y6WKUYNPDMBUZSNWEYDL2KKNR6WKHSKGP63S6KAG6F2MWMNTVMBQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.837910455 (499h15m56.7089935s) [noise=true , address=77.90.31.11:4571 , piece_id=OERZACIKIG7HBEV3LL42KUAGKN6AWZEXNQSA52FIYTUKDFWCBW7Q, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.838146514 (499h15m56.709229559s) [noise=true , address=64.49.13.26:4945 , piece_id=KXCFTLWR4HSBTP4GJJ4HEPA6BBEMILYTJEMXY4S5WB24HRMQJOKA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.838469329 (499h15m56.709552364s) [noise=true , address=45.91.116.5:4301 , piece_id=E3ZRKZUKM335ARHV3Z5YGLKQYHG6QV4GCCNHAVMN34BM7UQFWIIA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.838690831 (499h15m56.709773866s) [noise=true , address=45.91.116.14:4204 , piece_id=JWL7OOMKTVAP7WEUB5WTL3F6F5IRNID4KIOYJSYZ655PJSNXAQKQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.839143724 (499h15m56.710226759s) [noise=true , address=45.91.116.9:4272 , piece_id=GGFDVKEW5LUCHQQUP6CWJHJQXUETYUG2UISPVP6XQJGSAJW2KP2Q, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.839678982 (499h15m56.710762027s) [noise=true , address=145.79.227.44:4208 , piece_id=ZLIX7XAUMKRUEWWTQST2KLCOWFJSAFSAQNLUCTTOWVZUBESEI23A, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:58.136536405 (499h15m58.00761944s) [upload=438194]: Next returned nil segment Dec 26 10:44:58.136548628 (499h15m58.007631673s) [upload=438194]: Waiting for error group managing segments... Dec 26 10:44:58.469618858 (499h15m58.340701893s) [noise=true , address=64.49.13.26:4945 , piece_id=KXCFTLWR4HSBTP4GJJ4HEPA6BBEMILYTJEMXY4S5WB24HRMQJOKA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.708819055 (499h15m58.5799021s) [noise=true , address=145.79.227.9:4390 , piece_id=JDQNHNGABBGANIHKSKP4VHUWIPQBNNEOPC4ZQCKR3K5TTR4N4SGQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.714466429 (499h15m58.585549474s) [noise=true , address=145.79.227.40:4076 , piece_id=Y6WKUYNPDMBUZSNWEYDL2KKNR6WKHSKGP63S6KAG6F2MWMNTVMBQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.745407696 (499h15m58.616490731s) [noise=true , address=145.79.227.20:4459 , piece_id=2EUW2AH7FBVZNJ27XVLRXKMXA737DWKAYBL7CR7PRXXRTO2M644A, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.888701070 (499h15m58.759784105s) [noise=true , address=77.90.31.18:4838 , piece_id=VYBIKY3MLEE2CXGS2MT23HX5HUFK25FKYKI4XLXPDRNWCC55ANXQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.948150873 (499h15m58.819233918s) [noise=true , address=45.91.116.5:4301 , piece_id=E3ZRKZUKM335ARHV3Z5YGLKQYHG6QV4GCCNHAVMN34BM7UQFWIIA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.971389744 (499h15m58.842472779s) [noise=true , address=77.90.31.19:4868 , piece_id=WH77PZSXC2VUWLQYLBR2NDEVT6KMD3VZK5WAL53AMRDW74P242LQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.021685314 (499h15m58.892768349s) [noise=true , address=45.91.116.4:4963 , piece_id=U4T64KQ67C3BDKG5TJSTQWUS4WWYCSVPUZOWJ6YANR2IQ2JD4EFQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.045418216 (499h15m58.916501251s) [noise=true , address=45.91.116.14:4204 , piece_id=JWL7OOMKTVAP7WEUB5WTL3F6F5IRNID4KIOYJSYZ655PJSNXAQKQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.056260371 (499h15m58.927343416s) [noise=true , address=45.91.116.9:4272 , piece_id=GGFDVKEW5LUCHQQUP6CWJHJQXUETYUG2UISPVP6XQJGSAJW2KP2Q, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.085713054 (499h15m58.956796089s) [noise=true , address=45.91.116.7:4017 , piece_id=L4GGY6AICMSFMUVYGI6ZMUORYRXNJFPLJY5XJRUAM7NVYQOHPHSQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.640980124 (499h15m59.512063169s) [noise=true , address=145.79.227.30:4778 , piece_id=BMEAUWANP7FP6CFRULMIAZUV4JNQEXOUN2EAXJY5XOWLDZINWOIA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.654515607 (499h15m59.525598652s) [noise=true , address=145.79.227.44:4208 , piece_id=ZLIX7XAUMKRUEWWTQST2KLCOWFJSAFSAQNLUCTTOWVZUBESEI23A, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.660349657 (499h15m59.531432712s) [noise=true , address=77.90.31.11:4571 , piece_id=OERZACIKIG7HBEV3LL42KUAGKN6AWZEXNQSA52FIYTUKDFWCBW7Q, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.660376087 (499h15m59.531459132s) [seg_pos={0 0}, upload=438194]: Segment reached optimal threshold of 14 pieces. Dec 26 10:44:59.660612988 (499h15m59.531696033s) [noise=true , address=64.49.13.169:4711 , piece_id=PVBVY4TVJLPKYBP6A7LBNZADLN25XBDWZFEK2QVN45YHJATIR7TA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:52964->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:52964->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52964->64.49.13.169:4711: use of closed network connection Dec 26 10:44:59.660620753 (499h15m59.531703788s) [noise=true , address=64.49.13.169:4711 , piece_id=PVBVY4TVJLPKYBP6A7LBNZADLN25XBDWZFEK2QVN45YHJATIR7TA, seg_pos={0 0}, upload=438194]: Long tail cancellation due to optimal threshold Dec 26 10:44:59.660893663 (499h15m59.531976698s) [noise=true , address=45.91.116.6:4988 , piece_id=VPEKJF74D7SQWKR47MAIY5UJQPBJIEU5AQDKLFGATYIRHFREAQ5A, seg_pos={0 0}, upload=438194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TrcoCHhAJeEykP9hJNpygoPSdgFV5v6GzL48JpyNPiAj5ZJRS): protocol: expected piece hash; context canceled; EOF Dec 26 10:44:59.660901057 (499h15m59.531984102s) [noise=true , address=45.91.116.6:4988 , piece_id=VPEKJF74D7SQWKR47MAIY5UJQPBJIEU5AQDKLFGATYIRHFREAQ5A, seg_pos={0 0}, upload=438194]: Long tail cancellation due to optimal threshold Dec 26 10:44:59.661226155 (499h15m59.5323092s) [noise=true , address=77.90.31.14:4678 , piece_id=XQHQZJUPB24G3BWGFTFDD42NM6K5HJVHUXBXWH5TW7BU25QGHTMA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12beu3SUZMPNWfbeHV4tLtx9UVZoEHh4Kt7oBhT59w3bGBE1ovP): protocol: write tcp 188.166.225.170:35884->77.90.31.14:4678: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 10:44:59.661249630 (499h15m59.532332685s) [noise=true , address=77.90.31.14:4678 , piece_id=XQHQZJUPB24G3BWGFTFDD42NM6K5HJVHUXBXWH5TW7BU25QGHTMA, seg_pos={0 0}, upload=438194]: Long tail cancellation due to optimal threshold Dec 26 10:44:59.661309705 (499h15m59.53239275s) [noise=true , address=77.90.31.21:4909 , piece_id=H6KT2AKZISCU4VEA6TXJJ3EFGCRBZPPXBEJJEIHZR3GEJZKXVRQQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121RT7tP6czMFAiZ3DjzSSsYgPt3xZXVbXVpkcYxaZYscNiMjL6): protocol: expected piece hash; context canceled; EOF Dec 26 10:44:59.661318962 (499h15m59.532401997s) [noise=true , address=77.90.31.21:4909 , piece_id=H6KT2AKZISCU4VEA6TXJJ3EFGCRBZPPXBEJJEIHZR3GEJZKXVRQQ, seg_pos={0 0}, upload=438194]: Long tail cancellation due to optimal threshold Dec 26 10:44:59.661345674 (499h15m59.532428719s) [seg_pos={0 0}, upload=438194]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 10:44:59.661485770 (499h15m59.532568815s) [upload=438194]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 10:44:59.661493655 (499h15m59.53257669s) [upload=438194]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 10:44:59.968467153 (499h15m59.839550198s) [upload=438194]: Done uploading segments... Dec 26 10:44:59.968484155 (499h15m59.83956719s) [upload=438194]: Upload finished. err: <nil> Duration: 57.61803815s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [.X.......X..X.X......XX......XXX.XX......X.X..XX.X.X...X....] duration: 5.939737476s trace: https://jaeger.dev.storj.io/trace/50561a690e9a382d FAIL (2025-12-26 11:35:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: GEKQWOSNACJU2W6AYFCDHIO7GHB674NOVZARJRTIAT3FR3LYMSJQ) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: 6NHEH7ZOYADN6WLEHKVC4NVAS6J5NGUQGOYWO6S7KXEDUED6RACA) context deadline exceeded ======LOGS====== 0h4m28.049098989s) [upload=438898]: Uploading segments... Dec 26 11:33:28.178049557 (500h4m28.049132592s) [upload=438898]: Got next segment. Inline: false Dec 26 11:33:28.178074825 (500h4m28.04915786s) [upload=438898]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 11:33:28.178079243 (500h4m28.049162278s) [upload=438898]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 11:33:28.422805907 (500h4m28.293888942s) [seg_pos={0 0}, upload=438898]: Begin upload segment... Dec 26 11:33:28.422886350 (500h4m28.293969395s) [seg_pos={0 0}, upload=438898]: Done begin upload segment. Dec 26 11:33:28.422961903 (500h4m28.294044948s) [noise=true , address=145.79.227.46:4603 , piece_id=2CF7OVADT5JQSUFVCKNIMPD6YUCHVYQ3F2L5OYHVSM2Q735L46CQ, seg_pos={0 0}, upload=438898]: Uploading piece... Dec 26 11:33:28.423011248 (500h4m28.294094293s) [noise=true , address=145.79.227.51:4410 , piece_id=4LO4TXF75PW4O3SLXD2U5HABUR5GT5Y6BPWNIMNRLDPT2PTY3LVQ, seg_pos={0 0}, upload=438898]: Uploading piece... Dec 26 11:33:28.423619065 (500h4m28.2947021s) [noise=true , address=145.79.227.44:4194 , piece_id=3OISDZXJ4I5C73TFZCAKEPQW6JTNCRA5OTJLUTN2QQVPGR2JQRGQ, seg_pos={0 0}, upload=438898]: Uploading piece... Dec 26 11:33:28.423948102 (500h4m28.295031137s) [noise=true , address=45.91.116.14:4192 , piece_id=GEKQWOSNACJU2W6AYFCDHIO7GHB674NOVZARJRTIAT3FR3LYMSJQ, seg_pos={0 0}, upload=438898]: Uploading piece... Dec 26 11:33:28.424233195 (500h4m28.29531623s) [noise=true , address=145.79.227.16:4912 , piece_id=T7E5PDUZHRO3WRKDHLSYLWQLSTOVQPE3NKX7LISG5S6RHUFDM36Q, seg_pos={0 0}, upload=438898]: Uploading piece... Dec 26 11:33:28.424520181 (500h4m28.295603216s) [noise=true , address=45.91.116.5:4299 , piece_id=QCXEG2TLNUTPC53AVZ74NB5G7VVCZR3GN7A3VTUFCQYIMJU4ZXLQ, seg_pos={0 0}, upload=438898]: Uploading piece... Dec 26 11:33:28.424807708 (500h4m28.295890753s) [noise=true , address=145.79.227.49:4339 , piece_id=ZMV6JF3W5S55F5OO2K6Z4SNBNOIULEBEG6RACJ526NIPVBVQ5PMA, seg_pos={0 0}, upload=438898]: Uploading piece... Dec 26 11:33:28.425093333 (500h4m28.296176368s) [noise=true , address=45.91.116.12:4225 , piece_id=OUFJURNDHBRTMORDTCDYP5V6NPGZEH3FS2HMIVDNKB25CTJMN47Q, seg_pos={0 0}, upload=438898]: Uploading piece... Dec 26 11:33:28.425412991 (500h4m28.296496036s) [noise=true , address=45.91.116.11:4120 , piece_id=UK7U3NV5HV2BVVUDT7MGROYATPLGM2B2FCUR5WB4IFXIOYWWWRFA, seg_pos={0 0}, upload=438898]: Uploading piece... Dec 26 11:33:28.426729629 (500h4m28.297812664s) [noise=true , address=145.79.227.29:4943 , piece_id=NALMWEB5P65553GXA7JYGN254ZQCZSGWDILZPYBQQBJOI2Q2CFVA, seg_pos={0 0}, upload=438898]: Uploading piece... Dec 26 11:33:28.427120402 (500h4m28.298203448s) [noise=true , address=45.91.116.8:4045 , piece_id=6NHEH7ZOYADN6WLEHKVC4NVAS6J5NGUQGOYWO6S7KXEDUED6RACA, seg_pos={0 0}, upload=438898]: Uploading piece... Dec 26 11:33:28.428470443 (500h4m28.299553478s) [noise=true , address=45.91.116.10:4084 , piece_id=FAINT7GUVTE6JKVDXLDIUQHW3TFNG2C4AMJ7WHGRQ6GY7Z5G2EPA, seg_pos={0 0}, upload=438898]: Uploading piece... Dec 26 11:33:28.429781790 (500h4m28.300864825s) [noise=true , address=77.90.31.10:4519 , piece_id=I2AXQLV3D5GTZXXA6SZJVRFXVIICUEGFGYOBNZKP42KHW6KKRZCA, seg_pos={0 0}, upload=438898]: Uploading piece... Dec 26 11:33:28.431046629 (500h4m28.302129664s) [noise=true , address=145.79.227.50:4374 , piece_id=4454XFKBWSOPOKYMCCLMDJDW52Z2QH7FG6QQ4DVVYIAZ5NVUNO5Q, seg_pos={0 0}, upload=438898]: Uploading piece... Dec 26 11:33:28.432307870 (500h4m28.303390905s) [noise=true , address=77.90.31.11:4556 , piece_id=YBOZQXNWZ4D73XLYY4UK25HJIHKND3OYNNAJ33O35T4VZUED7YXA, seg_pos={0 0}, upload=438898]: Uploading piece... Dec 26 11:33:28.433521131 (500h4m28.304604166s) [noise=true , address=145.79.227.38:4979 , piece_id=ASE2VKWERDIBIYWWKKDYIRY337XXM2IDZN7LUF4RQGZM4N3BJSHA, seg_pos={0 0}, upload=438898]: Uploading piece... Dec 26 11:33:30.148590360 (500h4m30.019673405s) [upload=438898]: Next returned nil segment Dec 26 11:33:30.148613774 (500h4m30.019696819s) [upload=438898]: Waiting for error group managing segments... Dec 26 11:33:30.426305860 (500h4m30.297388895s) [noise=true , address=145.79.227.51:4410 , piece_id=4LO4TXF75PW4O3SLXD2U5HABUR5GT5Y6BPWNIMNRLDPT2PTY3LVQ, seg_pos={0 0}, upload=438898]: Done uploading piece. err: <nil> Dec 26 11:33:30.435012920 (500h4m30.306095965s) [noise=true , address=145.79.227.16:4912 , piece_id=T7E5PDUZHRO3WRKDHLSYLWQLSTOVQPE3NKX7LISG5S6RHUFDM36Q, seg_pos={0 0}, upload=438898]: Done uploading piece. err: <nil> Dec 26 11:33:30.449914575 (500h4m30.32099762s) [noise=true , address=145.79.227.50:4374 , piece_id=4454XFKBWSOPOKYMCCLMDJDW52Z2QH7FG6QQ4DVVYIAZ5NVUNO5Q, seg_pos={0 0}, upload=438898]: Done uploading piece. err: <nil> Dec 26 11:33:30.456513110 (500h4m30.327596155s) [noise=true , address=145.79.227.38:4979 , piece_id=ASE2VKWERDIBIYWWKKDYIRY337XXM2IDZN7LUF4RQGZM4N3BJSHA, seg_pos={0 0}, upload=438898]: Done uploading piece. err: <nil> Dec 26 11:33:30.457390621 (500h4m30.328473656s) [noise=true , address=145.79.227.29:4943 , piece_id=NALMWEB5P65553GXA7JYGN254ZQCZSGWDILZPYBQQBJOI2Q2CFVA, seg_pos={0 0}, upload=438898]: Done uploading piece. err: <nil> Dec 26 11:33:30.462934878 (500h4m30.334017913s) [noise=true , address=145.79.227.46:4603 , piece_id=2CF7OVADT5JQSUFVCKNIMPD6YUCHVYQ3F2L5OYHVSM2Q735L46CQ, seg_pos={0 0}, upload=438898]: Done uploading piece. err: <nil> Dec 26 11:33:30.464048539 (500h4m30.335131584s) [noise=true , address=145.79.227.49:4339 , piece_id=ZMV6JF3W5S55F5OO2K6Z4SNBNOIULEBEG6RACJ526NIPVBVQ5PMA, seg_pos={0 0}, upload=438898]: Done uploading piece. err: <nil> Dec 26 11:33:30.647928789 (500h4m30.519011824s) [noise=true , address=45.91.116.11:4120 , piece_id=UK7U3NV5HV2BVVUDT7MGROYATPLGM2B2FCUR5WB4IFXIOYWWWRFA, seg_pos={0 0}, upload=438898]: Done uploading piece. err: <nil> Dec 26 11:33:30.666692729 (500h4m30.537775774s) [noise=true , address=45.91.116.14:4192 , piece_id=GEKQWOSNACJU2W6AYFCDHIO7GHB674NOVZARJRTIAT3FR3LYMSJQ, seg_pos={0 0}, upload=438898]: Done uploading piece. err: <nil> Dec 26 11:33:30.804039442 (500h4m30.675122477s) [noise=true , address=45.91.116.12:4225 , piece_id=OUFJURNDHBRTMORDTCDYP5V6NPGZEH3FS2HMIVDNKB25CTJMN47Q, seg_pos={0 0}, upload=438898]: Done uploading piece. err: <nil> Dec 26 11:33:30.844837208 (500h4m30.715920243s) [noise=true , address=77.90.31.11:4556 , piece_id=YBOZQXNWZ4D73XLYY4UK25HJIHKND3OYNNAJ33O35T4VZUED7YXA, seg_pos={0 0}, upload=438898]: Done uploading piece. err: <nil> Dec 26 11:33:31.128170436 (500h4m30.999253471s) [noise=true , address=45.91.116.8:4045 , piece_id=6NHEH7ZOYADN6WLEHKVC4NVAS6J5NGUQGOYWO6S7KXEDUED6RACA, seg_pos={0 0}, upload=438898]: Done uploading piece. err: <nil> Dec 26 11:33:31.698403993 (500h4m31.569487028s) [noise=true , address=145.79.227.44:4194 , piece_id=3OISDZXJ4I5C73TFZCAKEPQW6JTNCRA5OTJLUTN2QQVPGR2JQRGQ, seg_pos={0 0}, upload=438898]: Done uploading piece. err: <nil> Dec 26 11:33:31.698424461 (500h4m31.569507506s) [seg_pos={0 0}, upload=438898]: Segment reached optimal threshold of 13 pieces. Dec 26 11:33:31.698816228 (500h4m31.569899273s) [noise=true , address=45.91.116.5:4299 , piece_id=QCXEG2TLNUTPC53AVZ74NB5G7VVCZR3GN7A3VTUFCQYIMJU4ZXLQ, seg_pos={0 0}, upload=438898]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: write tcp 188.166.225.170:57020->45.91.116.5:4299: use of closed network connection; write tcp 188.166.225.170:57020->45.91.116.5:4299: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57020->45.91.116.5:4299: use of closed network connection Dec 26 11:33:31.698823371 (500h4m31.569906406s) [noise=true , address=45.91.116.5:4299 , piece_id=QCXEG2TLNUTPC53AVZ74NB5G7VVCZR3GN7A3VTUFCQYIMJU4ZXLQ, seg_pos={0 0}, upload=438898]: Long tail cancellation due to optimal threshold Dec 26 11:33:31.698853148 (500h4m31.569936183s) [noise=true , address=45.91.116.10:4084 , piece_id=FAINT7GUVTE6JKVDXLDIUQHW3TFNG2C4AMJ7WHGRQ6GY7Z5G2EPA, seg_pos={0 0}, upload=438898]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: write tcp 188.166.225.170:39374->45.91.116.10:4084: use of closed network connection; write tcp 188.166.225.170:39374->45.91.116.10:4084: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39374->45.91.116.10:4084: use of closed network connection Dec 26 11:33:31.698877294 (500h4m31.569960329s) [noise=true , address=45.91.116.10:4084 , piece_id=FAINT7GUVTE6JKVDXLDIUQHW3TFNG2C4AMJ7WHGRQ6GY7Z5G2EPA, seg_pos={0 0}, upload=438898]: Long tail cancellation due to optimal threshold Dec 26 11:33:31.699107853 (500h4m31.570190888s) [noise=true , address=77.90.31.10:4519 , piece_id=I2AXQLV3D5GTZXXA6SZJVRFXVIICUEGFGYOBNZKP42KHW6KKRZCA, seg_pos={0 0}, upload=438898]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Dec 26 11:33:31.699115096 (500h4m31.570198131s) [noise=true , address=77.90.31.10:4519 , piece_id=I2AXQLV3D5GTZXXA6SZJVRFXVIICUEGFGYOBNZKP42KHW6KKRZCA, seg_pos={0 0}, upload=438898]: Long tail cancellation due to optimal threshold Dec 26 11:33:31.699134524 (500h4m31.570217559s) [seg_pos={0 0}, upload=438898]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 11:33:31.699223853 (500h4m31.570306888s) [upload=438898]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 11:33:31.699228854 (500h4m31.570311899s) [upload=438898]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 11:33:31.982204981 (500h4m31.853288016s) [upload=438898]: Done uploading segments... Dec 26 11:33:31.982223997 (500h4m31.853307032s) [upload=438898]: Upload finished. err: <nil> Duration: 5.939768605s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [..X.......X..X.X......XX......XXX.XX......X.X..XX.X.X...X...] duration: 6.61754537s trace: https://jaeger.dev.storj.io/trace/50561a690e9a382d FAIL (2025-12-26 11:35:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: GEKQWOSNACJU2W6AYFCDHIO7GHB674NOVZARJRTIAT3FR3LYMSJQ) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: 6NHEH7ZOYADN6WLEHKVC4NVAS6J5NGUQGOYWO6S7KXEDUED6RACA) context deadline exceeded ======LOGS====== 0h4m28.049098989s) [upload=438898]: Uploading segments... Dec 26 11:33:28.178049557 (500h4m28.049132592s) [upload=438898]: Got next segment. Inline: false Dec 26 11:33:28.178074825 (500h4m28.04915786s) [upload=438898]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 11:33:28.178079243 (500h4m28.049162278s) [upload=438898]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 11:33:28.422805907 (500h4m28.293888942s) [seg_pos={0 0}, upload=438898]: Begin upload segment... Dec 26 11:33:28.422886350 (500h4m28.293969395s) [seg_pos={0 0}, upload=438898]: Done begin upload segment. Dec 26 11:33:28.422961903 (500h4m28.294044948s) [noise=true , address=145.79.227.46:4603 , piece_id=2CF7OVADT5JQSUFVCKNIMPD6YUCHVYQ3F2L5OYHVSM2Q735L46CQ, seg_pos={0 0}, upload=438898]: Uploading piece... Dec 26 11:33:28.423011248 (500h4m28.294094293s) [noise=true , address=145.79.227.51:4410 , piece_id=4LO4TXF75PW4O3SLXD2U5HABUR5GT5Y6BPWNIMNRLDPT2PTY3LVQ, seg_pos={0 0}, upload=438898]: Uploading piece... Dec 26 11:33:28.423619065 (500h4m28.2947021s) [noise=true , address=145.79.227.44:4194 , piece_id=3OISDZXJ4I5C73TFZCAKEPQW6JTNCRA5OTJLUTN2QQVPGR2JQRGQ, seg_pos={0 0}, upload=438898]: Uploading piece... Dec 26 11:33:28.423948102 (500h4m28.295031137s) [noise=true , address=45.91.116.14:4192 , piece_id=GEKQWOSNACJU2W6AYFCDHIO7GHB674NOVZARJRTIAT3FR3LYMSJQ, seg_pos={0 0}, upload=438898]: Uploading piece... Dec 26 11:33:28.424233195 (500h4m28.29531623s) [noise=true , address=145.79.227.16:4912 , piece_id=T7E5PDUZHRO3WRKDHLSYLWQLSTOVQPE3NKX7LISG5S6RHUFDM36Q, seg_pos={0 0}, upload=438898]: Uploading piece... Dec 26 11:33:28.424520181 (500h4m28.295603216s) [noise=true , address=45.91.116.5:4299 , piece_id=QCXEG2TLNUTPC53AVZ74NB5G7VVCZR3GN7A3VTUFCQYIMJU4ZXLQ, seg_pos={0 0}, upload=438898]: Uploading piece... Dec 26 11:33:28.424807708 (500h4m28.295890753s) [noise=true , address=145.79.227.49:4339 , piece_id=ZMV6JF3W5S55F5OO2K6Z4SNBNOIULEBEG6RACJ526NIPVBVQ5PMA, seg_pos={0 0}, upload=438898]: Uploading piece... Dec 26 11:33:28.425093333 (500h4m28.296176368s) [noise=true , address=45.91.116.12:4225 , piece_id=OUFJURNDHBRTMORDTCDYP5V6NPGZEH3FS2HMIVDNKB25CTJMN47Q, seg_pos={0 0}, upload=438898]: Uploading piece... Dec 26 11:33:28.425412991 (500h4m28.296496036s) [noise=true , address=45.91.116.11:4120 , piece_id=UK7U3NV5HV2BVVUDT7MGROYATPLGM2B2FCUR5WB4IFXIOYWWWRFA, seg_pos={0 0}, upload=438898]: Uploading piece... Dec 26 11:33:28.426729629 (500h4m28.297812664s) [noise=true , address=145.79.227.29:4943 , piece_id=NALMWEB5P65553GXA7JYGN254ZQCZSGWDILZPYBQQBJOI2Q2CFVA, seg_pos={0 0}, upload=438898]: Uploading piece... Dec 26 11:33:28.427120402 (500h4m28.298203448s) [noise=true , address=45.91.116.8:4045 , piece_id=6NHEH7ZOYADN6WLEHKVC4NVAS6J5NGUQGOYWO6S7KXEDUED6RACA, seg_pos={0 0}, upload=438898]: Uploading piece... Dec 26 11:33:28.428470443 (500h4m28.299553478s) [noise=true , address=45.91.116.10:4084 , piece_id=FAINT7GUVTE6JKVDXLDIUQHW3TFNG2C4AMJ7WHGRQ6GY7Z5G2EPA, seg_pos={0 0}, upload=438898]: Uploading piece... Dec 26 11:33:28.429781790 (500h4m28.300864825s) [noise=true , address=77.90.31.10:4519 , piece_id=I2AXQLV3D5GTZXXA6SZJVRFXVIICUEGFGYOBNZKP42KHW6KKRZCA, seg_pos={0 0}, upload=438898]: Uploading piece... Dec 26 11:33:28.431046629 (500h4m28.302129664s) [noise=true , address=145.79.227.50:4374 , piece_id=4454XFKBWSOPOKYMCCLMDJDW52Z2QH7FG6QQ4DVVYIAZ5NVUNO5Q, seg_pos={0 0}, upload=438898]: Uploading piece... Dec 26 11:33:28.432307870 (500h4m28.303390905s) [noise=true , address=77.90.31.11:4556 , piece_id=YBOZQXNWZ4D73XLYY4UK25HJIHKND3OYNNAJ33O35T4VZUED7YXA, seg_pos={0 0}, upload=438898]: Uploading piece... Dec 26 11:33:28.433521131 (500h4m28.304604166s) [noise=true , address=145.79.227.38:4979 , piece_id=ASE2VKWERDIBIYWWKKDYIRY337XXM2IDZN7LUF4RQGZM4N3BJSHA, seg_pos={0 0}, upload=438898]: Uploading piece... Dec 26 11:33:30.148590360 (500h4m30.019673405s) [upload=438898]: Next returned nil segment Dec 26 11:33:30.148613774 (500h4m30.019696819s) [upload=438898]: Waiting for error group managing segments... Dec 26 11:33:30.426305860 (500h4m30.297388895s) [noise=true , address=145.79.227.51:4410 , piece_id=4LO4TXF75PW4O3SLXD2U5HABUR5GT5Y6BPWNIMNRLDPT2PTY3LVQ, seg_pos={0 0}, upload=438898]: Done uploading piece. err: <nil> Dec 26 11:33:30.435012920 (500h4m30.306095965s) [noise=true , address=145.79.227.16:4912 , piece_id=T7E5PDUZHRO3WRKDHLSYLWQLSTOVQPE3NKX7LISG5S6RHUFDM36Q, seg_pos={0 0}, upload=438898]: Done uploading piece. err: <nil> Dec 26 11:33:30.449914575 (500h4m30.32099762s) [noise=true , address=145.79.227.50:4374 , piece_id=4454XFKBWSOPOKYMCCLMDJDW52Z2QH7FG6QQ4DVVYIAZ5NVUNO5Q, seg_pos={0 0}, upload=438898]: Done uploading piece. err: <nil> Dec 26 11:33:30.456513110 (500h4m30.327596155s) [noise=true , address=145.79.227.38:4979 , piece_id=ASE2VKWERDIBIYWWKKDYIRY337XXM2IDZN7LUF4RQGZM4N3BJSHA, seg_pos={0 0}, upload=438898]: Done uploading piece. err: <nil> Dec 26 11:33:30.457390621 (500h4m30.328473656s) [noise=true , address=145.79.227.29:4943 , piece_id=NALMWEB5P65553GXA7JYGN254ZQCZSGWDILZPYBQQBJOI2Q2CFVA, seg_pos={0 0}, upload=438898]: Done uploading piece. err: <nil> Dec 26 11:33:30.462934878 (500h4m30.334017913s) [noise=true , address=145.79.227.46:4603 , piece_id=2CF7OVADT5JQSUFVCKNIMPD6YUCHVYQ3F2L5OYHVSM2Q735L46CQ, seg_pos={0 0}, upload=438898]: Done uploading piece. err: <nil> Dec 26 11:33:30.464048539 (500h4m30.335131584s) [noise=true , address=145.79.227.49:4339 , piece_id=ZMV6JF3W5S55F5OO2K6Z4SNBNOIULEBEG6RACJ526NIPVBVQ5PMA, seg_pos={0 0}, upload=438898]: Done uploading piece. err: <nil> Dec 26 11:33:30.647928789 (500h4m30.519011824s) [noise=true , address=45.91.116.11:4120 , piece_id=UK7U3NV5HV2BVVUDT7MGROYATPLGM2B2FCUR5WB4IFXIOYWWWRFA, seg_pos={0 0}, upload=438898]: Done uploading piece. err: <nil> Dec 26 11:33:30.666692729 (500h4m30.537775774s) [noise=true , address=45.91.116.14:4192 , piece_id=GEKQWOSNACJU2W6AYFCDHIO7GHB674NOVZARJRTIAT3FR3LYMSJQ, seg_pos={0 0}, upload=438898]: Done uploading piece. err: <nil> Dec 26 11:33:30.804039442 (500h4m30.675122477s) [noise=true , address=45.91.116.12:4225 , piece_id=OUFJURNDHBRTMORDTCDYP5V6NPGZEH3FS2HMIVDNKB25CTJMN47Q, seg_pos={0 0}, upload=438898]: Done uploading piece. err: <nil> Dec 26 11:33:30.844837208 (500h4m30.715920243s) [noise=true , address=77.90.31.11:4556 , piece_id=YBOZQXNWZ4D73XLYY4UK25HJIHKND3OYNNAJ33O35T4VZUED7YXA, seg_pos={0 0}, upload=438898]: Done uploading piece. err: <nil> Dec 26 11:33:31.128170436 (500h4m30.999253471s) [noise=true , address=45.91.116.8:4045 , piece_id=6NHEH7ZOYADN6WLEHKVC4NVAS6J5NGUQGOYWO6S7KXEDUED6RACA, seg_pos={0 0}, upload=438898]: Done uploading piece. err: <nil> Dec 26 11:33:31.698403993 (500h4m31.569487028s) [noise=true , address=145.79.227.44:4194 , piece_id=3OISDZXJ4I5C73TFZCAKEPQW6JTNCRA5OTJLUTN2QQVPGR2JQRGQ, seg_pos={0 0}, upload=438898]: Done uploading piece. err: <nil> Dec 26 11:33:31.698424461 (500h4m31.569507506s) [seg_pos={0 0}, upload=438898]: Segment reached optimal threshold of 13 pieces. Dec 26 11:33:31.698816228 (500h4m31.569899273s) [noise=true , address=45.91.116.5:4299 , piece_id=QCXEG2TLNUTPC53AVZ74NB5G7VVCZR3GN7A3VTUFCQYIMJU4ZXLQ, seg_pos={0 0}, upload=438898]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: write tcp 188.166.225.170:57020->45.91.116.5:4299: use of closed network connection; write tcp 188.166.225.170:57020->45.91.116.5:4299: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57020->45.91.116.5:4299: use of closed network connection Dec 26 11:33:31.698823371 (500h4m31.569906406s) [noise=true , address=45.91.116.5:4299 , piece_id=QCXEG2TLNUTPC53AVZ74NB5G7VVCZR3GN7A3VTUFCQYIMJU4ZXLQ, seg_pos={0 0}, upload=438898]: Long tail cancellation due to optimal threshold Dec 26 11:33:31.698853148 (500h4m31.569936183s) [noise=true , address=45.91.116.10:4084 , piece_id=FAINT7GUVTE6JKVDXLDIUQHW3TFNG2C4AMJ7WHGRQ6GY7Z5G2EPA, seg_pos={0 0}, upload=438898]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: write tcp 188.166.225.170:39374->45.91.116.10:4084: use of closed network connection; write tcp 188.166.225.170:39374->45.91.116.10:4084: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39374->45.91.116.10:4084: use of closed network connection Dec 26 11:33:31.698877294 (500h4m31.569960329s) [noise=true , address=45.91.116.10:4084 , piece_id=FAINT7GUVTE6JKVDXLDIUQHW3TFNG2C4AMJ7WHGRQ6GY7Z5G2EPA, seg_pos={0 0}, upload=438898]: Long tail cancellation due to optimal threshold Dec 26 11:33:31.699107853 (500h4m31.570190888s) [noise=true , address=77.90.31.10:4519 , piece_id=I2AXQLV3D5GTZXXA6SZJVRFXVIICUEGFGYOBNZKP42KHW6KKRZCA, seg_pos={0 0}, upload=438898]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Dec 26 11:33:31.699115096 (500h4m31.570198131s) [noise=true , address=77.90.31.10:4519 , piece_id=I2AXQLV3D5GTZXXA6SZJVRFXVIICUEGFGYOBNZKP42KHW6KKRZCA, seg_pos={0 0}, upload=438898]: Long tail cancellation due to optimal threshold Dec 26 11:33:31.699134524 (500h4m31.570217559s) [seg_pos={0 0}, upload=438898]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 11:33:31.699223853 (500h4m31.570306888s) [upload=438898]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 11:33:31.699228854 (500h4m31.570311899s) [upload=438898]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 11:33:31.982204981 (500h4m31.853288016s) [upload=438898]: Done uploading segments... Dec 26 11:33:31.982223997 (500h4m31.853307032s) [upload=438898]: Upload finished. err: <nil> Duration: 6.61757139s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 42.441399737s trace: https://jaeger.dev.storj.io/trace/785a90e8be84a595 FAIL (2025-12-26 11:35:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== KXVRQQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.834123964 (499h15m56.705207009s) [noise=true , address=45.91.116.4:4963 , piece_id=U4T64KQ67C3BDKG5TJSTQWUS4WWYCSVPUZOWJ6YANR2IQ2JD4EFQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.834585043 (499h15m56.705668078s) [noise=true , address=145.79.227.9:4390 , piece_id=JDQNHNGABBGANIHKSKP4VHUWIPQBNNEOPC4ZQCKR3K5TTR4N4SGQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.834961800 (499h15m56.706044845s) [noise=true , address=145.79.227.30:4778 , piece_id=BMEAUWANP7FP6CFRULMIAZUV4JNQEXOUN2EAXJY5XOWLDZINWOIA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.835331334 (499h15m56.706414379s) [noise=true , address=77.90.31.18:4838 , piece_id=VYBIKY3MLEE2CXGS2MT23HX5HUFK25FKYKI4XLXPDRNWCC55ANXQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.835738399 (499h15m56.706821434s) [noise=true , address=145.79.227.20:4459 , piece_id=2EUW2AH7FBVZNJ27XVLRXKMXA737DWKAYBL7CR7PRXXRTO2M644A, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.836118653 (499h15m56.707201698s) [noise=true , address=77.90.31.14:4678 , piece_id=XQHQZJUPB24G3BWGFTFDD42NM6K5HJVHUXBXWH5TW7BU25QGHTMA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.836501923 (499h15m56.707584968s) [noise=true , address=45.91.116.6:4988 , piece_id=VPEKJF74D7SQWKR47MAIY5UJQPBJIEU5AQDKLFGATYIRHFREAQ5A, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.836904940 (499h15m56.707987985s) [noise=true , address=64.49.13.169:4711 , piece_id=PVBVY4TVJLPKYBP6A7LBNZADLN25XBDWZFEK2QVN45YHJATIR7TA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.837388641 (499h15m56.708471676s) [noise=true , address=145.79.227.40:4076 , piece_id=Y6WKUYNPDMBUZSNWEYDL2KKNR6WKHSKGP63S6KAG6F2MWMNTVMBQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.837910455 (499h15m56.7089935s) [noise=true , address=77.90.31.11:4571 , piece_id=OERZACIKIG7HBEV3LL42KUAGKN6AWZEXNQSA52FIYTUKDFWCBW7Q, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.838146514 (499h15m56.709229559s) [noise=true , address=64.49.13.26:4945 , piece_id=KXCFTLWR4HSBTP4GJJ4HEPA6BBEMILYTJEMXY4S5WB24HRMQJOKA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.838469329 (499h15m56.709552364s) [noise=true , address=45.91.116.5:4301 , piece_id=E3ZRKZUKM335ARHV3Z5YGLKQYHG6QV4GCCNHAVMN34BM7UQFWIIA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.838690831 (499h15m56.709773866s) [noise=true , address=45.91.116.14:4204 , piece_id=JWL7OOMKTVAP7WEUB5WTL3F6F5IRNID4KIOYJSYZ655PJSNXAQKQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.839143724 (499h15m56.710226759s) [noise=true , address=45.91.116.9:4272 , piece_id=GGFDVKEW5LUCHQQUP6CWJHJQXUETYUG2UISPVP6XQJGSAJW2KP2Q, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.839678982 (499h15m56.710762027s) [noise=true , address=145.79.227.44:4208 , piece_id=ZLIX7XAUMKRUEWWTQST2KLCOWFJSAFSAQNLUCTTOWVZUBESEI23A, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:58.136536405 (499h15m58.00761944s) [upload=438194]: Next returned nil segment Dec 26 10:44:58.136548628 (499h15m58.007631673s) [upload=438194]: Waiting for error group managing segments... Dec 26 10:44:58.469618858 (499h15m58.340701893s) [noise=true , address=64.49.13.26:4945 , piece_id=KXCFTLWR4HSBTP4GJJ4HEPA6BBEMILYTJEMXY4S5WB24HRMQJOKA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.708819055 (499h15m58.5799021s) [noise=true , address=145.79.227.9:4390 , piece_id=JDQNHNGABBGANIHKSKP4VHUWIPQBNNEOPC4ZQCKR3K5TTR4N4SGQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.714466429 (499h15m58.585549474s) [noise=true , address=145.79.227.40:4076 , piece_id=Y6WKUYNPDMBUZSNWEYDL2KKNR6WKHSKGP63S6KAG6F2MWMNTVMBQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.745407696 (499h15m58.616490731s) [noise=true , address=145.79.227.20:4459 , piece_id=2EUW2AH7FBVZNJ27XVLRXKMXA737DWKAYBL7CR7PRXXRTO2M644A, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.888701070 (499h15m58.759784105s) [noise=true , address=77.90.31.18:4838 , piece_id=VYBIKY3MLEE2CXGS2MT23HX5HUFK25FKYKI4XLXPDRNWCC55ANXQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.948150873 (499h15m58.819233918s) [noise=true , address=45.91.116.5:4301 , piece_id=E3ZRKZUKM335ARHV3Z5YGLKQYHG6QV4GCCNHAVMN34BM7UQFWIIA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.971389744 (499h15m58.842472779s) [noise=true , address=77.90.31.19:4868 , piece_id=WH77PZSXC2VUWLQYLBR2NDEVT6KMD3VZK5WAL53AMRDW74P242LQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.021685314 (499h15m58.892768349s) [noise=true , address=45.91.116.4:4963 , piece_id=U4T64KQ67C3BDKG5TJSTQWUS4WWYCSVPUZOWJ6YANR2IQ2JD4EFQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.045418216 (499h15m58.916501251s) [noise=true , address=45.91.116.14:4204 , piece_id=JWL7OOMKTVAP7WEUB5WTL3F6F5IRNID4KIOYJSYZ655PJSNXAQKQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.056260371 (499h15m58.927343416s) [noise=true , address=45.91.116.9:4272 , piece_id=GGFDVKEW5LUCHQQUP6CWJHJQXUETYUG2UISPVP6XQJGSAJW2KP2Q, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.085713054 (499h15m58.956796089s) [noise=true , address=45.91.116.7:4017 , piece_id=L4GGY6AICMSFMUVYGI6ZMUORYRXNJFPLJY5XJRUAM7NVYQOHPHSQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.640980124 (499h15m59.512063169s) [noise=true , address=145.79.227.30:4778 , piece_id=BMEAUWANP7FP6CFRULMIAZUV4JNQEXOUN2EAXJY5XOWLDZINWOIA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.654515607 (499h15m59.525598652s) [noise=true , address=145.79.227.44:4208 , piece_id=ZLIX7XAUMKRUEWWTQST2KLCOWFJSAFSAQNLUCTTOWVZUBESEI23A, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.660349657 (499h15m59.531432712s) [noise=true , address=77.90.31.11:4571 , piece_id=OERZACIKIG7HBEV3LL42KUAGKN6AWZEXNQSA52FIYTUKDFWCBW7Q, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.660376087 (499h15m59.531459132s) [seg_pos={0 0}, upload=438194]: Segment reached optimal threshold of 14 pieces. Dec 26 10:44:59.660612988 (499h15m59.531696033s) [noise=true , address=64.49.13.169:4711 , piece_id=PVBVY4TVJLPKYBP6A7LBNZADLN25XBDWZFEK2QVN45YHJATIR7TA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:52964->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:52964->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52964->64.49.13.169:4711: use of closed network connection Dec 26 10:44:59.660620753 (499h15m59.531703788s) [noise=true , address=64.49.13.169:4711 , piece_id=PVBVY4TVJLPKYBP6A7LBNZADLN25XBDWZFEK2QVN45YHJATIR7TA, seg_pos={0 0}, upload=438194]: Long tail cancellation due to optimal threshold Dec 26 10:44:59.660893663 (499h15m59.531976698s) [noise=true , address=45.91.116.6:4988 , piece_id=VPEKJF74D7SQWKR47MAIY5UJQPBJIEU5AQDKLFGATYIRHFREAQ5A, seg_pos={0 0}, upload=438194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TrcoCHhAJeEykP9hJNpygoPSdgFV5v6GzL48JpyNPiAj5ZJRS): protocol: expected piece hash; context canceled; EOF Dec 26 10:44:59.660901057 (499h15m59.531984102s) [noise=true , address=45.91.116.6:4988 , piece_id=VPEKJF74D7SQWKR47MAIY5UJQPBJIEU5AQDKLFGATYIRHFREAQ5A, seg_pos={0 0}, upload=438194]: Long tail cancellation due to optimal threshold Dec 26 10:44:59.661226155 (499h15m59.5323092s) [noise=true , address=77.90.31.14:4678 , piece_id=XQHQZJUPB24G3BWGFTFDD42NM6K5HJVHUXBXWH5TW7BU25QGHTMA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12beu3SUZMPNWfbeHV4tLtx9UVZoEHh4Kt7oBhT59w3bGBE1ovP): protocol: write tcp 188.166.225.170:35884->77.90.31.14:4678: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 10:44:59.661249630 (499h15m59.532332685s) [noise=true , address=77.90.31.14:4678 , piece_id=XQHQZJUPB24G3BWGFTFDD42NM6K5HJVHUXBXWH5TW7BU25QGHTMA, seg_pos={0 0}, upload=438194]: Long tail cancellation due to optimal threshold Dec 26 10:44:59.661309705 (499h15m59.53239275s) [noise=true , address=77.90.31.21:4909 , piece_id=H6KT2AKZISCU4VEA6TXJJ3EFGCRBZPPXBEJJEIHZR3GEJZKXVRQQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121RT7tP6czMFAiZ3DjzSSsYgPt3xZXVbXVpkcYxaZYscNiMjL6): protocol: expected piece hash; context canceled; EOF Dec 26 10:44:59.661318962 (499h15m59.532401997s) [noise=true , address=77.90.31.21:4909 , piece_id=H6KT2AKZISCU4VEA6TXJJ3EFGCRBZPPXBEJJEIHZR3GEJZKXVRQQ, seg_pos={0 0}, upload=438194]: Long tail cancellation due to optimal threshold Dec 26 10:44:59.661345674 (499h15m59.532428719s) [seg_pos={0 0}, upload=438194]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 10:44:59.661485770 (499h15m59.532568815s) [upload=438194]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 10:44:59.661493655 (499h15m59.53257669s) [upload=438194]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 10:44:59.968467153 (499h15m59.839550198s) [upload=438194]: Done uploading segments... Dec 26 10:44:59.968484155 (499h15m59.83956719s) [upload=438194]: Upload finished. err: <nil> Duration: 42.441413603s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [X..X.......X..X.X......XX......XXX.XX......X.X..XX.X.X...X..] duration: 5.966243899s trace: https://jaeger.dev.storj.io/trace/50561a690e9a382d FAIL (2025-12-26 11:35:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: GEKQWOSNACJU2W6AYFCDHIO7GHB674NOVZARJRTIAT3FR3LYMSJQ) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: 6NHEH7ZOYADN6WLEHKVC4NVAS6J5NGUQGOYWO6S7KXEDUED6RACA) context deadline exceeded ======LOGS====== 0h4m28.049098989s) [upload=438898]: Uploading segments... Dec 26 11:33:28.178049557 (500h4m28.049132592s) [upload=438898]: Got next segment. Inline: false Dec 26 11:33:28.178074825 (500h4m28.04915786s) [upload=438898]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 11:33:28.178079243 (500h4m28.049162278s) [upload=438898]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 11:33:28.422805907 (500h4m28.293888942s) [seg_pos={0 0}, upload=438898]: Begin upload segment... Dec 26 11:33:28.422886350 (500h4m28.293969395s) [seg_pos={0 0}, upload=438898]: Done begin upload segment. Dec 26 11:33:28.422961903 (500h4m28.294044948s) [noise=true , address=145.79.227.46:4603 , piece_id=2CF7OVADT5JQSUFVCKNIMPD6YUCHVYQ3F2L5OYHVSM2Q735L46CQ, seg_pos={0 0}, upload=438898]: Uploading piece... Dec 26 11:33:28.423011248 (500h4m28.294094293s) [noise=true , address=145.79.227.51:4410 , piece_id=4LO4TXF75PW4O3SLXD2U5HABUR5GT5Y6BPWNIMNRLDPT2PTY3LVQ, seg_pos={0 0}, upload=438898]: Uploading piece... Dec 26 11:33:28.423619065 (500h4m28.2947021s) [noise=true , address=145.79.227.44:4194 , piece_id=3OISDZXJ4I5C73TFZCAKEPQW6JTNCRA5OTJLUTN2QQVPGR2JQRGQ, seg_pos={0 0}, upload=438898]: Uploading piece... Dec 26 11:33:28.423948102 (500h4m28.295031137s) [noise=true , address=45.91.116.14:4192 , piece_id=GEKQWOSNACJU2W6AYFCDHIO7GHB674NOVZARJRTIAT3FR3LYMSJQ, seg_pos={0 0}, upload=438898]: Uploading piece... Dec 26 11:33:28.424233195 (500h4m28.29531623s) [noise=true , address=145.79.227.16:4912 , piece_id=T7E5PDUZHRO3WRKDHLSYLWQLSTOVQPE3NKX7LISG5S6RHUFDM36Q, seg_pos={0 0}, upload=438898]: Uploading piece... Dec 26 11:33:28.424520181 (500h4m28.295603216s) [noise=true , address=45.91.116.5:4299 , piece_id=QCXEG2TLNUTPC53AVZ74NB5G7VVCZR3GN7A3VTUFCQYIMJU4ZXLQ, seg_pos={0 0}, upload=438898]: Uploading piece... Dec 26 11:33:28.424807708 (500h4m28.295890753s) [noise=true , address=145.79.227.49:4339 , piece_id=ZMV6JF3W5S55F5OO2K6Z4SNBNOIULEBEG6RACJ526NIPVBVQ5PMA, seg_pos={0 0}, upload=438898]: Uploading piece... Dec 26 11:33:28.425093333 (500h4m28.296176368s) [noise=true , address=45.91.116.12:4225 , piece_id=OUFJURNDHBRTMORDTCDYP5V6NPGZEH3FS2HMIVDNKB25CTJMN47Q, seg_pos={0 0}, upload=438898]: Uploading piece... Dec 26 11:33:28.425412991 (500h4m28.296496036s) [noise=true , address=45.91.116.11:4120 , piece_id=UK7U3NV5HV2BVVUDT7MGROYATPLGM2B2FCUR5WB4IFXIOYWWWRFA, seg_pos={0 0}, upload=438898]: Uploading piece... Dec 26 11:33:28.426729629 (500h4m28.297812664s) [noise=true , address=145.79.227.29:4943 , piece_id=NALMWEB5P65553GXA7JYGN254ZQCZSGWDILZPYBQQBJOI2Q2CFVA, seg_pos={0 0}, upload=438898]: Uploading piece... Dec 26 11:33:28.427120402 (500h4m28.298203448s) [noise=true , address=45.91.116.8:4045 , piece_id=6NHEH7ZOYADN6WLEHKVC4NVAS6J5NGUQGOYWO6S7KXEDUED6RACA, seg_pos={0 0}, upload=438898]: Uploading piece... Dec 26 11:33:28.428470443 (500h4m28.299553478s) [noise=true , address=45.91.116.10:4084 , piece_id=FAINT7GUVTE6JKVDXLDIUQHW3TFNG2C4AMJ7WHGRQ6GY7Z5G2EPA, seg_pos={0 0}, upload=438898]: Uploading piece... Dec 26 11:33:28.429781790 (500h4m28.300864825s) [noise=true , address=77.90.31.10:4519 , piece_id=I2AXQLV3D5GTZXXA6SZJVRFXVIICUEGFGYOBNZKP42KHW6KKRZCA, seg_pos={0 0}, upload=438898]: Uploading piece... Dec 26 11:33:28.431046629 (500h4m28.302129664s) [noise=true , address=145.79.227.50:4374 , piece_id=4454XFKBWSOPOKYMCCLMDJDW52Z2QH7FG6QQ4DVVYIAZ5NVUNO5Q, seg_pos={0 0}, upload=438898]: Uploading piece... Dec 26 11:33:28.432307870 (500h4m28.303390905s) [noise=true , address=77.90.31.11:4556 , piece_id=YBOZQXNWZ4D73XLYY4UK25HJIHKND3OYNNAJ33O35T4VZUED7YXA, seg_pos={0 0}, upload=438898]: Uploading piece... Dec 26 11:33:28.433521131 (500h4m28.304604166s) [noise=true , address=145.79.227.38:4979 , piece_id=ASE2VKWERDIBIYWWKKDYIRY337XXM2IDZN7LUF4RQGZM4N3BJSHA, seg_pos={0 0}, upload=438898]: Uploading piece... Dec 26 11:33:30.148590360 (500h4m30.019673405s) [upload=438898]: Next returned nil segment Dec 26 11:33:30.148613774 (500h4m30.019696819s) [upload=438898]: Waiting for error group managing segments... Dec 26 11:33:30.426305860 (500h4m30.297388895s) [noise=true , address=145.79.227.51:4410 , piece_id=4LO4TXF75PW4O3SLXD2U5HABUR5GT5Y6BPWNIMNRLDPT2PTY3LVQ, seg_pos={0 0}, upload=438898]: Done uploading piece. err: <nil> Dec 26 11:33:30.435012920 (500h4m30.306095965s) [noise=true , address=145.79.227.16:4912 , piece_id=T7E5PDUZHRO3WRKDHLSYLWQLSTOVQPE3NKX7LISG5S6RHUFDM36Q, seg_pos={0 0}, upload=438898]: Done uploading piece. err: <nil> Dec 26 11:33:30.449914575 (500h4m30.32099762s) [noise=true , address=145.79.227.50:4374 , piece_id=4454XFKBWSOPOKYMCCLMDJDW52Z2QH7FG6QQ4DVVYIAZ5NVUNO5Q, seg_pos={0 0}, upload=438898]: Done uploading piece. err: <nil> Dec 26 11:33:30.456513110 (500h4m30.327596155s) [noise=true , address=145.79.227.38:4979 , piece_id=ASE2VKWERDIBIYWWKKDYIRY337XXM2IDZN7LUF4RQGZM4N3BJSHA, seg_pos={0 0}, upload=438898]: Done uploading piece. err: <nil> Dec 26 11:33:30.457390621 (500h4m30.328473656s) [noise=true , address=145.79.227.29:4943 , piece_id=NALMWEB5P65553GXA7JYGN254ZQCZSGWDILZPYBQQBJOI2Q2CFVA, seg_pos={0 0}, upload=438898]: Done uploading piece. err: <nil> Dec 26 11:33:30.462934878 (500h4m30.334017913s) [noise=true , address=145.79.227.46:4603 , piece_id=2CF7OVADT5JQSUFVCKNIMPD6YUCHVYQ3F2L5OYHVSM2Q735L46CQ, seg_pos={0 0}, upload=438898]: Done uploading piece. err: <nil> Dec 26 11:33:30.464048539 (500h4m30.335131584s) [noise=true , address=145.79.227.49:4339 , piece_id=ZMV6JF3W5S55F5OO2K6Z4SNBNOIULEBEG6RACJ526NIPVBVQ5PMA, seg_pos={0 0}, upload=438898]: Done uploading piece. err: <nil> Dec 26 11:33:30.647928789 (500h4m30.519011824s) [noise=true , address=45.91.116.11:4120 , piece_id=UK7U3NV5HV2BVVUDT7MGROYATPLGM2B2FCUR5WB4IFXIOYWWWRFA, seg_pos={0 0}, upload=438898]: Done uploading piece. err: <nil> Dec 26 11:33:30.666692729 (500h4m30.537775774s) [noise=true , address=45.91.116.14:4192 , piece_id=GEKQWOSNACJU2W6AYFCDHIO7GHB674NOVZARJRTIAT3FR3LYMSJQ, seg_pos={0 0}, upload=438898]: Done uploading piece. err: <nil> Dec 26 11:33:30.804039442 (500h4m30.675122477s) [noise=true , address=45.91.116.12:4225 , piece_id=OUFJURNDHBRTMORDTCDYP5V6NPGZEH3FS2HMIVDNKB25CTJMN47Q, seg_pos={0 0}, upload=438898]: Done uploading piece. err: <nil> Dec 26 11:33:30.844837208 (500h4m30.715920243s) [noise=true , address=77.90.31.11:4556 , piece_id=YBOZQXNWZ4D73XLYY4UK25HJIHKND3OYNNAJ33O35T4VZUED7YXA, seg_pos={0 0}, upload=438898]: Done uploading piece. err: <nil> Dec 26 11:33:31.128170436 (500h4m30.999253471s) [noise=true , address=45.91.116.8:4045 , piece_id=6NHEH7ZOYADN6WLEHKVC4NVAS6J5NGUQGOYWO6S7KXEDUED6RACA, seg_pos={0 0}, upload=438898]: Done uploading piece. err: <nil> Dec 26 11:33:31.698403993 (500h4m31.569487028s) [noise=true , address=145.79.227.44:4194 , piece_id=3OISDZXJ4I5C73TFZCAKEPQW6JTNCRA5OTJLUTN2QQVPGR2JQRGQ, seg_pos={0 0}, upload=438898]: Done uploading piece. err: <nil> Dec 26 11:33:31.698424461 (500h4m31.569507506s) [seg_pos={0 0}, upload=438898]: Segment reached optimal threshold of 13 pieces. Dec 26 11:33:31.698816228 (500h4m31.569899273s) [noise=true , address=45.91.116.5:4299 , piece_id=QCXEG2TLNUTPC53AVZ74NB5G7VVCZR3GN7A3VTUFCQYIMJU4ZXLQ, seg_pos={0 0}, upload=438898]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: write tcp 188.166.225.170:57020->45.91.116.5:4299: use of closed network connection; write tcp 188.166.225.170:57020->45.91.116.5:4299: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57020->45.91.116.5:4299: use of closed network connection Dec 26 11:33:31.698823371 (500h4m31.569906406s) [noise=true , address=45.91.116.5:4299 , piece_id=QCXEG2TLNUTPC53AVZ74NB5G7VVCZR3GN7A3VTUFCQYIMJU4ZXLQ, seg_pos={0 0}, upload=438898]: Long tail cancellation due to optimal threshold Dec 26 11:33:31.698853148 (500h4m31.569936183s) [noise=true , address=45.91.116.10:4084 , piece_id=FAINT7GUVTE6JKVDXLDIUQHW3TFNG2C4AMJ7WHGRQ6GY7Z5G2EPA, seg_pos={0 0}, upload=438898]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: write tcp 188.166.225.170:39374->45.91.116.10:4084: use of closed network connection; write tcp 188.166.225.170:39374->45.91.116.10:4084: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39374->45.91.116.10:4084: use of closed network connection Dec 26 11:33:31.698877294 (500h4m31.569960329s) [noise=true , address=45.91.116.10:4084 , piece_id=FAINT7GUVTE6JKVDXLDIUQHW3TFNG2C4AMJ7WHGRQ6GY7Z5G2EPA, seg_pos={0 0}, upload=438898]: Long tail cancellation due to optimal threshold Dec 26 11:33:31.699107853 (500h4m31.570190888s) [noise=true , address=77.90.31.10:4519 , piece_id=I2AXQLV3D5GTZXXA6SZJVRFXVIICUEGFGYOBNZKP42KHW6KKRZCA, seg_pos={0 0}, upload=438898]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Dec 26 11:33:31.699115096 (500h4m31.570198131s) [noise=true , address=77.90.31.10:4519 , piece_id=I2AXQLV3D5GTZXXA6SZJVRFXVIICUEGFGYOBNZKP42KHW6KKRZCA, seg_pos={0 0}, upload=438898]: Long tail cancellation due to optimal threshold Dec 26 11:33:31.699134524 (500h4m31.570217559s) [seg_pos={0 0}, upload=438898]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 11:33:31.699223853 (500h4m31.570306888s) [upload=438898]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 11:33:31.699228854 (500h4m31.570311899s) [upload=438898]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 11:33:31.982204981 (500h4m31.853288016s) [upload=438898]: Done uploading segments... Dec 26 11:33:31.982223997 (500h4m31.853307032s) [upload=438898]: Upload finished. err: <nil> Duration: 5.966338699s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 59.696140205s trace: https://jaeger.dev.storj.io/trace/785a90e8be84a595 FAIL (2025-12-26 11:34:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== KXVRQQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.834123964 (499h15m56.705207009s) [noise=true , address=45.91.116.4:4963 , piece_id=U4T64KQ67C3BDKG5TJSTQWUS4WWYCSVPUZOWJ6YANR2IQ2JD4EFQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.834585043 (499h15m56.705668078s) [noise=true , address=145.79.227.9:4390 , piece_id=JDQNHNGABBGANIHKSKP4VHUWIPQBNNEOPC4ZQCKR3K5TTR4N4SGQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.834961800 (499h15m56.706044845s) [noise=true , address=145.79.227.30:4778 , piece_id=BMEAUWANP7FP6CFRULMIAZUV4JNQEXOUN2EAXJY5XOWLDZINWOIA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.835331334 (499h15m56.706414379s) [noise=true , address=77.90.31.18:4838 , piece_id=VYBIKY3MLEE2CXGS2MT23HX5HUFK25FKYKI4XLXPDRNWCC55ANXQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.835738399 (499h15m56.706821434s) [noise=true , address=145.79.227.20:4459 , piece_id=2EUW2AH7FBVZNJ27XVLRXKMXA737DWKAYBL7CR7PRXXRTO2M644A, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.836118653 (499h15m56.707201698s) [noise=true , address=77.90.31.14:4678 , piece_id=XQHQZJUPB24G3BWGFTFDD42NM6K5HJVHUXBXWH5TW7BU25QGHTMA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.836501923 (499h15m56.707584968s) [noise=true , address=45.91.116.6:4988 , piece_id=VPEKJF74D7SQWKR47MAIY5UJQPBJIEU5AQDKLFGATYIRHFREAQ5A, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.836904940 (499h15m56.707987985s) [noise=true , address=64.49.13.169:4711 , piece_id=PVBVY4TVJLPKYBP6A7LBNZADLN25XBDWZFEK2QVN45YHJATIR7TA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.837388641 (499h15m56.708471676s) [noise=true , address=145.79.227.40:4076 , piece_id=Y6WKUYNPDMBUZSNWEYDL2KKNR6WKHSKGP63S6KAG6F2MWMNTVMBQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.837910455 (499h15m56.7089935s) [noise=true , address=77.90.31.11:4571 , piece_id=OERZACIKIG7HBEV3LL42KUAGKN6AWZEXNQSA52FIYTUKDFWCBW7Q, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.838146514 (499h15m56.709229559s) [noise=true , address=64.49.13.26:4945 , piece_id=KXCFTLWR4HSBTP4GJJ4HEPA6BBEMILYTJEMXY4S5WB24HRMQJOKA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.838469329 (499h15m56.709552364s) [noise=true , address=45.91.116.5:4301 , piece_id=E3ZRKZUKM335ARHV3Z5YGLKQYHG6QV4GCCNHAVMN34BM7UQFWIIA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.838690831 (499h15m56.709773866s) [noise=true , address=45.91.116.14:4204 , piece_id=JWL7OOMKTVAP7WEUB5WTL3F6F5IRNID4KIOYJSYZ655PJSNXAQKQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.839143724 (499h15m56.710226759s) [noise=true , address=45.91.116.9:4272 , piece_id=GGFDVKEW5LUCHQQUP6CWJHJQXUETYUG2UISPVP6XQJGSAJW2KP2Q, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.839678982 (499h15m56.710762027s) [noise=true , address=145.79.227.44:4208 , piece_id=ZLIX7XAUMKRUEWWTQST2KLCOWFJSAFSAQNLUCTTOWVZUBESEI23A, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:58.136536405 (499h15m58.00761944s) [upload=438194]: Next returned nil segment Dec 26 10:44:58.136548628 (499h15m58.007631673s) [upload=438194]: Waiting for error group managing segments... Dec 26 10:44:58.469618858 (499h15m58.340701893s) [noise=true , address=64.49.13.26:4945 , piece_id=KXCFTLWR4HSBTP4GJJ4HEPA6BBEMILYTJEMXY4S5WB24HRMQJOKA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.708819055 (499h15m58.5799021s) [noise=true , address=145.79.227.9:4390 , piece_id=JDQNHNGABBGANIHKSKP4VHUWIPQBNNEOPC4ZQCKR3K5TTR4N4SGQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.714466429 (499h15m58.585549474s) [noise=true , address=145.79.227.40:4076 , piece_id=Y6WKUYNPDMBUZSNWEYDL2KKNR6WKHSKGP63S6KAG6F2MWMNTVMBQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.745407696 (499h15m58.616490731s) [noise=true , address=145.79.227.20:4459 , piece_id=2EUW2AH7FBVZNJ27XVLRXKMXA737DWKAYBL7CR7PRXXRTO2M644A, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.888701070 (499h15m58.759784105s) [noise=true , address=77.90.31.18:4838 , piece_id=VYBIKY3MLEE2CXGS2MT23HX5HUFK25FKYKI4XLXPDRNWCC55ANXQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.948150873 (499h15m58.819233918s) [noise=true , address=45.91.116.5:4301 , piece_id=E3ZRKZUKM335ARHV3Z5YGLKQYHG6QV4GCCNHAVMN34BM7UQFWIIA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.971389744 (499h15m58.842472779s) [noise=true , address=77.90.31.19:4868 , piece_id=WH77PZSXC2VUWLQYLBR2NDEVT6KMD3VZK5WAL53AMRDW74P242LQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.021685314 (499h15m58.892768349s) [noise=true , address=45.91.116.4:4963 , piece_id=U4T64KQ67C3BDKG5TJSTQWUS4WWYCSVPUZOWJ6YANR2IQ2JD4EFQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.045418216 (499h15m58.916501251s) [noise=true , address=45.91.116.14:4204 , piece_id=JWL7OOMKTVAP7WEUB5WTL3F6F5IRNID4KIOYJSYZ655PJSNXAQKQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.056260371 (499h15m58.927343416s) [noise=true , address=45.91.116.9:4272 , piece_id=GGFDVKEW5LUCHQQUP6CWJHJQXUETYUG2UISPVP6XQJGSAJW2KP2Q, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.085713054 (499h15m58.956796089s) [noise=true , address=45.91.116.7:4017 , piece_id=L4GGY6AICMSFMUVYGI6ZMUORYRXNJFPLJY5XJRUAM7NVYQOHPHSQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.640980124 (499h15m59.512063169s) [noise=true , address=145.79.227.30:4778 , piece_id=BMEAUWANP7FP6CFRULMIAZUV4JNQEXOUN2EAXJY5XOWLDZINWOIA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.654515607 (499h15m59.525598652s) [noise=true , address=145.79.227.44:4208 , piece_id=ZLIX7XAUMKRUEWWTQST2KLCOWFJSAFSAQNLUCTTOWVZUBESEI23A, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.660349657 (499h15m59.531432712s) [noise=true , address=77.90.31.11:4571 , piece_id=OERZACIKIG7HBEV3LL42KUAGKN6AWZEXNQSA52FIYTUKDFWCBW7Q, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.660376087 (499h15m59.531459132s) [seg_pos={0 0}, upload=438194]: Segment reached optimal threshold of 14 pieces. Dec 26 10:44:59.660612988 (499h15m59.531696033s) [noise=true , address=64.49.13.169:4711 , piece_id=PVBVY4TVJLPKYBP6A7LBNZADLN25XBDWZFEK2QVN45YHJATIR7TA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:52964->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:52964->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52964->64.49.13.169:4711: use of closed network connection Dec 26 10:44:59.660620753 (499h15m59.531703788s) [noise=true , address=64.49.13.169:4711 , piece_id=PVBVY4TVJLPKYBP6A7LBNZADLN25XBDWZFEK2QVN45YHJATIR7TA, seg_pos={0 0}, upload=438194]: Long tail cancellation due to optimal threshold Dec 26 10:44:59.660893663 (499h15m59.531976698s) [noise=true , address=45.91.116.6:4988 , piece_id=VPEKJF74D7SQWKR47MAIY5UJQPBJIEU5AQDKLFGATYIRHFREAQ5A, seg_pos={0 0}, upload=438194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TrcoCHhAJeEykP9hJNpygoPSdgFV5v6GzL48JpyNPiAj5ZJRS): protocol: expected piece hash; context canceled; EOF Dec 26 10:44:59.660901057 (499h15m59.531984102s) [noise=true , address=45.91.116.6:4988 , piece_id=VPEKJF74D7SQWKR47MAIY5UJQPBJIEU5AQDKLFGATYIRHFREAQ5A, seg_pos={0 0}, upload=438194]: Long tail cancellation due to optimal threshold Dec 26 10:44:59.661226155 (499h15m59.5323092s) [noise=true , address=77.90.31.14:4678 , piece_id=XQHQZJUPB24G3BWGFTFDD42NM6K5HJVHUXBXWH5TW7BU25QGHTMA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12beu3SUZMPNWfbeHV4tLtx9UVZoEHh4Kt7oBhT59w3bGBE1ovP): protocol: write tcp 188.166.225.170:35884->77.90.31.14:4678: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 10:44:59.661249630 (499h15m59.532332685s) [noise=true , address=77.90.31.14:4678 , piece_id=XQHQZJUPB24G3BWGFTFDD42NM6K5HJVHUXBXWH5TW7BU25QGHTMA, seg_pos={0 0}, upload=438194]: Long tail cancellation due to optimal threshold Dec 26 10:44:59.661309705 (499h15m59.53239275s) [noise=true , address=77.90.31.21:4909 , piece_id=H6KT2AKZISCU4VEA6TXJJ3EFGCRBZPPXBEJJEIHZR3GEJZKXVRQQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121RT7tP6czMFAiZ3DjzSSsYgPt3xZXVbXVpkcYxaZYscNiMjL6): protocol: expected piece hash; context canceled; EOF Dec 26 10:44:59.661318962 (499h15m59.532401997s) [noise=true , address=77.90.31.21:4909 , piece_id=H6KT2AKZISCU4VEA6TXJJ3EFGCRBZPPXBEJJEIHZR3GEJZKXVRQQ, seg_pos={0 0}, upload=438194]: Long tail cancellation due to optimal threshold Dec 26 10:44:59.661345674 (499h15m59.532428719s) [seg_pos={0 0}, upload=438194]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 10:44:59.661485770 (499h15m59.532568815s) [upload=438194]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 10:44:59.661493655 (499h15m59.53257669s) [upload=438194]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 10:44:59.968467153 (499h15m59.839550198s) [upload=438194]: Done uploading segments... Dec 26 10:44:59.968484155 (499h15m59.83956719s) [upload=438194]: Upload finished. err: <nil> Duration: 59.696171013s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [XX..X.......X..X.X......XX......XXX.XX......X.X..XX.X.X...X.] duration: 5.973813311s trace: https://jaeger.dev.storj.io/trace/50561a690e9a382d FAIL (2025-12-26 11:34:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: GEKQWOSNACJU2W6AYFCDHIO7GHB674NOVZARJRTIAT3FR3LYMSJQ) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: 6NHEH7ZOYADN6WLEHKVC4NVAS6J5NGUQGOYWO6S7KXEDUED6RACA) context deadline exceeded ======LOGS====== 0h4m28.049098989s) [upload=438898]: Uploading segments... Dec 26 11:33:28.178049557 (500h4m28.049132592s) [upload=438898]: Got next segment. Inline: false Dec 26 11:33:28.178074825 (500h4m28.04915786s) [upload=438898]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 11:33:28.178079243 (500h4m28.049162278s) [upload=438898]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 11:33:28.422805907 (500h4m28.293888942s) [seg_pos={0 0}, upload=438898]: Begin upload segment... Dec 26 11:33:28.422886350 (500h4m28.293969395s) [seg_pos={0 0}, upload=438898]: Done begin upload segment. Dec 26 11:33:28.422961903 (500h4m28.294044948s) [noise=true , address=145.79.227.46:4603 , piece_id=2CF7OVADT5JQSUFVCKNIMPD6YUCHVYQ3F2L5OYHVSM2Q735L46CQ, seg_pos={0 0}, upload=438898]: Uploading piece... Dec 26 11:33:28.423011248 (500h4m28.294094293s) [noise=true , address=145.79.227.51:4410 , piece_id=4LO4TXF75PW4O3SLXD2U5HABUR5GT5Y6BPWNIMNRLDPT2PTY3LVQ, seg_pos={0 0}, upload=438898]: Uploading piece... Dec 26 11:33:28.423619065 (500h4m28.2947021s) [noise=true , address=145.79.227.44:4194 , piece_id=3OISDZXJ4I5C73TFZCAKEPQW6JTNCRA5OTJLUTN2QQVPGR2JQRGQ, seg_pos={0 0}, upload=438898]: Uploading piece... Dec 26 11:33:28.423948102 (500h4m28.295031137s) [noise=true , address=45.91.116.14:4192 , piece_id=GEKQWOSNACJU2W6AYFCDHIO7GHB674NOVZARJRTIAT3FR3LYMSJQ, seg_pos={0 0}, upload=438898]: Uploading piece... Dec 26 11:33:28.424233195 (500h4m28.29531623s) [noise=true , address=145.79.227.16:4912 , piece_id=T7E5PDUZHRO3WRKDHLSYLWQLSTOVQPE3NKX7LISG5S6RHUFDM36Q, seg_pos={0 0}, upload=438898]: Uploading piece... Dec 26 11:33:28.424520181 (500h4m28.295603216s) [noise=true , address=45.91.116.5:4299 , piece_id=QCXEG2TLNUTPC53AVZ74NB5G7VVCZR3GN7A3VTUFCQYIMJU4ZXLQ, seg_pos={0 0}, upload=438898]: Uploading piece... Dec 26 11:33:28.424807708 (500h4m28.295890753s) [noise=true , address=145.79.227.49:4339 , piece_id=ZMV6JF3W5S55F5OO2K6Z4SNBNOIULEBEG6RACJ526NIPVBVQ5PMA, seg_pos={0 0}, upload=438898]: Uploading piece... Dec 26 11:33:28.425093333 (500h4m28.296176368s) [noise=true , address=45.91.116.12:4225 , piece_id=OUFJURNDHBRTMORDTCDYP5V6NPGZEH3FS2HMIVDNKB25CTJMN47Q, seg_pos={0 0}, upload=438898]: Uploading piece... Dec 26 11:33:28.425412991 (500h4m28.296496036s) [noise=true , address=45.91.116.11:4120 , piece_id=UK7U3NV5HV2BVVUDT7MGROYATPLGM2B2FCUR5WB4IFXIOYWWWRFA, seg_pos={0 0}, upload=438898]: Uploading piece... Dec 26 11:33:28.426729629 (500h4m28.297812664s) [noise=true , address=145.79.227.29:4943 , piece_id=NALMWEB5P65553GXA7JYGN254ZQCZSGWDILZPYBQQBJOI2Q2CFVA, seg_pos={0 0}, upload=438898]: Uploading piece... Dec 26 11:33:28.427120402 (500h4m28.298203448s) [noise=true , address=45.91.116.8:4045 , piece_id=6NHEH7ZOYADN6WLEHKVC4NVAS6J5NGUQGOYWO6S7KXEDUED6RACA, seg_pos={0 0}, upload=438898]: Uploading piece... Dec 26 11:33:28.428470443 (500h4m28.299553478s) [noise=true , address=45.91.116.10:4084 , piece_id=FAINT7GUVTE6JKVDXLDIUQHW3TFNG2C4AMJ7WHGRQ6GY7Z5G2EPA, seg_pos={0 0}, upload=438898]: Uploading piece... Dec 26 11:33:28.429781790 (500h4m28.300864825s) [noise=true , address=77.90.31.10:4519 , piece_id=I2AXQLV3D5GTZXXA6SZJVRFXVIICUEGFGYOBNZKP42KHW6KKRZCA, seg_pos={0 0}, upload=438898]: Uploading piece... Dec 26 11:33:28.431046629 (500h4m28.302129664s) [noise=true , address=145.79.227.50:4374 , piece_id=4454XFKBWSOPOKYMCCLMDJDW52Z2QH7FG6QQ4DVVYIAZ5NVUNO5Q, seg_pos={0 0}, upload=438898]: Uploading piece... Dec 26 11:33:28.432307870 (500h4m28.303390905s) [noise=true , address=77.90.31.11:4556 , piece_id=YBOZQXNWZ4D73XLYY4UK25HJIHKND3OYNNAJ33O35T4VZUED7YXA, seg_pos={0 0}, upload=438898]: Uploading piece... Dec 26 11:33:28.433521131 (500h4m28.304604166s) [noise=true , address=145.79.227.38:4979 , piece_id=ASE2VKWERDIBIYWWKKDYIRY337XXM2IDZN7LUF4RQGZM4N3BJSHA, seg_pos={0 0}, upload=438898]: Uploading piece... Dec 26 11:33:30.148590360 (500h4m30.019673405s) [upload=438898]: Next returned nil segment Dec 26 11:33:30.148613774 (500h4m30.019696819s) [upload=438898]: Waiting for error group managing segments... Dec 26 11:33:30.426305860 (500h4m30.297388895s) [noise=true , address=145.79.227.51:4410 , piece_id=4LO4TXF75PW4O3SLXD2U5HABUR5GT5Y6BPWNIMNRLDPT2PTY3LVQ, seg_pos={0 0}, upload=438898]: Done uploading piece. err: <nil> Dec 26 11:33:30.435012920 (500h4m30.306095965s) [noise=true , address=145.79.227.16:4912 , piece_id=T7E5PDUZHRO3WRKDHLSYLWQLSTOVQPE3NKX7LISG5S6RHUFDM36Q, seg_pos={0 0}, upload=438898]: Done uploading piece. err: <nil> Dec 26 11:33:30.449914575 (500h4m30.32099762s) [noise=true , address=145.79.227.50:4374 , piece_id=4454XFKBWSOPOKYMCCLMDJDW52Z2QH7FG6QQ4DVVYIAZ5NVUNO5Q, seg_pos={0 0}, upload=438898]: Done uploading piece. err: <nil> Dec 26 11:33:30.456513110 (500h4m30.327596155s) [noise=true , address=145.79.227.38:4979 , piece_id=ASE2VKWERDIBIYWWKKDYIRY337XXM2IDZN7LUF4RQGZM4N3BJSHA, seg_pos={0 0}, upload=438898]: Done uploading piece. err: <nil> Dec 26 11:33:30.457390621 (500h4m30.328473656s) [noise=true , address=145.79.227.29:4943 , piece_id=NALMWEB5P65553GXA7JYGN254ZQCZSGWDILZPYBQQBJOI2Q2CFVA, seg_pos={0 0}, upload=438898]: Done uploading piece. err: <nil> Dec 26 11:33:30.462934878 (500h4m30.334017913s) [noise=true , address=145.79.227.46:4603 , piece_id=2CF7OVADT5JQSUFVCKNIMPD6YUCHVYQ3F2L5OYHVSM2Q735L46CQ, seg_pos={0 0}, upload=438898]: Done uploading piece. err: <nil> Dec 26 11:33:30.464048539 (500h4m30.335131584s) [noise=true , address=145.79.227.49:4339 , piece_id=ZMV6JF3W5S55F5OO2K6Z4SNBNOIULEBEG6RACJ526NIPVBVQ5PMA, seg_pos={0 0}, upload=438898]: Done uploading piece. err: <nil> Dec 26 11:33:30.647928789 (500h4m30.519011824s) [noise=true , address=45.91.116.11:4120 , piece_id=UK7U3NV5HV2BVVUDT7MGROYATPLGM2B2FCUR5WB4IFXIOYWWWRFA, seg_pos={0 0}, upload=438898]: Done uploading piece. err: <nil> Dec 26 11:33:30.666692729 (500h4m30.537775774s) [noise=true , address=45.91.116.14:4192 , piece_id=GEKQWOSNACJU2W6AYFCDHIO7GHB674NOVZARJRTIAT3FR3LYMSJQ, seg_pos={0 0}, upload=438898]: Done uploading piece. err: <nil> Dec 26 11:33:30.804039442 (500h4m30.675122477s) [noise=true , address=45.91.116.12:4225 , piece_id=OUFJURNDHBRTMORDTCDYP5V6NPGZEH3FS2HMIVDNKB25CTJMN47Q, seg_pos={0 0}, upload=438898]: Done uploading piece. err: <nil> Dec 26 11:33:30.844837208 (500h4m30.715920243s) [noise=true , address=77.90.31.11:4556 , piece_id=YBOZQXNWZ4D73XLYY4UK25HJIHKND3OYNNAJ33O35T4VZUED7YXA, seg_pos={0 0}, upload=438898]: Done uploading piece. err: <nil> Dec 26 11:33:31.128170436 (500h4m30.999253471s) [noise=true , address=45.91.116.8:4045 , piece_id=6NHEH7ZOYADN6WLEHKVC4NVAS6J5NGUQGOYWO6S7KXEDUED6RACA, seg_pos={0 0}, upload=438898]: Done uploading piece. err: <nil> Dec 26 11:33:31.698403993 (500h4m31.569487028s) [noise=true , address=145.79.227.44:4194 , piece_id=3OISDZXJ4I5C73TFZCAKEPQW6JTNCRA5OTJLUTN2QQVPGR2JQRGQ, seg_pos={0 0}, upload=438898]: Done uploading piece. err: <nil> Dec 26 11:33:31.698424461 (500h4m31.569507506s) [seg_pos={0 0}, upload=438898]: Segment reached optimal threshold of 13 pieces. Dec 26 11:33:31.698816228 (500h4m31.569899273s) [noise=true , address=45.91.116.5:4299 , piece_id=QCXEG2TLNUTPC53AVZ74NB5G7VVCZR3GN7A3VTUFCQYIMJU4ZXLQ, seg_pos={0 0}, upload=438898]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: write tcp 188.166.225.170:57020->45.91.116.5:4299: use of closed network connection; write tcp 188.166.225.170:57020->45.91.116.5:4299: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57020->45.91.116.5:4299: use of closed network connection Dec 26 11:33:31.698823371 (500h4m31.569906406s) [noise=true , address=45.91.116.5:4299 , piece_id=QCXEG2TLNUTPC53AVZ74NB5G7VVCZR3GN7A3VTUFCQYIMJU4ZXLQ, seg_pos={0 0}, upload=438898]: Long tail cancellation due to optimal threshold Dec 26 11:33:31.698853148 (500h4m31.569936183s) [noise=true , address=45.91.116.10:4084 , piece_id=FAINT7GUVTE6JKVDXLDIUQHW3TFNG2C4AMJ7WHGRQ6GY7Z5G2EPA, seg_pos={0 0}, upload=438898]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: write tcp 188.166.225.170:39374->45.91.116.10:4084: use of closed network connection; write tcp 188.166.225.170:39374->45.91.116.10:4084: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39374->45.91.116.10:4084: use of closed network connection Dec 26 11:33:31.698877294 (500h4m31.569960329s) [noise=true , address=45.91.116.10:4084 , piece_id=FAINT7GUVTE6JKVDXLDIUQHW3TFNG2C4AMJ7WHGRQ6GY7Z5G2EPA, seg_pos={0 0}, upload=438898]: Long tail cancellation due to optimal threshold Dec 26 11:33:31.699107853 (500h4m31.570190888s) [noise=true , address=77.90.31.10:4519 , piece_id=I2AXQLV3D5GTZXXA6SZJVRFXVIICUEGFGYOBNZKP42KHW6KKRZCA, seg_pos={0 0}, upload=438898]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Dec 26 11:33:31.699115096 (500h4m31.570198131s) [noise=true , address=77.90.31.10:4519 , piece_id=I2AXQLV3D5GTZXXA6SZJVRFXVIICUEGFGYOBNZKP42KHW6KKRZCA, seg_pos={0 0}, upload=438898]: Long tail cancellation due to optimal threshold Dec 26 11:33:31.699134524 (500h4m31.570217559s) [seg_pos={0 0}, upload=438898]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 11:33:31.699223853 (500h4m31.570306888s) [upload=438898]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 11:33:31.699228854 (500h4m31.570311899s) [upload=438898]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 11:33:31.982204981 (500h4m31.853288016s) [upload=438898]: Done uploading segments... Dec 26 11:33:31.982223997 (500h4m31.853307032s) [upload=438898]: Upload finished. err: <nil> Duration: 5.973839561s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [.XX..X.......X..X.X......XX......XXX.XX......X.X..XX.X.X...X] duration: 30.003589762s trace: https://jaeger.dev.storj.io/trace/50561a690e9a382d FAIL (2025-12-26 11:33:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: GEKQWOSNACJU2W6AYFCDHIO7GHB674NOVZARJRTIAT3FR3LYMSJQ) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: 6NHEH7ZOYADN6WLEHKVC4NVAS6J5NGUQGOYWO6S7KXEDUED6RACA) context deadline exceeded ======LOGS====== 0h4m28.049098989s) [upload=438898]: Uploading segments... Dec 26 11:33:28.178049557 (500h4m28.049132592s) [upload=438898]: Got next segment. Inline: false Dec 26 11:33:28.178074825 (500h4m28.04915786s) [upload=438898]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 11:33:28.178079243 (500h4m28.049162278s) [upload=438898]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 11:33:28.422805907 (500h4m28.293888942s) [seg_pos={0 0}, upload=438898]: Begin upload segment... Dec 26 11:33:28.422886350 (500h4m28.293969395s) [seg_pos={0 0}, upload=438898]: Done begin upload segment. Dec 26 11:33:28.422961903 (500h4m28.294044948s) [noise=true , address=145.79.227.46:4603 , piece_id=2CF7OVADT5JQSUFVCKNIMPD6YUCHVYQ3F2L5OYHVSM2Q735L46CQ, seg_pos={0 0}, upload=438898]: Uploading piece... Dec 26 11:33:28.423011248 (500h4m28.294094293s) [noise=true , address=145.79.227.51:4410 , piece_id=4LO4TXF75PW4O3SLXD2U5HABUR5GT5Y6BPWNIMNRLDPT2PTY3LVQ, seg_pos={0 0}, upload=438898]: Uploading piece... Dec 26 11:33:28.423619065 (500h4m28.2947021s) [noise=true , address=145.79.227.44:4194 , piece_id=3OISDZXJ4I5C73TFZCAKEPQW6JTNCRA5OTJLUTN2QQVPGR2JQRGQ, seg_pos={0 0}, upload=438898]: Uploading piece... Dec 26 11:33:28.423948102 (500h4m28.295031137s) [noise=true , address=45.91.116.14:4192 , piece_id=GEKQWOSNACJU2W6AYFCDHIO7GHB674NOVZARJRTIAT3FR3LYMSJQ, seg_pos={0 0}, upload=438898]: Uploading piece... Dec 26 11:33:28.424233195 (500h4m28.29531623s) [noise=true , address=145.79.227.16:4912 , piece_id=T7E5PDUZHRO3WRKDHLSYLWQLSTOVQPE3NKX7LISG5S6RHUFDM36Q, seg_pos={0 0}, upload=438898]: Uploading piece... Dec 26 11:33:28.424520181 (500h4m28.295603216s) [noise=true , address=45.91.116.5:4299 , piece_id=QCXEG2TLNUTPC53AVZ74NB5G7VVCZR3GN7A3VTUFCQYIMJU4ZXLQ, seg_pos={0 0}, upload=438898]: Uploading piece... Dec 26 11:33:28.424807708 (500h4m28.295890753s) [noise=true , address=145.79.227.49:4339 , piece_id=ZMV6JF3W5S55F5OO2K6Z4SNBNOIULEBEG6RACJ526NIPVBVQ5PMA, seg_pos={0 0}, upload=438898]: Uploading piece... Dec 26 11:33:28.425093333 (500h4m28.296176368s) [noise=true , address=45.91.116.12:4225 , piece_id=OUFJURNDHBRTMORDTCDYP5V6NPGZEH3FS2HMIVDNKB25CTJMN47Q, seg_pos={0 0}, upload=438898]: Uploading piece... Dec 26 11:33:28.425412991 (500h4m28.296496036s) [noise=true , address=45.91.116.11:4120 , piece_id=UK7U3NV5HV2BVVUDT7MGROYATPLGM2B2FCUR5WB4IFXIOYWWWRFA, seg_pos={0 0}, upload=438898]: Uploading piece... Dec 26 11:33:28.426729629 (500h4m28.297812664s) [noise=true , address=145.79.227.29:4943 , piece_id=NALMWEB5P65553GXA7JYGN254ZQCZSGWDILZPYBQQBJOI2Q2CFVA, seg_pos={0 0}, upload=438898]: Uploading piece... Dec 26 11:33:28.427120402 (500h4m28.298203448s) [noise=true , address=45.91.116.8:4045 , piece_id=6NHEH7ZOYADN6WLEHKVC4NVAS6J5NGUQGOYWO6S7KXEDUED6RACA, seg_pos={0 0}, upload=438898]: Uploading piece... Dec 26 11:33:28.428470443 (500h4m28.299553478s) [noise=true , address=45.91.116.10:4084 , piece_id=FAINT7GUVTE6JKVDXLDIUQHW3TFNG2C4AMJ7WHGRQ6GY7Z5G2EPA, seg_pos={0 0}, upload=438898]: Uploading piece... Dec 26 11:33:28.429781790 (500h4m28.300864825s) [noise=true , address=77.90.31.10:4519 , piece_id=I2AXQLV3D5GTZXXA6SZJVRFXVIICUEGFGYOBNZKP42KHW6KKRZCA, seg_pos={0 0}, upload=438898]: Uploading piece... Dec 26 11:33:28.431046629 (500h4m28.302129664s) [noise=true , address=145.79.227.50:4374 , piece_id=4454XFKBWSOPOKYMCCLMDJDW52Z2QH7FG6QQ4DVVYIAZ5NVUNO5Q, seg_pos={0 0}, upload=438898]: Uploading piece... Dec 26 11:33:28.432307870 (500h4m28.303390905s) [noise=true , address=77.90.31.11:4556 , piece_id=YBOZQXNWZ4D73XLYY4UK25HJIHKND3OYNNAJ33O35T4VZUED7YXA, seg_pos={0 0}, upload=438898]: Uploading piece... Dec 26 11:33:28.433521131 (500h4m28.304604166s) [noise=true , address=145.79.227.38:4979 , piece_id=ASE2VKWERDIBIYWWKKDYIRY337XXM2IDZN7LUF4RQGZM4N3BJSHA, seg_pos={0 0}, upload=438898]: Uploading piece... Dec 26 11:33:30.148590360 (500h4m30.019673405s) [upload=438898]: Next returned nil segment Dec 26 11:33:30.148613774 (500h4m30.019696819s) [upload=438898]: Waiting for error group managing segments... Dec 26 11:33:30.426305860 (500h4m30.297388895s) [noise=true , address=145.79.227.51:4410 , piece_id=4LO4TXF75PW4O3SLXD2U5HABUR5GT5Y6BPWNIMNRLDPT2PTY3LVQ, seg_pos={0 0}, upload=438898]: Done uploading piece. err: <nil> Dec 26 11:33:30.435012920 (500h4m30.306095965s) [noise=true , address=145.79.227.16:4912 , piece_id=T7E5PDUZHRO3WRKDHLSYLWQLSTOVQPE3NKX7LISG5S6RHUFDM36Q, seg_pos={0 0}, upload=438898]: Done uploading piece. err: <nil> Dec 26 11:33:30.449914575 (500h4m30.32099762s) [noise=true , address=145.79.227.50:4374 , piece_id=4454XFKBWSOPOKYMCCLMDJDW52Z2QH7FG6QQ4DVVYIAZ5NVUNO5Q, seg_pos={0 0}, upload=438898]: Done uploading piece. err: <nil> Dec 26 11:33:30.456513110 (500h4m30.327596155s) [noise=true , address=145.79.227.38:4979 , piece_id=ASE2VKWERDIBIYWWKKDYIRY337XXM2IDZN7LUF4RQGZM4N3BJSHA, seg_pos={0 0}, upload=438898]: Done uploading piece. err: <nil> Dec 26 11:33:30.457390621 (500h4m30.328473656s) [noise=true , address=145.79.227.29:4943 , piece_id=NALMWEB5P65553GXA7JYGN254ZQCZSGWDILZPYBQQBJOI2Q2CFVA, seg_pos={0 0}, upload=438898]: Done uploading piece. err: <nil> Dec 26 11:33:30.462934878 (500h4m30.334017913s) [noise=true , address=145.79.227.46:4603 , piece_id=2CF7OVADT5JQSUFVCKNIMPD6YUCHVYQ3F2L5OYHVSM2Q735L46CQ, seg_pos={0 0}, upload=438898]: Done uploading piece. err: <nil> Dec 26 11:33:30.464048539 (500h4m30.335131584s) [noise=true , address=145.79.227.49:4339 , piece_id=ZMV6JF3W5S55F5OO2K6Z4SNBNOIULEBEG6RACJ526NIPVBVQ5PMA, seg_pos={0 0}, upload=438898]: Done uploading piece. err: <nil> Dec 26 11:33:30.647928789 (500h4m30.519011824s) [noise=true , address=45.91.116.11:4120 , piece_id=UK7U3NV5HV2BVVUDT7MGROYATPLGM2B2FCUR5WB4IFXIOYWWWRFA, seg_pos={0 0}, upload=438898]: Done uploading piece. err: <nil> Dec 26 11:33:30.666692729 (500h4m30.537775774s) [noise=true , address=45.91.116.14:4192 , piece_id=GEKQWOSNACJU2W6AYFCDHIO7GHB674NOVZARJRTIAT3FR3LYMSJQ, seg_pos={0 0}, upload=438898]: Done uploading piece. err: <nil> Dec 26 11:33:30.804039442 (500h4m30.675122477s) [noise=true , address=45.91.116.12:4225 , piece_id=OUFJURNDHBRTMORDTCDYP5V6NPGZEH3FS2HMIVDNKB25CTJMN47Q, seg_pos={0 0}, upload=438898]: Done uploading piece. err: <nil> Dec 26 11:33:30.844837208 (500h4m30.715920243s) [noise=true , address=77.90.31.11:4556 , piece_id=YBOZQXNWZ4D73XLYY4UK25HJIHKND3OYNNAJ33O35T4VZUED7YXA, seg_pos={0 0}, upload=438898]: Done uploading piece. err: <nil> Dec 26 11:33:31.128170436 (500h4m30.999253471s) [noise=true , address=45.91.116.8:4045 , piece_id=6NHEH7ZOYADN6WLEHKVC4NVAS6J5NGUQGOYWO6S7KXEDUED6RACA, seg_pos={0 0}, upload=438898]: Done uploading piece. err: <nil> Dec 26 11:33:31.698403993 (500h4m31.569487028s) [noise=true , address=145.79.227.44:4194 , piece_id=3OISDZXJ4I5C73TFZCAKEPQW6JTNCRA5OTJLUTN2QQVPGR2JQRGQ, seg_pos={0 0}, upload=438898]: Done uploading piece. err: <nil> Dec 26 11:33:31.698424461 (500h4m31.569507506s) [seg_pos={0 0}, upload=438898]: Segment reached optimal threshold of 13 pieces. Dec 26 11:33:31.698816228 (500h4m31.569899273s) [noise=true , address=45.91.116.5:4299 , piece_id=QCXEG2TLNUTPC53AVZ74NB5G7VVCZR3GN7A3VTUFCQYIMJU4ZXLQ, seg_pos={0 0}, upload=438898]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: write tcp 188.166.225.170:57020->45.91.116.5:4299: use of closed network connection; write tcp 188.166.225.170:57020->45.91.116.5:4299: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57020->45.91.116.5:4299: use of closed network connection Dec 26 11:33:31.698823371 (500h4m31.569906406s) [noise=true , address=45.91.116.5:4299 , piece_id=QCXEG2TLNUTPC53AVZ74NB5G7VVCZR3GN7A3VTUFCQYIMJU4ZXLQ, seg_pos={0 0}, upload=438898]: Long tail cancellation due to optimal threshold Dec 26 11:33:31.698853148 (500h4m31.569936183s) [noise=true , address=45.91.116.10:4084 , piece_id=FAINT7GUVTE6JKVDXLDIUQHW3TFNG2C4AMJ7WHGRQ6GY7Z5G2EPA, seg_pos={0 0}, upload=438898]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: write tcp 188.166.225.170:39374->45.91.116.10:4084: use of closed network connection; write tcp 188.166.225.170:39374->45.91.116.10:4084: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39374->45.91.116.10:4084: use of closed network connection Dec 26 11:33:31.698877294 (500h4m31.569960329s) [noise=true , address=45.91.116.10:4084 , piece_id=FAINT7GUVTE6JKVDXLDIUQHW3TFNG2C4AMJ7WHGRQ6GY7Z5G2EPA, seg_pos={0 0}, upload=438898]: Long tail cancellation due to optimal threshold Dec 26 11:33:31.699107853 (500h4m31.570190888s) [noise=true , address=77.90.31.10:4519 , piece_id=I2AXQLV3D5GTZXXA6SZJVRFXVIICUEGFGYOBNZKP42KHW6KKRZCA, seg_pos={0 0}, upload=438898]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Dec 26 11:33:31.699115096 (500h4m31.570198131s) [noise=true , address=77.90.31.10:4519 , piece_id=I2AXQLV3D5GTZXXA6SZJVRFXVIICUEGFGYOBNZKP42KHW6KKRZCA, seg_pos={0 0}, upload=438898]: Long tail cancellation due to optimal threshold Dec 26 11:33:31.699134524 (500h4m31.570217559s) [seg_pos={0 0}, upload=438898]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 11:33:31.699223853 (500h4m31.570306888s) [upload=438898]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 11:33:31.699228854 (500h4m31.570311899s) [upload=438898]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 11:33:31.982204981 (500h4m31.853288016s) [upload=438898]: Done uploading segments... Dec 26 11:33:31.982223997 (500h4m31.853307032s) [upload=438898]: Upload finished. err: <nil> Duration: 30.003607736s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 44.382593696s trace: https://jaeger.dev.storj.io/trace/785a90e8be84a595 FAIL (2025-12-26 11:33:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== KXVRQQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.834123964 (499h15m56.705207009s) [noise=true , address=45.91.116.4:4963 , piece_id=U4T64KQ67C3BDKG5TJSTQWUS4WWYCSVPUZOWJ6YANR2IQ2JD4EFQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.834585043 (499h15m56.705668078s) [noise=true , address=145.79.227.9:4390 , piece_id=JDQNHNGABBGANIHKSKP4VHUWIPQBNNEOPC4ZQCKR3K5TTR4N4SGQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.834961800 (499h15m56.706044845s) [noise=true , address=145.79.227.30:4778 , piece_id=BMEAUWANP7FP6CFRULMIAZUV4JNQEXOUN2EAXJY5XOWLDZINWOIA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.835331334 (499h15m56.706414379s) [noise=true , address=77.90.31.18:4838 , piece_id=VYBIKY3MLEE2CXGS2MT23HX5HUFK25FKYKI4XLXPDRNWCC55ANXQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.835738399 (499h15m56.706821434s) [noise=true , address=145.79.227.20:4459 , piece_id=2EUW2AH7FBVZNJ27XVLRXKMXA737DWKAYBL7CR7PRXXRTO2M644A, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.836118653 (499h15m56.707201698s) [noise=true , address=77.90.31.14:4678 , piece_id=XQHQZJUPB24G3BWGFTFDD42NM6K5HJVHUXBXWH5TW7BU25QGHTMA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.836501923 (499h15m56.707584968s) [noise=true , address=45.91.116.6:4988 , piece_id=VPEKJF74D7SQWKR47MAIY5UJQPBJIEU5AQDKLFGATYIRHFREAQ5A, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.836904940 (499h15m56.707987985s) [noise=true , address=64.49.13.169:4711 , piece_id=PVBVY4TVJLPKYBP6A7LBNZADLN25XBDWZFEK2QVN45YHJATIR7TA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.837388641 (499h15m56.708471676s) [noise=true , address=145.79.227.40:4076 , piece_id=Y6WKUYNPDMBUZSNWEYDL2KKNR6WKHSKGP63S6KAG6F2MWMNTVMBQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.837910455 (499h15m56.7089935s) [noise=true , address=77.90.31.11:4571 , piece_id=OERZACIKIG7HBEV3LL42KUAGKN6AWZEXNQSA52FIYTUKDFWCBW7Q, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.838146514 (499h15m56.709229559s) [noise=true , address=64.49.13.26:4945 , piece_id=KXCFTLWR4HSBTP4GJJ4HEPA6BBEMILYTJEMXY4S5WB24HRMQJOKA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.838469329 (499h15m56.709552364s) [noise=true , address=45.91.116.5:4301 , piece_id=E3ZRKZUKM335ARHV3Z5YGLKQYHG6QV4GCCNHAVMN34BM7UQFWIIA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.838690831 (499h15m56.709773866s) [noise=true , address=45.91.116.14:4204 , piece_id=JWL7OOMKTVAP7WEUB5WTL3F6F5IRNID4KIOYJSYZ655PJSNXAQKQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.839143724 (499h15m56.710226759s) [noise=true , address=45.91.116.9:4272 , piece_id=GGFDVKEW5LUCHQQUP6CWJHJQXUETYUG2UISPVP6XQJGSAJW2KP2Q, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.839678982 (499h15m56.710762027s) [noise=true , address=145.79.227.44:4208 , piece_id=ZLIX7XAUMKRUEWWTQST2KLCOWFJSAFSAQNLUCTTOWVZUBESEI23A, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:58.136536405 (499h15m58.00761944s) [upload=438194]: Next returned nil segment Dec 26 10:44:58.136548628 (499h15m58.007631673s) [upload=438194]: Waiting for error group managing segments... Dec 26 10:44:58.469618858 (499h15m58.340701893s) [noise=true , address=64.49.13.26:4945 , piece_id=KXCFTLWR4HSBTP4GJJ4HEPA6BBEMILYTJEMXY4S5WB24HRMQJOKA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.708819055 (499h15m58.5799021s) [noise=true , address=145.79.227.9:4390 , piece_id=JDQNHNGABBGANIHKSKP4VHUWIPQBNNEOPC4ZQCKR3K5TTR4N4SGQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.714466429 (499h15m58.585549474s) [noise=true , address=145.79.227.40:4076 , piece_id=Y6WKUYNPDMBUZSNWEYDL2KKNR6WKHSKGP63S6KAG6F2MWMNTVMBQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.745407696 (499h15m58.616490731s) [noise=true , address=145.79.227.20:4459 , piece_id=2EUW2AH7FBVZNJ27XVLRXKMXA737DWKAYBL7CR7PRXXRTO2M644A, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.888701070 (499h15m58.759784105s) [noise=true , address=77.90.31.18:4838 , piece_id=VYBIKY3MLEE2CXGS2MT23HX5HUFK25FKYKI4XLXPDRNWCC55ANXQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.948150873 (499h15m58.819233918s) [noise=true , address=45.91.116.5:4301 , piece_id=E3ZRKZUKM335ARHV3Z5YGLKQYHG6QV4GCCNHAVMN34BM7UQFWIIA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.971389744 (499h15m58.842472779s) [noise=true , address=77.90.31.19:4868 , piece_id=WH77PZSXC2VUWLQYLBR2NDEVT6KMD3VZK5WAL53AMRDW74P242LQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.021685314 (499h15m58.892768349s) [noise=true , address=45.91.116.4:4963 , piece_id=U4T64KQ67C3BDKG5TJSTQWUS4WWYCSVPUZOWJ6YANR2IQ2JD4EFQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.045418216 (499h15m58.916501251s) [noise=true , address=45.91.116.14:4204 , piece_id=JWL7OOMKTVAP7WEUB5WTL3F6F5IRNID4KIOYJSYZ655PJSNXAQKQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.056260371 (499h15m58.927343416s) [noise=true , address=45.91.116.9:4272 , piece_id=GGFDVKEW5LUCHQQUP6CWJHJQXUETYUG2UISPVP6XQJGSAJW2KP2Q, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.085713054 (499h15m58.956796089s) [noise=true , address=45.91.116.7:4017 , piece_id=L4GGY6AICMSFMUVYGI6ZMUORYRXNJFPLJY5XJRUAM7NVYQOHPHSQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.640980124 (499h15m59.512063169s) [noise=true , address=145.79.227.30:4778 , piece_id=BMEAUWANP7FP6CFRULMIAZUV4JNQEXOUN2EAXJY5XOWLDZINWOIA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.654515607 (499h15m59.525598652s) [noise=true , address=145.79.227.44:4208 , piece_id=ZLIX7XAUMKRUEWWTQST2KLCOWFJSAFSAQNLUCTTOWVZUBESEI23A, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.660349657 (499h15m59.531432712s) [noise=true , address=77.90.31.11:4571 , piece_id=OERZACIKIG7HBEV3LL42KUAGKN6AWZEXNQSA52FIYTUKDFWCBW7Q, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.660376087 (499h15m59.531459132s) [seg_pos={0 0}, upload=438194]: Segment reached optimal threshold of 14 pieces. Dec 26 10:44:59.660612988 (499h15m59.531696033s) [noise=true , address=64.49.13.169:4711 , piece_id=PVBVY4TVJLPKYBP6A7LBNZADLN25XBDWZFEK2QVN45YHJATIR7TA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:52964->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:52964->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52964->64.49.13.169:4711: use of closed network connection Dec 26 10:44:59.660620753 (499h15m59.531703788s) [noise=true , address=64.49.13.169:4711 , piece_id=PVBVY4TVJLPKYBP6A7LBNZADLN25XBDWZFEK2QVN45YHJATIR7TA, seg_pos={0 0}, upload=438194]: Long tail cancellation due to optimal threshold Dec 26 10:44:59.660893663 (499h15m59.531976698s) [noise=true , address=45.91.116.6:4988 , piece_id=VPEKJF74D7SQWKR47MAIY5UJQPBJIEU5AQDKLFGATYIRHFREAQ5A, seg_pos={0 0}, upload=438194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TrcoCHhAJeEykP9hJNpygoPSdgFV5v6GzL48JpyNPiAj5ZJRS): protocol: expected piece hash; context canceled; EOF Dec 26 10:44:59.660901057 (499h15m59.531984102s) [noise=true , address=45.91.116.6:4988 , piece_id=VPEKJF74D7SQWKR47MAIY5UJQPBJIEU5AQDKLFGATYIRHFREAQ5A, seg_pos={0 0}, upload=438194]: Long tail cancellation due to optimal threshold Dec 26 10:44:59.661226155 (499h15m59.5323092s) [noise=true , address=77.90.31.14:4678 , piece_id=XQHQZJUPB24G3BWGFTFDD42NM6K5HJVHUXBXWH5TW7BU25QGHTMA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12beu3SUZMPNWfbeHV4tLtx9UVZoEHh4Kt7oBhT59w3bGBE1ovP): protocol: write tcp 188.166.225.170:35884->77.90.31.14:4678: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 10:44:59.661249630 (499h15m59.532332685s) [noise=true , address=77.90.31.14:4678 , piece_id=XQHQZJUPB24G3BWGFTFDD42NM6K5HJVHUXBXWH5TW7BU25QGHTMA, seg_pos={0 0}, upload=438194]: Long tail cancellation due to optimal threshold Dec 26 10:44:59.661309705 (499h15m59.53239275s) [noise=true , address=77.90.31.21:4909 , piece_id=H6KT2AKZISCU4VEA6TXJJ3EFGCRBZPPXBEJJEIHZR3GEJZKXVRQQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121RT7tP6czMFAiZ3DjzSSsYgPt3xZXVbXVpkcYxaZYscNiMjL6): protocol: expected piece hash; context canceled; EOF Dec 26 10:44:59.661318962 (499h15m59.532401997s) [noise=true , address=77.90.31.21:4909 , piece_id=H6KT2AKZISCU4VEA6TXJJ3EFGCRBZPPXBEJJEIHZR3GEJZKXVRQQ, seg_pos={0 0}, upload=438194]: Long tail cancellation due to optimal threshold Dec 26 10:44:59.661345674 (499h15m59.532428719s) [seg_pos={0 0}, upload=438194]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 10:44:59.661485770 (499h15m59.532568815s) [upload=438194]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 10:44:59.661493655 (499h15m59.53257669s) [upload=438194]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 10:44:59.968467153 (499h15m59.839550198s) [upload=438194]: Done uploading segments... Dec 26 10:44:59.968484155 (499h15m59.83956719s) [upload=438194]: Upload finished. err: <nil> Duration: 44.38260653s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 54.628400309s trace: https://jaeger.dev.storj.io/trace/785a90e8be84a595 FAIL (2025-12-26 11:32:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== KXVRQQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.834123964 (499h15m56.705207009s) [noise=true , address=45.91.116.4:4963 , piece_id=U4T64KQ67C3BDKG5TJSTQWUS4WWYCSVPUZOWJ6YANR2IQ2JD4EFQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.834585043 (499h15m56.705668078s) [noise=true , address=145.79.227.9:4390 , piece_id=JDQNHNGABBGANIHKSKP4VHUWIPQBNNEOPC4ZQCKR3K5TTR4N4SGQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.834961800 (499h15m56.706044845s) [noise=true , address=145.79.227.30:4778 , piece_id=BMEAUWANP7FP6CFRULMIAZUV4JNQEXOUN2EAXJY5XOWLDZINWOIA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.835331334 (499h15m56.706414379s) [noise=true , address=77.90.31.18:4838 , piece_id=VYBIKY3MLEE2CXGS2MT23HX5HUFK25FKYKI4XLXPDRNWCC55ANXQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.835738399 (499h15m56.706821434s) [noise=true , address=145.79.227.20:4459 , piece_id=2EUW2AH7FBVZNJ27XVLRXKMXA737DWKAYBL7CR7PRXXRTO2M644A, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.836118653 (499h15m56.707201698s) [noise=true , address=77.90.31.14:4678 , piece_id=XQHQZJUPB24G3BWGFTFDD42NM6K5HJVHUXBXWH5TW7BU25QGHTMA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.836501923 (499h15m56.707584968s) [noise=true , address=45.91.116.6:4988 , piece_id=VPEKJF74D7SQWKR47MAIY5UJQPBJIEU5AQDKLFGATYIRHFREAQ5A, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.836904940 (499h15m56.707987985s) [noise=true , address=64.49.13.169:4711 , piece_id=PVBVY4TVJLPKYBP6A7LBNZADLN25XBDWZFEK2QVN45YHJATIR7TA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.837388641 (499h15m56.708471676s) [noise=true , address=145.79.227.40:4076 , piece_id=Y6WKUYNPDMBUZSNWEYDL2KKNR6WKHSKGP63S6KAG6F2MWMNTVMBQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.837910455 (499h15m56.7089935s) [noise=true , address=77.90.31.11:4571 , piece_id=OERZACIKIG7HBEV3LL42KUAGKN6AWZEXNQSA52FIYTUKDFWCBW7Q, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.838146514 (499h15m56.709229559s) [noise=true , address=64.49.13.26:4945 , piece_id=KXCFTLWR4HSBTP4GJJ4HEPA6BBEMILYTJEMXY4S5WB24HRMQJOKA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.838469329 (499h15m56.709552364s) [noise=true , address=45.91.116.5:4301 , piece_id=E3ZRKZUKM335ARHV3Z5YGLKQYHG6QV4GCCNHAVMN34BM7UQFWIIA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.838690831 (499h15m56.709773866s) [noise=true , address=45.91.116.14:4204 , piece_id=JWL7OOMKTVAP7WEUB5WTL3F6F5IRNID4KIOYJSYZ655PJSNXAQKQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.839143724 (499h15m56.710226759s) [noise=true , address=45.91.116.9:4272 , piece_id=GGFDVKEW5LUCHQQUP6CWJHJQXUETYUG2UISPVP6XQJGSAJW2KP2Q, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.839678982 (499h15m56.710762027s) [noise=true , address=145.79.227.44:4208 , piece_id=ZLIX7XAUMKRUEWWTQST2KLCOWFJSAFSAQNLUCTTOWVZUBESEI23A, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:58.136536405 (499h15m58.00761944s) [upload=438194]: Next returned nil segment Dec 26 10:44:58.136548628 (499h15m58.007631673s) [upload=438194]: Waiting for error group managing segments... Dec 26 10:44:58.469618858 (499h15m58.340701893s) [noise=true , address=64.49.13.26:4945 , piece_id=KXCFTLWR4HSBTP4GJJ4HEPA6BBEMILYTJEMXY4S5WB24HRMQJOKA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.708819055 (499h15m58.5799021s) [noise=true , address=145.79.227.9:4390 , piece_id=JDQNHNGABBGANIHKSKP4VHUWIPQBNNEOPC4ZQCKR3K5TTR4N4SGQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.714466429 (499h15m58.585549474s) [noise=true , address=145.79.227.40:4076 , piece_id=Y6WKUYNPDMBUZSNWEYDL2KKNR6WKHSKGP63S6KAG6F2MWMNTVMBQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.745407696 (499h15m58.616490731s) [noise=true , address=145.79.227.20:4459 , piece_id=2EUW2AH7FBVZNJ27XVLRXKMXA737DWKAYBL7CR7PRXXRTO2M644A, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.888701070 (499h15m58.759784105s) [noise=true , address=77.90.31.18:4838 , piece_id=VYBIKY3MLEE2CXGS2MT23HX5HUFK25FKYKI4XLXPDRNWCC55ANXQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.948150873 (499h15m58.819233918s) [noise=true , address=45.91.116.5:4301 , piece_id=E3ZRKZUKM335ARHV3Z5YGLKQYHG6QV4GCCNHAVMN34BM7UQFWIIA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.971389744 (499h15m58.842472779s) [noise=true , address=77.90.31.19:4868 , piece_id=WH77PZSXC2VUWLQYLBR2NDEVT6KMD3VZK5WAL53AMRDW74P242LQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.021685314 (499h15m58.892768349s) [noise=true , address=45.91.116.4:4963 , piece_id=U4T64KQ67C3BDKG5TJSTQWUS4WWYCSVPUZOWJ6YANR2IQ2JD4EFQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.045418216 (499h15m58.916501251s) [noise=true , address=45.91.116.14:4204 , piece_id=JWL7OOMKTVAP7WEUB5WTL3F6F5IRNID4KIOYJSYZ655PJSNXAQKQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.056260371 (499h15m58.927343416s) [noise=true , address=45.91.116.9:4272 , piece_id=GGFDVKEW5LUCHQQUP6CWJHJQXUETYUG2UISPVP6XQJGSAJW2KP2Q, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.085713054 (499h15m58.956796089s) [noise=true , address=45.91.116.7:4017 , piece_id=L4GGY6AICMSFMUVYGI6ZMUORYRXNJFPLJY5XJRUAM7NVYQOHPHSQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.640980124 (499h15m59.512063169s) [noise=true , address=145.79.227.30:4778 , piece_id=BMEAUWANP7FP6CFRULMIAZUV4JNQEXOUN2EAXJY5XOWLDZINWOIA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.654515607 (499h15m59.525598652s) [noise=true , address=145.79.227.44:4208 , piece_id=ZLIX7XAUMKRUEWWTQST2KLCOWFJSAFSAQNLUCTTOWVZUBESEI23A, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.660349657 (499h15m59.531432712s) [noise=true , address=77.90.31.11:4571 , piece_id=OERZACIKIG7HBEV3LL42KUAGKN6AWZEXNQSA52FIYTUKDFWCBW7Q, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.660376087 (499h15m59.531459132s) [seg_pos={0 0}, upload=438194]: Segment reached optimal threshold of 14 pieces. Dec 26 10:44:59.660612988 (499h15m59.531696033s) [noise=true , address=64.49.13.169:4711 , piece_id=PVBVY4TVJLPKYBP6A7LBNZADLN25XBDWZFEK2QVN45YHJATIR7TA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:52964->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:52964->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52964->64.49.13.169:4711: use of closed network connection Dec 26 10:44:59.660620753 (499h15m59.531703788s) [noise=true , address=64.49.13.169:4711 , piece_id=PVBVY4TVJLPKYBP6A7LBNZADLN25XBDWZFEK2QVN45YHJATIR7TA, seg_pos={0 0}, upload=438194]: Long tail cancellation due to optimal threshold Dec 26 10:44:59.660893663 (499h15m59.531976698s) [noise=true , address=45.91.116.6:4988 , piece_id=VPEKJF74D7SQWKR47MAIY5UJQPBJIEU5AQDKLFGATYIRHFREAQ5A, seg_pos={0 0}, upload=438194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TrcoCHhAJeEykP9hJNpygoPSdgFV5v6GzL48JpyNPiAj5ZJRS): protocol: expected piece hash; context canceled; EOF Dec 26 10:44:59.660901057 (499h15m59.531984102s) [noise=true , address=45.91.116.6:4988 , piece_id=VPEKJF74D7SQWKR47MAIY5UJQPBJIEU5AQDKLFGATYIRHFREAQ5A, seg_pos={0 0}, upload=438194]: Long tail cancellation due to optimal threshold Dec 26 10:44:59.661226155 (499h15m59.5323092s) [noise=true , address=77.90.31.14:4678 , piece_id=XQHQZJUPB24G3BWGFTFDD42NM6K5HJVHUXBXWH5TW7BU25QGHTMA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12beu3SUZMPNWfbeHV4tLtx9UVZoEHh4Kt7oBhT59w3bGBE1ovP): protocol: write tcp 188.166.225.170:35884->77.90.31.14:4678: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 10:44:59.661249630 (499h15m59.532332685s) [noise=true , address=77.90.31.14:4678 , piece_id=XQHQZJUPB24G3BWGFTFDD42NM6K5HJVHUXBXWH5TW7BU25QGHTMA, seg_pos={0 0}, upload=438194]: Long tail cancellation due to optimal threshold Dec 26 10:44:59.661309705 (499h15m59.53239275s) [noise=true , address=77.90.31.21:4909 , piece_id=H6KT2AKZISCU4VEA6TXJJ3EFGCRBZPPXBEJJEIHZR3GEJZKXVRQQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121RT7tP6czMFAiZ3DjzSSsYgPt3xZXVbXVpkcYxaZYscNiMjL6): protocol: expected piece hash; context canceled; EOF Dec 26 10:44:59.661318962 (499h15m59.532401997s) [noise=true , address=77.90.31.21:4909 , piece_id=H6KT2AKZISCU4VEA6TXJJ3EFGCRBZPPXBEJJEIHZR3GEJZKXVRQQ, seg_pos={0 0}, upload=438194]: Long tail cancellation due to optimal threshold Dec 26 10:44:59.661345674 (499h15m59.532428719s) [seg_pos={0 0}, upload=438194]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 10:44:59.661485770 (499h15m59.532568815s) [upload=438194]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 10:44:59.661493655 (499h15m59.53257669s) [upload=438194]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 10:44:59.968467153 (499h15m59.839550198s) [upload=438194]: Done uploading segments... Dec 26 10:44:59.968484155 (499h15m59.83956719s) [upload=438194]: Upload finished. err: <nil> Duration: 54.628431939s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [X.XX..X.......X..X.X......XX......XXX.XX......X.X..XX.X.X...] duration: 5.731343213s trace: https://jaeger.dev.storj.io/trace/3cbd70d9ed10fddd FAIL (2025-12-26 11:32:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: B6TLEJ2RBOWDTM3UVHA3OIM7WWQRHQM3KPKAP6DWQZ2ZF6OJ33CA) context deadline exceeded; piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: 5X5LBZXMQS4GLANHB4LZ736EAH4TVWPYX3DSWIBVBNH6LBS32I7A) context deadline exceeded ======LOGS====== Dec 26 11:30:14.902827760 (500h1m14.773910795s) [upload=438849]: Starting upload Dec 26 11:30:14.902885119 (500h1m14.773968165s) [upload=438849]: Done starting upload Dec 26 11:30:14.902969400 (500h1m14.774052445s) [upload=438849]: Uploading segments... Dec 26 11:30:14.903144593 (500h1m14.774227628s) [upload=438849]: Got next segment. Inline: false Dec 26 11:30:14.903197464 (500h1m14.774280499s) [upload=438849]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 11:30:14.903204277 (500h1m14.774287322s) [upload=438849]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 11:30:15.148960661 (500h1m15.020043696s) [seg_pos={0 0}, upload=438849]: Begin upload segment... Dec 26 11:30:15.149007871 (500h1m15.020090906s) [seg_pos={0 0}, upload=438849]: Done begin upload segment. Dec 26 11:30:15.149035434 (500h1m15.020118469s) [noise=true , address=45.91.116.6:4975 , piece_id=DSTBEQCKUXUWR2XDRIMEE3E267VXY5GTFPJ4PQXBFUD3SW4JMDSA, seg_pos={0 0}, upload=438849]: Uploading piece... Dec 26 11:30:15.149137539 (500h1m15.020220574s) [noise=true , address=145.79.227.48:4301 , piece_id=2YNXNO5PMSVPDKKMN2ZEFSRJHUBLM2G4XKCV6IEYQMZZLVB5PR6A, seg_pos={0 0}, upload=438849]: Uploading piece... Dec 26 11:30:15.149873229 (500h1m15.020956274s) [noise=true , address=145.79.227.31:4799 , piece_id=DHNGACX5WUD3YY7PKFUMJ3SZK2XI7WS7ESZD4LLVTNBL6K2Q3CPQ, seg_pos={0 0}, upload=438849]: Uploading piece... Dec 26 11:30:15.150375936 (500h1m15.021458981s) [noise=true , address=45.91.116.5:4298 , piece_id=RRVN7ERVO46M5GJPCDA2FDBF2WWZSDG3DUKQNCKSF5KQONMB76AQ, seg_pos={0 0}, upload=438849]: Uploading piece... Dec 26 11:30:15.150875928 (500h1m15.021958973s) [noise=true , address=145.79.227.6:4258 , piece_id=BKCG5FZVFBCSI7LHD2YTDY7BFMRTNRD4JMER4CTTEWNY7Q53BI3A, seg_pos={0 0}, upload=438849]: Uploading piece... Dec 26 11:30:15.151287341 (500h1m15.022370376s) [noise=true , address=77.90.31.14:4662 , piece_id=K7CLEXMKD3EVCWHJW2Z6KGDFX26HR4W52GLUCKLG5K4DWGO5F2CA, seg_pos={0 0}, upload=438849]: Uploading piece... Dec 26 11:30:15.151711319 (500h1m15.022794364s) [noise=true , address=45.91.116.4:4939 , piece_id=KGKZX7W5UBC3IMN6WSQEISB4CK7IAZ7ECCJS526YGYBJGUUEKAAQ, seg_pos={0 0}, upload=438849]: Uploading piece... Dec 26 11:30:15.153014350 (500h1m15.024097395s) [noise=true , address=45.91.116.10:4082 , piece_id=B6TLEJ2RBOWDTM3UVHA3OIM7WWQRHQM3KPKAP6DWQZ2ZF6OJ33CA, seg_pos={0 0}, upload=438849]: Uploading piece... Dec 26 11:30:15.153391188 (500h1m15.024474223s) [noise=true , address=145.79.227.46:4602 , piece_id=2UT5E72P2YCEXJZ6FWLM5RWFVUYDGBTQ4H5IXJDSJDJ5WHSY7NZQ, seg_pos={0 0}, upload=438849]: Uploading piece... Dec 26 11:30:15.154893739 (500h1m15.025976784s) [noise=true , address=45.91.116.8:4047 , piece_id=5IPJK4IQD554GLHPKUMYX5UCPPVEVXMTKQ6QQK3NAX5KUNHB5LHA, seg_pos={0 0}, upload=438849]: Uploading piece... Dec 26 11:30:15.155366759 (500h1m15.026449804s) [noise=true , address=145.79.227.39:4013 , piece_id=LJT2PBTVO7XRTS4BLZM4QHGQW3ZQV6FJHICZTI55NJECR6YZFECQ, seg_pos={0 0}, upload=438849]: Uploading piece... Dec 26 11:30:15.157001762 (500h1m15.028084808s) [noise=true , address=145.79.227.52:4446 , piece_id=ELGUDSP4GF2ICCMYQQD2TQSCJSQ5IYDLQZGZBJAAC34CRATQNFHA, seg_pos={0 0}, upload=438849]: Uploading piece... Dec 26 11:30:15.158394304 (500h1m15.029477339s) [noise=true , address=145.79.227.41:4087 , piece_id=6ECDUGJINW6FFQLHWVFYN56OFU7QYAMHWXPDDJ3URD7TOC4KTOAA, seg_pos={0 0}, upload=438849]: Uploading piece... Dec 26 11:30:15.160400314 (500h1m15.031483359s) [noise=true , address=145.79.227.34:4906 , piece_id=F2V55V6UN5MN4CMP7Q5GKWEGWBPZBBWSWS2BKYP5DYGZBYNHORQQ, seg_pos={0 0}, upload=438849]: Uploading piece... Dec 26 11:30:15.161370011 (500h1m15.032453056s) [noise=true , address=45.91.116.13:4154 , piece_id=5X5LBZXMQS4GLANHB4LZ736EAH4TVWPYX3DSWIBVBNH6LBS32I7A, seg_pos={0 0}, upload=438849]: Uploading piece... Dec 26 11:30:15.163119021 (500h1m15.034202066s) [noise=true , address=145.79.227.42:4122 , piece_id=WM5IV4MTYKYM4SJUPAFUOHLYNJKAE7ZCFV327RIHY3GYG6BOM7PQ, seg_pos={0 0}, upload=438849]: Uploading piece... Dec 26 11:30:16.875538755 (500h1m16.74662179s) [upload=438849]: Next returned nil segment Dec 26 11:30:16.875553864 (500h1m16.746636909s) [upload=438849]: Waiting for error group managing segments... Dec 26 11:30:17.161371171 (500h1m17.032454216s) [noise=true , address=145.79.227.6:4258 , piece_id=BKCG5FZVFBCSI7LHD2YTDY7BFMRTNRD4JMER4CTTEWNY7Q53BI3A, seg_pos={0 0}, upload=438849]: Done uploading piece. err: <nil> Dec 26 11:30:17.163325181 (500h1m17.034408216s) [noise=true , address=145.79.227.31:4799 , piece_id=DHNGACX5WUD3YY7PKFUMJ3SZK2XI7WS7ESZD4LLVTNBL6K2Q3CPQ, seg_pos={0 0}, upload=438849]: Done uploading piece. err: <nil> Dec 26 11:30:17.166911150 (500h1m17.037994195s) [noise=true , address=145.79.227.42:4122 , piece_id=WM5IV4MTYKYM4SJUPAFUOHLYNJKAE7ZCFV327RIHY3GYG6BOM7PQ, seg_pos={0 0}, upload=438849]: Done uploading piece. err: <nil> Dec 26 11:30:17.168159116 (500h1m17.039242161s) [noise=true , address=145.79.227.39:4013 , piece_id=LJT2PBTVO7XRTS4BLZM4QHGQW3ZQV6FJHICZTI55NJECR6YZFECQ, seg_pos={0 0}, upload=438849]: Done uploading piece. err: <nil> Dec 26 11:30:17.171042407 (500h1m17.042125452s) [noise=true , address=145.79.227.52:4446 , piece_id=ELGUDSP4GF2ICCMYQQD2TQSCJSQ5IYDLQZGZBJAAC34CRATQNFHA, seg_pos={0 0}, upload=438849]: Done uploading piece. err: <nil> Dec 26 11:30:17.183473047 (500h1m17.054556082s) [noise=true , address=145.79.227.41:4087 , piece_id=6ECDUGJINW6FFQLHWVFYN56OFU7QYAMHWXPDDJ3URD7TOC4KTOAA, seg_pos={0 0}, upload=438849]: Done uploading piece. err: <nil> Dec 26 11:30:17.188319034 (500h1m17.059402079s) [noise=true , address=145.79.227.46:4602 , piece_id=2UT5E72P2YCEXJZ6FWLM5RWFVUYDGBTQ4H5IXJDSJDJ5WHSY7NZQ, seg_pos={0 0}, upload=438849]: Done uploading piece. err: <nil> Dec 26 11:30:17.188376233 (500h1m17.059459278s) [noise=true , address=145.79.227.48:4301 , piece_id=2YNXNO5PMSVPDKKMN2ZEFSRJHUBLM2G4XKCV6IEYQMZZLVB5PR6A, seg_pos={0 0}, upload=438849]: Done uploading piece. err: <nil> Dec 26 11:30:17.352678334 (500h1m17.223761379s) [noise=true , address=77.90.31.14:4662 , piece_id=K7CLEXMKD3EVCWHJW2Z6KGDFX26HR4W52GLUCKLG5K4DWGO5F2CA, seg_pos={0 0}, upload=438849]: Done uploading piece. err: <nil> Dec 26 11:30:17.385303118 (500h1m17.256386153s) [noise=true , address=45.91.116.4:4939 , piece_id=KGKZX7W5UBC3IMN6WSQEISB4CK7IAZ7ECCJS526YGYBJGUUEKAAQ, seg_pos={0 0}, upload=438849]: Done uploading piece. err: <nil> Dec 26 11:30:17.418483329 (500h1m17.289566374s) [noise=true , address=45.91.116.13:4154 , piece_id=5X5LBZXMQS4GLANHB4LZ736EAH4TVWPYX3DSWIBVBNH6LBS32I7A, seg_pos={0 0}, upload=438849]: Done uploading piece. err: <nil> Dec 26 11:30:17.421974506 (500h1m17.293057541s) [noise=true , address=45.91.116.6:4975 , piece_id=DSTBEQCKUXUWR2XDRIMEE3E267VXY5GTFPJ4PQXBFUD3SW4JMDSA, seg_pos={0 0}, upload=438849]: Done uploading piece. err: <nil> Dec 26 11:30:17.521180402 (500h1m17.392263438s) [noise=true , address=45.91.116.10:4082 , piece_id=B6TLEJ2RBOWDTM3UVHA3OIM7WWQRHQM3KPKAP6DWQZ2ZF6OJ33CA, seg_pos={0 0}, upload=438849]: Done uploading piece. err: <nil> Dec 26 11:30:17.521200039 (500h1m17.392283084s) [seg_pos={0 0}, upload=438849]: Segment reached optimal threshold of 13 pieces. Dec 26 11:30:17.521427854 (500h1m17.392510899s) [noise=true , address=45.91.116.5:4298 , piece_id=RRVN7ERVO46M5GJPCDA2FDBF2WWZSDG3DUKQNCKSF5KQONMB76AQ, seg_pos={0 0}, upload=438849]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 26 11:30:17.521435518 (500h1m17.392518553s) [noise=true , address=45.91.116.5:4298 , piece_id=RRVN7ERVO46M5GJPCDA2FDBF2WWZSDG3DUKQNCKSF5KQONMB76AQ, seg_pos={0 0}, upload=438849]: Long tail cancellation due to optimal threshold Dec 26 11:30:17.521668923 (500h1m17.392751978s) [noise=true , address=145.79.227.34:4906 , piece_id=F2V55V6UN5MN4CMP7Q5GKWEGWBPZBBWSWS2BKYP5DYGZBYNHORQQ, seg_pos={0 0}, upload=438849]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 26 11:30:17.521692036 (500h1m17.392775091s) [noise=true , address=145.79.227.34:4906 , piece_id=F2V55V6UN5MN4CMP7Q5GKWEGWBPZBBWSWS2BKYP5DYGZBYNHORQQ, seg_pos={0 0}, upload=438849]: Long tail cancellation due to optimal threshold Dec 26 11:30:17.521723326 (500h1m17.392806371s) [noise=true , address=45.91.116.8:4047 , piece_id=5IPJK4IQD554GLHPKUMYX5UCPPVEVXMTKQ6QQK3NAX5KUNHB5LHA, seg_pos={0 0}, upload=438849]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 26 11:30:17.521733265 (500h1m17.39281631s) [noise=true , address=45.91.116.8:4047 , piece_id=5IPJK4IQD554GLHPKUMYX5UCPPVEVXMTKQ6QQK3NAX5KUNHB5LHA, seg_pos={0 0}, upload=438849]: Long tail cancellation due to optimal threshold Dec 26 11:30:17.521769053 (500h1m17.392852099s) [seg_pos={0 0}, upload=438849]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 11:30:17.521914570 (500h1m17.392997625s) [upload=438849]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 11:30:17.521922165 (500h1m17.39300521s) [upload=438849]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 11:30:17.808609149 (500h1m17.679692194s) [upload=438849]: Done uploading segments... Dec 26 11:30:17.808629166 (500h1m17.679712201s) [upload=438849]: Upload finished. err: <nil> Duration: 5.731375835s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [XX.XX..X.......X..X.X......XX......XXX.XX......X.X..XX.X.X..] duration: 5.934779472s trace: https://jaeger.dev.storj.io/trace/3cbd70d9ed10fddd FAIL (2025-12-26 11:32:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: B6TLEJ2RBOWDTM3UVHA3OIM7WWQRHQM3KPKAP6DWQZ2ZF6OJ33CA) context deadline exceeded; piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: 5X5LBZXMQS4GLANHB4LZ736EAH4TVWPYX3DSWIBVBNH6LBS32I7A) context deadline exceeded ======LOGS====== Dec 26 11:30:14.902827760 (500h1m14.773910795s) [upload=438849]: Starting upload Dec 26 11:30:14.902885119 (500h1m14.773968165s) [upload=438849]: Done starting upload Dec 26 11:30:14.902969400 (500h1m14.774052445s) [upload=438849]: Uploading segments... Dec 26 11:30:14.903144593 (500h1m14.774227628s) [upload=438849]: Got next segment. Inline: false Dec 26 11:30:14.903197464 (500h1m14.774280499s) [upload=438849]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 11:30:14.903204277 (500h1m14.774287322s) [upload=438849]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 11:30:15.148960661 (500h1m15.020043696s) [seg_pos={0 0}, upload=438849]: Begin upload segment... Dec 26 11:30:15.149007871 (500h1m15.020090906s) [seg_pos={0 0}, upload=438849]: Done begin upload segment. Dec 26 11:30:15.149035434 (500h1m15.020118469s) [noise=true , address=45.91.116.6:4975 , piece_id=DSTBEQCKUXUWR2XDRIMEE3E267VXY5GTFPJ4PQXBFUD3SW4JMDSA, seg_pos={0 0}, upload=438849]: Uploading piece... Dec 26 11:30:15.149137539 (500h1m15.020220574s) [noise=true , address=145.79.227.48:4301 , piece_id=2YNXNO5PMSVPDKKMN2ZEFSRJHUBLM2G4XKCV6IEYQMZZLVB5PR6A, seg_pos={0 0}, upload=438849]: Uploading piece... Dec 26 11:30:15.149873229 (500h1m15.020956274s) [noise=true , address=145.79.227.31:4799 , piece_id=DHNGACX5WUD3YY7PKFUMJ3SZK2XI7WS7ESZD4LLVTNBL6K2Q3CPQ, seg_pos={0 0}, upload=438849]: Uploading piece... Dec 26 11:30:15.150375936 (500h1m15.021458981s) [noise=true , address=45.91.116.5:4298 , piece_id=RRVN7ERVO46M5GJPCDA2FDBF2WWZSDG3DUKQNCKSF5KQONMB76AQ, seg_pos={0 0}, upload=438849]: Uploading piece... Dec 26 11:30:15.150875928 (500h1m15.021958973s) [noise=true , address=145.79.227.6:4258 , piece_id=BKCG5FZVFBCSI7LHD2YTDY7BFMRTNRD4JMER4CTTEWNY7Q53BI3A, seg_pos={0 0}, upload=438849]: Uploading piece... Dec 26 11:30:15.151287341 (500h1m15.022370376s) [noise=true , address=77.90.31.14:4662 , piece_id=K7CLEXMKD3EVCWHJW2Z6KGDFX26HR4W52GLUCKLG5K4DWGO5F2CA, seg_pos={0 0}, upload=438849]: Uploading piece... Dec 26 11:30:15.151711319 (500h1m15.022794364s) [noise=true , address=45.91.116.4:4939 , piece_id=KGKZX7W5UBC3IMN6WSQEISB4CK7IAZ7ECCJS526YGYBJGUUEKAAQ, seg_pos={0 0}, upload=438849]: Uploading piece... Dec 26 11:30:15.153014350 (500h1m15.024097395s) [noise=true , address=45.91.116.10:4082 , piece_id=B6TLEJ2RBOWDTM3UVHA3OIM7WWQRHQM3KPKAP6DWQZ2ZF6OJ33CA, seg_pos={0 0}, upload=438849]: Uploading piece... Dec 26 11:30:15.153391188 (500h1m15.024474223s) [noise=true , address=145.79.227.46:4602 , piece_id=2UT5E72P2YCEXJZ6FWLM5RWFVUYDGBTQ4H5IXJDSJDJ5WHSY7NZQ, seg_pos={0 0}, upload=438849]: Uploading piece... Dec 26 11:30:15.154893739 (500h1m15.025976784s) [noise=true , address=45.91.116.8:4047 , piece_id=5IPJK4IQD554GLHPKUMYX5UCPPVEVXMTKQ6QQK3NAX5KUNHB5LHA, seg_pos={0 0}, upload=438849]: Uploading piece... Dec 26 11:30:15.155366759 (500h1m15.026449804s) [noise=true , address=145.79.227.39:4013 , piece_id=LJT2PBTVO7XRTS4BLZM4QHGQW3ZQV6FJHICZTI55NJECR6YZFECQ, seg_pos={0 0}, upload=438849]: Uploading piece... Dec 26 11:30:15.157001762 (500h1m15.028084808s) [noise=true , address=145.79.227.52:4446 , piece_id=ELGUDSP4GF2ICCMYQQD2TQSCJSQ5IYDLQZGZBJAAC34CRATQNFHA, seg_pos={0 0}, upload=438849]: Uploading piece... Dec 26 11:30:15.158394304 (500h1m15.029477339s) [noise=true , address=145.79.227.41:4087 , piece_id=6ECDUGJINW6FFQLHWVFYN56OFU7QYAMHWXPDDJ3URD7TOC4KTOAA, seg_pos={0 0}, upload=438849]: Uploading piece... Dec 26 11:30:15.160400314 (500h1m15.031483359s) [noise=true , address=145.79.227.34:4906 , piece_id=F2V55V6UN5MN4CMP7Q5GKWEGWBPZBBWSWS2BKYP5DYGZBYNHORQQ, seg_pos={0 0}, upload=438849]: Uploading piece... Dec 26 11:30:15.161370011 (500h1m15.032453056s) [noise=true , address=45.91.116.13:4154 , piece_id=5X5LBZXMQS4GLANHB4LZ736EAH4TVWPYX3DSWIBVBNH6LBS32I7A, seg_pos={0 0}, upload=438849]: Uploading piece... Dec 26 11:30:15.163119021 (500h1m15.034202066s) [noise=true , address=145.79.227.42:4122 , piece_id=WM5IV4MTYKYM4SJUPAFUOHLYNJKAE7ZCFV327RIHY3GYG6BOM7PQ, seg_pos={0 0}, upload=438849]: Uploading piece... Dec 26 11:30:16.875538755 (500h1m16.74662179s) [upload=438849]: Next returned nil segment Dec 26 11:30:16.875553864 (500h1m16.746636909s) [upload=438849]: Waiting for error group managing segments... Dec 26 11:30:17.161371171 (500h1m17.032454216s) [noise=true , address=145.79.227.6:4258 , piece_id=BKCG5FZVFBCSI7LHD2YTDY7BFMRTNRD4JMER4CTTEWNY7Q53BI3A, seg_pos={0 0}, upload=438849]: Done uploading piece. err: <nil> Dec 26 11:30:17.163325181 (500h1m17.034408216s) [noise=true , address=145.79.227.31:4799 , piece_id=DHNGACX5WUD3YY7PKFUMJ3SZK2XI7WS7ESZD4LLVTNBL6K2Q3CPQ, seg_pos={0 0}, upload=438849]: Done uploading piece. err: <nil> Dec 26 11:30:17.166911150 (500h1m17.037994195s) [noise=true , address=145.79.227.42:4122 , piece_id=WM5IV4MTYKYM4SJUPAFUOHLYNJKAE7ZCFV327RIHY3GYG6BOM7PQ, seg_pos={0 0}, upload=438849]: Done uploading piece. err: <nil> Dec 26 11:30:17.168159116 (500h1m17.039242161s) [noise=true , address=145.79.227.39:4013 , piece_id=LJT2PBTVO7XRTS4BLZM4QHGQW3ZQV6FJHICZTI55NJECR6YZFECQ, seg_pos={0 0}, upload=438849]: Done uploading piece. err: <nil> Dec 26 11:30:17.171042407 (500h1m17.042125452s) [noise=true , address=145.79.227.52:4446 , piece_id=ELGUDSP4GF2ICCMYQQD2TQSCJSQ5IYDLQZGZBJAAC34CRATQNFHA, seg_pos={0 0}, upload=438849]: Done uploading piece. err: <nil> Dec 26 11:30:17.183473047 (500h1m17.054556082s) [noise=true , address=145.79.227.41:4087 , piece_id=6ECDUGJINW6FFQLHWVFYN56OFU7QYAMHWXPDDJ3URD7TOC4KTOAA, seg_pos={0 0}, upload=438849]: Done uploading piece. err: <nil> Dec 26 11:30:17.188319034 (500h1m17.059402079s) [noise=true , address=145.79.227.46:4602 , piece_id=2UT5E72P2YCEXJZ6FWLM5RWFVUYDGBTQ4H5IXJDSJDJ5WHSY7NZQ, seg_pos={0 0}, upload=438849]: Done uploading piece. err: <nil> Dec 26 11:30:17.188376233 (500h1m17.059459278s) [noise=true , address=145.79.227.48:4301 , piece_id=2YNXNO5PMSVPDKKMN2ZEFSRJHUBLM2G4XKCV6IEYQMZZLVB5PR6A, seg_pos={0 0}, upload=438849]: Done uploading piece. err: <nil> Dec 26 11:30:17.352678334 (500h1m17.223761379s) [noise=true , address=77.90.31.14:4662 , piece_id=K7CLEXMKD3EVCWHJW2Z6KGDFX26HR4W52GLUCKLG5K4DWGO5F2CA, seg_pos={0 0}, upload=438849]: Done uploading piece. err: <nil> Dec 26 11:30:17.385303118 (500h1m17.256386153s) [noise=true , address=45.91.116.4:4939 , piece_id=KGKZX7W5UBC3IMN6WSQEISB4CK7IAZ7ECCJS526YGYBJGUUEKAAQ, seg_pos={0 0}, upload=438849]: Done uploading piece. err: <nil> Dec 26 11:30:17.418483329 (500h1m17.289566374s) [noise=true , address=45.91.116.13:4154 , piece_id=5X5LBZXMQS4GLANHB4LZ736EAH4TVWPYX3DSWIBVBNH6LBS32I7A, seg_pos={0 0}, upload=438849]: Done uploading piece. err: <nil> Dec 26 11:30:17.421974506 (500h1m17.293057541s) [noise=true , address=45.91.116.6:4975 , piece_id=DSTBEQCKUXUWR2XDRIMEE3E267VXY5GTFPJ4PQXBFUD3SW4JMDSA, seg_pos={0 0}, upload=438849]: Done uploading piece. err: <nil> Dec 26 11:30:17.521180402 (500h1m17.392263438s) [noise=true , address=45.91.116.10:4082 , piece_id=B6TLEJ2RBOWDTM3UVHA3OIM7WWQRHQM3KPKAP6DWQZ2ZF6OJ33CA, seg_pos={0 0}, upload=438849]: Done uploading piece. err: <nil> Dec 26 11:30:17.521200039 (500h1m17.392283084s) [seg_pos={0 0}, upload=438849]: Segment reached optimal threshold of 13 pieces. Dec 26 11:30:17.521427854 (500h1m17.392510899s) [noise=true , address=45.91.116.5:4298 , piece_id=RRVN7ERVO46M5GJPCDA2FDBF2WWZSDG3DUKQNCKSF5KQONMB76AQ, seg_pos={0 0}, upload=438849]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 26 11:30:17.521435518 (500h1m17.392518553s) [noise=true , address=45.91.116.5:4298 , piece_id=RRVN7ERVO46M5GJPCDA2FDBF2WWZSDG3DUKQNCKSF5KQONMB76AQ, seg_pos={0 0}, upload=438849]: Long tail cancellation due to optimal threshold Dec 26 11:30:17.521668923 (500h1m17.392751978s) [noise=true , address=145.79.227.34:4906 , piece_id=F2V55V6UN5MN4CMP7Q5GKWEGWBPZBBWSWS2BKYP5DYGZBYNHORQQ, seg_pos={0 0}, upload=438849]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 26 11:30:17.521692036 (500h1m17.392775091s) [noise=true , address=145.79.227.34:4906 , piece_id=F2V55V6UN5MN4CMP7Q5GKWEGWBPZBBWSWS2BKYP5DYGZBYNHORQQ, seg_pos={0 0}, upload=438849]: Long tail cancellation due to optimal threshold Dec 26 11:30:17.521723326 (500h1m17.392806371s) [noise=true , address=45.91.116.8:4047 , piece_id=5IPJK4IQD554GLHPKUMYX5UCPPVEVXMTKQ6QQK3NAX5KUNHB5LHA, seg_pos={0 0}, upload=438849]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 26 11:30:17.521733265 (500h1m17.39281631s) [noise=true , address=45.91.116.8:4047 , piece_id=5IPJK4IQD554GLHPKUMYX5UCPPVEVXMTKQ6QQK3NAX5KUNHB5LHA, seg_pos={0 0}, upload=438849]: Long tail cancellation due to optimal threshold Dec 26 11:30:17.521769053 (500h1m17.392852099s) [seg_pos={0 0}, upload=438849]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 11:30:17.521914570 (500h1m17.392997625s) [upload=438849]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 11:30:17.521922165 (500h1m17.39300521s) [upload=438849]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 11:30:17.808609149 (500h1m17.679692194s) [upload=438849]: Done uploading segments... Dec 26 11:30:17.808629166 (500h1m17.679712201s) [upload=438849]: Upload finished. err: <nil> Duration: 5.934813629s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 51.496884768s trace: https://jaeger.dev.storj.io/trace/785a90e8be84a595 FAIL (2025-12-26 11:31:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== KXVRQQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.834123964 (499h15m56.705207009s) [noise=true , address=45.91.116.4:4963 , piece_id=U4T64KQ67C3BDKG5TJSTQWUS4WWYCSVPUZOWJ6YANR2IQ2JD4EFQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.834585043 (499h15m56.705668078s) [noise=true , address=145.79.227.9:4390 , piece_id=JDQNHNGABBGANIHKSKP4VHUWIPQBNNEOPC4ZQCKR3K5TTR4N4SGQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.834961800 (499h15m56.706044845s) [noise=true , address=145.79.227.30:4778 , piece_id=BMEAUWANP7FP6CFRULMIAZUV4JNQEXOUN2EAXJY5XOWLDZINWOIA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.835331334 (499h15m56.706414379s) [noise=true , address=77.90.31.18:4838 , piece_id=VYBIKY3MLEE2CXGS2MT23HX5HUFK25FKYKI4XLXPDRNWCC55ANXQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.835738399 (499h15m56.706821434s) [noise=true , address=145.79.227.20:4459 , piece_id=2EUW2AH7FBVZNJ27XVLRXKMXA737DWKAYBL7CR7PRXXRTO2M644A, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.836118653 (499h15m56.707201698s) [noise=true , address=77.90.31.14:4678 , piece_id=XQHQZJUPB24G3BWGFTFDD42NM6K5HJVHUXBXWH5TW7BU25QGHTMA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.836501923 (499h15m56.707584968s) [noise=true , address=45.91.116.6:4988 , piece_id=VPEKJF74D7SQWKR47MAIY5UJQPBJIEU5AQDKLFGATYIRHFREAQ5A, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.836904940 (499h15m56.707987985s) [noise=true , address=64.49.13.169:4711 , piece_id=PVBVY4TVJLPKYBP6A7LBNZADLN25XBDWZFEK2QVN45YHJATIR7TA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.837388641 (499h15m56.708471676s) [noise=true , address=145.79.227.40:4076 , piece_id=Y6WKUYNPDMBUZSNWEYDL2KKNR6WKHSKGP63S6KAG6F2MWMNTVMBQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.837910455 (499h15m56.7089935s) [noise=true , address=77.90.31.11:4571 , piece_id=OERZACIKIG7HBEV3LL42KUAGKN6AWZEXNQSA52FIYTUKDFWCBW7Q, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.838146514 (499h15m56.709229559s) [noise=true , address=64.49.13.26:4945 , piece_id=KXCFTLWR4HSBTP4GJJ4HEPA6BBEMILYTJEMXY4S5WB24HRMQJOKA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.838469329 (499h15m56.709552364s) [noise=true , address=45.91.116.5:4301 , piece_id=E3ZRKZUKM335ARHV3Z5YGLKQYHG6QV4GCCNHAVMN34BM7UQFWIIA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.838690831 (499h15m56.709773866s) [noise=true , address=45.91.116.14:4204 , piece_id=JWL7OOMKTVAP7WEUB5WTL3F6F5IRNID4KIOYJSYZ655PJSNXAQKQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.839143724 (499h15m56.710226759s) [noise=true , address=45.91.116.9:4272 , piece_id=GGFDVKEW5LUCHQQUP6CWJHJQXUETYUG2UISPVP6XQJGSAJW2KP2Q, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.839678982 (499h15m56.710762027s) [noise=true , address=145.79.227.44:4208 , piece_id=ZLIX7XAUMKRUEWWTQST2KLCOWFJSAFSAQNLUCTTOWVZUBESEI23A, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:58.136536405 (499h15m58.00761944s) [upload=438194]: Next returned nil segment Dec 26 10:44:58.136548628 (499h15m58.007631673s) [upload=438194]: Waiting for error group managing segments... Dec 26 10:44:58.469618858 (499h15m58.340701893s) [noise=true , address=64.49.13.26:4945 , piece_id=KXCFTLWR4HSBTP4GJJ4HEPA6BBEMILYTJEMXY4S5WB24HRMQJOKA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.708819055 (499h15m58.5799021s) [noise=true , address=145.79.227.9:4390 , piece_id=JDQNHNGABBGANIHKSKP4VHUWIPQBNNEOPC4ZQCKR3K5TTR4N4SGQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.714466429 (499h15m58.585549474s) [noise=true , address=145.79.227.40:4076 , piece_id=Y6WKUYNPDMBUZSNWEYDL2KKNR6WKHSKGP63S6KAG6F2MWMNTVMBQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.745407696 (499h15m58.616490731s) [noise=true , address=145.79.227.20:4459 , piece_id=2EUW2AH7FBVZNJ27XVLRXKMXA737DWKAYBL7CR7PRXXRTO2M644A, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.888701070 (499h15m58.759784105s) [noise=true , address=77.90.31.18:4838 , piece_id=VYBIKY3MLEE2CXGS2MT23HX5HUFK25FKYKI4XLXPDRNWCC55ANXQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.948150873 (499h15m58.819233918s) [noise=true , address=45.91.116.5:4301 , piece_id=E3ZRKZUKM335ARHV3Z5YGLKQYHG6QV4GCCNHAVMN34BM7UQFWIIA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.971389744 (499h15m58.842472779s) [noise=true , address=77.90.31.19:4868 , piece_id=WH77PZSXC2VUWLQYLBR2NDEVT6KMD3VZK5WAL53AMRDW74P242LQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.021685314 (499h15m58.892768349s) [noise=true , address=45.91.116.4:4963 , piece_id=U4T64KQ67C3BDKG5TJSTQWUS4WWYCSVPUZOWJ6YANR2IQ2JD4EFQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.045418216 (499h15m58.916501251s) [noise=true , address=45.91.116.14:4204 , piece_id=JWL7OOMKTVAP7WEUB5WTL3F6F5IRNID4KIOYJSYZ655PJSNXAQKQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.056260371 (499h15m58.927343416s) [noise=true , address=45.91.116.9:4272 , piece_id=GGFDVKEW5LUCHQQUP6CWJHJQXUETYUG2UISPVP6XQJGSAJW2KP2Q, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.085713054 (499h15m58.956796089s) [noise=true , address=45.91.116.7:4017 , piece_id=L4GGY6AICMSFMUVYGI6ZMUORYRXNJFPLJY5XJRUAM7NVYQOHPHSQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.640980124 (499h15m59.512063169s) [noise=true , address=145.79.227.30:4778 , piece_id=BMEAUWANP7FP6CFRULMIAZUV4JNQEXOUN2EAXJY5XOWLDZINWOIA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.654515607 (499h15m59.525598652s) [noise=true , address=145.79.227.44:4208 , piece_id=ZLIX7XAUMKRUEWWTQST2KLCOWFJSAFSAQNLUCTTOWVZUBESEI23A, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.660349657 (499h15m59.531432712s) [noise=true , address=77.90.31.11:4571 , piece_id=OERZACIKIG7HBEV3LL42KUAGKN6AWZEXNQSA52FIYTUKDFWCBW7Q, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.660376087 (499h15m59.531459132s) [seg_pos={0 0}, upload=438194]: Segment reached optimal threshold of 14 pieces. Dec 26 10:44:59.660612988 (499h15m59.531696033s) [noise=true , address=64.49.13.169:4711 , piece_id=PVBVY4TVJLPKYBP6A7LBNZADLN25XBDWZFEK2QVN45YHJATIR7TA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:52964->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:52964->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52964->64.49.13.169:4711: use of closed network connection Dec 26 10:44:59.660620753 (499h15m59.531703788s) [noise=true , address=64.49.13.169:4711 , piece_id=PVBVY4TVJLPKYBP6A7LBNZADLN25XBDWZFEK2QVN45YHJATIR7TA, seg_pos={0 0}, upload=438194]: Long tail cancellation due to optimal threshold Dec 26 10:44:59.660893663 (499h15m59.531976698s) [noise=true , address=45.91.116.6:4988 , piece_id=VPEKJF74D7SQWKR47MAIY5UJQPBJIEU5AQDKLFGATYIRHFREAQ5A, seg_pos={0 0}, upload=438194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TrcoCHhAJeEykP9hJNpygoPSdgFV5v6GzL48JpyNPiAj5ZJRS): protocol: expected piece hash; context canceled; EOF Dec 26 10:44:59.660901057 (499h15m59.531984102s) [noise=true , address=45.91.116.6:4988 , piece_id=VPEKJF74D7SQWKR47MAIY5UJQPBJIEU5AQDKLFGATYIRHFREAQ5A, seg_pos={0 0}, upload=438194]: Long tail cancellation due to optimal threshold Dec 26 10:44:59.661226155 (499h15m59.5323092s) [noise=true , address=77.90.31.14:4678 , piece_id=XQHQZJUPB24G3BWGFTFDD42NM6K5HJVHUXBXWH5TW7BU25QGHTMA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12beu3SUZMPNWfbeHV4tLtx9UVZoEHh4Kt7oBhT59w3bGBE1ovP): protocol: write tcp 188.166.225.170:35884->77.90.31.14:4678: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 10:44:59.661249630 (499h15m59.532332685s) [noise=true , address=77.90.31.14:4678 , piece_id=XQHQZJUPB24G3BWGFTFDD42NM6K5HJVHUXBXWH5TW7BU25QGHTMA, seg_pos={0 0}, upload=438194]: Long tail cancellation due to optimal threshold Dec 26 10:44:59.661309705 (499h15m59.53239275s) [noise=true , address=77.90.31.21:4909 , piece_id=H6KT2AKZISCU4VEA6TXJJ3EFGCRBZPPXBEJJEIHZR3GEJZKXVRQQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121RT7tP6czMFAiZ3DjzSSsYgPt3xZXVbXVpkcYxaZYscNiMjL6): protocol: expected piece hash; context canceled; EOF Dec 26 10:44:59.661318962 (499h15m59.532401997s) [noise=true , address=77.90.31.21:4909 , piece_id=H6KT2AKZISCU4VEA6TXJJ3EFGCRBZPPXBEJJEIHZR3GEJZKXVRQQ, seg_pos={0 0}, upload=438194]: Long tail cancellation due to optimal threshold Dec 26 10:44:59.661345674 (499h15m59.532428719s) [seg_pos={0 0}, upload=438194]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 10:44:59.661485770 (499h15m59.532568815s) [upload=438194]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 10:44:59.661493655 (499h15m59.53257669s) [upload=438194]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 10:44:59.968467153 (499h15m59.839550198s) [upload=438194]: Done uploading segments... Dec 26 10:44:59.968484155 (499h15m59.83956719s) [upload=438194]: Upload finished. err: <nil> Duration: 51.49690146s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 42.314943637s trace: https://jaeger.dev.storj.io/trace/785a90e8be84a595 FAIL (2025-12-26 11:31:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== KXVRQQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.834123964 (499h15m56.705207009s) [noise=true , address=45.91.116.4:4963 , piece_id=U4T64KQ67C3BDKG5TJSTQWUS4WWYCSVPUZOWJ6YANR2IQ2JD4EFQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.834585043 (499h15m56.705668078s) [noise=true , address=145.79.227.9:4390 , piece_id=JDQNHNGABBGANIHKSKP4VHUWIPQBNNEOPC4ZQCKR3K5TTR4N4SGQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.834961800 (499h15m56.706044845s) [noise=true , address=145.79.227.30:4778 , piece_id=BMEAUWANP7FP6CFRULMIAZUV4JNQEXOUN2EAXJY5XOWLDZINWOIA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.835331334 (499h15m56.706414379s) [noise=true , address=77.90.31.18:4838 , piece_id=VYBIKY3MLEE2CXGS2MT23HX5HUFK25FKYKI4XLXPDRNWCC55ANXQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.835738399 (499h15m56.706821434s) [noise=true , address=145.79.227.20:4459 , piece_id=2EUW2AH7FBVZNJ27XVLRXKMXA737DWKAYBL7CR7PRXXRTO2M644A, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.836118653 (499h15m56.707201698s) [noise=true , address=77.90.31.14:4678 , piece_id=XQHQZJUPB24G3BWGFTFDD42NM6K5HJVHUXBXWH5TW7BU25QGHTMA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.836501923 (499h15m56.707584968s) [noise=true , address=45.91.116.6:4988 , piece_id=VPEKJF74D7SQWKR47MAIY5UJQPBJIEU5AQDKLFGATYIRHFREAQ5A, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.836904940 (499h15m56.707987985s) [noise=true , address=64.49.13.169:4711 , piece_id=PVBVY4TVJLPKYBP6A7LBNZADLN25XBDWZFEK2QVN45YHJATIR7TA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.837388641 (499h15m56.708471676s) [noise=true , address=145.79.227.40:4076 , piece_id=Y6WKUYNPDMBUZSNWEYDL2KKNR6WKHSKGP63S6KAG6F2MWMNTVMBQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.837910455 (499h15m56.7089935s) [noise=true , address=77.90.31.11:4571 , piece_id=OERZACIKIG7HBEV3LL42KUAGKN6AWZEXNQSA52FIYTUKDFWCBW7Q, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.838146514 (499h15m56.709229559s) [noise=true , address=64.49.13.26:4945 , piece_id=KXCFTLWR4HSBTP4GJJ4HEPA6BBEMILYTJEMXY4S5WB24HRMQJOKA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.838469329 (499h15m56.709552364s) [noise=true , address=45.91.116.5:4301 , piece_id=E3ZRKZUKM335ARHV3Z5YGLKQYHG6QV4GCCNHAVMN34BM7UQFWIIA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.838690831 (499h15m56.709773866s) [noise=true , address=45.91.116.14:4204 , piece_id=JWL7OOMKTVAP7WEUB5WTL3F6F5IRNID4KIOYJSYZ655PJSNXAQKQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.839143724 (499h15m56.710226759s) [noise=true , address=45.91.116.9:4272 , piece_id=GGFDVKEW5LUCHQQUP6CWJHJQXUETYUG2UISPVP6XQJGSAJW2KP2Q, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.839678982 (499h15m56.710762027s) [noise=true , address=145.79.227.44:4208 , piece_id=ZLIX7XAUMKRUEWWTQST2KLCOWFJSAFSAQNLUCTTOWVZUBESEI23A, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:58.136536405 (499h15m58.00761944s) [upload=438194]: Next returned nil segment Dec 26 10:44:58.136548628 (499h15m58.007631673s) [upload=438194]: Waiting for error group managing segments... Dec 26 10:44:58.469618858 (499h15m58.340701893s) [noise=true , address=64.49.13.26:4945 , piece_id=KXCFTLWR4HSBTP4GJJ4HEPA6BBEMILYTJEMXY4S5WB24HRMQJOKA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.708819055 (499h15m58.5799021s) [noise=true , address=145.79.227.9:4390 , piece_id=JDQNHNGABBGANIHKSKP4VHUWIPQBNNEOPC4ZQCKR3K5TTR4N4SGQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.714466429 (499h15m58.585549474s) [noise=true , address=145.79.227.40:4076 , piece_id=Y6WKUYNPDMBUZSNWEYDL2KKNR6WKHSKGP63S6KAG6F2MWMNTVMBQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.745407696 (499h15m58.616490731s) [noise=true , address=145.79.227.20:4459 , piece_id=2EUW2AH7FBVZNJ27XVLRXKMXA737DWKAYBL7CR7PRXXRTO2M644A, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.888701070 (499h15m58.759784105s) [noise=true , address=77.90.31.18:4838 , piece_id=VYBIKY3MLEE2CXGS2MT23HX5HUFK25FKYKI4XLXPDRNWCC55ANXQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.948150873 (499h15m58.819233918s) [noise=true , address=45.91.116.5:4301 , piece_id=E3ZRKZUKM335ARHV3Z5YGLKQYHG6QV4GCCNHAVMN34BM7UQFWIIA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.971389744 (499h15m58.842472779s) [noise=true , address=77.90.31.19:4868 , piece_id=WH77PZSXC2VUWLQYLBR2NDEVT6KMD3VZK5WAL53AMRDW74P242LQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.021685314 (499h15m58.892768349s) [noise=true , address=45.91.116.4:4963 , piece_id=U4T64KQ67C3BDKG5TJSTQWUS4WWYCSVPUZOWJ6YANR2IQ2JD4EFQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.045418216 (499h15m58.916501251s) [noise=true , address=45.91.116.14:4204 , piece_id=JWL7OOMKTVAP7WEUB5WTL3F6F5IRNID4KIOYJSYZ655PJSNXAQKQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.056260371 (499h15m58.927343416s) [noise=true , address=45.91.116.9:4272 , piece_id=GGFDVKEW5LUCHQQUP6CWJHJQXUETYUG2UISPVP6XQJGSAJW2KP2Q, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.085713054 (499h15m58.956796089s) [noise=true , address=45.91.116.7:4017 , piece_id=L4GGY6AICMSFMUVYGI6ZMUORYRXNJFPLJY5XJRUAM7NVYQOHPHSQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.640980124 (499h15m59.512063169s) [noise=true , address=145.79.227.30:4778 , piece_id=BMEAUWANP7FP6CFRULMIAZUV4JNQEXOUN2EAXJY5XOWLDZINWOIA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.654515607 (499h15m59.525598652s) [noise=true , address=145.79.227.44:4208 , piece_id=ZLIX7XAUMKRUEWWTQST2KLCOWFJSAFSAQNLUCTTOWVZUBESEI23A, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.660349657 (499h15m59.531432712s) [noise=true , address=77.90.31.11:4571 , piece_id=OERZACIKIG7HBEV3LL42KUAGKN6AWZEXNQSA52FIYTUKDFWCBW7Q, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.660376087 (499h15m59.531459132s) [seg_pos={0 0}, upload=438194]: Segment reached optimal threshold of 14 pieces. Dec 26 10:44:59.660612988 (499h15m59.531696033s) [noise=true , address=64.49.13.169:4711 , piece_id=PVBVY4TVJLPKYBP6A7LBNZADLN25XBDWZFEK2QVN45YHJATIR7TA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:52964->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:52964->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52964->64.49.13.169:4711: use of closed network connection Dec 26 10:44:59.660620753 (499h15m59.531703788s) [noise=true , address=64.49.13.169:4711 , piece_id=PVBVY4TVJLPKYBP6A7LBNZADLN25XBDWZFEK2QVN45YHJATIR7TA, seg_pos={0 0}, upload=438194]: Long tail cancellation due to optimal threshold Dec 26 10:44:59.660893663 (499h15m59.531976698s) [noise=true , address=45.91.116.6:4988 , piece_id=VPEKJF74D7SQWKR47MAIY5UJQPBJIEU5AQDKLFGATYIRHFREAQ5A, seg_pos={0 0}, upload=438194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TrcoCHhAJeEykP9hJNpygoPSdgFV5v6GzL48JpyNPiAj5ZJRS): protocol: expected piece hash; context canceled; EOF Dec 26 10:44:59.660901057 (499h15m59.531984102s) [noise=true , address=45.91.116.6:4988 , piece_id=VPEKJF74D7SQWKR47MAIY5UJQPBJIEU5AQDKLFGATYIRHFREAQ5A, seg_pos={0 0}, upload=438194]: Long tail cancellation due to optimal threshold Dec 26 10:44:59.661226155 (499h15m59.5323092s) [noise=true , address=77.90.31.14:4678 , piece_id=XQHQZJUPB24G3BWGFTFDD42NM6K5HJVHUXBXWH5TW7BU25QGHTMA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12beu3SUZMPNWfbeHV4tLtx9UVZoEHh4Kt7oBhT59w3bGBE1ovP): protocol: write tcp 188.166.225.170:35884->77.90.31.14:4678: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 10:44:59.661249630 (499h15m59.532332685s) [noise=true , address=77.90.31.14:4678 , piece_id=XQHQZJUPB24G3BWGFTFDD42NM6K5HJVHUXBXWH5TW7BU25QGHTMA, seg_pos={0 0}, upload=438194]: Long tail cancellation due to optimal threshold Dec 26 10:44:59.661309705 (499h15m59.53239275s) [noise=true , address=77.90.31.21:4909 , piece_id=H6KT2AKZISCU4VEA6TXJJ3EFGCRBZPPXBEJJEIHZR3GEJZKXVRQQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121RT7tP6czMFAiZ3DjzSSsYgPt3xZXVbXVpkcYxaZYscNiMjL6): protocol: expected piece hash; context canceled; EOF Dec 26 10:44:59.661318962 (499h15m59.532401997s) [noise=true , address=77.90.31.21:4909 , piece_id=H6KT2AKZISCU4VEA6TXJJ3EFGCRBZPPXBEJJEIHZR3GEJZKXVRQQ, seg_pos={0 0}, upload=438194]: Long tail cancellation due to optimal threshold Dec 26 10:44:59.661345674 (499h15m59.532428719s) [seg_pos={0 0}, upload=438194]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 10:44:59.661485770 (499h15m59.532568815s) [upload=438194]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 10:44:59.661493655 (499h15m59.53257669s) [upload=438194]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 10:44:59.968467153 (499h15m59.839550198s) [upload=438194]: Done uploading segments... Dec 26 10:44:59.968484155 (499h15m59.83956719s) [upload=438194]: Upload finished. err: <nil> Duration: 42.314959917s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [.XX.XX..X.......X..X.X......XX......XXX.XX......X.X..XX.X.X.] duration: 5.974054321s trace: https://jaeger.dev.storj.io/trace/3cbd70d9ed10fddd FAIL (2025-12-26 11:30:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: B6TLEJ2RBOWDTM3UVHA3OIM7WWQRHQM3KPKAP6DWQZ2ZF6OJ33CA) context deadline exceeded; piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: 5X5LBZXMQS4GLANHB4LZ736EAH4TVWPYX3DSWIBVBNH6LBS32I7A) context deadline exceeded ======LOGS====== Dec 26 11:30:14.902827760 (500h1m14.773910795s) [upload=438849]: Starting upload Dec 26 11:30:14.902885119 (500h1m14.773968165s) [upload=438849]: Done starting upload Dec 26 11:30:14.902969400 (500h1m14.774052445s) [upload=438849]: Uploading segments... Dec 26 11:30:14.903144593 (500h1m14.774227628s) [upload=438849]: Got next segment. Inline: false Dec 26 11:30:14.903197464 (500h1m14.774280499s) [upload=438849]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 11:30:14.903204277 (500h1m14.774287322s) [upload=438849]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 11:30:15.148960661 (500h1m15.020043696s) [seg_pos={0 0}, upload=438849]: Begin upload segment... Dec 26 11:30:15.149007871 (500h1m15.020090906s) [seg_pos={0 0}, upload=438849]: Done begin upload segment. Dec 26 11:30:15.149035434 (500h1m15.020118469s) [noise=true , address=45.91.116.6:4975 , piece_id=DSTBEQCKUXUWR2XDRIMEE3E267VXY5GTFPJ4PQXBFUD3SW4JMDSA, seg_pos={0 0}, upload=438849]: Uploading piece... Dec 26 11:30:15.149137539 (500h1m15.020220574s) [noise=true , address=145.79.227.48:4301 , piece_id=2YNXNO5PMSVPDKKMN2ZEFSRJHUBLM2G4XKCV6IEYQMZZLVB5PR6A, seg_pos={0 0}, upload=438849]: Uploading piece... Dec 26 11:30:15.149873229 (500h1m15.020956274s) [noise=true , address=145.79.227.31:4799 , piece_id=DHNGACX5WUD3YY7PKFUMJ3SZK2XI7WS7ESZD4LLVTNBL6K2Q3CPQ, seg_pos={0 0}, upload=438849]: Uploading piece... Dec 26 11:30:15.150375936 (500h1m15.021458981s) [noise=true , address=45.91.116.5:4298 , piece_id=RRVN7ERVO46M5GJPCDA2FDBF2WWZSDG3DUKQNCKSF5KQONMB76AQ, seg_pos={0 0}, upload=438849]: Uploading piece... Dec 26 11:30:15.150875928 (500h1m15.021958973s) [noise=true , address=145.79.227.6:4258 , piece_id=BKCG5FZVFBCSI7LHD2YTDY7BFMRTNRD4JMER4CTTEWNY7Q53BI3A, seg_pos={0 0}, upload=438849]: Uploading piece... Dec 26 11:30:15.151287341 (500h1m15.022370376s) [noise=true , address=77.90.31.14:4662 , piece_id=K7CLEXMKD3EVCWHJW2Z6KGDFX26HR4W52GLUCKLG5K4DWGO5F2CA, seg_pos={0 0}, upload=438849]: Uploading piece... Dec 26 11:30:15.151711319 (500h1m15.022794364s) [noise=true , address=45.91.116.4:4939 , piece_id=KGKZX7W5UBC3IMN6WSQEISB4CK7IAZ7ECCJS526YGYBJGUUEKAAQ, seg_pos={0 0}, upload=438849]: Uploading piece... Dec 26 11:30:15.153014350 (500h1m15.024097395s) [noise=true , address=45.91.116.10:4082 , piece_id=B6TLEJ2RBOWDTM3UVHA3OIM7WWQRHQM3KPKAP6DWQZ2ZF6OJ33CA, seg_pos={0 0}, upload=438849]: Uploading piece... Dec 26 11:30:15.153391188 (500h1m15.024474223s) [noise=true , address=145.79.227.46:4602 , piece_id=2UT5E72P2YCEXJZ6FWLM5RWFVUYDGBTQ4H5IXJDSJDJ5WHSY7NZQ, seg_pos={0 0}, upload=438849]: Uploading piece... Dec 26 11:30:15.154893739 (500h1m15.025976784s) [noise=true , address=45.91.116.8:4047 , piece_id=5IPJK4IQD554GLHPKUMYX5UCPPVEVXMTKQ6QQK3NAX5KUNHB5LHA, seg_pos={0 0}, upload=438849]: Uploading piece... Dec 26 11:30:15.155366759 (500h1m15.026449804s) [noise=true , address=145.79.227.39:4013 , piece_id=LJT2PBTVO7XRTS4BLZM4QHGQW3ZQV6FJHICZTI55NJECR6YZFECQ, seg_pos={0 0}, upload=438849]: Uploading piece... Dec 26 11:30:15.157001762 (500h1m15.028084808s) [noise=true , address=145.79.227.52:4446 , piece_id=ELGUDSP4GF2ICCMYQQD2TQSCJSQ5IYDLQZGZBJAAC34CRATQNFHA, seg_pos={0 0}, upload=438849]: Uploading piece... Dec 26 11:30:15.158394304 (500h1m15.029477339s) [noise=true , address=145.79.227.41:4087 , piece_id=6ECDUGJINW6FFQLHWVFYN56OFU7QYAMHWXPDDJ3URD7TOC4KTOAA, seg_pos={0 0}, upload=438849]: Uploading piece... Dec 26 11:30:15.160400314 (500h1m15.031483359s) [noise=true , address=145.79.227.34:4906 , piece_id=F2V55V6UN5MN4CMP7Q5GKWEGWBPZBBWSWS2BKYP5DYGZBYNHORQQ, seg_pos={0 0}, upload=438849]: Uploading piece... Dec 26 11:30:15.161370011 (500h1m15.032453056s) [noise=true , address=45.91.116.13:4154 , piece_id=5X5LBZXMQS4GLANHB4LZ736EAH4TVWPYX3DSWIBVBNH6LBS32I7A, seg_pos={0 0}, upload=438849]: Uploading piece... Dec 26 11:30:15.163119021 (500h1m15.034202066s) [noise=true , address=145.79.227.42:4122 , piece_id=WM5IV4MTYKYM4SJUPAFUOHLYNJKAE7ZCFV327RIHY3GYG6BOM7PQ, seg_pos={0 0}, upload=438849]: Uploading piece... Dec 26 11:30:16.875538755 (500h1m16.74662179s) [upload=438849]: Next returned nil segment Dec 26 11:30:16.875553864 (500h1m16.746636909s) [upload=438849]: Waiting for error group managing segments... Dec 26 11:30:17.161371171 (500h1m17.032454216s) [noise=true , address=145.79.227.6:4258 , piece_id=BKCG5FZVFBCSI7LHD2YTDY7BFMRTNRD4JMER4CTTEWNY7Q53BI3A, seg_pos={0 0}, upload=438849]: Done uploading piece. err: <nil> Dec 26 11:30:17.163325181 (500h1m17.034408216s) [noise=true , address=145.79.227.31:4799 , piece_id=DHNGACX5WUD3YY7PKFUMJ3SZK2XI7WS7ESZD4LLVTNBL6K2Q3CPQ, seg_pos={0 0}, upload=438849]: Done uploading piece. err: <nil> Dec 26 11:30:17.166911150 (500h1m17.037994195s) [noise=true , address=145.79.227.42:4122 , piece_id=WM5IV4MTYKYM4SJUPAFUOHLYNJKAE7ZCFV327RIHY3GYG6BOM7PQ, seg_pos={0 0}, upload=438849]: Done uploading piece. err: <nil> Dec 26 11:30:17.168159116 (500h1m17.039242161s) [noise=true , address=145.79.227.39:4013 , piece_id=LJT2PBTVO7XRTS4BLZM4QHGQW3ZQV6FJHICZTI55NJECR6YZFECQ, seg_pos={0 0}, upload=438849]: Done uploading piece. err: <nil> Dec 26 11:30:17.171042407 (500h1m17.042125452s) [noise=true , address=145.79.227.52:4446 , piece_id=ELGUDSP4GF2ICCMYQQD2TQSCJSQ5IYDLQZGZBJAAC34CRATQNFHA, seg_pos={0 0}, upload=438849]: Done uploading piece. err: <nil> Dec 26 11:30:17.183473047 (500h1m17.054556082s) [noise=true , address=145.79.227.41:4087 , piece_id=6ECDUGJINW6FFQLHWVFYN56OFU7QYAMHWXPDDJ3URD7TOC4KTOAA, seg_pos={0 0}, upload=438849]: Done uploading piece. err: <nil> Dec 26 11:30:17.188319034 (500h1m17.059402079s) [noise=true , address=145.79.227.46:4602 , piece_id=2UT5E72P2YCEXJZ6FWLM5RWFVUYDGBTQ4H5IXJDSJDJ5WHSY7NZQ, seg_pos={0 0}, upload=438849]: Done uploading piece. err: <nil> Dec 26 11:30:17.188376233 (500h1m17.059459278s) [noise=true , address=145.79.227.48:4301 , piece_id=2YNXNO5PMSVPDKKMN2ZEFSRJHUBLM2G4XKCV6IEYQMZZLVB5PR6A, seg_pos={0 0}, upload=438849]: Done uploading piece. err: <nil> Dec 26 11:30:17.352678334 (500h1m17.223761379s) [noise=true , address=77.90.31.14:4662 , piece_id=K7CLEXMKD3EVCWHJW2Z6KGDFX26HR4W52GLUCKLG5K4DWGO5F2CA, seg_pos={0 0}, upload=438849]: Done uploading piece. err: <nil> Dec 26 11:30:17.385303118 (500h1m17.256386153s) [noise=true , address=45.91.116.4:4939 , piece_id=KGKZX7W5UBC3IMN6WSQEISB4CK7IAZ7ECCJS526YGYBJGUUEKAAQ, seg_pos={0 0}, upload=438849]: Done uploading piece. err: <nil> Dec 26 11:30:17.418483329 (500h1m17.289566374s) [noise=true , address=45.91.116.13:4154 , piece_id=5X5LBZXMQS4GLANHB4LZ736EAH4TVWPYX3DSWIBVBNH6LBS32I7A, seg_pos={0 0}, upload=438849]: Done uploading piece. err: <nil> Dec 26 11:30:17.421974506 (500h1m17.293057541s) [noise=true , address=45.91.116.6:4975 , piece_id=DSTBEQCKUXUWR2XDRIMEE3E267VXY5GTFPJ4PQXBFUD3SW4JMDSA, seg_pos={0 0}, upload=438849]: Done uploading piece. err: <nil> Dec 26 11:30:17.521180402 (500h1m17.392263438s) [noise=true , address=45.91.116.10:4082 , piece_id=B6TLEJ2RBOWDTM3UVHA3OIM7WWQRHQM3KPKAP6DWQZ2ZF6OJ33CA, seg_pos={0 0}, upload=438849]: Done uploading piece. err: <nil> Dec 26 11:30:17.521200039 (500h1m17.392283084s) [seg_pos={0 0}, upload=438849]: Segment reached optimal threshold of 13 pieces. Dec 26 11:30:17.521427854 (500h1m17.392510899s) [noise=true , address=45.91.116.5:4298 , piece_id=RRVN7ERVO46M5GJPCDA2FDBF2WWZSDG3DUKQNCKSF5KQONMB76AQ, seg_pos={0 0}, upload=438849]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 26 11:30:17.521435518 (500h1m17.392518553s) [noise=true , address=45.91.116.5:4298 , piece_id=RRVN7ERVO46M5GJPCDA2FDBF2WWZSDG3DUKQNCKSF5KQONMB76AQ, seg_pos={0 0}, upload=438849]: Long tail cancellation due to optimal threshold Dec 26 11:30:17.521668923 (500h1m17.392751978s) [noise=true , address=145.79.227.34:4906 , piece_id=F2V55V6UN5MN4CMP7Q5GKWEGWBPZBBWSWS2BKYP5DYGZBYNHORQQ, seg_pos={0 0}, upload=438849]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 26 11:30:17.521692036 (500h1m17.392775091s) [noise=true , address=145.79.227.34:4906 , piece_id=F2V55V6UN5MN4CMP7Q5GKWEGWBPZBBWSWS2BKYP5DYGZBYNHORQQ, seg_pos={0 0}, upload=438849]: Long tail cancellation due to optimal threshold Dec 26 11:30:17.521723326 (500h1m17.392806371s) [noise=true , address=45.91.116.8:4047 , piece_id=5IPJK4IQD554GLHPKUMYX5UCPPVEVXMTKQ6QQK3NAX5KUNHB5LHA, seg_pos={0 0}, upload=438849]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 26 11:30:17.521733265 (500h1m17.39281631s) [noise=true , address=45.91.116.8:4047 , piece_id=5IPJK4IQD554GLHPKUMYX5UCPPVEVXMTKQ6QQK3NAX5KUNHB5LHA, seg_pos={0 0}, upload=438849]: Long tail cancellation due to optimal threshold Dec 26 11:30:17.521769053 (500h1m17.392852099s) [seg_pos={0 0}, upload=438849]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 11:30:17.521914570 (500h1m17.392997625s) [upload=438849]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 11:30:17.521922165 (500h1m17.39300521s) [upload=438849]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 11:30:17.808609149 (500h1m17.679692194s) [upload=438849]: Done uploading segments... Dec 26 11:30:17.808629166 (500h1m17.679712201s) [upload=438849]: Upload finished. err: <nil> Duration: 5.974074168s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [..XX.XX..X.......X..X.X......XX......XXX.XX......X.X..XX.X.X] duration: 30.002695288s trace: https://jaeger.dev.storj.io/trace/3cbd70d9ed10fddd FAIL (2025-12-26 11:30:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: B6TLEJ2RBOWDTM3UVHA3OIM7WWQRHQM3KPKAP6DWQZ2ZF6OJ33CA) context deadline exceeded; piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: 5X5LBZXMQS4GLANHB4LZ736EAH4TVWPYX3DSWIBVBNH6LBS32I7A) context deadline exceeded ======LOGS====== Dec 26 11:30:14.902827760 (500h1m14.773910795s) [upload=438849]: Starting upload Dec 26 11:30:14.902885119 (500h1m14.773968165s) [upload=438849]: Done starting upload Dec 26 11:30:14.902969400 (500h1m14.774052445s) [upload=438849]: Uploading segments... Dec 26 11:30:14.903144593 (500h1m14.774227628s) [upload=438849]: Got next segment. Inline: false Dec 26 11:30:14.903197464 (500h1m14.774280499s) [upload=438849]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 11:30:14.903204277 (500h1m14.774287322s) [upload=438849]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 11:30:15.148960661 (500h1m15.020043696s) [seg_pos={0 0}, upload=438849]: Begin upload segment... Dec 26 11:30:15.149007871 (500h1m15.020090906s) [seg_pos={0 0}, upload=438849]: Done begin upload segment. Dec 26 11:30:15.149035434 (500h1m15.020118469s) [noise=true , address=45.91.116.6:4975 , piece_id=DSTBEQCKUXUWR2XDRIMEE3E267VXY5GTFPJ4PQXBFUD3SW4JMDSA, seg_pos={0 0}, upload=438849]: Uploading piece... Dec 26 11:30:15.149137539 (500h1m15.020220574s) [noise=true , address=145.79.227.48:4301 , piece_id=2YNXNO5PMSVPDKKMN2ZEFSRJHUBLM2G4XKCV6IEYQMZZLVB5PR6A, seg_pos={0 0}, upload=438849]: Uploading piece... Dec 26 11:30:15.149873229 (500h1m15.020956274s) [noise=true , address=145.79.227.31:4799 , piece_id=DHNGACX5WUD3YY7PKFUMJ3SZK2XI7WS7ESZD4LLVTNBL6K2Q3CPQ, seg_pos={0 0}, upload=438849]: Uploading piece... Dec 26 11:30:15.150375936 (500h1m15.021458981s) [noise=true , address=45.91.116.5:4298 , piece_id=RRVN7ERVO46M5GJPCDA2FDBF2WWZSDG3DUKQNCKSF5KQONMB76AQ, seg_pos={0 0}, upload=438849]: Uploading piece... Dec 26 11:30:15.150875928 (500h1m15.021958973s) [noise=true , address=145.79.227.6:4258 , piece_id=BKCG5FZVFBCSI7LHD2YTDY7BFMRTNRD4JMER4CTTEWNY7Q53BI3A, seg_pos={0 0}, upload=438849]: Uploading piece... Dec 26 11:30:15.151287341 (500h1m15.022370376s) [noise=true , address=77.90.31.14:4662 , piece_id=K7CLEXMKD3EVCWHJW2Z6KGDFX26HR4W52GLUCKLG5K4DWGO5F2CA, seg_pos={0 0}, upload=438849]: Uploading piece... Dec 26 11:30:15.151711319 (500h1m15.022794364s) [noise=true , address=45.91.116.4:4939 , piece_id=KGKZX7W5UBC3IMN6WSQEISB4CK7IAZ7ECCJS526YGYBJGUUEKAAQ, seg_pos={0 0}, upload=438849]: Uploading piece... Dec 26 11:30:15.153014350 (500h1m15.024097395s) [noise=true , address=45.91.116.10:4082 , piece_id=B6TLEJ2RBOWDTM3UVHA3OIM7WWQRHQM3KPKAP6DWQZ2ZF6OJ33CA, seg_pos={0 0}, upload=438849]: Uploading piece... Dec 26 11:30:15.153391188 (500h1m15.024474223s) [noise=true , address=145.79.227.46:4602 , piece_id=2UT5E72P2YCEXJZ6FWLM5RWFVUYDGBTQ4H5IXJDSJDJ5WHSY7NZQ, seg_pos={0 0}, upload=438849]: Uploading piece... Dec 26 11:30:15.154893739 (500h1m15.025976784s) [noise=true , address=45.91.116.8:4047 , piece_id=5IPJK4IQD554GLHPKUMYX5UCPPVEVXMTKQ6QQK3NAX5KUNHB5LHA, seg_pos={0 0}, upload=438849]: Uploading piece... Dec 26 11:30:15.155366759 (500h1m15.026449804s) [noise=true , address=145.79.227.39:4013 , piece_id=LJT2PBTVO7XRTS4BLZM4QHGQW3ZQV6FJHICZTI55NJECR6YZFECQ, seg_pos={0 0}, upload=438849]: Uploading piece... Dec 26 11:30:15.157001762 (500h1m15.028084808s) [noise=true , address=145.79.227.52:4446 , piece_id=ELGUDSP4GF2ICCMYQQD2TQSCJSQ5IYDLQZGZBJAAC34CRATQNFHA, seg_pos={0 0}, upload=438849]: Uploading piece... Dec 26 11:30:15.158394304 (500h1m15.029477339s) [noise=true , address=145.79.227.41:4087 , piece_id=6ECDUGJINW6FFQLHWVFYN56OFU7QYAMHWXPDDJ3URD7TOC4KTOAA, seg_pos={0 0}, upload=438849]: Uploading piece... Dec 26 11:30:15.160400314 (500h1m15.031483359s) [noise=true , address=145.79.227.34:4906 , piece_id=F2V55V6UN5MN4CMP7Q5GKWEGWBPZBBWSWS2BKYP5DYGZBYNHORQQ, seg_pos={0 0}, upload=438849]: Uploading piece... Dec 26 11:30:15.161370011 (500h1m15.032453056s) [noise=true , address=45.91.116.13:4154 , piece_id=5X5LBZXMQS4GLANHB4LZ736EAH4TVWPYX3DSWIBVBNH6LBS32I7A, seg_pos={0 0}, upload=438849]: Uploading piece... Dec 26 11:30:15.163119021 (500h1m15.034202066s) [noise=true , address=145.79.227.42:4122 , piece_id=WM5IV4MTYKYM4SJUPAFUOHLYNJKAE7ZCFV327RIHY3GYG6BOM7PQ, seg_pos={0 0}, upload=438849]: Uploading piece... Dec 26 11:30:16.875538755 (500h1m16.74662179s) [upload=438849]: Next returned nil segment Dec 26 11:30:16.875553864 (500h1m16.746636909s) [upload=438849]: Waiting for error group managing segments... Dec 26 11:30:17.161371171 (500h1m17.032454216s) [noise=true , address=145.79.227.6:4258 , piece_id=BKCG5FZVFBCSI7LHD2YTDY7BFMRTNRD4JMER4CTTEWNY7Q53BI3A, seg_pos={0 0}, upload=438849]: Done uploading piece. err: <nil> Dec 26 11:30:17.163325181 (500h1m17.034408216s) [noise=true , address=145.79.227.31:4799 , piece_id=DHNGACX5WUD3YY7PKFUMJ3SZK2XI7WS7ESZD4LLVTNBL6K2Q3CPQ, seg_pos={0 0}, upload=438849]: Done uploading piece. err: <nil> Dec 26 11:30:17.166911150 (500h1m17.037994195s) [noise=true , address=145.79.227.42:4122 , piece_id=WM5IV4MTYKYM4SJUPAFUOHLYNJKAE7ZCFV327RIHY3GYG6BOM7PQ, seg_pos={0 0}, upload=438849]: Done uploading piece. err: <nil> Dec 26 11:30:17.168159116 (500h1m17.039242161s) [noise=true , address=145.79.227.39:4013 , piece_id=LJT2PBTVO7XRTS4BLZM4QHGQW3ZQV6FJHICZTI55NJECR6YZFECQ, seg_pos={0 0}, upload=438849]: Done uploading piece. err: <nil> Dec 26 11:30:17.171042407 (500h1m17.042125452s) [noise=true , address=145.79.227.52:4446 , piece_id=ELGUDSP4GF2ICCMYQQD2TQSCJSQ5IYDLQZGZBJAAC34CRATQNFHA, seg_pos={0 0}, upload=438849]: Done uploading piece. err: <nil> Dec 26 11:30:17.183473047 (500h1m17.054556082s) [noise=true , address=145.79.227.41:4087 , piece_id=6ECDUGJINW6FFQLHWVFYN56OFU7QYAMHWXPDDJ3URD7TOC4KTOAA, seg_pos={0 0}, upload=438849]: Done uploading piece. err: <nil> Dec 26 11:30:17.188319034 (500h1m17.059402079s) [noise=true , address=145.79.227.46:4602 , piece_id=2UT5E72P2YCEXJZ6FWLM5RWFVUYDGBTQ4H5IXJDSJDJ5WHSY7NZQ, seg_pos={0 0}, upload=438849]: Done uploading piece. err: <nil> Dec 26 11:30:17.188376233 (500h1m17.059459278s) [noise=true , address=145.79.227.48:4301 , piece_id=2YNXNO5PMSVPDKKMN2ZEFSRJHUBLM2G4XKCV6IEYQMZZLVB5PR6A, seg_pos={0 0}, upload=438849]: Done uploading piece. err: <nil> Dec 26 11:30:17.352678334 (500h1m17.223761379s) [noise=true , address=77.90.31.14:4662 , piece_id=K7CLEXMKD3EVCWHJW2Z6KGDFX26HR4W52GLUCKLG5K4DWGO5F2CA, seg_pos={0 0}, upload=438849]: Done uploading piece. err: <nil> Dec 26 11:30:17.385303118 (500h1m17.256386153s) [noise=true , address=45.91.116.4:4939 , piece_id=KGKZX7W5UBC3IMN6WSQEISB4CK7IAZ7ECCJS526YGYBJGUUEKAAQ, seg_pos={0 0}, upload=438849]: Done uploading piece. err: <nil> Dec 26 11:30:17.418483329 (500h1m17.289566374s) [noise=true , address=45.91.116.13:4154 , piece_id=5X5LBZXMQS4GLANHB4LZ736EAH4TVWPYX3DSWIBVBNH6LBS32I7A, seg_pos={0 0}, upload=438849]: Done uploading piece. err: <nil> Dec 26 11:30:17.421974506 (500h1m17.293057541s) [noise=true , address=45.91.116.6:4975 , piece_id=DSTBEQCKUXUWR2XDRIMEE3E267VXY5GTFPJ4PQXBFUD3SW4JMDSA, seg_pos={0 0}, upload=438849]: Done uploading piece. err: <nil> Dec 26 11:30:17.521180402 (500h1m17.392263438s) [noise=true , address=45.91.116.10:4082 , piece_id=B6TLEJ2RBOWDTM3UVHA3OIM7WWQRHQM3KPKAP6DWQZ2ZF6OJ33CA, seg_pos={0 0}, upload=438849]: Done uploading piece. err: <nil> Dec 26 11:30:17.521200039 (500h1m17.392283084s) [seg_pos={0 0}, upload=438849]: Segment reached optimal threshold of 13 pieces. Dec 26 11:30:17.521427854 (500h1m17.392510899s) [noise=true , address=45.91.116.5:4298 , piece_id=RRVN7ERVO46M5GJPCDA2FDBF2WWZSDG3DUKQNCKSF5KQONMB76AQ, seg_pos={0 0}, upload=438849]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 26 11:30:17.521435518 (500h1m17.392518553s) [noise=true , address=45.91.116.5:4298 , piece_id=RRVN7ERVO46M5GJPCDA2FDBF2WWZSDG3DUKQNCKSF5KQONMB76AQ, seg_pos={0 0}, upload=438849]: Long tail cancellation due to optimal threshold Dec 26 11:30:17.521668923 (500h1m17.392751978s) [noise=true , address=145.79.227.34:4906 , piece_id=F2V55V6UN5MN4CMP7Q5GKWEGWBPZBBWSWS2BKYP5DYGZBYNHORQQ, seg_pos={0 0}, upload=438849]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 26 11:30:17.521692036 (500h1m17.392775091s) [noise=true , address=145.79.227.34:4906 , piece_id=F2V55V6UN5MN4CMP7Q5GKWEGWBPZBBWSWS2BKYP5DYGZBYNHORQQ, seg_pos={0 0}, upload=438849]: Long tail cancellation due to optimal threshold Dec 26 11:30:17.521723326 (500h1m17.392806371s) [noise=true , address=45.91.116.8:4047 , piece_id=5IPJK4IQD554GLHPKUMYX5UCPPVEVXMTKQ6QQK3NAX5KUNHB5LHA, seg_pos={0 0}, upload=438849]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 26 11:30:17.521733265 (500h1m17.39281631s) [noise=true , address=45.91.116.8:4047 , piece_id=5IPJK4IQD554GLHPKUMYX5UCPPVEVXMTKQ6QQK3NAX5KUNHB5LHA, seg_pos={0 0}, upload=438849]: Long tail cancellation due to optimal threshold Dec 26 11:30:17.521769053 (500h1m17.392852099s) [seg_pos={0 0}, upload=438849]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 11:30:17.521914570 (500h1m17.392997625s) [upload=438849]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 11:30:17.521922165 (500h1m17.39300521s) [upload=438849]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 11:30:17.808609149 (500h1m17.679692194s) [upload=438849]: Done uploading segments... Dec 26 11:30:17.808629166 (500h1m17.679712201s) [upload=438849]: Upload finished. err: <nil> Duration: 30.002720856s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 45.922685197s trace: https://jaeger.dev.storj.io/trace/785a90e8be84a595 FAIL (2025-12-26 11:30:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== KXVRQQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.834123964 (499h15m56.705207009s) [noise=true , address=45.91.116.4:4963 , piece_id=U4T64KQ67C3BDKG5TJSTQWUS4WWYCSVPUZOWJ6YANR2IQ2JD4EFQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.834585043 (499h15m56.705668078s) [noise=true , address=145.79.227.9:4390 , piece_id=JDQNHNGABBGANIHKSKP4VHUWIPQBNNEOPC4ZQCKR3K5TTR4N4SGQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.834961800 (499h15m56.706044845s) [noise=true , address=145.79.227.30:4778 , piece_id=BMEAUWANP7FP6CFRULMIAZUV4JNQEXOUN2EAXJY5XOWLDZINWOIA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.835331334 (499h15m56.706414379s) [noise=true , address=77.90.31.18:4838 , piece_id=VYBIKY3MLEE2CXGS2MT23HX5HUFK25FKYKI4XLXPDRNWCC55ANXQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.835738399 (499h15m56.706821434s) [noise=true , address=145.79.227.20:4459 , piece_id=2EUW2AH7FBVZNJ27XVLRXKMXA737DWKAYBL7CR7PRXXRTO2M644A, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.836118653 (499h15m56.707201698s) [noise=true , address=77.90.31.14:4678 , piece_id=XQHQZJUPB24G3BWGFTFDD42NM6K5HJVHUXBXWH5TW7BU25QGHTMA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.836501923 (499h15m56.707584968s) [noise=true , address=45.91.116.6:4988 , piece_id=VPEKJF74D7SQWKR47MAIY5UJQPBJIEU5AQDKLFGATYIRHFREAQ5A, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.836904940 (499h15m56.707987985s) [noise=true , address=64.49.13.169:4711 , piece_id=PVBVY4TVJLPKYBP6A7LBNZADLN25XBDWZFEK2QVN45YHJATIR7TA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.837388641 (499h15m56.708471676s) [noise=true , address=145.79.227.40:4076 , piece_id=Y6WKUYNPDMBUZSNWEYDL2KKNR6WKHSKGP63S6KAG6F2MWMNTVMBQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.837910455 (499h15m56.7089935s) [noise=true , address=77.90.31.11:4571 , piece_id=OERZACIKIG7HBEV3LL42KUAGKN6AWZEXNQSA52FIYTUKDFWCBW7Q, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.838146514 (499h15m56.709229559s) [noise=true , address=64.49.13.26:4945 , piece_id=KXCFTLWR4HSBTP4GJJ4HEPA6BBEMILYTJEMXY4S5WB24HRMQJOKA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.838469329 (499h15m56.709552364s) [noise=true , address=45.91.116.5:4301 , piece_id=E3ZRKZUKM335ARHV3Z5YGLKQYHG6QV4GCCNHAVMN34BM7UQFWIIA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.838690831 (499h15m56.709773866s) [noise=true , address=45.91.116.14:4204 , piece_id=JWL7OOMKTVAP7WEUB5WTL3F6F5IRNID4KIOYJSYZ655PJSNXAQKQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.839143724 (499h15m56.710226759s) [noise=true , address=45.91.116.9:4272 , piece_id=GGFDVKEW5LUCHQQUP6CWJHJQXUETYUG2UISPVP6XQJGSAJW2KP2Q, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.839678982 (499h15m56.710762027s) [noise=true , address=145.79.227.44:4208 , piece_id=ZLIX7XAUMKRUEWWTQST2KLCOWFJSAFSAQNLUCTTOWVZUBESEI23A, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:58.136536405 (499h15m58.00761944s) [upload=438194]: Next returned nil segment Dec 26 10:44:58.136548628 (499h15m58.007631673s) [upload=438194]: Waiting for error group managing segments... Dec 26 10:44:58.469618858 (499h15m58.340701893s) [noise=true , address=64.49.13.26:4945 , piece_id=KXCFTLWR4HSBTP4GJJ4HEPA6BBEMILYTJEMXY4S5WB24HRMQJOKA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.708819055 (499h15m58.5799021s) [noise=true , address=145.79.227.9:4390 , piece_id=JDQNHNGABBGANIHKSKP4VHUWIPQBNNEOPC4ZQCKR3K5TTR4N4SGQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.714466429 (499h15m58.585549474s) [noise=true , address=145.79.227.40:4076 , piece_id=Y6WKUYNPDMBUZSNWEYDL2KKNR6WKHSKGP63S6KAG6F2MWMNTVMBQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.745407696 (499h15m58.616490731s) [noise=true , address=145.79.227.20:4459 , piece_id=2EUW2AH7FBVZNJ27XVLRXKMXA737DWKAYBL7CR7PRXXRTO2M644A, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.888701070 (499h15m58.759784105s) [noise=true , address=77.90.31.18:4838 , piece_id=VYBIKY3MLEE2CXGS2MT23HX5HUFK25FKYKI4XLXPDRNWCC55ANXQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.948150873 (499h15m58.819233918s) [noise=true , address=45.91.116.5:4301 , piece_id=E3ZRKZUKM335ARHV3Z5YGLKQYHG6QV4GCCNHAVMN34BM7UQFWIIA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.971389744 (499h15m58.842472779s) [noise=true , address=77.90.31.19:4868 , piece_id=WH77PZSXC2VUWLQYLBR2NDEVT6KMD3VZK5WAL53AMRDW74P242LQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.021685314 (499h15m58.892768349s) [noise=true , address=45.91.116.4:4963 , piece_id=U4T64KQ67C3BDKG5TJSTQWUS4WWYCSVPUZOWJ6YANR2IQ2JD4EFQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.045418216 (499h15m58.916501251s) [noise=true , address=45.91.116.14:4204 , piece_id=JWL7OOMKTVAP7WEUB5WTL3F6F5IRNID4KIOYJSYZ655PJSNXAQKQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.056260371 (499h15m58.927343416s) [noise=true , address=45.91.116.9:4272 , piece_id=GGFDVKEW5LUCHQQUP6CWJHJQXUETYUG2UISPVP6XQJGSAJW2KP2Q, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.085713054 (499h15m58.956796089s) [noise=true , address=45.91.116.7:4017 , piece_id=L4GGY6AICMSFMUVYGI6ZMUORYRXNJFPLJY5XJRUAM7NVYQOHPHSQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.640980124 (499h15m59.512063169s) [noise=true , address=145.79.227.30:4778 , piece_id=BMEAUWANP7FP6CFRULMIAZUV4JNQEXOUN2EAXJY5XOWLDZINWOIA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.654515607 (499h15m59.525598652s) [noise=true , address=145.79.227.44:4208 , piece_id=ZLIX7XAUMKRUEWWTQST2KLCOWFJSAFSAQNLUCTTOWVZUBESEI23A, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.660349657 (499h15m59.531432712s) [noise=true , address=77.90.31.11:4571 , piece_id=OERZACIKIG7HBEV3LL42KUAGKN6AWZEXNQSA52FIYTUKDFWCBW7Q, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.660376087 (499h15m59.531459132s) [seg_pos={0 0}, upload=438194]: Segment reached optimal threshold of 14 pieces. Dec 26 10:44:59.660612988 (499h15m59.531696033s) [noise=true , address=64.49.13.169:4711 , piece_id=PVBVY4TVJLPKYBP6A7LBNZADLN25XBDWZFEK2QVN45YHJATIR7TA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:52964->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:52964->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52964->64.49.13.169:4711: use of closed network connection Dec 26 10:44:59.660620753 (499h15m59.531703788s) [noise=true , address=64.49.13.169:4711 , piece_id=PVBVY4TVJLPKYBP6A7LBNZADLN25XBDWZFEK2QVN45YHJATIR7TA, seg_pos={0 0}, upload=438194]: Long tail cancellation due to optimal threshold Dec 26 10:44:59.660893663 (499h15m59.531976698s) [noise=true , address=45.91.116.6:4988 , piece_id=VPEKJF74D7SQWKR47MAIY5UJQPBJIEU5AQDKLFGATYIRHFREAQ5A, seg_pos={0 0}, upload=438194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TrcoCHhAJeEykP9hJNpygoPSdgFV5v6GzL48JpyNPiAj5ZJRS): protocol: expected piece hash; context canceled; EOF Dec 26 10:44:59.660901057 (499h15m59.531984102s) [noise=true , address=45.91.116.6:4988 , piece_id=VPEKJF74D7SQWKR47MAIY5UJQPBJIEU5AQDKLFGATYIRHFREAQ5A, seg_pos={0 0}, upload=438194]: Long tail cancellation due to optimal threshold Dec 26 10:44:59.661226155 (499h15m59.5323092s) [noise=true , address=77.90.31.14:4678 , piece_id=XQHQZJUPB24G3BWGFTFDD42NM6K5HJVHUXBXWH5TW7BU25QGHTMA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12beu3SUZMPNWfbeHV4tLtx9UVZoEHh4Kt7oBhT59w3bGBE1ovP): protocol: write tcp 188.166.225.170:35884->77.90.31.14:4678: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 10:44:59.661249630 (499h15m59.532332685s) [noise=true , address=77.90.31.14:4678 , piece_id=XQHQZJUPB24G3BWGFTFDD42NM6K5HJVHUXBXWH5TW7BU25QGHTMA, seg_pos={0 0}, upload=438194]: Long tail cancellation due to optimal threshold Dec 26 10:44:59.661309705 (499h15m59.53239275s) [noise=true , address=77.90.31.21:4909 , piece_id=H6KT2AKZISCU4VEA6TXJJ3EFGCRBZPPXBEJJEIHZR3GEJZKXVRQQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121RT7tP6czMFAiZ3DjzSSsYgPt3xZXVbXVpkcYxaZYscNiMjL6): protocol: expected piece hash; context canceled; EOF Dec 26 10:44:59.661318962 (499h15m59.532401997s) [noise=true , address=77.90.31.21:4909 , piece_id=H6KT2AKZISCU4VEA6TXJJ3EFGCRBZPPXBEJJEIHZR3GEJZKXVRQQ, seg_pos={0 0}, upload=438194]: Long tail cancellation due to optimal threshold Dec 26 10:44:59.661345674 (499h15m59.532428719s) [seg_pos={0 0}, upload=438194]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 10:44:59.661485770 (499h15m59.532568815s) [upload=438194]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 10:44:59.661493655 (499h15m59.53257669s) [upload=438194]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 10:44:59.968467153 (499h15m59.839550198s) [upload=438194]: Done uploading segments... Dec 26 10:44:59.968484155 (499h15m59.83956719s) [upload=438194]: Upload finished. err: <nil> Duration: 45.922706006s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 53.268848405s trace: https://jaeger.dev.storj.io/trace/785a90e8be84a595 FAIL (2025-12-26 11:29:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== KXVRQQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.834123964 (499h15m56.705207009s) [noise=true , address=45.91.116.4:4963 , piece_id=U4T64KQ67C3BDKG5TJSTQWUS4WWYCSVPUZOWJ6YANR2IQ2JD4EFQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.834585043 (499h15m56.705668078s) [noise=true , address=145.79.227.9:4390 , piece_id=JDQNHNGABBGANIHKSKP4VHUWIPQBNNEOPC4ZQCKR3K5TTR4N4SGQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.834961800 (499h15m56.706044845s) [noise=true , address=145.79.227.30:4778 , piece_id=BMEAUWANP7FP6CFRULMIAZUV4JNQEXOUN2EAXJY5XOWLDZINWOIA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.835331334 (499h15m56.706414379s) [noise=true , address=77.90.31.18:4838 , piece_id=VYBIKY3MLEE2CXGS2MT23HX5HUFK25FKYKI4XLXPDRNWCC55ANXQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.835738399 (499h15m56.706821434s) [noise=true , address=145.79.227.20:4459 , piece_id=2EUW2AH7FBVZNJ27XVLRXKMXA737DWKAYBL7CR7PRXXRTO2M644A, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.836118653 (499h15m56.707201698s) [noise=true , address=77.90.31.14:4678 , piece_id=XQHQZJUPB24G3BWGFTFDD42NM6K5HJVHUXBXWH5TW7BU25QGHTMA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.836501923 (499h15m56.707584968s) [noise=true , address=45.91.116.6:4988 , piece_id=VPEKJF74D7SQWKR47MAIY5UJQPBJIEU5AQDKLFGATYIRHFREAQ5A, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.836904940 (499h15m56.707987985s) [noise=true , address=64.49.13.169:4711 , piece_id=PVBVY4TVJLPKYBP6A7LBNZADLN25XBDWZFEK2QVN45YHJATIR7TA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.837388641 (499h15m56.708471676s) [noise=true , address=145.79.227.40:4076 , piece_id=Y6WKUYNPDMBUZSNWEYDL2KKNR6WKHSKGP63S6KAG6F2MWMNTVMBQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.837910455 (499h15m56.7089935s) [noise=true , address=77.90.31.11:4571 , piece_id=OERZACIKIG7HBEV3LL42KUAGKN6AWZEXNQSA52FIYTUKDFWCBW7Q, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.838146514 (499h15m56.709229559s) [noise=true , address=64.49.13.26:4945 , piece_id=KXCFTLWR4HSBTP4GJJ4HEPA6BBEMILYTJEMXY4S5WB24HRMQJOKA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.838469329 (499h15m56.709552364s) [noise=true , address=45.91.116.5:4301 , piece_id=E3ZRKZUKM335ARHV3Z5YGLKQYHG6QV4GCCNHAVMN34BM7UQFWIIA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.838690831 (499h15m56.709773866s) [noise=true , address=45.91.116.14:4204 , piece_id=JWL7OOMKTVAP7WEUB5WTL3F6F5IRNID4KIOYJSYZ655PJSNXAQKQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.839143724 (499h15m56.710226759s) [noise=true , address=45.91.116.9:4272 , piece_id=GGFDVKEW5LUCHQQUP6CWJHJQXUETYUG2UISPVP6XQJGSAJW2KP2Q, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.839678982 (499h15m56.710762027s) [noise=true , address=145.79.227.44:4208 , piece_id=ZLIX7XAUMKRUEWWTQST2KLCOWFJSAFSAQNLUCTTOWVZUBESEI23A, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:58.136536405 (499h15m58.00761944s) [upload=438194]: Next returned nil segment Dec 26 10:44:58.136548628 (499h15m58.007631673s) [upload=438194]: Waiting for error group managing segments... Dec 26 10:44:58.469618858 (499h15m58.340701893s) [noise=true , address=64.49.13.26:4945 , piece_id=KXCFTLWR4HSBTP4GJJ4HEPA6BBEMILYTJEMXY4S5WB24HRMQJOKA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.708819055 (499h15m58.5799021s) [noise=true , address=145.79.227.9:4390 , piece_id=JDQNHNGABBGANIHKSKP4VHUWIPQBNNEOPC4ZQCKR3K5TTR4N4SGQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.714466429 (499h15m58.585549474s) [noise=true , address=145.79.227.40:4076 , piece_id=Y6WKUYNPDMBUZSNWEYDL2KKNR6WKHSKGP63S6KAG6F2MWMNTVMBQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.745407696 (499h15m58.616490731s) [noise=true , address=145.79.227.20:4459 , piece_id=2EUW2AH7FBVZNJ27XVLRXKMXA737DWKAYBL7CR7PRXXRTO2M644A, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.888701070 (499h15m58.759784105s) [noise=true , address=77.90.31.18:4838 , piece_id=VYBIKY3MLEE2CXGS2MT23HX5HUFK25FKYKI4XLXPDRNWCC55ANXQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.948150873 (499h15m58.819233918s) [noise=true , address=45.91.116.5:4301 , piece_id=E3ZRKZUKM335ARHV3Z5YGLKQYHG6QV4GCCNHAVMN34BM7UQFWIIA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.971389744 (499h15m58.842472779s) [noise=true , address=77.90.31.19:4868 , piece_id=WH77PZSXC2VUWLQYLBR2NDEVT6KMD3VZK5WAL53AMRDW74P242LQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.021685314 (499h15m58.892768349s) [noise=true , address=45.91.116.4:4963 , piece_id=U4T64KQ67C3BDKG5TJSTQWUS4WWYCSVPUZOWJ6YANR2IQ2JD4EFQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.045418216 (499h15m58.916501251s) [noise=true , address=45.91.116.14:4204 , piece_id=JWL7OOMKTVAP7WEUB5WTL3F6F5IRNID4KIOYJSYZ655PJSNXAQKQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.056260371 (499h15m58.927343416s) [noise=true , address=45.91.116.9:4272 , piece_id=GGFDVKEW5LUCHQQUP6CWJHJQXUETYUG2UISPVP6XQJGSAJW2KP2Q, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.085713054 (499h15m58.956796089s) [noise=true , address=45.91.116.7:4017 , piece_id=L4GGY6AICMSFMUVYGI6ZMUORYRXNJFPLJY5XJRUAM7NVYQOHPHSQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.640980124 (499h15m59.512063169s) [noise=true , address=145.79.227.30:4778 , piece_id=BMEAUWANP7FP6CFRULMIAZUV4JNQEXOUN2EAXJY5XOWLDZINWOIA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.654515607 (499h15m59.525598652s) [noise=true , address=145.79.227.44:4208 , piece_id=ZLIX7XAUMKRUEWWTQST2KLCOWFJSAFSAQNLUCTTOWVZUBESEI23A, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.660349657 (499h15m59.531432712s) [noise=true , address=77.90.31.11:4571 , piece_id=OERZACIKIG7HBEV3LL42KUAGKN6AWZEXNQSA52FIYTUKDFWCBW7Q, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.660376087 (499h15m59.531459132s) [seg_pos={0 0}, upload=438194]: Segment reached optimal threshold of 14 pieces. Dec 26 10:44:59.660612988 (499h15m59.531696033s) [noise=true , address=64.49.13.169:4711 , piece_id=PVBVY4TVJLPKYBP6A7LBNZADLN25XBDWZFEK2QVN45YHJATIR7TA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:52964->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:52964->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52964->64.49.13.169:4711: use of closed network connection Dec 26 10:44:59.660620753 (499h15m59.531703788s) [noise=true , address=64.49.13.169:4711 , piece_id=PVBVY4TVJLPKYBP6A7LBNZADLN25XBDWZFEK2QVN45YHJATIR7TA, seg_pos={0 0}, upload=438194]: Long tail cancellation due to optimal threshold Dec 26 10:44:59.660893663 (499h15m59.531976698s) [noise=true , address=45.91.116.6:4988 , piece_id=VPEKJF74D7SQWKR47MAIY5UJQPBJIEU5AQDKLFGATYIRHFREAQ5A, seg_pos={0 0}, upload=438194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TrcoCHhAJeEykP9hJNpygoPSdgFV5v6GzL48JpyNPiAj5ZJRS): protocol: expected piece hash; context canceled; EOF Dec 26 10:44:59.660901057 (499h15m59.531984102s) [noise=true , address=45.91.116.6:4988 , piece_id=VPEKJF74D7SQWKR47MAIY5UJQPBJIEU5AQDKLFGATYIRHFREAQ5A, seg_pos={0 0}, upload=438194]: Long tail cancellation due to optimal threshold Dec 26 10:44:59.661226155 (499h15m59.5323092s) [noise=true , address=77.90.31.14:4678 , piece_id=XQHQZJUPB24G3BWGFTFDD42NM6K5HJVHUXBXWH5TW7BU25QGHTMA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12beu3SUZMPNWfbeHV4tLtx9UVZoEHh4Kt7oBhT59w3bGBE1ovP): protocol: write tcp 188.166.225.170:35884->77.90.31.14:4678: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 10:44:59.661249630 (499h15m59.532332685s) [noise=true , address=77.90.31.14:4678 , piece_id=XQHQZJUPB24G3BWGFTFDD42NM6K5HJVHUXBXWH5TW7BU25QGHTMA, seg_pos={0 0}, upload=438194]: Long tail cancellation due to optimal threshold Dec 26 10:44:59.661309705 (499h15m59.53239275s) [noise=true , address=77.90.31.21:4909 , piece_id=H6KT2AKZISCU4VEA6TXJJ3EFGCRBZPPXBEJJEIHZR3GEJZKXVRQQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121RT7tP6czMFAiZ3DjzSSsYgPt3xZXVbXVpkcYxaZYscNiMjL6): protocol: expected piece hash; context canceled; EOF Dec 26 10:44:59.661318962 (499h15m59.532401997s) [noise=true , address=77.90.31.21:4909 , piece_id=H6KT2AKZISCU4VEA6TXJJ3EFGCRBZPPXBEJJEIHZR3GEJZKXVRQQ, seg_pos={0 0}, upload=438194]: Long tail cancellation due to optimal threshold Dec 26 10:44:59.661345674 (499h15m59.532428719s) [seg_pos={0 0}, upload=438194]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 10:44:59.661485770 (499h15m59.532568815s) [upload=438194]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 10:44:59.661493655 (499h15m59.53257669s) [upload=438194]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 10:44:59.968467153 (499h15m59.839550198s) [upload=438194]: Done uploading segments... Dec 26 10:44:59.968484155 (499h15m59.83956719s) [upload=438194]: Upload finished. err: <nil> Duration: 53.268862712s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [...XX.XX..X.......X..X.X......XX......XXX.XX......X.X..XX.X.] duration: 5.709121319s trace: https://jaeger.dev.storj.io/trace/4880c1e60e28bf3b FAIL (2025-12-26 11:29:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: 4DPDSVRPEKE73NJLGZ3ZZVTCSBGVKTNPE6H3WSUGZKHYWQQNZSJA) context deadline exceeded; piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: LTVQYSXTNHPW5UDTKPEGRIIVEW2FIZILXQQWSXQXKR5TQBFVMWRA) context deadline exceeded ======LOGS====== Dec 26 11:28:32.119998417 (499h59m31.991081462s) [upload=438828]: Starting upload Dec 26 11:28:32.120081886 (499h59m31.991164931s) [upload=438828]: Done starting upload Dec 26 11:28:32.120170655 (499h59m31.9912537s) [upload=438828]: Uploading segments... Dec 26 11:28:32.120378310 (499h59m31.991461345s) [upload=438828]: Got next segment. Inline: false Dec 26 11:28:32.120431562 (499h59m31.991514607s) [upload=438828]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 11:28:32.120438214 (499h59m31.991521259s) [upload=438828]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 11:28:32.362871038 (499h59m32.233954073s) [seg_pos={0 0}, upload=438828]: Begin upload segment... Dec 26 11:28:32.362937363 (499h59m32.234020398s) [seg_pos={0 0}, upload=438828]: Done begin upload segment. Dec 26 11:28:32.363081228 (499h59m32.234164273s) [noise=true , address=145.79.227.16:4910 , piece_id=G2ODCPTTKMKXVN4QKZYCVTNUNOWHPM62ZMZXP6TKUA27QUUEFF4Q, seg_pos={0 0}, upload=438828]: Uploading piece... Dec 26 11:28:32.363098380 (499h59m32.234181415s) [noise=true , address=77.90.31.19:4842 , piece_id=4DPDSVRPEKE73NJLGZ3ZZVTCSBGVKTNPE6H3WSUGZKHYWQQNZSJA, seg_pos={0 0}, upload=438828]: Uploading piece... Dec 26 11:28:32.363612469 (499h59m32.234695504s) [noise=true , address=145.79.227.11:4730 , piece_id=U7KYGNUPHTTTZUREOFHK75GQK6NNXYL7C46XQ7WJT7HK2EZO3UXA, seg_pos={0 0}, upload=438828]: Uploading piece... Dec 26 11:28:32.364005177 (499h59m32.235088212s) [noise=true , address=145.79.227.34:4906 , piece_id=I3R52PLH3H4CL6CVB2XNN3ZIDQWSS3KQANHAJVN7ZVFGJGCVHB6Q, seg_pos={0 0}, upload=438828]: Uploading piece... Dec 26 11:28:32.364435816 (499h59m32.235518851s) [noise=true , address=77.90.31.14:4664 , piece_id=SX655XS36RMZ2GXC6JOI6XYFHRJFOGY5AJ6BJB2FOVNF5AEZMZ7Q, seg_pos={0 0}, upload=438828]: Uploading piece... Dec 26 11:28:32.364822383 (499h59m32.235905428s) [noise=true , address=145.79.227.29:4941 , piece_id=E2P54E42ZVD2A7WAFE6G6EBZDJDKXOGTOU2FKTSHZUOLXEJRFJDA, seg_pos={0 0}, upload=438828]: Uploading piece... Dec 26 11:28:32.366170099 (499h59m32.237253134s) [noise=true , address=45.91.116.11:4120 , piece_id=22RHAZHWO2GY6457J3IVRNAXHEO7EP3BI7TQIR67YTMSRETVLQGA, seg_pos={0 0}, upload=438828]: Uploading piece... Dec 26 11:28:32.366559190 (499h59m32.237642225s) [noise=true , address=145.79.227.2:4113 , piece_id=4JGZ3CFRIQOZAEAAVSRDRXW3M32U57YBTCOAEI6ZZ6ASEKBRHWQQ, seg_pos={0 0}, upload=438828]: Uploading piece... Dec 26 11:28:32.367869695 (499h59m32.23895274s) [noise=true , address=145.79.227.50:4374 , piece_id=AUVHF22K66FZKYWEMQKJUPR4FQAYGSD4MUJCK5JGBWMTEZRWFLNA, seg_pos={0 0}, upload=438828]: Uploading piece... Dec 26 11:28:32.369078918 (499h59m32.240161963s) [noise=true , address=45.91.116.6:4976 , piece_id=YEYFWLSEKAXEJANHOYMJ7CPMMWO54GWMDBQEID4BFEJPT546OGTA, seg_pos={0 0}, upload=438828]: Uploading piece... Dec 26 11:28:32.370581149 (499h59m32.241664184s) [noise=true , address=145.79.227.36:4519 , piece_id=UW362K7Y2UPFGQ7RGWTVBIM3OT6D6UGRESWZFTH32LJY5Z2K4ORA, seg_pos={0 0}, upload=438828]: Uploading piece... Dec 26 11:28:32.372120179 (499h59m32.243203224s) [noise=true , address=45.91.116.7:4009 , piece_id=53IDZ635DHWIGLFKDYDGWAI3YGMRALPKRBB56Q4X6X5NCVUHHGSQ, seg_pos={0 0}, upload=438828]: Uploading piece... Dec 26 11:28:32.373906381 (499h59m32.244989426s) [noise=true , address=145.79.227.39:4013 , piece_id=LXPZEDYMOSBJVJIUKBO6EJRJYBS2Y7A2NSDFBLH4HUOFWD5BZQ6Q, seg_pos={0 0}, upload=438828]: Uploading piece... Dec 26 11:28:32.375501227 (499h59m32.246584272s) [noise=true , address=145.79.227.20:4439 , piece_id=FPCKEMNE3X7PR4A7L44HTIGTDDSUOELPLZ6ZB2JWEKEF23DPDOFQ, seg_pos={0 0}, upload=438828]: Uploading piece... Dec 26 11:28:32.376735918 (499h59m32.247818954s) [noise=true , address=145.79.227.17:4971 , piece_id=BZNF2WP5VR6QOETDCNGCNZGNKMMJWNZNAIJXUOR2WYFBJSGJO7AA, seg_pos={0 0}, upload=438828]: Uploading piece... Dec 26 11:28:32.377558604 (499h59m32.248641639s) [noise=true , address=45.91.116.14:4191 , piece_id=LTVQYSXTNHPW5UDTKPEGRIIVEW2FIZILXQQWSXQXKR5TQBFVMWRA, seg_pos={0 0}, upload=438828]: Uploading piece... Dec 26 11:28:34.068995904 (499h59m33.94007895s) [upload=438828]: Next returned nil segment Dec 26 11:28:34.069012576 (499h59m33.940095621s) [upload=438828]: Waiting for error group managing segments... Dec 26 11:28:34.367247445 (499h59m34.23833048s) [noise=true , address=145.79.227.29:4941 , piece_id=E2P54E42ZVD2A7WAFE6G6EBZDJDKXOGTOU2FKTSHZUOLXEJRFJDA, seg_pos={0 0}, upload=438828]: Done uploading piece. err: <nil> Dec 26 11:28:34.368859775 (499h59m34.23994282s) [noise=true , address=145.79.227.16:4910 , piece_id=G2ODCPTTKMKXVN4QKZYCVTNUNOWHPM62ZMZXP6TKUA27QUUEFF4Q, seg_pos={0 0}, upload=438828]: Done uploading piece. err: <nil> Dec 26 11:28:34.379134269 (499h59m34.250217304s) [noise=true , address=145.79.227.17:4971 , piece_id=BZNF2WP5VR6QOETDCNGCNZGNKMMJWNZNAIJXUOR2WYFBJSGJO7AA, seg_pos={0 0}, upload=438828]: Done uploading piece. err: <nil> Dec 26 11:28:34.379265028 (499h59m34.250348074s) [noise=true , address=145.79.227.50:4374 , piece_id=AUVHF22K66FZKYWEMQKJUPR4FQAYGSD4MUJCK5JGBWMTEZRWFLNA, seg_pos={0 0}, upload=438828]: Done uploading piece. err: <nil> Dec 26 11:28:34.380221059 (499h59m34.251304115s) [noise=true , address=145.79.227.36:4519 , piece_id=UW362K7Y2UPFGQ7RGWTVBIM3OT6D6UGRESWZFTH32LJY5Z2K4ORA, seg_pos={0 0}, upload=438828]: Done uploading piece. err: <nil> Dec 26 11:28:34.383624800 (499h59m34.254707845s) [noise=true , address=145.79.227.39:4013 , piece_id=LXPZEDYMOSBJVJIUKBO6EJRJYBS2Y7A2NSDFBLH4HUOFWD5BZQ6Q, seg_pos={0 0}, upload=438828]: Done uploading piece. err: <nil> Dec 26 11:28:34.384385368 (499h59m34.255468403s) [noise=true , address=145.79.227.20:4439 , piece_id=FPCKEMNE3X7PR4A7L44HTIGTDDSUOELPLZ6ZB2JWEKEF23DPDOFQ, seg_pos={0 0}, upload=438828]: Done uploading piece. err: <nil> Dec 26 11:28:34.391253136 (499h59m34.262336172s) [noise=true , address=145.79.227.11:4730 , piece_id=U7KYGNUPHTTTZUREOFHK75GQK6NNXYL7C46XQ7WJT7HK2EZO3UXA, seg_pos={0 0}, upload=438828]: Done uploading piece. err: <nil> Dec 26 11:28:34.520252463 (499h59m34.391335498s) [noise=true , address=145.79.227.2:4113 , piece_id=4JGZ3CFRIQOZAEAAVSRDRXW3M32U57YBTCOAEI6ZZ6ASEKBRHWQQ, seg_pos={0 0}, upload=438828]: Done uploading piece. err: <nil> Dec 26 11:28:34.572631651 (499h59m34.443714696s) [noise=true , address=77.90.31.19:4842 , piece_id=4DPDSVRPEKE73NJLGZ3ZZVTCSBGVKTNPE6H3WSUGZKHYWQQNZSJA, seg_pos={0 0}, upload=438828]: Done uploading piece. err: <nil> Dec 26 11:28:35.071333734 (499h59m34.942416779s) [noise=true , address=45.91.116.14:4191 , piece_id=LTVQYSXTNHPW5UDTKPEGRIIVEW2FIZILXQQWSXQXKR5TQBFVMWRA, seg_pos={0 0}, upload=438828]: Done uploading piece. err: <nil> Dec 26 11:28:35.311673611 (499h59m35.182756646s) [noise=true , address=45.91.116.11:4120 , piece_id=22RHAZHWO2GY6457J3IVRNAXHEO7EP3BI7TQIR67YTMSRETVLQGA, seg_pos={0 0}, upload=438828]: Done uploading piece. err: <nil> Dec 26 11:28:35.321516343 (499h59m35.192599378s) [noise=true , address=77.90.31.14:4664 , piece_id=SX655XS36RMZ2GXC6JOI6XYFHRJFOGY5AJ6BJB2FOVNF5AEZMZ7Q, seg_pos={0 0}, upload=438828]: Done uploading piece. err: <nil> Dec 26 11:28:35.321535089 (499h59m35.192618124s) [seg_pos={0 0}, upload=438828]: Segment reached optimal threshold of 13 pieces. Dec 26 11:28:35.321694762 (499h59m35.192777797s) [noise=true , address=45.91.116.6:4976 , piece_id=YEYFWLSEKAXEJANHOYMJ7CPMMWO54GWMDBQEID4BFEJPT546OGTA, seg_pos={0 0}, upload=438828]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Dec 26 11:28:35.321701777 (499h59m35.192784812s) [noise=true , address=45.91.116.6:4976 , piece_id=YEYFWLSEKAXEJANHOYMJ7CPMMWO54GWMDBQEID4BFEJPT546OGTA, seg_pos={0 0}, upload=438828]: Long tail cancellation due to optimal threshold Dec 26 11:28:35.321979454 (499h59m35.193062499s) [noise=true , address=145.79.227.34:4906 , piece_id=I3R52PLH3H4CL6CVB2XNN3ZIDQWSS3KQANHAJVN7ZVFGJGCVHB6Q, seg_pos={0 0}, upload=438828]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 26 11:28:35.321997970 (499h59m35.193081015s) [noise=true , address=145.79.227.34:4906 , piece_id=I3R52PLH3H4CL6CVB2XNN3ZIDQWSS3KQANHAJVN7ZVFGJGCVHB6Q, seg_pos={0 0}, upload=438828]: Long tail cancellation due to optimal threshold Dec 26 11:28:35.322020323 (499h59m35.193103368s) [noise=true , address=45.91.116.7:4009 , piece_id=53IDZ635DHWIGLFKDYDGWAI3YGMRALPKRBB56Q4X6X5NCVUHHGSQ, seg_pos={0 0}, upload=438828]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 26 11:28:35.322027697 (499h59m35.193110742s) [noise=true , address=45.91.116.7:4009 , piece_id=53IDZ635DHWIGLFKDYDGWAI3YGMRALPKRBB56Q4X6X5NCVUHHGSQ, seg_pos={0 0}, upload=438828]: Long tail cancellation due to optimal threshold Dec 26 11:28:35.322045951 (499h59m35.193128986s) [seg_pos={0 0}, upload=438828]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 11:28:35.322138127 (499h59m35.193221172s) [upload=438828]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 11:28:35.322152965 (499h59m35.193236s) [upload=438828]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 11:28:35.605414402 (499h59m35.476497447s) [upload=438828]: Done uploading segments... Dec 26 11:28:35.605432517 (499h59m35.476515552s) [upload=438828]: Upload finished. err: <nil> Duration: 5.709151216s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [....XX.XX..X.......X..X.X......XX......XXX.XX......X.X..XX.X] duration: 30.002375025s trace: https://jaeger.dev.storj.io/trace/4880c1e60e28bf3b FAIL (2025-12-26 11:29:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: 4DPDSVRPEKE73NJLGZ3ZZVTCSBGVKTNPE6H3WSUGZKHYWQQNZSJA) context deadline exceeded; piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: LTVQYSXTNHPW5UDTKPEGRIIVEW2FIZILXQQWSXQXKR5TQBFVMWRA) context deadline exceeded ======LOGS====== Dec 26 11:28:32.119998417 (499h59m31.991081462s) [upload=438828]: Starting upload Dec 26 11:28:32.120081886 (499h59m31.991164931s) [upload=438828]: Done starting upload Dec 26 11:28:32.120170655 (499h59m31.9912537s) [upload=438828]: Uploading segments... Dec 26 11:28:32.120378310 (499h59m31.991461345s) [upload=438828]: Got next segment. Inline: false Dec 26 11:28:32.120431562 (499h59m31.991514607s) [upload=438828]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 11:28:32.120438214 (499h59m31.991521259s) [upload=438828]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 11:28:32.362871038 (499h59m32.233954073s) [seg_pos={0 0}, upload=438828]: Begin upload segment... Dec 26 11:28:32.362937363 (499h59m32.234020398s) [seg_pos={0 0}, upload=438828]: Done begin upload segment. Dec 26 11:28:32.363081228 (499h59m32.234164273s) [noise=true , address=145.79.227.16:4910 , piece_id=G2ODCPTTKMKXVN4QKZYCVTNUNOWHPM62ZMZXP6TKUA27QUUEFF4Q, seg_pos={0 0}, upload=438828]: Uploading piece... Dec 26 11:28:32.363098380 (499h59m32.234181415s) [noise=true , address=77.90.31.19:4842 , piece_id=4DPDSVRPEKE73NJLGZ3ZZVTCSBGVKTNPE6H3WSUGZKHYWQQNZSJA, seg_pos={0 0}, upload=438828]: Uploading piece... Dec 26 11:28:32.363612469 (499h59m32.234695504s) [noise=true , address=145.79.227.11:4730 , piece_id=U7KYGNUPHTTTZUREOFHK75GQK6NNXYL7C46XQ7WJT7HK2EZO3UXA, seg_pos={0 0}, upload=438828]: Uploading piece... Dec 26 11:28:32.364005177 (499h59m32.235088212s) [noise=true , address=145.79.227.34:4906 , piece_id=I3R52PLH3H4CL6CVB2XNN3ZIDQWSS3KQANHAJVN7ZVFGJGCVHB6Q, seg_pos={0 0}, upload=438828]: Uploading piece... Dec 26 11:28:32.364435816 (499h59m32.235518851s) [noise=true , address=77.90.31.14:4664 , piece_id=SX655XS36RMZ2GXC6JOI6XYFHRJFOGY5AJ6BJB2FOVNF5AEZMZ7Q, seg_pos={0 0}, upload=438828]: Uploading piece... Dec 26 11:28:32.364822383 (499h59m32.235905428s) [noise=true , address=145.79.227.29:4941 , piece_id=E2P54E42ZVD2A7WAFE6G6EBZDJDKXOGTOU2FKTSHZUOLXEJRFJDA, seg_pos={0 0}, upload=438828]: Uploading piece... Dec 26 11:28:32.366170099 (499h59m32.237253134s) [noise=true , address=45.91.116.11:4120 , piece_id=22RHAZHWO2GY6457J3IVRNAXHEO7EP3BI7TQIR67YTMSRETVLQGA, seg_pos={0 0}, upload=438828]: Uploading piece... Dec 26 11:28:32.366559190 (499h59m32.237642225s) [noise=true , address=145.79.227.2:4113 , piece_id=4JGZ3CFRIQOZAEAAVSRDRXW3M32U57YBTCOAEI6ZZ6ASEKBRHWQQ, seg_pos={0 0}, upload=438828]: Uploading piece... Dec 26 11:28:32.367869695 (499h59m32.23895274s) [noise=true , address=145.79.227.50:4374 , piece_id=AUVHF22K66FZKYWEMQKJUPR4FQAYGSD4MUJCK5JGBWMTEZRWFLNA, seg_pos={0 0}, upload=438828]: Uploading piece... Dec 26 11:28:32.369078918 (499h59m32.240161963s) [noise=true , address=45.91.116.6:4976 , piece_id=YEYFWLSEKAXEJANHOYMJ7CPMMWO54GWMDBQEID4BFEJPT546OGTA, seg_pos={0 0}, upload=438828]: Uploading piece... Dec 26 11:28:32.370581149 (499h59m32.241664184s) [noise=true , address=145.79.227.36:4519 , piece_id=UW362K7Y2UPFGQ7RGWTVBIM3OT6D6UGRESWZFTH32LJY5Z2K4ORA, seg_pos={0 0}, upload=438828]: Uploading piece... Dec 26 11:28:32.372120179 (499h59m32.243203224s) [noise=true , address=45.91.116.7:4009 , piece_id=53IDZ635DHWIGLFKDYDGWAI3YGMRALPKRBB56Q4X6X5NCVUHHGSQ, seg_pos={0 0}, upload=438828]: Uploading piece... Dec 26 11:28:32.373906381 (499h59m32.244989426s) [noise=true , address=145.79.227.39:4013 , piece_id=LXPZEDYMOSBJVJIUKBO6EJRJYBS2Y7A2NSDFBLH4HUOFWD5BZQ6Q, seg_pos={0 0}, upload=438828]: Uploading piece... Dec 26 11:28:32.375501227 (499h59m32.246584272s) [noise=true , address=145.79.227.20:4439 , piece_id=FPCKEMNE3X7PR4A7L44HTIGTDDSUOELPLZ6ZB2JWEKEF23DPDOFQ, seg_pos={0 0}, upload=438828]: Uploading piece... Dec 26 11:28:32.376735918 (499h59m32.247818954s) [noise=true , address=145.79.227.17:4971 , piece_id=BZNF2WP5VR6QOETDCNGCNZGNKMMJWNZNAIJXUOR2WYFBJSGJO7AA, seg_pos={0 0}, upload=438828]: Uploading piece... Dec 26 11:28:32.377558604 (499h59m32.248641639s) [noise=true , address=45.91.116.14:4191 , piece_id=LTVQYSXTNHPW5UDTKPEGRIIVEW2FIZILXQQWSXQXKR5TQBFVMWRA, seg_pos={0 0}, upload=438828]: Uploading piece... Dec 26 11:28:34.068995904 (499h59m33.94007895s) [upload=438828]: Next returned nil segment Dec 26 11:28:34.069012576 (499h59m33.940095621s) [upload=438828]: Waiting for error group managing segments... Dec 26 11:28:34.367247445 (499h59m34.23833048s) [noise=true , address=145.79.227.29:4941 , piece_id=E2P54E42ZVD2A7WAFE6G6EBZDJDKXOGTOU2FKTSHZUOLXEJRFJDA, seg_pos={0 0}, upload=438828]: Done uploading piece. err: <nil> Dec 26 11:28:34.368859775 (499h59m34.23994282s) [noise=true , address=145.79.227.16:4910 , piece_id=G2ODCPTTKMKXVN4QKZYCVTNUNOWHPM62ZMZXP6TKUA27QUUEFF4Q, seg_pos={0 0}, upload=438828]: Done uploading piece. err: <nil> Dec 26 11:28:34.379134269 (499h59m34.250217304s) [noise=true , address=145.79.227.17:4971 , piece_id=BZNF2WP5VR6QOETDCNGCNZGNKMMJWNZNAIJXUOR2WYFBJSGJO7AA, seg_pos={0 0}, upload=438828]: Done uploading piece. err: <nil> Dec 26 11:28:34.379265028 (499h59m34.250348074s) [noise=true , address=145.79.227.50:4374 , piece_id=AUVHF22K66FZKYWEMQKJUPR4FQAYGSD4MUJCK5JGBWMTEZRWFLNA, seg_pos={0 0}, upload=438828]: Done uploading piece. err: <nil> Dec 26 11:28:34.380221059 (499h59m34.251304115s) [noise=true , address=145.79.227.36:4519 , piece_id=UW362K7Y2UPFGQ7RGWTVBIM3OT6D6UGRESWZFTH32LJY5Z2K4ORA, seg_pos={0 0}, upload=438828]: Done uploading piece. err: <nil> Dec 26 11:28:34.383624800 (499h59m34.254707845s) [noise=true , address=145.79.227.39:4013 , piece_id=LXPZEDYMOSBJVJIUKBO6EJRJYBS2Y7A2NSDFBLH4HUOFWD5BZQ6Q, seg_pos={0 0}, upload=438828]: Done uploading piece. err: <nil> Dec 26 11:28:34.384385368 (499h59m34.255468403s) [noise=true , address=145.79.227.20:4439 , piece_id=FPCKEMNE3X7PR4A7L44HTIGTDDSUOELPLZ6ZB2JWEKEF23DPDOFQ, seg_pos={0 0}, upload=438828]: Done uploading piece. err: <nil> Dec 26 11:28:34.391253136 (499h59m34.262336172s) [noise=true , address=145.79.227.11:4730 , piece_id=U7KYGNUPHTTTZUREOFHK75GQK6NNXYL7C46XQ7WJT7HK2EZO3UXA, seg_pos={0 0}, upload=438828]: Done uploading piece. err: <nil> Dec 26 11:28:34.520252463 (499h59m34.391335498s) [noise=true , address=145.79.227.2:4113 , piece_id=4JGZ3CFRIQOZAEAAVSRDRXW3M32U57YBTCOAEI6ZZ6ASEKBRHWQQ, seg_pos={0 0}, upload=438828]: Done uploading piece. err: <nil> Dec 26 11:28:34.572631651 (499h59m34.443714696s) [noise=true , address=77.90.31.19:4842 , piece_id=4DPDSVRPEKE73NJLGZ3ZZVTCSBGVKTNPE6H3WSUGZKHYWQQNZSJA, seg_pos={0 0}, upload=438828]: Done uploading piece. err: <nil> Dec 26 11:28:35.071333734 (499h59m34.942416779s) [noise=true , address=45.91.116.14:4191 , piece_id=LTVQYSXTNHPW5UDTKPEGRIIVEW2FIZILXQQWSXQXKR5TQBFVMWRA, seg_pos={0 0}, upload=438828]: Done uploading piece. err: <nil> Dec 26 11:28:35.311673611 (499h59m35.182756646s) [noise=true , address=45.91.116.11:4120 , piece_id=22RHAZHWO2GY6457J3IVRNAXHEO7EP3BI7TQIR67YTMSRETVLQGA, seg_pos={0 0}, upload=438828]: Done uploading piece. err: <nil> Dec 26 11:28:35.321516343 (499h59m35.192599378s) [noise=true , address=77.90.31.14:4664 , piece_id=SX655XS36RMZ2GXC6JOI6XYFHRJFOGY5AJ6BJB2FOVNF5AEZMZ7Q, seg_pos={0 0}, upload=438828]: Done uploading piece. err: <nil> Dec 26 11:28:35.321535089 (499h59m35.192618124s) [seg_pos={0 0}, upload=438828]: Segment reached optimal threshold of 13 pieces. Dec 26 11:28:35.321694762 (499h59m35.192777797s) [noise=true , address=45.91.116.6:4976 , piece_id=YEYFWLSEKAXEJANHOYMJ7CPMMWO54GWMDBQEID4BFEJPT546OGTA, seg_pos={0 0}, upload=438828]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Dec 26 11:28:35.321701777 (499h59m35.192784812s) [noise=true , address=45.91.116.6:4976 , piece_id=YEYFWLSEKAXEJANHOYMJ7CPMMWO54GWMDBQEID4BFEJPT546OGTA, seg_pos={0 0}, upload=438828]: Long tail cancellation due to optimal threshold Dec 26 11:28:35.321979454 (499h59m35.193062499s) [noise=true , address=145.79.227.34:4906 , piece_id=I3R52PLH3H4CL6CVB2XNN3ZIDQWSS3KQANHAJVN7ZVFGJGCVHB6Q, seg_pos={0 0}, upload=438828]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 26 11:28:35.321997970 (499h59m35.193081015s) [noise=true , address=145.79.227.34:4906 , piece_id=I3R52PLH3H4CL6CVB2XNN3ZIDQWSS3KQANHAJVN7ZVFGJGCVHB6Q, seg_pos={0 0}, upload=438828]: Long tail cancellation due to optimal threshold Dec 26 11:28:35.322020323 (499h59m35.193103368s) [noise=true , address=45.91.116.7:4009 , piece_id=53IDZ635DHWIGLFKDYDGWAI3YGMRALPKRBB56Q4X6X5NCVUHHGSQ, seg_pos={0 0}, upload=438828]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 26 11:28:35.322027697 (499h59m35.193110742s) [noise=true , address=45.91.116.7:4009 , piece_id=53IDZ635DHWIGLFKDYDGWAI3YGMRALPKRBB56Q4X6X5NCVUHHGSQ, seg_pos={0 0}, upload=438828]: Long tail cancellation due to optimal threshold Dec 26 11:28:35.322045951 (499h59m35.193128986s) [seg_pos={0 0}, upload=438828]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 11:28:35.322138127 (499h59m35.193221172s) [upload=438828]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 11:28:35.322152965 (499h59m35.193236s) [upload=438828]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 11:28:35.605414402 (499h59m35.476497447s) [upload=438828]: Done uploading segments... Dec 26 11:28:35.605432517 (499h59m35.476515552s) [upload=438828]: Upload finished. err: <nil> Duration: 30.002402767s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 46.361950995s trace: https://jaeger.dev.storj.io/trace/785a90e8be84a595 FAIL (2025-12-26 11:28:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== KXVRQQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.834123964 (499h15m56.705207009s) [noise=true , address=45.91.116.4:4963 , piece_id=U4T64KQ67C3BDKG5TJSTQWUS4WWYCSVPUZOWJ6YANR2IQ2JD4EFQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.834585043 (499h15m56.705668078s) [noise=true , address=145.79.227.9:4390 , piece_id=JDQNHNGABBGANIHKSKP4VHUWIPQBNNEOPC4ZQCKR3K5TTR4N4SGQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.834961800 (499h15m56.706044845s) [noise=true , address=145.79.227.30:4778 , piece_id=BMEAUWANP7FP6CFRULMIAZUV4JNQEXOUN2EAXJY5XOWLDZINWOIA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.835331334 (499h15m56.706414379s) [noise=true , address=77.90.31.18:4838 , piece_id=VYBIKY3MLEE2CXGS2MT23HX5HUFK25FKYKI4XLXPDRNWCC55ANXQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.835738399 (499h15m56.706821434s) [noise=true , address=145.79.227.20:4459 , piece_id=2EUW2AH7FBVZNJ27XVLRXKMXA737DWKAYBL7CR7PRXXRTO2M644A, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.836118653 (499h15m56.707201698s) [noise=true , address=77.90.31.14:4678 , piece_id=XQHQZJUPB24G3BWGFTFDD42NM6K5HJVHUXBXWH5TW7BU25QGHTMA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.836501923 (499h15m56.707584968s) [noise=true , address=45.91.116.6:4988 , piece_id=VPEKJF74D7SQWKR47MAIY5UJQPBJIEU5AQDKLFGATYIRHFREAQ5A, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.836904940 (499h15m56.707987985s) [noise=true , address=64.49.13.169:4711 , piece_id=PVBVY4TVJLPKYBP6A7LBNZADLN25XBDWZFEK2QVN45YHJATIR7TA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.837388641 (499h15m56.708471676s) [noise=true , address=145.79.227.40:4076 , piece_id=Y6WKUYNPDMBUZSNWEYDL2KKNR6WKHSKGP63S6KAG6F2MWMNTVMBQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.837910455 (499h15m56.7089935s) [noise=true , address=77.90.31.11:4571 , piece_id=OERZACIKIG7HBEV3LL42KUAGKN6AWZEXNQSA52FIYTUKDFWCBW7Q, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.838146514 (499h15m56.709229559s) [noise=true , address=64.49.13.26:4945 , piece_id=KXCFTLWR4HSBTP4GJJ4HEPA6BBEMILYTJEMXY4S5WB24HRMQJOKA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.838469329 (499h15m56.709552364s) [noise=true , address=45.91.116.5:4301 , piece_id=E3ZRKZUKM335ARHV3Z5YGLKQYHG6QV4GCCNHAVMN34BM7UQFWIIA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.838690831 (499h15m56.709773866s) [noise=true , address=45.91.116.14:4204 , piece_id=JWL7OOMKTVAP7WEUB5WTL3F6F5IRNID4KIOYJSYZ655PJSNXAQKQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.839143724 (499h15m56.710226759s) [noise=true , address=45.91.116.9:4272 , piece_id=GGFDVKEW5LUCHQQUP6CWJHJQXUETYUG2UISPVP6XQJGSAJW2KP2Q, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.839678982 (499h15m56.710762027s) [noise=true , address=145.79.227.44:4208 , piece_id=ZLIX7XAUMKRUEWWTQST2KLCOWFJSAFSAQNLUCTTOWVZUBESEI23A, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:58.136536405 (499h15m58.00761944s) [upload=438194]: Next returned nil segment Dec 26 10:44:58.136548628 (499h15m58.007631673s) [upload=438194]: Waiting for error group managing segments... Dec 26 10:44:58.469618858 (499h15m58.340701893s) [noise=true , address=64.49.13.26:4945 , piece_id=KXCFTLWR4HSBTP4GJJ4HEPA6BBEMILYTJEMXY4S5WB24HRMQJOKA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.708819055 (499h15m58.5799021s) [noise=true , address=145.79.227.9:4390 , piece_id=JDQNHNGABBGANIHKSKP4VHUWIPQBNNEOPC4ZQCKR3K5TTR4N4SGQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.714466429 (499h15m58.585549474s) [noise=true , address=145.79.227.40:4076 , piece_id=Y6WKUYNPDMBUZSNWEYDL2KKNR6WKHSKGP63S6KAG6F2MWMNTVMBQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.745407696 (499h15m58.616490731s) [noise=true , address=145.79.227.20:4459 , piece_id=2EUW2AH7FBVZNJ27XVLRXKMXA737DWKAYBL7CR7PRXXRTO2M644A, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.888701070 (499h15m58.759784105s) [noise=true , address=77.90.31.18:4838 , piece_id=VYBIKY3MLEE2CXGS2MT23HX5HUFK25FKYKI4XLXPDRNWCC55ANXQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.948150873 (499h15m58.819233918s) [noise=true , address=45.91.116.5:4301 , piece_id=E3ZRKZUKM335ARHV3Z5YGLKQYHG6QV4GCCNHAVMN34BM7UQFWIIA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.971389744 (499h15m58.842472779s) [noise=true , address=77.90.31.19:4868 , piece_id=WH77PZSXC2VUWLQYLBR2NDEVT6KMD3VZK5WAL53AMRDW74P242LQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.021685314 (499h15m58.892768349s) [noise=true , address=45.91.116.4:4963 , piece_id=U4T64KQ67C3BDKG5TJSTQWUS4WWYCSVPUZOWJ6YANR2IQ2JD4EFQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.045418216 (499h15m58.916501251s) [noise=true , address=45.91.116.14:4204 , piece_id=JWL7OOMKTVAP7WEUB5WTL3F6F5IRNID4KIOYJSYZ655PJSNXAQKQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.056260371 (499h15m58.927343416s) [noise=true , address=45.91.116.9:4272 , piece_id=GGFDVKEW5LUCHQQUP6CWJHJQXUETYUG2UISPVP6XQJGSAJW2KP2Q, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.085713054 (499h15m58.956796089s) [noise=true , address=45.91.116.7:4017 , piece_id=L4GGY6AICMSFMUVYGI6ZMUORYRXNJFPLJY5XJRUAM7NVYQOHPHSQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.640980124 (499h15m59.512063169s) [noise=true , address=145.79.227.30:4778 , piece_id=BMEAUWANP7FP6CFRULMIAZUV4JNQEXOUN2EAXJY5XOWLDZINWOIA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.654515607 (499h15m59.525598652s) [noise=true , address=145.79.227.44:4208 , piece_id=ZLIX7XAUMKRUEWWTQST2KLCOWFJSAFSAQNLUCTTOWVZUBESEI23A, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.660349657 (499h15m59.531432712s) [noise=true , address=77.90.31.11:4571 , piece_id=OERZACIKIG7HBEV3LL42KUAGKN6AWZEXNQSA52FIYTUKDFWCBW7Q, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.660376087 (499h15m59.531459132s) [seg_pos={0 0}, upload=438194]: Segment reached optimal threshold of 14 pieces. Dec 26 10:44:59.660612988 (499h15m59.531696033s) [noise=true , address=64.49.13.169:4711 , piece_id=PVBVY4TVJLPKYBP6A7LBNZADLN25XBDWZFEK2QVN45YHJATIR7TA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:52964->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:52964->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52964->64.49.13.169:4711: use of closed network connection Dec 26 10:44:59.660620753 (499h15m59.531703788s) [noise=true , address=64.49.13.169:4711 , piece_id=PVBVY4TVJLPKYBP6A7LBNZADLN25XBDWZFEK2QVN45YHJATIR7TA, seg_pos={0 0}, upload=438194]: Long tail cancellation due to optimal threshold Dec 26 10:44:59.660893663 (499h15m59.531976698s) [noise=true , address=45.91.116.6:4988 , piece_id=VPEKJF74D7SQWKR47MAIY5UJQPBJIEU5AQDKLFGATYIRHFREAQ5A, seg_pos={0 0}, upload=438194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TrcoCHhAJeEykP9hJNpygoPSdgFV5v6GzL48JpyNPiAj5ZJRS): protocol: expected piece hash; context canceled; EOF Dec 26 10:44:59.660901057 (499h15m59.531984102s) [noise=true , address=45.91.116.6:4988 , piece_id=VPEKJF74D7SQWKR47MAIY5UJQPBJIEU5AQDKLFGATYIRHFREAQ5A, seg_pos={0 0}, upload=438194]: Long tail cancellation due to optimal threshold Dec 26 10:44:59.661226155 (499h15m59.5323092s) [noise=true , address=77.90.31.14:4678 , piece_id=XQHQZJUPB24G3BWGFTFDD42NM6K5HJVHUXBXWH5TW7BU25QGHTMA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12beu3SUZMPNWfbeHV4tLtx9UVZoEHh4Kt7oBhT59w3bGBE1ovP): protocol: write tcp 188.166.225.170:35884->77.90.31.14:4678: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 10:44:59.661249630 (499h15m59.532332685s) [noise=true , address=77.90.31.14:4678 , piece_id=XQHQZJUPB24G3BWGFTFDD42NM6K5HJVHUXBXWH5TW7BU25QGHTMA, seg_pos={0 0}, upload=438194]: Long tail cancellation due to optimal threshold Dec 26 10:44:59.661309705 (499h15m59.53239275s) [noise=true , address=77.90.31.21:4909 , piece_id=H6KT2AKZISCU4VEA6TXJJ3EFGCRBZPPXBEJJEIHZR3GEJZKXVRQQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121RT7tP6czMFAiZ3DjzSSsYgPt3xZXVbXVpkcYxaZYscNiMjL6): protocol: expected piece hash; context canceled; EOF Dec 26 10:44:59.661318962 (499h15m59.532401997s) [noise=true , address=77.90.31.21:4909 , piece_id=H6KT2AKZISCU4VEA6TXJJ3EFGCRBZPPXBEJJEIHZR3GEJZKXVRQQ, seg_pos={0 0}, upload=438194]: Long tail cancellation due to optimal threshold Dec 26 10:44:59.661345674 (499h15m59.532428719s) [seg_pos={0 0}, upload=438194]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 10:44:59.661485770 (499h15m59.532568815s) [upload=438194]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 10:44:59.661493655 (499h15m59.53257669s) [upload=438194]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 10:44:59.968467153 (499h15m59.839550198s) [upload=438194]: Done uploading segments... Dec 26 10:44:59.968484155 (499h15m59.83956719s) [upload=438194]: Upload finished. err: <nil> Duration: 46.361965873s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [.....XX.XX..X.......X..X.X......XX......XXX.XX......X.X..XX.] duration: 5.690527028s trace: https://jaeger.dev.storj.io/trace/73e14189d001f0be FAIL (2025-12-26 11:27:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: ET35CSDDTCHRYY6QZLSI62CMUX6KYE74YTJF4QIXF42GMB5TZKLQ) context deadline exceeded; piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: KZOIBWRAQDAYPRXGBADCSWDIHFLGBX76LI7UAEX3U2WFX64HH76A) context deadline exceeded ======LOGS====== : false Dec 26 11:27:18.444688377 (499h58m18.315771412s) [upload=438811]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 11:27:18.444695730 (499h58m18.315778806s) [upload=438811]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 11:27:18.687533805 (499h58m18.55861688s) [seg_pos={0 0}, upload=438811]: Begin upload segment... Dec 26 11:27:18.687597567 (499h58m18.558680602s) [seg_pos={0 0}, upload=438811]: Done begin upload segment. Dec 26 11:27:18.687631101 (499h58m18.558714146s) [noise=true , address=145.79.227.50:4374 , piece_id=3Q3JYQS5T3BJ42KJSDGYGRT72IOZYUS4ZDHLZR7MFPWMDCSPXN6Q, seg_pos={0 0}, upload=438811]: Uploading piece... Dec 26 11:27:18.687730058 (499h58m18.558813123s) [noise=true , address=45.91.116.11:4120 , piece_id=PAWAYSU7KQ36ADDG3WO4QEGVCWLFUS7R6FEWYYXC54TA27QVE46Q, seg_pos={0 0}, upload=438811]: Uploading piece... Dec 26 11:27:18.688346993 (499h58m18.559430038s) [noise=true , address=145.79.227.41:4085 , piece_id=7RTIJNFY3V6XBCGGBJRAT2FPJ2TJ4YKQ5RLIKET4X4RAS4J3QOKA, seg_pos={0 0}, upload=438811]: Uploading piece... Dec 26 11:27:18.688804154 (499h58m18.559887189s) [noise=true , address=45.91.116.5:4298 , piece_id=35CTBAKJSQNUUXBBNTR6RY2KWECNLL5AVDOKK3324EUN56TZNDLA, seg_pos={0 0}, upload=438811]: Uploading piece... Dec 26 11:27:18.689256535 (499h58m18.56033958s) [noise=true , address=45.91.116.10:4083 , piece_id=E3WD3G4C2OJQ2NKJVIUJWOEXTEXGKZRNVFFWC65QJZX7U3YMESQA, seg_pos={0 0}, upload=438811]: Uploading piece... Dec 26 11:27:18.689754454 (499h58m18.560837489s) [noise=true , address=45.91.116.14:4192 , piece_id=ET35CSDDTCHRYY6QZLSI62CMUX6KYE74YTJF4QIXF42GMB5TZKLQ, seg_pos={0 0}, upload=438811]: Uploading piece... Dec 26 11:27:18.690287669 (499h58m18.561370704s) [noise=true , address=145.79.227.51:4410 , piece_id=GI6DENEBYMU5VYD2AF3LSGGBHG6RRPO4JPQPANR6ATVWBVR3THCQ, seg_pos={0 0}, upload=438811]: Uploading piece... Dec 26 11:27:18.691277603 (499h58m18.562360649s) [noise=true , address=145.79.227.7:4293 , piece_id=7AEOIDGAQOKI3OLTXPHSX3VP27VF4CQ3U6GERHEKVKMIINWRI32Q, seg_pos={0 0}, upload=438811]: Uploading piece... Dec 26 11:27:18.691958100 (499h58m18.563041135s) [noise=true , address=45.91.116.8:4045 , piece_id=NLC7JOZFKFMJAWIQKIRS5UCRLLIS2EHB23LEWXJMBDI3H2JFCGEQ, seg_pos={0 0}, upload=438811]: Uploading piece... Dec 26 11:27:18.692472329 (499h58m18.563555364s) [noise=true , address=145.79.227.33:4870 , piece_id=TWA7X2QDSVCAHT3L35IWUPLVROP57A47WEANTWAZJHHP55LO4VBA, seg_pos={0 0}, upload=438811]: Uploading piece... Dec 26 11:27:18.693509654 (499h58m18.564592689s) [noise=true , address=145.79.227.22:4511 , piece_id=SVD7USFL67GMBF7AAS4C7ZQB3CBIPC7D7WPAS6LM2ZKRKR2QHRDQ, seg_pos={0 0}, upload=438811]: Uploading piece... Dec 26 11:27:18.693643859 (499h58m18.564726904s) [noise=true , address=45.91.116.12:4226 , piece_id=KZOIBWRAQDAYPRXGBADCSWDIHFLGBX76LI7UAEX3U2WFX64HH76A, seg_pos={0 0}, upload=438811]: Uploading piece... Dec 26 11:27:18.694914189 (499h58m18.565997234s) [noise=true , address=45.91.116.6:4973 , piece_id=MNDHSLSI5BVR6ZDKIMYIXN4QTW7C3REEWJEA5SBMJ5NUALST4YEA, seg_pos={0 0}, upload=438811]: Uploading piece... Dec 26 11:27:18.696476534 (499h58m18.567559569s) [noise=true , address=145.79.227.2:4114 , piece_id=MG5FUE6XTOTGSV5X5AAEIUE7C6OT2RHKI7MH7TBLOZW4FTFZKCOQ, seg_pos={0 0}, upload=438811]: Uploading piece... Dec 26 11:27:18.697132242 (499h58m18.568215277s) [noise=true , address=145.79.227.4:4185 , piece_id=X4S6LO3XO2J3XN3UXSP4Y6KJJQW5XZJQSY77URWDCJQ4D5FACNNQ, seg_pos={0 0}, upload=438811]: Uploading piece... Dec 26 11:27:18.701266985 (499h58m18.57235003s) [noise=true , address=145.79.227.29:4941 , piece_id=U4I2AZ5YDAY34KCZXA5QETDCCGT57H62KAZRBZWHTAPBUKF4TVOA, seg_pos={0 0}, upload=438811]: Uploading piece... Dec 26 11:27:20.412851469 (499h58m20.283934514s) [upload=438811]: Next returned nil segment Dec 26 11:27:20.412868721 (499h58m20.283951756s) [upload=438811]: Waiting for error group managing segments... Dec 26 11:27:20.694011124 (499h58m20.565094159s) [noise=true , address=145.79.227.4:4185 , piece_id=X4S6LO3XO2J3XN3UXSP4Y6KJJQW5XZJQSY77URWDCJQ4D5FACNNQ, seg_pos={0 0}, upload=438811]: Done uploading piece. err: <nil> Dec 26 11:27:20.702024172 (499h58m20.573107207s) [noise=true , address=145.79.227.33:4870 , piece_id=TWA7X2QDSVCAHT3L35IWUPLVROP57A47WEANTWAZJHHP55LO4VBA, seg_pos={0 0}, upload=438811]: Done uploading piece. err: <nil> Dec 26 11:27:20.702150723 (499h58m20.573233758s) [noise=true , address=145.79.227.51:4410 , piece_id=GI6DENEBYMU5VYD2AF3LSGGBHG6RRPO4JPQPANR6ATVWBVR3THCQ, seg_pos={0 0}, upload=438811]: Done uploading piece. err: <nil> Dec 26 11:27:20.703021421 (499h58m20.574104466s) [noise=true , address=145.79.227.50:4374 , piece_id=3Q3JYQS5T3BJ42KJSDGYGRT72IOZYUS4ZDHLZR7MFPWMDCSPXN6Q, seg_pos={0 0}, upload=438811]: Done uploading piece. err: <nil> Dec 26 11:27:20.705665987 (499h58m20.576749022s) [noise=true , address=145.79.227.41:4085 , piece_id=7RTIJNFY3V6XBCGGBJRAT2FPJ2TJ4YKQ5RLIKET4X4RAS4J3QOKA, seg_pos={0 0}, upload=438811]: Done uploading piece. err: <nil> Dec 26 11:27:20.719063959 (499h58m20.590147014s) [noise=true , address=145.79.227.29:4941 , piece_id=U4I2AZ5YDAY34KCZXA5QETDCCGT57H62KAZRBZWHTAPBUKF4TVOA, seg_pos={0 0}, upload=438811]: Done uploading piece. err: <nil> Dec 26 11:27:20.721280740 (499h58m20.592363776s) [noise=true , address=145.79.227.7:4293 , piece_id=7AEOIDGAQOKI3OLTXPHSX3VP27VF4CQ3U6GERHEKVKMIINWRI32Q, seg_pos={0 0}, upload=438811]: Done uploading piece. err: <nil> Dec 26 11:27:20.721717071 (499h58m20.592800106s) [noise=true , address=145.79.227.2:4114 , piece_id=MG5FUE6XTOTGSV5X5AAEIUE7C6OT2RHKI7MH7TBLOZW4FTFZKCOQ, seg_pos={0 0}, upload=438811]: Done uploading piece. err: <nil> Dec 26 11:27:20.721952008 (499h58m20.593035043s) [noise=true , address=145.79.227.22:4511 , piece_id=SVD7USFL67GMBF7AAS4C7ZQB3CBIPC7D7WPAS6LM2ZKRKR2QHRDQ, seg_pos={0 0}, upload=438811]: Done uploading piece. err: <nil> Dec 26 11:27:20.916318675 (499h58m20.78740172s) [noise=true , address=45.91.116.11:4120 , piece_id=PAWAYSU7KQ36ADDG3WO4QEGVCWLFUS7R6FEWYYXC54TA27QVE46Q, seg_pos={0 0}, upload=438811]: Done uploading piece. err: <nil> Dec 26 11:27:20.974834679 (499h58m20.845917714s) [noise=true , address=45.91.116.10:4083 , piece_id=E3WD3G4C2OJQ2NKJVIUJWOEXTEXGKZRNVFFWC65QJZX7U3YMESQA, seg_pos={0 0}, upload=438811]: Done uploading piece. err: <nil> Dec 26 11:27:20.994390988 (499h58m20.865474023s) [noise=true , address=45.91.116.14:4192 , piece_id=ET35CSDDTCHRYY6QZLSI62CMUX6KYE74YTJF4QIXF42GMB5TZKLQ, seg_pos={0 0}, upload=438811]: Done uploading piece. err: <nil> Dec 26 11:27:21.015748255 (499h58m20.88683129s) [noise=true , address=45.91.116.12:4226 , piece_id=KZOIBWRAQDAYPRXGBADCSWDIHFLGBX76LI7UAEX3U2WFX64HH76A, seg_pos={0 0}, upload=438811]: Done uploading piece. err: <nil> Dec 26 11:27:21.015806446 (499h58m20.886889492s) [seg_pos={0 0}, upload=438811]: Segment reached optimal threshold of 13 pieces. Dec 26 11:27:21.016021756 (499h58m20.887104791s) [noise=true , address=45.91.116.6:4973 , piece_id=MNDHSLSI5BVR6ZDKIMYIXN4QTW7C3REEWJEA5SBMJ5NUALST4YEA, seg_pos={0 0}, upload=438811]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: write tcp 188.166.225.170:36000->45.91.116.6:4973: use of closed network connection; write tcp 188.166.225.170:36000->45.91.116.6:4973: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36000->45.91.116.6:4973: use of closed network connection Dec 26 11:27:21.016028549 (499h58m20.887111594s) [noise=true , address=45.91.116.6:4973 , piece_id=MNDHSLSI5BVR6ZDKIMYIXN4QTW7C3REEWJEA5SBMJ5NUALST4YEA, seg_pos={0 0}, upload=438811]: Long tail cancellation due to optimal threshold Dec 26 11:27:21.016355050 (499h58m20.887438085s) [noise=true , address=45.91.116.8:4045 , piece_id=NLC7JOZFKFMJAWIQKIRS5UCRLLIS2EHB23LEWXJMBDI3H2JFCGEQ, seg_pos={0 0}, upload=438811]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: read tcp 188.166.225.170:50116->45.91.116.8:4045: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 11:27:21.016362836 (499h58m20.887445881s) [noise=true , address=45.91.116.5:4298 , piece_id=35CTBAKJSQNUUXBBNTR6RY2KWECNLL5AVDOKK3324EUN56TZNDLA, seg_pos={0 0}, upload=438811]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: write tcp 188.166.225.170:53666->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:53666->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53666->45.91.116.5:4298: use of closed network connection Dec 26 11:27:21.016385238 (499h58m20.887468283s) [noise=true , address=45.91.116.5:4298 , piece_id=35CTBAKJSQNUUXBBNTR6RY2KWECNLL5AVDOKK3324EUN56TZNDLA, seg_pos={0 0}, upload=438811]: Long tail cancellation due to optimal threshold Dec 26 11:27:21.016417560 (499h58m20.887500605s) [noise=true , address=45.91.116.8:4045 , piece_id=NLC7JOZFKFMJAWIQKIRS5UCRLLIS2EHB23LEWXJMBDI3H2JFCGEQ, seg_pos={0 0}, upload=438811]: Long tail cancellation due to optimal threshold Dec 26 11:27:21.016458808 (499h58m20.887541843s) [seg_pos={0 0}, upload=438811]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 11:27:21.016545383 (499h58m20.887628428s) [upload=438811]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 11:27:21.016551014 (499h58m20.887634059s) [upload=438811]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 11:27:21.310306233 (499h58m21.181389298s) [upload=438811]: Done uploading segments... Dec 26 11:27:21.310328285 (499h58m21.18141132s) [upload=438811]: Upload finished. err: <nil> Duration: 5.690557616s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [X.........X.................................................] duration: 41.673582369s trace: https://jaeger.dev.storj.io/trace/785a90e8be84a595 FAIL (2025-12-26 11:27:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== KXVRQQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.834123964 (499h15m56.705207009s) [noise=true , address=45.91.116.4:4963 , piece_id=U4T64KQ67C3BDKG5TJSTQWUS4WWYCSVPUZOWJ6YANR2IQ2JD4EFQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.834585043 (499h15m56.705668078s) [noise=true , address=145.79.227.9:4390 , piece_id=JDQNHNGABBGANIHKSKP4VHUWIPQBNNEOPC4ZQCKR3K5TTR4N4SGQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.834961800 (499h15m56.706044845s) [noise=true , address=145.79.227.30:4778 , piece_id=BMEAUWANP7FP6CFRULMIAZUV4JNQEXOUN2EAXJY5XOWLDZINWOIA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.835331334 (499h15m56.706414379s) [noise=true , address=77.90.31.18:4838 , piece_id=VYBIKY3MLEE2CXGS2MT23HX5HUFK25FKYKI4XLXPDRNWCC55ANXQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.835738399 (499h15m56.706821434s) [noise=true , address=145.79.227.20:4459 , piece_id=2EUW2AH7FBVZNJ27XVLRXKMXA737DWKAYBL7CR7PRXXRTO2M644A, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.836118653 (499h15m56.707201698s) [noise=true , address=77.90.31.14:4678 , piece_id=XQHQZJUPB24G3BWGFTFDD42NM6K5HJVHUXBXWH5TW7BU25QGHTMA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.836501923 (499h15m56.707584968s) [noise=true , address=45.91.116.6:4988 , piece_id=VPEKJF74D7SQWKR47MAIY5UJQPBJIEU5AQDKLFGATYIRHFREAQ5A, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.836904940 (499h15m56.707987985s) [noise=true , address=64.49.13.169:4711 , piece_id=PVBVY4TVJLPKYBP6A7LBNZADLN25XBDWZFEK2QVN45YHJATIR7TA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.837388641 (499h15m56.708471676s) [noise=true , address=145.79.227.40:4076 , piece_id=Y6WKUYNPDMBUZSNWEYDL2KKNR6WKHSKGP63S6KAG6F2MWMNTVMBQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.837910455 (499h15m56.7089935s) [noise=true , address=77.90.31.11:4571 , piece_id=OERZACIKIG7HBEV3LL42KUAGKN6AWZEXNQSA52FIYTUKDFWCBW7Q, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.838146514 (499h15m56.709229559s) [noise=true , address=64.49.13.26:4945 , piece_id=KXCFTLWR4HSBTP4GJJ4HEPA6BBEMILYTJEMXY4S5WB24HRMQJOKA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.838469329 (499h15m56.709552364s) [noise=true , address=45.91.116.5:4301 , piece_id=E3ZRKZUKM335ARHV3Z5YGLKQYHG6QV4GCCNHAVMN34BM7UQFWIIA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.838690831 (499h15m56.709773866s) [noise=true , address=45.91.116.14:4204 , piece_id=JWL7OOMKTVAP7WEUB5WTL3F6F5IRNID4KIOYJSYZ655PJSNXAQKQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.839143724 (499h15m56.710226759s) [noise=true , address=45.91.116.9:4272 , piece_id=GGFDVKEW5LUCHQQUP6CWJHJQXUETYUG2UISPVP6XQJGSAJW2KP2Q, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.839678982 (499h15m56.710762027s) [noise=true , address=145.79.227.44:4208 , piece_id=ZLIX7XAUMKRUEWWTQST2KLCOWFJSAFSAQNLUCTTOWVZUBESEI23A, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:58.136536405 (499h15m58.00761944s) [upload=438194]: Next returned nil segment Dec 26 10:44:58.136548628 (499h15m58.007631673s) [upload=438194]: Waiting for error group managing segments... Dec 26 10:44:58.469618858 (499h15m58.340701893s) [noise=true , address=64.49.13.26:4945 , piece_id=KXCFTLWR4HSBTP4GJJ4HEPA6BBEMILYTJEMXY4S5WB24HRMQJOKA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.708819055 (499h15m58.5799021s) [noise=true , address=145.79.227.9:4390 , piece_id=JDQNHNGABBGANIHKSKP4VHUWIPQBNNEOPC4ZQCKR3K5TTR4N4SGQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.714466429 (499h15m58.585549474s) [noise=true , address=145.79.227.40:4076 , piece_id=Y6WKUYNPDMBUZSNWEYDL2KKNR6WKHSKGP63S6KAG6F2MWMNTVMBQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.745407696 (499h15m58.616490731s) [noise=true , address=145.79.227.20:4459 , piece_id=2EUW2AH7FBVZNJ27XVLRXKMXA737DWKAYBL7CR7PRXXRTO2M644A, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.888701070 (499h15m58.759784105s) [noise=true , address=77.90.31.18:4838 , piece_id=VYBIKY3MLEE2CXGS2MT23HX5HUFK25FKYKI4XLXPDRNWCC55ANXQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.948150873 (499h15m58.819233918s) [noise=true , address=45.91.116.5:4301 , piece_id=E3ZRKZUKM335ARHV3Z5YGLKQYHG6QV4GCCNHAVMN34BM7UQFWIIA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.971389744 (499h15m58.842472779s) [noise=true , address=77.90.31.19:4868 , piece_id=WH77PZSXC2VUWLQYLBR2NDEVT6KMD3VZK5WAL53AMRDW74P242LQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.021685314 (499h15m58.892768349s) [noise=true , address=45.91.116.4:4963 , piece_id=U4T64KQ67C3BDKG5TJSTQWUS4WWYCSVPUZOWJ6YANR2IQ2JD4EFQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.045418216 (499h15m58.916501251s) [noise=true , address=45.91.116.14:4204 , piece_id=JWL7OOMKTVAP7WEUB5WTL3F6F5IRNID4KIOYJSYZ655PJSNXAQKQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.056260371 (499h15m58.927343416s) [noise=true , address=45.91.116.9:4272 , piece_id=GGFDVKEW5LUCHQQUP6CWJHJQXUETYUG2UISPVP6XQJGSAJW2KP2Q, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.085713054 (499h15m58.956796089s) [noise=true , address=45.91.116.7:4017 , piece_id=L4GGY6AICMSFMUVYGI6ZMUORYRXNJFPLJY5XJRUAM7NVYQOHPHSQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.640980124 (499h15m59.512063169s) [noise=true , address=145.79.227.30:4778 , piece_id=BMEAUWANP7FP6CFRULMIAZUV4JNQEXOUN2EAXJY5XOWLDZINWOIA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.654515607 (499h15m59.525598652s) [noise=true , address=145.79.227.44:4208 , piece_id=ZLIX7XAUMKRUEWWTQST2KLCOWFJSAFSAQNLUCTTOWVZUBESEI23A, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.660349657 (499h15m59.531432712s) [noise=true , address=77.90.31.11:4571 , piece_id=OERZACIKIG7HBEV3LL42KUAGKN6AWZEXNQSA52FIYTUKDFWCBW7Q, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.660376087 (499h15m59.531459132s) [seg_pos={0 0}, upload=438194]: Segment reached optimal threshold of 14 pieces. Dec 26 10:44:59.660612988 (499h15m59.531696033s) [noise=true , address=64.49.13.169:4711 , piece_id=PVBVY4TVJLPKYBP6A7LBNZADLN25XBDWZFEK2QVN45YHJATIR7TA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:52964->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:52964->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52964->64.49.13.169:4711: use of closed network connection Dec 26 10:44:59.660620753 (499h15m59.531703788s) [noise=true , address=64.49.13.169:4711 , piece_id=PVBVY4TVJLPKYBP6A7LBNZADLN25XBDWZFEK2QVN45YHJATIR7TA, seg_pos={0 0}, upload=438194]: Long tail cancellation due to optimal threshold Dec 26 10:44:59.660893663 (499h15m59.531976698s) [noise=true , address=45.91.116.6:4988 , piece_id=VPEKJF74D7SQWKR47MAIY5UJQPBJIEU5AQDKLFGATYIRHFREAQ5A, seg_pos={0 0}, upload=438194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TrcoCHhAJeEykP9hJNpygoPSdgFV5v6GzL48JpyNPiAj5ZJRS): protocol: expected piece hash; context canceled; EOF Dec 26 10:44:59.660901057 (499h15m59.531984102s) [noise=true , address=45.91.116.6:4988 , piece_id=VPEKJF74D7SQWKR47MAIY5UJQPBJIEU5AQDKLFGATYIRHFREAQ5A, seg_pos={0 0}, upload=438194]: Long tail cancellation due to optimal threshold Dec 26 10:44:59.661226155 (499h15m59.5323092s) [noise=true , address=77.90.31.14:4678 , piece_id=XQHQZJUPB24G3BWGFTFDD42NM6K5HJVHUXBXWH5TW7BU25QGHTMA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12beu3SUZMPNWfbeHV4tLtx9UVZoEHh4Kt7oBhT59w3bGBE1ovP): protocol: write tcp 188.166.225.170:35884->77.90.31.14:4678: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 10:44:59.661249630 (499h15m59.532332685s) [noise=true , address=77.90.31.14:4678 , piece_id=XQHQZJUPB24G3BWGFTFDD42NM6K5HJVHUXBXWH5TW7BU25QGHTMA, seg_pos={0 0}, upload=438194]: Long tail cancellation due to optimal threshold Dec 26 10:44:59.661309705 (499h15m59.53239275s) [noise=true , address=77.90.31.21:4909 , piece_id=H6KT2AKZISCU4VEA6TXJJ3EFGCRBZPPXBEJJEIHZR3GEJZKXVRQQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121RT7tP6czMFAiZ3DjzSSsYgPt3xZXVbXVpkcYxaZYscNiMjL6): protocol: expected piece hash; context canceled; EOF Dec 26 10:44:59.661318962 (499h15m59.532401997s) [noise=true , address=77.90.31.21:4909 , piece_id=H6KT2AKZISCU4VEA6TXJJ3EFGCRBZPPXBEJJEIHZR3GEJZKXVRQQ, seg_pos={0 0}, upload=438194]: Long tail cancellation due to optimal threshold Dec 26 10:44:59.661345674 (499h15m59.532428719s) [seg_pos={0 0}, upload=438194]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 10:44:59.661485770 (499h15m59.532568815s) [upload=438194]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 10:44:59.661493655 (499h15m59.53257669s) [upload=438194]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 10:44:59.968467153 (499h15m59.839550198s) [upload=438194]: Done uploading segments... Dec 26 10:44:59.968484155 (499h15m59.83956719s) [upload=438194]: Upload finished. err: <nil> Duration: 41.673610643s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [......XX.XX..X.......X..X.X......XX......XXX.XX......X.X..XX] duration: 30.003071227s trace: https://jaeger.dev.storj.io/trace/73e14189d001f0be FAIL (2025-12-26 11:27:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: ET35CSDDTCHRYY6QZLSI62CMUX6KYE74YTJF4QIXF42GMB5TZKLQ) context deadline exceeded; piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: KZOIBWRAQDAYPRXGBADCSWDIHFLGBX76LI7UAEX3U2WFX64HH76A) context deadline exceeded ======LOGS====== : false Dec 26 11:27:18.444688377 (499h58m18.315771412s) [upload=438811]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 11:27:18.444695730 (499h58m18.315778806s) [upload=438811]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 11:27:18.687533805 (499h58m18.55861688s) [seg_pos={0 0}, upload=438811]: Begin upload segment... Dec 26 11:27:18.687597567 (499h58m18.558680602s) [seg_pos={0 0}, upload=438811]: Done begin upload segment. Dec 26 11:27:18.687631101 (499h58m18.558714146s) [noise=true , address=145.79.227.50:4374 , piece_id=3Q3JYQS5T3BJ42KJSDGYGRT72IOZYUS4ZDHLZR7MFPWMDCSPXN6Q, seg_pos={0 0}, upload=438811]: Uploading piece... Dec 26 11:27:18.687730058 (499h58m18.558813123s) [noise=true , address=45.91.116.11:4120 , piece_id=PAWAYSU7KQ36ADDG3WO4QEGVCWLFUS7R6FEWYYXC54TA27QVE46Q, seg_pos={0 0}, upload=438811]: Uploading piece... Dec 26 11:27:18.688346993 (499h58m18.559430038s) [noise=true , address=145.79.227.41:4085 , piece_id=7RTIJNFY3V6XBCGGBJRAT2FPJ2TJ4YKQ5RLIKET4X4RAS4J3QOKA, seg_pos={0 0}, upload=438811]: Uploading piece... Dec 26 11:27:18.688804154 (499h58m18.559887189s) [noise=true , address=45.91.116.5:4298 , piece_id=35CTBAKJSQNUUXBBNTR6RY2KWECNLL5AVDOKK3324EUN56TZNDLA, seg_pos={0 0}, upload=438811]: Uploading piece... Dec 26 11:27:18.689256535 (499h58m18.56033958s) [noise=true , address=45.91.116.10:4083 , piece_id=E3WD3G4C2OJQ2NKJVIUJWOEXTEXGKZRNVFFWC65QJZX7U3YMESQA, seg_pos={0 0}, upload=438811]: Uploading piece... Dec 26 11:27:18.689754454 (499h58m18.560837489s) [noise=true , address=45.91.116.14:4192 , piece_id=ET35CSDDTCHRYY6QZLSI62CMUX6KYE74YTJF4QIXF42GMB5TZKLQ, seg_pos={0 0}, upload=438811]: Uploading piece... Dec 26 11:27:18.690287669 (499h58m18.561370704s) [noise=true , address=145.79.227.51:4410 , piece_id=GI6DENEBYMU5VYD2AF3LSGGBHG6RRPO4JPQPANR6ATVWBVR3THCQ, seg_pos={0 0}, upload=438811]: Uploading piece... Dec 26 11:27:18.691277603 (499h58m18.562360649s) [noise=true , address=145.79.227.7:4293 , piece_id=7AEOIDGAQOKI3OLTXPHSX3VP27VF4CQ3U6GERHEKVKMIINWRI32Q, seg_pos={0 0}, upload=438811]: Uploading piece... Dec 26 11:27:18.691958100 (499h58m18.563041135s) [noise=true , address=45.91.116.8:4045 , piece_id=NLC7JOZFKFMJAWIQKIRS5UCRLLIS2EHB23LEWXJMBDI3H2JFCGEQ, seg_pos={0 0}, upload=438811]: Uploading piece... Dec 26 11:27:18.692472329 (499h58m18.563555364s) [noise=true , address=145.79.227.33:4870 , piece_id=TWA7X2QDSVCAHT3L35IWUPLVROP57A47WEANTWAZJHHP55LO4VBA, seg_pos={0 0}, upload=438811]: Uploading piece... Dec 26 11:27:18.693509654 (499h58m18.564592689s) [noise=true , address=145.79.227.22:4511 , piece_id=SVD7USFL67GMBF7AAS4C7ZQB3CBIPC7D7WPAS6LM2ZKRKR2QHRDQ, seg_pos={0 0}, upload=438811]: Uploading piece... Dec 26 11:27:18.693643859 (499h58m18.564726904s) [noise=true , address=45.91.116.12:4226 , piece_id=KZOIBWRAQDAYPRXGBADCSWDIHFLGBX76LI7UAEX3U2WFX64HH76A, seg_pos={0 0}, upload=438811]: Uploading piece... Dec 26 11:27:18.694914189 (499h58m18.565997234s) [noise=true , address=45.91.116.6:4973 , piece_id=MNDHSLSI5BVR6ZDKIMYIXN4QTW7C3REEWJEA5SBMJ5NUALST4YEA, seg_pos={0 0}, upload=438811]: Uploading piece... Dec 26 11:27:18.696476534 (499h58m18.567559569s) [noise=true , address=145.79.227.2:4114 , piece_id=MG5FUE6XTOTGSV5X5AAEIUE7C6OT2RHKI7MH7TBLOZW4FTFZKCOQ, seg_pos={0 0}, upload=438811]: Uploading piece... Dec 26 11:27:18.697132242 (499h58m18.568215277s) [noise=true , address=145.79.227.4:4185 , piece_id=X4S6LO3XO2J3XN3UXSP4Y6KJJQW5XZJQSY77URWDCJQ4D5FACNNQ, seg_pos={0 0}, upload=438811]: Uploading piece... Dec 26 11:27:18.701266985 (499h58m18.57235003s) [noise=true , address=145.79.227.29:4941 , piece_id=U4I2AZ5YDAY34KCZXA5QETDCCGT57H62KAZRBZWHTAPBUKF4TVOA, seg_pos={0 0}, upload=438811]: Uploading piece... Dec 26 11:27:20.412851469 (499h58m20.283934514s) [upload=438811]: Next returned nil segment Dec 26 11:27:20.412868721 (499h58m20.283951756s) [upload=438811]: Waiting for error group managing segments... Dec 26 11:27:20.694011124 (499h58m20.565094159s) [noise=true , address=145.79.227.4:4185 , piece_id=X4S6LO3XO2J3XN3UXSP4Y6KJJQW5XZJQSY77URWDCJQ4D5FACNNQ, seg_pos={0 0}, upload=438811]: Done uploading piece. err: <nil> Dec 26 11:27:20.702024172 (499h58m20.573107207s) [noise=true , address=145.79.227.33:4870 , piece_id=TWA7X2QDSVCAHT3L35IWUPLVROP57A47WEANTWAZJHHP55LO4VBA, seg_pos={0 0}, upload=438811]: Done uploading piece. err: <nil> Dec 26 11:27:20.702150723 (499h58m20.573233758s) [noise=true , address=145.79.227.51:4410 , piece_id=GI6DENEBYMU5VYD2AF3LSGGBHG6RRPO4JPQPANR6ATVWBVR3THCQ, seg_pos={0 0}, upload=438811]: Done uploading piece. err: <nil> Dec 26 11:27:20.703021421 (499h58m20.574104466s) [noise=true , address=145.79.227.50:4374 , piece_id=3Q3JYQS5T3BJ42KJSDGYGRT72IOZYUS4ZDHLZR7MFPWMDCSPXN6Q, seg_pos={0 0}, upload=438811]: Done uploading piece. err: <nil> Dec 26 11:27:20.705665987 (499h58m20.576749022s) [noise=true , address=145.79.227.41:4085 , piece_id=7RTIJNFY3V6XBCGGBJRAT2FPJ2TJ4YKQ5RLIKET4X4RAS4J3QOKA, seg_pos={0 0}, upload=438811]: Done uploading piece. err: <nil> Dec 26 11:27:20.719063959 (499h58m20.590147014s) [noise=true , address=145.79.227.29:4941 , piece_id=U4I2AZ5YDAY34KCZXA5QETDCCGT57H62KAZRBZWHTAPBUKF4TVOA, seg_pos={0 0}, upload=438811]: Done uploading piece. err: <nil> Dec 26 11:27:20.721280740 (499h58m20.592363776s) [noise=true , address=145.79.227.7:4293 , piece_id=7AEOIDGAQOKI3OLTXPHSX3VP27VF4CQ3U6GERHEKVKMIINWRI32Q, seg_pos={0 0}, upload=438811]: Done uploading piece. err: <nil> Dec 26 11:27:20.721717071 (499h58m20.592800106s) [noise=true , address=145.79.227.2:4114 , piece_id=MG5FUE6XTOTGSV5X5AAEIUE7C6OT2RHKI7MH7TBLOZW4FTFZKCOQ, seg_pos={0 0}, upload=438811]: Done uploading piece. err: <nil> Dec 26 11:27:20.721952008 (499h58m20.593035043s) [noise=true , address=145.79.227.22:4511 , piece_id=SVD7USFL67GMBF7AAS4C7ZQB3CBIPC7D7WPAS6LM2ZKRKR2QHRDQ, seg_pos={0 0}, upload=438811]: Done uploading piece. err: <nil> Dec 26 11:27:20.916318675 (499h58m20.78740172s) [noise=true , address=45.91.116.11:4120 , piece_id=PAWAYSU7KQ36ADDG3WO4QEGVCWLFUS7R6FEWYYXC54TA27QVE46Q, seg_pos={0 0}, upload=438811]: Done uploading piece. err: <nil> Dec 26 11:27:20.974834679 (499h58m20.845917714s) [noise=true , address=45.91.116.10:4083 , piece_id=E3WD3G4C2OJQ2NKJVIUJWOEXTEXGKZRNVFFWC65QJZX7U3YMESQA, seg_pos={0 0}, upload=438811]: Done uploading piece. err: <nil> Dec 26 11:27:20.994390988 (499h58m20.865474023s) [noise=true , address=45.91.116.14:4192 , piece_id=ET35CSDDTCHRYY6QZLSI62CMUX6KYE74YTJF4QIXF42GMB5TZKLQ, seg_pos={0 0}, upload=438811]: Done uploading piece. err: <nil> Dec 26 11:27:21.015748255 (499h58m20.88683129s) [noise=true , address=45.91.116.12:4226 , piece_id=KZOIBWRAQDAYPRXGBADCSWDIHFLGBX76LI7UAEX3U2WFX64HH76A, seg_pos={0 0}, upload=438811]: Done uploading piece. err: <nil> Dec 26 11:27:21.015806446 (499h58m20.886889492s) [seg_pos={0 0}, upload=438811]: Segment reached optimal threshold of 13 pieces. Dec 26 11:27:21.016021756 (499h58m20.887104791s) [noise=true , address=45.91.116.6:4973 , piece_id=MNDHSLSI5BVR6ZDKIMYIXN4QTW7C3REEWJEA5SBMJ5NUALST4YEA, seg_pos={0 0}, upload=438811]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: write tcp 188.166.225.170:36000->45.91.116.6:4973: use of closed network connection; write tcp 188.166.225.170:36000->45.91.116.6:4973: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36000->45.91.116.6:4973: use of closed network connection Dec 26 11:27:21.016028549 (499h58m20.887111594s) [noise=true , address=45.91.116.6:4973 , piece_id=MNDHSLSI5BVR6ZDKIMYIXN4QTW7C3REEWJEA5SBMJ5NUALST4YEA, seg_pos={0 0}, upload=438811]: Long tail cancellation due to optimal threshold Dec 26 11:27:21.016355050 (499h58m20.887438085s) [noise=true , address=45.91.116.8:4045 , piece_id=NLC7JOZFKFMJAWIQKIRS5UCRLLIS2EHB23LEWXJMBDI3H2JFCGEQ, seg_pos={0 0}, upload=438811]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: read tcp 188.166.225.170:50116->45.91.116.8:4045: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 11:27:21.016362836 (499h58m20.887445881s) [noise=true , address=45.91.116.5:4298 , piece_id=35CTBAKJSQNUUXBBNTR6RY2KWECNLL5AVDOKK3324EUN56TZNDLA, seg_pos={0 0}, upload=438811]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: write tcp 188.166.225.170:53666->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:53666->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53666->45.91.116.5:4298: use of closed network connection Dec 26 11:27:21.016385238 (499h58m20.887468283s) [noise=true , address=45.91.116.5:4298 , piece_id=35CTBAKJSQNUUXBBNTR6RY2KWECNLL5AVDOKK3324EUN56TZNDLA, seg_pos={0 0}, upload=438811]: Long tail cancellation due to optimal threshold Dec 26 11:27:21.016417560 (499h58m20.887500605s) [noise=true , address=45.91.116.8:4045 , piece_id=NLC7JOZFKFMJAWIQKIRS5UCRLLIS2EHB23LEWXJMBDI3H2JFCGEQ, seg_pos={0 0}, upload=438811]: Long tail cancellation due to optimal threshold Dec 26 11:27:21.016458808 (499h58m20.887541843s) [seg_pos={0 0}, upload=438811]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 11:27:21.016545383 (499h58m20.887628428s) [upload=438811]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 11:27:21.016551014 (499h58m20.887634059s) [upload=438811]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 11:27:21.310306233 (499h58m21.181389298s) [upload=438811]: Done uploading segments... Dec 26 11:27:21.310328285 (499h58m21.18141132s) [upload=438811]: Upload finished. err: <nil> Duration: 30.003096506s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.X.........X................................................] duration: 36.345454748s trace: https://jaeger.dev.storj.io/trace/785a90e8be84a595 FAIL (2025-12-26 11:27:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== KXVRQQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.834123964 (499h15m56.705207009s) [noise=true , address=45.91.116.4:4963 , piece_id=U4T64KQ67C3BDKG5TJSTQWUS4WWYCSVPUZOWJ6YANR2IQ2JD4EFQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.834585043 (499h15m56.705668078s) [noise=true , address=145.79.227.9:4390 , piece_id=JDQNHNGABBGANIHKSKP4VHUWIPQBNNEOPC4ZQCKR3K5TTR4N4SGQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.834961800 (499h15m56.706044845s) [noise=true , address=145.79.227.30:4778 , piece_id=BMEAUWANP7FP6CFRULMIAZUV4JNQEXOUN2EAXJY5XOWLDZINWOIA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.835331334 (499h15m56.706414379s) [noise=true , address=77.90.31.18:4838 , piece_id=VYBIKY3MLEE2CXGS2MT23HX5HUFK25FKYKI4XLXPDRNWCC55ANXQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.835738399 (499h15m56.706821434s) [noise=true , address=145.79.227.20:4459 , piece_id=2EUW2AH7FBVZNJ27XVLRXKMXA737DWKAYBL7CR7PRXXRTO2M644A, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.836118653 (499h15m56.707201698s) [noise=true , address=77.90.31.14:4678 , piece_id=XQHQZJUPB24G3BWGFTFDD42NM6K5HJVHUXBXWH5TW7BU25QGHTMA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.836501923 (499h15m56.707584968s) [noise=true , address=45.91.116.6:4988 , piece_id=VPEKJF74D7SQWKR47MAIY5UJQPBJIEU5AQDKLFGATYIRHFREAQ5A, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.836904940 (499h15m56.707987985s) [noise=true , address=64.49.13.169:4711 , piece_id=PVBVY4TVJLPKYBP6A7LBNZADLN25XBDWZFEK2QVN45YHJATIR7TA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.837388641 (499h15m56.708471676s) [noise=true , address=145.79.227.40:4076 , piece_id=Y6WKUYNPDMBUZSNWEYDL2KKNR6WKHSKGP63S6KAG6F2MWMNTVMBQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.837910455 (499h15m56.7089935s) [noise=true , address=77.90.31.11:4571 , piece_id=OERZACIKIG7HBEV3LL42KUAGKN6AWZEXNQSA52FIYTUKDFWCBW7Q, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.838146514 (499h15m56.709229559s) [noise=true , address=64.49.13.26:4945 , piece_id=KXCFTLWR4HSBTP4GJJ4HEPA6BBEMILYTJEMXY4S5WB24HRMQJOKA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.838469329 (499h15m56.709552364s) [noise=true , address=45.91.116.5:4301 , piece_id=E3ZRKZUKM335ARHV3Z5YGLKQYHG6QV4GCCNHAVMN34BM7UQFWIIA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.838690831 (499h15m56.709773866s) [noise=true , address=45.91.116.14:4204 , piece_id=JWL7OOMKTVAP7WEUB5WTL3F6F5IRNID4KIOYJSYZ655PJSNXAQKQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.839143724 (499h15m56.710226759s) [noise=true , address=45.91.116.9:4272 , piece_id=GGFDVKEW5LUCHQQUP6CWJHJQXUETYUG2UISPVP6XQJGSAJW2KP2Q, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.839678982 (499h15m56.710762027s) [noise=true , address=145.79.227.44:4208 , piece_id=ZLIX7XAUMKRUEWWTQST2KLCOWFJSAFSAQNLUCTTOWVZUBESEI23A, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:58.136536405 (499h15m58.00761944s) [upload=438194]: Next returned nil segment Dec 26 10:44:58.136548628 (499h15m58.007631673s) [upload=438194]: Waiting for error group managing segments... Dec 26 10:44:58.469618858 (499h15m58.340701893s) [noise=true , address=64.49.13.26:4945 , piece_id=KXCFTLWR4HSBTP4GJJ4HEPA6BBEMILYTJEMXY4S5WB24HRMQJOKA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.708819055 (499h15m58.5799021s) [noise=true , address=145.79.227.9:4390 , piece_id=JDQNHNGABBGANIHKSKP4VHUWIPQBNNEOPC4ZQCKR3K5TTR4N4SGQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.714466429 (499h15m58.585549474s) [noise=true , address=145.79.227.40:4076 , piece_id=Y6WKUYNPDMBUZSNWEYDL2KKNR6WKHSKGP63S6KAG6F2MWMNTVMBQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.745407696 (499h15m58.616490731s) [noise=true , address=145.79.227.20:4459 , piece_id=2EUW2AH7FBVZNJ27XVLRXKMXA737DWKAYBL7CR7PRXXRTO2M644A, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.888701070 (499h15m58.759784105s) [noise=true , address=77.90.31.18:4838 , piece_id=VYBIKY3MLEE2CXGS2MT23HX5HUFK25FKYKI4XLXPDRNWCC55ANXQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.948150873 (499h15m58.819233918s) [noise=true , address=45.91.116.5:4301 , piece_id=E3ZRKZUKM335ARHV3Z5YGLKQYHG6QV4GCCNHAVMN34BM7UQFWIIA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.971389744 (499h15m58.842472779s) [noise=true , address=77.90.31.19:4868 , piece_id=WH77PZSXC2VUWLQYLBR2NDEVT6KMD3VZK5WAL53AMRDW74P242LQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.021685314 (499h15m58.892768349s) [noise=true , address=45.91.116.4:4963 , piece_id=U4T64KQ67C3BDKG5TJSTQWUS4WWYCSVPUZOWJ6YANR2IQ2JD4EFQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.045418216 (499h15m58.916501251s) [noise=true , address=45.91.116.14:4204 , piece_id=JWL7OOMKTVAP7WEUB5WTL3F6F5IRNID4KIOYJSYZ655PJSNXAQKQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.056260371 (499h15m58.927343416s) [noise=true , address=45.91.116.9:4272 , piece_id=GGFDVKEW5LUCHQQUP6CWJHJQXUETYUG2UISPVP6XQJGSAJW2KP2Q, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.085713054 (499h15m58.956796089s) [noise=true , address=45.91.116.7:4017 , piece_id=L4GGY6AICMSFMUVYGI6ZMUORYRXNJFPLJY5XJRUAM7NVYQOHPHSQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.640980124 (499h15m59.512063169s) [noise=true , address=145.79.227.30:4778 , piece_id=BMEAUWANP7FP6CFRULMIAZUV4JNQEXOUN2EAXJY5XOWLDZINWOIA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.654515607 (499h15m59.525598652s) [noise=true , address=145.79.227.44:4208 , piece_id=ZLIX7XAUMKRUEWWTQST2KLCOWFJSAFSAQNLUCTTOWVZUBESEI23A, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.660349657 (499h15m59.531432712s) [noise=true , address=77.90.31.11:4571 , piece_id=OERZACIKIG7HBEV3LL42KUAGKN6AWZEXNQSA52FIYTUKDFWCBW7Q, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.660376087 (499h15m59.531459132s) [seg_pos={0 0}, upload=438194]: Segment reached optimal threshold of 14 pieces. Dec 26 10:44:59.660612988 (499h15m59.531696033s) [noise=true , address=64.49.13.169:4711 , piece_id=PVBVY4TVJLPKYBP6A7LBNZADLN25XBDWZFEK2QVN45YHJATIR7TA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:52964->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:52964->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52964->64.49.13.169:4711: use of closed network connection Dec 26 10:44:59.660620753 (499h15m59.531703788s) [noise=true , address=64.49.13.169:4711 , piece_id=PVBVY4TVJLPKYBP6A7LBNZADLN25XBDWZFEK2QVN45YHJATIR7TA, seg_pos={0 0}, upload=438194]: Long tail cancellation due to optimal threshold Dec 26 10:44:59.660893663 (499h15m59.531976698s) [noise=true , address=45.91.116.6:4988 , piece_id=VPEKJF74D7SQWKR47MAIY5UJQPBJIEU5AQDKLFGATYIRHFREAQ5A, seg_pos={0 0}, upload=438194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TrcoCHhAJeEykP9hJNpygoPSdgFV5v6GzL48JpyNPiAj5ZJRS): protocol: expected piece hash; context canceled; EOF Dec 26 10:44:59.660901057 (499h15m59.531984102s) [noise=true , address=45.91.116.6:4988 , piece_id=VPEKJF74D7SQWKR47MAIY5UJQPBJIEU5AQDKLFGATYIRHFREAQ5A, seg_pos={0 0}, upload=438194]: Long tail cancellation due to optimal threshold Dec 26 10:44:59.661226155 (499h15m59.5323092s) [noise=true , address=77.90.31.14:4678 , piece_id=XQHQZJUPB24G3BWGFTFDD42NM6K5HJVHUXBXWH5TW7BU25QGHTMA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12beu3SUZMPNWfbeHV4tLtx9UVZoEHh4Kt7oBhT59w3bGBE1ovP): protocol: write tcp 188.166.225.170:35884->77.90.31.14:4678: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 10:44:59.661249630 (499h15m59.532332685s) [noise=true , address=77.90.31.14:4678 , piece_id=XQHQZJUPB24G3BWGFTFDD42NM6K5HJVHUXBXWH5TW7BU25QGHTMA, seg_pos={0 0}, upload=438194]: Long tail cancellation due to optimal threshold Dec 26 10:44:59.661309705 (499h15m59.53239275s) [noise=true , address=77.90.31.21:4909 , piece_id=H6KT2AKZISCU4VEA6TXJJ3EFGCRBZPPXBEJJEIHZR3GEJZKXVRQQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121RT7tP6czMFAiZ3DjzSSsYgPt3xZXVbXVpkcYxaZYscNiMjL6): protocol: expected piece hash; context canceled; EOF Dec 26 10:44:59.661318962 (499h15m59.532401997s) [noise=true , address=77.90.31.21:4909 , piece_id=H6KT2AKZISCU4VEA6TXJJ3EFGCRBZPPXBEJJEIHZR3GEJZKXVRQQ, seg_pos={0 0}, upload=438194]: Long tail cancellation due to optimal threshold Dec 26 10:44:59.661345674 (499h15m59.532428719s) [seg_pos={0 0}, upload=438194]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 10:44:59.661485770 (499h15m59.532568815s) [upload=438194]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 10:44:59.661493655 (499h15m59.53257669s) [upload=438194]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 10:44:59.968467153 (499h15m59.839550198s) [upload=438194]: Done uploading segments... Dec 26 10:44:59.968484155 (499h15m59.83956719s) [upload=438194]: Upload finished. err: <nil> Duration: 36.345470691s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [.......XX.XX..X.......X..X.X......XX......XXX.XX......X.X..X] duration: 30.003163151s trace: https://jaeger.dev.storj.io/trace/2c481bddd90bfe25 FAIL (2025-12-26 11:26:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== segments... Dec 26 11:26:25.087661154 (499h57m24.958744199s) [upload=438793]: Got next segment. Inline: false Dec 26 11:26:25.087697443 (499h57m24.958780488s) [upload=438793]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 11:26:25.087703184 (499h57m24.958786229s) [upload=438793]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 11:26:25.330371575 (499h57m25.20145463s) [seg_pos={0 0}, upload=438793]: Begin upload segment... Dec 26 11:26:25.330435236 (499h57m25.201518281s) [seg_pos={0 0}, upload=438793]: Done begin upload segment. Dec 26 11:26:25.330471885 (499h57m25.20155492s) [noise=true , address=145.79.227.52:4447 , piece_id=I5LJEFC5NQQ7UDWPYSTM2FZLN743YF7GMM5LKJGELWYBVPCF62XQ, seg_pos={0 0}, upload=438793]: Uploading piece... Dec 26 11:26:25.330548863 (499h57m25.201631908s) [noise=true , address=145.79.227.11:4729 , piece_id=W5VVE5NZMM3FKNLOX5JADY2NRGPOFGL4KUG5CRKJG3CERAF5GMDQ, seg_pos={0 0}, upload=438793]: Uploading piece... Dec 26 11:26:25.331252813 (499h57m25.202335858s) [noise=true , address=145.79.227.2:4114 , piece_id=R73CBHQW5RFK24IK2XDEPEIZSETMDKUKHBOQLFT635LZE3OB4L7Q, seg_pos={0 0}, upload=438793]: Uploading piece... Dec 26 11:26:25.331540921 (499h57m25.202623966s) [noise=true , address=77.90.31.14:4663 , piece_id=WX4VGEA6BSK7HPAXUFXTL6FS7X55LG3SBY3E5B665EK6AYLMQZRA, seg_pos={0 0}, upload=438793]: Uploading piece... Dec 26 11:26:25.331825703 (499h57m25.202908738s) [noise=true , address=45.91.116.6:4976 , piece_id=JLVQOWENXYQ4ORJJOY7SCJNT4PJNKIGMMVR6LUS6DSPDKNZAFL3A, seg_pos={0 0}, upload=438793]: Uploading piece... Dec 26 11:26:25.332112430 (499h57m25.203195465s) [noise=true , address=77.90.31.10:4520 , piece_id=Y3OLEBF35XD2IGXN5GNN63PKMSAGKROS5LC3XDBSKZW2HZG23GCA, seg_pos={0 0}, upload=438793]: Uploading piece... Dec 26 11:26:25.332422681 (499h57m25.203505716s) [noise=true , address=145.79.227.37:4565 , piece_id=FJBR4RBOBYQPUFQKV4XNCE5UQTVE5CJQFM56JZM3JHOSKUL4W7ZQ, seg_pos={0 0}, upload=438793]: Uploading piece... Dec 26 11:26:25.332771505 (499h57m25.20385455s) [noise=true , address=77.90.31.15:4700 , piece_id=BPNKJEMNQAP4WLWHCHDHNKGDR7OEHLQ4O7IEY7NTUVUEPQHOHNPQ, seg_pos={0 0}, upload=438793]: Uploading piece... Dec 26 11:26:25.333235188 (499h57m25.204318233s) [noise=true , address=45.91.116.4:4939 , piece_id=FTIGBPEICBQS26U2LKOXIYOAJI3JJY6IVTEWDVIXISD3N3NC5YSQ, seg_pos={0 0}, upload=438793]: Uploading piece... Dec 26 11:26:25.335421912 (499h57m25.206504957s) [noise=true , address=145.79.227.31:4799 , piece_id=7XTRRVF4UJNMDRDOXUSN3TJNXBIZHALWRJCBFSSRWSNJZADB4LRA, seg_pos={0 0}, upload=438793]: Uploading piece... Dec 26 11:26:25.336802250 (499h57m25.207885285s) [noise=true , address=45.91.116.13:4155 , piece_id=3AAEEWOBALZWCD2EG4NQJ7VQKZPVFG25Y3B5AAUN52KMT5LYU42Q, seg_pos={0 0}, upload=438793]: Uploading piece... Dec 26 11:26:25.338933569 (499h57m25.210016644s) [noise=true , address=145.79.227.47:4267 , piece_id=4KYKGZGEOMRXUTSCQZ6OUNHKGUDBDUXZUIJRSGZS5F7FENTNBK6Q, seg_pos={0 0}, upload=438793]: Uploading piece... Dec 26 11:26:25.343003548 (499h57m25.214086583s) [noise=true , address=145.79.227.48:4301 , piece_id=FFAH3FYEM2HCUTTBEA4AZP7RBLJGCD4A4BKUASHNJAHG54NMJGLQ, seg_pos={0 0}, upload=438793]: Uploading piece... Dec 26 11:26:25.345766209 (499h57m25.216849254s) [noise=true , address=145.79.227.6:4257 , piece_id=GEEGYUN52ZEKW6F3HO7LRSXI56DRASPPUKWU4M6C4OGQKOFIUFQQ, seg_pos={0 0}, upload=438793]: Uploading piece... Dec 26 11:26:25.347858463 (499h57m25.218941498s) [noise=true , address=145.79.227.36:4519 , piece_id=6YMX3M5T4JHU32HOSOT5RPXWD2FVRDFDA3ZH7T3K3EFQZDUUTBEQ, seg_pos={0 0}, upload=438793]: Uploading piece... Dec 26 11:26:25.350061197 (499h57m25.221144232s) [noise=true , address=145.79.227.20:4439 , piece_id=RVPOUV6DQSLEPWR6K2UIV7GCXC5KFJQCHDT3QNRVNNPO5YAMVBNA, seg_pos={0 0}, upload=438793]: Uploading piece... Dec 26 11:26:27.053789862 (499h57m26.924872897s) [upload=438793]: Next returned nil segment Dec 26 11:26:27.053837343 (499h57m26.924920378s) [upload=438793]: Waiting for error group managing segments... Dec 26 11:26:27.336138682 (499h57m27.207221727s) [noise=true , address=145.79.227.11:4729 , piece_id=W5VVE5NZMM3FKNLOX5JADY2NRGPOFGL4KUG5CRKJG3CERAF5GMDQ, seg_pos={0 0}, upload=438793]: Done uploading piece. err: <nil> Dec 26 11:26:27.340576673 (499h57m27.211659718s) [noise=true , address=145.79.227.2:4114 , piece_id=R73CBHQW5RFK24IK2XDEPEIZSETMDKUKHBOQLFT635LZE3OB4L7Q, seg_pos={0 0}, upload=438793]: Done uploading piece. err: <nil> Dec 26 11:26:27.353987740 (499h57m27.225070775s) [noise=true , address=145.79.227.6:4257 , piece_id=GEEGYUN52ZEKW6F3HO7LRSXI56DRASPPUKWU4M6C4OGQKOFIUFQQ, seg_pos={0 0}, upload=438793]: Done uploading piece. err: <nil> Dec 26 11:26:27.355514006 (499h57m27.226597041s) [noise=true , address=145.79.227.37:4565 , piece_id=FJBR4RBOBYQPUFQKV4XNCE5UQTVE5CJQFM56JZM3JHOSKUL4W7ZQ, seg_pos={0 0}, upload=438793]: Done uploading piece. err: <nil> Dec 26 11:26:27.359529702 (499h57m27.230612738s) [noise=true , address=145.79.227.36:4519 , piece_id=6YMX3M5T4JHU32HOSOT5RPXWD2FVRDFDA3ZH7T3K3EFQZDUUTBEQ, seg_pos={0 0}, upload=438793]: Done uploading piece. err: <nil> Dec 26 11:26:27.370335159 (499h57m27.241418194s) [noise=true , address=145.79.227.52:4447 , piece_id=I5LJEFC5NQQ7UDWPYSTM2FZLN743YF7GMM5LKJGELWYBVPCF62XQ, seg_pos={0 0}, upload=438793]: Done uploading piece. err: <nil> Dec 26 11:26:27.371619784 (499h57m27.242702829s) [noise=true , address=145.79.227.31:4799 , piece_id=7XTRRVF4UJNMDRDOXUSN3TJNXBIZHALWRJCBFSSRWSNJZADB4LRA, seg_pos={0 0}, upload=438793]: Done uploading piece. err: <nil> Dec 26 11:26:27.380191477 (499h57m27.251274522s) [noise=true , address=145.79.227.48:4301 , piece_id=FFAH3FYEM2HCUTTBEA4AZP7RBLJGCD4A4BKUASHNJAHG54NMJGLQ, seg_pos={0 0}, upload=438793]: Done uploading piece. err: <nil> Dec 26 11:26:27.390608963 (499h57m27.261692008s) [noise=true , address=145.79.227.20:4439 , piece_id=RVPOUV6DQSLEPWR6K2UIV7GCXC5KFJQCHDT3QNRVNNPO5YAMVBNA, seg_pos={0 0}, upload=438793]: Done uploading piece. err: <nil> Dec 26 11:26:27.390818071 (499h57m27.261901106s) [noise=true , address=145.79.227.47:4267 , piece_id=4KYKGZGEOMRXUTSCQZ6OUNHKGUDBDUXZUIJRSGZS5F7FENTNBK6Q, seg_pos={0 0}, upload=438793]: Done uploading piece. err: <nil> Dec 26 11:26:27.545188117 (499h57m27.416271162s) [noise=true , address=77.90.31.14:4663 , piece_id=WX4VGEA6BSK7HPAXUFXTL6FS7X55LG3SBY3E5B665EK6AYLMQZRA, seg_pos={0 0}, upload=438793]: Done uploading piece. err: <nil> Dec 26 11:26:27.591339758 (499h57m27.462422803s) [noise=true , address=45.91.116.6:4976 , piece_id=JLVQOWENXYQ4ORJJOY7SCJNT4PJNKIGMMVR6LUS6DSPDKNZAFL3A, seg_pos={0 0}, upload=438793]: Done uploading piece. err: <nil> Dec 26 11:26:27.631624416 (499h57m27.502707461s) [noise=true , address=45.91.116.13:4155 , piece_id=3AAEEWOBALZWCD2EG4NQJ7VQKZPVFG25Y3B5AAUN52KMT5LYU42Q, seg_pos={0 0}, upload=438793]: Done uploading piece. err: <nil> Dec 26 11:26:27.631654584 (499h57m27.502737619s) [seg_pos={0 0}, upload=438793]: Segment reached optimal threshold of 13 pieces. Dec 26 11:26:27.631882899 (499h57m27.502965934s) [noise=true , address=77.90.31.15:4700 , piece_id=BPNKJEMNQAP4WLWHCHDHNKGDR7OEHLQ4O7IEY7NTUVUEPQHOHNPQ, seg_pos={0 0}, upload=438793]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: write tcp 188.166.225.170:39482->77.90.31.15:4700: use of closed network connection; write tcp 188.166.225.170:39482->77.90.31.15:4700: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39482->77.90.31.15:4700: use of closed network connection Dec 26 11:26:27.631890233 (499h57m27.502973268s) [noise=true , address=77.90.31.15:4700 , piece_id=BPNKJEMNQAP4WLWHCHDHNKGDR7OEHLQ4O7IEY7NTUVUEPQHOHNPQ, seg_pos={0 0}, upload=438793]: Long tail cancellation due to optimal threshold Dec 26 11:26:27.631955347 (499h57m27.503038382s) [noise=true , address=45.91.116.4:4939 , piece_id=FTIGBPEICBQS26U2LKOXIYOAJI3JJY6IVTEWDVIXISD3N3NC5YSQ, seg_pos={0 0}, upload=438793]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: write tcp 188.166.225.170:44642->45.91.116.4:4939: use of closed network connection; write tcp 188.166.225.170:44642->45.91.116.4:4939: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44642->45.91.116.4:4939: use of closed network connection Dec 26 11:26:27.631971718 (499h57m27.503054753s) [noise=true , address=45.91.116.4:4939 , piece_id=FTIGBPEICBQS26U2LKOXIYOAJI3JJY6IVTEWDVIXISD3N3NC5YSQ, seg_pos={0 0}, upload=438793]: Long tail cancellation due to optimal threshold Dec 26 11:26:27.632143986 (499h57m27.503227021s) [noise=true , address=77.90.31.10:4520 , piece_id=Y3OLEBF35XD2IGXN5GNN63PKMSAGKROS5LC3XDBSKZW2HZG23GCA, seg_pos={0 0}, upload=438793]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Dec 26 11:26:27.632151480 (499h57m27.503234525s) [noise=true , address=77.90.31.10:4520 , piece_id=Y3OLEBF35XD2IGXN5GNN63PKMSAGKROS5LC3XDBSKZW2HZG23GCA, seg_pos={0 0}, upload=438793]: Long tail cancellation due to optimal threshold Dec 26 11:26:27.632170286 (499h57m27.503253321s) [seg_pos={0 0}, upload=438793]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 11:26:27.632238606 (499h57m27.503321641s) [upload=438793]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 11:26:27.632243155 (499h57m27.5033262s) [upload=438793]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 11:26:27.916735536 (499h57m27.787818581s) [upload=438793]: Done uploading segments... Dec 26 11:26:27.916754161 (499h57m27.787837196s) [upload=438793]: Upload finished. err: <nil> Duration: 30.003187016s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..X.........X...............................................] duration: 56.841628549s trace: https://jaeger.dev.storj.io/trace/785a90e8be84a595 FAIL (2025-12-26 11:26:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== KXVRQQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.834123964 (499h15m56.705207009s) [noise=true , address=45.91.116.4:4963 , piece_id=U4T64KQ67C3BDKG5TJSTQWUS4WWYCSVPUZOWJ6YANR2IQ2JD4EFQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.834585043 (499h15m56.705668078s) [noise=true , address=145.79.227.9:4390 , piece_id=JDQNHNGABBGANIHKSKP4VHUWIPQBNNEOPC4ZQCKR3K5TTR4N4SGQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.834961800 (499h15m56.706044845s) [noise=true , address=145.79.227.30:4778 , piece_id=BMEAUWANP7FP6CFRULMIAZUV4JNQEXOUN2EAXJY5XOWLDZINWOIA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.835331334 (499h15m56.706414379s) [noise=true , address=77.90.31.18:4838 , piece_id=VYBIKY3MLEE2CXGS2MT23HX5HUFK25FKYKI4XLXPDRNWCC55ANXQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.835738399 (499h15m56.706821434s) [noise=true , address=145.79.227.20:4459 , piece_id=2EUW2AH7FBVZNJ27XVLRXKMXA737DWKAYBL7CR7PRXXRTO2M644A, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.836118653 (499h15m56.707201698s) [noise=true , address=77.90.31.14:4678 , piece_id=XQHQZJUPB24G3BWGFTFDD42NM6K5HJVHUXBXWH5TW7BU25QGHTMA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.836501923 (499h15m56.707584968s) [noise=true , address=45.91.116.6:4988 , piece_id=VPEKJF74D7SQWKR47MAIY5UJQPBJIEU5AQDKLFGATYIRHFREAQ5A, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.836904940 (499h15m56.707987985s) [noise=true , address=64.49.13.169:4711 , piece_id=PVBVY4TVJLPKYBP6A7LBNZADLN25XBDWZFEK2QVN45YHJATIR7TA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.837388641 (499h15m56.708471676s) [noise=true , address=145.79.227.40:4076 , piece_id=Y6WKUYNPDMBUZSNWEYDL2KKNR6WKHSKGP63S6KAG6F2MWMNTVMBQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.837910455 (499h15m56.7089935s) [noise=true , address=77.90.31.11:4571 , piece_id=OERZACIKIG7HBEV3LL42KUAGKN6AWZEXNQSA52FIYTUKDFWCBW7Q, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.838146514 (499h15m56.709229559s) [noise=true , address=64.49.13.26:4945 , piece_id=KXCFTLWR4HSBTP4GJJ4HEPA6BBEMILYTJEMXY4S5WB24HRMQJOKA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.838469329 (499h15m56.709552364s) [noise=true , address=45.91.116.5:4301 , piece_id=E3ZRKZUKM335ARHV3Z5YGLKQYHG6QV4GCCNHAVMN34BM7UQFWIIA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.838690831 (499h15m56.709773866s) [noise=true , address=45.91.116.14:4204 , piece_id=JWL7OOMKTVAP7WEUB5WTL3F6F5IRNID4KIOYJSYZ655PJSNXAQKQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.839143724 (499h15m56.710226759s) [noise=true , address=45.91.116.9:4272 , piece_id=GGFDVKEW5LUCHQQUP6CWJHJQXUETYUG2UISPVP6XQJGSAJW2KP2Q, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.839678982 (499h15m56.710762027s) [noise=true , address=145.79.227.44:4208 , piece_id=ZLIX7XAUMKRUEWWTQST2KLCOWFJSAFSAQNLUCTTOWVZUBESEI23A, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:58.136536405 (499h15m58.00761944s) [upload=438194]: Next returned nil segment Dec 26 10:44:58.136548628 (499h15m58.007631673s) [upload=438194]: Waiting for error group managing segments... Dec 26 10:44:58.469618858 (499h15m58.340701893s) [noise=true , address=64.49.13.26:4945 , piece_id=KXCFTLWR4HSBTP4GJJ4HEPA6BBEMILYTJEMXY4S5WB24HRMQJOKA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.708819055 (499h15m58.5799021s) [noise=true , address=145.79.227.9:4390 , piece_id=JDQNHNGABBGANIHKSKP4VHUWIPQBNNEOPC4ZQCKR3K5TTR4N4SGQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.714466429 (499h15m58.585549474s) [noise=true , address=145.79.227.40:4076 , piece_id=Y6WKUYNPDMBUZSNWEYDL2KKNR6WKHSKGP63S6KAG6F2MWMNTVMBQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.745407696 (499h15m58.616490731s) [noise=true , address=145.79.227.20:4459 , piece_id=2EUW2AH7FBVZNJ27XVLRXKMXA737DWKAYBL7CR7PRXXRTO2M644A, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.888701070 (499h15m58.759784105s) [noise=true , address=77.90.31.18:4838 , piece_id=VYBIKY3MLEE2CXGS2MT23HX5HUFK25FKYKI4XLXPDRNWCC55ANXQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.948150873 (499h15m58.819233918s) [noise=true , address=45.91.116.5:4301 , piece_id=E3ZRKZUKM335ARHV3Z5YGLKQYHG6QV4GCCNHAVMN34BM7UQFWIIA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.971389744 (499h15m58.842472779s) [noise=true , address=77.90.31.19:4868 , piece_id=WH77PZSXC2VUWLQYLBR2NDEVT6KMD3VZK5WAL53AMRDW74P242LQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.021685314 (499h15m58.892768349s) [noise=true , address=45.91.116.4:4963 , piece_id=U4T64KQ67C3BDKG5TJSTQWUS4WWYCSVPUZOWJ6YANR2IQ2JD4EFQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.045418216 (499h15m58.916501251s) [noise=true , address=45.91.116.14:4204 , piece_id=JWL7OOMKTVAP7WEUB5WTL3F6F5IRNID4KIOYJSYZ655PJSNXAQKQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.056260371 (499h15m58.927343416s) [noise=true , address=45.91.116.9:4272 , piece_id=GGFDVKEW5LUCHQQUP6CWJHJQXUETYUG2UISPVP6XQJGSAJW2KP2Q, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.085713054 (499h15m58.956796089s) [noise=true , address=45.91.116.7:4017 , piece_id=L4GGY6AICMSFMUVYGI6ZMUORYRXNJFPLJY5XJRUAM7NVYQOHPHSQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.640980124 (499h15m59.512063169s) [noise=true , address=145.79.227.30:4778 , piece_id=BMEAUWANP7FP6CFRULMIAZUV4JNQEXOUN2EAXJY5XOWLDZINWOIA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.654515607 (499h15m59.525598652s) [noise=true , address=145.79.227.44:4208 , piece_id=ZLIX7XAUMKRUEWWTQST2KLCOWFJSAFSAQNLUCTTOWVZUBESEI23A, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.660349657 (499h15m59.531432712s) [noise=true , address=77.90.31.11:4571 , piece_id=OERZACIKIG7HBEV3LL42KUAGKN6AWZEXNQSA52FIYTUKDFWCBW7Q, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.660376087 (499h15m59.531459132s) [seg_pos={0 0}, upload=438194]: Segment reached optimal threshold of 14 pieces. Dec 26 10:44:59.660612988 (499h15m59.531696033s) [noise=true , address=64.49.13.169:4711 , piece_id=PVBVY4TVJLPKYBP6A7LBNZADLN25XBDWZFEK2QVN45YHJATIR7TA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:52964->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:52964->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52964->64.49.13.169:4711: use of closed network connection Dec 26 10:44:59.660620753 (499h15m59.531703788s) [noise=true , address=64.49.13.169:4711 , piece_id=PVBVY4TVJLPKYBP6A7LBNZADLN25XBDWZFEK2QVN45YHJATIR7TA, seg_pos={0 0}, upload=438194]: Long tail cancellation due to optimal threshold Dec 26 10:44:59.660893663 (499h15m59.531976698s) [noise=true , address=45.91.116.6:4988 , piece_id=VPEKJF74D7SQWKR47MAIY5UJQPBJIEU5AQDKLFGATYIRHFREAQ5A, seg_pos={0 0}, upload=438194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TrcoCHhAJeEykP9hJNpygoPSdgFV5v6GzL48JpyNPiAj5ZJRS): protocol: expected piece hash; context canceled; EOF Dec 26 10:44:59.660901057 (499h15m59.531984102s) [noise=true , address=45.91.116.6:4988 , piece_id=VPEKJF74D7SQWKR47MAIY5UJQPBJIEU5AQDKLFGATYIRHFREAQ5A, seg_pos={0 0}, upload=438194]: Long tail cancellation due to optimal threshold Dec 26 10:44:59.661226155 (499h15m59.5323092s) [noise=true , address=77.90.31.14:4678 , piece_id=XQHQZJUPB24G3BWGFTFDD42NM6K5HJVHUXBXWH5TW7BU25QGHTMA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12beu3SUZMPNWfbeHV4tLtx9UVZoEHh4Kt7oBhT59w3bGBE1ovP): protocol: write tcp 188.166.225.170:35884->77.90.31.14:4678: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 10:44:59.661249630 (499h15m59.532332685s) [noise=true , address=77.90.31.14:4678 , piece_id=XQHQZJUPB24G3BWGFTFDD42NM6K5HJVHUXBXWH5TW7BU25QGHTMA, seg_pos={0 0}, upload=438194]: Long tail cancellation due to optimal threshold Dec 26 10:44:59.661309705 (499h15m59.53239275s) [noise=true , address=77.90.31.21:4909 , piece_id=H6KT2AKZISCU4VEA6TXJJ3EFGCRBZPPXBEJJEIHZR3GEJZKXVRQQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121RT7tP6czMFAiZ3DjzSSsYgPt3xZXVbXVpkcYxaZYscNiMjL6): protocol: expected piece hash; context canceled; EOF Dec 26 10:44:59.661318962 (499h15m59.532401997s) [noise=true , address=77.90.31.21:4909 , piece_id=H6KT2AKZISCU4VEA6TXJJ3EFGCRBZPPXBEJJEIHZR3GEJZKXVRQQ, seg_pos={0 0}, upload=438194]: Long tail cancellation due to optimal threshold Dec 26 10:44:59.661345674 (499h15m59.532428719s) [seg_pos={0 0}, upload=438194]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 10:44:59.661485770 (499h15m59.532568815s) [upload=438194]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 10:44:59.661493655 (499h15m59.53257669s) [upload=438194]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 10:44:59.968467153 (499h15m59.839550198s) [upload=438194]: Done uploading segments... Dec 26 10:44:59.968484155 (499h15m59.83956719s) [upload=438194]: Upload finished. err: <nil> Duration: 56.841653896s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [X.......XX.XX..X.......X..X.X......XX......XXX.XX......X.X..] duration: 5.749588421s trace: https://jaeger.dev.storj.io/trace/4423565d91067fd6 FAIL (2025-12-26 11:25:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ... Dec 26 11:24:26.232078238 (499h55m26.103161283s) [upload=438769]: Got next segment. Inline: false Dec 26 11:24:26.232116671 (499h55m26.103199706s) [upload=438769]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 11:24:26.232122402 (499h55m26.103205447s) [upload=438769]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 11:24:26.477514509 (499h55m26.348597555s) [seg_pos={0 0}, upload=438769]: Begin upload segment... Dec 26 11:24:26.477569434 (499h55m26.348652469s) [seg_pos={0 0}, upload=438769]: Done begin upload segment. Dec 26 11:24:26.477607276 (499h55m26.348690311s) [noise=true , address=45.91.116.5:4300 , piece_id=KWWAMZ7EHQN23VNEY6XBZLDXADE52MK6ZOKPQTFUBPUZOD7RJV5Q, seg_pos={0 0}, upload=438769]: Uploading piece... Dec 26 11:24:26.477667741 (499h55m26.348750786s) [noise=true , address=145.79.227.39:4013 , piece_id=BGNS2EZ37Q4OHUVJOWZECB3TNF3LHZ47XIHCUZN3FKUMX626ZRRA, seg_pos={0 0}, upload=438769]: Uploading piece... Dec 26 11:24:26.478778406 (499h55m26.349861451s) [noise=true , address=145.79.227.20:4439 , piece_id=Z66QISQ7QKQALWSGNBQQCE223ZKZLZE75PJXA7SFPROPP7ZAIRRA, seg_pos={0 0}, upload=438769]: Uploading piece... Dec 26 11:24:26.479171675 (499h55m26.35025471s) [noise=true , address=77.90.31.13:4629 , piece_id=G2JDDTJJHRORQ223UBZLTNEZLVDMOJZWM6PMM6DBPA7UJPOEZOXQ, seg_pos={0 0}, upload=438769]: Uploading piece... Dec 26 11:24:26.479261726 (499h55m26.350344771s) [noise=true , address=45.91.116.11:4119 , piece_id=QH5OISUFZCYTMEULVCLHF45JEXOELQDV2HWVS6TMNX7G4YQD5LJA, seg_pos={0 0}, upload=438769]: Uploading piece... Dec 26 11:24:26.479942523 (499h55m26.351025568s) [noise=true , address=45.91.116.8:4047 , piece_id=L5G7B3WHLO4SJ6TTGPGO7LAIBGRJUZQWGDIDVSYCNYUGO2VZHT5Q, seg_pos={0 0}, upload=438769]: Uploading piece... Dec 26 11:24:26.481238090 (499h55m26.352321125s) [noise=true , address=45.91.116.13:4155 , piece_id=6YLWXRB46E7VZ6BVU4FURJWQL5RS65CP6QTL32GMD5XRNKEDTWOA, seg_pos={0 0}, upload=438769]: Uploading piece... Dec 26 11:24:26.482140899 (499h55m26.353223944s) [noise=true , address=77.90.31.16:4735 , piece_id=R3LEGBFUSVF6LJNOVF3VDO44HEAREXRANJ2P2PZZZNV5TXI7KUWA, seg_pos={0 0}, upload=438769]: Uploading piece... Dec 26 11:24:26.482700735 (499h55m26.35378377s) [noise=true , address=145.79.227.37:4566 , piece_id=NIS3EZDPQBQ4B6WJ5YAOQWIWOCZWIWJ6QJ2KC5IVODSDCJVQV5CQ, seg_pos={0 0}, upload=438769]: Uploading piece... Dec 26 11:24:26.483083013 (499h55m26.354166058s) [noise=true , address=45.91.116.10:4081 , piece_id=U5HL2D26TB72RFY5PT7J2YQEB2AZA75OFJQP6VLGOWBCRMXSY4VA, seg_pos={0 0}, upload=438769]: Uploading piece... Dec 26 11:24:26.483575250 (499h55m26.354658295s) [noise=true , address=45.91.116.12:4226 , piece_id=HIQPR64F3L3VORXZWG5YTGU4T5FFREMT3ZOXKGIYR46X454MMQJQ, seg_pos={0 0}, upload=438769]: Uploading piece... Dec 26 11:24:26.484251507 (499h55m26.355334542s) [noise=true , address=145.79.227.48:4301 , piece_id=2FM4EHNDLRURQNX77ZHURIBHWJ3UZRDKAMCHPK4UHYCRUOORCNBA, seg_pos={0 0}, upload=438769]: Uploading piece... Dec 26 11:24:26.487142833 (499h55m26.358225868s) [noise=true , address=145.79.227.29:4942 , piece_id=B2V6WTBCTWD7ZMHIN46NXDW7GYKTE7B6YDUOAQRM24EH7REE46BQ, seg_pos={0 0}, upload=438769]: Uploading piece... Dec 26 11:24:26.489065373 (499h55m26.360148409s) [noise=true , address=45.91.116.7:4012 , piece_id=SGV2HXRO6PQWSK5S3OT4C6UR3OCFNJ3GMEMEJNJUGZOLGTVBP5ZA, seg_pos={0 0}, upload=438769]: Uploading piece... Dec 26 11:24:26.490417198 (499h55m26.361500233s) [noise=true , address=145.79.227.52:4446 , piece_id=QT6AURNCLTTSRKEERI3FMKRDLF6K6MEVQMLZL5O34M2YO4MXCLRQ, seg_pos={0 0}, upload=438769]: Uploading piece... Dec 26 11:24:26.494672611 (499h55m26.365755646s) [noise=true , address=145.79.227.16:4910 , piece_id=QOORN3OTMYVCFECSIM6C46QYRH6DNWB6XES4DZ445E7UWH5SQM4Q, seg_pos={0 0}, upload=438769]: Uploading piece... Dec 26 11:24:28.199910059 (499h55m28.070993094s) [upload=438769]: Next returned nil segment Dec 26 11:24:28.199925618 (499h55m28.071008653s) [upload=438769]: Waiting for error group managing segments... Dec 26 11:24:28.483828588 (499h55m28.354911633s) [noise=true , address=145.79.227.37:4566 , piece_id=NIS3EZDPQBQ4B6WJ5YAOQWIWOCZWIWJ6QJ2KC5IVODSDCJVQV5CQ, seg_pos={0 0}, upload=438769]: Done uploading piece. err: <nil> Dec 26 11:24:28.486064325 (499h55m28.35714737s) [noise=true , address=145.79.227.20:4439 , piece_id=Z66QISQ7QKQALWSGNBQQCE223ZKZLZE75PJXA7SFPROPP7ZAIRRA, seg_pos={0 0}, upload=438769]: Done uploading piece. err: <nil> Dec 26 11:24:28.492430918 (499h55m28.363513963s) [noise=true , address=145.79.227.48:4301 , piece_id=2FM4EHNDLRURQNX77ZHURIBHWJ3UZRDKAMCHPK4UHYCRUOORCNBA, seg_pos={0 0}, upload=438769]: Done uploading piece. err: <nil> Dec 26 11:24:28.503261771 (499h55m28.374344806s) [noise=true , address=145.79.227.39:4013 , piece_id=BGNS2EZ37Q4OHUVJOWZECB3TNF3LHZ47XIHCUZN3FKUMX626ZRRA, seg_pos={0 0}, upload=438769]: Done uploading piece. err: <nil> Dec 26 11:24:28.505613890 (499h55m28.376696925s) [noise=true , address=145.79.227.52:4446 , piece_id=QT6AURNCLTTSRKEERI3FMKRDLF6K6MEVQMLZL5O34M2YO4MXCLRQ, seg_pos={0 0}, upload=438769]: Done uploading piece. err: <nil> Dec 26 11:24:28.517816536 (499h55m28.388899582s) [noise=true , address=145.79.227.29:4942 , piece_id=B2V6WTBCTWD7ZMHIN46NXDW7GYKTE7B6YDUOAQRM24EH7REE46BQ, seg_pos={0 0}, upload=438769]: Done uploading piece. err: <nil> Dec 26 11:24:28.527772374 (499h55m28.398855409s) [noise=true , address=145.79.227.16:4910 , piece_id=QOORN3OTMYVCFECSIM6C46QYRH6DNWB6XES4DZ445E7UWH5SQM4Q, seg_pos={0 0}, upload=438769]: Done uploading piece. err: <nil> Dec 26 11:24:28.845118774 (499h55m28.716201819s) [noise=true , address=45.91.116.10:4081 , piece_id=U5HL2D26TB72RFY5PT7J2YQEB2AZA75OFJQP6VLGOWBCRMXSY4VA, seg_pos={0 0}, upload=438769]: Done uploading piece. err: <nil> Dec 26 11:24:28.860180034 (499h55m28.731263069s) [noise=true , address=45.91.116.11:4119 , piece_id=QH5OISUFZCYTMEULVCLHF45JEXOELQDV2HWVS6TMNX7G4YQD5LJA, seg_pos={0 0}, upload=438769]: Done uploading piece. err: <nil> Dec 26 11:24:28.878376293 (499h55m28.749459328s) [noise=true , address=45.91.116.7:4012 , piece_id=SGV2HXRO6PQWSK5S3OT4C6UR3OCFNJ3GMEMEJNJUGZOLGTVBP5ZA, seg_pos={0 0}, upload=438769]: Done uploading piece. err: <nil> Dec 26 11:24:28.949507468 (499h55m28.820590514s) [noise=true , address=77.90.31.16:4735 , piece_id=R3LEGBFUSVF6LJNOVF3VDO44HEAREXRANJ2P2PZZZNV5TXI7KUWA, seg_pos={0 0}, upload=438769]: Done uploading piece. err: <nil> Dec 26 11:24:29.045945892 (499h55m28.917028938s) [noise=true , address=45.91.116.5:4300 , piece_id=KWWAMZ7EHQN23VNEY6XBZLDXADE52MK6ZOKPQTFUBPUZOD7RJV5Q, seg_pos={0 0}, upload=438769]: Done uploading piece. err: <nil> Dec 26 11:24:29.183983449 (499h55m29.055066495s) [noise=true , address=45.91.116.8:4047 , piece_id=L5G7B3WHLO4SJ6TTGPGO7LAIBGRJUZQWGDIDVSYCNYUGO2VZHT5Q, seg_pos={0 0}, upload=438769]: Done uploading piece. err: <nil> Dec 26 11:24:29.184010211 (499h55m29.055093246s) [seg_pos={0 0}, upload=438769]: Segment reached optimal threshold of 13 pieces. Dec 26 11:24:29.184242834 (499h55m29.055325879s) [noise=true , address=77.90.31.13:4629 , piece_id=G2JDDTJJHRORQ223UBZLTNEZLVDMOJZWM6PMM6DBPA7UJPOEZOXQ, seg_pos={0 0}, upload=438769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: expected piece hash; context canceled; EOF Dec 26 11:24:29.184252101 (499h55m29.055335146s) [noise=true , address=77.90.31.13:4629 , piece_id=G2JDDTJJHRORQ223UBZLTNEZLVDMOJZWM6PMM6DBPA7UJPOEZOXQ, seg_pos={0 0}, upload=438769]: Long tail cancellation due to optimal threshold Dec 26 11:24:29.184470006 (499h55m29.055553051s) [noise=true , address=45.91.116.12:4226 , piece_id=HIQPR64F3L3VORXZWG5YTGU4T5FFREMT3ZOXKGIYR46X454MMQJQ, seg_pos={0 0}, upload=438769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: write tcp 188.166.225.170:36500->45.91.116.12:4226: use of closed network connection; write tcp 188.166.225.170:36500->45.91.116.12:4226: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36500->45.91.116.12:4226: use of closed network connection Dec 26 11:24:29.184490976 (499h55m29.055574011s) [noise=true , address=45.91.116.12:4226 , piece_id=HIQPR64F3L3VORXZWG5YTGU4T5FFREMT3ZOXKGIYR46X454MMQJQ, seg_pos={0 0}, upload=438769]: Long tail cancellation due to optimal threshold Dec 26 11:24:29.184563194 (499h55m29.055646239s) [noise=true , address=45.91.116.13:4155 , piece_id=6YLWXRB46E7VZ6BVU4FURJWQL5RS65CP6QTL32GMD5XRNKEDTWOA, seg_pos={0 0}, upload=438769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: write tcp 188.166.225.170:34548->45.91.116.13:4155: use of closed network connection; write tcp 188.166.225.170:34548->45.91.116.13:4155: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34548->45.91.116.13:4155: use of closed network connection Dec 26 11:24:29.184576740 (499h55m29.055659775s) [noise=true , address=45.91.116.13:4155 , piece_id=6YLWXRB46E7VZ6BVU4FURJWQL5RS65CP6QTL32GMD5XRNKEDTWOA, seg_pos={0 0}, upload=438769]: Long tail cancellation due to optimal threshold Dec 26 11:24:29.184629009 (499h55m29.055712054s) [seg_pos={0 0}, upload=438769]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 11:24:29.184746142 (499h55m29.055829187s) [upload=438769]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 11:24:29.184757143 (499h55m29.055840189s) [upload=438769]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 11:24:29.478019373 (499h55m29.349102428s) [upload=438769]: Done uploading segments... Dec 26 11:24:29.478044351 (499h55m29.349127386s) [upload=438769]: Upload finished. err: <nil> Duration: 5.74961481s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [.X.......XX.XX..X.......X..X.X......XX......XXX.XX......X.X.] duration: 5.873456855s trace: https://jaeger.dev.storj.io/trace/4423565d91067fd6 FAIL (2025-12-26 11:25:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ... Dec 26 11:24:26.232078238 (499h55m26.103161283s) [upload=438769]: Got next segment. Inline: false Dec 26 11:24:26.232116671 (499h55m26.103199706s) [upload=438769]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 11:24:26.232122402 (499h55m26.103205447s) [upload=438769]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 11:24:26.477514509 (499h55m26.348597555s) [seg_pos={0 0}, upload=438769]: Begin upload segment... Dec 26 11:24:26.477569434 (499h55m26.348652469s) [seg_pos={0 0}, upload=438769]: Done begin upload segment. Dec 26 11:24:26.477607276 (499h55m26.348690311s) [noise=true , address=45.91.116.5:4300 , piece_id=KWWAMZ7EHQN23VNEY6XBZLDXADE52MK6ZOKPQTFUBPUZOD7RJV5Q, seg_pos={0 0}, upload=438769]: Uploading piece... Dec 26 11:24:26.477667741 (499h55m26.348750786s) [noise=true , address=145.79.227.39:4013 , piece_id=BGNS2EZ37Q4OHUVJOWZECB3TNF3LHZ47XIHCUZN3FKUMX626ZRRA, seg_pos={0 0}, upload=438769]: Uploading piece... Dec 26 11:24:26.478778406 (499h55m26.349861451s) [noise=true , address=145.79.227.20:4439 , piece_id=Z66QISQ7QKQALWSGNBQQCE223ZKZLZE75PJXA7SFPROPP7ZAIRRA, seg_pos={0 0}, upload=438769]: Uploading piece... Dec 26 11:24:26.479171675 (499h55m26.35025471s) [noise=true , address=77.90.31.13:4629 , piece_id=G2JDDTJJHRORQ223UBZLTNEZLVDMOJZWM6PMM6DBPA7UJPOEZOXQ, seg_pos={0 0}, upload=438769]: Uploading piece... Dec 26 11:24:26.479261726 (499h55m26.350344771s) [noise=true , address=45.91.116.11:4119 , piece_id=QH5OISUFZCYTMEULVCLHF45JEXOELQDV2HWVS6TMNX7G4YQD5LJA, seg_pos={0 0}, upload=438769]: Uploading piece... Dec 26 11:24:26.479942523 (499h55m26.351025568s) [noise=true , address=45.91.116.8:4047 , piece_id=L5G7B3WHLO4SJ6TTGPGO7LAIBGRJUZQWGDIDVSYCNYUGO2VZHT5Q, seg_pos={0 0}, upload=438769]: Uploading piece... Dec 26 11:24:26.481238090 (499h55m26.352321125s) [noise=true , address=45.91.116.13:4155 , piece_id=6YLWXRB46E7VZ6BVU4FURJWQL5RS65CP6QTL32GMD5XRNKEDTWOA, seg_pos={0 0}, upload=438769]: Uploading piece... Dec 26 11:24:26.482140899 (499h55m26.353223944s) [noise=true , address=77.90.31.16:4735 , piece_id=R3LEGBFUSVF6LJNOVF3VDO44HEAREXRANJ2P2PZZZNV5TXI7KUWA, seg_pos={0 0}, upload=438769]: Uploading piece... Dec 26 11:24:26.482700735 (499h55m26.35378377s) [noise=true , address=145.79.227.37:4566 , piece_id=NIS3EZDPQBQ4B6WJ5YAOQWIWOCZWIWJ6QJ2KC5IVODSDCJVQV5CQ, seg_pos={0 0}, upload=438769]: Uploading piece... Dec 26 11:24:26.483083013 (499h55m26.354166058s) [noise=true , address=45.91.116.10:4081 , piece_id=U5HL2D26TB72RFY5PT7J2YQEB2AZA75OFJQP6VLGOWBCRMXSY4VA, seg_pos={0 0}, upload=438769]: Uploading piece... Dec 26 11:24:26.483575250 (499h55m26.354658295s) [noise=true , address=45.91.116.12:4226 , piece_id=HIQPR64F3L3VORXZWG5YTGU4T5FFREMT3ZOXKGIYR46X454MMQJQ, seg_pos={0 0}, upload=438769]: Uploading piece... Dec 26 11:24:26.484251507 (499h55m26.355334542s) [noise=true , address=145.79.227.48:4301 , piece_id=2FM4EHNDLRURQNX77ZHURIBHWJ3UZRDKAMCHPK4UHYCRUOORCNBA, seg_pos={0 0}, upload=438769]: Uploading piece... Dec 26 11:24:26.487142833 (499h55m26.358225868s) [noise=true , address=145.79.227.29:4942 , piece_id=B2V6WTBCTWD7ZMHIN46NXDW7GYKTE7B6YDUOAQRM24EH7REE46BQ, seg_pos={0 0}, upload=438769]: Uploading piece... Dec 26 11:24:26.489065373 (499h55m26.360148409s) [noise=true , address=45.91.116.7:4012 , piece_id=SGV2HXRO6PQWSK5S3OT4C6UR3OCFNJ3GMEMEJNJUGZOLGTVBP5ZA, seg_pos={0 0}, upload=438769]: Uploading piece... Dec 26 11:24:26.490417198 (499h55m26.361500233s) [noise=true , address=145.79.227.52:4446 , piece_id=QT6AURNCLTTSRKEERI3FMKRDLF6K6MEVQMLZL5O34M2YO4MXCLRQ, seg_pos={0 0}, upload=438769]: Uploading piece... Dec 26 11:24:26.494672611 (499h55m26.365755646s) [noise=true , address=145.79.227.16:4910 , piece_id=QOORN3OTMYVCFECSIM6C46QYRH6DNWB6XES4DZ445E7UWH5SQM4Q, seg_pos={0 0}, upload=438769]: Uploading piece... Dec 26 11:24:28.199910059 (499h55m28.070993094s) [upload=438769]: Next returned nil segment Dec 26 11:24:28.199925618 (499h55m28.071008653s) [upload=438769]: Waiting for error group managing segments... Dec 26 11:24:28.483828588 (499h55m28.354911633s) [noise=true , address=145.79.227.37:4566 , piece_id=NIS3EZDPQBQ4B6WJ5YAOQWIWOCZWIWJ6QJ2KC5IVODSDCJVQV5CQ, seg_pos={0 0}, upload=438769]: Done uploading piece. err: <nil> Dec 26 11:24:28.486064325 (499h55m28.35714737s) [noise=true , address=145.79.227.20:4439 , piece_id=Z66QISQ7QKQALWSGNBQQCE223ZKZLZE75PJXA7SFPROPP7ZAIRRA, seg_pos={0 0}, upload=438769]: Done uploading piece. err: <nil> Dec 26 11:24:28.492430918 (499h55m28.363513963s) [noise=true , address=145.79.227.48:4301 , piece_id=2FM4EHNDLRURQNX77ZHURIBHWJ3UZRDKAMCHPK4UHYCRUOORCNBA, seg_pos={0 0}, upload=438769]: Done uploading piece. err: <nil> Dec 26 11:24:28.503261771 (499h55m28.374344806s) [noise=true , address=145.79.227.39:4013 , piece_id=BGNS2EZ37Q4OHUVJOWZECB3TNF3LHZ47XIHCUZN3FKUMX626ZRRA, seg_pos={0 0}, upload=438769]: Done uploading piece. err: <nil> Dec 26 11:24:28.505613890 (499h55m28.376696925s) [noise=true , address=145.79.227.52:4446 , piece_id=QT6AURNCLTTSRKEERI3FMKRDLF6K6MEVQMLZL5O34M2YO4MXCLRQ, seg_pos={0 0}, upload=438769]: Done uploading piece. err: <nil> Dec 26 11:24:28.517816536 (499h55m28.388899582s) [noise=true , address=145.79.227.29:4942 , piece_id=B2V6WTBCTWD7ZMHIN46NXDW7GYKTE7B6YDUOAQRM24EH7REE46BQ, seg_pos={0 0}, upload=438769]: Done uploading piece. err: <nil> Dec 26 11:24:28.527772374 (499h55m28.398855409s) [noise=true , address=145.79.227.16:4910 , piece_id=QOORN3OTMYVCFECSIM6C46QYRH6DNWB6XES4DZ445E7UWH5SQM4Q, seg_pos={0 0}, upload=438769]: Done uploading piece. err: <nil> Dec 26 11:24:28.845118774 (499h55m28.716201819s) [noise=true , address=45.91.116.10:4081 , piece_id=U5HL2D26TB72RFY5PT7J2YQEB2AZA75OFJQP6VLGOWBCRMXSY4VA, seg_pos={0 0}, upload=438769]: Done uploading piece. err: <nil> Dec 26 11:24:28.860180034 (499h55m28.731263069s) [noise=true , address=45.91.116.11:4119 , piece_id=QH5OISUFZCYTMEULVCLHF45JEXOELQDV2HWVS6TMNX7G4YQD5LJA, seg_pos={0 0}, upload=438769]: Done uploading piece. err: <nil> Dec 26 11:24:28.878376293 (499h55m28.749459328s) [noise=true , address=45.91.116.7:4012 , piece_id=SGV2HXRO6PQWSK5S3OT4C6UR3OCFNJ3GMEMEJNJUGZOLGTVBP5ZA, seg_pos={0 0}, upload=438769]: Done uploading piece. err: <nil> Dec 26 11:24:28.949507468 (499h55m28.820590514s) [noise=true , address=77.90.31.16:4735 , piece_id=R3LEGBFUSVF6LJNOVF3VDO44HEAREXRANJ2P2PZZZNV5TXI7KUWA, seg_pos={0 0}, upload=438769]: Done uploading piece. err: <nil> Dec 26 11:24:29.045945892 (499h55m28.917028938s) [noise=true , address=45.91.116.5:4300 , piece_id=KWWAMZ7EHQN23VNEY6XBZLDXADE52MK6ZOKPQTFUBPUZOD7RJV5Q, seg_pos={0 0}, upload=438769]: Done uploading piece. err: <nil> Dec 26 11:24:29.183983449 (499h55m29.055066495s) [noise=true , address=45.91.116.8:4047 , piece_id=L5G7B3WHLO4SJ6TTGPGO7LAIBGRJUZQWGDIDVSYCNYUGO2VZHT5Q, seg_pos={0 0}, upload=438769]: Done uploading piece. err: <nil> Dec 26 11:24:29.184010211 (499h55m29.055093246s) [seg_pos={0 0}, upload=438769]: Segment reached optimal threshold of 13 pieces. Dec 26 11:24:29.184242834 (499h55m29.055325879s) [noise=true , address=77.90.31.13:4629 , piece_id=G2JDDTJJHRORQ223UBZLTNEZLVDMOJZWM6PMM6DBPA7UJPOEZOXQ, seg_pos={0 0}, upload=438769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: expected piece hash; context canceled; EOF Dec 26 11:24:29.184252101 (499h55m29.055335146s) [noise=true , address=77.90.31.13:4629 , piece_id=G2JDDTJJHRORQ223UBZLTNEZLVDMOJZWM6PMM6DBPA7UJPOEZOXQ, seg_pos={0 0}, upload=438769]: Long tail cancellation due to optimal threshold Dec 26 11:24:29.184470006 (499h55m29.055553051s) [noise=true , address=45.91.116.12:4226 , piece_id=HIQPR64F3L3VORXZWG5YTGU4T5FFREMT3ZOXKGIYR46X454MMQJQ, seg_pos={0 0}, upload=438769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: write tcp 188.166.225.170:36500->45.91.116.12:4226: use of closed network connection; write tcp 188.166.225.170:36500->45.91.116.12:4226: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36500->45.91.116.12:4226: use of closed network connection Dec 26 11:24:29.184490976 (499h55m29.055574011s) [noise=true , address=45.91.116.12:4226 , piece_id=HIQPR64F3L3VORXZWG5YTGU4T5FFREMT3ZOXKGIYR46X454MMQJQ, seg_pos={0 0}, upload=438769]: Long tail cancellation due to optimal threshold Dec 26 11:24:29.184563194 (499h55m29.055646239s) [noise=true , address=45.91.116.13:4155 , piece_id=6YLWXRB46E7VZ6BVU4FURJWQL5RS65CP6QTL32GMD5XRNKEDTWOA, seg_pos={0 0}, upload=438769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: write tcp 188.166.225.170:34548->45.91.116.13:4155: use of closed network connection; write tcp 188.166.225.170:34548->45.91.116.13:4155: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34548->45.91.116.13:4155: use of closed network connection Dec 26 11:24:29.184576740 (499h55m29.055659775s) [noise=true , address=45.91.116.13:4155 , piece_id=6YLWXRB46E7VZ6BVU4FURJWQL5RS65CP6QTL32GMD5XRNKEDTWOA, seg_pos={0 0}, upload=438769]: Long tail cancellation due to optimal threshold Dec 26 11:24:29.184629009 (499h55m29.055712054s) [seg_pos={0 0}, upload=438769]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 11:24:29.184746142 (499h55m29.055829187s) [upload=438769]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 11:24:29.184757143 (499h55m29.055840189s) [upload=438769]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 11:24:29.478019373 (499h55m29.349102428s) [upload=438769]: Done uploading segments... Dec 26 11:24:29.478044351 (499h55m29.349127386s) [upload=438769]: Upload finished. err: <nil> Duration: 5.873512392s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...X.........X..............................................] duration: 38.15590528s trace: https://jaeger.dev.storj.io/trace/785a90e8be84a595 FAIL (2025-12-26 11:25:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== KXVRQQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.834123964 (499h15m56.705207009s) [noise=true , address=45.91.116.4:4963 , piece_id=U4T64KQ67C3BDKG5TJSTQWUS4WWYCSVPUZOWJ6YANR2IQ2JD4EFQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.834585043 (499h15m56.705668078s) [noise=true , address=145.79.227.9:4390 , piece_id=JDQNHNGABBGANIHKSKP4VHUWIPQBNNEOPC4ZQCKR3K5TTR4N4SGQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.834961800 (499h15m56.706044845s) [noise=true , address=145.79.227.30:4778 , piece_id=BMEAUWANP7FP6CFRULMIAZUV4JNQEXOUN2EAXJY5XOWLDZINWOIA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.835331334 (499h15m56.706414379s) [noise=true , address=77.90.31.18:4838 , piece_id=VYBIKY3MLEE2CXGS2MT23HX5HUFK25FKYKI4XLXPDRNWCC55ANXQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.835738399 (499h15m56.706821434s) [noise=true , address=145.79.227.20:4459 , piece_id=2EUW2AH7FBVZNJ27XVLRXKMXA737DWKAYBL7CR7PRXXRTO2M644A, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.836118653 (499h15m56.707201698s) [noise=true , address=77.90.31.14:4678 , piece_id=XQHQZJUPB24G3BWGFTFDD42NM6K5HJVHUXBXWH5TW7BU25QGHTMA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.836501923 (499h15m56.707584968s) [noise=true , address=45.91.116.6:4988 , piece_id=VPEKJF74D7SQWKR47MAIY5UJQPBJIEU5AQDKLFGATYIRHFREAQ5A, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.836904940 (499h15m56.707987985s) [noise=true , address=64.49.13.169:4711 , piece_id=PVBVY4TVJLPKYBP6A7LBNZADLN25XBDWZFEK2QVN45YHJATIR7TA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.837388641 (499h15m56.708471676s) [noise=true , address=145.79.227.40:4076 , piece_id=Y6WKUYNPDMBUZSNWEYDL2KKNR6WKHSKGP63S6KAG6F2MWMNTVMBQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.837910455 (499h15m56.7089935s) [noise=true , address=77.90.31.11:4571 , piece_id=OERZACIKIG7HBEV3LL42KUAGKN6AWZEXNQSA52FIYTUKDFWCBW7Q, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.838146514 (499h15m56.709229559s) [noise=true , address=64.49.13.26:4945 , piece_id=KXCFTLWR4HSBTP4GJJ4HEPA6BBEMILYTJEMXY4S5WB24HRMQJOKA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.838469329 (499h15m56.709552364s) [noise=true , address=45.91.116.5:4301 , piece_id=E3ZRKZUKM335ARHV3Z5YGLKQYHG6QV4GCCNHAVMN34BM7UQFWIIA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.838690831 (499h15m56.709773866s) [noise=true , address=45.91.116.14:4204 , piece_id=JWL7OOMKTVAP7WEUB5WTL3F6F5IRNID4KIOYJSYZ655PJSNXAQKQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.839143724 (499h15m56.710226759s) [noise=true , address=45.91.116.9:4272 , piece_id=GGFDVKEW5LUCHQQUP6CWJHJQXUETYUG2UISPVP6XQJGSAJW2KP2Q, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.839678982 (499h15m56.710762027s) [noise=true , address=145.79.227.44:4208 , piece_id=ZLIX7XAUMKRUEWWTQST2KLCOWFJSAFSAQNLUCTTOWVZUBESEI23A, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:58.136536405 (499h15m58.00761944s) [upload=438194]: Next returned nil segment Dec 26 10:44:58.136548628 (499h15m58.007631673s) [upload=438194]: Waiting for error group managing segments... Dec 26 10:44:58.469618858 (499h15m58.340701893s) [noise=true , address=64.49.13.26:4945 , piece_id=KXCFTLWR4HSBTP4GJJ4HEPA6BBEMILYTJEMXY4S5WB24HRMQJOKA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.708819055 (499h15m58.5799021s) [noise=true , address=145.79.227.9:4390 , piece_id=JDQNHNGABBGANIHKSKP4VHUWIPQBNNEOPC4ZQCKR3K5TTR4N4SGQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.714466429 (499h15m58.585549474s) [noise=true , address=145.79.227.40:4076 , piece_id=Y6WKUYNPDMBUZSNWEYDL2KKNR6WKHSKGP63S6KAG6F2MWMNTVMBQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.745407696 (499h15m58.616490731s) [noise=true , address=145.79.227.20:4459 , piece_id=2EUW2AH7FBVZNJ27XVLRXKMXA737DWKAYBL7CR7PRXXRTO2M644A, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.888701070 (499h15m58.759784105s) [noise=true , address=77.90.31.18:4838 , piece_id=VYBIKY3MLEE2CXGS2MT23HX5HUFK25FKYKI4XLXPDRNWCC55ANXQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.948150873 (499h15m58.819233918s) [noise=true , address=45.91.116.5:4301 , piece_id=E3ZRKZUKM335ARHV3Z5YGLKQYHG6QV4GCCNHAVMN34BM7UQFWIIA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.971389744 (499h15m58.842472779s) [noise=true , address=77.90.31.19:4868 , piece_id=WH77PZSXC2VUWLQYLBR2NDEVT6KMD3VZK5WAL53AMRDW74P242LQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.021685314 (499h15m58.892768349s) [noise=true , address=45.91.116.4:4963 , piece_id=U4T64KQ67C3BDKG5TJSTQWUS4WWYCSVPUZOWJ6YANR2IQ2JD4EFQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.045418216 (499h15m58.916501251s) [noise=true , address=45.91.116.14:4204 , piece_id=JWL7OOMKTVAP7WEUB5WTL3F6F5IRNID4KIOYJSYZ655PJSNXAQKQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.056260371 (499h15m58.927343416s) [noise=true , address=45.91.116.9:4272 , piece_id=GGFDVKEW5LUCHQQUP6CWJHJQXUETYUG2UISPVP6XQJGSAJW2KP2Q, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.085713054 (499h15m58.956796089s) [noise=true , address=45.91.116.7:4017 , piece_id=L4GGY6AICMSFMUVYGI6ZMUORYRXNJFPLJY5XJRUAM7NVYQOHPHSQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.640980124 (499h15m59.512063169s) [noise=true , address=145.79.227.30:4778 , piece_id=BMEAUWANP7FP6CFRULMIAZUV4JNQEXOUN2EAXJY5XOWLDZINWOIA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.654515607 (499h15m59.525598652s) [noise=true , address=145.79.227.44:4208 , piece_id=ZLIX7XAUMKRUEWWTQST2KLCOWFJSAFSAQNLUCTTOWVZUBESEI23A, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.660349657 (499h15m59.531432712s) [noise=true , address=77.90.31.11:4571 , piece_id=OERZACIKIG7HBEV3LL42KUAGKN6AWZEXNQSA52FIYTUKDFWCBW7Q, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.660376087 (499h15m59.531459132s) [seg_pos={0 0}, upload=438194]: Segment reached optimal threshold of 14 pieces. Dec 26 10:44:59.660612988 (499h15m59.531696033s) [noise=true , address=64.49.13.169:4711 , piece_id=PVBVY4TVJLPKYBP6A7LBNZADLN25XBDWZFEK2QVN45YHJATIR7TA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:52964->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:52964->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52964->64.49.13.169:4711: use of closed network connection Dec 26 10:44:59.660620753 (499h15m59.531703788s) [noise=true , address=64.49.13.169:4711 , piece_id=PVBVY4TVJLPKYBP6A7LBNZADLN25XBDWZFEK2QVN45YHJATIR7TA, seg_pos={0 0}, upload=438194]: Long tail cancellation due to optimal threshold Dec 26 10:44:59.660893663 (499h15m59.531976698s) [noise=true , address=45.91.116.6:4988 , piece_id=VPEKJF74D7SQWKR47MAIY5UJQPBJIEU5AQDKLFGATYIRHFREAQ5A, seg_pos={0 0}, upload=438194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TrcoCHhAJeEykP9hJNpygoPSdgFV5v6GzL48JpyNPiAj5ZJRS): protocol: expected piece hash; context canceled; EOF Dec 26 10:44:59.660901057 (499h15m59.531984102s) [noise=true , address=45.91.116.6:4988 , piece_id=VPEKJF74D7SQWKR47MAIY5UJQPBJIEU5AQDKLFGATYIRHFREAQ5A, seg_pos={0 0}, upload=438194]: Long tail cancellation due to optimal threshold Dec 26 10:44:59.661226155 (499h15m59.5323092s) [noise=true , address=77.90.31.14:4678 , piece_id=XQHQZJUPB24G3BWGFTFDD42NM6K5HJVHUXBXWH5TW7BU25QGHTMA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12beu3SUZMPNWfbeHV4tLtx9UVZoEHh4Kt7oBhT59w3bGBE1ovP): protocol: write tcp 188.166.225.170:35884->77.90.31.14:4678: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 10:44:59.661249630 (499h15m59.532332685s) [noise=true , address=77.90.31.14:4678 , piece_id=XQHQZJUPB24G3BWGFTFDD42NM6K5HJVHUXBXWH5TW7BU25QGHTMA, seg_pos={0 0}, upload=438194]: Long tail cancellation due to optimal threshold Dec 26 10:44:59.661309705 (499h15m59.53239275s) [noise=true , address=77.90.31.21:4909 , piece_id=H6KT2AKZISCU4VEA6TXJJ3EFGCRBZPPXBEJJEIHZR3GEJZKXVRQQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121RT7tP6czMFAiZ3DjzSSsYgPt3xZXVbXVpkcYxaZYscNiMjL6): protocol: expected piece hash; context canceled; EOF Dec 26 10:44:59.661318962 (499h15m59.532401997s) [noise=true , address=77.90.31.21:4909 , piece_id=H6KT2AKZISCU4VEA6TXJJ3EFGCRBZPPXBEJJEIHZR3GEJZKXVRQQ, seg_pos={0 0}, upload=438194]: Long tail cancellation due to optimal threshold Dec 26 10:44:59.661345674 (499h15m59.532428719s) [seg_pos={0 0}, upload=438194]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 10:44:59.661485770 (499h15m59.532568815s) [upload=438194]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 10:44:59.661493655 (499h15m59.53257669s) [upload=438194]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 10:44:59.968467153 (499h15m59.839550198s) [upload=438194]: Done uploading segments... Dec 26 10:44:59.968484155 (499h15m59.83956719s) [upload=438194]: Upload finished. err: <nil> Duration: 38.155993027s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [..X.......XX.XX..X.......X..X.X......XX......XXX.XX......X.X] duration: 30.003358905s trace: https://jaeger.dev.storj.io/trace/4423565d91067fd6 FAIL (2025-12-26 11:24:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ... Dec 26 11:24:26.232078238 (499h55m26.103161283s) [upload=438769]: Got next segment. Inline: false Dec 26 11:24:26.232116671 (499h55m26.103199706s) [upload=438769]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 11:24:26.232122402 (499h55m26.103205447s) [upload=438769]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 11:24:26.477514509 (499h55m26.348597555s) [seg_pos={0 0}, upload=438769]: Begin upload segment... Dec 26 11:24:26.477569434 (499h55m26.348652469s) [seg_pos={0 0}, upload=438769]: Done begin upload segment. Dec 26 11:24:26.477607276 (499h55m26.348690311s) [noise=true , address=45.91.116.5:4300 , piece_id=KWWAMZ7EHQN23VNEY6XBZLDXADE52MK6ZOKPQTFUBPUZOD7RJV5Q, seg_pos={0 0}, upload=438769]: Uploading piece... Dec 26 11:24:26.477667741 (499h55m26.348750786s) [noise=true , address=145.79.227.39:4013 , piece_id=BGNS2EZ37Q4OHUVJOWZECB3TNF3LHZ47XIHCUZN3FKUMX626ZRRA, seg_pos={0 0}, upload=438769]: Uploading piece... Dec 26 11:24:26.478778406 (499h55m26.349861451s) [noise=true , address=145.79.227.20:4439 , piece_id=Z66QISQ7QKQALWSGNBQQCE223ZKZLZE75PJXA7SFPROPP7ZAIRRA, seg_pos={0 0}, upload=438769]: Uploading piece... Dec 26 11:24:26.479171675 (499h55m26.35025471s) [noise=true , address=77.90.31.13:4629 , piece_id=G2JDDTJJHRORQ223UBZLTNEZLVDMOJZWM6PMM6DBPA7UJPOEZOXQ, seg_pos={0 0}, upload=438769]: Uploading piece... Dec 26 11:24:26.479261726 (499h55m26.350344771s) [noise=true , address=45.91.116.11:4119 , piece_id=QH5OISUFZCYTMEULVCLHF45JEXOELQDV2HWVS6TMNX7G4YQD5LJA, seg_pos={0 0}, upload=438769]: Uploading piece... Dec 26 11:24:26.479942523 (499h55m26.351025568s) [noise=true , address=45.91.116.8:4047 , piece_id=L5G7B3WHLO4SJ6TTGPGO7LAIBGRJUZQWGDIDVSYCNYUGO2VZHT5Q, seg_pos={0 0}, upload=438769]: Uploading piece... Dec 26 11:24:26.481238090 (499h55m26.352321125s) [noise=true , address=45.91.116.13:4155 , piece_id=6YLWXRB46E7VZ6BVU4FURJWQL5RS65CP6QTL32GMD5XRNKEDTWOA, seg_pos={0 0}, upload=438769]: Uploading piece... Dec 26 11:24:26.482140899 (499h55m26.353223944s) [noise=true , address=77.90.31.16:4735 , piece_id=R3LEGBFUSVF6LJNOVF3VDO44HEAREXRANJ2P2PZZZNV5TXI7KUWA, seg_pos={0 0}, upload=438769]: Uploading piece... Dec 26 11:24:26.482700735 (499h55m26.35378377s) [noise=true , address=145.79.227.37:4566 , piece_id=NIS3EZDPQBQ4B6WJ5YAOQWIWOCZWIWJ6QJ2KC5IVODSDCJVQV5CQ, seg_pos={0 0}, upload=438769]: Uploading piece... Dec 26 11:24:26.483083013 (499h55m26.354166058s) [noise=true , address=45.91.116.10:4081 , piece_id=U5HL2D26TB72RFY5PT7J2YQEB2AZA75OFJQP6VLGOWBCRMXSY4VA, seg_pos={0 0}, upload=438769]: Uploading piece... Dec 26 11:24:26.483575250 (499h55m26.354658295s) [noise=true , address=45.91.116.12:4226 , piece_id=HIQPR64F3L3VORXZWG5YTGU4T5FFREMT3ZOXKGIYR46X454MMQJQ, seg_pos={0 0}, upload=438769]: Uploading piece... Dec 26 11:24:26.484251507 (499h55m26.355334542s) [noise=true , address=145.79.227.48:4301 , piece_id=2FM4EHNDLRURQNX77ZHURIBHWJ3UZRDKAMCHPK4UHYCRUOORCNBA, seg_pos={0 0}, upload=438769]: Uploading piece... Dec 26 11:24:26.487142833 (499h55m26.358225868s) [noise=true , address=145.79.227.29:4942 , piece_id=B2V6WTBCTWD7ZMHIN46NXDW7GYKTE7B6YDUOAQRM24EH7REE46BQ, seg_pos={0 0}, upload=438769]: Uploading piece... Dec 26 11:24:26.489065373 (499h55m26.360148409s) [noise=true , address=45.91.116.7:4012 , piece_id=SGV2HXRO6PQWSK5S3OT4C6UR3OCFNJ3GMEMEJNJUGZOLGTVBP5ZA, seg_pos={0 0}, upload=438769]: Uploading piece... Dec 26 11:24:26.490417198 (499h55m26.361500233s) [noise=true , address=145.79.227.52:4446 , piece_id=QT6AURNCLTTSRKEERI3FMKRDLF6K6MEVQMLZL5O34M2YO4MXCLRQ, seg_pos={0 0}, upload=438769]: Uploading piece... Dec 26 11:24:26.494672611 (499h55m26.365755646s) [noise=true , address=145.79.227.16:4910 , piece_id=QOORN3OTMYVCFECSIM6C46QYRH6DNWB6XES4DZ445E7UWH5SQM4Q, seg_pos={0 0}, upload=438769]: Uploading piece... Dec 26 11:24:28.199910059 (499h55m28.070993094s) [upload=438769]: Next returned nil segment Dec 26 11:24:28.199925618 (499h55m28.071008653s) [upload=438769]: Waiting for error group managing segments... Dec 26 11:24:28.483828588 (499h55m28.354911633s) [noise=true , address=145.79.227.37:4566 , piece_id=NIS3EZDPQBQ4B6WJ5YAOQWIWOCZWIWJ6QJ2KC5IVODSDCJVQV5CQ, seg_pos={0 0}, upload=438769]: Done uploading piece. err: <nil> Dec 26 11:24:28.486064325 (499h55m28.35714737s) [noise=true , address=145.79.227.20:4439 , piece_id=Z66QISQ7QKQALWSGNBQQCE223ZKZLZE75PJXA7SFPROPP7ZAIRRA, seg_pos={0 0}, upload=438769]: Done uploading piece. err: <nil> Dec 26 11:24:28.492430918 (499h55m28.363513963s) [noise=true , address=145.79.227.48:4301 , piece_id=2FM4EHNDLRURQNX77ZHURIBHWJ3UZRDKAMCHPK4UHYCRUOORCNBA, seg_pos={0 0}, upload=438769]: Done uploading piece. err: <nil> Dec 26 11:24:28.503261771 (499h55m28.374344806s) [noise=true , address=145.79.227.39:4013 , piece_id=BGNS2EZ37Q4OHUVJOWZECB3TNF3LHZ47XIHCUZN3FKUMX626ZRRA, seg_pos={0 0}, upload=438769]: Done uploading piece. err: <nil> Dec 26 11:24:28.505613890 (499h55m28.376696925s) [noise=true , address=145.79.227.52:4446 , piece_id=QT6AURNCLTTSRKEERI3FMKRDLF6K6MEVQMLZL5O34M2YO4MXCLRQ, seg_pos={0 0}, upload=438769]: Done uploading piece. err: <nil> Dec 26 11:24:28.517816536 (499h55m28.388899582s) [noise=true , address=145.79.227.29:4942 , piece_id=B2V6WTBCTWD7ZMHIN46NXDW7GYKTE7B6YDUOAQRM24EH7REE46BQ, seg_pos={0 0}, upload=438769]: Done uploading piece. err: <nil> Dec 26 11:24:28.527772374 (499h55m28.398855409s) [noise=true , address=145.79.227.16:4910 , piece_id=QOORN3OTMYVCFECSIM6C46QYRH6DNWB6XES4DZ445E7UWH5SQM4Q, seg_pos={0 0}, upload=438769]: Done uploading piece. err: <nil> Dec 26 11:24:28.845118774 (499h55m28.716201819s) [noise=true , address=45.91.116.10:4081 , piece_id=U5HL2D26TB72RFY5PT7J2YQEB2AZA75OFJQP6VLGOWBCRMXSY4VA, seg_pos={0 0}, upload=438769]: Done uploading piece. err: <nil> Dec 26 11:24:28.860180034 (499h55m28.731263069s) [noise=true , address=45.91.116.11:4119 , piece_id=QH5OISUFZCYTMEULVCLHF45JEXOELQDV2HWVS6TMNX7G4YQD5LJA, seg_pos={0 0}, upload=438769]: Done uploading piece. err: <nil> Dec 26 11:24:28.878376293 (499h55m28.749459328s) [noise=true , address=45.91.116.7:4012 , piece_id=SGV2HXRO6PQWSK5S3OT4C6UR3OCFNJ3GMEMEJNJUGZOLGTVBP5ZA, seg_pos={0 0}, upload=438769]: Done uploading piece. err: <nil> Dec 26 11:24:28.949507468 (499h55m28.820590514s) [noise=true , address=77.90.31.16:4735 , piece_id=R3LEGBFUSVF6LJNOVF3VDO44HEAREXRANJ2P2PZZZNV5TXI7KUWA, seg_pos={0 0}, upload=438769]: Done uploading piece. err: <nil> Dec 26 11:24:29.045945892 (499h55m28.917028938s) [noise=true , address=45.91.116.5:4300 , piece_id=KWWAMZ7EHQN23VNEY6XBZLDXADE52MK6ZOKPQTFUBPUZOD7RJV5Q, seg_pos={0 0}, upload=438769]: Done uploading piece. err: <nil> Dec 26 11:24:29.183983449 (499h55m29.055066495s) [noise=true , address=45.91.116.8:4047 , piece_id=L5G7B3WHLO4SJ6TTGPGO7LAIBGRJUZQWGDIDVSYCNYUGO2VZHT5Q, seg_pos={0 0}, upload=438769]: Done uploading piece. err: <nil> Dec 26 11:24:29.184010211 (499h55m29.055093246s) [seg_pos={0 0}, upload=438769]: Segment reached optimal threshold of 13 pieces. Dec 26 11:24:29.184242834 (499h55m29.055325879s) [noise=true , address=77.90.31.13:4629 , piece_id=G2JDDTJJHRORQ223UBZLTNEZLVDMOJZWM6PMM6DBPA7UJPOEZOXQ, seg_pos={0 0}, upload=438769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: expected piece hash; context canceled; EOF Dec 26 11:24:29.184252101 (499h55m29.055335146s) [noise=true , address=77.90.31.13:4629 , piece_id=G2JDDTJJHRORQ223UBZLTNEZLVDMOJZWM6PMM6DBPA7UJPOEZOXQ, seg_pos={0 0}, upload=438769]: Long tail cancellation due to optimal threshold Dec 26 11:24:29.184470006 (499h55m29.055553051s) [noise=true , address=45.91.116.12:4226 , piece_id=HIQPR64F3L3VORXZWG5YTGU4T5FFREMT3ZOXKGIYR46X454MMQJQ, seg_pos={0 0}, upload=438769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: write tcp 188.166.225.170:36500->45.91.116.12:4226: use of closed network connection; write tcp 188.166.225.170:36500->45.91.116.12:4226: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36500->45.91.116.12:4226: use of closed network connection Dec 26 11:24:29.184490976 (499h55m29.055574011s) [noise=true , address=45.91.116.12:4226 , piece_id=HIQPR64F3L3VORXZWG5YTGU4T5FFREMT3ZOXKGIYR46X454MMQJQ, seg_pos={0 0}, upload=438769]: Long tail cancellation due to optimal threshold Dec 26 11:24:29.184563194 (499h55m29.055646239s) [noise=true , address=45.91.116.13:4155 , piece_id=6YLWXRB46E7VZ6BVU4FURJWQL5RS65CP6QTL32GMD5XRNKEDTWOA, seg_pos={0 0}, upload=438769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: write tcp 188.166.225.170:34548->45.91.116.13:4155: use of closed network connection; write tcp 188.166.225.170:34548->45.91.116.13:4155: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34548->45.91.116.13:4155: use of closed network connection Dec 26 11:24:29.184576740 (499h55m29.055659775s) [noise=true , address=45.91.116.13:4155 , piece_id=6YLWXRB46E7VZ6BVU4FURJWQL5RS65CP6QTL32GMD5XRNKEDTWOA, seg_pos={0 0}, upload=438769]: Long tail cancellation due to optimal threshold Dec 26 11:24:29.184629009 (499h55m29.055712054s) [seg_pos={0 0}, upload=438769]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 11:24:29.184746142 (499h55m29.055829187s) [upload=438769]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 11:24:29.184757143 (499h55m29.055840189s) [upload=438769]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 11:24:29.478019373 (499h55m29.349102428s) [upload=438769]: Done uploading segments... Dec 26 11:24:29.478044351 (499h55m29.349127386s) [upload=438769]: Upload finished. err: <nil> Duration: 30.003380455s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [....X.........X.............................................] duration: 36.984092907s trace: https://jaeger.dev.storj.io/trace/785a90e8be84a595 FAIL (2025-12-26 11:24:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== KXVRQQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.834123964 (499h15m56.705207009s) [noise=true , address=45.91.116.4:4963 , piece_id=U4T64KQ67C3BDKG5TJSTQWUS4WWYCSVPUZOWJ6YANR2IQ2JD4EFQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.834585043 (499h15m56.705668078s) [noise=true , address=145.79.227.9:4390 , piece_id=JDQNHNGABBGANIHKSKP4VHUWIPQBNNEOPC4ZQCKR3K5TTR4N4SGQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.834961800 (499h15m56.706044845s) [noise=true , address=145.79.227.30:4778 , piece_id=BMEAUWANP7FP6CFRULMIAZUV4JNQEXOUN2EAXJY5XOWLDZINWOIA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.835331334 (499h15m56.706414379s) [noise=true , address=77.90.31.18:4838 , piece_id=VYBIKY3MLEE2CXGS2MT23HX5HUFK25FKYKI4XLXPDRNWCC55ANXQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.835738399 (499h15m56.706821434s) [noise=true , address=145.79.227.20:4459 , piece_id=2EUW2AH7FBVZNJ27XVLRXKMXA737DWKAYBL7CR7PRXXRTO2M644A, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.836118653 (499h15m56.707201698s) [noise=true , address=77.90.31.14:4678 , piece_id=XQHQZJUPB24G3BWGFTFDD42NM6K5HJVHUXBXWH5TW7BU25QGHTMA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.836501923 (499h15m56.707584968s) [noise=true , address=45.91.116.6:4988 , piece_id=VPEKJF74D7SQWKR47MAIY5UJQPBJIEU5AQDKLFGATYIRHFREAQ5A, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.836904940 (499h15m56.707987985s) [noise=true , address=64.49.13.169:4711 , piece_id=PVBVY4TVJLPKYBP6A7LBNZADLN25XBDWZFEK2QVN45YHJATIR7TA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.837388641 (499h15m56.708471676s) [noise=true , address=145.79.227.40:4076 , piece_id=Y6WKUYNPDMBUZSNWEYDL2KKNR6WKHSKGP63S6KAG6F2MWMNTVMBQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.837910455 (499h15m56.7089935s) [noise=true , address=77.90.31.11:4571 , piece_id=OERZACIKIG7HBEV3LL42KUAGKN6AWZEXNQSA52FIYTUKDFWCBW7Q, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.838146514 (499h15m56.709229559s) [noise=true , address=64.49.13.26:4945 , piece_id=KXCFTLWR4HSBTP4GJJ4HEPA6BBEMILYTJEMXY4S5WB24HRMQJOKA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.838469329 (499h15m56.709552364s) [noise=true , address=45.91.116.5:4301 , piece_id=E3ZRKZUKM335ARHV3Z5YGLKQYHG6QV4GCCNHAVMN34BM7UQFWIIA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.838690831 (499h15m56.709773866s) [noise=true , address=45.91.116.14:4204 , piece_id=JWL7OOMKTVAP7WEUB5WTL3F6F5IRNID4KIOYJSYZ655PJSNXAQKQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.839143724 (499h15m56.710226759s) [noise=true , address=45.91.116.9:4272 , piece_id=GGFDVKEW5LUCHQQUP6CWJHJQXUETYUG2UISPVP6XQJGSAJW2KP2Q, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.839678982 (499h15m56.710762027s) [noise=true , address=145.79.227.44:4208 , piece_id=ZLIX7XAUMKRUEWWTQST2KLCOWFJSAFSAQNLUCTTOWVZUBESEI23A, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:58.136536405 (499h15m58.00761944s) [upload=438194]: Next returned nil segment Dec 26 10:44:58.136548628 (499h15m58.007631673s) [upload=438194]: Waiting for error group managing segments... Dec 26 10:44:58.469618858 (499h15m58.340701893s) [noise=true , address=64.49.13.26:4945 , piece_id=KXCFTLWR4HSBTP4GJJ4HEPA6BBEMILYTJEMXY4S5WB24HRMQJOKA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.708819055 (499h15m58.5799021s) [noise=true , address=145.79.227.9:4390 , piece_id=JDQNHNGABBGANIHKSKP4VHUWIPQBNNEOPC4ZQCKR3K5TTR4N4SGQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.714466429 (499h15m58.585549474s) [noise=true , address=145.79.227.40:4076 , piece_id=Y6WKUYNPDMBUZSNWEYDL2KKNR6WKHSKGP63S6KAG6F2MWMNTVMBQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.745407696 (499h15m58.616490731s) [noise=true , address=145.79.227.20:4459 , piece_id=2EUW2AH7FBVZNJ27XVLRXKMXA737DWKAYBL7CR7PRXXRTO2M644A, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.888701070 (499h15m58.759784105s) [noise=true , address=77.90.31.18:4838 , piece_id=VYBIKY3MLEE2CXGS2MT23HX5HUFK25FKYKI4XLXPDRNWCC55ANXQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.948150873 (499h15m58.819233918s) [noise=true , address=45.91.116.5:4301 , piece_id=E3ZRKZUKM335ARHV3Z5YGLKQYHG6QV4GCCNHAVMN34BM7UQFWIIA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.971389744 (499h15m58.842472779s) [noise=true , address=77.90.31.19:4868 , piece_id=WH77PZSXC2VUWLQYLBR2NDEVT6KMD3VZK5WAL53AMRDW74P242LQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.021685314 (499h15m58.892768349s) [noise=true , address=45.91.116.4:4963 , piece_id=U4T64KQ67C3BDKG5TJSTQWUS4WWYCSVPUZOWJ6YANR2IQ2JD4EFQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.045418216 (499h15m58.916501251s) [noise=true , address=45.91.116.14:4204 , piece_id=JWL7OOMKTVAP7WEUB5WTL3F6F5IRNID4KIOYJSYZ655PJSNXAQKQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.056260371 (499h15m58.927343416s) [noise=true , address=45.91.116.9:4272 , piece_id=GGFDVKEW5LUCHQQUP6CWJHJQXUETYUG2UISPVP6XQJGSAJW2KP2Q, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.085713054 (499h15m58.956796089s) [noise=true , address=45.91.116.7:4017 , piece_id=L4GGY6AICMSFMUVYGI6ZMUORYRXNJFPLJY5XJRUAM7NVYQOHPHSQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.640980124 (499h15m59.512063169s) [noise=true , address=145.79.227.30:4778 , piece_id=BMEAUWANP7FP6CFRULMIAZUV4JNQEXOUN2EAXJY5XOWLDZINWOIA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.654515607 (499h15m59.525598652s) [noise=true , address=145.79.227.44:4208 , piece_id=ZLIX7XAUMKRUEWWTQST2KLCOWFJSAFSAQNLUCTTOWVZUBESEI23A, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.660349657 (499h15m59.531432712s) [noise=true , address=77.90.31.11:4571 , piece_id=OERZACIKIG7HBEV3LL42KUAGKN6AWZEXNQSA52FIYTUKDFWCBW7Q, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.660376087 (499h15m59.531459132s) [seg_pos={0 0}, upload=438194]: Segment reached optimal threshold of 14 pieces. Dec 26 10:44:59.660612988 (499h15m59.531696033s) [noise=true , address=64.49.13.169:4711 , piece_id=PVBVY4TVJLPKYBP6A7LBNZADLN25XBDWZFEK2QVN45YHJATIR7TA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:52964->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:52964->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52964->64.49.13.169:4711: use of closed network connection Dec 26 10:44:59.660620753 (499h15m59.531703788s) [noise=true , address=64.49.13.169:4711 , piece_id=PVBVY4TVJLPKYBP6A7LBNZADLN25XBDWZFEK2QVN45YHJATIR7TA, seg_pos={0 0}, upload=438194]: Long tail cancellation due to optimal threshold Dec 26 10:44:59.660893663 (499h15m59.531976698s) [noise=true , address=45.91.116.6:4988 , piece_id=VPEKJF74D7SQWKR47MAIY5UJQPBJIEU5AQDKLFGATYIRHFREAQ5A, seg_pos={0 0}, upload=438194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TrcoCHhAJeEykP9hJNpygoPSdgFV5v6GzL48JpyNPiAj5ZJRS): protocol: expected piece hash; context canceled; EOF Dec 26 10:44:59.660901057 (499h15m59.531984102s) [noise=true , address=45.91.116.6:4988 , piece_id=VPEKJF74D7SQWKR47MAIY5UJQPBJIEU5AQDKLFGATYIRHFREAQ5A, seg_pos={0 0}, upload=438194]: Long tail cancellation due to optimal threshold Dec 26 10:44:59.661226155 (499h15m59.5323092s) [noise=true , address=77.90.31.14:4678 , piece_id=XQHQZJUPB24G3BWGFTFDD42NM6K5HJVHUXBXWH5TW7BU25QGHTMA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12beu3SUZMPNWfbeHV4tLtx9UVZoEHh4Kt7oBhT59w3bGBE1ovP): protocol: write tcp 188.166.225.170:35884->77.90.31.14:4678: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 10:44:59.661249630 (499h15m59.532332685s) [noise=true , address=77.90.31.14:4678 , piece_id=XQHQZJUPB24G3BWGFTFDD42NM6K5HJVHUXBXWH5TW7BU25QGHTMA, seg_pos={0 0}, upload=438194]: Long tail cancellation due to optimal threshold Dec 26 10:44:59.661309705 (499h15m59.53239275s) [noise=true , address=77.90.31.21:4909 , piece_id=H6KT2AKZISCU4VEA6TXJJ3EFGCRBZPPXBEJJEIHZR3GEJZKXVRQQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121RT7tP6czMFAiZ3DjzSSsYgPt3xZXVbXVpkcYxaZYscNiMjL6): protocol: expected piece hash; context canceled; EOF Dec 26 10:44:59.661318962 (499h15m59.532401997s) [noise=true , address=77.90.31.21:4909 , piece_id=H6KT2AKZISCU4VEA6TXJJ3EFGCRBZPPXBEJJEIHZR3GEJZKXVRQQ, seg_pos={0 0}, upload=438194]: Long tail cancellation due to optimal threshold Dec 26 10:44:59.661345674 (499h15m59.532428719s) [seg_pos={0 0}, upload=438194]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 10:44:59.661485770 (499h15m59.532568815s) [upload=438194]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 10:44:59.661493655 (499h15m59.53257669s) [upload=438194]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 10:44:59.968467153 (499h15m59.839550198s) [upload=438194]: Done uploading segments... Dec 26 10:44:59.968484155 (499h15m59.83956719s) [upload=438194]: Upload finished. err: <nil> Duration: 36.984117314s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [X..X.......XX.XX..X.......X..X.X......XX......XXX.XX......X.] duration: 5.742285193s trace: https://jaeger.dev.storj.io/trace/4546d6426bae145a FAIL (2025-12-26 11:23:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks, Piece ID: 2HW265IPDYSAEPMQ7ZMJG62BDHGXK6R52L6VJD7WJHNDNFUGR66A) context deadline exceeded; piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: M33DIVITO4OF4GWFO2Z25ZGZKMHZ5FHEWBINRGPCTWELCCGM7IQA) context deadline exceeded ======LOGS====== Dec 26 11:23:05.282030906 (499h54m5.153113941s) [upload=438749]: Starting upload Dec 26 11:23:05.282127220 (499h54m5.153210265s) [upload=438749]: Done starting upload Dec 26 11:23:05.282169110 (499h54m5.153252155s) [upload=438749]: Uploading segments... Dec 26 11:23:05.282365875 (499h54m5.15344892s) [upload=438749]: Got next segment. Inline: false Dec 26 11:23:05.282426130 (499h54m5.153509185s) [upload=438749]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 11:23:05.282433674 (499h54m5.153516719s) [upload=438749]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 11:23:05.527463031 (499h54m5.398546066s) [seg_pos={0 0}, upload=438749]: Begin upload segment... Dec 26 11:23:05.527509429 (499h54m5.398592464s) [seg_pos={0 0}, upload=438749]: Done begin upload segment. Dec 26 11:23:05.527535729 (499h54m5.398618775s) [noise=true , address=145.79.227.39:4013 , piece_id=BXK4IFLFG3GU6FV5I4TTVCC5JEYD5GXF5MK4P32ZOANUZE2YIDFQ, seg_pos={0 0}, upload=438749]: Uploading piece... Dec 26 11:23:05.527614549 (499h54m5.398697594s) [noise=true , address=45.91.116.10:4081 , piece_id=M33DIVITO4OF4GWFO2Z25ZGZKMHZ5FHEWBINRGPCTWELCCGM7IQA, seg_pos={0 0}, upload=438749]: Uploading piece... Dec 26 11:23:05.528144107 (499h54m5.399227152s) [noise=true , address=145.79.227.2:4115 , piece_id=5UFQSVAZFC7OK6APENJJHHXT2U7LU2Q7THMGNBNAN75RLLS3NI7Q, seg_pos={0 0}, upload=438749]: Uploading piece... Dec 26 11:23:05.528564257 (499h54m5.399647302s) [noise=true , address=45.91.116.5:4298 , piece_id=B7E5QXF4UWB5CPBNAM2CEVRXG7RUZ3DJTHE4HGJ2MAF5H7JGZCIQ, seg_pos={0 0}, upload=438749]: Uploading piece... Dec 26 11:23:05.529022269 (499h54m5.400105304s) [noise=true , address=145.79.227.45:4231 , piece_id=QRTFSLSY2GMELAVMAYTIJKIT5HPGUY2Z5NJ7USSSM45HBNR2BBZQ, seg_pos={0 0}, upload=438749]: Uploading piece... Dec 26 11:23:05.529033501 (499h54m5.400116547s) [noise=true , address=145.79.227.29:4942 , piece_id=G45QYXMXTAVG2UFEEMNE2RQAUE7LIDTIB6447OTHHN3CENBMJPCA, seg_pos={0 0}, upload=438749]: Uploading piece... Dec 26 11:23:05.529348331 (499h54m5.400431366s) [noise=true , address=77.90.31.14:4661 , piece_id=JKDQIUMSGA6SU3FT2SPPUNMLDXYVQKI2MQOJLHTOL6EJUB5H7YDA, seg_pos={0 0}, upload=438749]: Uploading piece... Dec 26 11:23:05.529470002 (499h54m5.400553047s) [noise=true , address=145.79.227.42:4122 , piece_id=EIPNKPL7SPBWTMFNTYZFAJVSOHTCZMK3JQQT2ZGPGRWOU352PPSQ, seg_pos={0 0}, upload=438749]: Uploading piece... Dec 26 11:23:05.529729125 (499h54m5.40081218s) [noise=true , address=145.79.227.40:4051 , piece_id=PQU4ANKZWO3CH2YXZ4WB23U66TEPJ65532JYNL5VVBM4D5VIFS6Q, seg_pos={0 0}, upload=438749]: Uploading piece... Dec 26 11:23:05.530113928 (499h54m5.401196973s) [noise=true , address=45.91.116.9:4261 , piece_id=ACDNNFWEDP5H7RBQIX4ERYPK42AH73JNAEXSS53F7CY5ENU7H3CA, seg_pos={0 0}, upload=438749]: Uploading piece... Dec 26 11:23:05.530280305 (499h54m5.40136335s) [noise=true , address=145.79.227.46:4602 , piece_id=SH2BUPUONDV47FFDBH2YTNZVMLGFYIIJKVKHRYURAW6UD4IYXE7A, seg_pos={0 0}, upload=438749]: Uploading piece... Dec 26 11:23:05.530681950 (499h54m5.401764995s) [noise=true , address=77.90.31.11:4556 , piece_id=2HW265IPDYSAEPMQ7ZMJG62BDHGXK6R52L6VJD7WJHNDNFUGR66A, seg_pos={0 0}, upload=438749]: Uploading piece... Dec 26 11:23:05.530904012 (499h54m5.401987058s) [noise=true , address=145.79.227.50:4375 , piece_id=5QE27PXPITZNHWKQ22PW23B57G5RIRIJZHMDJB4UO42OHUWOMCRA, seg_pos={0 0}, upload=438749]: Uploading piece... Dec 26 11:23:05.531115195 (499h54m5.40219823s) [noise=true , address=145.79.227.35:4481 , piece_id=CLWXBSA3KK6FH3WMTZQFI3556ATOBGY7K5U7AOILXFBAD52FZZOQ, seg_pos={0 0}, upload=438749]: Uploading piece... Dec 26 11:23:05.531505848 (499h54m5.402588883s) [noise=true , address=145.79.227.7:4293 , piece_id=MBAQ535SN7PMD72O6ZX7FOVXGNVYE3ZVKVNAOXTXNO4JKFCCZZSQ, seg_pos={0 0}, upload=438749]: Uploading piece... Dec 26 11:23:05.532227633 (499h54m5.403310668s) [noise=true , address=145.79.227.34:4907 , piece_id=L7DT4YAL6ZESQISOQ7OLEVXKEEVBPNLNH2RQPM7NOVLWZ3KXQKGQ, seg_pos={0 0}, upload=438749]: Uploading piece... Dec 26 11:23:07.244514333 (499h54m7.115597378s) [upload=438749]: Next returned nil segment Dec 26 11:23:07.244532538 (499h54m7.115615583s) [upload=438749]: Waiting for error group managing segments... Dec 26 11:23:07.535091360 (499h54m7.406174405s) [noise=true , address=145.79.227.45:4231 , piece_id=QRTFSLSY2GMELAVMAYTIJKIT5HPGUY2Z5NJ7USSSM45HBNR2BBZQ, seg_pos={0 0}, upload=438749]: Done uploading piece. err: <nil> Dec 26 11:23:07.535367176 (499h54m7.406450211s) [noise=true , address=145.79.227.7:4293 , piece_id=MBAQ535SN7PMD72O6ZX7FOVXGNVYE3ZVKVNAOXTXNO4JKFCCZZSQ, seg_pos={0 0}, upload=438749]: Done uploading piece. err: <nil> Dec 26 11:23:07.537477364 (499h54m7.408560409s) [noise=true , address=145.79.227.42:4122 , piece_id=EIPNKPL7SPBWTMFNTYZFAJVSOHTCZMK3JQQT2ZGPGRWOU352PPSQ, seg_pos={0 0}, upload=438749]: Done uploading piece. err: <nil> Dec 26 11:23:07.538950970 (499h54m7.410034015s) [noise=true , address=145.79.227.50:4375 , piece_id=5QE27PXPITZNHWKQ22PW23B57G5RIRIJZHMDJB4UO42OHUWOMCRA, seg_pos={0 0}, upload=438749]: Done uploading piece. err: <nil> Dec 26 11:23:07.539194664 (499h54m7.410277709s) [noise=true , address=145.79.227.2:4115 , piece_id=5UFQSVAZFC7OK6APENJJHHXT2U7LU2Q7THMGNBNAN75RLLS3NI7Q, seg_pos={0 0}, upload=438749]: Done uploading piece. err: <nil> Dec 26 11:23:07.545069681 (499h54m7.416152726s) [noise=true , address=145.79.227.35:4481 , piece_id=CLWXBSA3KK6FH3WMTZQFI3556ATOBGY7K5U7AOILXFBAD52FZZOQ, seg_pos={0 0}, upload=438749]: Done uploading piece. err: <nil> Dec 26 11:23:07.545182826 (499h54m7.416265861s) [noise=true , address=145.79.227.29:4942 , piece_id=G45QYXMXTAVG2UFEEMNE2RQAUE7LIDTIB6447OTHHN3CENBMJPCA, seg_pos={0 0}, upload=438749]: Done uploading piece. err: <nil> Dec 26 11:23:07.550817406 (499h54m7.421900451s) [noise=true , address=145.79.227.39:4013 , piece_id=BXK4IFLFG3GU6FV5I4TTVCC5JEYD5GXF5MK4P32ZOANUZE2YIDFQ, seg_pos={0 0}, upload=438749]: Done uploading piece. err: <nil> Dec 26 11:23:07.557971217 (499h54m7.429054262s) [noise=true , address=145.79.227.46:4602 , piece_id=SH2BUPUONDV47FFDBH2YTNZVMLGFYIIJKVKHRYURAW6UD4IYXE7A, seg_pos={0 0}, upload=438749]: Done uploading piece. err: <nil> Dec 26 11:23:07.802469373 (499h54m7.673552408s) [noise=true , address=77.90.31.14:4661 , piece_id=JKDQIUMSGA6SU3FT2SPPUNMLDXYVQKI2MQOJLHTOL6EJUB5H7YDA, seg_pos={0 0}, upload=438749]: Done uploading piece. err: <nil> Dec 26 11:23:07.946368241 (499h54m7.817451286s) [noise=true , address=45.91.116.5:4298 , piece_id=B7E5QXF4UWB5CPBNAM2CEVRXG7RUZ3DJTHE4HGJ2MAF5H7JGZCIQ, seg_pos={0 0}, upload=438749]: Done uploading piece. err: <nil> Dec 26 11:23:08.207859955 (499h54m8.078943001s) [noise=true , address=77.90.31.11:4556 , piece_id=2HW265IPDYSAEPMQ7ZMJG62BDHGXK6R52L6VJD7WJHNDNFUGR66A, seg_pos={0 0}, upload=438749]: Done uploading piece. err: <nil> Dec 26 11:23:08.254357524 (499h54m8.125440559s) [noise=true , address=45.91.116.10:4081 , piece_id=M33DIVITO4OF4GWFO2Z25ZGZKMHZ5FHEWBINRGPCTWELCCGM7IQA, seg_pos={0 0}, upload=438749]: Done uploading piece. err: <nil> Dec 26 11:23:08.254377683 (499h54m8.125460728s) [seg_pos={0 0}, upload=438749]: Segment reached optimal threshold of 13 pieces. Dec 26 11:23:08.254570700 (499h54m8.125653745s) [noise=true , address=145.79.227.34:4907 , piece_id=L7DT4YAL6ZESQISOQ7OLEVXKEEVBPNLNH2RQPM7NOVLWZ3KXQKGQ, seg_pos={0 0}, upload=438749]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 26 11:23:08.254576751 (499h54m8.125659786s) [noise=true , address=145.79.227.34:4907 , piece_id=L7DT4YAL6ZESQISOQ7OLEVXKEEVBPNLNH2RQPM7NOVLWZ3KXQKGQ, seg_pos={0 0}, upload=438749]: Long tail cancellation due to optimal threshold Dec 26 11:23:08.254808894 (499h54m8.125891929s) [noise=true , address=45.91.116.9:4261 , piece_id=ACDNNFWEDP5H7RBQIX4ERYPK42AH73JNAEXSS53F7CY5ENU7H3CA, seg_pos={0 0}, upload=438749]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: expected piece hash; context canceled; EOF Dec 26 11:23:08.254814985 (499h54m8.12589803s) [noise=true , address=45.91.116.9:4261 , piece_id=ACDNNFWEDP5H7RBQIX4ERYPK42AH73JNAEXSS53F7CY5ENU7H3CA, seg_pos={0 0}, upload=438749]: Long tail cancellation due to optimal threshold Dec 26 11:23:08.254919073 (499h54m8.126002118s) [noise=true , address=145.79.227.40:4051 , piece_id=PQU4ANKZWO3CH2YXZ4WB23U66TEPJ65532JYNL5VVBM4D5VIFS6Q, seg_pos={0 0}, upload=438749]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 26 11:23:08.254932259 (499h54m8.126015314s) [noise=true , address=145.79.227.40:4051 , piece_id=PQU4ANKZWO3CH2YXZ4WB23U66TEPJ65532JYNL5VVBM4D5VIFS6Q, seg_pos={0 0}, upload=438749]: Long tail cancellation due to optimal threshold Dec 26 11:23:08.254968458 (499h54m8.126051493s) [seg_pos={0 0}, upload=438749]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 11:23:08.255051456 (499h54m8.126134501s) [upload=438749]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 11:23:08.255056545 (499h54m8.12613958s) [upload=438749]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 11:23:08.537031563 (499h54m8.408114608s) [upload=438749]: Done uploading segments... Dec 26 11:23:08.537055609 (499h54m8.408138654s) [upload=438749]: Upload finished. err: <nil> Duration: 5.74231026s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [.X..X.......XX.XX..X.......X..X.X......XX......XXX.XX......X] duration: 30.002666565s trace: https://jaeger.dev.storj.io/trace/4546d6426bae145a FAIL (2025-12-26 11:23:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks, Piece ID: 2HW265IPDYSAEPMQ7ZMJG62BDHGXK6R52L6VJD7WJHNDNFUGR66A) context deadline exceeded; piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: M33DIVITO4OF4GWFO2Z25ZGZKMHZ5FHEWBINRGPCTWELCCGM7IQA) context deadline exceeded ======LOGS====== Dec 26 11:23:05.282030906 (499h54m5.153113941s) [upload=438749]: Starting upload Dec 26 11:23:05.282127220 (499h54m5.153210265s) [upload=438749]: Done starting upload Dec 26 11:23:05.282169110 (499h54m5.153252155s) [upload=438749]: Uploading segments... Dec 26 11:23:05.282365875 (499h54m5.15344892s) [upload=438749]: Got next segment. Inline: false Dec 26 11:23:05.282426130 (499h54m5.153509185s) [upload=438749]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 11:23:05.282433674 (499h54m5.153516719s) [upload=438749]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 11:23:05.527463031 (499h54m5.398546066s) [seg_pos={0 0}, upload=438749]: Begin upload segment... Dec 26 11:23:05.527509429 (499h54m5.398592464s) [seg_pos={0 0}, upload=438749]: Done begin upload segment. Dec 26 11:23:05.527535729 (499h54m5.398618775s) [noise=true , address=145.79.227.39:4013 , piece_id=BXK4IFLFG3GU6FV5I4TTVCC5JEYD5GXF5MK4P32ZOANUZE2YIDFQ, seg_pos={0 0}, upload=438749]: Uploading piece... Dec 26 11:23:05.527614549 (499h54m5.398697594s) [noise=true , address=45.91.116.10:4081 , piece_id=M33DIVITO4OF4GWFO2Z25ZGZKMHZ5FHEWBINRGPCTWELCCGM7IQA, seg_pos={0 0}, upload=438749]: Uploading piece... Dec 26 11:23:05.528144107 (499h54m5.399227152s) [noise=true , address=145.79.227.2:4115 , piece_id=5UFQSVAZFC7OK6APENJJHHXT2U7LU2Q7THMGNBNAN75RLLS3NI7Q, seg_pos={0 0}, upload=438749]: Uploading piece... Dec 26 11:23:05.528564257 (499h54m5.399647302s) [noise=true , address=45.91.116.5:4298 , piece_id=B7E5QXF4UWB5CPBNAM2CEVRXG7RUZ3DJTHE4HGJ2MAF5H7JGZCIQ, seg_pos={0 0}, upload=438749]: Uploading piece... Dec 26 11:23:05.529022269 (499h54m5.400105304s) [noise=true , address=145.79.227.45:4231 , piece_id=QRTFSLSY2GMELAVMAYTIJKIT5HPGUY2Z5NJ7USSSM45HBNR2BBZQ, seg_pos={0 0}, upload=438749]: Uploading piece... Dec 26 11:23:05.529033501 (499h54m5.400116547s) [noise=true , address=145.79.227.29:4942 , piece_id=G45QYXMXTAVG2UFEEMNE2RQAUE7LIDTIB6447OTHHN3CENBMJPCA, seg_pos={0 0}, upload=438749]: Uploading piece... Dec 26 11:23:05.529348331 (499h54m5.400431366s) [noise=true , address=77.90.31.14:4661 , piece_id=JKDQIUMSGA6SU3FT2SPPUNMLDXYVQKI2MQOJLHTOL6EJUB5H7YDA, seg_pos={0 0}, upload=438749]: Uploading piece... Dec 26 11:23:05.529470002 (499h54m5.400553047s) [noise=true , address=145.79.227.42:4122 , piece_id=EIPNKPL7SPBWTMFNTYZFAJVSOHTCZMK3JQQT2ZGPGRWOU352PPSQ, seg_pos={0 0}, upload=438749]: Uploading piece... Dec 26 11:23:05.529729125 (499h54m5.40081218s) [noise=true , address=145.79.227.40:4051 , piece_id=PQU4ANKZWO3CH2YXZ4WB23U66TEPJ65532JYNL5VVBM4D5VIFS6Q, seg_pos={0 0}, upload=438749]: Uploading piece... Dec 26 11:23:05.530113928 (499h54m5.401196973s) [noise=true , address=45.91.116.9:4261 , piece_id=ACDNNFWEDP5H7RBQIX4ERYPK42AH73JNAEXSS53F7CY5ENU7H3CA, seg_pos={0 0}, upload=438749]: Uploading piece... Dec 26 11:23:05.530280305 (499h54m5.40136335s) [noise=true , address=145.79.227.46:4602 , piece_id=SH2BUPUONDV47FFDBH2YTNZVMLGFYIIJKVKHRYURAW6UD4IYXE7A, seg_pos={0 0}, upload=438749]: Uploading piece... Dec 26 11:23:05.530681950 (499h54m5.401764995s) [noise=true , address=77.90.31.11:4556 , piece_id=2HW265IPDYSAEPMQ7ZMJG62BDHGXK6R52L6VJD7WJHNDNFUGR66A, seg_pos={0 0}, upload=438749]: Uploading piece... Dec 26 11:23:05.530904012 (499h54m5.401987058s) [noise=true , address=145.79.227.50:4375 , piece_id=5QE27PXPITZNHWKQ22PW23B57G5RIRIJZHMDJB4UO42OHUWOMCRA, seg_pos={0 0}, upload=438749]: Uploading piece... Dec 26 11:23:05.531115195 (499h54m5.40219823s) [noise=true , address=145.79.227.35:4481 , piece_id=CLWXBSA3KK6FH3WMTZQFI3556ATOBGY7K5U7AOILXFBAD52FZZOQ, seg_pos={0 0}, upload=438749]: Uploading piece... Dec 26 11:23:05.531505848 (499h54m5.402588883s) [noise=true , address=145.79.227.7:4293 , piece_id=MBAQ535SN7PMD72O6ZX7FOVXGNVYE3ZVKVNAOXTXNO4JKFCCZZSQ, seg_pos={0 0}, upload=438749]: Uploading piece... Dec 26 11:23:05.532227633 (499h54m5.403310668s) [noise=true , address=145.79.227.34:4907 , piece_id=L7DT4YAL6ZESQISOQ7OLEVXKEEVBPNLNH2RQPM7NOVLWZ3KXQKGQ, seg_pos={0 0}, upload=438749]: Uploading piece... Dec 26 11:23:07.244514333 (499h54m7.115597378s) [upload=438749]: Next returned nil segment Dec 26 11:23:07.244532538 (499h54m7.115615583s) [upload=438749]: Waiting for error group managing segments... Dec 26 11:23:07.535091360 (499h54m7.406174405s) [noise=true , address=145.79.227.45:4231 , piece_id=QRTFSLSY2GMELAVMAYTIJKIT5HPGUY2Z5NJ7USSSM45HBNR2BBZQ, seg_pos={0 0}, upload=438749]: Done uploading piece. err: <nil> Dec 26 11:23:07.535367176 (499h54m7.406450211s) [noise=true , address=145.79.227.7:4293 , piece_id=MBAQ535SN7PMD72O6ZX7FOVXGNVYE3ZVKVNAOXTXNO4JKFCCZZSQ, seg_pos={0 0}, upload=438749]: Done uploading piece. err: <nil> Dec 26 11:23:07.537477364 (499h54m7.408560409s) [noise=true , address=145.79.227.42:4122 , piece_id=EIPNKPL7SPBWTMFNTYZFAJVSOHTCZMK3JQQT2ZGPGRWOU352PPSQ, seg_pos={0 0}, upload=438749]: Done uploading piece. err: <nil> Dec 26 11:23:07.538950970 (499h54m7.410034015s) [noise=true , address=145.79.227.50:4375 , piece_id=5QE27PXPITZNHWKQ22PW23B57G5RIRIJZHMDJB4UO42OHUWOMCRA, seg_pos={0 0}, upload=438749]: Done uploading piece. err: <nil> Dec 26 11:23:07.539194664 (499h54m7.410277709s) [noise=true , address=145.79.227.2:4115 , piece_id=5UFQSVAZFC7OK6APENJJHHXT2U7LU2Q7THMGNBNAN75RLLS3NI7Q, seg_pos={0 0}, upload=438749]: Done uploading piece. err: <nil> Dec 26 11:23:07.545069681 (499h54m7.416152726s) [noise=true , address=145.79.227.35:4481 , piece_id=CLWXBSA3KK6FH3WMTZQFI3556ATOBGY7K5U7AOILXFBAD52FZZOQ, seg_pos={0 0}, upload=438749]: Done uploading piece. err: <nil> Dec 26 11:23:07.545182826 (499h54m7.416265861s) [noise=true , address=145.79.227.29:4942 , piece_id=G45QYXMXTAVG2UFEEMNE2RQAUE7LIDTIB6447OTHHN3CENBMJPCA, seg_pos={0 0}, upload=438749]: Done uploading piece. err: <nil> Dec 26 11:23:07.550817406 (499h54m7.421900451s) [noise=true , address=145.79.227.39:4013 , piece_id=BXK4IFLFG3GU6FV5I4TTVCC5JEYD5GXF5MK4P32ZOANUZE2YIDFQ, seg_pos={0 0}, upload=438749]: Done uploading piece. err: <nil> Dec 26 11:23:07.557971217 (499h54m7.429054262s) [noise=true , address=145.79.227.46:4602 , piece_id=SH2BUPUONDV47FFDBH2YTNZVMLGFYIIJKVKHRYURAW6UD4IYXE7A, seg_pos={0 0}, upload=438749]: Done uploading piece. err: <nil> Dec 26 11:23:07.802469373 (499h54m7.673552408s) [noise=true , address=77.90.31.14:4661 , piece_id=JKDQIUMSGA6SU3FT2SPPUNMLDXYVQKI2MQOJLHTOL6EJUB5H7YDA, seg_pos={0 0}, upload=438749]: Done uploading piece. err: <nil> Dec 26 11:23:07.946368241 (499h54m7.817451286s) [noise=true , address=45.91.116.5:4298 , piece_id=B7E5QXF4UWB5CPBNAM2CEVRXG7RUZ3DJTHE4HGJ2MAF5H7JGZCIQ, seg_pos={0 0}, upload=438749]: Done uploading piece. err: <nil> Dec 26 11:23:08.207859955 (499h54m8.078943001s) [noise=true , address=77.90.31.11:4556 , piece_id=2HW265IPDYSAEPMQ7ZMJG62BDHGXK6R52L6VJD7WJHNDNFUGR66A, seg_pos={0 0}, upload=438749]: Done uploading piece. err: <nil> Dec 26 11:23:08.254357524 (499h54m8.125440559s) [noise=true , address=45.91.116.10:4081 , piece_id=M33DIVITO4OF4GWFO2Z25ZGZKMHZ5FHEWBINRGPCTWELCCGM7IQA, seg_pos={0 0}, upload=438749]: Done uploading piece. err: <nil> Dec 26 11:23:08.254377683 (499h54m8.125460728s) [seg_pos={0 0}, upload=438749]: Segment reached optimal threshold of 13 pieces. Dec 26 11:23:08.254570700 (499h54m8.125653745s) [noise=true , address=145.79.227.34:4907 , piece_id=L7DT4YAL6ZESQISOQ7OLEVXKEEVBPNLNH2RQPM7NOVLWZ3KXQKGQ, seg_pos={0 0}, upload=438749]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 26 11:23:08.254576751 (499h54m8.125659786s) [noise=true , address=145.79.227.34:4907 , piece_id=L7DT4YAL6ZESQISOQ7OLEVXKEEVBPNLNH2RQPM7NOVLWZ3KXQKGQ, seg_pos={0 0}, upload=438749]: Long tail cancellation due to optimal threshold Dec 26 11:23:08.254808894 (499h54m8.125891929s) [noise=true , address=45.91.116.9:4261 , piece_id=ACDNNFWEDP5H7RBQIX4ERYPK42AH73JNAEXSS53F7CY5ENU7H3CA, seg_pos={0 0}, upload=438749]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: expected piece hash; context canceled; EOF Dec 26 11:23:08.254814985 (499h54m8.12589803s) [noise=true , address=45.91.116.9:4261 , piece_id=ACDNNFWEDP5H7RBQIX4ERYPK42AH73JNAEXSS53F7CY5ENU7H3CA, seg_pos={0 0}, upload=438749]: Long tail cancellation due to optimal threshold Dec 26 11:23:08.254919073 (499h54m8.126002118s) [noise=true , address=145.79.227.40:4051 , piece_id=PQU4ANKZWO3CH2YXZ4WB23U66TEPJ65532JYNL5VVBM4D5VIFS6Q, seg_pos={0 0}, upload=438749]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 26 11:23:08.254932259 (499h54m8.126015314s) [noise=true , address=145.79.227.40:4051 , piece_id=PQU4ANKZWO3CH2YXZ4WB23U66TEPJ65532JYNL5VVBM4D5VIFS6Q, seg_pos={0 0}, upload=438749]: Long tail cancellation due to optimal threshold Dec 26 11:23:08.254968458 (499h54m8.126051493s) [seg_pos={0 0}, upload=438749]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 11:23:08.255051456 (499h54m8.126134501s) [upload=438749]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 11:23:08.255056545 (499h54m8.12613958s) [upload=438749]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 11:23:08.537031563 (499h54m8.408114608s) [upload=438749]: Done uploading segments... Dec 26 11:23:08.537055609 (499h54m8.408138654s) [upload=438749]: Upload finished. err: <nil> Duration: 30.002692294s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....X.........X............................................] duration: 28.77820204s trace: https://jaeger.dev.storj.io/trace/785a90e8be84a595 FAIL (2025-12-26 11:22:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== KXVRQQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.834123964 (499h15m56.705207009s) [noise=true , address=45.91.116.4:4963 , piece_id=U4T64KQ67C3BDKG5TJSTQWUS4WWYCSVPUZOWJ6YANR2IQ2JD4EFQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.834585043 (499h15m56.705668078s) [noise=true , address=145.79.227.9:4390 , piece_id=JDQNHNGABBGANIHKSKP4VHUWIPQBNNEOPC4ZQCKR3K5TTR4N4SGQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.834961800 (499h15m56.706044845s) [noise=true , address=145.79.227.30:4778 , piece_id=BMEAUWANP7FP6CFRULMIAZUV4JNQEXOUN2EAXJY5XOWLDZINWOIA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.835331334 (499h15m56.706414379s) [noise=true , address=77.90.31.18:4838 , piece_id=VYBIKY3MLEE2CXGS2MT23HX5HUFK25FKYKI4XLXPDRNWCC55ANXQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.835738399 (499h15m56.706821434s) [noise=true , address=145.79.227.20:4459 , piece_id=2EUW2AH7FBVZNJ27XVLRXKMXA737DWKAYBL7CR7PRXXRTO2M644A, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.836118653 (499h15m56.707201698s) [noise=true , address=77.90.31.14:4678 , piece_id=XQHQZJUPB24G3BWGFTFDD42NM6K5HJVHUXBXWH5TW7BU25QGHTMA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.836501923 (499h15m56.707584968s) [noise=true , address=45.91.116.6:4988 , piece_id=VPEKJF74D7SQWKR47MAIY5UJQPBJIEU5AQDKLFGATYIRHFREAQ5A, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.836904940 (499h15m56.707987985s) [noise=true , address=64.49.13.169:4711 , piece_id=PVBVY4TVJLPKYBP6A7LBNZADLN25XBDWZFEK2QVN45YHJATIR7TA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.837388641 (499h15m56.708471676s) [noise=true , address=145.79.227.40:4076 , piece_id=Y6WKUYNPDMBUZSNWEYDL2KKNR6WKHSKGP63S6KAG6F2MWMNTVMBQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.837910455 (499h15m56.7089935s) [noise=true , address=77.90.31.11:4571 , piece_id=OERZACIKIG7HBEV3LL42KUAGKN6AWZEXNQSA52FIYTUKDFWCBW7Q, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.838146514 (499h15m56.709229559s) [noise=true , address=64.49.13.26:4945 , piece_id=KXCFTLWR4HSBTP4GJJ4HEPA6BBEMILYTJEMXY4S5WB24HRMQJOKA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.838469329 (499h15m56.709552364s) [noise=true , address=45.91.116.5:4301 , piece_id=E3ZRKZUKM335ARHV3Z5YGLKQYHG6QV4GCCNHAVMN34BM7UQFWIIA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.838690831 (499h15m56.709773866s) [noise=true , address=45.91.116.14:4204 , piece_id=JWL7OOMKTVAP7WEUB5WTL3F6F5IRNID4KIOYJSYZ655PJSNXAQKQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.839143724 (499h15m56.710226759s) [noise=true , address=45.91.116.9:4272 , piece_id=GGFDVKEW5LUCHQQUP6CWJHJQXUETYUG2UISPVP6XQJGSAJW2KP2Q, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.839678982 (499h15m56.710762027s) [noise=true , address=145.79.227.44:4208 , piece_id=ZLIX7XAUMKRUEWWTQST2KLCOWFJSAFSAQNLUCTTOWVZUBESEI23A, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:58.136536405 (499h15m58.00761944s) [upload=438194]: Next returned nil segment Dec 26 10:44:58.136548628 (499h15m58.007631673s) [upload=438194]: Waiting for error group managing segments... Dec 26 10:44:58.469618858 (499h15m58.340701893s) [noise=true , address=64.49.13.26:4945 , piece_id=KXCFTLWR4HSBTP4GJJ4HEPA6BBEMILYTJEMXY4S5WB24HRMQJOKA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.708819055 (499h15m58.5799021s) [noise=true , address=145.79.227.9:4390 , piece_id=JDQNHNGABBGANIHKSKP4VHUWIPQBNNEOPC4ZQCKR3K5TTR4N4SGQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.714466429 (499h15m58.585549474s) [noise=true , address=145.79.227.40:4076 , piece_id=Y6WKUYNPDMBUZSNWEYDL2KKNR6WKHSKGP63S6KAG6F2MWMNTVMBQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.745407696 (499h15m58.616490731s) [noise=true , address=145.79.227.20:4459 , piece_id=2EUW2AH7FBVZNJ27XVLRXKMXA737DWKAYBL7CR7PRXXRTO2M644A, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.888701070 (499h15m58.759784105s) [noise=true , address=77.90.31.18:4838 , piece_id=VYBIKY3MLEE2CXGS2MT23HX5HUFK25FKYKI4XLXPDRNWCC55ANXQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.948150873 (499h15m58.819233918s) [noise=true , address=45.91.116.5:4301 , piece_id=E3ZRKZUKM335ARHV3Z5YGLKQYHG6QV4GCCNHAVMN34BM7UQFWIIA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.971389744 (499h15m58.842472779s) [noise=true , address=77.90.31.19:4868 , piece_id=WH77PZSXC2VUWLQYLBR2NDEVT6KMD3VZK5WAL53AMRDW74P242LQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.021685314 (499h15m58.892768349s) [noise=true , address=45.91.116.4:4963 , piece_id=U4T64KQ67C3BDKG5TJSTQWUS4WWYCSVPUZOWJ6YANR2IQ2JD4EFQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.045418216 (499h15m58.916501251s) [noise=true , address=45.91.116.14:4204 , piece_id=JWL7OOMKTVAP7WEUB5WTL3F6F5IRNID4KIOYJSYZ655PJSNXAQKQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.056260371 (499h15m58.927343416s) [noise=true , address=45.91.116.9:4272 , piece_id=GGFDVKEW5LUCHQQUP6CWJHJQXUETYUG2UISPVP6XQJGSAJW2KP2Q, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.085713054 (499h15m58.956796089s) [noise=true , address=45.91.116.7:4017 , piece_id=L4GGY6AICMSFMUVYGI6ZMUORYRXNJFPLJY5XJRUAM7NVYQOHPHSQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.640980124 (499h15m59.512063169s) [noise=true , address=145.79.227.30:4778 , piece_id=BMEAUWANP7FP6CFRULMIAZUV4JNQEXOUN2EAXJY5XOWLDZINWOIA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.654515607 (499h15m59.525598652s) [noise=true , address=145.79.227.44:4208 , piece_id=ZLIX7XAUMKRUEWWTQST2KLCOWFJSAFSAQNLUCTTOWVZUBESEI23A, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.660349657 (499h15m59.531432712s) [noise=true , address=77.90.31.11:4571 , piece_id=OERZACIKIG7HBEV3LL42KUAGKN6AWZEXNQSA52FIYTUKDFWCBW7Q, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.660376087 (499h15m59.531459132s) [seg_pos={0 0}, upload=438194]: Segment reached optimal threshold of 14 pieces. Dec 26 10:44:59.660612988 (499h15m59.531696033s) [noise=true , address=64.49.13.169:4711 , piece_id=PVBVY4TVJLPKYBP6A7LBNZADLN25XBDWZFEK2QVN45YHJATIR7TA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:52964->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:52964->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52964->64.49.13.169:4711: use of closed network connection Dec 26 10:44:59.660620753 (499h15m59.531703788s) [noise=true , address=64.49.13.169:4711 , piece_id=PVBVY4TVJLPKYBP6A7LBNZADLN25XBDWZFEK2QVN45YHJATIR7TA, seg_pos={0 0}, upload=438194]: Long tail cancellation due to optimal threshold Dec 26 10:44:59.660893663 (499h15m59.531976698s) [noise=true , address=45.91.116.6:4988 , piece_id=VPEKJF74D7SQWKR47MAIY5UJQPBJIEU5AQDKLFGATYIRHFREAQ5A, seg_pos={0 0}, upload=438194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TrcoCHhAJeEykP9hJNpygoPSdgFV5v6GzL48JpyNPiAj5ZJRS): protocol: expected piece hash; context canceled; EOF Dec 26 10:44:59.660901057 (499h15m59.531984102s) [noise=true , address=45.91.116.6:4988 , piece_id=VPEKJF74D7SQWKR47MAIY5UJQPBJIEU5AQDKLFGATYIRHFREAQ5A, seg_pos={0 0}, upload=438194]: Long tail cancellation due to optimal threshold Dec 26 10:44:59.661226155 (499h15m59.5323092s) [noise=true , address=77.90.31.14:4678 , piece_id=XQHQZJUPB24G3BWGFTFDD42NM6K5HJVHUXBXWH5TW7BU25QGHTMA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12beu3SUZMPNWfbeHV4tLtx9UVZoEHh4Kt7oBhT59w3bGBE1ovP): protocol: write tcp 188.166.225.170:35884->77.90.31.14:4678: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 10:44:59.661249630 (499h15m59.532332685s) [noise=true , address=77.90.31.14:4678 , piece_id=XQHQZJUPB24G3BWGFTFDD42NM6K5HJVHUXBXWH5TW7BU25QGHTMA, seg_pos={0 0}, upload=438194]: Long tail cancellation due to optimal threshold Dec 26 10:44:59.661309705 (499h15m59.53239275s) [noise=true , address=77.90.31.21:4909 , piece_id=H6KT2AKZISCU4VEA6TXJJ3EFGCRBZPPXBEJJEIHZR3GEJZKXVRQQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121RT7tP6czMFAiZ3DjzSSsYgPt3xZXVbXVpkcYxaZYscNiMjL6): protocol: expected piece hash; context canceled; EOF Dec 26 10:44:59.661318962 (499h15m59.532401997s) [noise=true , address=77.90.31.21:4909 , piece_id=H6KT2AKZISCU4VEA6TXJJ3EFGCRBZPPXBEJJEIHZR3GEJZKXVRQQ, seg_pos={0 0}, upload=438194]: Long tail cancellation due to optimal threshold Dec 26 10:44:59.661345674 (499h15m59.532428719s) [seg_pos={0 0}, upload=438194]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 10:44:59.661485770 (499h15m59.532568815s) [upload=438194]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 10:44:59.661493655 (499h15m59.53257669s) [upload=438194]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 10:44:59.968467153 (499h15m59.839550198s) [upload=438194]: Done uploading segments... Dec 26 10:44:59.968484155 (499h15m59.83956719s) [upload=438194]: Upload finished. err: <nil> Duration: 28.778226936s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [..X..X.......XX.XX..X.......X..X.X......XX......XXX.XX......] duration: 6.062443547s trace: https://jaeger.dev.storj.io/trace/193e6c819c3089fd FAIL (2025-12-26 11:22:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 716 (499h48m12.134924751s) [upload=438666]: Done starting upload Dec 26 11:17:12.263890689 (499h48m12.134973744s) [upload=438666]: Uploading segments... Dec 26 11:17:12.264121739 (499h48m12.135204784s) [upload=438666]: Got next segment. Inline: false Dec 26 11:17:12.264154531 (499h48m12.135237577s) [upload=438666]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 11:17:12.264161665 (499h48m12.13524471s) [upload=438666]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 11:17:12.513895599 (499h48m12.384978644s) [seg_pos={0 0}, upload=438666]: Begin upload segment... Dec 26 11:17:12.513949633 (499h48m12.385032678s) [seg_pos={0 0}, upload=438666]: Done begin upload segment. Dec 26 11:17:12.513986573 (499h48m12.385069608s) [noise=true , address=145.79.227.38:4978 , piece_id=QJZMHIFPCA57RDXEY36X5FNX4GY2JPTIIJJTKJJEO37SE3Y7YHGA, seg_pos={0 0}, upload=438666]: Uploading piece... Dec 26 11:17:12.518924846 (499h48m12.390007881s) [noise=true , address=45.91.116.6:4975 , piece_id=CDAJNYJWSPWIAFSHHL3UKEPD2GT3JIW7FCHM6NMPNJQ57FKVKM4Q, seg_pos={0 0}, upload=438666]: Uploading piece... Dec 26 11:17:12.520769569 (499h48m12.391852634s) [noise=true , address=145.79.227.50:4375 , piece_id=IL3SV5AKAYTDLLX3SNJ2Y2YHUQUWI52CKRYLIYQMSQWZZEVCNCNQ, seg_pos={0 0}, upload=438666]: Uploading piece... Dec 26 11:17:12.522359848 (499h48m12.393442883s) [noise=true , address=77.90.31.16:4735 , piece_id=ZOR2FJ5C6C3EIGQQWH6M5UOVHAHSBISDN2YN2PD5C72D532ES57Q, seg_pos={0 0}, upload=438666]: Uploading piece... Dec 26 11:17:12.523493195 (499h48m12.39457624s) [noise=true , address=77.90.31.17:4773 , piece_id=PJ326GIID2MZ2JXZMUQGCZAMWM5WZTQHXFJ4CV3LEZ72K4B6ZBRA, seg_pos={0 0}, upload=438666]: Uploading piece... Dec 26 11:17:12.526770225 (499h48m12.39785326s) [noise=true , address=145.79.227.24:4583 , piece_id=Z56FPKVQK2CZVLGL6CTK3IYGS3GN6C3JAPHSYLR4ZJHYOKKTLXMQ, seg_pos={0 0}, upload=438666]: Uploading piece... Dec 26 11:17:12.530787165 (499h48m12.40187021s) [noise=true , address=145.79.227.13:4803 , piece_id=PPZ74A3ZVV7I6HHUASUPXAMYD7BJTPIEZNR7TNO2EGKZGXRU5OKA, seg_pos={0 0}, upload=438666]: Uploading piece... Dec 26 11:17:12.534103499 (499h48m12.405186544s) [noise=true , address=145.79.227.49:4338 , piece_id=GZIWJ42YTCA2AQ2O2ALV4RFE4RWAZLFIDNEDZC6AI67YQXIRMA7A, seg_pos={0 0}, upload=438666]: Uploading piece... Dec 26 11:17:12.536031159 (499h48m12.407114204s) [noise=true , address=145.79.227.19:4077 , piece_id=RI4DDZEMOINLI6BUM7RXBDLX5H7NCYJMBEOAHLR3NSLZHNJD7ZGQ, seg_pos={0 0}, upload=438666]: Uploading piece... Dec 26 11:17:12.540743423 (499h48m12.411826468s) [noise=true , address=77.90.31.10:4519 , piece_id=47QEELPAZKYSYILZF4MHYFDNQTUR2YQNSME7LBRUJXEBRI4TWQDQ, seg_pos={0 0}, upload=438666]: Uploading piece... Dec 26 11:17:12.542682845 (499h48m12.41376591s) [noise=true , address=45.91.116.10:4084 , piece_id=TZSYRWBTD63NLCH5L72WTT2EKBZV3D2GBKT6LSTZARXLOPQ2EAFQ, seg_pos={0 0}, upload=438666]: Uploading piece... Dec 26 11:17:12.543247732 (499h48m12.414330777s) [noise=true , address=45.91.116.12:4226 , piece_id=T4NHHKAYFIRVVYFUBSK34OZHY5BH4SEHHMZKLXR6NCA7DZMRDDDQ, seg_pos={0 0}, upload=438666]: Uploading piece... Dec 26 11:17:12.544640514 (499h48m12.415723559s) [noise=true , address=145.79.227.4:4187 , piece_id=EGTOIDYEADC2MB77FTKGIR374CLCTVTLLSYMPTMGV3GTM6L4ANKA, seg_pos={0 0}, upload=438666]: Uploading piece... Dec 26 11:17:12.546342665 (499h48m12.41742571s) [noise=true , address=145.79.227.29:4941 , piece_id=JCG4GRZBB4JLBGAVL4MR7P57OQINGGF42W4W7G3PBXJFSGYUFRNA, seg_pos={0 0}, upload=438666]: Uploading piece... Dec 26 11:17:12.553741163 (499h48m12.424824208s) [noise=true , address=145.79.227.7:4293 , piece_id=JBHIXSZ3JBPNAPXFDBX4ZLMIQNKVKB2ZZIYVWLBK3SWYZIZHSU2Q, seg_pos={0 0}, upload=438666]: Uploading piece... Dec 26 11:17:12.555548043 (499h48m12.426631088s) [noise=true , address=77.90.31.14:4661 , piece_id=GVZWQLR6ETAA5DVIY4FNBE37DC6B72JGOHQQRJXLSNRCHQNK3CVQ, seg_pos={0 0}, upload=438666]: Uploading piece... Dec 26 11:17:14.277062566 (499h48m14.148145611s) [upload=438666]: Next returned nil segment Dec 26 11:17:14.277080911 (499h48m14.148163956s) [upload=438666]: Waiting for error group managing segments... Dec 26 11:17:14.568798930 (499h48m14.439881965s) [noise=true , address=145.79.227.19:4077 , piece_id=RI4DDZEMOINLI6BUM7RXBDLX5H7NCYJMBEOAHLR3NSLZHNJD7ZGQ, seg_pos={0 0}, upload=438666]: Done uploading piece. err: <nil> Dec 26 11:17:14.573695865 (499h48m14.44477891s) [noise=true , address=145.79.227.50:4375 , piece_id=IL3SV5AKAYTDLLX3SNJ2Y2YHUQUWI52CKRYLIYQMSQWZZEVCNCNQ, seg_pos={0 0}, upload=438666]: Done uploading piece. err: <nil> Dec 26 11:17:14.578731304 (499h48m14.449814339s) [noise=true , address=145.79.227.13:4803 , piece_id=PPZ74A3ZVV7I6HHUASUPXAMYD7BJTPIEZNR7TNO2EGKZGXRU5OKA, seg_pos={0 0}, upload=438666]: Done uploading piece. err: <nil> Dec 26 11:17:14.578885477 (499h48m14.449968522s) [noise=true , address=145.79.227.4:4187 , piece_id=EGTOIDYEADC2MB77FTKGIR374CLCTVTLLSYMPTMGV3GTM6L4ANKA, seg_pos={0 0}, upload=438666]: Done uploading piece. err: <nil> Dec 26 11:17:14.579341024 (499h48m14.45042406s) [noise=true , address=145.79.227.38:4978 , piece_id=QJZMHIFPCA57RDXEY36X5FNX4GY2JPTIIJJTKJJEO37SE3Y7YHGA, seg_pos={0 0}, upload=438666]: Done uploading piece. err: <nil> Dec 26 11:17:14.580493308 (499h48m14.451576344s) [noise=true , address=145.79.227.29:4941 , piece_id=JCG4GRZBB4JLBGAVL4MR7P57OQINGGF42W4W7G3PBXJFSGYUFRNA, seg_pos={0 0}, upload=438666]: Done uploading piece. err: <nil> Dec 26 11:17:14.581557736 (499h48m14.452640781s) [noise=true , address=145.79.227.49:4338 , piece_id=GZIWJ42YTCA2AQ2O2ALV4RFE4RWAZLFIDNEDZC6AI67YQXIRMA7A, seg_pos={0 0}, upload=438666]: Done uploading piece. err: <nil> Dec 26 11:17:14.587511112 (499h48m14.458594157s) [noise=true , address=145.79.227.7:4293 , piece_id=JBHIXSZ3JBPNAPXFDBX4ZLMIQNKVKB2ZZIYVWLBK3SWYZIZHSU2Q, seg_pos={0 0}, upload=438666]: Done uploading piece. err: <nil> Dec 26 11:17:14.595857755 (499h48m14.46694079s) [noise=true , address=145.79.227.24:4583 , piece_id=Z56FPKVQK2CZVLGL6CTK3IYGS3GN6C3JAPHSYLR4ZJHYOKKTLXMQ, seg_pos={0 0}, upload=438666]: Done uploading piece. err: <nil> Dec 26 11:17:14.814376751 (499h48m14.685459796s) [noise=true , address=77.90.31.14:4661 , piece_id=GVZWQLR6ETAA5DVIY4FNBE37DC6B72JGOHQQRJXLSNRCHQNK3CVQ, seg_pos={0 0}, upload=438666]: Done uploading piece. err: <nil> Dec 26 11:17:14.820988090 (499h48m14.692071136s) [noise=true , address=77.90.31.17:4773 , piece_id=PJ326GIID2MZ2JXZMUQGCZAMWM5WZTQHXFJ4CV3LEZ72K4B6ZBRA, seg_pos={0 0}, upload=438666]: Done uploading piece. err: <nil> Dec 26 11:17:14.853580242 (499h48m14.724663287s) [noise=true , address=77.90.31.16:4735 , piece_id=ZOR2FJ5C6C3EIGQQWH6M5UOVHAHSBISDN2YN2PD5C72D532ES57Q, seg_pos={0 0}, upload=438666]: Done uploading piece. err: <nil> Dec 26 11:17:15.021385681 (499h48m14.892468716s) [noise=true , address=77.90.31.10:4519 , piece_id=47QEELPAZKYSYILZF4MHYFDNQTUR2YQNSME7LBRUJXEBRI4TWQDQ, seg_pos={0 0}, upload=438666]: Done uploading piece. err: <nil> Dec 26 11:17:15.021407592 (499h48m14.892490627s) [seg_pos={0 0}, upload=438666]: Segment reached optimal threshold of 13 pieces. Dec 26 11:17:15.021671616 (499h48m14.892754651s) [noise=true , address=45.91.116.6:4975 , piece_id=CDAJNYJWSPWIAFSHHL3UKEPD2GT3JIW7FCHM6NMPNJQ57FKVKM4Q, seg_pos={0 0}, upload=438666]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: read tcp 188.166.225.170:39522->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 11:17:15.021677657 (499h48m14.892760692s) [noise=true , address=45.91.116.6:4975 , piece_id=CDAJNYJWSPWIAFSHHL3UKEPD2GT3JIW7FCHM6NMPNJQ57FKVKM4Q, seg_pos={0 0}, upload=438666]: Long tail cancellation due to optimal threshold Dec 26 11:17:15.021742240 (499h48m14.892825275s) [noise=true , address=45.91.116.10:4084 , piece_id=TZSYRWBTD63NLCH5L72WTT2EKBZV3D2GBKT6LSTZARXLOPQ2EAFQ, seg_pos={0 0}, upload=438666]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF Dec 26 11:17:15.021780964 (499h48m14.892864009s) [noise=true , address=45.91.116.10:4084 , piece_id=TZSYRWBTD63NLCH5L72WTT2EKBZV3D2GBKT6LSTZARXLOPQ2EAFQ, seg_pos={0 0}, upload=438666]: Long tail cancellation due to optimal threshold Dec 26 11:17:15.021928154 (499h48m14.893011199s) [noise=true , address=45.91.116.12:4226 , piece_id=T4NHHKAYFIRVVYFUBSK34OZHY5BH4SEHHMZKLXR6NCA7DZMRDDDQ, seg_pos={0 0}, upload=438666]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: write tcp 188.166.225.170:53630->45.91.116.12:4226: use of closed network connection; write tcp 188.166.225.170:53630->45.91.116.12:4226: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53630->45.91.116.12:4226: use of closed network connection Dec 26 11:17:15.021934826 (499h48m14.893017871s) [noise=true , address=45.91.116.12:4226 , piece_id=T4NHHKAYFIRVVYFUBSK34OZHY5BH4SEHHMZKLXR6NCA7DZMRDDDQ, seg_pos={0 0}, upload=438666]: Long tail cancellation due to optimal threshold Dec 26 11:17:15.021956368 (499h48m14.893039403s) [seg_pos={0 0}, upload=438666]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 11:17:15.022023355 (499h48m14.89310639s) [upload=438666]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 11:17:15.022028755 (499h48m14.89311179s) [upload=438666]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 11:17:15.321887056 (499h48m15.192970091s) [upload=438666]: Done uploading segments... Dec 26 11:17:15.321911402 (499h48m15.192994437s) [upload=438666]: Upload finished. err: <nil> Duration: 6.062489707s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [......X.........X...........................................] duration: 49.536357639s trace: https://jaeger.dev.storj.io/trace/785a90e8be84a595 FAIL (2025-12-26 11:22:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== KXVRQQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.834123964 (499h15m56.705207009s) [noise=true , address=45.91.116.4:4963 , piece_id=U4T64KQ67C3BDKG5TJSTQWUS4WWYCSVPUZOWJ6YANR2IQ2JD4EFQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.834585043 (499h15m56.705668078s) [noise=true , address=145.79.227.9:4390 , piece_id=JDQNHNGABBGANIHKSKP4VHUWIPQBNNEOPC4ZQCKR3K5TTR4N4SGQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.834961800 (499h15m56.706044845s) [noise=true , address=145.79.227.30:4778 , piece_id=BMEAUWANP7FP6CFRULMIAZUV4JNQEXOUN2EAXJY5XOWLDZINWOIA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.835331334 (499h15m56.706414379s) [noise=true , address=77.90.31.18:4838 , piece_id=VYBIKY3MLEE2CXGS2MT23HX5HUFK25FKYKI4XLXPDRNWCC55ANXQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.835738399 (499h15m56.706821434s) [noise=true , address=145.79.227.20:4459 , piece_id=2EUW2AH7FBVZNJ27XVLRXKMXA737DWKAYBL7CR7PRXXRTO2M644A, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.836118653 (499h15m56.707201698s) [noise=true , address=77.90.31.14:4678 , piece_id=XQHQZJUPB24G3BWGFTFDD42NM6K5HJVHUXBXWH5TW7BU25QGHTMA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.836501923 (499h15m56.707584968s) [noise=true , address=45.91.116.6:4988 , piece_id=VPEKJF74D7SQWKR47MAIY5UJQPBJIEU5AQDKLFGATYIRHFREAQ5A, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.836904940 (499h15m56.707987985s) [noise=true , address=64.49.13.169:4711 , piece_id=PVBVY4TVJLPKYBP6A7LBNZADLN25XBDWZFEK2QVN45YHJATIR7TA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.837388641 (499h15m56.708471676s) [noise=true , address=145.79.227.40:4076 , piece_id=Y6WKUYNPDMBUZSNWEYDL2KKNR6WKHSKGP63S6KAG6F2MWMNTVMBQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.837910455 (499h15m56.7089935s) [noise=true , address=77.90.31.11:4571 , piece_id=OERZACIKIG7HBEV3LL42KUAGKN6AWZEXNQSA52FIYTUKDFWCBW7Q, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.838146514 (499h15m56.709229559s) [noise=true , address=64.49.13.26:4945 , piece_id=KXCFTLWR4HSBTP4GJJ4HEPA6BBEMILYTJEMXY4S5WB24HRMQJOKA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.838469329 (499h15m56.709552364s) [noise=true , address=45.91.116.5:4301 , piece_id=E3ZRKZUKM335ARHV3Z5YGLKQYHG6QV4GCCNHAVMN34BM7UQFWIIA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.838690831 (499h15m56.709773866s) [noise=true , address=45.91.116.14:4204 , piece_id=JWL7OOMKTVAP7WEUB5WTL3F6F5IRNID4KIOYJSYZ655PJSNXAQKQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.839143724 (499h15m56.710226759s) [noise=true , address=45.91.116.9:4272 , piece_id=GGFDVKEW5LUCHQQUP6CWJHJQXUETYUG2UISPVP6XQJGSAJW2KP2Q, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.839678982 (499h15m56.710762027s) [noise=true , address=145.79.227.44:4208 , piece_id=ZLIX7XAUMKRUEWWTQST2KLCOWFJSAFSAQNLUCTTOWVZUBESEI23A, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:58.136536405 (499h15m58.00761944s) [upload=438194]: Next returned nil segment Dec 26 10:44:58.136548628 (499h15m58.007631673s) [upload=438194]: Waiting for error group managing segments... Dec 26 10:44:58.469618858 (499h15m58.340701893s) [noise=true , address=64.49.13.26:4945 , piece_id=KXCFTLWR4HSBTP4GJJ4HEPA6BBEMILYTJEMXY4S5WB24HRMQJOKA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.708819055 (499h15m58.5799021s) [noise=true , address=145.79.227.9:4390 , piece_id=JDQNHNGABBGANIHKSKP4VHUWIPQBNNEOPC4ZQCKR3K5TTR4N4SGQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.714466429 (499h15m58.585549474s) [noise=true , address=145.79.227.40:4076 , piece_id=Y6WKUYNPDMBUZSNWEYDL2KKNR6WKHSKGP63S6KAG6F2MWMNTVMBQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.745407696 (499h15m58.616490731s) [noise=true , address=145.79.227.20:4459 , piece_id=2EUW2AH7FBVZNJ27XVLRXKMXA737DWKAYBL7CR7PRXXRTO2M644A, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.888701070 (499h15m58.759784105s) [noise=true , address=77.90.31.18:4838 , piece_id=VYBIKY3MLEE2CXGS2MT23HX5HUFK25FKYKI4XLXPDRNWCC55ANXQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.948150873 (499h15m58.819233918s) [noise=true , address=45.91.116.5:4301 , piece_id=E3ZRKZUKM335ARHV3Z5YGLKQYHG6QV4GCCNHAVMN34BM7UQFWIIA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.971389744 (499h15m58.842472779s) [noise=true , address=77.90.31.19:4868 , piece_id=WH77PZSXC2VUWLQYLBR2NDEVT6KMD3VZK5WAL53AMRDW74P242LQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.021685314 (499h15m58.892768349s) [noise=true , address=45.91.116.4:4963 , piece_id=U4T64KQ67C3BDKG5TJSTQWUS4WWYCSVPUZOWJ6YANR2IQ2JD4EFQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.045418216 (499h15m58.916501251s) [noise=true , address=45.91.116.14:4204 , piece_id=JWL7OOMKTVAP7WEUB5WTL3F6F5IRNID4KIOYJSYZ655PJSNXAQKQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.056260371 (499h15m58.927343416s) [noise=true , address=45.91.116.9:4272 , piece_id=GGFDVKEW5LUCHQQUP6CWJHJQXUETYUG2UISPVP6XQJGSAJW2KP2Q, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.085713054 (499h15m58.956796089s) [noise=true , address=45.91.116.7:4017 , piece_id=L4GGY6AICMSFMUVYGI6ZMUORYRXNJFPLJY5XJRUAM7NVYQOHPHSQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.640980124 (499h15m59.512063169s) [noise=true , address=145.79.227.30:4778 , piece_id=BMEAUWANP7FP6CFRULMIAZUV4JNQEXOUN2EAXJY5XOWLDZINWOIA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.654515607 (499h15m59.525598652s) [noise=true , address=145.79.227.44:4208 , piece_id=ZLIX7XAUMKRUEWWTQST2KLCOWFJSAFSAQNLUCTTOWVZUBESEI23A, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.660349657 (499h15m59.531432712s) [noise=true , address=77.90.31.11:4571 , piece_id=OERZACIKIG7HBEV3LL42KUAGKN6AWZEXNQSA52FIYTUKDFWCBW7Q, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.660376087 (499h15m59.531459132s) [seg_pos={0 0}, upload=438194]: Segment reached optimal threshold of 14 pieces. Dec 26 10:44:59.660612988 (499h15m59.531696033s) [noise=true , address=64.49.13.169:4711 , piece_id=PVBVY4TVJLPKYBP6A7LBNZADLN25XBDWZFEK2QVN45YHJATIR7TA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:52964->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:52964->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52964->64.49.13.169:4711: use of closed network connection Dec 26 10:44:59.660620753 (499h15m59.531703788s) [noise=true , address=64.49.13.169:4711 , piece_id=PVBVY4TVJLPKYBP6A7LBNZADLN25XBDWZFEK2QVN45YHJATIR7TA, seg_pos={0 0}, upload=438194]: Long tail cancellation due to optimal threshold Dec 26 10:44:59.660893663 (499h15m59.531976698s) [noise=true , address=45.91.116.6:4988 , piece_id=VPEKJF74D7SQWKR47MAIY5UJQPBJIEU5AQDKLFGATYIRHFREAQ5A, seg_pos={0 0}, upload=438194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TrcoCHhAJeEykP9hJNpygoPSdgFV5v6GzL48JpyNPiAj5ZJRS): protocol: expected piece hash; context canceled; EOF Dec 26 10:44:59.660901057 (499h15m59.531984102s) [noise=true , address=45.91.116.6:4988 , piece_id=VPEKJF74D7SQWKR47MAIY5UJQPBJIEU5AQDKLFGATYIRHFREAQ5A, seg_pos={0 0}, upload=438194]: Long tail cancellation due to optimal threshold Dec 26 10:44:59.661226155 (499h15m59.5323092s) [noise=true , address=77.90.31.14:4678 , piece_id=XQHQZJUPB24G3BWGFTFDD42NM6K5HJVHUXBXWH5TW7BU25QGHTMA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12beu3SUZMPNWfbeHV4tLtx9UVZoEHh4Kt7oBhT59w3bGBE1ovP): protocol: write tcp 188.166.225.170:35884->77.90.31.14:4678: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 10:44:59.661249630 (499h15m59.532332685s) [noise=true , address=77.90.31.14:4678 , piece_id=XQHQZJUPB24G3BWGFTFDD42NM6K5HJVHUXBXWH5TW7BU25QGHTMA, seg_pos={0 0}, upload=438194]: Long tail cancellation due to optimal threshold Dec 26 10:44:59.661309705 (499h15m59.53239275s) [noise=true , address=77.90.31.21:4909 , piece_id=H6KT2AKZISCU4VEA6TXJJ3EFGCRBZPPXBEJJEIHZR3GEJZKXVRQQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121RT7tP6czMFAiZ3DjzSSsYgPt3xZXVbXVpkcYxaZYscNiMjL6): protocol: expected piece hash; context canceled; EOF Dec 26 10:44:59.661318962 (499h15m59.532401997s) [noise=true , address=77.90.31.21:4909 , piece_id=H6KT2AKZISCU4VEA6TXJJ3EFGCRBZPPXBEJJEIHZR3GEJZKXVRQQ, seg_pos={0 0}, upload=438194]: Long tail cancellation due to optimal threshold Dec 26 10:44:59.661345674 (499h15m59.532428719s) [seg_pos={0 0}, upload=438194]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 10:44:59.661485770 (499h15m59.532568815s) [upload=438194]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 10:44:59.661493655 (499h15m59.53257669s) [upload=438194]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 10:44:59.968467153 (499h15m59.839550198s) [upload=438194]: Done uploading segments... Dec 26 10:44:59.968484155 (499h15m59.83956719s) [upload=438194]: Upload finished. err: <nil> Duration: 49.536372638s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [X..X..X.......XX.XX..X.......X..X.X......XX......XXX.XX.....] duration: 6.041500316s trace: https://jaeger.dev.storj.io/trace/193e6c819c3089fd FAIL (2025-12-26 11:21:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 716 (499h48m12.134924751s) [upload=438666]: Done starting upload Dec 26 11:17:12.263890689 (499h48m12.134973744s) [upload=438666]: Uploading segments... Dec 26 11:17:12.264121739 (499h48m12.135204784s) [upload=438666]: Got next segment. Inline: false Dec 26 11:17:12.264154531 (499h48m12.135237577s) [upload=438666]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 11:17:12.264161665 (499h48m12.13524471s) [upload=438666]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 11:17:12.513895599 (499h48m12.384978644s) [seg_pos={0 0}, upload=438666]: Begin upload segment... Dec 26 11:17:12.513949633 (499h48m12.385032678s) [seg_pos={0 0}, upload=438666]: Done begin upload segment. Dec 26 11:17:12.513986573 (499h48m12.385069608s) [noise=true , address=145.79.227.38:4978 , piece_id=QJZMHIFPCA57RDXEY36X5FNX4GY2JPTIIJJTKJJEO37SE3Y7YHGA, seg_pos={0 0}, upload=438666]: Uploading piece... Dec 26 11:17:12.518924846 (499h48m12.390007881s) [noise=true , address=45.91.116.6:4975 , piece_id=CDAJNYJWSPWIAFSHHL3UKEPD2GT3JIW7FCHM6NMPNJQ57FKVKM4Q, seg_pos={0 0}, upload=438666]: Uploading piece... Dec 26 11:17:12.520769569 (499h48m12.391852634s) [noise=true , address=145.79.227.50:4375 , piece_id=IL3SV5AKAYTDLLX3SNJ2Y2YHUQUWI52CKRYLIYQMSQWZZEVCNCNQ, seg_pos={0 0}, upload=438666]: Uploading piece... Dec 26 11:17:12.522359848 (499h48m12.393442883s) [noise=true , address=77.90.31.16:4735 , piece_id=ZOR2FJ5C6C3EIGQQWH6M5UOVHAHSBISDN2YN2PD5C72D532ES57Q, seg_pos={0 0}, upload=438666]: Uploading piece... Dec 26 11:17:12.523493195 (499h48m12.39457624s) [noise=true , address=77.90.31.17:4773 , piece_id=PJ326GIID2MZ2JXZMUQGCZAMWM5WZTQHXFJ4CV3LEZ72K4B6ZBRA, seg_pos={0 0}, upload=438666]: Uploading piece... Dec 26 11:17:12.526770225 (499h48m12.39785326s) [noise=true , address=145.79.227.24:4583 , piece_id=Z56FPKVQK2CZVLGL6CTK3IYGS3GN6C3JAPHSYLR4ZJHYOKKTLXMQ, seg_pos={0 0}, upload=438666]: Uploading piece... Dec 26 11:17:12.530787165 (499h48m12.40187021s) [noise=true , address=145.79.227.13:4803 , piece_id=PPZ74A3ZVV7I6HHUASUPXAMYD7BJTPIEZNR7TNO2EGKZGXRU5OKA, seg_pos={0 0}, upload=438666]: Uploading piece... Dec 26 11:17:12.534103499 (499h48m12.405186544s) [noise=true , address=145.79.227.49:4338 , piece_id=GZIWJ42YTCA2AQ2O2ALV4RFE4RWAZLFIDNEDZC6AI67YQXIRMA7A, seg_pos={0 0}, upload=438666]: Uploading piece... Dec 26 11:17:12.536031159 (499h48m12.407114204s) [noise=true , address=145.79.227.19:4077 , piece_id=RI4DDZEMOINLI6BUM7RXBDLX5H7NCYJMBEOAHLR3NSLZHNJD7ZGQ, seg_pos={0 0}, upload=438666]: Uploading piece... Dec 26 11:17:12.540743423 (499h48m12.411826468s) [noise=true , address=77.90.31.10:4519 , piece_id=47QEELPAZKYSYILZF4MHYFDNQTUR2YQNSME7LBRUJXEBRI4TWQDQ, seg_pos={0 0}, upload=438666]: Uploading piece... Dec 26 11:17:12.542682845 (499h48m12.41376591s) [noise=true , address=45.91.116.10:4084 , piece_id=TZSYRWBTD63NLCH5L72WTT2EKBZV3D2GBKT6LSTZARXLOPQ2EAFQ, seg_pos={0 0}, upload=438666]: Uploading piece... Dec 26 11:17:12.543247732 (499h48m12.414330777s) [noise=true , address=45.91.116.12:4226 , piece_id=T4NHHKAYFIRVVYFUBSK34OZHY5BH4SEHHMZKLXR6NCA7DZMRDDDQ, seg_pos={0 0}, upload=438666]: Uploading piece... Dec 26 11:17:12.544640514 (499h48m12.415723559s) [noise=true , address=145.79.227.4:4187 , piece_id=EGTOIDYEADC2MB77FTKGIR374CLCTVTLLSYMPTMGV3GTM6L4ANKA, seg_pos={0 0}, upload=438666]: Uploading piece... Dec 26 11:17:12.546342665 (499h48m12.41742571s) [noise=true , address=145.79.227.29:4941 , piece_id=JCG4GRZBB4JLBGAVL4MR7P57OQINGGF42W4W7G3PBXJFSGYUFRNA, seg_pos={0 0}, upload=438666]: Uploading piece... Dec 26 11:17:12.553741163 (499h48m12.424824208s) [noise=true , address=145.79.227.7:4293 , piece_id=JBHIXSZ3JBPNAPXFDBX4ZLMIQNKVKB2ZZIYVWLBK3SWYZIZHSU2Q, seg_pos={0 0}, upload=438666]: Uploading piece... Dec 26 11:17:12.555548043 (499h48m12.426631088s) [noise=true , address=77.90.31.14:4661 , piece_id=GVZWQLR6ETAA5DVIY4FNBE37DC6B72JGOHQQRJXLSNRCHQNK3CVQ, seg_pos={0 0}, upload=438666]: Uploading piece... Dec 26 11:17:14.277062566 (499h48m14.148145611s) [upload=438666]: Next returned nil segment Dec 26 11:17:14.277080911 (499h48m14.148163956s) [upload=438666]: Waiting for error group managing segments... Dec 26 11:17:14.568798930 (499h48m14.439881965s) [noise=true , address=145.79.227.19:4077 , piece_id=RI4DDZEMOINLI6BUM7RXBDLX5H7NCYJMBEOAHLR3NSLZHNJD7ZGQ, seg_pos={0 0}, upload=438666]: Done uploading piece. err: <nil> Dec 26 11:17:14.573695865 (499h48m14.44477891s) [noise=true , address=145.79.227.50:4375 , piece_id=IL3SV5AKAYTDLLX3SNJ2Y2YHUQUWI52CKRYLIYQMSQWZZEVCNCNQ, seg_pos={0 0}, upload=438666]: Done uploading piece. err: <nil> Dec 26 11:17:14.578731304 (499h48m14.449814339s) [noise=true , address=145.79.227.13:4803 , piece_id=PPZ74A3ZVV7I6HHUASUPXAMYD7BJTPIEZNR7TNO2EGKZGXRU5OKA, seg_pos={0 0}, upload=438666]: Done uploading piece. err: <nil> Dec 26 11:17:14.578885477 (499h48m14.449968522s) [noise=true , address=145.79.227.4:4187 , piece_id=EGTOIDYEADC2MB77FTKGIR374CLCTVTLLSYMPTMGV3GTM6L4ANKA, seg_pos={0 0}, upload=438666]: Done uploading piece. err: <nil> Dec 26 11:17:14.579341024 (499h48m14.45042406s) [noise=true , address=145.79.227.38:4978 , piece_id=QJZMHIFPCA57RDXEY36X5FNX4GY2JPTIIJJTKJJEO37SE3Y7YHGA, seg_pos={0 0}, upload=438666]: Done uploading piece. err: <nil> Dec 26 11:17:14.580493308 (499h48m14.451576344s) [noise=true , address=145.79.227.29:4941 , piece_id=JCG4GRZBB4JLBGAVL4MR7P57OQINGGF42W4W7G3PBXJFSGYUFRNA, seg_pos={0 0}, upload=438666]: Done uploading piece. err: <nil> Dec 26 11:17:14.581557736 (499h48m14.452640781s) [noise=true , address=145.79.227.49:4338 , piece_id=GZIWJ42YTCA2AQ2O2ALV4RFE4RWAZLFIDNEDZC6AI67YQXIRMA7A, seg_pos={0 0}, upload=438666]: Done uploading piece. err: <nil> Dec 26 11:17:14.587511112 (499h48m14.458594157s) [noise=true , address=145.79.227.7:4293 , piece_id=JBHIXSZ3JBPNAPXFDBX4ZLMIQNKVKB2ZZIYVWLBK3SWYZIZHSU2Q, seg_pos={0 0}, upload=438666]: Done uploading piece. err: <nil> Dec 26 11:17:14.595857755 (499h48m14.46694079s) [noise=true , address=145.79.227.24:4583 , piece_id=Z56FPKVQK2CZVLGL6CTK3IYGS3GN6C3JAPHSYLR4ZJHYOKKTLXMQ, seg_pos={0 0}, upload=438666]: Done uploading piece. err: <nil> Dec 26 11:17:14.814376751 (499h48m14.685459796s) [noise=true , address=77.90.31.14:4661 , piece_id=GVZWQLR6ETAA5DVIY4FNBE37DC6B72JGOHQQRJXLSNRCHQNK3CVQ, seg_pos={0 0}, upload=438666]: Done uploading piece. err: <nil> Dec 26 11:17:14.820988090 (499h48m14.692071136s) [noise=true , address=77.90.31.17:4773 , piece_id=PJ326GIID2MZ2JXZMUQGCZAMWM5WZTQHXFJ4CV3LEZ72K4B6ZBRA, seg_pos={0 0}, upload=438666]: Done uploading piece. err: <nil> Dec 26 11:17:14.853580242 (499h48m14.724663287s) [noise=true , address=77.90.31.16:4735 , piece_id=ZOR2FJ5C6C3EIGQQWH6M5UOVHAHSBISDN2YN2PD5C72D532ES57Q, seg_pos={0 0}, upload=438666]: Done uploading piece. err: <nil> Dec 26 11:17:15.021385681 (499h48m14.892468716s) [noise=true , address=77.90.31.10:4519 , piece_id=47QEELPAZKYSYILZF4MHYFDNQTUR2YQNSME7LBRUJXEBRI4TWQDQ, seg_pos={0 0}, upload=438666]: Done uploading piece. err: <nil> Dec 26 11:17:15.021407592 (499h48m14.892490627s) [seg_pos={0 0}, upload=438666]: Segment reached optimal threshold of 13 pieces. Dec 26 11:17:15.021671616 (499h48m14.892754651s) [noise=true , address=45.91.116.6:4975 , piece_id=CDAJNYJWSPWIAFSHHL3UKEPD2GT3JIW7FCHM6NMPNJQ57FKVKM4Q, seg_pos={0 0}, upload=438666]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: read tcp 188.166.225.170:39522->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 11:17:15.021677657 (499h48m14.892760692s) [noise=true , address=45.91.116.6:4975 , piece_id=CDAJNYJWSPWIAFSHHL3UKEPD2GT3JIW7FCHM6NMPNJQ57FKVKM4Q, seg_pos={0 0}, upload=438666]: Long tail cancellation due to optimal threshold Dec 26 11:17:15.021742240 (499h48m14.892825275s) [noise=true , address=45.91.116.10:4084 , piece_id=TZSYRWBTD63NLCH5L72WTT2EKBZV3D2GBKT6LSTZARXLOPQ2EAFQ, seg_pos={0 0}, upload=438666]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF Dec 26 11:17:15.021780964 (499h48m14.892864009s) [noise=true , address=45.91.116.10:4084 , piece_id=TZSYRWBTD63NLCH5L72WTT2EKBZV3D2GBKT6LSTZARXLOPQ2EAFQ, seg_pos={0 0}, upload=438666]: Long tail cancellation due to optimal threshold Dec 26 11:17:15.021928154 (499h48m14.893011199s) [noise=true , address=45.91.116.12:4226 , piece_id=T4NHHKAYFIRVVYFUBSK34OZHY5BH4SEHHMZKLXR6NCA7DZMRDDDQ, seg_pos={0 0}, upload=438666]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: write tcp 188.166.225.170:53630->45.91.116.12:4226: use of closed network connection; write tcp 188.166.225.170:53630->45.91.116.12:4226: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53630->45.91.116.12:4226: use of closed network connection Dec 26 11:17:15.021934826 (499h48m14.893017871s) [noise=true , address=45.91.116.12:4226 , piece_id=T4NHHKAYFIRVVYFUBSK34OZHY5BH4SEHHMZKLXR6NCA7DZMRDDDQ, seg_pos={0 0}, upload=438666]: Long tail cancellation due to optimal threshold Dec 26 11:17:15.021956368 (499h48m14.893039403s) [seg_pos={0 0}, upload=438666]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 11:17:15.022023355 (499h48m14.89310639s) [upload=438666]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 11:17:15.022028755 (499h48m14.89311179s) [upload=438666]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 11:17:15.321887056 (499h48m15.192970091s) [upload=438666]: Done uploading segments... Dec 26 11:17:15.321911402 (499h48m15.192994437s) [upload=438666]: Upload finished. err: <nil> Duration: 6.041608092s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......X.........X..........................................] duration: 31.795661003s trace: https://jaeger.dev.storj.io/trace/785a90e8be84a595 FAIL (2025-12-26 11:21:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== KXVRQQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.834123964 (499h15m56.705207009s) [noise=true , address=45.91.116.4:4963 , piece_id=U4T64KQ67C3BDKG5TJSTQWUS4WWYCSVPUZOWJ6YANR2IQ2JD4EFQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.834585043 (499h15m56.705668078s) [noise=true , address=145.79.227.9:4390 , piece_id=JDQNHNGABBGANIHKSKP4VHUWIPQBNNEOPC4ZQCKR3K5TTR4N4SGQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.834961800 (499h15m56.706044845s) [noise=true , address=145.79.227.30:4778 , piece_id=BMEAUWANP7FP6CFRULMIAZUV4JNQEXOUN2EAXJY5XOWLDZINWOIA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.835331334 (499h15m56.706414379s) [noise=true , address=77.90.31.18:4838 , piece_id=VYBIKY3MLEE2CXGS2MT23HX5HUFK25FKYKI4XLXPDRNWCC55ANXQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.835738399 (499h15m56.706821434s) [noise=true , address=145.79.227.20:4459 , piece_id=2EUW2AH7FBVZNJ27XVLRXKMXA737DWKAYBL7CR7PRXXRTO2M644A, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.836118653 (499h15m56.707201698s) [noise=true , address=77.90.31.14:4678 , piece_id=XQHQZJUPB24G3BWGFTFDD42NM6K5HJVHUXBXWH5TW7BU25QGHTMA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.836501923 (499h15m56.707584968s) [noise=true , address=45.91.116.6:4988 , piece_id=VPEKJF74D7SQWKR47MAIY5UJQPBJIEU5AQDKLFGATYIRHFREAQ5A, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.836904940 (499h15m56.707987985s) [noise=true , address=64.49.13.169:4711 , piece_id=PVBVY4TVJLPKYBP6A7LBNZADLN25XBDWZFEK2QVN45YHJATIR7TA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.837388641 (499h15m56.708471676s) [noise=true , address=145.79.227.40:4076 , piece_id=Y6WKUYNPDMBUZSNWEYDL2KKNR6WKHSKGP63S6KAG6F2MWMNTVMBQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.837910455 (499h15m56.7089935s) [noise=true , address=77.90.31.11:4571 , piece_id=OERZACIKIG7HBEV3LL42KUAGKN6AWZEXNQSA52FIYTUKDFWCBW7Q, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.838146514 (499h15m56.709229559s) [noise=true , address=64.49.13.26:4945 , piece_id=KXCFTLWR4HSBTP4GJJ4HEPA6BBEMILYTJEMXY4S5WB24HRMQJOKA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.838469329 (499h15m56.709552364s) [noise=true , address=45.91.116.5:4301 , piece_id=E3ZRKZUKM335ARHV3Z5YGLKQYHG6QV4GCCNHAVMN34BM7UQFWIIA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.838690831 (499h15m56.709773866s) [noise=true , address=45.91.116.14:4204 , piece_id=JWL7OOMKTVAP7WEUB5WTL3F6F5IRNID4KIOYJSYZ655PJSNXAQKQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.839143724 (499h15m56.710226759s) [noise=true , address=45.91.116.9:4272 , piece_id=GGFDVKEW5LUCHQQUP6CWJHJQXUETYUG2UISPVP6XQJGSAJW2KP2Q, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.839678982 (499h15m56.710762027s) [noise=true , address=145.79.227.44:4208 , piece_id=ZLIX7XAUMKRUEWWTQST2KLCOWFJSAFSAQNLUCTTOWVZUBESEI23A, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:58.136536405 (499h15m58.00761944s) [upload=438194]: Next returned nil segment Dec 26 10:44:58.136548628 (499h15m58.007631673s) [upload=438194]: Waiting for error group managing segments... Dec 26 10:44:58.469618858 (499h15m58.340701893s) [noise=true , address=64.49.13.26:4945 , piece_id=KXCFTLWR4HSBTP4GJJ4HEPA6BBEMILYTJEMXY4S5WB24HRMQJOKA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.708819055 (499h15m58.5799021s) [noise=true , address=145.79.227.9:4390 , piece_id=JDQNHNGABBGANIHKSKP4VHUWIPQBNNEOPC4ZQCKR3K5TTR4N4SGQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.714466429 (499h15m58.585549474s) [noise=true , address=145.79.227.40:4076 , piece_id=Y6WKUYNPDMBUZSNWEYDL2KKNR6WKHSKGP63S6KAG6F2MWMNTVMBQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.745407696 (499h15m58.616490731s) [noise=true , address=145.79.227.20:4459 , piece_id=2EUW2AH7FBVZNJ27XVLRXKMXA737DWKAYBL7CR7PRXXRTO2M644A, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.888701070 (499h15m58.759784105s) [noise=true , address=77.90.31.18:4838 , piece_id=VYBIKY3MLEE2CXGS2MT23HX5HUFK25FKYKI4XLXPDRNWCC55ANXQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.948150873 (499h15m58.819233918s) [noise=true , address=45.91.116.5:4301 , piece_id=E3ZRKZUKM335ARHV3Z5YGLKQYHG6QV4GCCNHAVMN34BM7UQFWIIA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.971389744 (499h15m58.842472779s) [noise=true , address=77.90.31.19:4868 , piece_id=WH77PZSXC2VUWLQYLBR2NDEVT6KMD3VZK5WAL53AMRDW74P242LQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.021685314 (499h15m58.892768349s) [noise=true , address=45.91.116.4:4963 , piece_id=U4T64KQ67C3BDKG5TJSTQWUS4WWYCSVPUZOWJ6YANR2IQ2JD4EFQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.045418216 (499h15m58.916501251s) [noise=true , address=45.91.116.14:4204 , piece_id=JWL7OOMKTVAP7WEUB5WTL3F6F5IRNID4KIOYJSYZ655PJSNXAQKQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.056260371 (499h15m58.927343416s) [noise=true , address=45.91.116.9:4272 , piece_id=GGFDVKEW5LUCHQQUP6CWJHJQXUETYUG2UISPVP6XQJGSAJW2KP2Q, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.085713054 (499h15m58.956796089s) [noise=true , address=45.91.116.7:4017 , piece_id=L4GGY6AICMSFMUVYGI6ZMUORYRXNJFPLJY5XJRUAM7NVYQOHPHSQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.640980124 (499h15m59.512063169s) [noise=true , address=145.79.227.30:4778 , piece_id=BMEAUWANP7FP6CFRULMIAZUV4JNQEXOUN2EAXJY5XOWLDZINWOIA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.654515607 (499h15m59.525598652s) [noise=true , address=145.79.227.44:4208 , piece_id=ZLIX7XAUMKRUEWWTQST2KLCOWFJSAFSAQNLUCTTOWVZUBESEI23A, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.660349657 (499h15m59.531432712s) [noise=true , address=77.90.31.11:4571 , piece_id=OERZACIKIG7HBEV3LL42KUAGKN6AWZEXNQSA52FIYTUKDFWCBW7Q, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.660376087 (499h15m59.531459132s) [seg_pos={0 0}, upload=438194]: Segment reached optimal threshold of 14 pieces. Dec 26 10:44:59.660612988 (499h15m59.531696033s) [noise=true , address=64.49.13.169:4711 , piece_id=PVBVY4TVJLPKYBP6A7LBNZADLN25XBDWZFEK2QVN45YHJATIR7TA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:52964->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:52964->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52964->64.49.13.169:4711: use of closed network connection Dec 26 10:44:59.660620753 (499h15m59.531703788s) [noise=true , address=64.49.13.169:4711 , piece_id=PVBVY4TVJLPKYBP6A7LBNZADLN25XBDWZFEK2QVN45YHJATIR7TA, seg_pos={0 0}, upload=438194]: Long tail cancellation due to optimal threshold Dec 26 10:44:59.660893663 (499h15m59.531976698s) [noise=true , address=45.91.116.6:4988 , piece_id=VPEKJF74D7SQWKR47MAIY5UJQPBJIEU5AQDKLFGATYIRHFREAQ5A, seg_pos={0 0}, upload=438194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TrcoCHhAJeEykP9hJNpygoPSdgFV5v6GzL48JpyNPiAj5ZJRS): protocol: expected piece hash; context canceled; EOF Dec 26 10:44:59.660901057 (499h15m59.531984102s) [noise=true , address=45.91.116.6:4988 , piece_id=VPEKJF74D7SQWKR47MAIY5UJQPBJIEU5AQDKLFGATYIRHFREAQ5A, seg_pos={0 0}, upload=438194]: Long tail cancellation due to optimal threshold Dec 26 10:44:59.661226155 (499h15m59.5323092s) [noise=true , address=77.90.31.14:4678 , piece_id=XQHQZJUPB24G3BWGFTFDD42NM6K5HJVHUXBXWH5TW7BU25QGHTMA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12beu3SUZMPNWfbeHV4tLtx9UVZoEHh4Kt7oBhT59w3bGBE1ovP): protocol: write tcp 188.166.225.170:35884->77.90.31.14:4678: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 10:44:59.661249630 (499h15m59.532332685s) [noise=true , address=77.90.31.14:4678 , piece_id=XQHQZJUPB24G3BWGFTFDD42NM6K5HJVHUXBXWH5TW7BU25QGHTMA, seg_pos={0 0}, upload=438194]: Long tail cancellation due to optimal threshold Dec 26 10:44:59.661309705 (499h15m59.53239275s) [noise=true , address=77.90.31.21:4909 , piece_id=H6KT2AKZISCU4VEA6TXJJ3EFGCRBZPPXBEJJEIHZR3GEJZKXVRQQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121RT7tP6czMFAiZ3DjzSSsYgPt3xZXVbXVpkcYxaZYscNiMjL6): protocol: expected piece hash; context canceled; EOF Dec 26 10:44:59.661318962 (499h15m59.532401997s) [noise=true , address=77.90.31.21:4909 , piece_id=H6KT2AKZISCU4VEA6TXJJ3EFGCRBZPPXBEJJEIHZR3GEJZKXVRQQ, seg_pos={0 0}, upload=438194]: Long tail cancellation due to optimal threshold Dec 26 10:44:59.661345674 (499h15m59.532428719s) [seg_pos={0 0}, upload=438194]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 10:44:59.661485770 (499h15m59.532568815s) [upload=438194]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 10:44:59.661493655 (499h15m59.53257669s) [upload=438194]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 10:44:59.968467153 (499h15m59.839550198s) [upload=438194]: Done uploading segments... Dec 26 10:44:59.968484155 (499h15m59.83956719s) [upload=438194]: Upload finished. err: <nil> Duration: 31.795678258s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [........X.........X.........................................] duration: 42.099547114s trace: https://jaeger.dev.storj.io/trace/785a90e8be84a595 FAIL (2025-12-26 11:20:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== KXVRQQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.834123964 (499h15m56.705207009s) [noise=true , address=45.91.116.4:4963 , piece_id=U4T64KQ67C3BDKG5TJSTQWUS4WWYCSVPUZOWJ6YANR2IQ2JD4EFQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.834585043 (499h15m56.705668078s) [noise=true , address=145.79.227.9:4390 , piece_id=JDQNHNGABBGANIHKSKP4VHUWIPQBNNEOPC4ZQCKR3K5TTR4N4SGQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.834961800 (499h15m56.706044845s) [noise=true , address=145.79.227.30:4778 , piece_id=BMEAUWANP7FP6CFRULMIAZUV4JNQEXOUN2EAXJY5XOWLDZINWOIA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.835331334 (499h15m56.706414379s) [noise=true , address=77.90.31.18:4838 , piece_id=VYBIKY3MLEE2CXGS2MT23HX5HUFK25FKYKI4XLXPDRNWCC55ANXQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.835738399 (499h15m56.706821434s) [noise=true , address=145.79.227.20:4459 , piece_id=2EUW2AH7FBVZNJ27XVLRXKMXA737DWKAYBL7CR7PRXXRTO2M644A, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.836118653 (499h15m56.707201698s) [noise=true , address=77.90.31.14:4678 , piece_id=XQHQZJUPB24G3BWGFTFDD42NM6K5HJVHUXBXWH5TW7BU25QGHTMA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.836501923 (499h15m56.707584968s) [noise=true , address=45.91.116.6:4988 , piece_id=VPEKJF74D7SQWKR47MAIY5UJQPBJIEU5AQDKLFGATYIRHFREAQ5A, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.836904940 (499h15m56.707987985s) [noise=true , address=64.49.13.169:4711 , piece_id=PVBVY4TVJLPKYBP6A7LBNZADLN25XBDWZFEK2QVN45YHJATIR7TA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.837388641 (499h15m56.708471676s) [noise=true , address=145.79.227.40:4076 , piece_id=Y6WKUYNPDMBUZSNWEYDL2KKNR6WKHSKGP63S6KAG6F2MWMNTVMBQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.837910455 (499h15m56.7089935s) [noise=true , address=77.90.31.11:4571 , piece_id=OERZACIKIG7HBEV3LL42KUAGKN6AWZEXNQSA52FIYTUKDFWCBW7Q, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.838146514 (499h15m56.709229559s) [noise=true , address=64.49.13.26:4945 , piece_id=KXCFTLWR4HSBTP4GJJ4HEPA6BBEMILYTJEMXY4S5WB24HRMQJOKA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.838469329 (499h15m56.709552364s) [noise=true , address=45.91.116.5:4301 , piece_id=E3ZRKZUKM335ARHV3Z5YGLKQYHG6QV4GCCNHAVMN34BM7UQFWIIA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.838690831 (499h15m56.709773866s) [noise=true , address=45.91.116.14:4204 , piece_id=JWL7OOMKTVAP7WEUB5WTL3F6F5IRNID4KIOYJSYZ655PJSNXAQKQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.839143724 (499h15m56.710226759s) [noise=true , address=45.91.116.9:4272 , piece_id=GGFDVKEW5LUCHQQUP6CWJHJQXUETYUG2UISPVP6XQJGSAJW2KP2Q, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.839678982 (499h15m56.710762027s) [noise=true , address=145.79.227.44:4208 , piece_id=ZLIX7XAUMKRUEWWTQST2KLCOWFJSAFSAQNLUCTTOWVZUBESEI23A, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:58.136536405 (499h15m58.00761944s) [upload=438194]: Next returned nil segment Dec 26 10:44:58.136548628 (499h15m58.007631673s) [upload=438194]: Waiting for error group managing segments... Dec 26 10:44:58.469618858 (499h15m58.340701893s) [noise=true , address=64.49.13.26:4945 , piece_id=KXCFTLWR4HSBTP4GJJ4HEPA6BBEMILYTJEMXY4S5WB24HRMQJOKA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.708819055 (499h15m58.5799021s) [noise=true , address=145.79.227.9:4390 , piece_id=JDQNHNGABBGANIHKSKP4VHUWIPQBNNEOPC4ZQCKR3K5TTR4N4SGQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.714466429 (499h15m58.585549474s) [noise=true , address=145.79.227.40:4076 , piece_id=Y6WKUYNPDMBUZSNWEYDL2KKNR6WKHSKGP63S6KAG6F2MWMNTVMBQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.745407696 (499h15m58.616490731s) [noise=true , address=145.79.227.20:4459 , piece_id=2EUW2AH7FBVZNJ27XVLRXKMXA737DWKAYBL7CR7PRXXRTO2M644A, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.888701070 (499h15m58.759784105s) [noise=true , address=77.90.31.18:4838 , piece_id=VYBIKY3MLEE2CXGS2MT23HX5HUFK25FKYKI4XLXPDRNWCC55ANXQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.948150873 (499h15m58.819233918s) [noise=true , address=45.91.116.5:4301 , piece_id=E3ZRKZUKM335ARHV3Z5YGLKQYHG6QV4GCCNHAVMN34BM7UQFWIIA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.971389744 (499h15m58.842472779s) [noise=true , address=77.90.31.19:4868 , piece_id=WH77PZSXC2VUWLQYLBR2NDEVT6KMD3VZK5WAL53AMRDW74P242LQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.021685314 (499h15m58.892768349s) [noise=true , address=45.91.116.4:4963 , piece_id=U4T64KQ67C3BDKG5TJSTQWUS4WWYCSVPUZOWJ6YANR2IQ2JD4EFQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.045418216 (499h15m58.916501251s) [noise=true , address=45.91.116.14:4204 , piece_id=JWL7OOMKTVAP7WEUB5WTL3F6F5IRNID4KIOYJSYZ655PJSNXAQKQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.056260371 (499h15m58.927343416s) [noise=true , address=45.91.116.9:4272 , piece_id=GGFDVKEW5LUCHQQUP6CWJHJQXUETYUG2UISPVP6XQJGSAJW2KP2Q, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.085713054 (499h15m58.956796089s) [noise=true , address=45.91.116.7:4017 , piece_id=L4GGY6AICMSFMUVYGI6ZMUORYRXNJFPLJY5XJRUAM7NVYQOHPHSQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.640980124 (499h15m59.512063169s) [noise=true , address=145.79.227.30:4778 , piece_id=BMEAUWANP7FP6CFRULMIAZUV4JNQEXOUN2EAXJY5XOWLDZINWOIA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.654515607 (499h15m59.525598652s) [noise=true , address=145.79.227.44:4208 , piece_id=ZLIX7XAUMKRUEWWTQST2KLCOWFJSAFSAQNLUCTTOWVZUBESEI23A, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.660349657 (499h15m59.531432712s) [noise=true , address=77.90.31.11:4571 , piece_id=OERZACIKIG7HBEV3LL42KUAGKN6AWZEXNQSA52FIYTUKDFWCBW7Q, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.660376087 (499h15m59.531459132s) [seg_pos={0 0}, upload=438194]: Segment reached optimal threshold of 14 pieces. Dec 26 10:44:59.660612988 (499h15m59.531696033s) [noise=true , address=64.49.13.169:4711 , piece_id=PVBVY4TVJLPKYBP6A7LBNZADLN25XBDWZFEK2QVN45YHJATIR7TA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:52964->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:52964->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52964->64.49.13.169:4711: use of closed network connection Dec 26 10:44:59.660620753 (499h15m59.531703788s) [noise=true , address=64.49.13.169:4711 , piece_id=PVBVY4TVJLPKYBP6A7LBNZADLN25XBDWZFEK2QVN45YHJATIR7TA, seg_pos={0 0}, upload=438194]: Long tail cancellation due to optimal threshold Dec 26 10:44:59.660893663 (499h15m59.531976698s) [noise=true , address=45.91.116.6:4988 , piece_id=VPEKJF74D7SQWKR47MAIY5UJQPBJIEU5AQDKLFGATYIRHFREAQ5A, seg_pos={0 0}, upload=438194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TrcoCHhAJeEykP9hJNpygoPSdgFV5v6GzL48JpyNPiAj5ZJRS): protocol: expected piece hash; context canceled; EOF Dec 26 10:44:59.660901057 (499h15m59.531984102s) [noise=true , address=45.91.116.6:4988 , piece_id=VPEKJF74D7SQWKR47MAIY5UJQPBJIEU5AQDKLFGATYIRHFREAQ5A, seg_pos={0 0}, upload=438194]: Long tail cancellation due to optimal threshold Dec 26 10:44:59.661226155 (499h15m59.5323092s) [noise=true , address=77.90.31.14:4678 , piece_id=XQHQZJUPB24G3BWGFTFDD42NM6K5HJVHUXBXWH5TW7BU25QGHTMA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12beu3SUZMPNWfbeHV4tLtx9UVZoEHh4Kt7oBhT59w3bGBE1ovP): protocol: write tcp 188.166.225.170:35884->77.90.31.14:4678: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 10:44:59.661249630 (499h15m59.532332685s) [noise=true , address=77.90.31.14:4678 , piece_id=XQHQZJUPB24G3BWGFTFDD42NM6K5HJVHUXBXWH5TW7BU25QGHTMA, seg_pos={0 0}, upload=438194]: Long tail cancellation due to optimal threshold Dec 26 10:44:59.661309705 (499h15m59.53239275s) [noise=true , address=77.90.31.21:4909 , piece_id=H6KT2AKZISCU4VEA6TXJJ3EFGCRBZPPXBEJJEIHZR3GEJZKXVRQQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121RT7tP6czMFAiZ3DjzSSsYgPt3xZXVbXVpkcYxaZYscNiMjL6): protocol: expected piece hash; context canceled; EOF Dec 26 10:44:59.661318962 (499h15m59.532401997s) [noise=true , address=77.90.31.21:4909 , piece_id=H6KT2AKZISCU4VEA6TXJJ3EFGCRBZPPXBEJJEIHZR3GEJZKXVRQQ, seg_pos={0 0}, upload=438194]: Long tail cancellation due to optimal threshold Dec 26 10:44:59.661345674 (499h15m59.532428719s) [seg_pos={0 0}, upload=438194]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 10:44:59.661485770 (499h15m59.532568815s) [upload=438194]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 10:44:59.661493655 (499h15m59.53257669s) [upload=438194]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 10:44:59.968467153 (499h15m59.839550198s) [upload=438194]: Done uploading segments... Dec 26 10:44:59.968484155 (499h15m59.83956719s) [upload=438194]: Upload finished. err: <nil> Duration: 42.099577322s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [.X..X..X.......XX.XX..X.......X..X.X......XX......XXX.XX....] duration: 6.370608055s trace: https://jaeger.dev.storj.io/trace/193e6c819c3089fd FAIL (2025-12-26 11:20:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 716 (499h48m12.134924751s) [upload=438666]: Done starting upload Dec 26 11:17:12.263890689 (499h48m12.134973744s) [upload=438666]: Uploading segments... Dec 26 11:17:12.264121739 (499h48m12.135204784s) [upload=438666]: Got next segment. Inline: false Dec 26 11:17:12.264154531 (499h48m12.135237577s) [upload=438666]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 11:17:12.264161665 (499h48m12.13524471s) [upload=438666]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 11:17:12.513895599 (499h48m12.384978644s) [seg_pos={0 0}, upload=438666]: Begin upload segment... Dec 26 11:17:12.513949633 (499h48m12.385032678s) [seg_pos={0 0}, upload=438666]: Done begin upload segment. Dec 26 11:17:12.513986573 (499h48m12.385069608s) [noise=true , address=145.79.227.38:4978 , piece_id=QJZMHIFPCA57RDXEY36X5FNX4GY2JPTIIJJTKJJEO37SE3Y7YHGA, seg_pos={0 0}, upload=438666]: Uploading piece... Dec 26 11:17:12.518924846 (499h48m12.390007881s) [noise=true , address=45.91.116.6:4975 , piece_id=CDAJNYJWSPWIAFSHHL3UKEPD2GT3JIW7FCHM6NMPNJQ57FKVKM4Q, seg_pos={0 0}, upload=438666]: Uploading piece... Dec 26 11:17:12.520769569 (499h48m12.391852634s) [noise=true , address=145.79.227.50:4375 , piece_id=IL3SV5AKAYTDLLX3SNJ2Y2YHUQUWI52CKRYLIYQMSQWZZEVCNCNQ, seg_pos={0 0}, upload=438666]: Uploading piece... Dec 26 11:17:12.522359848 (499h48m12.393442883s) [noise=true , address=77.90.31.16:4735 , piece_id=ZOR2FJ5C6C3EIGQQWH6M5UOVHAHSBISDN2YN2PD5C72D532ES57Q, seg_pos={0 0}, upload=438666]: Uploading piece... Dec 26 11:17:12.523493195 (499h48m12.39457624s) [noise=true , address=77.90.31.17:4773 , piece_id=PJ326GIID2MZ2JXZMUQGCZAMWM5WZTQHXFJ4CV3LEZ72K4B6ZBRA, seg_pos={0 0}, upload=438666]: Uploading piece... Dec 26 11:17:12.526770225 (499h48m12.39785326s) [noise=true , address=145.79.227.24:4583 , piece_id=Z56FPKVQK2CZVLGL6CTK3IYGS3GN6C3JAPHSYLR4ZJHYOKKTLXMQ, seg_pos={0 0}, upload=438666]: Uploading piece... Dec 26 11:17:12.530787165 (499h48m12.40187021s) [noise=true , address=145.79.227.13:4803 , piece_id=PPZ74A3ZVV7I6HHUASUPXAMYD7BJTPIEZNR7TNO2EGKZGXRU5OKA, seg_pos={0 0}, upload=438666]: Uploading piece... Dec 26 11:17:12.534103499 (499h48m12.405186544s) [noise=true , address=145.79.227.49:4338 , piece_id=GZIWJ42YTCA2AQ2O2ALV4RFE4RWAZLFIDNEDZC6AI67YQXIRMA7A, seg_pos={0 0}, upload=438666]: Uploading piece... Dec 26 11:17:12.536031159 (499h48m12.407114204s) [noise=true , address=145.79.227.19:4077 , piece_id=RI4DDZEMOINLI6BUM7RXBDLX5H7NCYJMBEOAHLR3NSLZHNJD7ZGQ, seg_pos={0 0}, upload=438666]: Uploading piece... Dec 26 11:17:12.540743423 (499h48m12.411826468s) [noise=true , address=77.90.31.10:4519 , piece_id=47QEELPAZKYSYILZF4MHYFDNQTUR2YQNSME7LBRUJXEBRI4TWQDQ, seg_pos={0 0}, upload=438666]: Uploading piece... Dec 26 11:17:12.542682845 (499h48m12.41376591s) [noise=true , address=45.91.116.10:4084 , piece_id=TZSYRWBTD63NLCH5L72WTT2EKBZV3D2GBKT6LSTZARXLOPQ2EAFQ, seg_pos={0 0}, upload=438666]: Uploading piece... Dec 26 11:17:12.543247732 (499h48m12.414330777s) [noise=true , address=45.91.116.12:4226 , piece_id=T4NHHKAYFIRVVYFUBSK34OZHY5BH4SEHHMZKLXR6NCA7DZMRDDDQ, seg_pos={0 0}, upload=438666]: Uploading piece... Dec 26 11:17:12.544640514 (499h48m12.415723559s) [noise=true , address=145.79.227.4:4187 , piece_id=EGTOIDYEADC2MB77FTKGIR374CLCTVTLLSYMPTMGV3GTM6L4ANKA, seg_pos={0 0}, upload=438666]: Uploading piece... Dec 26 11:17:12.546342665 (499h48m12.41742571s) [noise=true , address=145.79.227.29:4941 , piece_id=JCG4GRZBB4JLBGAVL4MR7P57OQINGGF42W4W7G3PBXJFSGYUFRNA, seg_pos={0 0}, upload=438666]: Uploading piece... Dec 26 11:17:12.553741163 (499h48m12.424824208s) [noise=true , address=145.79.227.7:4293 , piece_id=JBHIXSZ3JBPNAPXFDBX4ZLMIQNKVKB2ZZIYVWLBK3SWYZIZHSU2Q, seg_pos={0 0}, upload=438666]: Uploading piece... Dec 26 11:17:12.555548043 (499h48m12.426631088s) [noise=true , address=77.90.31.14:4661 , piece_id=GVZWQLR6ETAA5DVIY4FNBE37DC6B72JGOHQQRJXLSNRCHQNK3CVQ, seg_pos={0 0}, upload=438666]: Uploading piece... Dec 26 11:17:14.277062566 (499h48m14.148145611s) [upload=438666]: Next returned nil segment Dec 26 11:17:14.277080911 (499h48m14.148163956s) [upload=438666]: Waiting for error group managing segments... Dec 26 11:17:14.568798930 (499h48m14.439881965s) [noise=true , address=145.79.227.19:4077 , piece_id=RI4DDZEMOINLI6BUM7RXBDLX5H7NCYJMBEOAHLR3NSLZHNJD7ZGQ, seg_pos={0 0}, upload=438666]: Done uploading piece. err: <nil> Dec 26 11:17:14.573695865 (499h48m14.44477891s) [noise=true , address=145.79.227.50:4375 , piece_id=IL3SV5AKAYTDLLX3SNJ2Y2YHUQUWI52CKRYLIYQMSQWZZEVCNCNQ, seg_pos={0 0}, upload=438666]: Done uploading piece. err: <nil> Dec 26 11:17:14.578731304 (499h48m14.449814339s) [noise=true , address=145.79.227.13:4803 , piece_id=PPZ74A3ZVV7I6HHUASUPXAMYD7BJTPIEZNR7TNO2EGKZGXRU5OKA, seg_pos={0 0}, upload=438666]: Done uploading piece. err: <nil> Dec 26 11:17:14.578885477 (499h48m14.449968522s) [noise=true , address=145.79.227.4:4187 , piece_id=EGTOIDYEADC2MB77FTKGIR374CLCTVTLLSYMPTMGV3GTM6L4ANKA, seg_pos={0 0}, upload=438666]: Done uploading piece. err: <nil> Dec 26 11:17:14.579341024 (499h48m14.45042406s) [noise=true , address=145.79.227.38:4978 , piece_id=QJZMHIFPCA57RDXEY36X5FNX4GY2JPTIIJJTKJJEO37SE3Y7YHGA, seg_pos={0 0}, upload=438666]: Done uploading piece. err: <nil> Dec 26 11:17:14.580493308 (499h48m14.451576344s) [noise=true , address=145.79.227.29:4941 , piece_id=JCG4GRZBB4JLBGAVL4MR7P57OQINGGF42W4W7G3PBXJFSGYUFRNA, seg_pos={0 0}, upload=438666]: Done uploading piece. err: <nil> Dec 26 11:17:14.581557736 (499h48m14.452640781s) [noise=true , address=145.79.227.49:4338 , piece_id=GZIWJ42YTCA2AQ2O2ALV4RFE4RWAZLFIDNEDZC6AI67YQXIRMA7A, seg_pos={0 0}, upload=438666]: Done uploading piece. err: <nil> Dec 26 11:17:14.587511112 (499h48m14.458594157s) [noise=true , address=145.79.227.7:4293 , piece_id=JBHIXSZ3JBPNAPXFDBX4ZLMIQNKVKB2ZZIYVWLBK3SWYZIZHSU2Q, seg_pos={0 0}, upload=438666]: Done uploading piece. err: <nil> Dec 26 11:17:14.595857755 (499h48m14.46694079s) [noise=true , address=145.79.227.24:4583 , piece_id=Z56FPKVQK2CZVLGL6CTK3IYGS3GN6C3JAPHSYLR4ZJHYOKKTLXMQ, seg_pos={0 0}, upload=438666]: Done uploading piece. err: <nil> Dec 26 11:17:14.814376751 (499h48m14.685459796s) [noise=true , address=77.90.31.14:4661 , piece_id=GVZWQLR6ETAA5DVIY4FNBE37DC6B72JGOHQQRJXLSNRCHQNK3CVQ, seg_pos={0 0}, upload=438666]: Done uploading piece. err: <nil> Dec 26 11:17:14.820988090 (499h48m14.692071136s) [noise=true , address=77.90.31.17:4773 , piece_id=PJ326GIID2MZ2JXZMUQGCZAMWM5WZTQHXFJ4CV3LEZ72K4B6ZBRA, seg_pos={0 0}, upload=438666]: Done uploading piece. err: <nil> Dec 26 11:17:14.853580242 (499h48m14.724663287s) [noise=true , address=77.90.31.16:4735 , piece_id=ZOR2FJ5C6C3EIGQQWH6M5UOVHAHSBISDN2YN2PD5C72D532ES57Q, seg_pos={0 0}, upload=438666]: Done uploading piece. err: <nil> Dec 26 11:17:15.021385681 (499h48m14.892468716s) [noise=true , address=77.90.31.10:4519 , piece_id=47QEELPAZKYSYILZF4MHYFDNQTUR2YQNSME7LBRUJXEBRI4TWQDQ, seg_pos={0 0}, upload=438666]: Done uploading piece. err: <nil> Dec 26 11:17:15.021407592 (499h48m14.892490627s) [seg_pos={0 0}, upload=438666]: Segment reached optimal threshold of 13 pieces. Dec 26 11:17:15.021671616 (499h48m14.892754651s) [noise=true , address=45.91.116.6:4975 , piece_id=CDAJNYJWSPWIAFSHHL3UKEPD2GT3JIW7FCHM6NMPNJQ57FKVKM4Q, seg_pos={0 0}, upload=438666]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: read tcp 188.166.225.170:39522->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 11:17:15.021677657 (499h48m14.892760692s) [noise=true , address=45.91.116.6:4975 , piece_id=CDAJNYJWSPWIAFSHHL3UKEPD2GT3JIW7FCHM6NMPNJQ57FKVKM4Q, seg_pos={0 0}, upload=438666]: Long tail cancellation due to optimal threshold Dec 26 11:17:15.021742240 (499h48m14.892825275s) [noise=true , address=45.91.116.10:4084 , piece_id=TZSYRWBTD63NLCH5L72WTT2EKBZV3D2GBKT6LSTZARXLOPQ2EAFQ, seg_pos={0 0}, upload=438666]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF Dec 26 11:17:15.021780964 (499h48m14.892864009s) [noise=true , address=45.91.116.10:4084 , piece_id=TZSYRWBTD63NLCH5L72WTT2EKBZV3D2GBKT6LSTZARXLOPQ2EAFQ, seg_pos={0 0}, upload=438666]: Long tail cancellation due to optimal threshold Dec 26 11:17:15.021928154 (499h48m14.893011199s) [noise=true , address=45.91.116.12:4226 , piece_id=T4NHHKAYFIRVVYFUBSK34OZHY5BH4SEHHMZKLXR6NCA7DZMRDDDQ, seg_pos={0 0}, upload=438666]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: write tcp 188.166.225.170:53630->45.91.116.12:4226: use of closed network connection; write tcp 188.166.225.170:53630->45.91.116.12:4226: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53630->45.91.116.12:4226: use of closed network connection Dec 26 11:17:15.021934826 (499h48m14.893017871s) [noise=true , address=45.91.116.12:4226 , piece_id=T4NHHKAYFIRVVYFUBSK34OZHY5BH4SEHHMZKLXR6NCA7DZMRDDDQ, seg_pos={0 0}, upload=438666]: Long tail cancellation due to optimal threshold Dec 26 11:17:15.021956368 (499h48m14.893039403s) [seg_pos={0 0}, upload=438666]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 11:17:15.022023355 (499h48m14.89310639s) [upload=438666]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 11:17:15.022028755 (499h48m14.89311179s) [upload=438666]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 11:17:15.321887056 (499h48m15.192970091s) [upload=438666]: Done uploading segments... Dec 26 11:17:15.321911402 (499h48m15.192994437s) [upload=438666]: Upload finished. err: <nil> Duration: 6.370658712s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)