Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 17 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXX....................................X.......] duration: 3.662451285s trace: https://jaeger.dev.storj.io/trace/30ef13d1b320910a FAIL (2025-11-21 12:10:51 UTC) orbiter warning: context deadline exceeded Duration: 3.662480782s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [X............X....X.X.X..X.X..X...X..XXX.X.XX...X...XX......] duration: 5.988195307s trace: https://jaeger.dev.storj.io/trace/a61d21c4ccf9666 FAIL (2025-11-21 12:10:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: 5W5AXIKSPSX6MGGV25AQZMKXOEUOUDT6LJ3ALXWTUXGGJJGPHMGQ) context deadline exceeded; piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: HEZ2FM5Q6XK6RPTD5TAU3TAHCDJKMGFMZKNOT77G5DWQFA2LLSGQ) context deadline exceeded ======LOGS====== (292h2m15.53524253s) [upload=257194]: Uploading segments... Nov 21 12:06:16.984911178 (292h2m15.536381649s) [upload=257194]: Got next segment. Inline: false Nov 21 12:06:16.984990128 (292h2m15.536460599s) [upload=257194]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 12:06:16.984998183 (292h2m15.536468644s) [upload=257194]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 12:06:17.228288865 (292h2m15.779759326s) [seg_pos={0 0}, upload=257194]: Begin upload segment... Nov 21 12:06:17.228357676 (292h2m15.779828137s) [seg_pos={0 0}, upload=257194]: Done begin upload segment. Nov 21 12:06:17.228448919 (292h2m15.77991938s) [noise=true , address=145.79.227.24:4583 , piece_id=P4YAEWAL5QJAEZ7QAILYI5FZFZBYLKADVQLMSVMYCAKZNLSVEMFQ, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.228492993 (292h2m15.779963454s) [noise=true , address=77.90.31.16:4735 , piece_id=YZ333DAUCNN3PG7TRJVU5LNO7AZBBHFMLM6VCNUSQEMDYRVOAKMQ, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.229164662 (292h2m15.780635123s) [noise=true , address=145.79.227.27:4690 , piece_id=GIYOKO5RULD7KKZXIA542RZO2O7WUP4GFU556MGRCST356HARMEA, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.229260765 (292h2m15.780731227s) [noise=true , address=145.79.227.30:4761 , piece_id=FNEHOUE5BJQTQILNEVTTZCHYDWCQGWEHIX6DX5RDQWYGCSYQKE7Q, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.230231854 (292h2m15.781702315s) [noise=true , address=77.90.31.13:4627 , piece_id=5W5AXIKSPSX6MGGV25AQZMKXOEUOUDT6LJ3ALXWTUXGGJJGPHMGQ, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.230413711 (292h2m15.781884172s) [noise=true , address=45.91.116.5:4298 , piece_id=BDMI2ONMMXWTHZBQDOD4CJGNJZK7OGLB4VRVOZEGCCMD53S3ZVQA, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.231487476 (292h2m15.782957927s) [noise=true , address=145.79.227.25:4619 , piece_id=5LQLNFMJSFGJ4BVZMPSBPSOWQKNH7NYIGYFKXQX2JH4ITV2OOQ4Q, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.231583307 (292h2m15.783053758s) [noise=true , address=145.79.227.30:4763 , piece_id=TPJMEV5GZFAC2UZESKDGVNWZ5GCMVMCQWLOSNUJC6PNXUEXEG4VA, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.231654042 (292h2m15.783124503s) [noise=true , address=77.90.31.17:4770 , piece_id=UUU7FZIUIX2IC7F5QIZC5JKA35O2RHZ5I6P4PSMAJCRB2A7OUGLQ, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.231699840 (292h2m15.783170301s) [noise=true , address=145.79.227.26:4654 , piece_id=EFU4C5HZSEVQXOTMLH62VRMI6O22HKUFXMKLODY2CJQKODI7ZDCQ, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.232691227 (292h2m15.784161688s) [noise=true , address=77.90.31.21:4879 , piece_id=WQJBQIFULX7LGWXDQB3WGPXR5K5LHYLOKIEDXMCYJSYWP6YHXMFA, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.232924111 (292h2m15.784394572s) [noise=true , address=77.90.31.10:4518 , piece_id=HEZ2FM5Q6XK6RPTD5TAU3TAHCDJKMGFMZKNOT77G5DWQFA2LLSGQ, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.234049513 (292h2m15.785520005s) [noise=true , address=145.79.227.23:4547 , piece_id=EWWBDUM52GIS5KX3FKYR5C7EXNDSDIICJ47NONYMOV6E6K2D2QNQ, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.235293583 (292h2m15.786764044s) [noise=true , address=145.79.227.16:4912 , piece_id=5VSFQ3LC3G4Y2VZB6RNGSWFKEBJ4ZHEKJI3MN6AZ4RQ5GIEC6TTQ, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.235803664 (292h2m15.787274125s) [noise=true , address=145.79.227.28:4725 , piece_id=V4SLAUQ244S53J6QLWM32NO2DQ2DS6NPDFZPZMX3CVY5DWM3KEMA, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.238435937 (292h2m15.789906398s) [noise=true , address=145.79.227.21:4473 , piece_id=BFGHAWIA7ERR5BI5I6M3YCQPQC7M6TOBEART2WCE6Q3RUVYP222A, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:18.900466542 (292h2m17.451937033s) [upload=257194]: Next returned nil segment Nov 21 12:06:18.900488314 (292h2m17.451958785s) [upload=257194]: Waiting for error group managing segments... Nov 21 12:06:19.223504608 (292h2m17.774975069s) [noise=true , address=145.79.227.26:4654 , piece_id=EFU4C5HZSEVQXOTMLH62VRMI6O22HKUFXMKLODY2CJQKODI7ZDCQ, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.224063582 (292h2m17.775534043s) [noise=true , address=145.79.227.30:4761 , piece_id=FNEHOUE5BJQTQILNEVTTZCHYDWCQGWEHIX6DX5RDQWYGCSYQKE7Q, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.224813419 (292h2m17.77628388s) [noise=true , address=145.79.227.27:4690 , piece_id=GIYOKO5RULD7KKZXIA542RZO2O7WUP4GFU556MGRCST356HARMEA, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.234614373 (292h2m17.786084824s) [noise=true , address=145.79.227.16:4912 , piece_id=5VSFQ3LC3G4Y2VZB6RNGSWFKEBJ4ZHEKJI3MN6AZ4RQ5GIEC6TTQ, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.235452939 (292h2m17.7869234s) [noise=true , address=145.79.227.21:4473 , piece_id=BFGHAWIA7ERR5BI5I6M3YCQPQC7M6TOBEART2WCE6Q3RUVYP222A, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.238949498 (292h2m17.790419959s) [noise=true , address=145.79.227.28:4725 , piece_id=V4SLAUQ244S53J6QLWM32NO2DQ2DS6NPDFZPZMX3CVY5DWM3KEMA, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.248638006 (292h2m17.800108467s) [noise=true , address=145.79.227.25:4619 , piece_id=5LQLNFMJSFGJ4BVZMPSBPSOWQKNH7NYIGYFKXQX2JH4ITV2OOQ4Q, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.249046234 (292h2m17.800516695s) [noise=true , address=145.79.227.30:4763 , piece_id=TPJMEV5GZFAC2UZESKDGVNWZ5GCMVMCQWLOSNUJC6PNXUEXEG4VA, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.249660604 (292h2m17.801131055s) [noise=true , address=145.79.227.24:4583 , piece_id=P4YAEWAL5QJAEZ7QAILYI5FZFZBYLKADVQLMSVMYCAKZNLSVEMFQ, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.250828577 (292h2m17.802299038s) [noise=true , address=145.79.227.23:4547 , piece_id=EWWBDUM52GIS5KX3FKYR5C7EXNDSDIICJ47NONYMOV6E6K2D2QNQ, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.830354733 (292h2m18.381825194s) [noise=true , address=77.90.31.13:4627 , piece_id=5W5AXIKSPSX6MGGV25AQZMKXOEUOUDT6LJ3ALXWTUXGGJJGPHMGQ, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.893988254 (292h2m18.445458715s) [noise=true , address=77.90.31.10:4518 , piece_id=HEZ2FM5Q6XK6RPTD5TAU3TAHCDJKMGFMZKNOT77G5DWQFA2LLSGQ, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.985985273 (292h2m18.537455734s) [noise=true , address=77.90.31.21:4879 , piece_id=WQJBQIFULX7LGWXDQB3WGPXR5K5LHYLOKIEDXMCYJSYWP6YHXMFA, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.986007055 (292h2m18.537477516s) [seg_pos={0 0}, upload=257194]: Segment reached optimal threshold of 13 pieces. Nov 21 12:06:19.986208177 (292h2m18.537678638s) [noise=true , address=45.91.116.5:4298 , piece_id=BDMI2ONMMXWTHZBQDOD4CJGNJZK7OGLB4VRVOZEGCCMD53S3ZVQA, seg_pos={0 0}, upload=257194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: write tcp 188.166.225.170:39292->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:39292->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39292->45.91.116.5:4298: use of closed network connection Nov 21 12:06:19.986215171 (292h2m18.537685632s) [noise=true , address=45.91.116.5:4298 , piece_id=BDMI2ONMMXWTHZBQDOD4CJGNJZK7OGLB4VRVOZEGCCMD53S3ZVQA, seg_pos={0 0}, upload=257194]: Long tail cancellation due to optimal threshold Nov 21 12:06:19.986329428 (292h2m18.537799889s) [noise=true , address=77.90.31.16:4735 , piece_id=YZ333DAUCNN3PG7TRJVU5LNO7AZBBHFMLM6VCNUSQEMDYRVOAKMQ, seg_pos={0 0}, upload=257194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: write tcp 188.166.225.170:36976->77.90.31.16:4735: use of closed network connection; write tcp 188.166.225.170:36976->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36976->77.90.31.16:4735: use of closed network connection Nov 21 12:06:19.986334608 (292h2m18.537805069s) [noise=true , address=77.90.31.16:4735 , piece_id=YZ333DAUCNN3PG7TRJVU5LNO7AZBBHFMLM6VCNUSQEMDYRVOAKMQ, seg_pos={0 0}, upload=257194]: Long tail cancellation due to optimal threshold Nov 21 12:06:19.986374995 (292h2m18.537845466s) [noise=true , address=77.90.31.17:4770 , piece_id=UUU7FZIUIX2IC7F5QIZC5JKA35O2RHZ5I6P4PSMAJCRB2A7OUGLQ, seg_pos={0 0}, upload=257194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: expected piece hash; context canceled; EOF Nov 21 12:06:19.986397208 (292h2m18.537867669s) [noise=true , address=77.90.31.17:4770 , piece_id=UUU7FZIUIX2IC7F5QIZC5JKA35O2RHZ5I6P4PSMAJCRB2A7OUGLQ, seg_pos={0 0}, upload=257194]: Long tail cancellation due to optimal threshold Nov 21 12:06:19.986466850 (292h2m18.537937311s) [seg_pos={0 0}, upload=257194]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 21 12:06:19.986571549 (292h2m18.538042011s) [upload=257194]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 21 12:06:19.986580406 (292h2m18.538050867s) [upload=257194]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 21 12:06:20.274288336 (292h2m18.825758797s) [upload=257194]: Done uploading segments... Nov 21 12:06:20.274311019 (292h2m18.82578147s) [upload=257194]: Upload finished. err: <nil> Duration: 5.988225856s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 18 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXX....................................X......] duration: 5.002539841s trace: https://jaeger.dev.storj.io/trace/30ef13d1b320910a FAIL (2025-11-21 12:10:07 UTC) orbiter warning: context deadline exceeded Duration: 5.002571801s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [.X............X....X.X.X..X.X..X...X..XXX.X.XX...X...XX.....] duration: 5.959832698s trace: https://jaeger.dev.storj.io/trace/a61d21c4ccf9666 FAIL (2025-11-21 12:09:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: 5W5AXIKSPSX6MGGV25AQZMKXOEUOUDT6LJ3ALXWTUXGGJJGPHMGQ) context deadline exceeded; piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: HEZ2FM5Q6XK6RPTD5TAU3TAHCDJKMGFMZKNOT77G5DWQFA2LLSGQ) context deadline exceeded ======LOGS====== (292h2m15.53524253s) [upload=257194]: Uploading segments... Nov 21 12:06:16.984911178 (292h2m15.536381649s) [upload=257194]: Got next segment. Inline: false Nov 21 12:06:16.984990128 (292h2m15.536460599s) [upload=257194]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 12:06:16.984998183 (292h2m15.536468644s) [upload=257194]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 12:06:17.228288865 (292h2m15.779759326s) [seg_pos={0 0}, upload=257194]: Begin upload segment... Nov 21 12:06:17.228357676 (292h2m15.779828137s) [seg_pos={0 0}, upload=257194]: Done begin upload segment. Nov 21 12:06:17.228448919 (292h2m15.77991938s) [noise=true , address=145.79.227.24:4583 , piece_id=P4YAEWAL5QJAEZ7QAILYI5FZFZBYLKADVQLMSVMYCAKZNLSVEMFQ, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.228492993 (292h2m15.779963454s) [noise=true , address=77.90.31.16:4735 , piece_id=YZ333DAUCNN3PG7TRJVU5LNO7AZBBHFMLM6VCNUSQEMDYRVOAKMQ, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.229164662 (292h2m15.780635123s) [noise=true , address=145.79.227.27:4690 , piece_id=GIYOKO5RULD7KKZXIA542RZO2O7WUP4GFU556MGRCST356HARMEA, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.229260765 (292h2m15.780731227s) [noise=true , address=145.79.227.30:4761 , piece_id=FNEHOUE5BJQTQILNEVTTZCHYDWCQGWEHIX6DX5RDQWYGCSYQKE7Q, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.230231854 (292h2m15.781702315s) [noise=true , address=77.90.31.13:4627 , piece_id=5W5AXIKSPSX6MGGV25AQZMKXOEUOUDT6LJ3ALXWTUXGGJJGPHMGQ, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.230413711 (292h2m15.781884172s) [noise=true , address=45.91.116.5:4298 , piece_id=BDMI2ONMMXWTHZBQDOD4CJGNJZK7OGLB4VRVOZEGCCMD53S3ZVQA, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.231487476 (292h2m15.782957927s) [noise=true , address=145.79.227.25:4619 , piece_id=5LQLNFMJSFGJ4BVZMPSBPSOWQKNH7NYIGYFKXQX2JH4ITV2OOQ4Q, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.231583307 (292h2m15.783053758s) [noise=true , address=145.79.227.30:4763 , piece_id=TPJMEV5GZFAC2UZESKDGVNWZ5GCMVMCQWLOSNUJC6PNXUEXEG4VA, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.231654042 (292h2m15.783124503s) [noise=true , address=77.90.31.17:4770 , piece_id=UUU7FZIUIX2IC7F5QIZC5JKA35O2RHZ5I6P4PSMAJCRB2A7OUGLQ, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.231699840 (292h2m15.783170301s) [noise=true , address=145.79.227.26:4654 , piece_id=EFU4C5HZSEVQXOTMLH62VRMI6O22HKUFXMKLODY2CJQKODI7ZDCQ, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.232691227 (292h2m15.784161688s) [noise=true , address=77.90.31.21:4879 , piece_id=WQJBQIFULX7LGWXDQB3WGPXR5K5LHYLOKIEDXMCYJSYWP6YHXMFA, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.232924111 (292h2m15.784394572s) [noise=true , address=77.90.31.10:4518 , piece_id=HEZ2FM5Q6XK6RPTD5TAU3TAHCDJKMGFMZKNOT77G5DWQFA2LLSGQ, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.234049513 (292h2m15.785520005s) [noise=true , address=145.79.227.23:4547 , piece_id=EWWBDUM52GIS5KX3FKYR5C7EXNDSDIICJ47NONYMOV6E6K2D2QNQ, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.235293583 (292h2m15.786764044s) [noise=true , address=145.79.227.16:4912 , piece_id=5VSFQ3LC3G4Y2VZB6RNGSWFKEBJ4ZHEKJI3MN6AZ4RQ5GIEC6TTQ, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.235803664 (292h2m15.787274125s) [noise=true , address=145.79.227.28:4725 , piece_id=V4SLAUQ244S53J6QLWM32NO2DQ2DS6NPDFZPZMX3CVY5DWM3KEMA, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.238435937 (292h2m15.789906398s) [noise=true , address=145.79.227.21:4473 , piece_id=BFGHAWIA7ERR5BI5I6M3YCQPQC7M6TOBEART2WCE6Q3RUVYP222A, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:18.900466542 (292h2m17.451937033s) [upload=257194]: Next returned nil segment Nov 21 12:06:18.900488314 (292h2m17.451958785s) [upload=257194]: Waiting for error group managing segments... Nov 21 12:06:19.223504608 (292h2m17.774975069s) [noise=true , address=145.79.227.26:4654 , piece_id=EFU4C5HZSEVQXOTMLH62VRMI6O22HKUFXMKLODY2CJQKODI7ZDCQ, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.224063582 (292h2m17.775534043s) [noise=true , address=145.79.227.30:4761 , piece_id=FNEHOUE5BJQTQILNEVTTZCHYDWCQGWEHIX6DX5RDQWYGCSYQKE7Q, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.224813419 (292h2m17.77628388s) [noise=true , address=145.79.227.27:4690 , piece_id=GIYOKO5RULD7KKZXIA542RZO2O7WUP4GFU556MGRCST356HARMEA, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.234614373 (292h2m17.786084824s) [noise=true , address=145.79.227.16:4912 , piece_id=5VSFQ3LC3G4Y2VZB6RNGSWFKEBJ4ZHEKJI3MN6AZ4RQ5GIEC6TTQ, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.235452939 (292h2m17.7869234s) [noise=true , address=145.79.227.21:4473 , piece_id=BFGHAWIA7ERR5BI5I6M3YCQPQC7M6TOBEART2WCE6Q3RUVYP222A, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.238949498 (292h2m17.790419959s) [noise=true , address=145.79.227.28:4725 , piece_id=V4SLAUQ244S53J6QLWM32NO2DQ2DS6NPDFZPZMX3CVY5DWM3KEMA, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.248638006 (292h2m17.800108467s) [noise=true , address=145.79.227.25:4619 , piece_id=5LQLNFMJSFGJ4BVZMPSBPSOWQKNH7NYIGYFKXQX2JH4ITV2OOQ4Q, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.249046234 (292h2m17.800516695s) [noise=true , address=145.79.227.30:4763 , piece_id=TPJMEV5GZFAC2UZESKDGVNWZ5GCMVMCQWLOSNUJC6PNXUEXEG4VA, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.249660604 (292h2m17.801131055s) [noise=true , address=145.79.227.24:4583 , piece_id=P4YAEWAL5QJAEZ7QAILYI5FZFZBYLKADVQLMSVMYCAKZNLSVEMFQ, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.250828577 (292h2m17.802299038s) [noise=true , address=145.79.227.23:4547 , piece_id=EWWBDUM52GIS5KX3FKYR5C7EXNDSDIICJ47NONYMOV6E6K2D2QNQ, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.830354733 (292h2m18.381825194s) [noise=true , address=77.90.31.13:4627 , piece_id=5W5AXIKSPSX6MGGV25AQZMKXOEUOUDT6LJ3ALXWTUXGGJJGPHMGQ, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.893988254 (292h2m18.445458715s) [noise=true , address=77.90.31.10:4518 , piece_id=HEZ2FM5Q6XK6RPTD5TAU3TAHCDJKMGFMZKNOT77G5DWQFA2LLSGQ, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.985985273 (292h2m18.537455734s) [noise=true , address=77.90.31.21:4879 , piece_id=WQJBQIFULX7LGWXDQB3WGPXR5K5LHYLOKIEDXMCYJSYWP6YHXMFA, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.986007055 (292h2m18.537477516s) [seg_pos={0 0}, upload=257194]: Segment reached optimal threshold of 13 pieces. Nov 21 12:06:19.986208177 (292h2m18.537678638s) [noise=true , address=45.91.116.5:4298 , piece_id=BDMI2ONMMXWTHZBQDOD4CJGNJZK7OGLB4VRVOZEGCCMD53S3ZVQA, seg_pos={0 0}, upload=257194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: write tcp 188.166.225.170:39292->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:39292->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39292->45.91.116.5:4298: use of closed network connection Nov 21 12:06:19.986215171 (292h2m18.537685632s) [noise=true , address=45.91.116.5:4298 , piece_id=BDMI2ONMMXWTHZBQDOD4CJGNJZK7OGLB4VRVOZEGCCMD53S3ZVQA, seg_pos={0 0}, upload=257194]: Long tail cancellation due to optimal threshold Nov 21 12:06:19.986329428 (292h2m18.537799889s) [noise=true , address=77.90.31.16:4735 , piece_id=YZ333DAUCNN3PG7TRJVU5LNO7AZBBHFMLM6VCNUSQEMDYRVOAKMQ, seg_pos={0 0}, upload=257194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: write tcp 188.166.225.170:36976->77.90.31.16:4735: use of closed network connection; write tcp 188.166.225.170:36976->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36976->77.90.31.16:4735: use of closed network connection Nov 21 12:06:19.986334608 (292h2m18.537805069s) [noise=true , address=77.90.31.16:4735 , piece_id=YZ333DAUCNN3PG7TRJVU5LNO7AZBBHFMLM6VCNUSQEMDYRVOAKMQ, seg_pos={0 0}, upload=257194]: Long tail cancellation due to optimal threshold Nov 21 12:06:19.986374995 (292h2m18.537845466s) [noise=true , address=77.90.31.17:4770 , piece_id=UUU7FZIUIX2IC7F5QIZC5JKA35O2RHZ5I6P4PSMAJCRB2A7OUGLQ, seg_pos={0 0}, upload=257194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: expected piece hash; context canceled; EOF Nov 21 12:06:19.986397208 (292h2m18.537867669s) [noise=true , address=77.90.31.17:4770 , piece_id=UUU7FZIUIX2IC7F5QIZC5JKA35O2RHZ5I6P4PSMAJCRB2A7OUGLQ, seg_pos={0 0}, upload=257194]: Long tail cancellation due to optimal threshold Nov 21 12:06:19.986466850 (292h2m18.537937311s) [seg_pos={0 0}, upload=257194]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 21 12:06:19.986571549 (292h2m18.538042011s) [upload=257194]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 21 12:06:19.986580406 (292h2m18.538050867s) [upload=257194]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 21 12:06:20.274288336 (292h2m18.825758797s) [upload=257194]: Done uploading segments... Nov 21 12:06:20.274311019 (292h2m18.82578147s) [upload=257194]: Upload finished. err: <nil> Duration: 5.95986507s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [..X............X....X.X.X..X.X..X...X..XXX.X.XX...X...XX....] duration: 6.02530416s trace: https://jaeger.dev.storj.io/trace/a61d21c4ccf9666 FAIL (2025-11-21 12:09:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: 5W5AXIKSPSX6MGGV25AQZMKXOEUOUDT6LJ3ALXWTUXGGJJGPHMGQ) context deadline exceeded; piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: HEZ2FM5Q6XK6RPTD5TAU3TAHCDJKMGFMZKNOT77G5DWQFA2LLSGQ) context deadline exceeded ======LOGS====== (292h2m15.53524253s) [upload=257194]: Uploading segments... Nov 21 12:06:16.984911178 (292h2m15.536381649s) [upload=257194]: Got next segment. Inline: false Nov 21 12:06:16.984990128 (292h2m15.536460599s) [upload=257194]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 12:06:16.984998183 (292h2m15.536468644s) [upload=257194]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 12:06:17.228288865 (292h2m15.779759326s) [seg_pos={0 0}, upload=257194]: Begin upload segment... Nov 21 12:06:17.228357676 (292h2m15.779828137s) [seg_pos={0 0}, upload=257194]: Done begin upload segment. Nov 21 12:06:17.228448919 (292h2m15.77991938s) [noise=true , address=145.79.227.24:4583 , piece_id=P4YAEWAL5QJAEZ7QAILYI5FZFZBYLKADVQLMSVMYCAKZNLSVEMFQ, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.228492993 (292h2m15.779963454s) [noise=true , address=77.90.31.16:4735 , piece_id=YZ333DAUCNN3PG7TRJVU5LNO7AZBBHFMLM6VCNUSQEMDYRVOAKMQ, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.229164662 (292h2m15.780635123s) [noise=true , address=145.79.227.27:4690 , piece_id=GIYOKO5RULD7KKZXIA542RZO2O7WUP4GFU556MGRCST356HARMEA, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.229260765 (292h2m15.780731227s) [noise=true , address=145.79.227.30:4761 , piece_id=FNEHOUE5BJQTQILNEVTTZCHYDWCQGWEHIX6DX5RDQWYGCSYQKE7Q, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.230231854 (292h2m15.781702315s) [noise=true , address=77.90.31.13:4627 , piece_id=5W5AXIKSPSX6MGGV25AQZMKXOEUOUDT6LJ3ALXWTUXGGJJGPHMGQ, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.230413711 (292h2m15.781884172s) [noise=true , address=45.91.116.5:4298 , piece_id=BDMI2ONMMXWTHZBQDOD4CJGNJZK7OGLB4VRVOZEGCCMD53S3ZVQA, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.231487476 (292h2m15.782957927s) [noise=true , address=145.79.227.25:4619 , piece_id=5LQLNFMJSFGJ4BVZMPSBPSOWQKNH7NYIGYFKXQX2JH4ITV2OOQ4Q, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.231583307 (292h2m15.783053758s) [noise=true , address=145.79.227.30:4763 , piece_id=TPJMEV5GZFAC2UZESKDGVNWZ5GCMVMCQWLOSNUJC6PNXUEXEG4VA, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.231654042 (292h2m15.783124503s) [noise=true , address=77.90.31.17:4770 , piece_id=UUU7FZIUIX2IC7F5QIZC5JKA35O2RHZ5I6P4PSMAJCRB2A7OUGLQ, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.231699840 (292h2m15.783170301s) [noise=true , address=145.79.227.26:4654 , piece_id=EFU4C5HZSEVQXOTMLH62VRMI6O22HKUFXMKLODY2CJQKODI7ZDCQ, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.232691227 (292h2m15.784161688s) [noise=true , address=77.90.31.21:4879 , piece_id=WQJBQIFULX7LGWXDQB3WGPXR5K5LHYLOKIEDXMCYJSYWP6YHXMFA, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.232924111 (292h2m15.784394572s) [noise=true , address=77.90.31.10:4518 , piece_id=HEZ2FM5Q6XK6RPTD5TAU3TAHCDJKMGFMZKNOT77G5DWQFA2LLSGQ, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.234049513 (292h2m15.785520005s) [noise=true , address=145.79.227.23:4547 , piece_id=EWWBDUM52GIS5KX3FKYR5C7EXNDSDIICJ47NONYMOV6E6K2D2QNQ, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.235293583 (292h2m15.786764044s) [noise=true , address=145.79.227.16:4912 , piece_id=5VSFQ3LC3G4Y2VZB6RNGSWFKEBJ4ZHEKJI3MN6AZ4RQ5GIEC6TTQ, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.235803664 (292h2m15.787274125s) [noise=true , address=145.79.227.28:4725 , piece_id=V4SLAUQ244S53J6QLWM32NO2DQ2DS6NPDFZPZMX3CVY5DWM3KEMA, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.238435937 (292h2m15.789906398s) [noise=true , address=145.79.227.21:4473 , piece_id=BFGHAWIA7ERR5BI5I6M3YCQPQC7M6TOBEART2WCE6Q3RUVYP222A, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:18.900466542 (292h2m17.451937033s) [upload=257194]: Next returned nil segment Nov 21 12:06:18.900488314 (292h2m17.451958785s) [upload=257194]: Waiting for error group managing segments... Nov 21 12:06:19.223504608 (292h2m17.774975069s) [noise=true , address=145.79.227.26:4654 , piece_id=EFU4C5HZSEVQXOTMLH62VRMI6O22HKUFXMKLODY2CJQKODI7ZDCQ, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.224063582 (292h2m17.775534043s) [noise=true , address=145.79.227.30:4761 , piece_id=FNEHOUE5BJQTQILNEVTTZCHYDWCQGWEHIX6DX5RDQWYGCSYQKE7Q, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.224813419 (292h2m17.77628388s) [noise=true , address=145.79.227.27:4690 , piece_id=GIYOKO5RULD7KKZXIA542RZO2O7WUP4GFU556MGRCST356HARMEA, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.234614373 (292h2m17.786084824s) [noise=true , address=145.79.227.16:4912 , piece_id=5VSFQ3LC3G4Y2VZB6RNGSWFKEBJ4ZHEKJI3MN6AZ4RQ5GIEC6TTQ, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.235452939 (292h2m17.7869234s) [noise=true , address=145.79.227.21:4473 , piece_id=BFGHAWIA7ERR5BI5I6M3YCQPQC7M6TOBEART2WCE6Q3RUVYP222A, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.238949498 (292h2m17.790419959s) [noise=true , address=145.79.227.28:4725 , piece_id=V4SLAUQ244S53J6QLWM32NO2DQ2DS6NPDFZPZMX3CVY5DWM3KEMA, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.248638006 (292h2m17.800108467s) [noise=true , address=145.79.227.25:4619 , piece_id=5LQLNFMJSFGJ4BVZMPSBPSOWQKNH7NYIGYFKXQX2JH4ITV2OOQ4Q, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.249046234 (292h2m17.800516695s) [noise=true , address=145.79.227.30:4763 , piece_id=TPJMEV5GZFAC2UZESKDGVNWZ5GCMVMCQWLOSNUJC6PNXUEXEG4VA, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.249660604 (292h2m17.801131055s) [noise=true , address=145.79.227.24:4583 , piece_id=P4YAEWAL5QJAEZ7QAILYI5FZFZBYLKADVQLMSVMYCAKZNLSVEMFQ, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.250828577 (292h2m17.802299038s) [noise=true , address=145.79.227.23:4547 , piece_id=EWWBDUM52GIS5KX3FKYR5C7EXNDSDIICJ47NONYMOV6E6K2D2QNQ, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.830354733 (292h2m18.381825194s) [noise=true , address=77.90.31.13:4627 , piece_id=5W5AXIKSPSX6MGGV25AQZMKXOEUOUDT6LJ3ALXWTUXGGJJGPHMGQ, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.893988254 (292h2m18.445458715s) [noise=true , address=77.90.31.10:4518 , piece_id=HEZ2FM5Q6XK6RPTD5TAU3TAHCDJKMGFMZKNOT77G5DWQFA2LLSGQ, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.985985273 (292h2m18.537455734s) [noise=true , address=77.90.31.21:4879 , piece_id=WQJBQIFULX7LGWXDQB3WGPXR5K5LHYLOKIEDXMCYJSYWP6YHXMFA, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.986007055 (292h2m18.537477516s) [seg_pos={0 0}, upload=257194]: Segment reached optimal threshold of 13 pieces. Nov 21 12:06:19.986208177 (292h2m18.537678638s) [noise=true , address=45.91.116.5:4298 , piece_id=BDMI2ONMMXWTHZBQDOD4CJGNJZK7OGLB4VRVOZEGCCMD53S3ZVQA, seg_pos={0 0}, upload=257194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: write tcp 188.166.225.170:39292->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:39292->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39292->45.91.116.5:4298: use of closed network connection Nov 21 12:06:19.986215171 (292h2m18.537685632s) [noise=true , address=45.91.116.5:4298 , piece_id=BDMI2ONMMXWTHZBQDOD4CJGNJZK7OGLB4VRVOZEGCCMD53S3ZVQA, seg_pos={0 0}, upload=257194]: Long tail cancellation due to optimal threshold Nov 21 12:06:19.986329428 (292h2m18.537799889s) [noise=true , address=77.90.31.16:4735 , piece_id=YZ333DAUCNN3PG7TRJVU5LNO7AZBBHFMLM6VCNUSQEMDYRVOAKMQ, seg_pos={0 0}, upload=257194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: write tcp 188.166.225.170:36976->77.90.31.16:4735: use of closed network connection; write tcp 188.166.225.170:36976->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36976->77.90.31.16:4735: use of closed network connection Nov 21 12:06:19.986334608 (292h2m18.537805069s) [noise=true , address=77.90.31.16:4735 , piece_id=YZ333DAUCNN3PG7TRJVU5LNO7AZBBHFMLM6VCNUSQEMDYRVOAKMQ, seg_pos={0 0}, upload=257194]: Long tail cancellation due to optimal threshold Nov 21 12:06:19.986374995 (292h2m18.537845466s) [noise=true , address=77.90.31.17:4770 , piece_id=UUU7FZIUIX2IC7F5QIZC5JKA35O2RHZ5I6P4PSMAJCRB2A7OUGLQ, seg_pos={0 0}, upload=257194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: expected piece hash; context canceled; EOF Nov 21 12:06:19.986397208 (292h2m18.537867669s) [noise=true , address=77.90.31.17:4770 , piece_id=UUU7FZIUIX2IC7F5QIZC5JKA35O2RHZ5I6P4PSMAJCRB2A7OUGLQ, seg_pos={0 0}, upload=257194]: Long tail cancellation due to optimal threshold Nov 21 12:06:19.986466850 (292h2m18.537937311s) [seg_pos={0 0}, upload=257194]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 21 12:06:19.986571549 (292h2m18.538042011s) [upload=257194]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 21 12:06:19.986580406 (292h2m18.538050867s) [upload=257194]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 21 12:06:20.274288336 (292h2m18.825758797s) [upload=257194]: Done uploading segments... Nov 21 12:06:20.274311019 (292h2m18.82578147s) [upload=257194]: Upload finished. err: <nil> Duration: 6.025337084s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 19 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXX....................................X.....] duration: 4.419309059s trace: https://jaeger.dev.storj.io/trace/30ef13d1b320910a FAIL (2025-11-21 12:09:14 UTC) orbiter warning: context deadline exceeded Duration: 4.419338285s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [...X............X....X.X.X..X.X..X...X..XXX.X.XX...X...XX...] duration: 6.126932484s trace: https://jaeger.dev.storj.io/trace/a61d21c4ccf9666 FAIL (2025-11-21 12:08:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: 5W5AXIKSPSX6MGGV25AQZMKXOEUOUDT6LJ3ALXWTUXGGJJGPHMGQ) context deadline exceeded; piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: HEZ2FM5Q6XK6RPTD5TAU3TAHCDJKMGFMZKNOT77G5DWQFA2LLSGQ) context deadline exceeded ======LOGS====== (292h2m15.53524253s) [upload=257194]: Uploading segments... Nov 21 12:06:16.984911178 (292h2m15.536381649s) [upload=257194]: Got next segment. Inline: false Nov 21 12:06:16.984990128 (292h2m15.536460599s) [upload=257194]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 12:06:16.984998183 (292h2m15.536468644s) [upload=257194]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 12:06:17.228288865 (292h2m15.779759326s) [seg_pos={0 0}, upload=257194]: Begin upload segment... Nov 21 12:06:17.228357676 (292h2m15.779828137s) [seg_pos={0 0}, upload=257194]: Done begin upload segment. Nov 21 12:06:17.228448919 (292h2m15.77991938s) [noise=true , address=145.79.227.24:4583 , piece_id=P4YAEWAL5QJAEZ7QAILYI5FZFZBYLKADVQLMSVMYCAKZNLSVEMFQ, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.228492993 (292h2m15.779963454s) [noise=true , address=77.90.31.16:4735 , piece_id=YZ333DAUCNN3PG7TRJVU5LNO7AZBBHFMLM6VCNUSQEMDYRVOAKMQ, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.229164662 (292h2m15.780635123s) [noise=true , address=145.79.227.27:4690 , piece_id=GIYOKO5RULD7KKZXIA542RZO2O7WUP4GFU556MGRCST356HARMEA, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.229260765 (292h2m15.780731227s) [noise=true , address=145.79.227.30:4761 , piece_id=FNEHOUE5BJQTQILNEVTTZCHYDWCQGWEHIX6DX5RDQWYGCSYQKE7Q, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.230231854 (292h2m15.781702315s) [noise=true , address=77.90.31.13:4627 , piece_id=5W5AXIKSPSX6MGGV25AQZMKXOEUOUDT6LJ3ALXWTUXGGJJGPHMGQ, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.230413711 (292h2m15.781884172s) [noise=true , address=45.91.116.5:4298 , piece_id=BDMI2ONMMXWTHZBQDOD4CJGNJZK7OGLB4VRVOZEGCCMD53S3ZVQA, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.231487476 (292h2m15.782957927s) [noise=true , address=145.79.227.25:4619 , piece_id=5LQLNFMJSFGJ4BVZMPSBPSOWQKNH7NYIGYFKXQX2JH4ITV2OOQ4Q, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.231583307 (292h2m15.783053758s) [noise=true , address=145.79.227.30:4763 , piece_id=TPJMEV5GZFAC2UZESKDGVNWZ5GCMVMCQWLOSNUJC6PNXUEXEG4VA, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.231654042 (292h2m15.783124503s) [noise=true , address=77.90.31.17:4770 , piece_id=UUU7FZIUIX2IC7F5QIZC5JKA35O2RHZ5I6P4PSMAJCRB2A7OUGLQ, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.231699840 (292h2m15.783170301s) [noise=true , address=145.79.227.26:4654 , piece_id=EFU4C5HZSEVQXOTMLH62VRMI6O22HKUFXMKLODY2CJQKODI7ZDCQ, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.232691227 (292h2m15.784161688s) [noise=true , address=77.90.31.21:4879 , piece_id=WQJBQIFULX7LGWXDQB3WGPXR5K5LHYLOKIEDXMCYJSYWP6YHXMFA, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.232924111 (292h2m15.784394572s) [noise=true , address=77.90.31.10:4518 , piece_id=HEZ2FM5Q6XK6RPTD5TAU3TAHCDJKMGFMZKNOT77G5DWQFA2LLSGQ, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.234049513 (292h2m15.785520005s) [noise=true , address=145.79.227.23:4547 , piece_id=EWWBDUM52GIS5KX3FKYR5C7EXNDSDIICJ47NONYMOV6E6K2D2QNQ, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.235293583 (292h2m15.786764044s) [noise=true , address=145.79.227.16:4912 , piece_id=5VSFQ3LC3G4Y2VZB6RNGSWFKEBJ4ZHEKJI3MN6AZ4RQ5GIEC6TTQ, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.235803664 (292h2m15.787274125s) [noise=true , address=145.79.227.28:4725 , piece_id=V4SLAUQ244S53J6QLWM32NO2DQ2DS6NPDFZPZMX3CVY5DWM3KEMA, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.238435937 (292h2m15.789906398s) [noise=true , address=145.79.227.21:4473 , piece_id=BFGHAWIA7ERR5BI5I6M3YCQPQC7M6TOBEART2WCE6Q3RUVYP222A, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:18.900466542 (292h2m17.451937033s) [upload=257194]: Next returned nil segment Nov 21 12:06:18.900488314 (292h2m17.451958785s) [upload=257194]: Waiting for error group managing segments... Nov 21 12:06:19.223504608 (292h2m17.774975069s) [noise=true , address=145.79.227.26:4654 , piece_id=EFU4C5HZSEVQXOTMLH62VRMI6O22HKUFXMKLODY2CJQKODI7ZDCQ, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.224063582 (292h2m17.775534043s) [noise=true , address=145.79.227.30:4761 , piece_id=FNEHOUE5BJQTQILNEVTTZCHYDWCQGWEHIX6DX5RDQWYGCSYQKE7Q, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.224813419 (292h2m17.77628388s) [noise=true , address=145.79.227.27:4690 , piece_id=GIYOKO5RULD7KKZXIA542RZO2O7WUP4GFU556MGRCST356HARMEA, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.234614373 (292h2m17.786084824s) [noise=true , address=145.79.227.16:4912 , piece_id=5VSFQ3LC3G4Y2VZB6RNGSWFKEBJ4ZHEKJI3MN6AZ4RQ5GIEC6TTQ, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.235452939 (292h2m17.7869234s) [noise=true , address=145.79.227.21:4473 , piece_id=BFGHAWIA7ERR5BI5I6M3YCQPQC7M6TOBEART2WCE6Q3RUVYP222A, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.238949498 (292h2m17.790419959s) [noise=true , address=145.79.227.28:4725 , piece_id=V4SLAUQ244S53J6QLWM32NO2DQ2DS6NPDFZPZMX3CVY5DWM3KEMA, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.248638006 (292h2m17.800108467s) [noise=true , address=145.79.227.25:4619 , piece_id=5LQLNFMJSFGJ4BVZMPSBPSOWQKNH7NYIGYFKXQX2JH4ITV2OOQ4Q, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.249046234 (292h2m17.800516695s) [noise=true , address=145.79.227.30:4763 , piece_id=TPJMEV5GZFAC2UZESKDGVNWZ5GCMVMCQWLOSNUJC6PNXUEXEG4VA, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.249660604 (292h2m17.801131055s) [noise=true , address=145.79.227.24:4583 , piece_id=P4YAEWAL5QJAEZ7QAILYI5FZFZBYLKADVQLMSVMYCAKZNLSVEMFQ, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.250828577 (292h2m17.802299038s) [noise=true , address=145.79.227.23:4547 , piece_id=EWWBDUM52GIS5KX3FKYR5C7EXNDSDIICJ47NONYMOV6E6K2D2QNQ, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.830354733 (292h2m18.381825194s) [noise=true , address=77.90.31.13:4627 , piece_id=5W5AXIKSPSX6MGGV25AQZMKXOEUOUDT6LJ3ALXWTUXGGJJGPHMGQ, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.893988254 (292h2m18.445458715s) [noise=true , address=77.90.31.10:4518 , piece_id=HEZ2FM5Q6XK6RPTD5TAU3TAHCDJKMGFMZKNOT77G5DWQFA2LLSGQ, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.985985273 (292h2m18.537455734s) [noise=true , address=77.90.31.21:4879 , piece_id=WQJBQIFULX7LGWXDQB3WGPXR5K5LHYLOKIEDXMCYJSYWP6YHXMFA, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.986007055 (292h2m18.537477516s) [seg_pos={0 0}, upload=257194]: Segment reached optimal threshold of 13 pieces. Nov 21 12:06:19.986208177 (292h2m18.537678638s) [noise=true , address=45.91.116.5:4298 , piece_id=BDMI2ONMMXWTHZBQDOD4CJGNJZK7OGLB4VRVOZEGCCMD53S3ZVQA, seg_pos={0 0}, upload=257194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: write tcp 188.166.225.170:39292->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:39292->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39292->45.91.116.5:4298: use of closed network connection Nov 21 12:06:19.986215171 (292h2m18.537685632s) [noise=true , address=45.91.116.5:4298 , piece_id=BDMI2ONMMXWTHZBQDOD4CJGNJZK7OGLB4VRVOZEGCCMD53S3ZVQA, seg_pos={0 0}, upload=257194]: Long tail cancellation due to optimal threshold Nov 21 12:06:19.986329428 (292h2m18.537799889s) [noise=true , address=77.90.31.16:4735 , piece_id=YZ333DAUCNN3PG7TRJVU5LNO7AZBBHFMLM6VCNUSQEMDYRVOAKMQ, seg_pos={0 0}, upload=257194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: write tcp 188.166.225.170:36976->77.90.31.16:4735: use of closed network connection; write tcp 188.166.225.170:36976->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36976->77.90.31.16:4735: use of closed network connection Nov 21 12:06:19.986334608 (292h2m18.537805069s) [noise=true , address=77.90.31.16:4735 , piece_id=YZ333DAUCNN3PG7TRJVU5LNO7AZBBHFMLM6VCNUSQEMDYRVOAKMQ, seg_pos={0 0}, upload=257194]: Long tail cancellation due to optimal threshold Nov 21 12:06:19.986374995 (292h2m18.537845466s) [noise=true , address=77.90.31.17:4770 , piece_id=UUU7FZIUIX2IC7F5QIZC5JKA35O2RHZ5I6P4PSMAJCRB2A7OUGLQ, seg_pos={0 0}, upload=257194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: expected piece hash; context canceled; EOF Nov 21 12:06:19.986397208 (292h2m18.537867669s) [noise=true , address=77.90.31.17:4770 , piece_id=UUU7FZIUIX2IC7F5QIZC5JKA35O2RHZ5I6P4PSMAJCRB2A7OUGLQ, seg_pos={0 0}, upload=257194]: Long tail cancellation due to optimal threshold Nov 21 12:06:19.986466850 (292h2m18.537937311s) [seg_pos={0 0}, upload=257194]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 21 12:06:19.986571549 (292h2m18.538042011s) [upload=257194]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 21 12:06:19.986580406 (292h2m18.538050867s) [upload=257194]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 21 12:06:20.274288336 (292h2m18.825758797s) [upload=257194]: Done uploading segments... Nov 21 12:06:20.274311019 (292h2m18.82578147s) [upload=257194]: Upload finished. err: <nil> Duration: 6.126949917s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 20 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXX....................................X....] duration: 3.458506652s trace: https://jaeger.dev.storj.io/trace/30ef13d1b320910a FAIL (2025-11-21 12:08:16 UTC) orbiter warning: context deadline exceeded Duration: 3.458590973s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [....X............X....X.X.X..X.X..X...X..XXX.X.XX...X...XX..] duration: 5.771707719s trace: https://jaeger.dev.storj.io/trace/a61d21c4ccf9666 FAIL (2025-11-21 12:07:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: 5W5AXIKSPSX6MGGV25AQZMKXOEUOUDT6LJ3ALXWTUXGGJJGPHMGQ) context deadline exceeded; piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: HEZ2FM5Q6XK6RPTD5TAU3TAHCDJKMGFMZKNOT77G5DWQFA2LLSGQ) context deadline exceeded ======LOGS====== (292h2m15.53524253s) [upload=257194]: Uploading segments... Nov 21 12:06:16.984911178 (292h2m15.536381649s) [upload=257194]: Got next segment. Inline: false Nov 21 12:06:16.984990128 (292h2m15.536460599s) [upload=257194]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 12:06:16.984998183 (292h2m15.536468644s) [upload=257194]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 12:06:17.228288865 (292h2m15.779759326s) [seg_pos={0 0}, upload=257194]: Begin upload segment... Nov 21 12:06:17.228357676 (292h2m15.779828137s) [seg_pos={0 0}, upload=257194]: Done begin upload segment. Nov 21 12:06:17.228448919 (292h2m15.77991938s) [noise=true , address=145.79.227.24:4583 , piece_id=P4YAEWAL5QJAEZ7QAILYI5FZFZBYLKADVQLMSVMYCAKZNLSVEMFQ, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.228492993 (292h2m15.779963454s) [noise=true , address=77.90.31.16:4735 , piece_id=YZ333DAUCNN3PG7TRJVU5LNO7AZBBHFMLM6VCNUSQEMDYRVOAKMQ, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.229164662 (292h2m15.780635123s) [noise=true , address=145.79.227.27:4690 , piece_id=GIYOKO5RULD7KKZXIA542RZO2O7WUP4GFU556MGRCST356HARMEA, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.229260765 (292h2m15.780731227s) [noise=true , address=145.79.227.30:4761 , piece_id=FNEHOUE5BJQTQILNEVTTZCHYDWCQGWEHIX6DX5RDQWYGCSYQKE7Q, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.230231854 (292h2m15.781702315s) [noise=true , address=77.90.31.13:4627 , piece_id=5W5AXIKSPSX6MGGV25AQZMKXOEUOUDT6LJ3ALXWTUXGGJJGPHMGQ, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.230413711 (292h2m15.781884172s) [noise=true , address=45.91.116.5:4298 , piece_id=BDMI2ONMMXWTHZBQDOD4CJGNJZK7OGLB4VRVOZEGCCMD53S3ZVQA, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.231487476 (292h2m15.782957927s) [noise=true , address=145.79.227.25:4619 , piece_id=5LQLNFMJSFGJ4BVZMPSBPSOWQKNH7NYIGYFKXQX2JH4ITV2OOQ4Q, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.231583307 (292h2m15.783053758s) [noise=true , address=145.79.227.30:4763 , piece_id=TPJMEV5GZFAC2UZESKDGVNWZ5GCMVMCQWLOSNUJC6PNXUEXEG4VA, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.231654042 (292h2m15.783124503s) [noise=true , address=77.90.31.17:4770 , piece_id=UUU7FZIUIX2IC7F5QIZC5JKA35O2RHZ5I6P4PSMAJCRB2A7OUGLQ, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.231699840 (292h2m15.783170301s) [noise=true , address=145.79.227.26:4654 , piece_id=EFU4C5HZSEVQXOTMLH62VRMI6O22HKUFXMKLODY2CJQKODI7ZDCQ, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.232691227 (292h2m15.784161688s) [noise=true , address=77.90.31.21:4879 , piece_id=WQJBQIFULX7LGWXDQB3WGPXR5K5LHYLOKIEDXMCYJSYWP6YHXMFA, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.232924111 (292h2m15.784394572s) [noise=true , address=77.90.31.10:4518 , piece_id=HEZ2FM5Q6XK6RPTD5TAU3TAHCDJKMGFMZKNOT77G5DWQFA2LLSGQ, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.234049513 (292h2m15.785520005s) [noise=true , address=145.79.227.23:4547 , piece_id=EWWBDUM52GIS5KX3FKYR5C7EXNDSDIICJ47NONYMOV6E6K2D2QNQ, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.235293583 (292h2m15.786764044s) [noise=true , address=145.79.227.16:4912 , piece_id=5VSFQ3LC3G4Y2VZB6RNGSWFKEBJ4ZHEKJI3MN6AZ4RQ5GIEC6TTQ, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.235803664 (292h2m15.787274125s) [noise=true , address=145.79.227.28:4725 , piece_id=V4SLAUQ244S53J6QLWM32NO2DQ2DS6NPDFZPZMX3CVY5DWM3KEMA, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.238435937 (292h2m15.789906398s) [noise=true , address=145.79.227.21:4473 , piece_id=BFGHAWIA7ERR5BI5I6M3YCQPQC7M6TOBEART2WCE6Q3RUVYP222A, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:18.900466542 (292h2m17.451937033s) [upload=257194]: Next returned nil segment Nov 21 12:06:18.900488314 (292h2m17.451958785s) [upload=257194]: Waiting for error group managing segments... Nov 21 12:06:19.223504608 (292h2m17.774975069s) [noise=true , address=145.79.227.26:4654 , piece_id=EFU4C5HZSEVQXOTMLH62VRMI6O22HKUFXMKLODY2CJQKODI7ZDCQ, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.224063582 (292h2m17.775534043s) [noise=true , address=145.79.227.30:4761 , piece_id=FNEHOUE5BJQTQILNEVTTZCHYDWCQGWEHIX6DX5RDQWYGCSYQKE7Q, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.224813419 (292h2m17.77628388s) [noise=true , address=145.79.227.27:4690 , piece_id=GIYOKO5RULD7KKZXIA542RZO2O7WUP4GFU556MGRCST356HARMEA, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.234614373 (292h2m17.786084824s) [noise=true , address=145.79.227.16:4912 , piece_id=5VSFQ3LC3G4Y2VZB6RNGSWFKEBJ4ZHEKJI3MN6AZ4RQ5GIEC6TTQ, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.235452939 (292h2m17.7869234s) [noise=true , address=145.79.227.21:4473 , piece_id=BFGHAWIA7ERR5BI5I6M3YCQPQC7M6TOBEART2WCE6Q3RUVYP222A, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.238949498 (292h2m17.790419959s) [noise=true , address=145.79.227.28:4725 , piece_id=V4SLAUQ244S53J6QLWM32NO2DQ2DS6NPDFZPZMX3CVY5DWM3KEMA, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.248638006 (292h2m17.800108467s) [noise=true , address=145.79.227.25:4619 , piece_id=5LQLNFMJSFGJ4BVZMPSBPSOWQKNH7NYIGYFKXQX2JH4ITV2OOQ4Q, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.249046234 (292h2m17.800516695s) [noise=true , address=145.79.227.30:4763 , piece_id=TPJMEV5GZFAC2UZESKDGVNWZ5GCMVMCQWLOSNUJC6PNXUEXEG4VA, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.249660604 (292h2m17.801131055s) [noise=true , address=145.79.227.24:4583 , piece_id=P4YAEWAL5QJAEZ7QAILYI5FZFZBYLKADVQLMSVMYCAKZNLSVEMFQ, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.250828577 (292h2m17.802299038s) [noise=true , address=145.79.227.23:4547 , piece_id=EWWBDUM52GIS5KX3FKYR5C7EXNDSDIICJ47NONYMOV6E6K2D2QNQ, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.830354733 (292h2m18.381825194s) [noise=true , address=77.90.31.13:4627 , piece_id=5W5AXIKSPSX6MGGV25AQZMKXOEUOUDT6LJ3ALXWTUXGGJJGPHMGQ, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.893988254 (292h2m18.445458715s) [noise=true , address=77.90.31.10:4518 , piece_id=HEZ2FM5Q6XK6RPTD5TAU3TAHCDJKMGFMZKNOT77G5DWQFA2LLSGQ, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.985985273 (292h2m18.537455734s) [noise=true , address=77.90.31.21:4879 , piece_id=WQJBQIFULX7LGWXDQB3WGPXR5K5LHYLOKIEDXMCYJSYWP6YHXMFA, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.986007055 (292h2m18.537477516s) [seg_pos={0 0}, upload=257194]: Segment reached optimal threshold of 13 pieces. Nov 21 12:06:19.986208177 (292h2m18.537678638s) [noise=true , address=45.91.116.5:4298 , piece_id=BDMI2ONMMXWTHZBQDOD4CJGNJZK7OGLB4VRVOZEGCCMD53S3ZVQA, seg_pos={0 0}, upload=257194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: write tcp 188.166.225.170:39292->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:39292->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39292->45.91.116.5:4298: use of closed network connection Nov 21 12:06:19.986215171 (292h2m18.537685632s) [noise=true , address=45.91.116.5:4298 , piece_id=BDMI2ONMMXWTHZBQDOD4CJGNJZK7OGLB4VRVOZEGCCMD53S3ZVQA, seg_pos={0 0}, upload=257194]: Long tail cancellation due to optimal threshold Nov 21 12:06:19.986329428 (292h2m18.537799889s) [noise=true , address=77.90.31.16:4735 , piece_id=YZ333DAUCNN3PG7TRJVU5LNO7AZBBHFMLM6VCNUSQEMDYRVOAKMQ, seg_pos={0 0}, upload=257194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: write tcp 188.166.225.170:36976->77.90.31.16:4735: use of closed network connection; write tcp 188.166.225.170:36976->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36976->77.90.31.16:4735: use of closed network connection Nov 21 12:06:19.986334608 (292h2m18.537805069s) [noise=true , address=77.90.31.16:4735 , piece_id=YZ333DAUCNN3PG7TRJVU5LNO7AZBBHFMLM6VCNUSQEMDYRVOAKMQ, seg_pos={0 0}, upload=257194]: Long tail cancellation due to optimal threshold Nov 21 12:06:19.986374995 (292h2m18.537845466s) [noise=true , address=77.90.31.17:4770 , piece_id=UUU7FZIUIX2IC7F5QIZC5JKA35O2RHZ5I6P4PSMAJCRB2A7OUGLQ, seg_pos={0 0}, upload=257194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: expected piece hash; context canceled; EOF Nov 21 12:06:19.986397208 (292h2m18.537867669s) [noise=true , address=77.90.31.17:4770 , piece_id=UUU7FZIUIX2IC7F5QIZC5JKA35O2RHZ5I6P4PSMAJCRB2A7OUGLQ, seg_pos={0 0}, upload=257194]: Long tail cancellation due to optimal threshold Nov 21 12:06:19.986466850 (292h2m18.537937311s) [seg_pos={0 0}, upload=257194]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 21 12:06:19.986571549 (292h2m18.538042011s) [upload=257194]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 21 12:06:19.986580406 (292h2m18.538050867s) [upload=257194]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 21 12:06:20.274288336 (292h2m18.825758797s) [upload=257194]: Done uploading segments... Nov 21 12:06:20.274311019 (292h2m18.82578147s) [upload=257194]: Upload finished. err: <nil> Duration: 5.771741825s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 21 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXX....................................X...] duration: 4.312614773s trace: https://jaeger.dev.storj.io/trace/30ef13d1b320910a FAIL (2025-11-21 12:07:41 UTC) orbiter warning: context deadline exceeded Duration: 4.312642886s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [X....X............X....X.X.X..X.X..X...X..XXX.X.XX...X...XX.] duration: 6.386728466s trace: https://jaeger.dev.storj.io/trace/a61d21c4ccf9666 FAIL (2025-11-21 12:07:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: 5W5AXIKSPSX6MGGV25AQZMKXOEUOUDT6LJ3ALXWTUXGGJJGPHMGQ) context deadline exceeded; piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: HEZ2FM5Q6XK6RPTD5TAU3TAHCDJKMGFMZKNOT77G5DWQFA2LLSGQ) context deadline exceeded ======LOGS====== (292h2m15.53524253s) [upload=257194]: Uploading segments... Nov 21 12:06:16.984911178 (292h2m15.536381649s) [upload=257194]: Got next segment. Inline: false Nov 21 12:06:16.984990128 (292h2m15.536460599s) [upload=257194]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 12:06:16.984998183 (292h2m15.536468644s) [upload=257194]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 12:06:17.228288865 (292h2m15.779759326s) [seg_pos={0 0}, upload=257194]: Begin upload segment... Nov 21 12:06:17.228357676 (292h2m15.779828137s) [seg_pos={0 0}, upload=257194]: Done begin upload segment. Nov 21 12:06:17.228448919 (292h2m15.77991938s) [noise=true , address=145.79.227.24:4583 , piece_id=P4YAEWAL5QJAEZ7QAILYI5FZFZBYLKADVQLMSVMYCAKZNLSVEMFQ, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.228492993 (292h2m15.779963454s) [noise=true , address=77.90.31.16:4735 , piece_id=YZ333DAUCNN3PG7TRJVU5LNO7AZBBHFMLM6VCNUSQEMDYRVOAKMQ, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.229164662 (292h2m15.780635123s) [noise=true , address=145.79.227.27:4690 , piece_id=GIYOKO5RULD7KKZXIA542RZO2O7WUP4GFU556MGRCST356HARMEA, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.229260765 (292h2m15.780731227s) [noise=true , address=145.79.227.30:4761 , piece_id=FNEHOUE5BJQTQILNEVTTZCHYDWCQGWEHIX6DX5RDQWYGCSYQKE7Q, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.230231854 (292h2m15.781702315s) [noise=true , address=77.90.31.13:4627 , piece_id=5W5AXIKSPSX6MGGV25AQZMKXOEUOUDT6LJ3ALXWTUXGGJJGPHMGQ, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.230413711 (292h2m15.781884172s) [noise=true , address=45.91.116.5:4298 , piece_id=BDMI2ONMMXWTHZBQDOD4CJGNJZK7OGLB4VRVOZEGCCMD53S3ZVQA, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.231487476 (292h2m15.782957927s) [noise=true , address=145.79.227.25:4619 , piece_id=5LQLNFMJSFGJ4BVZMPSBPSOWQKNH7NYIGYFKXQX2JH4ITV2OOQ4Q, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.231583307 (292h2m15.783053758s) [noise=true , address=145.79.227.30:4763 , piece_id=TPJMEV5GZFAC2UZESKDGVNWZ5GCMVMCQWLOSNUJC6PNXUEXEG4VA, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.231654042 (292h2m15.783124503s) [noise=true , address=77.90.31.17:4770 , piece_id=UUU7FZIUIX2IC7F5QIZC5JKA35O2RHZ5I6P4PSMAJCRB2A7OUGLQ, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.231699840 (292h2m15.783170301s) [noise=true , address=145.79.227.26:4654 , piece_id=EFU4C5HZSEVQXOTMLH62VRMI6O22HKUFXMKLODY2CJQKODI7ZDCQ, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.232691227 (292h2m15.784161688s) [noise=true , address=77.90.31.21:4879 , piece_id=WQJBQIFULX7LGWXDQB3WGPXR5K5LHYLOKIEDXMCYJSYWP6YHXMFA, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.232924111 (292h2m15.784394572s) [noise=true , address=77.90.31.10:4518 , piece_id=HEZ2FM5Q6XK6RPTD5TAU3TAHCDJKMGFMZKNOT77G5DWQFA2LLSGQ, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.234049513 (292h2m15.785520005s) [noise=true , address=145.79.227.23:4547 , piece_id=EWWBDUM52GIS5KX3FKYR5C7EXNDSDIICJ47NONYMOV6E6K2D2QNQ, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.235293583 (292h2m15.786764044s) [noise=true , address=145.79.227.16:4912 , piece_id=5VSFQ3LC3G4Y2VZB6RNGSWFKEBJ4ZHEKJI3MN6AZ4RQ5GIEC6TTQ, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.235803664 (292h2m15.787274125s) [noise=true , address=145.79.227.28:4725 , piece_id=V4SLAUQ244S53J6QLWM32NO2DQ2DS6NPDFZPZMX3CVY5DWM3KEMA, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.238435937 (292h2m15.789906398s) [noise=true , address=145.79.227.21:4473 , piece_id=BFGHAWIA7ERR5BI5I6M3YCQPQC7M6TOBEART2WCE6Q3RUVYP222A, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:18.900466542 (292h2m17.451937033s) [upload=257194]: Next returned nil segment Nov 21 12:06:18.900488314 (292h2m17.451958785s) [upload=257194]: Waiting for error group managing segments... Nov 21 12:06:19.223504608 (292h2m17.774975069s) [noise=true , address=145.79.227.26:4654 , piece_id=EFU4C5HZSEVQXOTMLH62VRMI6O22HKUFXMKLODY2CJQKODI7ZDCQ, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.224063582 (292h2m17.775534043s) [noise=true , address=145.79.227.30:4761 , piece_id=FNEHOUE5BJQTQILNEVTTZCHYDWCQGWEHIX6DX5RDQWYGCSYQKE7Q, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.224813419 (292h2m17.77628388s) [noise=true , address=145.79.227.27:4690 , piece_id=GIYOKO5RULD7KKZXIA542RZO2O7WUP4GFU556MGRCST356HARMEA, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.234614373 (292h2m17.786084824s) [noise=true , address=145.79.227.16:4912 , piece_id=5VSFQ3LC3G4Y2VZB6RNGSWFKEBJ4ZHEKJI3MN6AZ4RQ5GIEC6TTQ, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.235452939 (292h2m17.7869234s) [noise=true , address=145.79.227.21:4473 , piece_id=BFGHAWIA7ERR5BI5I6M3YCQPQC7M6TOBEART2WCE6Q3RUVYP222A, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.238949498 (292h2m17.790419959s) [noise=true , address=145.79.227.28:4725 , piece_id=V4SLAUQ244S53J6QLWM32NO2DQ2DS6NPDFZPZMX3CVY5DWM3KEMA, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.248638006 (292h2m17.800108467s) [noise=true , address=145.79.227.25:4619 , piece_id=5LQLNFMJSFGJ4BVZMPSBPSOWQKNH7NYIGYFKXQX2JH4ITV2OOQ4Q, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.249046234 (292h2m17.800516695s) [noise=true , address=145.79.227.30:4763 , piece_id=TPJMEV5GZFAC2UZESKDGVNWZ5GCMVMCQWLOSNUJC6PNXUEXEG4VA, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.249660604 (292h2m17.801131055s) [noise=true , address=145.79.227.24:4583 , piece_id=P4YAEWAL5QJAEZ7QAILYI5FZFZBYLKADVQLMSVMYCAKZNLSVEMFQ, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.250828577 (292h2m17.802299038s) [noise=true , address=145.79.227.23:4547 , piece_id=EWWBDUM52GIS5KX3FKYR5C7EXNDSDIICJ47NONYMOV6E6K2D2QNQ, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.830354733 (292h2m18.381825194s) [noise=true , address=77.90.31.13:4627 , piece_id=5W5AXIKSPSX6MGGV25AQZMKXOEUOUDT6LJ3ALXWTUXGGJJGPHMGQ, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.893988254 (292h2m18.445458715s) [noise=true , address=77.90.31.10:4518 , piece_id=HEZ2FM5Q6XK6RPTD5TAU3TAHCDJKMGFMZKNOT77G5DWQFA2LLSGQ, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.985985273 (292h2m18.537455734s) [noise=true , address=77.90.31.21:4879 , piece_id=WQJBQIFULX7LGWXDQB3WGPXR5K5LHYLOKIEDXMCYJSYWP6YHXMFA, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.986007055 (292h2m18.537477516s) [seg_pos={0 0}, upload=257194]: Segment reached optimal threshold of 13 pieces. Nov 21 12:06:19.986208177 (292h2m18.537678638s) [noise=true , address=45.91.116.5:4298 , piece_id=BDMI2ONMMXWTHZBQDOD4CJGNJZK7OGLB4VRVOZEGCCMD53S3ZVQA, seg_pos={0 0}, upload=257194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: write tcp 188.166.225.170:39292->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:39292->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39292->45.91.116.5:4298: use of closed network connection Nov 21 12:06:19.986215171 (292h2m18.537685632s) [noise=true , address=45.91.116.5:4298 , piece_id=BDMI2ONMMXWTHZBQDOD4CJGNJZK7OGLB4VRVOZEGCCMD53S3ZVQA, seg_pos={0 0}, upload=257194]: Long tail cancellation due to optimal threshold Nov 21 12:06:19.986329428 (292h2m18.537799889s) [noise=true , address=77.90.31.16:4735 , piece_id=YZ333DAUCNN3PG7TRJVU5LNO7AZBBHFMLM6VCNUSQEMDYRVOAKMQ, seg_pos={0 0}, upload=257194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: write tcp 188.166.225.170:36976->77.90.31.16:4735: use of closed network connection; write tcp 188.166.225.170:36976->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36976->77.90.31.16:4735: use of closed network connection Nov 21 12:06:19.986334608 (292h2m18.537805069s) [noise=true , address=77.90.31.16:4735 , piece_id=YZ333DAUCNN3PG7TRJVU5LNO7AZBBHFMLM6VCNUSQEMDYRVOAKMQ, seg_pos={0 0}, upload=257194]: Long tail cancellation due to optimal threshold Nov 21 12:06:19.986374995 (292h2m18.537845466s) [noise=true , address=77.90.31.17:4770 , piece_id=UUU7FZIUIX2IC7F5QIZC5JKA35O2RHZ5I6P4PSMAJCRB2A7OUGLQ, seg_pos={0 0}, upload=257194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: expected piece hash; context canceled; EOF Nov 21 12:06:19.986397208 (292h2m18.537867669s) [noise=true , address=77.90.31.17:4770 , piece_id=UUU7FZIUIX2IC7F5QIZC5JKA35O2RHZ5I6P4PSMAJCRB2A7OUGLQ, seg_pos={0 0}, upload=257194]: Long tail cancellation due to optimal threshold Nov 21 12:06:19.986466850 (292h2m18.537937311s) [seg_pos={0 0}, upload=257194]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 21 12:06:19.986571549 (292h2m18.538042011s) [upload=257194]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 21 12:06:19.986580406 (292h2m18.538050867s) [upload=257194]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 21 12:06:20.274288336 (292h2m18.825758797s) [upload=257194]: Done uploading segments... Nov 21 12:06:20.274311019 (292h2m18.82578147s) [upload=257194]: Upload finished. err: <nil> Duration: 6.38675706s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 22 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXX....................................X..] duration: 3.519007989s trace: https://jaeger.dev.storj.io/trace/30ef13d1b320910a FAIL (2025-11-21 12:07:09 UTC) orbiter warning: context deadline exceeded Duration: 3.519034249s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [.X....X............X....X.X.X..X.X..X...X..XXX.X.XX...X...XX] duration: 30.003055085s trace: https://jaeger.dev.storj.io/trace/a61d21c4ccf9666 FAIL (2025-11-21 12:06:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: 5W5AXIKSPSX6MGGV25AQZMKXOEUOUDT6LJ3ALXWTUXGGJJGPHMGQ) context deadline exceeded; piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: HEZ2FM5Q6XK6RPTD5TAU3TAHCDJKMGFMZKNOT77G5DWQFA2LLSGQ) context deadline exceeded ======LOGS====== (292h2m15.53524253s) [upload=257194]: Uploading segments... Nov 21 12:06:16.984911178 (292h2m15.536381649s) [upload=257194]: Got next segment. Inline: false Nov 21 12:06:16.984990128 (292h2m15.536460599s) [upload=257194]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 12:06:16.984998183 (292h2m15.536468644s) [upload=257194]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 12:06:17.228288865 (292h2m15.779759326s) [seg_pos={0 0}, upload=257194]: Begin upload segment... Nov 21 12:06:17.228357676 (292h2m15.779828137s) [seg_pos={0 0}, upload=257194]: Done begin upload segment. Nov 21 12:06:17.228448919 (292h2m15.77991938s) [noise=true , address=145.79.227.24:4583 , piece_id=P4YAEWAL5QJAEZ7QAILYI5FZFZBYLKADVQLMSVMYCAKZNLSVEMFQ, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.228492993 (292h2m15.779963454s) [noise=true , address=77.90.31.16:4735 , piece_id=YZ333DAUCNN3PG7TRJVU5LNO7AZBBHFMLM6VCNUSQEMDYRVOAKMQ, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.229164662 (292h2m15.780635123s) [noise=true , address=145.79.227.27:4690 , piece_id=GIYOKO5RULD7KKZXIA542RZO2O7WUP4GFU556MGRCST356HARMEA, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.229260765 (292h2m15.780731227s) [noise=true , address=145.79.227.30:4761 , piece_id=FNEHOUE5BJQTQILNEVTTZCHYDWCQGWEHIX6DX5RDQWYGCSYQKE7Q, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.230231854 (292h2m15.781702315s) [noise=true , address=77.90.31.13:4627 , piece_id=5W5AXIKSPSX6MGGV25AQZMKXOEUOUDT6LJ3ALXWTUXGGJJGPHMGQ, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.230413711 (292h2m15.781884172s) [noise=true , address=45.91.116.5:4298 , piece_id=BDMI2ONMMXWTHZBQDOD4CJGNJZK7OGLB4VRVOZEGCCMD53S3ZVQA, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.231487476 (292h2m15.782957927s) [noise=true , address=145.79.227.25:4619 , piece_id=5LQLNFMJSFGJ4BVZMPSBPSOWQKNH7NYIGYFKXQX2JH4ITV2OOQ4Q, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.231583307 (292h2m15.783053758s) [noise=true , address=145.79.227.30:4763 , piece_id=TPJMEV5GZFAC2UZESKDGVNWZ5GCMVMCQWLOSNUJC6PNXUEXEG4VA, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.231654042 (292h2m15.783124503s) [noise=true , address=77.90.31.17:4770 , piece_id=UUU7FZIUIX2IC7F5QIZC5JKA35O2RHZ5I6P4PSMAJCRB2A7OUGLQ, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.231699840 (292h2m15.783170301s) [noise=true , address=145.79.227.26:4654 , piece_id=EFU4C5HZSEVQXOTMLH62VRMI6O22HKUFXMKLODY2CJQKODI7ZDCQ, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.232691227 (292h2m15.784161688s) [noise=true , address=77.90.31.21:4879 , piece_id=WQJBQIFULX7LGWXDQB3WGPXR5K5LHYLOKIEDXMCYJSYWP6YHXMFA, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.232924111 (292h2m15.784394572s) [noise=true , address=77.90.31.10:4518 , piece_id=HEZ2FM5Q6XK6RPTD5TAU3TAHCDJKMGFMZKNOT77G5DWQFA2LLSGQ, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.234049513 (292h2m15.785520005s) [noise=true , address=145.79.227.23:4547 , piece_id=EWWBDUM52GIS5KX3FKYR5C7EXNDSDIICJ47NONYMOV6E6K2D2QNQ, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.235293583 (292h2m15.786764044s) [noise=true , address=145.79.227.16:4912 , piece_id=5VSFQ3LC3G4Y2VZB6RNGSWFKEBJ4ZHEKJI3MN6AZ4RQ5GIEC6TTQ, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.235803664 (292h2m15.787274125s) [noise=true , address=145.79.227.28:4725 , piece_id=V4SLAUQ244S53J6QLWM32NO2DQ2DS6NPDFZPZMX3CVY5DWM3KEMA, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:17.238435937 (292h2m15.789906398s) [noise=true , address=145.79.227.21:4473 , piece_id=BFGHAWIA7ERR5BI5I6M3YCQPQC7M6TOBEART2WCE6Q3RUVYP222A, seg_pos={0 0}, upload=257194]: Uploading piece... Nov 21 12:06:18.900466542 (292h2m17.451937033s) [upload=257194]: Next returned nil segment Nov 21 12:06:18.900488314 (292h2m17.451958785s) [upload=257194]: Waiting for error group managing segments... Nov 21 12:06:19.223504608 (292h2m17.774975069s) [noise=true , address=145.79.227.26:4654 , piece_id=EFU4C5HZSEVQXOTMLH62VRMI6O22HKUFXMKLODY2CJQKODI7ZDCQ, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.224063582 (292h2m17.775534043s) [noise=true , address=145.79.227.30:4761 , piece_id=FNEHOUE5BJQTQILNEVTTZCHYDWCQGWEHIX6DX5RDQWYGCSYQKE7Q, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.224813419 (292h2m17.77628388s) [noise=true , address=145.79.227.27:4690 , piece_id=GIYOKO5RULD7KKZXIA542RZO2O7WUP4GFU556MGRCST356HARMEA, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.234614373 (292h2m17.786084824s) [noise=true , address=145.79.227.16:4912 , piece_id=5VSFQ3LC3G4Y2VZB6RNGSWFKEBJ4ZHEKJI3MN6AZ4RQ5GIEC6TTQ, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.235452939 (292h2m17.7869234s) [noise=true , address=145.79.227.21:4473 , piece_id=BFGHAWIA7ERR5BI5I6M3YCQPQC7M6TOBEART2WCE6Q3RUVYP222A, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.238949498 (292h2m17.790419959s) [noise=true , address=145.79.227.28:4725 , piece_id=V4SLAUQ244S53J6QLWM32NO2DQ2DS6NPDFZPZMX3CVY5DWM3KEMA, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.248638006 (292h2m17.800108467s) [noise=true , address=145.79.227.25:4619 , piece_id=5LQLNFMJSFGJ4BVZMPSBPSOWQKNH7NYIGYFKXQX2JH4ITV2OOQ4Q, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.249046234 (292h2m17.800516695s) [noise=true , address=145.79.227.30:4763 , piece_id=TPJMEV5GZFAC2UZESKDGVNWZ5GCMVMCQWLOSNUJC6PNXUEXEG4VA, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.249660604 (292h2m17.801131055s) [noise=true , address=145.79.227.24:4583 , piece_id=P4YAEWAL5QJAEZ7QAILYI5FZFZBYLKADVQLMSVMYCAKZNLSVEMFQ, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.250828577 (292h2m17.802299038s) [noise=true , address=145.79.227.23:4547 , piece_id=EWWBDUM52GIS5KX3FKYR5C7EXNDSDIICJ47NONYMOV6E6K2D2QNQ, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.830354733 (292h2m18.381825194s) [noise=true , address=77.90.31.13:4627 , piece_id=5W5AXIKSPSX6MGGV25AQZMKXOEUOUDT6LJ3ALXWTUXGGJJGPHMGQ, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.893988254 (292h2m18.445458715s) [noise=true , address=77.90.31.10:4518 , piece_id=HEZ2FM5Q6XK6RPTD5TAU3TAHCDJKMGFMZKNOT77G5DWQFA2LLSGQ, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.985985273 (292h2m18.537455734s) [noise=true , address=77.90.31.21:4879 , piece_id=WQJBQIFULX7LGWXDQB3WGPXR5K5LHYLOKIEDXMCYJSYWP6YHXMFA, seg_pos={0 0}, upload=257194]: Done uploading piece. err: <nil> Nov 21 12:06:19.986007055 (292h2m18.537477516s) [seg_pos={0 0}, upload=257194]: Segment reached optimal threshold of 13 pieces. Nov 21 12:06:19.986208177 (292h2m18.537678638s) [noise=true , address=45.91.116.5:4298 , piece_id=BDMI2ONMMXWTHZBQDOD4CJGNJZK7OGLB4VRVOZEGCCMD53S3ZVQA, seg_pos={0 0}, upload=257194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: write tcp 188.166.225.170:39292->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:39292->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39292->45.91.116.5:4298: use of closed network connection Nov 21 12:06:19.986215171 (292h2m18.537685632s) [noise=true , address=45.91.116.5:4298 , piece_id=BDMI2ONMMXWTHZBQDOD4CJGNJZK7OGLB4VRVOZEGCCMD53S3ZVQA, seg_pos={0 0}, upload=257194]: Long tail cancellation due to optimal threshold Nov 21 12:06:19.986329428 (292h2m18.537799889s) [noise=true , address=77.90.31.16:4735 , piece_id=YZ333DAUCNN3PG7TRJVU5LNO7AZBBHFMLM6VCNUSQEMDYRVOAKMQ, seg_pos={0 0}, upload=257194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: write tcp 188.166.225.170:36976->77.90.31.16:4735: use of closed network connection; write tcp 188.166.225.170:36976->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36976->77.90.31.16:4735: use of closed network connection Nov 21 12:06:19.986334608 (292h2m18.537805069s) [noise=true , address=77.90.31.16:4735 , piece_id=YZ333DAUCNN3PG7TRJVU5LNO7AZBBHFMLM6VCNUSQEMDYRVOAKMQ, seg_pos={0 0}, upload=257194]: Long tail cancellation due to optimal threshold Nov 21 12:06:19.986374995 (292h2m18.537845466s) [noise=true , address=77.90.31.17:4770 , piece_id=UUU7FZIUIX2IC7F5QIZC5JKA35O2RHZ5I6P4PSMAJCRB2A7OUGLQ, seg_pos={0 0}, upload=257194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: expected piece hash; context canceled; EOF Nov 21 12:06:19.986397208 (292h2m18.537867669s) [noise=true , address=77.90.31.17:4770 , piece_id=UUU7FZIUIX2IC7F5QIZC5JKA35O2RHZ5I6P4PSMAJCRB2A7OUGLQ, seg_pos={0 0}, upload=257194]: Long tail cancellation due to optimal threshold Nov 21 12:06:19.986466850 (292h2m18.537937311s) [seg_pos={0 0}, upload=257194]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 21 12:06:19.986571549 (292h2m18.538042011s) [upload=257194]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 21 12:06:19.986580406 (292h2m18.538050867s) [upload=257194]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 21 12:06:20.274288336 (292h2m18.825758797s) [upload=257194]: Done uploading segments... Nov 21 12:06:20.274311019 (292h2m18.82578147s) [upload=257194]: Upload finished. err: <nil> Duration: 30.003069501s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 23 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXX....................................X.] duration: 3.665817793s trace: https://jaeger.dev.storj.io/trace/30ef13d1b320910a FAIL (2025-11-21 12:06:30 UTC) orbiter warning: context deadline exceeded Duration: 3.66584792s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [X.X....X............X....X.X.X..X.X..X...X..XXX.X.XX...X...X] duration: 30.004432449s trace: https://jaeger.dev.storj.io/trace/79f38c169f8d1093 FAIL (2025-11-21 12:06:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: 3FRD5DEQM64AD6MCXPZU4DEBDXQQIYXXSUDBZOLRJNXRF2JUDLZA) context deadline exceeded; piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: XPEYRMNEX5LOBKBE4WFO53FBP6D5AGCH3ZPVGUW5ZSNITPZCBUBA) context deadline exceeded ======LOGS====== Nov 21 12:05:46.752156381 (292h1m45.303626852s) [upload=257188]: Starting upload Nov 21 12:05:46.752350189 (292h1m45.30382065s) [upload=257188]: Done starting upload Nov 21 12:05:46.752490426 (292h1m45.303960887s) [upload=257188]: Uploading segments... Nov 21 12:05:46.752645121 (292h1m45.304115582s) [upload=257188]: Got next segment. Inline: false Nov 21 12:05:46.752699144 (292h1m45.304169605s) [upload=257188]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 12:05:46.752706789 (292h1m45.30417725s) [upload=257188]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 12:05:46.996725466 (292h1m45.548195937s) [seg_pos={0 0}, upload=257188]: Begin upload segment... Nov 21 12:05:46.996783206 (292h1m45.548253667s) [seg_pos={0 0}, upload=257188]: Done begin upload segment. Nov 21 12:05:46.996822601 (292h1m45.548293062s) [noise=true , address=45.91.116.11:4119 , piece_id=3FRD5DEQM64AD6MCXPZU4DEBDXQQIYXXSUDBZOLRJNXRF2JUDLZA, seg_pos={0 0}, upload=257188]: Uploading piece... Nov 21 12:05:46.996841827 (292h1m45.548312288s) [noise=true , address=145.79.227.26:4654 , piece_id=77OFOBNOZ4B5C2WZAIO5JHDHNDMJTOTLFKEZ26EMEOB4AS4D6XXA, seg_pos={0 0}, upload=257188]: Uploading piece... Nov 21 12:05:46.996907933 (292h1m45.548378394s) [noise=true , address=145.79.227.23:4546 , piece_id=TBF6DVZX35H66MECGKWUMA7ZGZQ766TAH7AG2NMVUM655N6XJZYQ, seg_pos={0 0}, upload=257188]: Uploading piece... Nov 21 12:05:46.997543654 (292h1m45.549014115s) [noise=true , address=145.79.227.22:4511 , piece_id=UYT5EOTRDBBLWC6PMFYDNAEI7SI6T3BDKSQ2MS75YG6V2ESWCTFQ, seg_pos={0 0}, upload=257188]: Uploading piece... Nov 21 12:05:46.998115503 (292h1m45.549585954s) [noise=true , address=77.90.31.10:4519 , piece_id=O6QXAPD4HLZT7KCJAYSXCGMGYKP5BH6KKCV3DY3X3BIUBHEFGCKQ, seg_pos={0 0}, upload=257188]: Uploading piece... Nov 21 12:05:46.998554960 (292h1m45.550025411s) [noise=true , address=145.79.227.14:4838 , piece_id=AV4MEOSDF6DEMU3YR2HZJUZYDQKOMOX3OWMSAYIPLTSBBITFB66Q, seg_pos={0 0}, upload=257188]: Uploading piece... Nov 21 12:05:46.998996039 (292h1m45.5504665s) [noise=true , address=145.79.227.30:4763 , piece_id=PLJR75AH3XPEHXMJX3HHELNMORZK4U3GMMA2DAELCECKET3QHAEA, seg_pos={0 0}, upload=257188]: Uploading piece... Nov 21 12:05:46.999404968 (292h1m45.550875429s) [noise=true , address=77.90.31.13:4627 , piece_id=XPEYRMNEX5LOBKBE4WFO53FBP6D5AGCH3ZPVGUW5ZSNITPZCBUBA, seg_pos={0 0}, upload=257188]: Uploading piece... Nov 21 12:05:46.999868020 (292h1m45.551338481s) [noise=true , address=45.91.116.7:4010 , piece_id=GJOREZHUYLBLECQJWYWSTNPXVMOLAHMRV7TMV7ARJOBX5SX4UKNQ, seg_pos={0 0}, upload=257188]: Uploading piece... Nov 21 12:05:47.001571093 (292h1m45.553041554s) [noise=true , address=77.90.31.21:4879 , piece_id=TAAA7DML4UBI5LH7HFVJFL2HRR4FVLDINMHBLJE272SKXJQJSGKQ, seg_pos={0 0}, upload=257188]: Uploading piece... Nov 21 12:05:47.003257915 (292h1m45.554728396s) [noise=true , address=45.91.116.4:4937 , piece_id=O3PR6L75HPE7XRXCBVKQCTXKKJGI34SL4DZAE4TXFQMRCTDYA36Q, seg_pos={0 0}, upload=257188]: Uploading piece... Nov 21 12:05:47.004934948 (292h1m45.556405419s) [noise=true , address=77.90.31.16:4735 , piece_id=XKFSUSV7CBHYIE2SBYZQWL4KWRTBJJVSEI3NYA5S6OY3UFOM6KOA, seg_pos={0 0}, upload=257188]: Uploading piece... Nov 21 12:05:47.006320106 (292h1m45.557790567s) [noise=true , address=145.79.227.21:4475 , piece_id=4P7UZTJTR6KGL5QLSCT3RWWZWYXFJCE2GHDABNT2HA7GHTAZL3FQ, seg_pos={0 0}, upload=257188]: Uploading piece... Nov 21 12:05:47.007762442 (292h1m45.559232903s) [noise=true , address=145.79.227.28:4727 , piece_id=LFJNMNDDCYH4EDDUCOFKEKW5K44KR36VOULDBO4UCZUKHW5FVZNA, seg_pos={0 0}, upload=257188]: Uploading piece... Nov 21 12:05:47.009144724 (292h1m45.560615186s) [noise=true , address=145.79.227.11:4731 , piece_id=GE5KFHPINZGYLG4HBZTLVHYOT4YRQIPL6IIOJY2XCJGFJ2WOZ7GA, seg_pos={0 0}, upload=257188]: Uploading piece... Nov 21 12:05:47.010457394 (292h1m45.561927855s) [noise=true , address=145.79.227.31:4798 , piece_id=5GYDGHE7FGTU4JMZOR36MBAMUVM2TA237J7V4KCEE6IHKWHUAVSQ, seg_pos={0 0}, upload=257188]: Uploading piece... Nov 21 12:05:48.607590393 (292h1m47.159060844s) [upload=257188]: Next returned nil segment Nov 21 12:05:48.607628535 (292h1m47.159098996s) [upload=257188]: Waiting for error group managing segments... Nov 21 12:05:48.989766426 (292h1m47.541236887s) [noise=true , address=145.79.227.30:4763 , piece_id=PLJR75AH3XPEHXMJX3HHELNMORZK4U3GMMA2DAELCECKET3QHAEA, seg_pos={0 0}, upload=257188]: Done uploading piece. err: <nil> Nov 21 12:05:48.990752234 (292h1m47.542222685s) [noise=true , address=145.79.227.22:4511 , piece_id=UYT5EOTRDBBLWC6PMFYDNAEI7SI6T3BDKSQ2MS75YG6V2ESWCTFQ, seg_pos={0 0}, upload=257188]: Done uploading piece. err: <nil> Nov 21 12:05:48.994693659 (292h1m47.54616412s) [noise=true , address=145.79.227.26:4654 , piece_id=77OFOBNOZ4B5C2WZAIO5JHDHNDMJTOTLFKEZ26EMEOB4AS4D6XXA, seg_pos={0 0}, upload=257188]: Done uploading piece. err: <nil> Nov 21 12:05:48.996162657 (292h1m47.547633118s) [noise=true , address=145.79.227.28:4727 , piece_id=LFJNMNDDCYH4EDDUCOFKEKW5K44KR36VOULDBO4UCZUKHW5FVZNA, seg_pos={0 0}, upload=257188]: Done uploading piece. err: <nil> Nov 21 12:05:48.997148564 (292h1m47.548619025s) [noise=true , address=145.79.227.21:4475 , piece_id=4P7UZTJTR6KGL5QLSCT3RWWZWYXFJCE2GHDABNT2HA7GHTAZL3FQ, seg_pos={0 0}, upload=257188]: Done uploading piece. err: <nil> Nov 21 12:05:49.005739312 (292h1m47.557245892s) [noise=true , address=145.79.227.14:4838 , piece_id=AV4MEOSDF6DEMU3YR2HZJUZYDQKOMOX3OWMSAYIPLTSBBITFB66Q, seg_pos={0 0}, upload=257188]: Done uploading piece. err: <nil> Nov 21 12:05:49.018990575 (292h1m47.570461036s) [noise=true , address=145.79.227.11:4731 , piece_id=GE5KFHPINZGYLG4HBZTLVHYOT4YRQIPL6IIOJY2XCJGFJ2WOZ7GA, seg_pos={0 0}, upload=257188]: Done uploading piece. err: <nil> Nov 21 12:05:49.020376043 (292h1m47.571846504s) [noise=true , address=145.79.227.23:4546 , piece_id=TBF6DVZX35H66MECGKWUMA7ZGZQ766TAH7AG2NMVUM655N6XJZYQ, seg_pos={0 0}, upload=257188]: Done uploading piece. err: <nil> Nov 21 12:05:49.023588430 (292h1m47.575058891s) [noise=true , address=145.79.227.31:4798 , piece_id=5GYDGHE7FGTU4JMZOR36MBAMUVM2TA237J7V4KCEE6IHKWHUAVSQ, seg_pos={0 0}, upload=257188]: Done uploading piece. err: <nil> Nov 21 12:05:49.620039073 (292h1m48.171509524s) [noise=true , address=45.91.116.7:4010 , piece_id=GJOREZHUYLBLECQJWYWSTNPXVMOLAHMRV7TMV7ARJOBX5SX4UKNQ, seg_pos={0 0}, upload=257188]: Done uploading piece. err: <nil> Nov 21 12:05:49.766608008 (292h1m48.318078469s) [noise=true , address=77.90.31.21:4879 , piece_id=TAAA7DML4UBI5LH7HFVJFL2HRR4FVLDINMHBLJE272SKXJQJSGKQ, seg_pos={0 0}, upload=257188]: Done uploading piece. err: <nil> Nov 21 12:05:50.079331471 (292h1m48.630801923s) [noise=true , address=45.91.116.11:4119 , piece_id=3FRD5DEQM64AD6MCXPZU4DEBDXQQIYXXSUDBZOLRJNXRF2JUDLZA, seg_pos={0 0}, upload=257188]: Done uploading piece. err: <nil> Nov 21 12:05:50.447696187 (292h1m48.999166648s) [noise=true , address=77.90.31.13:4627 , piece_id=XPEYRMNEX5LOBKBE4WFO53FBP6D5AGCH3ZPVGUW5ZSNITPZCBUBA, seg_pos={0 0}, upload=257188]: Done uploading piece. err: <nil> Nov 21 12:05:50.447723859 (292h1m48.99919432s) [seg_pos={0 0}, upload=257188]: Segment reached optimal threshold of 13 pieces. Nov 21 12:05:50.447933539 (292h1m48.999404s) [noise=true , address=77.90.31.10:4519 , piece_id=O6QXAPD4HLZT7KCJAYSXCGMGYKP5BH6KKCV3DY3X3BIUBHEFGCKQ, seg_pos={0 0}, upload=257188]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: write tcp 188.166.225.170:52098->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:52098->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52098->77.90.31.10:4519: use of closed network connection Nov 21 12:05:50.447940642 (292h1m48.999411113s) [noise=true , address=77.90.31.10:4519 , piece_id=O6QXAPD4HLZT7KCJAYSXCGMGYKP5BH6KKCV3DY3X3BIUBHEFGCKQ, seg_pos={0 0}, upload=257188]: Long tail cancellation due to optimal threshold Nov 21 12:05:50.448112389 (292h1m48.99958285s) [noise=true , address=45.91.116.4:4937 , piece_id=O3PR6L75HPE7XRXCBVKQCTXKKJGI34SL4DZAE4TXFQMRCTDYA36Q, seg_pos={0 0}, upload=257188]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Nov 21 12:05:50.448119774 (292h1m48.999590235s) [noise=true , address=45.91.116.4:4937 , piece_id=O3PR6L75HPE7XRXCBVKQCTXKKJGI34SL4DZAE4TXFQMRCTDYA36Q, seg_pos={0 0}, upload=257188]: Long tail cancellation due to optimal threshold Nov 21 12:05:50.448144931 (292h1m48.999615402s) [noise=true , address=77.90.31.16:4735 , piece_id=XKFSUSV7CBHYIE2SBYZQWL4KWRTBJJVSEI3NYA5S6OY3UFOM6KOA, seg_pos={0 0}, upload=257188]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Nov 21 12:05:50.448167184 (292h1m48.999637645s) [noise=true , address=77.90.31.16:4735 , piece_id=XKFSUSV7CBHYIE2SBYZQWL4KWRTBJJVSEI3NYA5S6OY3UFOM6KOA, seg_pos={0 0}, upload=257188]: Long tail cancellation due to optimal threshold Nov 21 12:05:50.448337548 (292h1m48.999807999s) [seg_pos={0 0}, upload=257188]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 21 12:05:50.448474259 (292h1m48.99994472s) [upload=257188]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 21 12:05:50.448490619 (292h1m48.999961091s) [upload=257188]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 21 12:05:50.740980595 (292h1m49.292451066s) [upload=257188]: Done uploading segments... Nov 21 12:05:50.741050117 (292h1m49.292520578s) [upload=257188]: Upload finished. err: <nil> Duration: 30.00445935s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 24 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXX....................................X] duration: 5.000754853s trace: https://jaeger.dev.storj.io/trace/30ef13d1b320910a FAIL (2025-11-21 12:05:46 UTC) orbiter warning: context deadline exceeded Duration: 5.000778427s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [.X.X....X............X....X.X.X..X.X..X...X..XXX.X.XX...X...] duration: 5.834833311s trace: https://jaeger.dev.storj.io/trace/400a3392fd596b9b FAIL (2025-11-21 12:05:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 21 12:03:20.894547296 (291h59m19.446017757s) [upload=257150]: Starting upload Nov 21 12:03:20.894637809 (291h59m19.44610827s) [upload=257150]: Done starting upload Nov 21 12:03:20.894713393 (291h59m19.446183854s) [upload=257150]: Uploading segments... Nov 21 12:03:20.894969650 (291h59m19.446440122s) [upload=257150]: Got next segment. Inline: false Nov 21 12:03:20.895013724 (291h59m19.446484185s) [upload=257150]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 12:03:20.895020538 (291h59m19.446491009s) [upload=257150]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 12:03:21.141706975 (291h59m19.693177446s) [seg_pos={0 0}, upload=257150]: Begin upload segment... Nov 21 12:03:21.141774353 (291h59m19.693244814s) [seg_pos={0 0}, upload=257150]: Done begin upload segment. Nov 21 12:03:21.141865167 (291h59m19.693335638s) [noise=true , address=77.90.31.21:4877 , piece_id=AJIDGRABKTO7SN745P7K73BTXEB72PZ473XBUPDLZVDX4F7KKKNQ, seg_pos={0 0}, upload=257150]: Uploading piece... Nov 21 12:03:21.145928293 (291h59m19.697398754s) [noise=true , address=145.79.227.17:4971 , piece_id=53IMICLBNCSB57326VC5XKP7CG4AQSBKKAFYKBQMELRRSZDKBSQQ, seg_pos={0 0}, upload=257150]: Uploading piece... Nov 21 12:03:21.146959898 (291h59m19.698430359s) [noise=true , address=145.79.227.27:4689 , piece_id=X4QKAUFXGYX54435VLH7QJ2HXSYZEL4SJ6V4VBNTIV77KBUMCZIA, seg_pos={0 0}, upload=257150]: Uploading piece... Nov 21 12:03:21.147497481 (291h59m19.698967942s) [noise=true , address=145.79.227.20:4439 , piece_id=MYM5JPZEKXUBXAND2MHK2ZDHODNYPSM6K6JJRND4IFBOS5I43GDQ, seg_pos={0 0}, upload=257150]: Uploading piece... Nov 21 12:03:21.147962968 (291h59m19.699433429s) [noise=true , address=145.79.227.25:4619 , piece_id=PMHMAYC6QSE3HJLCEAMMBHHFOYOS6XLAMPRM5M33BBIILQX2Y7GA, seg_pos={0 0}, upload=257150]: Uploading piece... Nov 21 12:03:21.151526994 (291h59m19.702997465s) [noise=true , address=145.79.227.22:4511 , piece_id=3HRMW3Y33TRQHQNCD7CFDHYXRJ2WCAGE6XDJFBITER76CCOTIFOQ, seg_pos={0 0}, upload=257150]: Uploading piece... Nov 21 12:03:21.151746542 (291h59m19.703217013s) [noise=true , address=145.79.227.14:4838 , piece_id=6D5V4CF2LSODZ4NPN67KUAJTZMK3ZAYGL546WJ4NIHV5SDLUHIZA, seg_pos={0 0}, upload=257150]: Uploading piece... Nov 21 12:03:21.152451976 (291h59m19.703922437s) [noise=true , address=45.91.116.14:4190 , piece_id=XXL6DRSFSQFP2KCZTKZMS4R24CUY3CYUP7C6B4ZR7UYMHEVVNSFA, seg_pos={0 0}, upload=257150]: Uploading piece... Nov 21 12:03:21.152738210 (291h59m19.704208671s) [noise=true , address=145.79.227.22:4509 , piece_id=5GC5WOJYILYZ2DUIIGLZITCL4XJZ4XWXKCWXZY62C4QW5FYVHELA, seg_pos={0 0}, upload=257150]: Uploading piece... Nov 21 12:03:21.153047469 (291h59m19.70451793s) [noise=true , address=145.79.227.32:4834 , piece_id=WRCEO7VGITAJMKFS7W534X24KCALWKNGWOYP622UOI5GYF4V3OXA, seg_pos={0 0}, upload=257150]: Uploading piece... Nov 21 12:03:21.154222217 (291h59m19.705692678s) [noise=true , address=77.90.31.10:4519 , piece_id=KVHNESUY7ZYPBXUVZQMWJTVUF64ITT57A3AUYZQIP5CPRKIN6RGA, seg_pos={0 0}, upload=257150]: Uploading piece... Nov 21 12:03:21.155401501 (291h59m19.706871962s) [noise=true , address=145.79.227.8:4329 , piece_id=NMX7FTR2Z4EFQLGOND32RE2TI3AIZ7TGGVJTYFDPP3CGFFFJV2LQ, seg_pos={0 0}, upload=257150]: Uploading piece... Nov 21 12:03:21.157354521 (291h59m19.708824972s) [noise=true , address=45.91.116.10:4082 , piece_id=PUYEZFZQLZCRUATWBPEP7VYU75XD4DDIABD6GBBASF4RKMKTG2FA, seg_pos={0 0}, upload=257150]: Uploading piece... Nov 21 12:03:21.157746998 (291h59m19.709217469s) [noise=true , address=145.79.227.11:4729 , piece_id=GH57FJM7JI7WN3ICOX2BCK4JQQDYORJ3JX2B7V4LDG6KALWOHX7Q, seg_pos={0 0}, upload=257150]: Uploading piece... Nov 21 12:03:21.163268371 (291h59m19.714738832s) [noise=true , address=145.79.227.16:4912 , piece_id=QCAG67ERHROPSHVUVCKEZJFK3B6K4YI7LRZWARMJCT5QXTSAHL2A, seg_pos={0 0}, upload=257150]: Uploading piece... Nov 21 12:03:21.165309478 (291h59m19.716779939s) [noise=true , address=77.90.31.18:4805 , piece_id=6LWJPVVTDVATLDHL66WS6H6KMLXL4ZRXVRZFXP2QCGYFUFRV3C4A, seg_pos={0 0}, upload=257150]: Uploading piece... Nov 21 12:03:22.765252658 (291h59m21.316723119s) [upload=257150]: Next returned nil segment Nov 21 12:03:22.765268569 (291h59m21.31673903s) [upload=257150]: Waiting for error group managing segments... Nov 21 12:03:23.146214842 (291h59m21.697685313s) [noise=true , address=145.79.227.22:4511 , piece_id=3HRMW3Y33TRQHQNCD7CFDHYXRJ2WCAGE6XDJFBITER76CCOTIFOQ, seg_pos={0 0}, upload=257150]: Done uploading piece. err: <nil> Nov 21 12:03:23.148157711 (291h59m21.699628172s) [noise=true , address=145.79.227.14:4838 , piece_id=6D5V4CF2LSODZ4NPN67KUAJTZMK3ZAYGL546WJ4NIHV5SDLUHIZA, seg_pos={0 0}, upload=257150]: Done uploading piece. err: <nil> Nov 21 12:03:23.151571021 (291h59m21.703041482s) [noise=true , address=145.79.227.25:4619 , piece_id=PMHMAYC6QSE3HJLCEAMMBHHFOYOS6XLAMPRM5M33BBIILQX2Y7GA, seg_pos={0 0}, upload=257150]: Done uploading piece. err: <nil> Nov 21 12:03:23.152335496 (291h59m21.703805977s) [noise=true , address=145.79.227.11:4729 , piece_id=GH57FJM7JI7WN3ICOX2BCK4JQQDYORJ3JX2B7V4LDG6KALWOHX7Q, seg_pos={0 0}, upload=257150]: Done uploading piece. err: <nil> Nov 21 12:03:23.153157361 (291h59m21.704627822s) [noise=true , address=145.79.227.17:4971 , piece_id=53IMICLBNCSB57326VC5XKP7CG4AQSBKKAFYKBQMELRRSZDKBSQQ, seg_pos={0 0}, upload=257150]: Done uploading piece. err: <nil> Nov 21 12:03:23.154678468 (291h59m21.706148949s) [noise=true , address=145.79.227.20:4439 , piece_id=MYM5JPZEKXUBXAND2MHK2ZDHODNYPSM6K6JJRND4IFBOS5I43GDQ, seg_pos={0 0}, upload=257150]: Done uploading piece. err: <nil> Nov 21 12:03:23.155458653 (291h59m21.706929114s) [noise=true , address=145.79.227.32:4834 , piece_id=WRCEO7VGITAJMKFS7W534X24KCALWKNGWOYP622UOI5GYF4V3OXA, seg_pos={0 0}, upload=257150]: Done uploading piece. err: <nil> Nov 21 12:03:23.166587054 (291h59m21.718057515s) [noise=true , address=145.79.227.22:4509 , piece_id=5GC5WOJYILYZ2DUIIGLZITCL4XJZ4XWXKCWXZY62C4QW5FYVHELA, seg_pos={0 0}, upload=257150]: Done uploading piece. err: <nil> Nov 21 12:03:23.167943206 (291h59m21.719413667s) [noise=true , address=145.79.227.16:4912 , piece_id=QCAG67ERHROPSHVUVCKEZJFK3B6K4YI7LRZWARMJCT5QXTSAHL2A, seg_pos={0 0}, upload=257150]: Done uploading piece. err: <nil> Nov 21 12:03:23.172565528 (291h59m21.724035989s) [noise=true , address=145.79.227.27:4689 , piece_id=X4QKAUFXGYX54435VLH7QJ2HXSYZEL4SJ6V4VBNTIV77KBUMCZIA, seg_pos={0 0}, upload=257150]: Done uploading piece. err: <nil> Nov 21 12:03:23.191491918 (291h59m21.742962379s) [noise=true , address=145.79.227.8:4329 , piece_id=NMX7FTR2Z4EFQLGOND32RE2TI3AIZ7TGGVJTYFDPP3CGFFFJV2LQ, seg_pos={0 0}, upload=257150]: Done uploading piece. err: <nil> Nov 21 12:03:23.769501866 (291h59m22.320972327s) [noise=true , address=45.91.116.10:4082 , piece_id=PUYEZFZQLZCRUATWBPEP7VYU75XD4DDIABD6GBBASF4RKMKTG2FA, seg_pos={0 0}, upload=257150]: Done uploading piece. err: <nil> Nov 21 12:03:23.881038302 (291h59m22.432508764s) [noise=true , address=77.90.31.18:4805 , piece_id=6LWJPVVTDVATLDHL66WS6H6KMLXL4ZRXVRZFXP2QCGYFUFRV3C4A, seg_pos={0 0}, upload=257150]: Done uploading piece. err: <nil> Nov 21 12:03:23.881065174 (291h59m22.432535635s) [seg_pos={0 0}, upload=257150]: Segment reached optimal threshold of 13 pieces. Nov 21 12:03:23.881321642 (291h59m22.432792103s) [noise=true , address=77.90.31.10:4519 , piece_id=KVHNESUY7ZYPBXUVZQMWJTVUF64ITT57A3AUYZQIP5CPRKIN6RGA, seg_pos={0 0}, upload=257150]: Done uploading piece. err: <nil> Nov 21 12:03:23.881450838 (291h59m22.432921299s) [noise=true , address=77.90.31.21:4877 , piece_id=AJIDGRABKTO7SN745P7K73BTXEB72PZ473XBUPDLZVDX4F7KKKNQ, seg_pos={0 0}, upload=257150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Nov 21 12:03:23.881472891 (291h59m22.432943362s) [noise=true , address=77.90.31.21:4877 , piece_id=AJIDGRABKTO7SN745P7K73BTXEB72PZ473XBUPDLZVDX4F7KKKNQ, seg_pos={0 0}, upload=257150]: Long tail cancellation due to optimal threshold Nov 21 12:03:23.881998511 (291h59m22.433468972s) [noise=true , address=45.91.116.14:4190 , piece_id=XXL6DRSFSQFP2KCZTKZMS4R24CUY3CYUP7C6B4ZR7UYMHEVVNSFA, seg_pos={0 0}, upload=257150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Nov 21 12:03:23.882010925 (291h59m22.433481386s) [noise=true , address=45.91.116.14:4190 , piece_id=XXL6DRSFSQFP2KCZTKZMS4R24CUY3CYUP7C6B4ZR7UYMHEVVNSFA, seg_pos={0 0}, upload=257150]: Long tail cancellation due to optimal threshold Nov 21 12:03:23.882103892 (291h59m22.433574353s) [seg_pos={0 0}, upload=257150]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Nov 21 12:03:23.882311507 (291h59m22.433781978s) [upload=257150]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 21 12:03:23.882321226 (291h59m22.433791687s) [upload=257150]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 21 12:03:24.186628483 (291h59m22.738098944s) [upload=257150]: Done uploading segments... Nov 21 12:03:24.186656958 (291h59m22.738127429s) [upload=257150]: Upload finished. err: <nil> Duration: 5.834866354s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 24 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXX....................................] duration: 4.269441704s trace: https://jaeger.dev.storj.io/trace/6af15a3a67b655d5 FAIL (2025-11-21 12:05:03 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 4.271675798s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [X.X.X....X............X....X.X.X..X.X..X...X..XXX.X.XX...X..] duration: 5.870608254s trace: https://jaeger.dev.storj.io/trace/400a3392fd596b9b FAIL (2025-11-21 12:04:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 21 12:03:20.894547296 (291h59m19.446017757s) [upload=257150]: Starting upload Nov 21 12:03:20.894637809 (291h59m19.44610827s) [upload=257150]: Done starting upload Nov 21 12:03:20.894713393 (291h59m19.446183854s) [upload=257150]: Uploading segments... Nov 21 12:03:20.894969650 (291h59m19.446440122s) [upload=257150]: Got next segment. Inline: false Nov 21 12:03:20.895013724 (291h59m19.446484185s) [upload=257150]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 12:03:20.895020538 (291h59m19.446491009s) [upload=257150]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 12:03:21.141706975 (291h59m19.693177446s) [seg_pos={0 0}, upload=257150]: Begin upload segment... Nov 21 12:03:21.141774353 (291h59m19.693244814s) [seg_pos={0 0}, upload=257150]: Done begin upload segment. Nov 21 12:03:21.141865167 (291h59m19.693335638s) [noise=true , address=77.90.31.21:4877 , piece_id=AJIDGRABKTO7SN745P7K73BTXEB72PZ473XBUPDLZVDX4F7KKKNQ, seg_pos={0 0}, upload=257150]: Uploading piece... Nov 21 12:03:21.145928293 (291h59m19.697398754s) [noise=true , address=145.79.227.17:4971 , piece_id=53IMICLBNCSB57326VC5XKP7CG4AQSBKKAFYKBQMELRRSZDKBSQQ, seg_pos={0 0}, upload=257150]: Uploading piece... Nov 21 12:03:21.146959898 (291h59m19.698430359s) [noise=true , address=145.79.227.27:4689 , piece_id=X4QKAUFXGYX54435VLH7QJ2HXSYZEL4SJ6V4VBNTIV77KBUMCZIA, seg_pos={0 0}, upload=257150]: Uploading piece... Nov 21 12:03:21.147497481 (291h59m19.698967942s) [noise=true , address=145.79.227.20:4439 , piece_id=MYM5JPZEKXUBXAND2MHK2ZDHODNYPSM6K6JJRND4IFBOS5I43GDQ, seg_pos={0 0}, upload=257150]: Uploading piece... Nov 21 12:03:21.147962968 (291h59m19.699433429s) [noise=true , address=145.79.227.25:4619 , piece_id=PMHMAYC6QSE3HJLCEAMMBHHFOYOS6XLAMPRM5M33BBIILQX2Y7GA, seg_pos={0 0}, upload=257150]: Uploading piece... Nov 21 12:03:21.151526994 (291h59m19.702997465s) [noise=true , address=145.79.227.22:4511 , piece_id=3HRMW3Y33TRQHQNCD7CFDHYXRJ2WCAGE6XDJFBITER76CCOTIFOQ, seg_pos={0 0}, upload=257150]: Uploading piece... Nov 21 12:03:21.151746542 (291h59m19.703217013s) [noise=true , address=145.79.227.14:4838 , piece_id=6D5V4CF2LSODZ4NPN67KUAJTZMK3ZAYGL546WJ4NIHV5SDLUHIZA, seg_pos={0 0}, upload=257150]: Uploading piece... Nov 21 12:03:21.152451976 (291h59m19.703922437s) [noise=true , address=45.91.116.14:4190 , piece_id=XXL6DRSFSQFP2KCZTKZMS4R24CUY3CYUP7C6B4ZR7UYMHEVVNSFA, seg_pos={0 0}, upload=257150]: Uploading piece... Nov 21 12:03:21.152738210 (291h59m19.704208671s) [noise=true , address=145.79.227.22:4509 , piece_id=5GC5WOJYILYZ2DUIIGLZITCL4XJZ4XWXKCWXZY62C4QW5FYVHELA, seg_pos={0 0}, upload=257150]: Uploading piece... Nov 21 12:03:21.153047469 (291h59m19.70451793s) [noise=true , address=145.79.227.32:4834 , piece_id=WRCEO7VGITAJMKFS7W534X24KCALWKNGWOYP622UOI5GYF4V3OXA, seg_pos={0 0}, upload=257150]: Uploading piece... Nov 21 12:03:21.154222217 (291h59m19.705692678s) [noise=true , address=77.90.31.10:4519 , piece_id=KVHNESUY7ZYPBXUVZQMWJTVUF64ITT57A3AUYZQIP5CPRKIN6RGA, seg_pos={0 0}, upload=257150]: Uploading piece... Nov 21 12:03:21.155401501 (291h59m19.706871962s) [noise=true , address=145.79.227.8:4329 , piece_id=NMX7FTR2Z4EFQLGOND32RE2TI3AIZ7TGGVJTYFDPP3CGFFFJV2LQ, seg_pos={0 0}, upload=257150]: Uploading piece... Nov 21 12:03:21.157354521 (291h59m19.708824972s) [noise=true , address=45.91.116.10:4082 , piece_id=PUYEZFZQLZCRUATWBPEP7VYU75XD4DDIABD6GBBASF4RKMKTG2FA, seg_pos={0 0}, upload=257150]: Uploading piece... Nov 21 12:03:21.157746998 (291h59m19.709217469s) [noise=true , address=145.79.227.11:4729 , piece_id=GH57FJM7JI7WN3ICOX2BCK4JQQDYORJ3JX2B7V4LDG6KALWOHX7Q, seg_pos={0 0}, upload=257150]: Uploading piece... Nov 21 12:03:21.163268371 (291h59m19.714738832s) [noise=true , address=145.79.227.16:4912 , piece_id=QCAG67ERHROPSHVUVCKEZJFK3B6K4YI7LRZWARMJCT5QXTSAHL2A, seg_pos={0 0}, upload=257150]: Uploading piece... Nov 21 12:03:21.165309478 (291h59m19.716779939s) [noise=true , address=77.90.31.18:4805 , piece_id=6LWJPVVTDVATLDHL66WS6H6KMLXL4ZRXVRZFXP2QCGYFUFRV3C4A, seg_pos={0 0}, upload=257150]: Uploading piece... Nov 21 12:03:22.765252658 (291h59m21.316723119s) [upload=257150]: Next returned nil segment Nov 21 12:03:22.765268569 (291h59m21.31673903s) [upload=257150]: Waiting for error group managing segments... Nov 21 12:03:23.146214842 (291h59m21.697685313s) [noise=true , address=145.79.227.22:4511 , piece_id=3HRMW3Y33TRQHQNCD7CFDHYXRJ2WCAGE6XDJFBITER76CCOTIFOQ, seg_pos={0 0}, upload=257150]: Done uploading piece. err: <nil> Nov 21 12:03:23.148157711 (291h59m21.699628172s) [noise=true , address=145.79.227.14:4838 , piece_id=6D5V4CF2LSODZ4NPN67KUAJTZMK3ZAYGL546WJ4NIHV5SDLUHIZA, seg_pos={0 0}, upload=257150]: Done uploading piece. err: <nil> Nov 21 12:03:23.151571021 (291h59m21.703041482s) [noise=true , address=145.79.227.25:4619 , piece_id=PMHMAYC6QSE3HJLCEAMMBHHFOYOS6XLAMPRM5M33BBIILQX2Y7GA, seg_pos={0 0}, upload=257150]: Done uploading piece. err: <nil> Nov 21 12:03:23.152335496 (291h59m21.703805977s) [noise=true , address=145.79.227.11:4729 , piece_id=GH57FJM7JI7WN3ICOX2BCK4JQQDYORJ3JX2B7V4LDG6KALWOHX7Q, seg_pos={0 0}, upload=257150]: Done uploading piece. err: <nil> Nov 21 12:03:23.153157361 (291h59m21.704627822s) [noise=true , address=145.79.227.17:4971 , piece_id=53IMICLBNCSB57326VC5XKP7CG4AQSBKKAFYKBQMELRRSZDKBSQQ, seg_pos={0 0}, upload=257150]: Done uploading piece. err: <nil> Nov 21 12:03:23.154678468 (291h59m21.706148949s) [noise=true , address=145.79.227.20:4439 , piece_id=MYM5JPZEKXUBXAND2MHK2ZDHODNYPSM6K6JJRND4IFBOS5I43GDQ, seg_pos={0 0}, upload=257150]: Done uploading piece. err: <nil> Nov 21 12:03:23.155458653 (291h59m21.706929114s) [noise=true , address=145.79.227.32:4834 , piece_id=WRCEO7VGITAJMKFS7W534X24KCALWKNGWOYP622UOI5GYF4V3OXA, seg_pos={0 0}, upload=257150]: Done uploading piece. err: <nil> Nov 21 12:03:23.166587054 (291h59m21.718057515s) [noise=true , address=145.79.227.22:4509 , piece_id=5GC5WOJYILYZ2DUIIGLZITCL4XJZ4XWXKCWXZY62C4QW5FYVHELA, seg_pos={0 0}, upload=257150]: Done uploading piece. err: <nil> Nov 21 12:03:23.167943206 (291h59m21.719413667s) [noise=true , address=145.79.227.16:4912 , piece_id=QCAG67ERHROPSHVUVCKEZJFK3B6K4YI7LRZWARMJCT5QXTSAHL2A, seg_pos={0 0}, upload=257150]: Done uploading piece. err: <nil> Nov 21 12:03:23.172565528 (291h59m21.724035989s) [noise=true , address=145.79.227.27:4689 , piece_id=X4QKAUFXGYX54435VLH7QJ2HXSYZEL4SJ6V4VBNTIV77KBUMCZIA, seg_pos={0 0}, upload=257150]: Done uploading piece. err: <nil> Nov 21 12:03:23.191491918 (291h59m21.742962379s) [noise=true , address=145.79.227.8:4329 , piece_id=NMX7FTR2Z4EFQLGOND32RE2TI3AIZ7TGGVJTYFDPP3CGFFFJV2LQ, seg_pos={0 0}, upload=257150]: Done uploading piece. err: <nil> Nov 21 12:03:23.769501866 (291h59m22.320972327s) [noise=true , address=45.91.116.10:4082 , piece_id=PUYEZFZQLZCRUATWBPEP7VYU75XD4DDIABD6GBBASF4RKMKTG2FA, seg_pos={0 0}, upload=257150]: Done uploading piece. err: <nil> Nov 21 12:03:23.881038302 (291h59m22.432508764s) [noise=true , address=77.90.31.18:4805 , piece_id=6LWJPVVTDVATLDHL66WS6H6KMLXL4ZRXVRZFXP2QCGYFUFRV3C4A, seg_pos={0 0}, upload=257150]: Done uploading piece. err: <nil> Nov 21 12:03:23.881065174 (291h59m22.432535635s) [seg_pos={0 0}, upload=257150]: Segment reached optimal threshold of 13 pieces. Nov 21 12:03:23.881321642 (291h59m22.432792103s) [noise=true , address=77.90.31.10:4519 , piece_id=KVHNESUY7ZYPBXUVZQMWJTVUF64ITT57A3AUYZQIP5CPRKIN6RGA, seg_pos={0 0}, upload=257150]: Done uploading piece. err: <nil> Nov 21 12:03:23.881450838 (291h59m22.432921299s) [noise=true , address=77.90.31.21:4877 , piece_id=AJIDGRABKTO7SN745P7K73BTXEB72PZ473XBUPDLZVDX4F7KKKNQ, seg_pos={0 0}, upload=257150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Nov 21 12:03:23.881472891 (291h59m22.432943362s) [noise=true , address=77.90.31.21:4877 , piece_id=AJIDGRABKTO7SN745P7K73BTXEB72PZ473XBUPDLZVDX4F7KKKNQ, seg_pos={0 0}, upload=257150]: Long tail cancellation due to optimal threshold Nov 21 12:03:23.881998511 (291h59m22.433468972s) [noise=true , address=45.91.116.14:4190 , piece_id=XXL6DRSFSQFP2KCZTKZMS4R24CUY3CYUP7C6B4ZR7UYMHEVVNSFA, seg_pos={0 0}, upload=257150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Nov 21 12:03:23.882010925 (291h59m22.433481386s) [noise=true , address=45.91.116.14:4190 , piece_id=XXL6DRSFSQFP2KCZTKZMS4R24CUY3CYUP7C6B4ZR7UYMHEVVNSFA, seg_pos={0 0}, upload=257150]: Long tail cancellation due to optimal threshold Nov 21 12:03:23.882103892 (291h59m22.433574353s) [seg_pos={0 0}, upload=257150]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Nov 21 12:03:23.882311507 (291h59m22.433781978s) [upload=257150]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 21 12:03:23.882321226 (291h59m22.433791687s) [upload=257150]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 21 12:03:24.186628483 (291h59m22.738098944s) [upload=257150]: Done uploading segments... Nov 21 12:03:24.186656958 (291h59m22.738127429s) [upload=257150]: Upload finished. err: <nil> Duration: 5.870636328s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 5.352294517s trace: https://jaeger.dev.storj.io/trace/a7f69ee106efb7f FAIL (2025-11-21 12:04:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== upload=256586]: Uploading piece... Nov 21 11:25:27.210346024 (291h21m25.761816485s) [noise=true , address=89.149.201.147:4008 , piece_id=HCX7I4ICPEHDWTYDFIQMKZ5EE5CGPQLYAIHF7WZMTUUU6CNZPHTA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.211425289 (291h21m25.76289575s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.213103795 (291h21m25.764574246s) [noise=true , address=185.167.204.205:4022 , piece_id=BES3YK6PYQV2AKOWSAGHF7DGGWGWSWPIJU2ICAQ5WBHKZ5VT5L6A, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.213408395 (291h21m25.764878856s) [noise=true , address=185.167.204.205:4027 , piece_id=IVPONKBF2PIHZQAW3L4YA4HOMQV5BQML6SKKCJLIEGT5SZSQ2AOA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.214501747 (291h21m25.765972208s) [noise=true , address=81.171.14.70:4042 , piece_id=XNREZEUZ5L5KZ3L7ZOL6IRK7IHU6DJN4RLEQZRFMDKDH2JWC24CQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.215013752 (291h21m25.766484213s) [noise=true , address=81.171.14.70:4034 , piece_id=ITRLKYVNFSYBLABNGNFO3HRJNMX3MWB345FIMWSIPBNAVZPA47UA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.215087572 (291h21m25.766558044s) [noise=true , address=185.167.204.205:4031 , piece_id=ZEI2YSIQC655R7WS43WADXGFT5RKT5LFSN6IZEQ6SHTDVOV7BZBQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.216432844 (291h21m25.767903305s) [noise=true , address=185.167.204.206:4056 , piece_id=EAMZIK6AVJL7IMFF5VOLW3SMCGYKSKB2VXJTNUNLJCVIT7EKXNMQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.216662281 (291h21m25.768132742s) [noise=true , address=89.149.201.147:4011 , piece_id=ZATNMD5HOK35AWGJJ5V66VDJ4YGUVZQJ2OBERQL32ZV5GW2HBSMQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.218008314 (291h21m25.769478775s) [noise=true , address=81.171.14.70:4032 , piece_id=CFH2RTR3BLBUB5ZIZCFC6UHT3FHT3B2XSCHQMKXNGDJJGPHEMCTQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:28.035150976 (291h21m26.586621437s) [upload=256586]: Next returned nil segment Nov 21 11:25:28.035169622 (291h21m26.586640083s) [upload=256586]: Waiting for error group managing segments... Nov 21 11:25:28.416646836 (291h21m26.968117317s) [noise=true , address=185.167.204.206:4064 , piece_id=SRFIWFQLRAGCNIKYLHMGT4QRUL3WH2QPLLIKAJ7DLFJEQUW5DPPA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.417338854 (291h21m26.968809325s) [noise=true , address=185.167.204.205:4033 , piece_id=FEWQVHPIKN3Q6O5AZ6AGJBLUPZ34PBZFB5X3XNSKGPOGI4AIO7JA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.421089406 (291h21m26.972559867s) [noise=true , address=185.167.204.206:4072 , piece_id=GYCO53OWGOMOHK4FHC5GENN4LTQGSZKZKOI5V534RLV3LHLPKC6Q, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.426252887 (291h21m26.977723348s) [noise=true , address=185.167.204.206:4046 , piece_id=BZ57ZA35ITEOA6CAZJGNODK5MUTXEZWDV7E2OGBZOSMS4OQZYAEA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.431720689 (291h21m26.98319115s) [noise=true , address=185.167.204.205:4027 , piece_id=IVPONKBF2PIHZQAW3L4YA4HOMQV5BQML6SKKCJLIEGT5SZSQ2AOA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.438243951 (291h21m26.989714412s) [noise=true , address=185.167.204.205:4022 , piece_id=BES3YK6PYQV2AKOWSAGHF7DGGWGWSWPIJU2ICAQ5WBHKZ5VT5L6A, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.443364331 (291h21m26.994834792s) [noise=true , address=185.167.204.206:4056 , piece_id=EAMZIK6AVJL7IMFF5VOLW3SMCGYKSKB2VXJTNUNLJCVIT7EKXNMQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.445594728 (291h21m26.997065179s) [noise=true , address=185.167.204.205:4016 , piece_id=EHQS3546UXRXTL2MGFCSRTPDDPAYXEYLMISPP5AWY337MHX7HFLQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.449325822 (291h21m27.000796283s) [noise=true , address=185.167.204.205:4031 , piece_id=ZEI2YSIQC655R7WS43WADXGFT5RKT5LFSN6IZEQ6SHTDVOV7BZBQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.449761211 (291h21m27.001231672s) [noise=true , address=185.167.204.205:4018 , piece_id=7OSLPWJLKFTIJPZEAFMZOAH6JAAMX6HKROI25FODFYBPNJWSMLWA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.450419845 (291h21m27.001890306s) [noise=true , address=185.167.204.206:4062 , piece_id=UF3S6NAVGS2KQJPGA5WXP7YG4RQO44P445SNPCWXKXMONZAYUTNQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.453201252 (291h21m27.004671713s) [noise=true , address=185.167.204.205:4007 , piece_id=MEBGOPZ6MFMQY4UZARS2LKJKEMNN5YGSLKPJGTWSKS45CUNXSRGQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.661209626 (291h21m28.212680087s) [noise=true , address=89.149.201.147:4016 , piece_id=TGRWGGGBKDANC7YKROWPRMIYNAVFEVE3RXOSUOINFI7KTRS2IYUQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.693169182 (291h21m28.244639643s) [noise=true , address=89.149.201.147:4002 , piece_id=UIYJSCMY3LCLRHZUJB6JFJHHXKMYTVKERTZW7F7X44MWJETPY6XA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.721204497 (291h21m28.272674958s) [noise=true , address=89.149.201.147:4008 , piece_id=HCX7I4ICPEHDWTYDFIQMKZ5EE5CGPQLYAIHF7WZMTUUU6CNZPHTA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.764584711 (291h21m28.316055172s) [noise=true , address=89.149.201.147:4001 , piece_id=QUS6NB3NQBLJFB6YCK3EBDPKCTWEWGKAT3NOR24TVI65SQKHWDOQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.768284075 (291h21m28.319754536s) [noise=true , address=89.149.201.147:4011 , piece_id=ZATNMD5HOK35AWGJJ5V66VDJ4YGUVZQJ2OBERQL32ZV5GW2HBSMQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.789681961 (291h21m28.341152422s) [noise=true , address=89.149.201.147:4003 , piece_id=RUAYGN2RGF5JJTWFEGIHWH3BHEUDSS5DN2QJ6YLSHPF5BGUXD3RA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:37.298598682 (291h21m35.850069143s) [noise=true , address=81.171.14.70:4043 , piece_id=T3NF6JR3VJIGDMPNUBLMJ3AOTQP5IHOGJYORMCSS7ZMOENQETZ7A, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:37.497478356 (291h21m36.048948818s) [noise=true , address=81.171.14.70:4032 , piece_id=CFH2RTR3BLBUB5ZIZCFC6UHT3FHT3B2XSCHQMKXNGDJJGPHEMCTQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:40.406166933 (291h21m38.957637394s) [noise=true , address=81.171.14.70:4045 , piece_id=BRWQ7G3E547PJE2RLGSESIB3HMP5C4M5NSKAJNGWSFXUBEO6SYEQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:42.511041157 (291h21m41.062511608s) [noise=true , address=81.171.14.70:4029 , piece_id=DKJM5E7EY7LOSBSEBQOZRC6CI5ZPPE47HFOFQJROPCUMUV567CAA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:12.012682521 (291h22m10.564152982s) [noise=true , address=81.171.14.70:4042 , piece_id=XNREZEUZ5L5KZ3L7ZOL6IRK7IHU6DJN4RLEQZRFMDKDH2JWC24CQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:15.596046283 (291h22m14.147516744s) [noise=true , address=81.171.14.70:4033 , piece_id=JRKPD4EP6FZM5HISOIDJYZY7FAWWQCADAS47FGW2JM5A7RFJR2BQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:18.453059659 (291h22m17.00453012s) [noise=true , address=81.171.14.70:4046 , piece_id=OSONRSF6KJS7F2QJVCC3FPZK4AV7QPG5YSM47CW5LIW7EME44AWA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:23.442583344 (291h22m21.994053805s) [noise=true , address=81.171.14.70:4034 , piece_id=ITRLKYVNFSYBLABNGNFO3HRJNMX3MWB345FIMWSIPBNAVZPA47UA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:23.442621967 (291h22m21.994092428s) [seg_pos={0 0}, upload=256586]: Segment reached optimal threshold of 26 pieces. Nov 21 11:26:23.442836976 (291h22m21.994307437s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MUVGecysnTSFcJoHUL7QMWb3h6dtXgUd7FbA672fQCyEWhePr): protocol: expected piece hash; context canceled; EOF Nov 21 11:26:23.442845934 (291h22m21.994316395s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Long tail cancellation due to optimal threshold Nov 21 11:26:23.443227491 (291h22m21.994697942s) [noise=true , address=81.171.14.70:4027 , piece_id=4FDFCCWF3EIROQAFLOHGS743523OYGSXWQSPQEVD3AMQ6WKDQI7Q, seg_pos={0 0}, upload=256586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122cYvgTxSvC2aEXnBkRk2o8ARTBXj9gqPoa12vW8hogSN6kpA4): protocol: expected piece hash; context canceled; EOF Nov 21 11:26:23.443237770 (291h22m21.994708231s) [noise=true , address=81.171.14.70:4027 , piece_id=4FDFCCWF3EIROQAFLOHGS743523OYGSXWQSPQEVD3AMQ6WKDQI7Q, seg_pos={0 0}, upload=256586]: Long tail cancellation due to optimal threshold Nov 21 11:26:23.443274720 (291h22m21.994745181s) [seg_pos={0 0}, upload=256586]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Nov 21 11:26:23.443378929 (291h22m21.99484939s) [upload=256586]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 21 11:26:23.443387044 (291h22m21.994857495s) [upload=256586]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 21 11:26:23.685139626 (291h22m22.236610077s) [upload=256586]: Done uploading segments... Nov 21 11:26:23.685160676 (291h22m22.236631127s) [upload=256586]: Upload finished. err: <nil> Duration: 5.352322631s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 25 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXX...................................] duration: 3.440219689s trace: https://jaeger.dev.storj.io/trace/6af15a3a67b655d5 FAIL (2025-11-21 12:04:11 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 3.440249836s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [.X.X.X....X............X....X.X.X..X.X..X...X..XXX.X.XX...X.] duration: 6.287913009s trace: https://jaeger.dev.storj.io/trace/400a3392fd596b9b FAIL (2025-11-21 12:03:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 21 12:03:20.894547296 (291h59m19.446017757s) [upload=257150]: Starting upload Nov 21 12:03:20.894637809 (291h59m19.44610827s) [upload=257150]: Done starting upload Nov 21 12:03:20.894713393 (291h59m19.446183854s) [upload=257150]: Uploading segments... Nov 21 12:03:20.894969650 (291h59m19.446440122s) [upload=257150]: Got next segment. Inline: false Nov 21 12:03:20.895013724 (291h59m19.446484185s) [upload=257150]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 12:03:20.895020538 (291h59m19.446491009s) [upload=257150]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 12:03:21.141706975 (291h59m19.693177446s) [seg_pos={0 0}, upload=257150]: Begin upload segment... Nov 21 12:03:21.141774353 (291h59m19.693244814s) [seg_pos={0 0}, upload=257150]: Done begin upload segment. Nov 21 12:03:21.141865167 (291h59m19.693335638s) [noise=true , address=77.90.31.21:4877 , piece_id=AJIDGRABKTO7SN745P7K73BTXEB72PZ473XBUPDLZVDX4F7KKKNQ, seg_pos={0 0}, upload=257150]: Uploading piece... Nov 21 12:03:21.145928293 (291h59m19.697398754s) [noise=true , address=145.79.227.17:4971 , piece_id=53IMICLBNCSB57326VC5XKP7CG4AQSBKKAFYKBQMELRRSZDKBSQQ, seg_pos={0 0}, upload=257150]: Uploading piece... Nov 21 12:03:21.146959898 (291h59m19.698430359s) [noise=true , address=145.79.227.27:4689 , piece_id=X4QKAUFXGYX54435VLH7QJ2HXSYZEL4SJ6V4VBNTIV77KBUMCZIA, seg_pos={0 0}, upload=257150]: Uploading piece... Nov 21 12:03:21.147497481 (291h59m19.698967942s) [noise=true , address=145.79.227.20:4439 , piece_id=MYM5JPZEKXUBXAND2MHK2ZDHODNYPSM6K6JJRND4IFBOS5I43GDQ, seg_pos={0 0}, upload=257150]: Uploading piece... Nov 21 12:03:21.147962968 (291h59m19.699433429s) [noise=true , address=145.79.227.25:4619 , piece_id=PMHMAYC6QSE3HJLCEAMMBHHFOYOS6XLAMPRM5M33BBIILQX2Y7GA, seg_pos={0 0}, upload=257150]: Uploading piece... Nov 21 12:03:21.151526994 (291h59m19.702997465s) [noise=true , address=145.79.227.22:4511 , piece_id=3HRMW3Y33TRQHQNCD7CFDHYXRJ2WCAGE6XDJFBITER76CCOTIFOQ, seg_pos={0 0}, upload=257150]: Uploading piece... Nov 21 12:03:21.151746542 (291h59m19.703217013s) [noise=true , address=145.79.227.14:4838 , piece_id=6D5V4CF2LSODZ4NPN67KUAJTZMK3ZAYGL546WJ4NIHV5SDLUHIZA, seg_pos={0 0}, upload=257150]: Uploading piece... Nov 21 12:03:21.152451976 (291h59m19.703922437s) [noise=true , address=45.91.116.14:4190 , piece_id=XXL6DRSFSQFP2KCZTKZMS4R24CUY3CYUP7C6B4ZR7UYMHEVVNSFA, seg_pos={0 0}, upload=257150]: Uploading piece... Nov 21 12:03:21.152738210 (291h59m19.704208671s) [noise=true , address=145.79.227.22:4509 , piece_id=5GC5WOJYILYZ2DUIIGLZITCL4XJZ4XWXKCWXZY62C4QW5FYVHELA, seg_pos={0 0}, upload=257150]: Uploading piece... Nov 21 12:03:21.153047469 (291h59m19.70451793s) [noise=true , address=145.79.227.32:4834 , piece_id=WRCEO7VGITAJMKFS7W534X24KCALWKNGWOYP622UOI5GYF4V3OXA, seg_pos={0 0}, upload=257150]: Uploading piece... Nov 21 12:03:21.154222217 (291h59m19.705692678s) [noise=true , address=77.90.31.10:4519 , piece_id=KVHNESUY7ZYPBXUVZQMWJTVUF64ITT57A3AUYZQIP5CPRKIN6RGA, seg_pos={0 0}, upload=257150]: Uploading piece... Nov 21 12:03:21.155401501 (291h59m19.706871962s) [noise=true , address=145.79.227.8:4329 , piece_id=NMX7FTR2Z4EFQLGOND32RE2TI3AIZ7TGGVJTYFDPP3CGFFFJV2LQ, seg_pos={0 0}, upload=257150]: Uploading piece... Nov 21 12:03:21.157354521 (291h59m19.708824972s) [noise=true , address=45.91.116.10:4082 , piece_id=PUYEZFZQLZCRUATWBPEP7VYU75XD4DDIABD6GBBASF4RKMKTG2FA, seg_pos={0 0}, upload=257150]: Uploading piece... Nov 21 12:03:21.157746998 (291h59m19.709217469s) [noise=true , address=145.79.227.11:4729 , piece_id=GH57FJM7JI7WN3ICOX2BCK4JQQDYORJ3JX2B7V4LDG6KALWOHX7Q, seg_pos={0 0}, upload=257150]: Uploading piece... Nov 21 12:03:21.163268371 (291h59m19.714738832s) [noise=true , address=145.79.227.16:4912 , piece_id=QCAG67ERHROPSHVUVCKEZJFK3B6K4YI7LRZWARMJCT5QXTSAHL2A, seg_pos={0 0}, upload=257150]: Uploading piece... Nov 21 12:03:21.165309478 (291h59m19.716779939s) [noise=true , address=77.90.31.18:4805 , piece_id=6LWJPVVTDVATLDHL66WS6H6KMLXL4ZRXVRZFXP2QCGYFUFRV3C4A, seg_pos={0 0}, upload=257150]: Uploading piece... Nov 21 12:03:22.765252658 (291h59m21.316723119s) [upload=257150]: Next returned nil segment Nov 21 12:03:22.765268569 (291h59m21.31673903s) [upload=257150]: Waiting for error group managing segments... Nov 21 12:03:23.146214842 (291h59m21.697685313s) [noise=true , address=145.79.227.22:4511 , piece_id=3HRMW3Y33TRQHQNCD7CFDHYXRJ2WCAGE6XDJFBITER76CCOTIFOQ, seg_pos={0 0}, upload=257150]: Done uploading piece. err: <nil> Nov 21 12:03:23.148157711 (291h59m21.699628172s) [noise=true , address=145.79.227.14:4838 , piece_id=6D5V4CF2LSODZ4NPN67KUAJTZMK3ZAYGL546WJ4NIHV5SDLUHIZA, seg_pos={0 0}, upload=257150]: Done uploading piece. err: <nil> Nov 21 12:03:23.151571021 (291h59m21.703041482s) [noise=true , address=145.79.227.25:4619 , piece_id=PMHMAYC6QSE3HJLCEAMMBHHFOYOS6XLAMPRM5M33BBIILQX2Y7GA, seg_pos={0 0}, upload=257150]: Done uploading piece. err: <nil> Nov 21 12:03:23.152335496 (291h59m21.703805977s) [noise=true , address=145.79.227.11:4729 , piece_id=GH57FJM7JI7WN3ICOX2BCK4JQQDYORJ3JX2B7V4LDG6KALWOHX7Q, seg_pos={0 0}, upload=257150]: Done uploading piece. err: <nil> Nov 21 12:03:23.153157361 (291h59m21.704627822s) [noise=true , address=145.79.227.17:4971 , piece_id=53IMICLBNCSB57326VC5XKP7CG4AQSBKKAFYKBQMELRRSZDKBSQQ, seg_pos={0 0}, upload=257150]: Done uploading piece. err: <nil> Nov 21 12:03:23.154678468 (291h59m21.706148949s) [noise=true , address=145.79.227.20:4439 , piece_id=MYM5JPZEKXUBXAND2MHK2ZDHODNYPSM6K6JJRND4IFBOS5I43GDQ, seg_pos={0 0}, upload=257150]: Done uploading piece. err: <nil> Nov 21 12:03:23.155458653 (291h59m21.706929114s) [noise=true , address=145.79.227.32:4834 , piece_id=WRCEO7VGITAJMKFS7W534X24KCALWKNGWOYP622UOI5GYF4V3OXA, seg_pos={0 0}, upload=257150]: Done uploading piece. err: <nil> Nov 21 12:03:23.166587054 (291h59m21.718057515s) [noise=true , address=145.79.227.22:4509 , piece_id=5GC5WOJYILYZ2DUIIGLZITCL4XJZ4XWXKCWXZY62C4QW5FYVHELA, seg_pos={0 0}, upload=257150]: Done uploading piece. err: <nil> Nov 21 12:03:23.167943206 (291h59m21.719413667s) [noise=true , address=145.79.227.16:4912 , piece_id=QCAG67ERHROPSHVUVCKEZJFK3B6K4YI7LRZWARMJCT5QXTSAHL2A, seg_pos={0 0}, upload=257150]: Done uploading piece. err: <nil> Nov 21 12:03:23.172565528 (291h59m21.724035989s) [noise=true , address=145.79.227.27:4689 , piece_id=X4QKAUFXGYX54435VLH7QJ2HXSYZEL4SJ6V4VBNTIV77KBUMCZIA, seg_pos={0 0}, upload=257150]: Done uploading piece. err: <nil> Nov 21 12:03:23.191491918 (291h59m21.742962379s) [noise=true , address=145.79.227.8:4329 , piece_id=NMX7FTR2Z4EFQLGOND32RE2TI3AIZ7TGGVJTYFDPP3CGFFFJV2LQ, seg_pos={0 0}, upload=257150]: Done uploading piece. err: <nil> Nov 21 12:03:23.769501866 (291h59m22.320972327s) [noise=true , address=45.91.116.10:4082 , piece_id=PUYEZFZQLZCRUATWBPEP7VYU75XD4DDIABD6GBBASF4RKMKTG2FA, seg_pos={0 0}, upload=257150]: Done uploading piece. err: <nil> Nov 21 12:03:23.881038302 (291h59m22.432508764s) [noise=true , address=77.90.31.18:4805 , piece_id=6LWJPVVTDVATLDHL66WS6H6KMLXL4ZRXVRZFXP2QCGYFUFRV3C4A, seg_pos={0 0}, upload=257150]: Done uploading piece. err: <nil> Nov 21 12:03:23.881065174 (291h59m22.432535635s) [seg_pos={0 0}, upload=257150]: Segment reached optimal threshold of 13 pieces. Nov 21 12:03:23.881321642 (291h59m22.432792103s) [noise=true , address=77.90.31.10:4519 , piece_id=KVHNESUY7ZYPBXUVZQMWJTVUF64ITT57A3AUYZQIP5CPRKIN6RGA, seg_pos={0 0}, upload=257150]: Done uploading piece. err: <nil> Nov 21 12:03:23.881450838 (291h59m22.432921299s) [noise=true , address=77.90.31.21:4877 , piece_id=AJIDGRABKTO7SN745P7K73BTXEB72PZ473XBUPDLZVDX4F7KKKNQ, seg_pos={0 0}, upload=257150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Nov 21 12:03:23.881472891 (291h59m22.432943362s) [noise=true , address=77.90.31.21:4877 , piece_id=AJIDGRABKTO7SN745P7K73BTXEB72PZ473XBUPDLZVDX4F7KKKNQ, seg_pos={0 0}, upload=257150]: Long tail cancellation due to optimal threshold Nov 21 12:03:23.881998511 (291h59m22.433468972s) [noise=true , address=45.91.116.14:4190 , piece_id=XXL6DRSFSQFP2KCZTKZMS4R24CUY3CYUP7C6B4ZR7UYMHEVVNSFA, seg_pos={0 0}, upload=257150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Nov 21 12:03:23.882010925 (291h59m22.433481386s) [noise=true , address=45.91.116.14:4190 , piece_id=XXL6DRSFSQFP2KCZTKZMS4R24CUY3CYUP7C6B4ZR7UYMHEVVNSFA, seg_pos={0 0}, upload=257150]: Long tail cancellation due to optimal threshold Nov 21 12:03:23.882103892 (291h59m22.433574353s) [seg_pos={0 0}, upload=257150]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Nov 21 12:03:23.882311507 (291h59m22.433781978s) [upload=257150]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 21 12:03:23.882321226 (291h59m22.433791687s) [upload=257150]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 21 12:03:24.186628483 (291h59m22.738098944s) [upload=257150]: Done uploading segments... Nov 21 12:03:24.186656958 (291h59m22.738127429s) [upload=257150]: Upload finished. err: <nil> Duration: 6.287935071s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 5.743195546s trace: https://jaeger.dev.storj.io/trace/a7f69ee106efb7f FAIL (2025-11-21 12:03:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== upload=256586]: Uploading piece... Nov 21 11:25:27.210346024 (291h21m25.761816485s) [noise=true , address=89.149.201.147:4008 , piece_id=HCX7I4ICPEHDWTYDFIQMKZ5EE5CGPQLYAIHF7WZMTUUU6CNZPHTA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.211425289 (291h21m25.76289575s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.213103795 (291h21m25.764574246s) [noise=true , address=185.167.204.205:4022 , piece_id=BES3YK6PYQV2AKOWSAGHF7DGGWGWSWPIJU2ICAQ5WBHKZ5VT5L6A, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.213408395 (291h21m25.764878856s) [noise=true , address=185.167.204.205:4027 , piece_id=IVPONKBF2PIHZQAW3L4YA4HOMQV5BQML6SKKCJLIEGT5SZSQ2AOA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.214501747 (291h21m25.765972208s) [noise=true , address=81.171.14.70:4042 , piece_id=XNREZEUZ5L5KZ3L7ZOL6IRK7IHU6DJN4RLEQZRFMDKDH2JWC24CQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.215013752 (291h21m25.766484213s) [noise=true , address=81.171.14.70:4034 , piece_id=ITRLKYVNFSYBLABNGNFO3HRJNMX3MWB345FIMWSIPBNAVZPA47UA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.215087572 (291h21m25.766558044s) [noise=true , address=185.167.204.205:4031 , piece_id=ZEI2YSIQC655R7WS43WADXGFT5RKT5LFSN6IZEQ6SHTDVOV7BZBQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.216432844 (291h21m25.767903305s) [noise=true , address=185.167.204.206:4056 , piece_id=EAMZIK6AVJL7IMFF5VOLW3SMCGYKSKB2VXJTNUNLJCVIT7EKXNMQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.216662281 (291h21m25.768132742s) [noise=true , address=89.149.201.147:4011 , piece_id=ZATNMD5HOK35AWGJJ5V66VDJ4YGUVZQJ2OBERQL32ZV5GW2HBSMQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.218008314 (291h21m25.769478775s) [noise=true , address=81.171.14.70:4032 , piece_id=CFH2RTR3BLBUB5ZIZCFC6UHT3FHT3B2XSCHQMKXNGDJJGPHEMCTQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:28.035150976 (291h21m26.586621437s) [upload=256586]: Next returned nil segment Nov 21 11:25:28.035169622 (291h21m26.586640083s) [upload=256586]: Waiting for error group managing segments... Nov 21 11:25:28.416646836 (291h21m26.968117317s) [noise=true , address=185.167.204.206:4064 , piece_id=SRFIWFQLRAGCNIKYLHMGT4QRUL3WH2QPLLIKAJ7DLFJEQUW5DPPA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.417338854 (291h21m26.968809325s) [noise=true , address=185.167.204.205:4033 , piece_id=FEWQVHPIKN3Q6O5AZ6AGJBLUPZ34PBZFB5X3XNSKGPOGI4AIO7JA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.421089406 (291h21m26.972559867s) [noise=true , address=185.167.204.206:4072 , piece_id=GYCO53OWGOMOHK4FHC5GENN4LTQGSZKZKOI5V534RLV3LHLPKC6Q, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.426252887 (291h21m26.977723348s) [noise=true , address=185.167.204.206:4046 , piece_id=BZ57ZA35ITEOA6CAZJGNODK5MUTXEZWDV7E2OGBZOSMS4OQZYAEA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.431720689 (291h21m26.98319115s) [noise=true , address=185.167.204.205:4027 , piece_id=IVPONKBF2PIHZQAW3L4YA4HOMQV5BQML6SKKCJLIEGT5SZSQ2AOA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.438243951 (291h21m26.989714412s) [noise=true , address=185.167.204.205:4022 , piece_id=BES3YK6PYQV2AKOWSAGHF7DGGWGWSWPIJU2ICAQ5WBHKZ5VT5L6A, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.443364331 (291h21m26.994834792s) [noise=true , address=185.167.204.206:4056 , piece_id=EAMZIK6AVJL7IMFF5VOLW3SMCGYKSKB2VXJTNUNLJCVIT7EKXNMQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.445594728 (291h21m26.997065179s) [noise=true , address=185.167.204.205:4016 , piece_id=EHQS3546UXRXTL2MGFCSRTPDDPAYXEYLMISPP5AWY337MHX7HFLQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.449325822 (291h21m27.000796283s) [noise=true , address=185.167.204.205:4031 , piece_id=ZEI2YSIQC655R7WS43WADXGFT5RKT5LFSN6IZEQ6SHTDVOV7BZBQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.449761211 (291h21m27.001231672s) [noise=true , address=185.167.204.205:4018 , piece_id=7OSLPWJLKFTIJPZEAFMZOAH6JAAMX6HKROI25FODFYBPNJWSMLWA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.450419845 (291h21m27.001890306s) [noise=true , address=185.167.204.206:4062 , piece_id=UF3S6NAVGS2KQJPGA5WXP7YG4RQO44P445SNPCWXKXMONZAYUTNQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.453201252 (291h21m27.004671713s) [noise=true , address=185.167.204.205:4007 , piece_id=MEBGOPZ6MFMQY4UZARS2LKJKEMNN5YGSLKPJGTWSKS45CUNXSRGQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.661209626 (291h21m28.212680087s) [noise=true , address=89.149.201.147:4016 , piece_id=TGRWGGGBKDANC7YKROWPRMIYNAVFEVE3RXOSUOINFI7KTRS2IYUQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.693169182 (291h21m28.244639643s) [noise=true , address=89.149.201.147:4002 , piece_id=UIYJSCMY3LCLRHZUJB6JFJHHXKMYTVKERTZW7F7X44MWJETPY6XA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.721204497 (291h21m28.272674958s) [noise=true , address=89.149.201.147:4008 , piece_id=HCX7I4ICPEHDWTYDFIQMKZ5EE5CGPQLYAIHF7WZMTUUU6CNZPHTA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.764584711 (291h21m28.316055172s) [noise=true , address=89.149.201.147:4001 , piece_id=QUS6NB3NQBLJFB6YCK3EBDPKCTWEWGKAT3NOR24TVI65SQKHWDOQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.768284075 (291h21m28.319754536s) [noise=true , address=89.149.201.147:4011 , piece_id=ZATNMD5HOK35AWGJJ5V66VDJ4YGUVZQJ2OBERQL32ZV5GW2HBSMQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.789681961 (291h21m28.341152422s) [noise=true , address=89.149.201.147:4003 , piece_id=RUAYGN2RGF5JJTWFEGIHWH3BHEUDSS5DN2QJ6YLSHPF5BGUXD3RA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:37.298598682 (291h21m35.850069143s) [noise=true , address=81.171.14.70:4043 , piece_id=T3NF6JR3VJIGDMPNUBLMJ3AOTQP5IHOGJYORMCSS7ZMOENQETZ7A, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:37.497478356 (291h21m36.048948818s) [noise=true , address=81.171.14.70:4032 , piece_id=CFH2RTR3BLBUB5ZIZCFC6UHT3FHT3B2XSCHQMKXNGDJJGPHEMCTQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:40.406166933 (291h21m38.957637394s) [noise=true , address=81.171.14.70:4045 , piece_id=BRWQ7G3E547PJE2RLGSESIB3HMP5C4M5NSKAJNGWSFXUBEO6SYEQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:42.511041157 (291h21m41.062511608s) [noise=true , address=81.171.14.70:4029 , piece_id=DKJM5E7EY7LOSBSEBQOZRC6CI5ZPPE47HFOFQJROPCUMUV567CAA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:12.012682521 (291h22m10.564152982s) [noise=true , address=81.171.14.70:4042 , piece_id=XNREZEUZ5L5KZ3L7ZOL6IRK7IHU6DJN4RLEQZRFMDKDH2JWC24CQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:15.596046283 (291h22m14.147516744s) [noise=true , address=81.171.14.70:4033 , piece_id=JRKPD4EP6FZM5HISOIDJYZY7FAWWQCADAS47FGW2JM5A7RFJR2BQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:18.453059659 (291h22m17.00453012s) [noise=true , address=81.171.14.70:4046 , piece_id=OSONRSF6KJS7F2QJVCC3FPZK4AV7QPG5YSM47CW5LIW7EME44AWA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:23.442583344 (291h22m21.994053805s) [noise=true , address=81.171.14.70:4034 , piece_id=ITRLKYVNFSYBLABNGNFO3HRJNMX3MWB345FIMWSIPBNAVZPA47UA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:23.442621967 (291h22m21.994092428s) [seg_pos={0 0}, upload=256586]: Segment reached optimal threshold of 26 pieces. Nov 21 11:26:23.442836976 (291h22m21.994307437s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MUVGecysnTSFcJoHUL7QMWb3h6dtXgUd7FbA672fQCyEWhePr): protocol: expected piece hash; context canceled; EOF Nov 21 11:26:23.442845934 (291h22m21.994316395s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Long tail cancellation due to optimal threshold Nov 21 11:26:23.443227491 (291h22m21.994697942s) [noise=true , address=81.171.14.70:4027 , piece_id=4FDFCCWF3EIROQAFLOHGS743523OYGSXWQSPQEVD3AMQ6WKDQI7Q, seg_pos={0 0}, upload=256586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122cYvgTxSvC2aEXnBkRk2o8ARTBXj9gqPoa12vW8hogSN6kpA4): protocol: expected piece hash; context canceled; EOF Nov 21 11:26:23.443237770 (291h22m21.994708231s) [noise=true , address=81.171.14.70:4027 , piece_id=4FDFCCWF3EIROQAFLOHGS743523OYGSXWQSPQEVD3AMQ6WKDQI7Q, seg_pos={0 0}, upload=256586]: Long tail cancellation due to optimal threshold Nov 21 11:26:23.443274720 (291h22m21.994745181s) [seg_pos={0 0}, upload=256586]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Nov 21 11:26:23.443378929 (291h22m21.99484939s) [upload=256586]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 21 11:26:23.443387044 (291h22m21.994857495s) [upload=256586]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 21 11:26:23.685139626 (291h22m22.236610077s) [upload=256586]: Done uploading segments... Nov 21 11:26:23.685160676 (291h22m22.236631127s) [upload=256586]: Upload finished. err: <nil> Duration: 5.743226484s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [..X.X.X....X............X....X.X.X..X.X..X...X..XXX.X.XX...X] duration: 30.003727288s trace: https://jaeger.dev.storj.io/trace/400a3392fd596b9b FAIL (2025-11-21 12:03:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 21 12:03:20.894547296 (291h59m19.446017757s) [upload=257150]: Starting upload Nov 21 12:03:20.894637809 (291h59m19.44610827s) [upload=257150]: Done starting upload Nov 21 12:03:20.894713393 (291h59m19.446183854s) [upload=257150]: Uploading segments... Nov 21 12:03:20.894969650 (291h59m19.446440122s) [upload=257150]: Got next segment. Inline: false Nov 21 12:03:20.895013724 (291h59m19.446484185s) [upload=257150]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 12:03:20.895020538 (291h59m19.446491009s) [upload=257150]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 12:03:21.141706975 (291h59m19.693177446s) [seg_pos={0 0}, upload=257150]: Begin upload segment... Nov 21 12:03:21.141774353 (291h59m19.693244814s) [seg_pos={0 0}, upload=257150]: Done begin upload segment. Nov 21 12:03:21.141865167 (291h59m19.693335638s) [noise=true , address=77.90.31.21:4877 , piece_id=AJIDGRABKTO7SN745P7K73BTXEB72PZ473XBUPDLZVDX4F7KKKNQ, seg_pos={0 0}, upload=257150]: Uploading piece... Nov 21 12:03:21.145928293 (291h59m19.697398754s) [noise=true , address=145.79.227.17:4971 , piece_id=53IMICLBNCSB57326VC5XKP7CG4AQSBKKAFYKBQMELRRSZDKBSQQ, seg_pos={0 0}, upload=257150]: Uploading piece... Nov 21 12:03:21.146959898 (291h59m19.698430359s) [noise=true , address=145.79.227.27:4689 , piece_id=X4QKAUFXGYX54435VLH7QJ2HXSYZEL4SJ6V4VBNTIV77KBUMCZIA, seg_pos={0 0}, upload=257150]: Uploading piece... Nov 21 12:03:21.147497481 (291h59m19.698967942s) [noise=true , address=145.79.227.20:4439 , piece_id=MYM5JPZEKXUBXAND2MHK2ZDHODNYPSM6K6JJRND4IFBOS5I43GDQ, seg_pos={0 0}, upload=257150]: Uploading piece... Nov 21 12:03:21.147962968 (291h59m19.699433429s) [noise=true , address=145.79.227.25:4619 , piece_id=PMHMAYC6QSE3HJLCEAMMBHHFOYOS6XLAMPRM5M33BBIILQX2Y7GA, seg_pos={0 0}, upload=257150]: Uploading piece... Nov 21 12:03:21.151526994 (291h59m19.702997465s) [noise=true , address=145.79.227.22:4511 , piece_id=3HRMW3Y33TRQHQNCD7CFDHYXRJ2WCAGE6XDJFBITER76CCOTIFOQ, seg_pos={0 0}, upload=257150]: Uploading piece... Nov 21 12:03:21.151746542 (291h59m19.703217013s) [noise=true , address=145.79.227.14:4838 , piece_id=6D5V4CF2LSODZ4NPN67KUAJTZMK3ZAYGL546WJ4NIHV5SDLUHIZA, seg_pos={0 0}, upload=257150]: Uploading piece... Nov 21 12:03:21.152451976 (291h59m19.703922437s) [noise=true , address=45.91.116.14:4190 , piece_id=XXL6DRSFSQFP2KCZTKZMS4R24CUY3CYUP7C6B4ZR7UYMHEVVNSFA, seg_pos={0 0}, upload=257150]: Uploading piece... Nov 21 12:03:21.152738210 (291h59m19.704208671s) [noise=true , address=145.79.227.22:4509 , piece_id=5GC5WOJYILYZ2DUIIGLZITCL4XJZ4XWXKCWXZY62C4QW5FYVHELA, seg_pos={0 0}, upload=257150]: Uploading piece... Nov 21 12:03:21.153047469 (291h59m19.70451793s) [noise=true , address=145.79.227.32:4834 , piece_id=WRCEO7VGITAJMKFS7W534X24KCALWKNGWOYP622UOI5GYF4V3OXA, seg_pos={0 0}, upload=257150]: Uploading piece... Nov 21 12:03:21.154222217 (291h59m19.705692678s) [noise=true , address=77.90.31.10:4519 , piece_id=KVHNESUY7ZYPBXUVZQMWJTVUF64ITT57A3AUYZQIP5CPRKIN6RGA, seg_pos={0 0}, upload=257150]: Uploading piece... Nov 21 12:03:21.155401501 (291h59m19.706871962s) [noise=true , address=145.79.227.8:4329 , piece_id=NMX7FTR2Z4EFQLGOND32RE2TI3AIZ7TGGVJTYFDPP3CGFFFJV2LQ, seg_pos={0 0}, upload=257150]: Uploading piece... Nov 21 12:03:21.157354521 (291h59m19.708824972s) [noise=true , address=45.91.116.10:4082 , piece_id=PUYEZFZQLZCRUATWBPEP7VYU75XD4DDIABD6GBBASF4RKMKTG2FA, seg_pos={0 0}, upload=257150]: Uploading piece... Nov 21 12:03:21.157746998 (291h59m19.709217469s) [noise=true , address=145.79.227.11:4729 , piece_id=GH57FJM7JI7WN3ICOX2BCK4JQQDYORJ3JX2B7V4LDG6KALWOHX7Q, seg_pos={0 0}, upload=257150]: Uploading piece... Nov 21 12:03:21.163268371 (291h59m19.714738832s) [noise=true , address=145.79.227.16:4912 , piece_id=QCAG67ERHROPSHVUVCKEZJFK3B6K4YI7LRZWARMJCT5QXTSAHL2A, seg_pos={0 0}, upload=257150]: Uploading piece... Nov 21 12:03:21.165309478 (291h59m19.716779939s) [noise=true , address=77.90.31.18:4805 , piece_id=6LWJPVVTDVATLDHL66WS6H6KMLXL4ZRXVRZFXP2QCGYFUFRV3C4A, seg_pos={0 0}, upload=257150]: Uploading piece... Nov 21 12:03:22.765252658 (291h59m21.316723119s) [upload=257150]: Next returned nil segment Nov 21 12:03:22.765268569 (291h59m21.31673903s) [upload=257150]: Waiting for error group managing segments... Nov 21 12:03:23.146214842 (291h59m21.697685313s) [noise=true , address=145.79.227.22:4511 , piece_id=3HRMW3Y33TRQHQNCD7CFDHYXRJ2WCAGE6XDJFBITER76CCOTIFOQ, seg_pos={0 0}, upload=257150]: Done uploading piece. err: <nil> Nov 21 12:03:23.148157711 (291h59m21.699628172s) [noise=true , address=145.79.227.14:4838 , piece_id=6D5V4CF2LSODZ4NPN67KUAJTZMK3ZAYGL546WJ4NIHV5SDLUHIZA, seg_pos={0 0}, upload=257150]: Done uploading piece. err: <nil> Nov 21 12:03:23.151571021 (291h59m21.703041482s) [noise=true , address=145.79.227.25:4619 , piece_id=PMHMAYC6QSE3HJLCEAMMBHHFOYOS6XLAMPRM5M33BBIILQX2Y7GA, seg_pos={0 0}, upload=257150]: Done uploading piece. err: <nil> Nov 21 12:03:23.152335496 (291h59m21.703805977s) [noise=true , address=145.79.227.11:4729 , piece_id=GH57FJM7JI7WN3ICOX2BCK4JQQDYORJ3JX2B7V4LDG6KALWOHX7Q, seg_pos={0 0}, upload=257150]: Done uploading piece. err: <nil> Nov 21 12:03:23.153157361 (291h59m21.704627822s) [noise=true , address=145.79.227.17:4971 , piece_id=53IMICLBNCSB57326VC5XKP7CG4AQSBKKAFYKBQMELRRSZDKBSQQ, seg_pos={0 0}, upload=257150]: Done uploading piece. err: <nil> Nov 21 12:03:23.154678468 (291h59m21.706148949s) [noise=true , address=145.79.227.20:4439 , piece_id=MYM5JPZEKXUBXAND2MHK2ZDHODNYPSM6K6JJRND4IFBOS5I43GDQ, seg_pos={0 0}, upload=257150]: Done uploading piece. err: <nil> Nov 21 12:03:23.155458653 (291h59m21.706929114s) [noise=true , address=145.79.227.32:4834 , piece_id=WRCEO7VGITAJMKFS7W534X24KCALWKNGWOYP622UOI5GYF4V3OXA, seg_pos={0 0}, upload=257150]: Done uploading piece. err: <nil> Nov 21 12:03:23.166587054 (291h59m21.718057515s) [noise=true , address=145.79.227.22:4509 , piece_id=5GC5WOJYILYZ2DUIIGLZITCL4XJZ4XWXKCWXZY62C4QW5FYVHELA, seg_pos={0 0}, upload=257150]: Done uploading piece. err: <nil> Nov 21 12:03:23.167943206 (291h59m21.719413667s) [noise=true , address=145.79.227.16:4912 , piece_id=QCAG67ERHROPSHVUVCKEZJFK3B6K4YI7LRZWARMJCT5QXTSAHL2A, seg_pos={0 0}, upload=257150]: Done uploading piece. err: <nil> Nov 21 12:03:23.172565528 (291h59m21.724035989s) [noise=true , address=145.79.227.27:4689 , piece_id=X4QKAUFXGYX54435VLH7QJ2HXSYZEL4SJ6V4VBNTIV77KBUMCZIA, seg_pos={0 0}, upload=257150]: Done uploading piece. err: <nil> Nov 21 12:03:23.191491918 (291h59m21.742962379s) [noise=true , address=145.79.227.8:4329 , piece_id=NMX7FTR2Z4EFQLGOND32RE2TI3AIZ7TGGVJTYFDPP3CGFFFJV2LQ, seg_pos={0 0}, upload=257150]: Done uploading piece. err: <nil> Nov 21 12:03:23.769501866 (291h59m22.320972327s) [noise=true , address=45.91.116.10:4082 , piece_id=PUYEZFZQLZCRUATWBPEP7VYU75XD4DDIABD6GBBASF4RKMKTG2FA, seg_pos={0 0}, upload=257150]: Done uploading piece. err: <nil> Nov 21 12:03:23.881038302 (291h59m22.432508764s) [noise=true , address=77.90.31.18:4805 , piece_id=6LWJPVVTDVATLDHL66WS6H6KMLXL4ZRXVRZFXP2QCGYFUFRV3C4A, seg_pos={0 0}, upload=257150]: Done uploading piece. err: <nil> Nov 21 12:03:23.881065174 (291h59m22.432535635s) [seg_pos={0 0}, upload=257150]: Segment reached optimal threshold of 13 pieces. Nov 21 12:03:23.881321642 (291h59m22.432792103s) [noise=true , address=77.90.31.10:4519 , piece_id=KVHNESUY7ZYPBXUVZQMWJTVUF64ITT57A3AUYZQIP5CPRKIN6RGA, seg_pos={0 0}, upload=257150]: Done uploading piece. err: <nil> Nov 21 12:03:23.881450838 (291h59m22.432921299s) [noise=true , address=77.90.31.21:4877 , piece_id=AJIDGRABKTO7SN745P7K73BTXEB72PZ473XBUPDLZVDX4F7KKKNQ, seg_pos={0 0}, upload=257150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Nov 21 12:03:23.881472891 (291h59m22.432943362s) [noise=true , address=77.90.31.21:4877 , piece_id=AJIDGRABKTO7SN745P7K73BTXEB72PZ473XBUPDLZVDX4F7KKKNQ, seg_pos={0 0}, upload=257150]: Long tail cancellation due to optimal threshold Nov 21 12:03:23.881998511 (291h59m22.433468972s) [noise=true , address=45.91.116.14:4190 , piece_id=XXL6DRSFSQFP2KCZTKZMS4R24CUY3CYUP7C6B4ZR7UYMHEVVNSFA, seg_pos={0 0}, upload=257150]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Nov 21 12:03:23.882010925 (291h59m22.433481386s) [noise=true , address=45.91.116.14:4190 , piece_id=XXL6DRSFSQFP2KCZTKZMS4R24CUY3CYUP7C6B4ZR7UYMHEVVNSFA, seg_pos={0 0}, upload=257150]: Long tail cancellation due to optimal threshold Nov 21 12:03:23.882103892 (291h59m22.433574353s) [seg_pos={0 0}, upload=257150]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Nov 21 12:03:23.882311507 (291h59m22.433781978s) [upload=257150]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 21 12:03:23.882321226 (291h59m22.433791687s) [upload=257150]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 21 12:03:24.186628483 (291h59m22.738098944s) [upload=257150]: Done uploading segments... Nov 21 12:03:24.186656958 (291h59m22.738127429s) [upload=257150]: Upload finished. err: <nil> Duration: 30.003753207s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 2 out of 60 is failed (threshold=3) [X.X.........................................................] duration: 6.517035373s trace: https://jaeger.dev.storj.io/trace/a7f69ee106efb7f FAIL (2025-11-21 12:03:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== upload=256586]: Uploading piece... Nov 21 11:25:27.210346024 (291h21m25.761816485s) [noise=true , address=89.149.201.147:4008 , piece_id=HCX7I4ICPEHDWTYDFIQMKZ5EE5CGPQLYAIHF7WZMTUUU6CNZPHTA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.211425289 (291h21m25.76289575s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.213103795 (291h21m25.764574246s) [noise=true , address=185.167.204.205:4022 , piece_id=BES3YK6PYQV2AKOWSAGHF7DGGWGWSWPIJU2ICAQ5WBHKZ5VT5L6A, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.213408395 (291h21m25.764878856s) [noise=true , address=185.167.204.205:4027 , piece_id=IVPONKBF2PIHZQAW3L4YA4HOMQV5BQML6SKKCJLIEGT5SZSQ2AOA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.214501747 (291h21m25.765972208s) [noise=true , address=81.171.14.70:4042 , piece_id=XNREZEUZ5L5KZ3L7ZOL6IRK7IHU6DJN4RLEQZRFMDKDH2JWC24CQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.215013752 (291h21m25.766484213s) [noise=true , address=81.171.14.70:4034 , piece_id=ITRLKYVNFSYBLABNGNFO3HRJNMX3MWB345FIMWSIPBNAVZPA47UA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.215087572 (291h21m25.766558044s) [noise=true , address=185.167.204.205:4031 , piece_id=ZEI2YSIQC655R7WS43WADXGFT5RKT5LFSN6IZEQ6SHTDVOV7BZBQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.216432844 (291h21m25.767903305s) [noise=true , address=185.167.204.206:4056 , piece_id=EAMZIK6AVJL7IMFF5VOLW3SMCGYKSKB2VXJTNUNLJCVIT7EKXNMQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.216662281 (291h21m25.768132742s) [noise=true , address=89.149.201.147:4011 , piece_id=ZATNMD5HOK35AWGJJ5V66VDJ4YGUVZQJ2OBERQL32ZV5GW2HBSMQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.218008314 (291h21m25.769478775s) [noise=true , address=81.171.14.70:4032 , piece_id=CFH2RTR3BLBUB5ZIZCFC6UHT3FHT3B2XSCHQMKXNGDJJGPHEMCTQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:28.035150976 (291h21m26.586621437s) [upload=256586]: Next returned nil segment Nov 21 11:25:28.035169622 (291h21m26.586640083s) [upload=256586]: Waiting for error group managing segments... Nov 21 11:25:28.416646836 (291h21m26.968117317s) [noise=true , address=185.167.204.206:4064 , piece_id=SRFIWFQLRAGCNIKYLHMGT4QRUL3WH2QPLLIKAJ7DLFJEQUW5DPPA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.417338854 (291h21m26.968809325s) [noise=true , address=185.167.204.205:4033 , piece_id=FEWQVHPIKN3Q6O5AZ6AGJBLUPZ34PBZFB5X3XNSKGPOGI4AIO7JA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.421089406 (291h21m26.972559867s) [noise=true , address=185.167.204.206:4072 , piece_id=GYCO53OWGOMOHK4FHC5GENN4LTQGSZKZKOI5V534RLV3LHLPKC6Q, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.426252887 (291h21m26.977723348s) [noise=true , address=185.167.204.206:4046 , piece_id=BZ57ZA35ITEOA6CAZJGNODK5MUTXEZWDV7E2OGBZOSMS4OQZYAEA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.431720689 (291h21m26.98319115s) [noise=true , address=185.167.204.205:4027 , piece_id=IVPONKBF2PIHZQAW3L4YA4HOMQV5BQML6SKKCJLIEGT5SZSQ2AOA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.438243951 (291h21m26.989714412s) [noise=true , address=185.167.204.205:4022 , piece_id=BES3YK6PYQV2AKOWSAGHF7DGGWGWSWPIJU2ICAQ5WBHKZ5VT5L6A, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.443364331 (291h21m26.994834792s) [noise=true , address=185.167.204.206:4056 , piece_id=EAMZIK6AVJL7IMFF5VOLW3SMCGYKSKB2VXJTNUNLJCVIT7EKXNMQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.445594728 (291h21m26.997065179s) [noise=true , address=185.167.204.205:4016 , piece_id=EHQS3546UXRXTL2MGFCSRTPDDPAYXEYLMISPP5AWY337MHX7HFLQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.449325822 (291h21m27.000796283s) [noise=true , address=185.167.204.205:4031 , piece_id=ZEI2YSIQC655R7WS43WADXGFT5RKT5LFSN6IZEQ6SHTDVOV7BZBQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.449761211 (291h21m27.001231672s) [noise=true , address=185.167.204.205:4018 , piece_id=7OSLPWJLKFTIJPZEAFMZOAH6JAAMX6HKROI25FODFYBPNJWSMLWA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.450419845 (291h21m27.001890306s) [noise=true , address=185.167.204.206:4062 , piece_id=UF3S6NAVGS2KQJPGA5WXP7YG4RQO44P445SNPCWXKXMONZAYUTNQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.453201252 (291h21m27.004671713s) [noise=true , address=185.167.204.205:4007 , piece_id=MEBGOPZ6MFMQY4UZARS2LKJKEMNN5YGSLKPJGTWSKS45CUNXSRGQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.661209626 (291h21m28.212680087s) [noise=true , address=89.149.201.147:4016 , piece_id=TGRWGGGBKDANC7YKROWPRMIYNAVFEVE3RXOSUOINFI7KTRS2IYUQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.693169182 (291h21m28.244639643s) [noise=true , address=89.149.201.147:4002 , piece_id=UIYJSCMY3LCLRHZUJB6JFJHHXKMYTVKERTZW7F7X44MWJETPY6XA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.721204497 (291h21m28.272674958s) [noise=true , address=89.149.201.147:4008 , piece_id=HCX7I4ICPEHDWTYDFIQMKZ5EE5CGPQLYAIHF7WZMTUUU6CNZPHTA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.764584711 (291h21m28.316055172s) [noise=true , address=89.149.201.147:4001 , piece_id=QUS6NB3NQBLJFB6YCK3EBDPKCTWEWGKAT3NOR24TVI65SQKHWDOQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.768284075 (291h21m28.319754536s) [noise=true , address=89.149.201.147:4011 , piece_id=ZATNMD5HOK35AWGJJ5V66VDJ4YGUVZQJ2OBERQL32ZV5GW2HBSMQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.789681961 (291h21m28.341152422s) [noise=true , address=89.149.201.147:4003 , piece_id=RUAYGN2RGF5JJTWFEGIHWH3BHEUDSS5DN2QJ6YLSHPF5BGUXD3RA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:37.298598682 (291h21m35.850069143s) [noise=true , address=81.171.14.70:4043 , piece_id=T3NF6JR3VJIGDMPNUBLMJ3AOTQP5IHOGJYORMCSS7ZMOENQETZ7A, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:37.497478356 (291h21m36.048948818s) [noise=true , address=81.171.14.70:4032 , piece_id=CFH2RTR3BLBUB5ZIZCFC6UHT3FHT3B2XSCHQMKXNGDJJGPHEMCTQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:40.406166933 (291h21m38.957637394s) [noise=true , address=81.171.14.70:4045 , piece_id=BRWQ7G3E547PJE2RLGSESIB3HMP5C4M5NSKAJNGWSFXUBEO6SYEQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:42.511041157 (291h21m41.062511608s) [noise=true , address=81.171.14.70:4029 , piece_id=DKJM5E7EY7LOSBSEBQOZRC6CI5ZPPE47HFOFQJROPCUMUV567CAA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:12.012682521 (291h22m10.564152982s) [noise=true , address=81.171.14.70:4042 , piece_id=XNREZEUZ5L5KZ3L7ZOL6IRK7IHU6DJN4RLEQZRFMDKDH2JWC24CQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:15.596046283 (291h22m14.147516744s) [noise=true , address=81.171.14.70:4033 , piece_id=JRKPD4EP6FZM5HISOIDJYZY7FAWWQCADAS47FGW2JM5A7RFJR2BQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:18.453059659 (291h22m17.00453012s) [noise=true , address=81.171.14.70:4046 , piece_id=OSONRSF6KJS7F2QJVCC3FPZK4AV7QPG5YSM47CW5LIW7EME44AWA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:23.442583344 (291h22m21.994053805s) [noise=true , address=81.171.14.70:4034 , piece_id=ITRLKYVNFSYBLABNGNFO3HRJNMX3MWB345FIMWSIPBNAVZPA47UA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:23.442621967 (291h22m21.994092428s) [seg_pos={0 0}, upload=256586]: Segment reached optimal threshold of 26 pieces. Nov 21 11:26:23.442836976 (291h22m21.994307437s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MUVGecysnTSFcJoHUL7QMWb3h6dtXgUd7FbA672fQCyEWhePr): protocol: expected piece hash; context canceled; EOF Nov 21 11:26:23.442845934 (291h22m21.994316395s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Long tail cancellation due to optimal threshold Nov 21 11:26:23.443227491 (291h22m21.994697942s) [noise=true , address=81.171.14.70:4027 , piece_id=4FDFCCWF3EIROQAFLOHGS743523OYGSXWQSPQEVD3AMQ6WKDQI7Q, seg_pos={0 0}, upload=256586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122cYvgTxSvC2aEXnBkRk2o8ARTBXj9gqPoa12vW8hogSN6kpA4): protocol: expected piece hash; context canceled; EOF Nov 21 11:26:23.443237770 (291h22m21.994708231s) [noise=true , address=81.171.14.70:4027 , piece_id=4FDFCCWF3EIROQAFLOHGS743523OYGSXWQSPQEVD3AMQ6WKDQI7Q, seg_pos={0 0}, upload=256586]: Long tail cancellation due to optimal threshold Nov 21 11:26:23.443274720 (291h22m21.994745181s) [seg_pos={0 0}, upload=256586]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Nov 21 11:26:23.443378929 (291h22m21.99484939s) [upload=256586]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 21 11:26:23.443387044 (291h22m21.994857495s) [upload=256586]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 21 11:26:23.685139626 (291h22m22.236610077s) [upload=256586]: Done uploading segments... Nov 21 11:26:23.685160676 (291h22m22.236631127s) [upload=256586]: Upload finished. err: <nil> Duration: 6.517075438s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 26 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXX..................................] duration: 3.709130076s trace: https://jaeger.dev.storj.io/trace/6af15a3a67b655d5 FAIL (2025-11-21 12:03:33 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 3.709170252s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 27 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXX.................................] duration: 3.478545369s trace: https://jaeger.dev.storj.io/trace/6af15a3a67b655d5 FAIL (2025-11-21 12:03:06 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 3.478577592s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 2 out of 60 is failed (threshold=3) [.X.X........................................................] duration: 5.867075239s trace: https://jaeger.dev.storj.io/trace/a7f69ee106efb7f FAIL (2025-11-21 12:02:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== upload=256586]: Uploading piece... Nov 21 11:25:27.210346024 (291h21m25.761816485s) [noise=true , address=89.149.201.147:4008 , piece_id=HCX7I4ICPEHDWTYDFIQMKZ5EE5CGPQLYAIHF7WZMTUUU6CNZPHTA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.211425289 (291h21m25.76289575s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.213103795 (291h21m25.764574246s) [noise=true , address=185.167.204.205:4022 , piece_id=BES3YK6PYQV2AKOWSAGHF7DGGWGWSWPIJU2ICAQ5WBHKZ5VT5L6A, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.213408395 (291h21m25.764878856s) [noise=true , address=185.167.204.205:4027 , piece_id=IVPONKBF2PIHZQAW3L4YA4HOMQV5BQML6SKKCJLIEGT5SZSQ2AOA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.214501747 (291h21m25.765972208s) [noise=true , address=81.171.14.70:4042 , piece_id=XNREZEUZ5L5KZ3L7ZOL6IRK7IHU6DJN4RLEQZRFMDKDH2JWC24CQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.215013752 (291h21m25.766484213s) [noise=true , address=81.171.14.70:4034 , piece_id=ITRLKYVNFSYBLABNGNFO3HRJNMX3MWB345FIMWSIPBNAVZPA47UA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.215087572 (291h21m25.766558044s) [noise=true , address=185.167.204.205:4031 , piece_id=ZEI2YSIQC655R7WS43WADXGFT5RKT5LFSN6IZEQ6SHTDVOV7BZBQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.216432844 (291h21m25.767903305s) [noise=true , address=185.167.204.206:4056 , piece_id=EAMZIK6AVJL7IMFF5VOLW3SMCGYKSKB2VXJTNUNLJCVIT7EKXNMQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.216662281 (291h21m25.768132742s) [noise=true , address=89.149.201.147:4011 , piece_id=ZATNMD5HOK35AWGJJ5V66VDJ4YGUVZQJ2OBERQL32ZV5GW2HBSMQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.218008314 (291h21m25.769478775s) [noise=true , address=81.171.14.70:4032 , piece_id=CFH2RTR3BLBUB5ZIZCFC6UHT3FHT3B2XSCHQMKXNGDJJGPHEMCTQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:28.035150976 (291h21m26.586621437s) [upload=256586]: Next returned nil segment Nov 21 11:25:28.035169622 (291h21m26.586640083s) [upload=256586]: Waiting for error group managing segments... Nov 21 11:25:28.416646836 (291h21m26.968117317s) [noise=true , address=185.167.204.206:4064 , piece_id=SRFIWFQLRAGCNIKYLHMGT4QRUL3WH2QPLLIKAJ7DLFJEQUW5DPPA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.417338854 (291h21m26.968809325s) [noise=true , address=185.167.204.205:4033 , piece_id=FEWQVHPIKN3Q6O5AZ6AGJBLUPZ34PBZFB5X3XNSKGPOGI4AIO7JA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.421089406 (291h21m26.972559867s) [noise=true , address=185.167.204.206:4072 , piece_id=GYCO53OWGOMOHK4FHC5GENN4LTQGSZKZKOI5V534RLV3LHLPKC6Q, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.426252887 (291h21m26.977723348s) [noise=true , address=185.167.204.206:4046 , piece_id=BZ57ZA35ITEOA6CAZJGNODK5MUTXEZWDV7E2OGBZOSMS4OQZYAEA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.431720689 (291h21m26.98319115s) [noise=true , address=185.167.204.205:4027 , piece_id=IVPONKBF2PIHZQAW3L4YA4HOMQV5BQML6SKKCJLIEGT5SZSQ2AOA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.438243951 (291h21m26.989714412s) [noise=true , address=185.167.204.205:4022 , piece_id=BES3YK6PYQV2AKOWSAGHF7DGGWGWSWPIJU2ICAQ5WBHKZ5VT5L6A, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.443364331 (291h21m26.994834792s) [noise=true , address=185.167.204.206:4056 , piece_id=EAMZIK6AVJL7IMFF5VOLW3SMCGYKSKB2VXJTNUNLJCVIT7EKXNMQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.445594728 (291h21m26.997065179s) [noise=true , address=185.167.204.205:4016 , piece_id=EHQS3546UXRXTL2MGFCSRTPDDPAYXEYLMISPP5AWY337MHX7HFLQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.449325822 (291h21m27.000796283s) [noise=true , address=185.167.204.205:4031 , piece_id=ZEI2YSIQC655R7WS43WADXGFT5RKT5LFSN6IZEQ6SHTDVOV7BZBQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.449761211 (291h21m27.001231672s) [noise=true , address=185.167.204.205:4018 , piece_id=7OSLPWJLKFTIJPZEAFMZOAH6JAAMX6HKROI25FODFYBPNJWSMLWA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.450419845 (291h21m27.001890306s) [noise=true , address=185.167.204.206:4062 , piece_id=UF3S6NAVGS2KQJPGA5WXP7YG4RQO44P445SNPCWXKXMONZAYUTNQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.453201252 (291h21m27.004671713s) [noise=true , address=185.167.204.205:4007 , piece_id=MEBGOPZ6MFMQY4UZARS2LKJKEMNN5YGSLKPJGTWSKS45CUNXSRGQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.661209626 (291h21m28.212680087s) [noise=true , address=89.149.201.147:4016 , piece_id=TGRWGGGBKDANC7YKROWPRMIYNAVFEVE3RXOSUOINFI7KTRS2IYUQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.693169182 (291h21m28.244639643s) [noise=true , address=89.149.201.147:4002 , piece_id=UIYJSCMY3LCLRHZUJB6JFJHHXKMYTVKERTZW7F7X44MWJETPY6XA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.721204497 (291h21m28.272674958s) [noise=true , address=89.149.201.147:4008 , piece_id=HCX7I4ICPEHDWTYDFIQMKZ5EE5CGPQLYAIHF7WZMTUUU6CNZPHTA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.764584711 (291h21m28.316055172s) [noise=true , address=89.149.201.147:4001 , piece_id=QUS6NB3NQBLJFB6YCK3EBDPKCTWEWGKAT3NOR24TVI65SQKHWDOQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.768284075 (291h21m28.319754536s) [noise=true , address=89.149.201.147:4011 , piece_id=ZATNMD5HOK35AWGJJ5V66VDJ4YGUVZQJ2OBERQL32ZV5GW2HBSMQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.789681961 (291h21m28.341152422s) [noise=true , address=89.149.201.147:4003 , piece_id=RUAYGN2RGF5JJTWFEGIHWH3BHEUDSS5DN2QJ6YLSHPF5BGUXD3RA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:37.298598682 (291h21m35.850069143s) [noise=true , address=81.171.14.70:4043 , piece_id=T3NF6JR3VJIGDMPNUBLMJ3AOTQP5IHOGJYORMCSS7ZMOENQETZ7A, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:37.497478356 (291h21m36.048948818s) [noise=true , address=81.171.14.70:4032 , piece_id=CFH2RTR3BLBUB5ZIZCFC6UHT3FHT3B2XSCHQMKXNGDJJGPHEMCTQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:40.406166933 (291h21m38.957637394s) [noise=true , address=81.171.14.70:4045 , piece_id=BRWQ7G3E547PJE2RLGSESIB3HMP5C4M5NSKAJNGWSFXUBEO6SYEQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:42.511041157 (291h21m41.062511608s) [noise=true , address=81.171.14.70:4029 , piece_id=DKJM5E7EY7LOSBSEBQOZRC6CI5ZPPE47HFOFQJROPCUMUV567CAA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:12.012682521 (291h22m10.564152982s) [noise=true , address=81.171.14.70:4042 , piece_id=XNREZEUZ5L5KZ3L7ZOL6IRK7IHU6DJN4RLEQZRFMDKDH2JWC24CQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:15.596046283 (291h22m14.147516744s) [noise=true , address=81.171.14.70:4033 , piece_id=JRKPD4EP6FZM5HISOIDJYZY7FAWWQCADAS47FGW2JM5A7RFJR2BQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:18.453059659 (291h22m17.00453012s) [noise=true , address=81.171.14.70:4046 , piece_id=OSONRSF6KJS7F2QJVCC3FPZK4AV7QPG5YSM47CW5LIW7EME44AWA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:23.442583344 (291h22m21.994053805s) [noise=true , address=81.171.14.70:4034 , piece_id=ITRLKYVNFSYBLABNGNFO3HRJNMX3MWB345FIMWSIPBNAVZPA47UA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:23.442621967 (291h22m21.994092428s) [seg_pos={0 0}, upload=256586]: Segment reached optimal threshold of 26 pieces. Nov 21 11:26:23.442836976 (291h22m21.994307437s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MUVGecysnTSFcJoHUL7QMWb3h6dtXgUd7FbA672fQCyEWhePr): protocol: expected piece hash; context canceled; EOF Nov 21 11:26:23.442845934 (291h22m21.994316395s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Long tail cancellation due to optimal threshold Nov 21 11:26:23.443227491 (291h22m21.994697942s) [noise=true , address=81.171.14.70:4027 , piece_id=4FDFCCWF3EIROQAFLOHGS743523OYGSXWQSPQEVD3AMQ6WKDQI7Q, seg_pos={0 0}, upload=256586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122cYvgTxSvC2aEXnBkRk2o8ARTBXj9gqPoa12vW8hogSN6kpA4): protocol: expected piece hash; context canceled; EOF Nov 21 11:26:23.443237770 (291h22m21.994708231s) [noise=true , address=81.171.14.70:4027 , piece_id=4FDFCCWF3EIROQAFLOHGS743523OYGSXWQSPQEVD3AMQ6WKDQI7Q, seg_pos={0 0}, upload=256586]: Long tail cancellation due to optimal threshold Nov 21 11:26:23.443274720 (291h22m21.994745181s) [seg_pos={0 0}, upload=256586]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Nov 21 11:26:23.443378929 (291h22m21.99484939s) [upload=256586]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 21 11:26:23.443387044 (291h22m21.994857495s) [upload=256586]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 21 11:26:23.685139626 (291h22m22.236610077s) [upload=256586]: Done uploading segments... Nov 21 11:26:23.685160676 (291h22m22.236631127s) [upload=256586]: Upload finished. err: <nil> Duration: 5.867108933s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 28 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXX................................] duration: 3.698777922s trace: https://jaeger.dev.storj.io/trace/6af15a3a67b655d5 FAIL (2025-11-21 12:02:40 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 3.698803841s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [...X.X.X....X............X....X.X.X..X.X..X...X..XXX.X.XX...] duration: 5.917425488s trace: https://jaeger.dev.storj.io/trace/2e92d97344640fc3 FAIL (2025-11-21 12:02:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 21 12:00:44.413717038 (291h56m42.965187489s) [upload=257111]: Starting upload Nov 21 12:00:44.413777513 (291h56m42.965247974s) [upload=257111]: Done starting upload Nov 21 12:00:44.413945853 (291h56m42.965416314s) [upload=257111]: Uploading segments... Nov 21 12:00:44.414062085 (291h56m42.965532536s) [upload=257111]: Got next segment. Inline: false Nov 21 12:00:44.414124955 (291h56m42.965595416s) [upload=257111]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 12:00:44.414131217 (291h56m42.965601678s) [upload=257111]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 12:00:44.659711608 (291h56m43.21118208s) [seg_pos={0 0}, upload=257111]: Begin upload segment... Nov 21 12:00:44.659783275 (291h56m43.211253736s) [seg_pos={0 0}, upload=257111]: Done begin upload segment. Nov 21 12:00:44.659821467 (291h56m43.211291919s) [noise=true , address=45.91.116.14:4190 , piece_id=EK762P55UQ2NFROSHJYUYIETP5CBQZJHKOTJRUFBKNPTRG7IQCYA, seg_pos={0 0}, upload=257111]: Uploading piece... Nov 21 12:00:44.661076968 (291h56m43.212547429s) [noise=true , address=145.79.227.15:4876 , piece_id=42VLAZCDHN6CYRZ5EMVPLIVJKQ7TZFJMXFA2AWHUU6S7P37ISMNQ, seg_pos={0 0}, upload=257111]: Uploading piece... Nov 21 12:00:44.661623990 (291h56m43.213094451s) [noise=true , address=77.90.31.13:4628 , piece_id=KV2WK5KY4B4N3NHYJ6UBAKBLRZWRIFMTVV65SWS2FCUF7UV2AJMQ, seg_pos={0 0}, upload=257111]: Uploading piece... Nov 21 12:00:44.662015575 (291h56m43.213486036s) [noise=true , address=145.79.227.23:4547 , piece_id=R7V3ZYNZBPLABNLWIB35BHRKZKFPBQ4RS2G5ZDIOBNVCFFLODDZA, seg_pos={0 0}, upload=257111]: Uploading piece... Nov 21 12:00:44.662104145 (291h56m43.213574626s) [noise=true , address=145.79.227.27:4690 , piece_id=EF7B5TVERZJ5RQWZMXXFWHQQXPIAKKTGEU5IRKQMNY4GIOZ7PMOQ, seg_pos={0 0}, upload=257111]: Uploading piece... Nov 21 12:00:44.662331166 (291h56m43.213801617s) [noise=true , address=77.90.31.10:4518 , piece_id=B4KCY7UJHBDHWMLOX7NKAQ75PQUTZTXEDCWEXR43EOY6QFYA3VTA, seg_pos={0 0}, upload=257111]: Uploading piece... Nov 21 12:00:44.662744884 (291h56m43.214215345s) [noise=true , address=145.79.227.24:4583 , piece_id=KJPKPFRWFZD5H7G3RNN2VYLP3SUHBWO3QRUR4T5J6RFYWUSZSSRA, seg_pos={0 0}, upload=257111]: Uploading piece... Nov 21 12:00:44.662792254 (291h56m43.214262715s) [noise=true , address=145.79.227.11:4731 , piece_id=NZEOWNXMOAIGI5QWGYVJHR4GKK5E7CIUVHGYPY7UBEKQMAIGQPCQ, seg_pos={0 0}, upload=257111]: Uploading piece... Nov 21 12:00:44.663124828 (291h56m43.214595289s) [noise=true , address=145.79.227.20:4438 , piece_id=DEKU5U5LMX3WRZ64WKE3MUQC3C2GBVTXPFVVEX6DN5PSEE53Q5DA, seg_pos={0 0}, upload=257111]: Uploading piece... Nov 21 12:00:44.663330720 (291h56m43.214801171s) [noise=true , address=145.79.227.30:4763 , piece_id=7WWH65VO7HUBZFP3ATOKMDTPY6HLJVWIA2XHVI2K2V46FEPK5EPA, seg_pos={0 0}, upload=257111]: Uploading piece... Nov 21 12:00:44.663457621 (291h56m43.214928082s) [noise=true , address=145.79.227.22:4511 , piece_id=4MDCBBI4LD4BQQ76YTC75IRLH4HNVG7GEBGBIFDI7Q2TTEHNOBXA, seg_pos={0 0}, upload=257111]: Uploading piece... Nov 21 12:00:44.663871199 (291h56m43.21534165s) [noise=true , address=145.79.227.28:4727 , piece_id=4W63CG2IW33MVQHJJJGPJGKZFA4JKBZMHHG5OMEKJ3Y63LVV25FQ, seg_pos={0 0}, upload=257111]: Uploading piece... Nov 21 12:00:44.663892760 (291h56m43.215363221s) [noise=true , address=45.91.116.8:4048 , piece_id=OUL6TXMXPESMVCTC7RKYMMGZA7ZIWMNMG4B5IWSHYFNMP2PC5PCA, seg_pos={0 0}, upload=257111]: Uploading piece... Nov 21 12:00:44.663946742 (291h56m43.215417203s) [noise=true , address=77.90.31.15:4698 , piece_id=ZQ5SBBVFVQQRRPRWQOC5MNLVROAAW3UNJOO3XXESCKCN2ELA5BIA, seg_pos={0 0}, upload=257111]: Uploading piece... Nov 21 12:00:44.664546976 (291h56m43.216017437s) [noise=true , address=145.79.227.31:4798 , piece_id=YWX2POKFFJISTJ5Y6WY5SHP3XIUVNH5CP24AXKU6BLW42RYWKROQ, seg_pos={0 0}, upload=257111]: Uploading piece... Nov 21 12:00:44.665139685 (291h56m43.216610136s) [noise=true , address=77.90.31.17:4770 , piece_id=I7AQRIKKA7FVGTOPEJODHU5B6Z26GHWZXCFQQDXHFRYMHUQVJYZA, seg_pos={0 0}, upload=257111]: Uploading piece... Nov 21 12:00:46.269248547 (291h56m44.820718998s) [upload=257111]: Next returned nil segment Nov 21 12:00:46.269292731 (291h56m44.820763192s) [upload=257111]: Waiting for error group managing segments... Nov 21 12:00:46.651938048 (291h56m45.203408509s) [noise=true , address=145.79.227.11:4731 , piece_id=NZEOWNXMOAIGI5QWGYVJHR4GKK5E7CIUVHGYPY7UBEKQMAIGQPCQ, seg_pos={0 0}, upload=257111]: Done uploading piece. err: <nil> Nov 21 12:00:46.653674856 (291h56m45.205145317s) [noise=true , address=145.79.227.27:4690 , piece_id=EF7B5TVERZJ5RQWZMXXFWHQQXPIAKKTGEU5IRKQMNY4GIOZ7PMOQ, seg_pos={0 0}, upload=257111]: Done uploading piece. err: <nil> Nov 21 12:00:46.656724623 (291h56m45.208195084s) [noise=true , address=145.79.227.22:4511 , piece_id=4MDCBBI4LD4BQQ76YTC75IRLH4HNVG7GEBGBIFDI7Q2TTEHNOBXA, seg_pos={0 0}, upload=257111]: Done uploading piece. err: <nil> Nov 21 12:00:46.663448046 (291h56m45.214918497s) [noise=true , address=145.79.227.23:4547 , piece_id=R7V3ZYNZBPLABNLWIB35BHRKZKFPBQ4RS2G5ZDIOBNVCFFLODDZA, seg_pos={0 0}, upload=257111]: Done uploading piece. err: <nil> Nov 21 12:00:46.672901036 (291h56m45.224371497s) [noise=true , address=145.79.227.24:4583 , piece_id=KJPKPFRWFZD5H7G3RNN2VYLP3SUHBWO3QRUR4T5J6RFYWUSZSSRA, seg_pos={0 0}, upload=257111]: Done uploading piece. err: <nil> Nov 21 12:00:46.677309982 (291h56m45.228780463s) [noise=true , address=145.79.227.31:4798 , piece_id=YWX2POKFFJISTJ5Y6WY5SHP3XIUVNH5CP24AXKU6BLW42RYWKROQ, seg_pos={0 0}, upload=257111]: Done uploading piece. err: <nil> Nov 21 12:00:46.678373266 (291h56m45.229843727s) [noise=true , address=145.79.227.28:4727 , piece_id=4W63CG2IW33MVQHJJJGPJGKZFA4JKBZMHHG5OMEKJ3Y63LVV25FQ, seg_pos={0 0}, upload=257111]: Done uploading piece. err: <nil> Nov 21 12:00:46.686989964 (291h56m45.238460425s) [noise=true , address=145.79.227.20:4438 , piece_id=DEKU5U5LMX3WRZ64WKE3MUQC3C2GBVTXPFVVEX6DN5PSEE53Q5DA, seg_pos={0 0}, upload=257111]: Done uploading piece. err: <nil> Nov 21 12:00:46.690017299 (291h56m45.24148776s) [noise=true , address=145.79.227.15:4876 , piece_id=42VLAZCDHN6CYRZ5EMVPLIVJKQ7TZFJMXFA2AWHUU6S7P37ISMNQ, seg_pos={0 0}, upload=257111]: Done uploading piece. err: <nil> Nov 21 12:00:46.693246267 (291h56m45.244716728s) [noise=true , address=145.79.227.30:4763 , piece_id=7WWH65VO7HUBZFP3ATOKMDTPY6HLJVWIA2XHVI2K2V46FEPK5EPA, seg_pos={0 0}, upload=257111]: Done uploading piece. err: <nil> Nov 21 12:00:47.227653148 (291h56m45.779123609s) [noise=true , address=77.90.31.17:4770 , piece_id=I7AQRIKKA7FVGTOPEJODHU5B6Z26GHWZXCFQQDXHFRYMHUQVJYZA, seg_pos={0 0}, upload=257111]: Done uploading piece. err: <nil> Nov 21 12:00:47.255119890 (291h56m45.806590351s) [noise=true , address=77.90.31.13:4628 , piece_id=KV2WK5KY4B4N3NHYJ6UBAKBLRZWRIFMTVV65SWS2FCUF7UV2AJMQ, seg_pos={0 0}, upload=257111]: Done uploading piece. err: <nil> Nov 21 12:00:47.278989602 (291h56m45.830460064s) [noise=true , address=77.90.31.10:4518 , piece_id=B4KCY7UJHBDHWMLOX7NKAQ75PQUTZTXEDCWEXR43EOY6QFYA3VTA, seg_pos={0 0}, upload=257111]: Done uploading piece. err: <nil> Nov 21 12:00:47.279015171 (291h56m45.830485632s) [seg_pos={0 0}, upload=257111]: Segment reached optimal threshold of 13 pieces. Nov 21 12:00:47.279217887 (291h56m45.830688348s) [noise=true , address=77.90.31.15:4698 , piece_id=ZQ5SBBVFVQQRRPRWQOC5MNLVROAAW3UNJOO3XXESCKCN2ELA5BIA, seg_pos={0 0}, upload=257111]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj): protocol: expected piece hash; context canceled; EOF Nov 21 12:00:47.279226153 (291h56m45.830696614s) [noise=true , address=77.90.31.15:4698 , piece_id=ZQ5SBBVFVQQRRPRWQOC5MNLVROAAW3UNJOO3XXESCKCN2ELA5BIA, seg_pos={0 0}, upload=257111]: Long tail cancellation due to optimal threshold Nov 21 12:00:47.279513941 (291h56m45.830984392s) [noise=true , address=45.91.116.14:4190 , piece_id=EK762P55UQ2NFROSHJYUYIETP5CBQZJHKOTJRUFBKNPTRG7IQCYA, seg_pos={0 0}, upload=257111]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Nov 21 12:00:47.279522787 (291h56m45.830993249s) [noise=true , address=45.91.116.14:4190 , piece_id=EK762P55UQ2NFROSHJYUYIETP5CBQZJHKOTJRUFBKNPTRG7IQCYA, seg_pos={0 0}, upload=257111]: Long tail cancellation due to optimal threshold Nov 21 12:00:47.279754489 (291h56m45.83122495s) [noise=true , address=45.91.116.8:4048 , piece_id=OUL6TXMXPESMVCTC7RKYMMGZA7ZIWMNMG4B5IWSHYFNMP2PC5PCA, seg_pos={0 0}, upload=257111]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: read tcp 188.166.225.170:53244->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 21 12:00:47.279762644 (291h56m45.831233105s) [noise=true , address=45.91.116.8:4048 , piece_id=OUL6TXMXPESMVCTC7RKYMMGZA7ZIWMNMG4B5IWSHYFNMP2PC5PCA, seg_pos={0 0}, upload=257111]: Long tail cancellation due to optimal threshold Nov 21 12:00:47.279782472 (291h56m45.831252934s) [seg_pos={0 0}, upload=257111]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 21 12:00:47.279865650 (291h56m45.831336102s) [upload=257111]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 21 12:00:47.279870900 (291h56m45.831341361s) [upload=257111]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 21 12:00:47.570435510 (291h56m46.121905962s) [upload=257111]: Done uploading segments... Nov 21 12:00:47.570460016 (291h56m46.121930478s) [upload=257111]: Upload finished. err: <nil> Duration: 5.917455255s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [X...X.X.X....X............X....X.X.X..X.X..X...X..XXX.X.XX..] duration: 7.15520731s trace: https://jaeger.dev.storj.io/trace/2e92d97344640fc3 FAIL (2025-11-21 12:02:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 21 12:00:44.413717038 (291h56m42.965187489s) [upload=257111]: Starting upload Nov 21 12:00:44.413777513 (291h56m42.965247974s) [upload=257111]: Done starting upload Nov 21 12:00:44.413945853 (291h56m42.965416314s) [upload=257111]: Uploading segments... Nov 21 12:00:44.414062085 (291h56m42.965532536s) [upload=257111]: Got next segment. Inline: false Nov 21 12:00:44.414124955 (291h56m42.965595416s) [upload=257111]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 12:00:44.414131217 (291h56m42.965601678s) [upload=257111]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 12:00:44.659711608 (291h56m43.21118208s) [seg_pos={0 0}, upload=257111]: Begin upload segment... Nov 21 12:00:44.659783275 (291h56m43.211253736s) [seg_pos={0 0}, upload=257111]: Done begin upload segment. Nov 21 12:00:44.659821467 (291h56m43.211291919s) [noise=true , address=45.91.116.14:4190 , piece_id=EK762P55UQ2NFROSHJYUYIETP5CBQZJHKOTJRUFBKNPTRG7IQCYA, seg_pos={0 0}, upload=257111]: Uploading piece... Nov 21 12:00:44.661076968 (291h56m43.212547429s) [noise=true , address=145.79.227.15:4876 , piece_id=42VLAZCDHN6CYRZ5EMVPLIVJKQ7TZFJMXFA2AWHUU6S7P37ISMNQ, seg_pos={0 0}, upload=257111]: Uploading piece... Nov 21 12:00:44.661623990 (291h56m43.213094451s) [noise=true , address=77.90.31.13:4628 , piece_id=KV2WK5KY4B4N3NHYJ6UBAKBLRZWRIFMTVV65SWS2FCUF7UV2AJMQ, seg_pos={0 0}, upload=257111]: Uploading piece... Nov 21 12:00:44.662015575 (291h56m43.213486036s) [noise=true , address=145.79.227.23:4547 , piece_id=R7V3ZYNZBPLABNLWIB35BHRKZKFPBQ4RS2G5ZDIOBNVCFFLODDZA, seg_pos={0 0}, upload=257111]: Uploading piece... Nov 21 12:00:44.662104145 (291h56m43.213574626s) [noise=true , address=145.79.227.27:4690 , piece_id=EF7B5TVERZJ5RQWZMXXFWHQQXPIAKKTGEU5IRKQMNY4GIOZ7PMOQ, seg_pos={0 0}, upload=257111]: Uploading piece... Nov 21 12:00:44.662331166 (291h56m43.213801617s) [noise=true , address=77.90.31.10:4518 , piece_id=B4KCY7UJHBDHWMLOX7NKAQ75PQUTZTXEDCWEXR43EOY6QFYA3VTA, seg_pos={0 0}, upload=257111]: Uploading piece... Nov 21 12:00:44.662744884 (291h56m43.214215345s) [noise=true , address=145.79.227.24:4583 , piece_id=KJPKPFRWFZD5H7G3RNN2VYLP3SUHBWO3QRUR4T5J6RFYWUSZSSRA, seg_pos={0 0}, upload=257111]: Uploading piece... Nov 21 12:00:44.662792254 (291h56m43.214262715s) [noise=true , address=145.79.227.11:4731 , piece_id=NZEOWNXMOAIGI5QWGYVJHR4GKK5E7CIUVHGYPY7UBEKQMAIGQPCQ, seg_pos={0 0}, upload=257111]: Uploading piece... Nov 21 12:00:44.663124828 (291h56m43.214595289s) [noise=true , address=145.79.227.20:4438 , piece_id=DEKU5U5LMX3WRZ64WKE3MUQC3C2GBVTXPFVVEX6DN5PSEE53Q5DA, seg_pos={0 0}, upload=257111]: Uploading piece... Nov 21 12:00:44.663330720 (291h56m43.214801171s) [noise=true , address=145.79.227.30:4763 , piece_id=7WWH65VO7HUBZFP3ATOKMDTPY6HLJVWIA2XHVI2K2V46FEPK5EPA, seg_pos={0 0}, upload=257111]: Uploading piece... Nov 21 12:00:44.663457621 (291h56m43.214928082s) [noise=true , address=145.79.227.22:4511 , piece_id=4MDCBBI4LD4BQQ76YTC75IRLH4HNVG7GEBGBIFDI7Q2TTEHNOBXA, seg_pos={0 0}, upload=257111]: Uploading piece... Nov 21 12:00:44.663871199 (291h56m43.21534165s) [noise=true , address=145.79.227.28:4727 , piece_id=4W63CG2IW33MVQHJJJGPJGKZFA4JKBZMHHG5OMEKJ3Y63LVV25FQ, seg_pos={0 0}, upload=257111]: Uploading piece... Nov 21 12:00:44.663892760 (291h56m43.215363221s) [noise=true , address=45.91.116.8:4048 , piece_id=OUL6TXMXPESMVCTC7RKYMMGZA7ZIWMNMG4B5IWSHYFNMP2PC5PCA, seg_pos={0 0}, upload=257111]: Uploading piece... Nov 21 12:00:44.663946742 (291h56m43.215417203s) [noise=true , address=77.90.31.15:4698 , piece_id=ZQ5SBBVFVQQRRPRWQOC5MNLVROAAW3UNJOO3XXESCKCN2ELA5BIA, seg_pos={0 0}, upload=257111]: Uploading piece... Nov 21 12:00:44.664546976 (291h56m43.216017437s) [noise=true , address=145.79.227.31:4798 , piece_id=YWX2POKFFJISTJ5Y6WY5SHP3XIUVNH5CP24AXKU6BLW42RYWKROQ, seg_pos={0 0}, upload=257111]: Uploading piece... Nov 21 12:00:44.665139685 (291h56m43.216610136s) [noise=true , address=77.90.31.17:4770 , piece_id=I7AQRIKKA7FVGTOPEJODHU5B6Z26GHWZXCFQQDXHFRYMHUQVJYZA, seg_pos={0 0}, upload=257111]: Uploading piece... Nov 21 12:00:46.269248547 (291h56m44.820718998s) [upload=257111]: Next returned nil segment Nov 21 12:00:46.269292731 (291h56m44.820763192s) [upload=257111]: Waiting for error group managing segments... Nov 21 12:00:46.651938048 (291h56m45.203408509s) [noise=true , address=145.79.227.11:4731 , piece_id=NZEOWNXMOAIGI5QWGYVJHR4GKK5E7CIUVHGYPY7UBEKQMAIGQPCQ, seg_pos={0 0}, upload=257111]: Done uploading piece. err: <nil> Nov 21 12:00:46.653674856 (291h56m45.205145317s) [noise=true , address=145.79.227.27:4690 , piece_id=EF7B5TVERZJ5RQWZMXXFWHQQXPIAKKTGEU5IRKQMNY4GIOZ7PMOQ, seg_pos={0 0}, upload=257111]: Done uploading piece. err: <nil> Nov 21 12:00:46.656724623 (291h56m45.208195084s) [noise=true , address=145.79.227.22:4511 , piece_id=4MDCBBI4LD4BQQ76YTC75IRLH4HNVG7GEBGBIFDI7Q2TTEHNOBXA, seg_pos={0 0}, upload=257111]: Done uploading piece. err: <nil> Nov 21 12:00:46.663448046 (291h56m45.214918497s) [noise=true , address=145.79.227.23:4547 , piece_id=R7V3ZYNZBPLABNLWIB35BHRKZKFPBQ4RS2G5ZDIOBNVCFFLODDZA, seg_pos={0 0}, upload=257111]: Done uploading piece. err: <nil> Nov 21 12:00:46.672901036 (291h56m45.224371497s) [noise=true , address=145.79.227.24:4583 , piece_id=KJPKPFRWFZD5H7G3RNN2VYLP3SUHBWO3QRUR4T5J6RFYWUSZSSRA, seg_pos={0 0}, upload=257111]: Done uploading piece. err: <nil> Nov 21 12:00:46.677309982 (291h56m45.228780463s) [noise=true , address=145.79.227.31:4798 , piece_id=YWX2POKFFJISTJ5Y6WY5SHP3XIUVNH5CP24AXKU6BLW42RYWKROQ, seg_pos={0 0}, upload=257111]: Done uploading piece. err: <nil> Nov 21 12:00:46.678373266 (291h56m45.229843727s) [noise=true , address=145.79.227.28:4727 , piece_id=4W63CG2IW33MVQHJJJGPJGKZFA4JKBZMHHG5OMEKJ3Y63LVV25FQ, seg_pos={0 0}, upload=257111]: Done uploading piece. err: <nil> Nov 21 12:00:46.686989964 (291h56m45.238460425s) [noise=true , address=145.79.227.20:4438 , piece_id=DEKU5U5LMX3WRZ64WKE3MUQC3C2GBVTXPFVVEX6DN5PSEE53Q5DA, seg_pos={0 0}, upload=257111]: Done uploading piece. err: <nil> Nov 21 12:00:46.690017299 (291h56m45.24148776s) [noise=true , address=145.79.227.15:4876 , piece_id=42VLAZCDHN6CYRZ5EMVPLIVJKQ7TZFJMXFA2AWHUU6S7P37ISMNQ, seg_pos={0 0}, upload=257111]: Done uploading piece. err: <nil> Nov 21 12:00:46.693246267 (291h56m45.244716728s) [noise=true , address=145.79.227.30:4763 , piece_id=7WWH65VO7HUBZFP3ATOKMDTPY6HLJVWIA2XHVI2K2V46FEPK5EPA, seg_pos={0 0}, upload=257111]: Done uploading piece. err: <nil> Nov 21 12:00:47.227653148 (291h56m45.779123609s) [noise=true , address=77.90.31.17:4770 , piece_id=I7AQRIKKA7FVGTOPEJODHU5B6Z26GHWZXCFQQDXHFRYMHUQVJYZA, seg_pos={0 0}, upload=257111]: Done uploading piece. err: <nil> Nov 21 12:00:47.255119890 (291h56m45.806590351s) [noise=true , address=77.90.31.13:4628 , piece_id=KV2WK5KY4B4N3NHYJ6UBAKBLRZWRIFMTVV65SWS2FCUF7UV2AJMQ, seg_pos={0 0}, upload=257111]: Done uploading piece. err: <nil> Nov 21 12:00:47.278989602 (291h56m45.830460064s) [noise=true , address=77.90.31.10:4518 , piece_id=B4KCY7UJHBDHWMLOX7NKAQ75PQUTZTXEDCWEXR43EOY6QFYA3VTA, seg_pos={0 0}, upload=257111]: Done uploading piece. err: <nil> Nov 21 12:00:47.279015171 (291h56m45.830485632s) [seg_pos={0 0}, upload=257111]: Segment reached optimal threshold of 13 pieces. Nov 21 12:00:47.279217887 (291h56m45.830688348s) [noise=true , address=77.90.31.15:4698 , piece_id=ZQ5SBBVFVQQRRPRWQOC5MNLVROAAW3UNJOO3XXESCKCN2ELA5BIA, seg_pos={0 0}, upload=257111]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj): protocol: expected piece hash; context canceled; EOF Nov 21 12:00:47.279226153 (291h56m45.830696614s) [noise=true , address=77.90.31.15:4698 , piece_id=ZQ5SBBVFVQQRRPRWQOC5MNLVROAAW3UNJOO3XXESCKCN2ELA5BIA, seg_pos={0 0}, upload=257111]: Long tail cancellation due to optimal threshold Nov 21 12:00:47.279513941 (291h56m45.830984392s) [noise=true , address=45.91.116.14:4190 , piece_id=EK762P55UQ2NFROSHJYUYIETP5CBQZJHKOTJRUFBKNPTRG7IQCYA, seg_pos={0 0}, upload=257111]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Nov 21 12:00:47.279522787 (291h56m45.830993249s) [noise=true , address=45.91.116.14:4190 , piece_id=EK762P55UQ2NFROSHJYUYIETP5CBQZJHKOTJRUFBKNPTRG7IQCYA, seg_pos={0 0}, upload=257111]: Long tail cancellation due to optimal threshold Nov 21 12:00:47.279754489 (291h56m45.83122495s) [noise=true , address=45.91.116.8:4048 , piece_id=OUL6TXMXPESMVCTC7RKYMMGZA7ZIWMNMG4B5IWSHYFNMP2PC5PCA, seg_pos={0 0}, upload=257111]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: read tcp 188.166.225.170:53244->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 21 12:00:47.279762644 (291h56m45.831233105s) [noise=true , address=45.91.116.8:4048 , piece_id=OUL6TXMXPESMVCTC7RKYMMGZA7ZIWMNMG4B5IWSHYFNMP2PC5PCA, seg_pos={0 0}, upload=257111]: Long tail cancellation due to optimal threshold Nov 21 12:00:47.279782472 (291h56m45.831252934s) [seg_pos={0 0}, upload=257111]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 21 12:00:47.279865650 (291h56m45.831336102s) [upload=257111]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 21 12:00:47.279870900 (291h56m45.831341361s) [upload=257111]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 21 12:00:47.570435510 (291h56m46.121905962s) [upload=257111]: Done uploading segments... Nov 21 12:00:47.570460016 (291h56m46.121930478s) [upload=257111]: Upload finished. err: <nil> Duration: 7.155241785s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 29 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXX...............................] duration: 3.712341702s trace: https://jaeger.dev.storj.io/trace/6af15a3a67b655d5 FAIL (2025-11-21 12:02:13 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 3.712372101s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 2 out of 60 is failed (threshold=3) [..X.X.......................................................] duration: 5.717016377s trace: https://jaeger.dev.storj.io/trace/a7f69ee106efb7f FAIL (2025-11-21 12:02:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== upload=256586]: Uploading piece... Nov 21 11:25:27.210346024 (291h21m25.761816485s) [noise=true , address=89.149.201.147:4008 , piece_id=HCX7I4ICPEHDWTYDFIQMKZ5EE5CGPQLYAIHF7WZMTUUU6CNZPHTA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.211425289 (291h21m25.76289575s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.213103795 (291h21m25.764574246s) [noise=true , address=185.167.204.205:4022 , piece_id=BES3YK6PYQV2AKOWSAGHF7DGGWGWSWPIJU2ICAQ5WBHKZ5VT5L6A, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.213408395 (291h21m25.764878856s) [noise=true , address=185.167.204.205:4027 , piece_id=IVPONKBF2PIHZQAW3L4YA4HOMQV5BQML6SKKCJLIEGT5SZSQ2AOA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.214501747 (291h21m25.765972208s) [noise=true , address=81.171.14.70:4042 , piece_id=XNREZEUZ5L5KZ3L7ZOL6IRK7IHU6DJN4RLEQZRFMDKDH2JWC24CQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.215013752 (291h21m25.766484213s) [noise=true , address=81.171.14.70:4034 , piece_id=ITRLKYVNFSYBLABNGNFO3HRJNMX3MWB345FIMWSIPBNAVZPA47UA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.215087572 (291h21m25.766558044s) [noise=true , address=185.167.204.205:4031 , piece_id=ZEI2YSIQC655R7WS43WADXGFT5RKT5LFSN6IZEQ6SHTDVOV7BZBQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.216432844 (291h21m25.767903305s) [noise=true , address=185.167.204.206:4056 , piece_id=EAMZIK6AVJL7IMFF5VOLW3SMCGYKSKB2VXJTNUNLJCVIT7EKXNMQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.216662281 (291h21m25.768132742s) [noise=true , address=89.149.201.147:4011 , piece_id=ZATNMD5HOK35AWGJJ5V66VDJ4YGUVZQJ2OBERQL32ZV5GW2HBSMQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.218008314 (291h21m25.769478775s) [noise=true , address=81.171.14.70:4032 , piece_id=CFH2RTR3BLBUB5ZIZCFC6UHT3FHT3B2XSCHQMKXNGDJJGPHEMCTQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:28.035150976 (291h21m26.586621437s) [upload=256586]: Next returned nil segment Nov 21 11:25:28.035169622 (291h21m26.586640083s) [upload=256586]: Waiting for error group managing segments... Nov 21 11:25:28.416646836 (291h21m26.968117317s) [noise=true , address=185.167.204.206:4064 , piece_id=SRFIWFQLRAGCNIKYLHMGT4QRUL3WH2QPLLIKAJ7DLFJEQUW5DPPA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.417338854 (291h21m26.968809325s) [noise=true , address=185.167.204.205:4033 , piece_id=FEWQVHPIKN3Q6O5AZ6AGJBLUPZ34PBZFB5X3XNSKGPOGI4AIO7JA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.421089406 (291h21m26.972559867s) [noise=true , address=185.167.204.206:4072 , piece_id=GYCO53OWGOMOHK4FHC5GENN4LTQGSZKZKOI5V534RLV3LHLPKC6Q, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.426252887 (291h21m26.977723348s) [noise=true , address=185.167.204.206:4046 , piece_id=BZ57ZA35ITEOA6CAZJGNODK5MUTXEZWDV7E2OGBZOSMS4OQZYAEA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.431720689 (291h21m26.98319115s) [noise=true , address=185.167.204.205:4027 , piece_id=IVPONKBF2PIHZQAW3L4YA4HOMQV5BQML6SKKCJLIEGT5SZSQ2AOA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.438243951 (291h21m26.989714412s) [noise=true , address=185.167.204.205:4022 , piece_id=BES3YK6PYQV2AKOWSAGHF7DGGWGWSWPIJU2ICAQ5WBHKZ5VT5L6A, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.443364331 (291h21m26.994834792s) [noise=true , address=185.167.204.206:4056 , piece_id=EAMZIK6AVJL7IMFF5VOLW3SMCGYKSKB2VXJTNUNLJCVIT7EKXNMQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.445594728 (291h21m26.997065179s) [noise=true , address=185.167.204.205:4016 , piece_id=EHQS3546UXRXTL2MGFCSRTPDDPAYXEYLMISPP5AWY337MHX7HFLQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.449325822 (291h21m27.000796283s) [noise=true , address=185.167.204.205:4031 , piece_id=ZEI2YSIQC655R7WS43WADXGFT5RKT5LFSN6IZEQ6SHTDVOV7BZBQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.449761211 (291h21m27.001231672s) [noise=true , address=185.167.204.205:4018 , piece_id=7OSLPWJLKFTIJPZEAFMZOAH6JAAMX6HKROI25FODFYBPNJWSMLWA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.450419845 (291h21m27.001890306s) [noise=true , address=185.167.204.206:4062 , piece_id=UF3S6NAVGS2KQJPGA5WXP7YG4RQO44P445SNPCWXKXMONZAYUTNQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.453201252 (291h21m27.004671713s) [noise=true , address=185.167.204.205:4007 , piece_id=MEBGOPZ6MFMQY4UZARS2LKJKEMNN5YGSLKPJGTWSKS45CUNXSRGQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.661209626 (291h21m28.212680087s) [noise=true , address=89.149.201.147:4016 , piece_id=TGRWGGGBKDANC7YKROWPRMIYNAVFEVE3RXOSUOINFI7KTRS2IYUQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.693169182 (291h21m28.244639643s) [noise=true , address=89.149.201.147:4002 , piece_id=UIYJSCMY3LCLRHZUJB6JFJHHXKMYTVKERTZW7F7X44MWJETPY6XA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.721204497 (291h21m28.272674958s) [noise=true , address=89.149.201.147:4008 , piece_id=HCX7I4ICPEHDWTYDFIQMKZ5EE5CGPQLYAIHF7WZMTUUU6CNZPHTA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.764584711 (291h21m28.316055172s) [noise=true , address=89.149.201.147:4001 , piece_id=QUS6NB3NQBLJFB6YCK3EBDPKCTWEWGKAT3NOR24TVI65SQKHWDOQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.768284075 (291h21m28.319754536s) [noise=true , address=89.149.201.147:4011 , piece_id=ZATNMD5HOK35AWGJJ5V66VDJ4YGUVZQJ2OBERQL32ZV5GW2HBSMQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.789681961 (291h21m28.341152422s) [noise=true , address=89.149.201.147:4003 , piece_id=RUAYGN2RGF5JJTWFEGIHWH3BHEUDSS5DN2QJ6YLSHPF5BGUXD3RA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:37.298598682 (291h21m35.850069143s) [noise=true , address=81.171.14.70:4043 , piece_id=T3NF6JR3VJIGDMPNUBLMJ3AOTQP5IHOGJYORMCSS7ZMOENQETZ7A, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:37.497478356 (291h21m36.048948818s) [noise=true , address=81.171.14.70:4032 , piece_id=CFH2RTR3BLBUB5ZIZCFC6UHT3FHT3B2XSCHQMKXNGDJJGPHEMCTQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:40.406166933 (291h21m38.957637394s) [noise=true , address=81.171.14.70:4045 , piece_id=BRWQ7G3E547PJE2RLGSESIB3HMP5C4M5NSKAJNGWSFXUBEO6SYEQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:42.511041157 (291h21m41.062511608s) [noise=true , address=81.171.14.70:4029 , piece_id=DKJM5E7EY7LOSBSEBQOZRC6CI5ZPPE47HFOFQJROPCUMUV567CAA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:12.012682521 (291h22m10.564152982s) [noise=true , address=81.171.14.70:4042 , piece_id=XNREZEUZ5L5KZ3L7ZOL6IRK7IHU6DJN4RLEQZRFMDKDH2JWC24CQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:15.596046283 (291h22m14.147516744s) [noise=true , address=81.171.14.70:4033 , piece_id=JRKPD4EP6FZM5HISOIDJYZY7FAWWQCADAS47FGW2JM5A7RFJR2BQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:18.453059659 (291h22m17.00453012s) [noise=true , address=81.171.14.70:4046 , piece_id=OSONRSF6KJS7F2QJVCC3FPZK4AV7QPG5YSM47CW5LIW7EME44AWA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:23.442583344 (291h22m21.994053805s) [noise=true , address=81.171.14.70:4034 , piece_id=ITRLKYVNFSYBLABNGNFO3HRJNMX3MWB345FIMWSIPBNAVZPA47UA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:23.442621967 (291h22m21.994092428s) [seg_pos={0 0}, upload=256586]: Segment reached optimal threshold of 26 pieces. Nov 21 11:26:23.442836976 (291h22m21.994307437s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MUVGecysnTSFcJoHUL7QMWb3h6dtXgUd7FbA672fQCyEWhePr): protocol: expected piece hash; context canceled; EOF Nov 21 11:26:23.442845934 (291h22m21.994316395s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Long tail cancellation due to optimal threshold Nov 21 11:26:23.443227491 (291h22m21.994697942s) [noise=true , address=81.171.14.70:4027 , piece_id=4FDFCCWF3EIROQAFLOHGS743523OYGSXWQSPQEVD3AMQ6WKDQI7Q, seg_pos={0 0}, upload=256586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122cYvgTxSvC2aEXnBkRk2o8ARTBXj9gqPoa12vW8hogSN6kpA4): protocol: expected piece hash; context canceled; EOF Nov 21 11:26:23.443237770 (291h22m21.994708231s) [noise=true , address=81.171.14.70:4027 , piece_id=4FDFCCWF3EIROQAFLOHGS743523OYGSXWQSPQEVD3AMQ6WKDQI7Q, seg_pos={0 0}, upload=256586]: Long tail cancellation due to optimal threshold Nov 21 11:26:23.443274720 (291h22m21.994745181s) [seg_pos={0 0}, upload=256586]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Nov 21 11:26:23.443378929 (291h22m21.99484939s) [upload=256586]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 21 11:26:23.443387044 (291h22m21.994857495s) [upload=256586]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 21 11:26:23.685139626 (291h22m22.236610077s) [upload=256586]: Done uploading segments... Nov 21 11:26:23.685160676 (291h22m22.236631127s) [upload=256586]: Upload finished. err: <nil> Duration: 5.71704442s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [XX...X.X.X....X............X....X.X.X..X.X..X...X..XXX.X.XX.] duration: 6.39675778s trace: https://jaeger.dev.storj.io/trace/2e92d97344640fc3 FAIL (2025-11-21 12:01:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 21 12:00:44.413717038 (291h56m42.965187489s) [upload=257111]: Starting upload Nov 21 12:00:44.413777513 (291h56m42.965247974s) [upload=257111]: Done starting upload Nov 21 12:00:44.413945853 (291h56m42.965416314s) [upload=257111]: Uploading segments... Nov 21 12:00:44.414062085 (291h56m42.965532536s) [upload=257111]: Got next segment. Inline: false Nov 21 12:00:44.414124955 (291h56m42.965595416s) [upload=257111]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 12:00:44.414131217 (291h56m42.965601678s) [upload=257111]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 12:00:44.659711608 (291h56m43.21118208s) [seg_pos={0 0}, upload=257111]: Begin upload segment... Nov 21 12:00:44.659783275 (291h56m43.211253736s) [seg_pos={0 0}, upload=257111]: Done begin upload segment. Nov 21 12:00:44.659821467 (291h56m43.211291919s) [noise=true , address=45.91.116.14:4190 , piece_id=EK762P55UQ2NFROSHJYUYIETP5CBQZJHKOTJRUFBKNPTRG7IQCYA, seg_pos={0 0}, upload=257111]: Uploading piece... Nov 21 12:00:44.661076968 (291h56m43.212547429s) [noise=true , address=145.79.227.15:4876 , piece_id=42VLAZCDHN6CYRZ5EMVPLIVJKQ7TZFJMXFA2AWHUU6S7P37ISMNQ, seg_pos={0 0}, upload=257111]: Uploading piece... Nov 21 12:00:44.661623990 (291h56m43.213094451s) [noise=true , address=77.90.31.13:4628 , piece_id=KV2WK5KY4B4N3NHYJ6UBAKBLRZWRIFMTVV65SWS2FCUF7UV2AJMQ, seg_pos={0 0}, upload=257111]: Uploading piece... Nov 21 12:00:44.662015575 (291h56m43.213486036s) [noise=true , address=145.79.227.23:4547 , piece_id=R7V3ZYNZBPLABNLWIB35BHRKZKFPBQ4RS2G5ZDIOBNVCFFLODDZA, seg_pos={0 0}, upload=257111]: Uploading piece... Nov 21 12:00:44.662104145 (291h56m43.213574626s) [noise=true , address=145.79.227.27:4690 , piece_id=EF7B5TVERZJ5RQWZMXXFWHQQXPIAKKTGEU5IRKQMNY4GIOZ7PMOQ, seg_pos={0 0}, upload=257111]: Uploading piece... Nov 21 12:00:44.662331166 (291h56m43.213801617s) [noise=true , address=77.90.31.10:4518 , piece_id=B4KCY7UJHBDHWMLOX7NKAQ75PQUTZTXEDCWEXR43EOY6QFYA3VTA, seg_pos={0 0}, upload=257111]: Uploading piece... Nov 21 12:00:44.662744884 (291h56m43.214215345s) [noise=true , address=145.79.227.24:4583 , piece_id=KJPKPFRWFZD5H7G3RNN2VYLP3SUHBWO3QRUR4T5J6RFYWUSZSSRA, seg_pos={0 0}, upload=257111]: Uploading piece... Nov 21 12:00:44.662792254 (291h56m43.214262715s) [noise=true , address=145.79.227.11:4731 , piece_id=NZEOWNXMOAIGI5QWGYVJHR4GKK5E7CIUVHGYPY7UBEKQMAIGQPCQ, seg_pos={0 0}, upload=257111]: Uploading piece... Nov 21 12:00:44.663124828 (291h56m43.214595289s) [noise=true , address=145.79.227.20:4438 , piece_id=DEKU5U5LMX3WRZ64WKE3MUQC3C2GBVTXPFVVEX6DN5PSEE53Q5DA, seg_pos={0 0}, upload=257111]: Uploading piece... Nov 21 12:00:44.663330720 (291h56m43.214801171s) [noise=true , address=145.79.227.30:4763 , piece_id=7WWH65VO7HUBZFP3ATOKMDTPY6HLJVWIA2XHVI2K2V46FEPK5EPA, seg_pos={0 0}, upload=257111]: Uploading piece... Nov 21 12:00:44.663457621 (291h56m43.214928082s) [noise=true , address=145.79.227.22:4511 , piece_id=4MDCBBI4LD4BQQ76YTC75IRLH4HNVG7GEBGBIFDI7Q2TTEHNOBXA, seg_pos={0 0}, upload=257111]: Uploading piece... Nov 21 12:00:44.663871199 (291h56m43.21534165s) [noise=true , address=145.79.227.28:4727 , piece_id=4W63CG2IW33MVQHJJJGPJGKZFA4JKBZMHHG5OMEKJ3Y63LVV25FQ, seg_pos={0 0}, upload=257111]: Uploading piece... Nov 21 12:00:44.663892760 (291h56m43.215363221s) [noise=true , address=45.91.116.8:4048 , piece_id=OUL6TXMXPESMVCTC7RKYMMGZA7ZIWMNMG4B5IWSHYFNMP2PC5PCA, seg_pos={0 0}, upload=257111]: Uploading piece... Nov 21 12:00:44.663946742 (291h56m43.215417203s) [noise=true , address=77.90.31.15:4698 , piece_id=ZQ5SBBVFVQQRRPRWQOC5MNLVROAAW3UNJOO3XXESCKCN2ELA5BIA, seg_pos={0 0}, upload=257111]: Uploading piece... Nov 21 12:00:44.664546976 (291h56m43.216017437s) [noise=true , address=145.79.227.31:4798 , piece_id=YWX2POKFFJISTJ5Y6WY5SHP3XIUVNH5CP24AXKU6BLW42RYWKROQ, seg_pos={0 0}, upload=257111]: Uploading piece... Nov 21 12:00:44.665139685 (291h56m43.216610136s) [noise=true , address=77.90.31.17:4770 , piece_id=I7AQRIKKA7FVGTOPEJODHU5B6Z26GHWZXCFQQDXHFRYMHUQVJYZA, seg_pos={0 0}, upload=257111]: Uploading piece... Nov 21 12:00:46.269248547 (291h56m44.820718998s) [upload=257111]: Next returned nil segment Nov 21 12:00:46.269292731 (291h56m44.820763192s) [upload=257111]: Waiting for error group managing segments... Nov 21 12:00:46.651938048 (291h56m45.203408509s) [noise=true , address=145.79.227.11:4731 , piece_id=NZEOWNXMOAIGI5QWGYVJHR4GKK5E7CIUVHGYPY7UBEKQMAIGQPCQ, seg_pos={0 0}, upload=257111]: Done uploading piece. err: <nil> Nov 21 12:00:46.653674856 (291h56m45.205145317s) [noise=true , address=145.79.227.27:4690 , piece_id=EF7B5TVERZJ5RQWZMXXFWHQQXPIAKKTGEU5IRKQMNY4GIOZ7PMOQ, seg_pos={0 0}, upload=257111]: Done uploading piece. err: <nil> Nov 21 12:00:46.656724623 (291h56m45.208195084s) [noise=true , address=145.79.227.22:4511 , piece_id=4MDCBBI4LD4BQQ76YTC75IRLH4HNVG7GEBGBIFDI7Q2TTEHNOBXA, seg_pos={0 0}, upload=257111]: Done uploading piece. err: <nil> Nov 21 12:00:46.663448046 (291h56m45.214918497s) [noise=true , address=145.79.227.23:4547 , piece_id=R7V3ZYNZBPLABNLWIB35BHRKZKFPBQ4RS2G5ZDIOBNVCFFLODDZA, seg_pos={0 0}, upload=257111]: Done uploading piece. err: <nil> Nov 21 12:00:46.672901036 (291h56m45.224371497s) [noise=true , address=145.79.227.24:4583 , piece_id=KJPKPFRWFZD5H7G3RNN2VYLP3SUHBWO3QRUR4T5J6RFYWUSZSSRA, seg_pos={0 0}, upload=257111]: Done uploading piece. err: <nil> Nov 21 12:00:46.677309982 (291h56m45.228780463s) [noise=true , address=145.79.227.31:4798 , piece_id=YWX2POKFFJISTJ5Y6WY5SHP3XIUVNH5CP24AXKU6BLW42RYWKROQ, seg_pos={0 0}, upload=257111]: Done uploading piece. err: <nil> Nov 21 12:00:46.678373266 (291h56m45.229843727s) [noise=true , address=145.79.227.28:4727 , piece_id=4W63CG2IW33MVQHJJJGPJGKZFA4JKBZMHHG5OMEKJ3Y63LVV25FQ, seg_pos={0 0}, upload=257111]: Done uploading piece. err: <nil> Nov 21 12:00:46.686989964 (291h56m45.238460425s) [noise=true , address=145.79.227.20:4438 , piece_id=DEKU5U5LMX3WRZ64WKE3MUQC3C2GBVTXPFVVEX6DN5PSEE53Q5DA, seg_pos={0 0}, upload=257111]: Done uploading piece. err: <nil> Nov 21 12:00:46.690017299 (291h56m45.24148776s) [noise=true , address=145.79.227.15:4876 , piece_id=42VLAZCDHN6CYRZ5EMVPLIVJKQ7TZFJMXFA2AWHUU6S7P37ISMNQ, seg_pos={0 0}, upload=257111]: Done uploading piece. err: <nil> Nov 21 12:00:46.693246267 (291h56m45.244716728s) [noise=true , address=145.79.227.30:4763 , piece_id=7WWH65VO7HUBZFP3ATOKMDTPY6HLJVWIA2XHVI2K2V46FEPK5EPA, seg_pos={0 0}, upload=257111]: Done uploading piece. err: <nil> Nov 21 12:00:47.227653148 (291h56m45.779123609s) [noise=true , address=77.90.31.17:4770 , piece_id=I7AQRIKKA7FVGTOPEJODHU5B6Z26GHWZXCFQQDXHFRYMHUQVJYZA, seg_pos={0 0}, upload=257111]: Done uploading piece. err: <nil> Nov 21 12:00:47.255119890 (291h56m45.806590351s) [noise=true , address=77.90.31.13:4628 , piece_id=KV2WK5KY4B4N3NHYJ6UBAKBLRZWRIFMTVV65SWS2FCUF7UV2AJMQ, seg_pos={0 0}, upload=257111]: Done uploading piece. err: <nil> Nov 21 12:00:47.278989602 (291h56m45.830460064s) [noise=true , address=77.90.31.10:4518 , piece_id=B4KCY7UJHBDHWMLOX7NKAQ75PQUTZTXEDCWEXR43EOY6QFYA3VTA, seg_pos={0 0}, upload=257111]: Done uploading piece. err: <nil> Nov 21 12:00:47.279015171 (291h56m45.830485632s) [seg_pos={0 0}, upload=257111]: Segment reached optimal threshold of 13 pieces. Nov 21 12:00:47.279217887 (291h56m45.830688348s) [noise=true , address=77.90.31.15:4698 , piece_id=ZQ5SBBVFVQQRRPRWQOC5MNLVROAAW3UNJOO3XXESCKCN2ELA5BIA, seg_pos={0 0}, upload=257111]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj): protocol: expected piece hash; context canceled; EOF Nov 21 12:00:47.279226153 (291h56m45.830696614s) [noise=true , address=77.90.31.15:4698 , piece_id=ZQ5SBBVFVQQRRPRWQOC5MNLVROAAW3UNJOO3XXESCKCN2ELA5BIA, seg_pos={0 0}, upload=257111]: Long tail cancellation due to optimal threshold Nov 21 12:00:47.279513941 (291h56m45.830984392s) [noise=true , address=45.91.116.14:4190 , piece_id=EK762P55UQ2NFROSHJYUYIETP5CBQZJHKOTJRUFBKNPTRG7IQCYA, seg_pos={0 0}, upload=257111]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Nov 21 12:00:47.279522787 (291h56m45.830993249s) [noise=true , address=45.91.116.14:4190 , piece_id=EK762P55UQ2NFROSHJYUYIETP5CBQZJHKOTJRUFBKNPTRG7IQCYA, seg_pos={0 0}, upload=257111]: Long tail cancellation due to optimal threshold Nov 21 12:00:47.279754489 (291h56m45.83122495s) [noise=true , address=45.91.116.8:4048 , piece_id=OUL6TXMXPESMVCTC7RKYMMGZA7ZIWMNMG4B5IWSHYFNMP2PC5PCA, seg_pos={0 0}, upload=257111]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: read tcp 188.166.225.170:53244->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 21 12:00:47.279762644 (291h56m45.831233105s) [noise=true , address=45.91.116.8:4048 , piece_id=OUL6TXMXPESMVCTC7RKYMMGZA7ZIWMNMG4B5IWSHYFNMP2PC5PCA, seg_pos={0 0}, upload=257111]: Long tail cancellation due to optimal threshold Nov 21 12:00:47.279782472 (291h56m45.831252934s) [seg_pos={0 0}, upload=257111]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 21 12:00:47.279865650 (291h56m45.831336102s) [upload=257111]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 21 12:00:47.279870900 (291h56m45.831341361s) [upload=257111]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 21 12:00:47.570435510 (291h56m46.121905962s) [upload=257111]: Done uploading segments... Nov 21 12:00:47.570460016 (291h56m46.121930478s) [upload=257111]: Upload finished. err: <nil> Duration: 6.396811202s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 2 out of 60 is failed (threshold=3) [...X.X......................................................] duration: 5.320575088s trace: https://jaeger.dev.storj.io/trace/a7f69ee106efb7f FAIL (2025-11-21 12:01:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== upload=256586]: Uploading piece... Nov 21 11:25:27.210346024 (291h21m25.761816485s) [noise=true , address=89.149.201.147:4008 , piece_id=HCX7I4ICPEHDWTYDFIQMKZ5EE5CGPQLYAIHF7WZMTUUU6CNZPHTA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.211425289 (291h21m25.76289575s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.213103795 (291h21m25.764574246s) [noise=true , address=185.167.204.205:4022 , piece_id=BES3YK6PYQV2AKOWSAGHF7DGGWGWSWPIJU2ICAQ5WBHKZ5VT5L6A, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.213408395 (291h21m25.764878856s) [noise=true , address=185.167.204.205:4027 , piece_id=IVPONKBF2PIHZQAW3L4YA4HOMQV5BQML6SKKCJLIEGT5SZSQ2AOA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.214501747 (291h21m25.765972208s) [noise=true , address=81.171.14.70:4042 , piece_id=XNREZEUZ5L5KZ3L7ZOL6IRK7IHU6DJN4RLEQZRFMDKDH2JWC24CQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.215013752 (291h21m25.766484213s) [noise=true , address=81.171.14.70:4034 , piece_id=ITRLKYVNFSYBLABNGNFO3HRJNMX3MWB345FIMWSIPBNAVZPA47UA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.215087572 (291h21m25.766558044s) [noise=true , address=185.167.204.205:4031 , piece_id=ZEI2YSIQC655R7WS43WADXGFT5RKT5LFSN6IZEQ6SHTDVOV7BZBQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.216432844 (291h21m25.767903305s) [noise=true , address=185.167.204.206:4056 , piece_id=EAMZIK6AVJL7IMFF5VOLW3SMCGYKSKB2VXJTNUNLJCVIT7EKXNMQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.216662281 (291h21m25.768132742s) [noise=true , address=89.149.201.147:4011 , piece_id=ZATNMD5HOK35AWGJJ5V66VDJ4YGUVZQJ2OBERQL32ZV5GW2HBSMQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.218008314 (291h21m25.769478775s) [noise=true , address=81.171.14.70:4032 , piece_id=CFH2RTR3BLBUB5ZIZCFC6UHT3FHT3B2XSCHQMKXNGDJJGPHEMCTQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:28.035150976 (291h21m26.586621437s) [upload=256586]: Next returned nil segment Nov 21 11:25:28.035169622 (291h21m26.586640083s) [upload=256586]: Waiting for error group managing segments... Nov 21 11:25:28.416646836 (291h21m26.968117317s) [noise=true , address=185.167.204.206:4064 , piece_id=SRFIWFQLRAGCNIKYLHMGT4QRUL3WH2QPLLIKAJ7DLFJEQUW5DPPA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.417338854 (291h21m26.968809325s) [noise=true , address=185.167.204.205:4033 , piece_id=FEWQVHPIKN3Q6O5AZ6AGJBLUPZ34PBZFB5X3XNSKGPOGI4AIO7JA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.421089406 (291h21m26.972559867s) [noise=true , address=185.167.204.206:4072 , piece_id=GYCO53OWGOMOHK4FHC5GENN4LTQGSZKZKOI5V534RLV3LHLPKC6Q, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.426252887 (291h21m26.977723348s) [noise=true , address=185.167.204.206:4046 , piece_id=BZ57ZA35ITEOA6CAZJGNODK5MUTXEZWDV7E2OGBZOSMS4OQZYAEA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.431720689 (291h21m26.98319115s) [noise=true , address=185.167.204.205:4027 , piece_id=IVPONKBF2PIHZQAW3L4YA4HOMQV5BQML6SKKCJLIEGT5SZSQ2AOA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.438243951 (291h21m26.989714412s) [noise=true , address=185.167.204.205:4022 , piece_id=BES3YK6PYQV2AKOWSAGHF7DGGWGWSWPIJU2ICAQ5WBHKZ5VT5L6A, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.443364331 (291h21m26.994834792s) [noise=true , address=185.167.204.206:4056 , piece_id=EAMZIK6AVJL7IMFF5VOLW3SMCGYKSKB2VXJTNUNLJCVIT7EKXNMQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.445594728 (291h21m26.997065179s) [noise=true , address=185.167.204.205:4016 , piece_id=EHQS3546UXRXTL2MGFCSRTPDDPAYXEYLMISPP5AWY337MHX7HFLQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.449325822 (291h21m27.000796283s) [noise=true , address=185.167.204.205:4031 , piece_id=ZEI2YSIQC655R7WS43WADXGFT5RKT5LFSN6IZEQ6SHTDVOV7BZBQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.449761211 (291h21m27.001231672s) [noise=true , address=185.167.204.205:4018 , piece_id=7OSLPWJLKFTIJPZEAFMZOAH6JAAMX6HKROI25FODFYBPNJWSMLWA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.450419845 (291h21m27.001890306s) [noise=true , address=185.167.204.206:4062 , piece_id=UF3S6NAVGS2KQJPGA5WXP7YG4RQO44P445SNPCWXKXMONZAYUTNQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.453201252 (291h21m27.004671713s) [noise=true , address=185.167.204.205:4007 , piece_id=MEBGOPZ6MFMQY4UZARS2LKJKEMNN5YGSLKPJGTWSKS45CUNXSRGQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.661209626 (291h21m28.212680087s) [noise=true , address=89.149.201.147:4016 , piece_id=TGRWGGGBKDANC7YKROWPRMIYNAVFEVE3RXOSUOINFI7KTRS2IYUQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.693169182 (291h21m28.244639643s) [noise=true , address=89.149.201.147:4002 , piece_id=UIYJSCMY3LCLRHZUJB6JFJHHXKMYTVKERTZW7F7X44MWJETPY6XA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.721204497 (291h21m28.272674958s) [noise=true , address=89.149.201.147:4008 , piece_id=HCX7I4ICPEHDWTYDFIQMKZ5EE5CGPQLYAIHF7WZMTUUU6CNZPHTA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.764584711 (291h21m28.316055172s) [noise=true , address=89.149.201.147:4001 , piece_id=QUS6NB3NQBLJFB6YCK3EBDPKCTWEWGKAT3NOR24TVI65SQKHWDOQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.768284075 (291h21m28.319754536s) [noise=true , address=89.149.201.147:4011 , piece_id=ZATNMD5HOK35AWGJJ5V66VDJ4YGUVZQJ2OBERQL32ZV5GW2HBSMQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.789681961 (291h21m28.341152422s) [noise=true , address=89.149.201.147:4003 , piece_id=RUAYGN2RGF5JJTWFEGIHWH3BHEUDSS5DN2QJ6YLSHPF5BGUXD3RA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:37.298598682 (291h21m35.850069143s) [noise=true , address=81.171.14.70:4043 , piece_id=T3NF6JR3VJIGDMPNUBLMJ3AOTQP5IHOGJYORMCSS7ZMOENQETZ7A, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:37.497478356 (291h21m36.048948818s) [noise=true , address=81.171.14.70:4032 , piece_id=CFH2RTR3BLBUB5ZIZCFC6UHT3FHT3B2XSCHQMKXNGDJJGPHEMCTQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:40.406166933 (291h21m38.957637394s) [noise=true , address=81.171.14.70:4045 , piece_id=BRWQ7G3E547PJE2RLGSESIB3HMP5C4M5NSKAJNGWSFXUBEO6SYEQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:42.511041157 (291h21m41.062511608s) [noise=true , address=81.171.14.70:4029 , piece_id=DKJM5E7EY7LOSBSEBQOZRC6CI5ZPPE47HFOFQJROPCUMUV567CAA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:12.012682521 (291h22m10.564152982s) [noise=true , address=81.171.14.70:4042 , piece_id=XNREZEUZ5L5KZ3L7ZOL6IRK7IHU6DJN4RLEQZRFMDKDH2JWC24CQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:15.596046283 (291h22m14.147516744s) [noise=true , address=81.171.14.70:4033 , piece_id=JRKPD4EP6FZM5HISOIDJYZY7FAWWQCADAS47FGW2JM5A7RFJR2BQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:18.453059659 (291h22m17.00453012s) [noise=true , address=81.171.14.70:4046 , piece_id=OSONRSF6KJS7F2QJVCC3FPZK4AV7QPG5YSM47CW5LIW7EME44AWA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:23.442583344 (291h22m21.994053805s) [noise=true , address=81.171.14.70:4034 , piece_id=ITRLKYVNFSYBLABNGNFO3HRJNMX3MWB345FIMWSIPBNAVZPA47UA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:23.442621967 (291h22m21.994092428s) [seg_pos={0 0}, upload=256586]: Segment reached optimal threshold of 26 pieces. Nov 21 11:26:23.442836976 (291h22m21.994307437s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MUVGecysnTSFcJoHUL7QMWb3h6dtXgUd7FbA672fQCyEWhePr): protocol: expected piece hash; context canceled; EOF Nov 21 11:26:23.442845934 (291h22m21.994316395s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Long tail cancellation due to optimal threshold Nov 21 11:26:23.443227491 (291h22m21.994697942s) [noise=true , address=81.171.14.70:4027 , piece_id=4FDFCCWF3EIROQAFLOHGS743523OYGSXWQSPQEVD3AMQ6WKDQI7Q, seg_pos={0 0}, upload=256586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122cYvgTxSvC2aEXnBkRk2o8ARTBXj9gqPoa12vW8hogSN6kpA4): protocol: expected piece hash; context canceled; EOF Nov 21 11:26:23.443237770 (291h22m21.994708231s) [noise=true , address=81.171.14.70:4027 , piece_id=4FDFCCWF3EIROQAFLOHGS743523OYGSXWQSPQEVD3AMQ6WKDQI7Q, seg_pos={0 0}, upload=256586]: Long tail cancellation due to optimal threshold Nov 21 11:26:23.443274720 (291h22m21.994745181s) [seg_pos={0 0}, upload=256586]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Nov 21 11:26:23.443378929 (291h22m21.99484939s) [upload=256586]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 21 11:26:23.443387044 (291h22m21.994857495s) [upload=256586]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 21 11:26:23.685139626 (291h22m22.236610077s) [upload=256586]: Done uploading segments... Nov 21 11:26:23.685160676 (291h22m22.236631127s) [upload=256586]: Upload finished. err: <nil> Duration: 5.320611366s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 30 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX..............................] duration: 3.460174567s trace: https://jaeger.dev.storj.io/trace/6af15a3a67b655d5 FAIL (2025-11-21 12:01:24 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 3.460230182s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [.XX...X.X.X....X............X....X.X.X..X.X..X...X..XXX.X.XX] duration: 30.003420546s trace: https://jaeger.dev.storj.io/trace/2e92d97344640fc3 FAIL (2025-11-21 12:01:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 21 12:00:44.413717038 (291h56m42.965187489s) [upload=257111]: Starting upload Nov 21 12:00:44.413777513 (291h56m42.965247974s) [upload=257111]: Done starting upload Nov 21 12:00:44.413945853 (291h56m42.965416314s) [upload=257111]: Uploading segments... Nov 21 12:00:44.414062085 (291h56m42.965532536s) [upload=257111]: Got next segment. Inline: false Nov 21 12:00:44.414124955 (291h56m42.965595416s) [upload=257111]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 12:00:44.414131217 (291h56m42.965601678s) [upload=257111]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 12:00:44.659711608 (291h56m43.21118208s) [seg_pos={0 0}, upload=257111]: Begin upload segment... Nov 21 12:00:44.659783275 (291h56m43.211253736s) [seg_pos={0 0}, upload=257111]: Done begin upload segment. Nov 21 12:00:44.659821467 (291h56m43.211291919s) [noise=true , address=45.91.116.14:4190 , piece_id=EK762P55UQ2NFROSHJYUYIETP5CBQZJHKOTJRUFBKNPTRG7IQCYA, seg_pos={0 0}, upload=257111]: Uploading piece... Nov 21 12:00:44.661076968 (291h56m43.212547429s) [noise=true , address=145.79.227.15:4876 , piece_id=42VLAZCDHN6CYRZ5EMVPLIVJKQ7TZFJMXFA2AWHUU6S7P37ISMNQ, seg_pos={0 0}, upload=257111]: Uploading piece... Nov 21 12:00:44.661623990 (291h56m43.213094451s) [noise=true , address=77.90.31.13:4628 , piece_id=KV2WK5KY4B4N3NHYJ6UBAKBLRZWRIFMTVV65SWS2FCUF7UV2AJMQ, seg_pos={0 0}, upload=257111]: Uploading piece... Nov 21 12:00:44.662015575 (291h56m43.213486036s) [noise=true , address=145.79.227.23:4547 , piece_id=R7V3ZYNZBPLABNLWIB35BHRKZKFPBQ4RS2G5ZDIOBNVCFFLODDZA, seg_pos={0 0}, upload=257111]: Uploading piece... Nov 21 12:00:44.662104145 (291h56m43.213574626s) [noise=true , address=145.79.227.27:4690 , piece_id=EF7B5TVERZJ5RQWZMXXFWHQQXPIAKKTGEU5IRKQMNY4GIOZ7PMOQ, seg_pos={0 0}, upload=257111]: Uploading piece... Nov 21 12:00:44.662331166 (291h56m43.213801617s) [noise=true , address=77.90.31.10:4518 , piece_id=B4KCY7UJHBDHWMLOX7NKAQ75PQUTZTXEDCWEXR43EOY6QFYA3VTA, seg_pos={0 0}, upload=257111]: Uploading piece... Nov 21 12:00:44.662744884 (291h56m43.214215345s) [noise=true , address=145.79.227.24:4583 , piece_id=KJPKPFRWFZD5H7G3RNN2VYLP3SUHBWO3QRUR4T5J6RFYWUSZSSRA, seg_pos={0 0}, upload=257111]: Uploading piece... Nov 21 12:00:44.662792254 (291h56m43.214262715s) [noise=true , address=145.79.227.11:4731 , piece_id=NZEOWNXMOAIGI5QWGYVJHR4GKK5E7CIUVHGYPY7UBEKQMAIGQPCQ, seg_pos={0 0}, upload=257111]: Uploading piece... Nov 21 12:00:44.663124828 (291h56m43.214595289s) [noise=true , address=145.79.227.20:4438 , piece_id=DEKU5U5LMX3WRZ64WKE3MUQC3C2GBVTXPFVVEX6DN5PSEE53Q5DA, seg_pos={0 0}, upload=257111]: Uploading piece... Nov 21 12:00:44.663330720 (291h56m43.214801171s) [noise=true , address=145.79.227.30:4763 , piece_id=7WWH65VO7HUBZFP3ATOKMDTPY6HLJVWIA2XHVI2K2V46FEPK5EPA, seg_pos={0 0}, upload=257111]: Uploading piece... Nov 21 12:00:44.663457621 (291h56m43.214928082s) [noise=true , address=145.79.227.22:4511 , piece_id=4MDCBBI4LD4BQQ76YTC75IRLH4HNVG7GEBGBIFDI7Q2TTEHNOBXA, seg_pos={0 0}, upload=257111]: Uploading piece... Nov 21 12:00:44.663871199 (291h56m43.21534165s) [noise=true , address=145.79.227.28:4727 , piece_id=4W63CG2IW33MVQHJJJGPJGKZFA4JKBZMHHG5OMEKJ3Y63LVV25FQ, seg_pos={0 0}, upload=257111]: Uploading piece... Nov 21 12:00:44.663892760 (291h56m43.215363221s) [noise=true , address=45.91.116.8:4048 , piece_id=OUL6TXMXPESMVCTC7RKYMMGZA7ZIWMNMG4B5IWSHYFNMP2PC5PCA, seg_pos={0 0}, upload=257111]: Uploading piece... Nov 21 12:00:44.663946742 (291h56m43.215417203s) [noise=true , address=77.90.31.15:4698 , piece_id=ZQ5SBBVFVQQRRPRWQOC5MNLVROAAW3UNJOO3XXESCKCN2ELA5BIA, seg_pos={0 0}, upload=257111]: Uploading piece... Nov 21 12:00:44.664546976 (291h56m43.216017437s) [noise=true , address=145.79.227.31:4798 , piece_id=YWX2POKFFJISTJ5Y6WY5SHP3XIUVNH5CP24AXKU6BLW42RYWKROQ, seg_pos={0 0}, upload=257111]: Uploading piece... Nov 21 12:00:44.665139685 (291h56m43.216610136s) [noise=true , address=77.90.31.17:4770 , piece_id=I7AQRIKKA7FVGTOPEJODHU5B6Z26GHWZXCFQQDXHFRYMHUQVJYZA, seg_pos={0 0}, upload=257111]: Uploading piece... Nov 21 12:00:46.269248547 (291h56m44.820718998s) [upload=257111]: Next returned nil segment Nov 21 12:00:46.269292731 (291h56m44.820763192s) [upload=257111]: Waiting for error group managing segments... Nov 21 12:00:46.651938048 (291h56m45.203408509s) [noise=true , address=145.79.227.11:4731 , piece_id=NZEOWNXMOAIGI5QWGYVJHR4GKK5E7CIUVHGYPY7UBEKQMAIGQPCQ, seg_pos={0 0}, upload=257111]: Done uploading piece. err: <nil> Nov 21 12:00:46.653674856 (291h56m45.205145317s) [noise=true , address=145.79.227.27:4690 , piece_id=EF7B5TVERZJ5RQWZMXXFWHQQXPIAKKTGEU5IRKQMNY4GIOZ7PMOQ, seg_pos={0 0}, upload=257111]: Done uploading piece. err: <nil> Nov 21 12:00:46.656724623 (291h56m45.208195084s) [noise=true , address=145.79.227.22:4511 , piece_id=4MDCBBI4LD4BQQ76YTC75IRLH4HNVG7GEBGBIFDI7Q2TTEHNOBXA, seg_pos={0 0}, upload=257111]: Done uploading piece. err: <nil> Nov 21 12:00:46.663448046 (291h56m45.214918497s) [noise=true , address=145.79.227.23:4547 , piece_id=R7V3ZYNZBPLABNLWIB35BHRKZKFPBQ4RS2G5ZDIOBNVCFFLODDZA, seg_pos={0 0}, upload=257111]: Done uploading piece. err: <nil> Nov 21 12:00:46.672901036 (291h56m45.224371497s) [noise=true , address=145.79.227.24:4583 , piece_id=KJPKPFRWFZD5H7G3RNN2VYLP3SUHBWO3QRUR4T5J6RFYWUSZSSRA, seg_pos={0 0}, upload=257111]: Done uploading piece. err: <nil> Nov 21 12:00:46.677309982 (291h56m45.228780463s) [noise=true , address=145.79.227.31:4798 , piece_id=YWX2POKFFJISTJ5Y6WY5SHP3XIUVNH5CP24AXKU6BLW42RYWKROQ, seg_pos={0 0}, upload=257111]: Done uploading piece. err: <nil> Nov 21 12:00:46.678373266 (291h56m45.229843727s) [noise=true , address=145.79.227.28:4727 , piece_id=4W63CG2IW33MVQHJJJGPJGKZFA4JKBZMHHG5OMEKJ3Y63LVV25FQ, seg_pos={0 0}, upload=257111]: Done uploading piece. err: <nil> Nov 21 12:00:46.686989964 (291h56m45.238460425s) [noise=true , address=145.79.227.20:4438 , piece_id=DEKU5U5LMX3WRZ64WKE3MUQC3C2GBVTXPFVVEX6DN5PSEE53Q5DA, seg_pos={0 0}, upload=257111]: Done uploading piece. err: <nil> Nov 21 12:00:46.690017299 (291h56m45.24148776s) [noise=true , address=145.79.227.15:4876 , piece_id=42VLAZCDHN6CYRZ5EMVPLIVJKQ7TZFJMXFA2AWHUU6S7P37ISMNQ, seg_pos={0 0}, upload=257111]: Done uploading piece. err: <nil> Nov 21 12:00:46.693246267 (291h56m45.244716728s) [noise=true , address=145.79.227.30:4763 , piece_id=7WWH65VO7HUBZFP3ATOKMDTPY6HLJVWIA2XHVI2K2V46FEPK5EPA, seg_pos={0 0}, upload=257111]: Done uploading piece. err: <nil> Nov 21 12:00:47.227653148 (291h56m45.779123609s) [noise=true , address=77.90.31.17:4770 , piece_id=I7AQRIKKA7FVGTOPEJODHU5B6Z26GHWZXCFQQDXHFRYMHUQVJYZA, seg_pos={0 0}, upload=257111]: Done uploading piece. err: <nil> Nov 21 12:00:47.255119890 (291h56m45.806590351s) [noise=true , address=77.90.31.13:4628 , piece_id=KV2WK5KY4B4N3NHYJ6UBAKBLRZWRIFMTVV65SWS2FCUF7UV2AJMQ, seg_pos={0 0}, upload=257111]: Done uploading piece. err: <nil> Nov 21 12:00:47.278989602 (291h56m45.830460064s) [noise=true , address=77.90.31.10:4518 , piece_id=B4KCY7UJHBDHWMLOX7NKAQ75PQUTZTXEDCWEXR43EOY6QFYA3VTA, seg_pos={0 0}, upload=257111]: Done uploading piece. err: <nil> Nov 21 12:00:47.279015171 (291h56m45.830485632s) [seg_pos={0 0}, upload=257111]: Segment reached optimal threshold of 13 pieces. Nov 21 12:00:47.279217887 (291h56m45.830688348s) [noise=true , address=77.90.31.15:4698 , piece_id=ZQ5SBBVFVQQRRPRWQOC5MNLVROAAW3UNJOO3XXESCKCN2ELA5BIA, seg_pos={0 0}, upload=257111]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj): protocol: expected piece hash; context canceled; EOF Nov 21 12:00:47.279226153 (291h56m45.830696614s) [noise=true , address=77.90.31.15:4698 , piece_id=ZQ5SBBVFVQQRRPRWQOC5MNLVROAAW3UNJOO3XXESCKCN2ELA5BIA, seg_pos={0 0}, upload=257111]: Long tail cancellation due to optimal threshold Nov 21 12:00:47.279513941 (291h56m45.830984392s) [noise=true , address=45.91.116.14:4190 , piece_id=EK762P55UQ2NFROSHJYUYIETP5CBQZJHKOTJRUFBKNPTRG7IQCYA, seg_pos={0 0}, upload=257111]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Nov 21 12:00:47.279522787 (291h56m45.830993249s) [noise=true , address=45.91.116.14:4190 , piece_id=EK762P55UQ2NFROSHJYUYIETP5CBQZJHKOTJRUFBKNPTRG7IQCYA, seg_pos={0 0}, upload=257111]: Long tail cancellation due to optimal threshold Nov 21 12:00:47.279754489 (291h56m45.83122495s) [noise=true , address=45.91.116.8:4048 , piece_id=OUL6TXMXPESMVCTC7RKYMMGZA7ZIWMNMG4B5IWSHYFNMP2PC5PCA, seg_pos={0 0}, upload=257111]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: read tcp 188.166.225.170:53244->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 21 12:00:47.279762644 (291h56m45.831233105s) [noise=true , address=45.91.116.8:4048 , piece_id=OUL6TXMXPESMVCTC7RKYMMGZA7ZIWMNMG4B5IWSHYFNMP2PC5PCA, seg_pos={0 0}, upload=257111]: Long tail cancellation due to optimal threshold Nov 21 12:00:47.279782472 (291h56m45.831252934s) [seg_pos={0 0}, upload=257111]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 21 12:00:47.279865650 (291h56m45.831336102s) [upload=257111]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 21 12:00:47.279870900 (291h56m45.831341361s) [upload=257111]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 21 12:00:47.570435510 (291h56m46.121905962s) [upload=257111]: Done uploading segments... Nov 21 12:00:47.570460016 (291h56m46.121930478s) [upload=257111]: Upload finished. err: <nil> Duration: 30.003448038s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 2 out of 60 is failed (threshold=3) [....X.X.....................................................] duration: 6.084197508s trace: https://jaeger.dev.storj.io/trace/a7f69ee106efb7f FAIL (2025-11-21 12:00:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== upload=256586]: Uploading piece... Nov 21 11:25:27.210346024 (291h21m25.761816485s) [noise=true , address=89.149.201.147:4008 , piece_id=HCX7I4ICPEHDWTYDFIQMKZ5EE5CGPQLYAIHF7WZMTUUU6CNZPHTA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.211425289 (291h21m25.76289575s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.213103795 (291h21m25.764574246s) [noise=true , address=185.167.204.205:4022 , piece_id=BES3YK6PYQV2AKOWSAGHF7DGGWGWSWPIJU2ICAQ5WBHKZ5VT5L6A, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.213408395 (291h21m25.764878856s) [noise=true , address=185.167.204.205:4027 , piece_id=IVPONKBF2PIHZQAW3L4YA4HOMQV5BQML6SKKCJLIEGT5SZSQ2AOA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.214501747 (291h21m25.765972208s) [noise=true , address=81.171.14.70:4042 , piece_id=XNREZEUZ5L5KZ3L7ZOL6IRK7IHU6DJN4RLEQZRFMDKDH2JWC24CQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.215013752 (291h21m25.766484213s) [noise=true , address=81.171.14.70:4034 , piece_id=ITRLKYVNFSYBLABNGNFO3HRJNMX3MWB345FIMWSIPBNAVZPA47UA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.215087572 (291h21m25.766558044s) [noise=true , address=185.167.204.205:4031 , piece_id=ZEI2YSIQC655R7WS43WADXGFT5RKT5LFSN6IZEQ6SHTDVOV7BZBQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.216432844 (291h21m25.767903305s) [noise=true , address=185.167.204.206:4056 , piece_id=EAMZIK6AVJL7IMFF5VOLW3SMCGYKSKB2VXJTNUNLJCVIT7EKXNMQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.216662281 (291h21m25.768132742s) [noise=true , address=89.149.201.147:4011 , piece_id=ZATNMD5HOK35AWGJJ5V66VDJ4YGUVZQJ2OBERQL32ZV5GW2HBSMQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.218008314 (291h21m25.769478775s) [noise=true , address=81.171.14.70:4032 , piece_id=CFH2RTR3BLBUB5ZIZCFC6UHT3FHT3B2XSCHQMKXNGDJJGPHEMCTQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:28.035150976 (291h21m26.586621437s) [upload=256586]: Next returned nil segment Nov 21 11:25:28.035169622 (291h21m26.586640083s) [upload=256586]: Waiting for error group managing segments... Nov 21 11:25:28.416646836 (291h21m26.968117317s) [noise=true , address=185.167.204.206:4064 , piece_id=SRFIWFQLRAGCNIKYLHMGT4QRUL3WH2QPLLIKAJ7DLFJEQUW5DPPA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.417338854 (291h21m26.968809325s) [noise=true , address=185.167.204.205:4033 , piece_id=FEWQVHPIKN3Q6O5AZ6AGJBLUPZ34PBZFB5X3XNSKGPOGI4AIO7JA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.421089406 (291h21m26.972559867s) [noise=true , address=185.167.204.206:4072 , piece_id=GYCO53OWGOMOHK4FHC5GENN4LTQGSZKZKOI5V534RLV3LHLPKC6Q, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.426252887 (291h21m26.977723348s) [noise=true , address=185.167.204.206:4046 , piece_id=BZ57ZA35ITEOA6CAZJGNODK5MUTXEZWDV7E2OGBZOSMS4OQZYAEA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.431720689 (291h21m26.98319115s) [noise=true , address=185.167.204.205:4027 , piece_id=IVPONKBF2PIHZQAW3L4YA4HOMQV5BQML6SKKCJLIEGT5SZSQ2AOA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.438243951 (291h21m26.989714412s) [noise=true , address=185.167.204.205:4022 , piece_id=BES3YK6PYQV2AKOWSAGHF7DGGWGWSWPIJU2ICAQ5WBHKZ5VT5L6A, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.443364331 (291h21m26.994834792s) [noise=true , address=185.167.204.206:4056 , piece_id=EAMZIK6AVJL7IMFF5VOLW3SMCGYKSKB2VXJTNUNLJCVIT7EKXNMQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.445594728 (291h21m26.997065179s) [noise=true , address=185.167.204.205:4016 , piece_id=EHQS3546UXRXTL2MGFCSRTPDDPAYXEYLMISPP5AWY337MHX7HFLQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.449325822 (291h21m27.000796283s) [noise=true , address=185.167.204.205:4031 , piece_id=ZEI2YSIQC655R7WS43WADXGFT5RKT5LFSN6IZEQ6SHTDVOV7BZBQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.449761211 (291h21m27.001231672s) [noise=true , address=185.167.204.205:4018 , piece_id=7OSLPWJLKFTIJPZEAFMZOAH6JAAMX6HKROI25FODFYBPNJWSMLWA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.450419845 (291h21m27.001890306s) [noise=true , address=185.167.204.206:4062 , piece_id=UF3S6NAVGS2KQJPGA5WXP7YG4RQO44P445SNPCWXKXMONZAYUTNQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.453201252 (291h21m27.004671713s) [noise=true , address=185.167.204.205:4007 , piece_id=MEBGOPZ6MFMQY4UZARS2LKJKEMNN5YGSLKPJGTWSKS45CUNXSRGQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.661209626 (291h21m28.212680087s) [noise=true , address=89.149.201.147:4016 , piece_id=TGRWGGGBKDANC7YKROWPRMIYNAVFEVE3RXOSUOINFI7KTRS2IYUQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.693169182 (291h21m28.244639643s) [noise=true , address=89.149.201.147:4002 , piece_id=UIYJSCMY3LCLRHZUJB6JFJHHXKMYTVKERTZW7F7X44MWJETPY6XA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.721204497 (291h21m28.272674958s) [noise=true , address=89.149.201.147:4008 , piece_id=HCX7I4ICPEHDWTYDFIQMKZ5EE5CGPQLYAIHF7WZMTUUU6CNZPHTA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.764584711 (291h21m28.316055172s) [noise=true , address=89.149.201.147:4001 , piece_id=QUS6NB3NQBLJFB6YCK3EBDPKCTWEWGKAT3NOR24TVI65SQKHWDOQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.768284075 (291h21m28.319754536s) [noise=true , address=89.149.201.147:4011 , piece_id=ZATNMD5HOK35AWGJJ5V66VDJ4YGUVZQJ2OBERQL32ZV5GW2HBSMQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.789681961 (291h21m28.341152422s) [noise=true , address=89.149.201.147:4003 , piece_id=RUAYGN2RGF5JJTWFEGIHWH3BHEUDSS5DN2QJ6YLSHPF5BGUXD3RA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:37.298598682 (291h21m35.850069143s) [noise=true , address=81.171.14.70:4043 , piece_id=T3NF6JR3VJIGDMPNUBLMJ3AOTQP5IHOGJYORMCSS7ZMOENQETZ7A, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:37.497478356 (291h21m36.048948818s) [noise=true , address=81.171.14.70:4032 , piece_id=CFH2RTR3BLBUB5ZIZCFC6UHT3FHT3B2XSCHQMKXNGDJJGPHEMCTQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:40.406166933 (291h21m38.957637394s) [noise=true , address=81.171.14.70:4045 , piece_id=BRWQ7G3E547PJE2RLGSESIB3HMP5C4M5NSKAJNGWSFXUBEO6SYEQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:42.511041157 (291h21m41.062511608s) [noise=true , address=81.171.14.70:4029 , piece_id=DKJM5E7EY7LOSBSEBQOZRC6CI5ZPPE47HFOFQJROPCUMUV567CAA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:12.012682521 (291h22m10.564152982s) [noise=true , address=81.171.14.70:4042 , piece_id=XNREZEUZ5L5KZ3L7ZOL6IRK7IHU6DJN4RLEQZRFMDKDH2JWC24CQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:15.596046283 (291h22m14.147516744s) [noise=true , address=81.171.14.70:4033 , piece_id=JRKPD4EP6FZM5HISOIDJYZY7FAWWQCADAS47FGW2JM5A7RFJR2BQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:18.453059659 (291h22m17.00453012s) [noise=true , address=81.171.14.70:4046 , piece_id=OSONRSF6KJS7F2QJVCC3FPZK4AV7QPG5YSM47CW5LIW7EME44AWA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:23.442583344 (291h22m21.994053805s) [noise=true , address=81.171.14.70:4034 , piece_id=ITRLKYVNFSYBLABNGNFO3HRJNMX3MWB345FIMWSIPBNAVZPA47UA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:23.442621967 (291h22m21.994092428s) [seg_pos={0 0}, upload=256586]: Segment reached optimal threshold of 26 pieces. Nov 21 11:26:23.442836976 (291h22m21.994307437s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MUVGecysnTSFcJoHUL7QMWb3h6dtXgUd7FbA672fQCyEWhePr): protocol: expected piece hash; context canceled; EOF Nov 21 11:26:23.442845934 (291h22m21.994316395s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Long tail cancellation due to optimal threshold Nov 21 11:26:23.443227491 (291h22m21.994697942s) [noise=true , address=81.171.14.70:4027 , piece_id=4FDFCCWF3EIROQAFLOHGS743523OYGSXWQSPQEVD3AMQ6WKDQI7Q, seg_pos={0 0}, upload=256586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122cYvgTxSvC2aEXnBkRk2o8ARTBXj9gqPoa12vW8hogSN6kpA4): protocol: expected piece hash; context canceled; EOF Nov 21 11:26:23.443237770 (291h22m21.994708231s) [noise=true , address=81.171.14.70:4027 , piece_id=4FDFCCWF3EIROQAFLOHGS743523OYGSXWQSPQEVD3AMQ6WKDQI7Q, seg_pos={0 0}, upload=256586]: Long tail cancellation due to optimal threshold Nov 21 11:26:23.443274720 (291h22m21.994745181s) [seg_pos={0 0}, upload=256586]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Nov 21 11:26:23.443378929 (291h22m21.99484939s) [upload=256586]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 21 11:26:23.443387044 (291h22m21.994857495s) [upload=256586]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 21 11:26:23.685139626 (291h22m22.236610077s) [upload=256586]: Done uploading segments... Nov 21 11:26:23.685160676 (291h22m22.236631127s) [upload=256586]: Upload finished. err: <nil> Duration: 6.084233658s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 31 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.............................] duration: 4.20643034s trace: https://jaeger.dev.storj.io/trace/6af15a3a67b655d5 FAIL (2025-11-21 12:00:17 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 4.20646122s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [X.XX...X.X.X....X............X....X.X.X..X.X..X...X..XXX.X.X] duration: 30.006575203s trace: https://jaeger.dev.storj.io/trace/2d0a737d6f0b3a75 FAIL (2025-11-21 12:00:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: DEHUL2WNISOUZACYFYT4RLV54ZKCSUYGFXNNKZBCXWGHPWCV4H3Q) context deadline exceeded; piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: G5XT2BY35LSKRGSYV3D5Z6R47LVSX5E4TKUWCR4Y6SMXT4W2FQPA) context deadline exceeded ======LOGS====== 4.198056631 (291h55m42.749527092s) [upload=257093]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 11:59:44.444211737 (291h55m42.995682188s) [seg_pos={0 0}, upload=257093]: Begin upload segment... Nov 21 11:59:44.444407079 (291h55m42.99587754s) [seg_pos={0 0}, upload=257093]: Done begin upload segment. Nov 21 11:59:44.444448357 (291h55m42.995918818s) [noise=true , address=77.90.31.12:4589 , piece_id=NQ5CNDUHIVXSO2WIMLFFWB5Q6MV55XJKAUHI3JHX32OA6TJ27LUA, seg_pos={0 0}, upload=257093]: Uploading piece... Nov 21 11:59:44.444723491 (291h55m42.996193952s) [noise=true , address=145.79.227.23:4546 , piece_id=GT7SBNAJWGUUVBOWMJWRFZC6AFT6MESITZOWCBDLQ7UNZ6XUHSZQ, seg_pos={0 0}, upload=257093]: Uploading piece... Nov 21 11:59:44.445731331 (291h55m42.997201792s) [noise=true , address=77.90.31.19:4842 , piece_id=G5XT2BY35LSKRGSYV3D5Z6R47LVSX5E4TKUWCR4Y6SMXT4W2FQPA, seg_pos={0 0}, upload=257093]: Uploading piece... Nov 21 11:59:44.446336312 (291h55m42.997806773s) [noise=true , address=145.79.227.28:4727 , piece_id=A7JE5X6DB7NAQMTZ3IGOTHZTRCES7PEKIBRDM4E55F25UADXHLGQ, seg_pos={0 0}, upload=257093]: Uploading piece... Nov 21 11:59:44.446851453 (291h55m42.998321904s) [noise=true , address=145.79.227.20:4437 , piece_id=Y2ZWMWSV42RJP3QDA77XZKQHMZ3VKFC7RSD732JIIHYTTJLW534A, seg_pos={0 0}, upload=257093]: Uploading piece... Nov 21 11:59:44.447395429 (291h55m42.998865891s) [noise=true , address=77.90.31.13:4628 , piece_id=GPQKPCGQ4XZXS5FFJEQKVGIVD7XUQOFMRHDNOGTGVVLGCQBG4S4A, seg_pos={0 0}, upload=257093]: Uploading piece... Nov 21 11:59:44.447849113 (291h55m42.999319574s) [noise=true , address=145.79.227.9:4367 , piece_id=WRYDKN4XXSWK3XEPHTECXOGZBXER62TX5W4QB6I3BTM7GAPZOLIA, seg_pos={0 0}, upload=257093]: Uploading piece... Nov 21 11:59:44.448351760 (291h55m42.999822231s) [noise=true , address=145.79.227.26:4653 , piece_id=N2HSKRSHAOMKO3RHAHAT6FRQJD4GGYO5WAJF4HIWG3K22JCPXUZA, seg_pos={0 0}, upload=257093]: Uploading piece... Nov 21 11:59:44.448837004 (291h55m43.000307465s) [noise=true , address=45.91.116.11:4119 , piece_id=DEHUL2WNISOUZACYFYT4RLV54ZKCSUYGFXNNKZBCXWGHPWCV4H3Q, seg_pos={0 0}, upload=257093]: Uploading piece... Nov 21 11:59:44.450042300 (291h55m43.001512751s) [noise=true , address=145.79.227.22:4510 , piece_id=PTMTSIGJH24IMSYU2LCVNRHGAKVLMKRPVU73OCLZSSJQTAYB4BNA, seg_pos={0 0}, upload=257093]: Uploading piece... Nov 21 11:59:44.450588840 (291h55m43.002059301s) [noise=true , address=145.79.227.8:4329 , piece_id=OAXMUVPXTZFV5YCYU7IYZ4UKQWHNYQ5HYHXIPQMJSE773HKPTNAQ, seg_pos={0 0}, upload=257093]: Uploading piece... Nov 21 11:59:44.451123318 (291h55m43.002593779s) [noise=true , address=45.91.116.13:4154 , piece_id=CQXYN3CKALT4PIQ2243HJH5YKDGCX5UDGHLFPUIMOMDC2MYYU4YA, seg_pos={0 0}, upload=257093]: Uploading piece... Nov 21 11:59:44.451686801 (291h55m43.003157262s) [noise=true , address=145.79.227.25:4617 , piece_id=5SWWVZFF65GNIUZ7W7EVTDMW6BAYXLBA2FTHAXBUSICSW462WKWA, seg_pos={0 0}, upload=257093]: Uploading piece... Nov 21 11:59:44.452322281 (291h55m43.003792742s) [noise=true , address=145.79.227.13:4801 , piece_id=RMOOF4H6XTER66HQXDHWBFDL5XJESLC6MERUF4N6VQQ3XGVXZAJA, seg_pos={0 0}, upload=257093]: Uploading piece... Nov 21 11:59:44.452883469 (291h55m43.00435393s) [noise=true , address=145.79.227.10:4402 , piece_id=BW73WYCTHSIOV3SB3BIAM6DMLPOQQAC57IRLLV7EHHTDSSHHRL7Q, seg_pos={0 0}, upload=257093]: Uploading piece... Nov 21 11:59:44.453437975 (291h55m43.004908426s) [noise=true , address=45.91.116.4:4937 , piece_id=HJCGEXX3BKHVUEOX2D4NVYDTQ2ECE2LCJ2T3GHRGF6OUEKFPZMAQ, seg_pos={0 0}, upload=257093]: Uploading piece... Nov 21 11:59:46.149272521 (291h55m44.700742982s) [upload=257093]: Next returned nil segment Nov 21 11:59:46.149287108 (291h55m44.700757569s) [upload=257093]: Waiting for error group managing segments... Nov 21 11:59:46.430745998 (291h55m44.982216459s) [noise=true , address=145.79.227.10:4402 , piece_id=BW73WYCTHSIOV3SB3BIAM6DMLPOQQAC57IRLLV7EHHTDSSHHRL7Q, seg_pos={0 0}, upload=257093]: Done uploading piece. err: <nil> Nov 21 11:59:46.435836702 (291h55m44.987307163s) [noise=true , address=145.79.227.8:4329 , piece_id=OAXMUVPXTZFV5YCYU7IYZ4UKQWHNYQ5HYHXIPQMJSE773HKPTNAQ, seg_pos={0 0}, upload=257093]: Done uploading piece. err: <nil> Nov 21 11:59:46.436122096 (291h55m44.987592557s) [noise=true , address=145.79.227.26:4653 , piece_id=N2HSKRSHAOMKO3RHAHAT6FRQJD4GGYO5WAJF4HIWG3K22JCPXUZA, seg_pos={0 0}, upload=257093]: Done uploading piece. err: <nil> Nov 21 11:59:46.437246056 (291h55m44.988716507s) [noise=true , address=145.79.227.25:4617 , piece_id=5SWWVZFF65GNIUZ7W7EVTDMW6BAYXLBA2FTHAXBUSICSW462WKWA, seg_pos={0 0}, upload=257093]: Done uploading piece. err: <nil> Nov 21 11:59:46.442802506 (291h55m44.994272977s) [noise=true , address=145.79.227.13:4801 , piece_id=RMOOF4H6XTER66HQXDHWBFDL5XJESLC6MERUF4N6VQQ3XGVXZAJA, seg_pos={0 0}, upload=257093]: Done uploading piece. err: <nil> Nov 21 11:59:46.455635032 (291h55m45.007105483s) [noise=true , address=145.79.227.20:4437 , piece_id=Y2ZWMWSV42RJP3QDA77XZKQHMZ3VKFC7RSD732JIIHYTTJLW534A, seg_pos={0 0}, upload=257093]: Done uploading piece. err: <nil> Nov 21 11:59:46.457499862 (291h55m45.008970324s) [noise=true , address=145.79.227.9:4367 , piece_id=WRYDKN4XXSWK3XEPHTECXOGZBXER62TX5W4QB6I3BTM7GAPZOLIA, seg_pos={0 0}, upload=257093]: Done uploading piece. err: <nil> Nov 21 11:59:46.459020679 (291h55m45.01049113s) [noise=true , address=145.79.227.22:4510 , piece_id=PTMTSIGJH24IMSYU2LCVNRHGAKVLMKRPVU73OCLZSSJQTAYB4BNA, seg_pos={0 0}, upload=257093]: Done uploading piece. err: <nil> Nov 21 11:59:46.459085482 (291h55m45.010555943s) [noise=true , address=145.79.227.28:4727 , piece_id=A7JE5X6DB7NAQMTZ3IGOTHZTRCES7PEKIBRDM4E55F25UADXHLGQ, seg_pos={0 0}, upload=257093]: Done uploading piece. err: <nil> Nov 21 11:59:46.460866684 (291h55m45.012337135s) [noise=true , address=145.79.227.23:4546 , piece_id=GT7SBNAJWGUUVBOWMJWRFZC6AFT6MESITZOWCBDLQ7UNZ6XUHSZQ, seg_pos={0 0}, upload=257093]: Done uploading piece. err: <nil> Nov 21 11:59:46.999746391 (291h55m45.551216843s) [noise=true , address=45.91.116.4:4937 , piece_id=HJCGEXX3BKHVUEOX2D4NVYDTQ2ECE2LCJ2T3GHRGF6OUEKFPZMAQ, seg_pos={0 0}, upload=257093]: Done uploading piece. err: <nil> Nov 21 11:59:47.025097955 (291h55m45.576568416s) [noise=true , address=77.90.31.19:4842 , piece_id=G5XT2BY35LSKRGSYV3D5Z6R47LVSX5E4TKUWCR4Y6SMXT4W2FQPA, seg_pos={0 0}, upload=257093]: Done uploading piece. err: <nil> Nov 21 11:59:47.121865960 (291h55m45.673336431s) [noise=true , address=45.91.116.11:4119 , piece_id=DEHUL2WNISOUZACYFYT4RLV54ZKCSUYGFXNNKZBCXWGHPWCV4H3Q, seg_pos={0 0}, upload=257093]: Done uploading piece. err: <nil> Nov 21 11:59:47.121893291 (291h55m45.673363742s) [seg_pos={0 0}, upload=257093]: Segment reached optimal threshold of 13 pieces. Nov 21 11:59:47.122112079 (291h55m45.67358254s) [noise=true , address=77.90.31.13:4628 , piece_id=GPQKPCGQ4XZXS5FFJEQKVGIVD7XUQOFMRHDNOGTGVVLGCQBG4S4A, seg_pos={0 0}, upload=257093]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a): protocol: write tcp 188.166.225.170:55566->77.90.31.13:4628: use of closed network connection; write tcp 188.166.225.170:55566->77.90.31.13:4628: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55566->77.90.31.13:4628: use of closed network connection Nov 21 11:59:47.122120284 (291h55m45.673590745s) [noise=true , address=77.90.31.13:4628 , piece_id=GPQKPCGQ4XZXS5FFJEQKVGIVD7XUQOFMRHDNOGTGVVLGCQBG4S4A, seg_pos={0 0}, upload=257093]: Long tail cancellation due to optimal threshold Nov 21 11:59:47.122314153 (291h55m45.673784614s) [noise=true , address=45.91.116.13:4154 , piece_id=CQXYN3CKALT4PIQ2243HJH5YKDGCX5UDGHLFPUIMOMDC2MYYU4YA, seg_pos={0 0}, upload=257093]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: write tcp 188.166.225.170:46116->45.91.116.13:4154: use of closed network connection; write tcp 188.166.225.170:46116->45.91.116.13:4154: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46116->45.91.116.13:4154: use of closed network connection Nov 21 11:59:47.122359410 (291h55m45.673829871s) [noise=true , address=45.91.116.13:4154 , piece_id=CQXYN3CKALT4PIQ2243HJH5YKDGCX5UDGHLFPUIMOMDC2MYYU4YA, seg_pos={0 0}, upload=257093]: Long tail cancellation due to optimal threshold Nov 21 11:59:47.122374598 (291h55m45.673845059s) [noise=true , address=77.90.31.12:4589 , piece_id=NQ5CNDUHIVXSO2WIMLFFWB5Q6MV55XJKAUHI3JHX32OA6TJ27LUA, seg_pos={0 0}, upload=257093]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: write tcp 188.166.225.170:33090->77.90.31.12:4589: use of closed network connection; write tcp 188.166.225.170:33090->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33090->77.90.31.12:4589: use of closed network connection Nov 21 11:59:47.122386972 (291h55m45.673857423s) [noise=true , address=77.90.31.12:4589 , piece_id=NQ5CNDUHIVXSO2WIMLFFWB5Q6MV55XJKAUHI3JHX32OA6TJ27LUA, seg_pos={0 0}, upload=257093]: Long tail cancellation due to optimal threshold Nov 21 11:59:47.122528912 (291h55m45.673999363s) [seg_pos={0 0}, upload=257093]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 21 11:59:47.122877316 (291h55m45.674347777s) [upload=257093]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 21 11:59:47.122886784 (291h55m45.674357245s) [upload=257093]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 21 11:59:47.406105435 (291h55m45.957575896s) [upload=257093]: Done uploading segments... Nov 21 11:59:47.406126023 (291h55m45.957596485s) [upload=257093]: Upload finished. err: <nil> Duration:* 30.006603746s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 32 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX............................] duration: 3.39210982s trace: https://jaeger.dev.storj.io/trace/6af15a3a67b655d5 FAIL (2025-11-21 11:59:50 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 3.392133516s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 2 out of 60 is failed (threshold=3) [.....X.X....................................................] duration: 5.721095522s trace: https://jaeger.dev.storj.io/trace/a7f69ee106efb7f FAIL (2025-11-21 11:59:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== upload=256586]: Uploading piece... Nov 21 11:25:27.210346024 (291h21m25.761816485s) [noise=true , address=89.149.201.147:4008 , piece_id=HCX7I4ICPEHDWTYDFIQMKZ5EE5CGPQLYAIHF7WZMTUUU6CNZPHTA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.211425289 (291h21m25.76289575s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.213103795 (291h21m25.764574246s) [noise=true , address=185.167.204.205:4022 , piece_id=BES3YK6PYQV2AKOWSAGHF7DGGWGWSWPIJU2ICAQ5WBHKZ5VT5L6A, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.213408395 (291h21m25.764878856s) [noise=true , address=185.167.204.205:4027 , piece_id=IVPONKBF2PIHZQAW3L4YA4HOMQV5BQML6SKKCJLIEGT5SZSQ2AOA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.214501747 (291h21m25.765972208s) [noise=true , address=81.171.14.70:4042 , piece_id=XNREZEUZ5L5KZ3L7ZOL6IRK7IHU6DJN4RLEQZRFMDKDH2JWC24CQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.215013752 (291h21m25.766484213s) [noise=true , address=81.171.14.70:4034 , piece_id=ITRLKYVNFSYBLABNGNFO3HRJNMX3MWB345FIMWSIPBNAVZPA47UA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.215087572 (291h21m25.766558044s) [noise=true , address=185.167.204.205:4031 , piece_id=ZEI2YSIQC655R7WS43WADXGFT5RKT5LFSN6IZEQ6SHTDVOV7BZBQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.216432844 (291h21m25.767903305s) [noise=true , address=185.167.204.206:4056 , piece_id=EAMZIK6AVJL7IMFF5VOLW3SMCGYKSKB2VXJTNUNLJCVIT7EKXNMQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.216662281 (291h21m25.768132742s) [noise=true , address=89.149.201.147:4011 , piece_id=ZATNMD5HOK35AWGJJ5V66VDJ4YGUVZQJ2OBERQL32ZV5GW2HBSMQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.218008314 (291h21m25.769478775s) [noise=true , address=81.171.14.70:4032 , piece_id=CFH2RTR3BLBUB5ZIZCFC6UHT3FHT3B2XSCHQMKXNGDJJGPHEMCTQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:28.035150976 (291h21m26.586621437s) [upload=256586]: Next returned nil segment Nov 21 11:25:28.035169622 (291h21m26.586640083s) [upload=256586]: Waiting for error group managing segments... Nov 21 11:25:28.416646836 (291h21m26.968117317s) [noise=true , address=185.167.204.206:4064 , piece_id=SRFIWFQLRAGCNIKYLHMGT4QRUL3WH2QPLLIKAJ7DLFJEQUW5DPPA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.417338854 (291h21m26.968809325s) [noise=true , address=185.167.204.205:4033 , piece_id=FEWQVHPIKN3Q6O5AZ6AGJBLUPZ34PBZFB5X3XNSKGPOGI4AIO7JA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.421089406 (291h21m26.972559867s) [noise=true , address=185.167.204.206:4072 , piece_id=GYCO53OWGOMOHK4FHC5GENN4LTQGSZKZKOI5V534RLV3LHLPKC6Q, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.426252887 (291h21m26.977723348s) [noise=true , address=185.167.204.206:4046 , piece_id=BZ57ZA35ITEOA6CAZJGNODK5MUTXEZWDV7E2OGBZOSMS4OQZYAEA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.431720689 (291h21m26.98319115s) [noise=true , address=185.167.204.205:4027 , piece_id=IVPONKBF2PIHZQAW3L4YA4HOMQV5BQML6SKKCJLIEGT5SZSQ2AOA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.438243951 (291h21m26.989714412s) [noise=true , address=185.167.204.205:4022 , piece_id=BES3YK6PYQV2AKOWSAGHF7DGGWGWSWPIJU2ICAQ5WBHKZ5VT5L6A, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.443364331 (291h21m26.994834792s) [noise=true , address=185.167.204.206:4056 , piece_id=EAMZIK6AVJL7IMFF5VOLW3SMCGYKSKB2VXJTNUNLJCVIT7EKXNMQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.445594728 (291h21m26.997065179s) [noise=true , address=185.167.204.205:4016 , piece_id=EHQS3546UXRXTL2MGFCSRTPDDPAYXEYLMISPP5AWY337MHX7HFLQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.449325822 (291h21m27.000796283s) [noise=true , address=185.167.204.205:4031 , piece_id=ZEI2YSIQC655R7WS43WADXGFT5RKT5LFSN6IZEQ6SHTDVOV7BZBQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.449761211 (291h21m27.001231672s) [noise=true , address=185.167.204.205:4018 , piece_id=7OSLPWJLKFTIJPZEAFMZOAH6JAAMX6HKROI25FODFYBPNJWSMLWA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.450419845 (291h21m27.001890306s) [noise=true , address=185.167.204.206:4062 , piece_id=UF3S6NAVGS2KQJPGA5WXP7YG4RQO44P445SNPCWXKXMONZAYUTNQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.453201252 (291h21m27.004671713s) [noise=true , address=185.167.204.205:4007 , piece_id=MEBGOPZ6MFMQY4UZARS2LKJKEMNN5YGSLKPJGTWSKS45CUNXSRGQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.661209626 (291h21m28.212680087s) [noise=true , address=89.149.201.147:4016 , piece_id=TGRWGGGBKDANC7YKROWPRMIYNAVFEVE3RXOSUOINFI7KTRS2IYUQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.693169182 (291h21m28.244639643s) [noise=true , address=89.149.201.147:4002 , piece_id=UIYJSCMY3LCLRHZUJB6JFJHHXKMYTVKERTZW7F7X44MWJETPY6XA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.721204497 (291h21m28.272674958s) [noise=true , address=89.149.201.147:4008 , piece_id=HCX7I4ICPEHDWTYDFIQMKZ5EE5CGPQLYAIHF7WZMTUUU6CNZPHTA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.764584711 (291h21m28.316055172s) [noise=true , address=89.149.201.147:4001 , piece_id=QUS6NB3NQBLJFB6YCK3EBDPKCTWEWGKAT3NOR24TVI65SQKHWDOQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.768284075 (291h21m28.319754536s) [noise=true , address=89.149.201.147:4011 , piece_id=ZATNMD5HOK35AWGJJ5V66VDJ4YGUVZQJ2OBERQL32ZV5GW2HBSMQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.789681961 (291h21m28.341152422s) [noise=true , address=89.149.201.147:4003 , piece_id=RUAYGN2RGF5JJTWFEGIHWH3BHEUDSS5DN2QJ6YLSHPF5BGUXD3RA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:37.298598682 (291h21m35.850069143s) [noise=true , address=81.171.14.70:4043 , piece_id=T3NF6JR3VJIGDMPNUBLMJ3AOTQP5IHOGJYORMCSS7ZMOENQETZ7A, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:37.497478356 (291h21m36.048948818s) [noise=true , address=81.171.14.70:4032 , piece_id=CFH2RTR3BLBUB5ZIZCFC6UHT3FHT3B2XSCHQMKXNGDJJGPHEMCTQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:40.406166933 (291h21m38.957637394s) [noise=true , address=81.171.14.70:4045 , piece_id=BRWQ7G3E547PJE2RLGSESIB3HMP5C4M5NSKAJNGWSFXUBEO6SYEQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:42.511041157 (291h21m41.062511608s) [noise=true , address=81.171.14.70:4029 , piece_id=DKJM5E7EY7LOSBSEBQOZRC6CI5ZPPE47HFOFQJROPCUMUV567CAA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:12.012682521 (291h22m10.564152982s) [noise=true , address=81.171.14.70:4042 , piece_id=XNREZEUZ5L5KZ3L7ZOL6IRK7IHU6DJN4RLEQZRFMDKDH2JWC24CQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:15.596046283 (291h22m14.147516744s) [noise=true , address=81.171.14.70:4033 , piece_id=JRKPD4EP6FZM5HISOIDJYZY7FAWWQCADAS47FGW2JM5A7RFJR2BQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:18.453059659 (291h22m17.00453012s) [noise=true , address=81.171.14.70:4046 , piece_id=OSONRSF6KJS7F2QJVCC3FPZK4AV7QPG5YSM47CW5LIW7EME44AWA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:23.442583344 (291h22m21.994053805s) [noise=true , address=81.171.14.70:4034 , piece_id=ITRLKYVNFSYBLABNGNFO3HRJNMX3MWB345FIMWSIPBNAVZPA47UA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:23.442621967 (291h22m21.994092428s) [seg_pos={0 0}, upload=256586]: Segment reached optimal threshold of 26 pieces. Nov 21 11:26:23.442836976 (291h22m21.994307437s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MUVGecysnTSFcJoHUL7QMWb3h6dtXgUd7FbA672fQCyEWhePr): protocol: expected piece hash; context canceled; EOF Nov 21 11:26:23.442845934 (291h22m21.994316395s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Long tail cancellation due to optimal threshold Nov 21 11:26:23.443227491 (291h22m21.994697942s) [noise=true , address=81.171.14.70:4027 , piece_id=4FDFCCWF3EIROQAFLOHGS743523OYGSXWQSPQEVD3AMQ6WKDQI7Q, seg_pos={0 0}, upload=256586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122cYvgTxSvC2aEXnBkRk2o8ARTBXj9gqPoa12vW8hogSN6kpA4): protocol: expected piece hash; context canceled; EOF Nov 21 11:26:23.443237770 (291h22m21.994708231s) [noise=true , address=81.171.14.70:4027 , piece_id=4FDFCCWF3EIROQAFLOHGS743523OYGSXWQSPQEVD3AMQ6WKDQI7Q, seg_pos={0 0}, upload=256586]: Long tail cancellation due to optimal threshold Nov 21 11:26:23.443274720 (291h22m21.994745181s) [seg_pos={0 0}, upload=256586]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Nov 21 11:26:23.443378929 (291h22m21.99484939s) [upload=256586]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 21 11:26:23.443387044 (291h22m21.994857495s) [upload=256586]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 21 11:26:23.685139626 (291h22m22.236610077s) [upload=256586]: Done uploading segments... Nov 21 11:26:23.685160676 (291h22m22.236631127s) [upload=256586]: Upload finished. err: <nil> Duration: 5.721129286s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [.X.XX...X.X.X....X............X....X.X.X..X.X..X...X..XXX.X.] duration: 6.403382444s trace: https://jaeger.dev.storj.io/trace/58acb9609da86b42 FAIL (2025-11-21 11:59:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== segments... Nov 21 11:58:23.798165294 (291h54m22.349635755s) [upload=257073]: Got next segment. Inline: false Nov 21 11:58:23.798209027 (291h54m22.349679488s) [upload=257073]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 11:58:23.798215149 (291h54m22.34968561s) [upload=257073]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 11:58:24.043750464 (291h54m22.595220925s) [seg_pos={0 0}, upload=257073]: Begin upload segment... Nov 21 11:58:24.043824775 (291h54m22.595295236s) [seg_pos={0 0}, upload=257073]: Done begin upload segment. Nov 21 11:58:24.043886623 (291h54m22.595357084s) [noise=true , address=145.79.227.10:4403 , piece_id=CT6D2DWCPLKM3RMZTHRZ5CGWK5XA4Z3WRFZFQQEARFG5FJRZRG2Q, seg_pos={0 0}, upload=257073]: Uploading piece... Nov 21 11:58:24.043914426 (291h54m22.595384887s) [noise=true , address=145.79.227.14:4838 , piece_id=I2R25G47MYOF4ZGAQP6K74XNHPIOHRRJXT56NVTTDRXXJ7TV7H6Q, seg_pos={0 0}, upload=257073]: Uploading piece... Nov 21 11:58:24.046406581 (291h54m22.597877052s) [noise=true , address=45.91.116.7:4010 , piece_id=DJIK5ZX2Y65BTPNB72YDZXZREXRNEWVSI2NQR5HHM54OS5QVLM6Q, seg_pos={0 0}, upload=257073]: Uploading piece... Nov 21 11:58:24.047221594 (291h54m22.598692045s) [noise=true , address=145.79.227.8:4331 , piece_id=JE53Y3SUDZDUAONS47KOY5R6D6JM3LE4XT7O5HDB34YXKZRS3QZA, seg_pos={0 0}, upload=257073]: Uploading piece... Nov 21 11:58:24.047751973 (291h54m22.599222435s) [noise=true , address=145.79.227.17:4971 , piece_id=6IE5YC2ZZPI23AUMEYNZF5S2WKO77EHR57HRLWK5STZUVAMGNUTQ, seg_pos={0 0}, upload=257073]: Uploading piece... Nov 21 11:58:24.048334221 (291h54m22.599804694s) [noise=true , address=145.79.227.23:4546 , piece_id=27GJOAY3EEJ2KQP3OEOC4DQ3N4AE3XK3266RJMQSNEIMCPSAIKIQ, seg_pos={0 0}, upload=257073]: Uploading piece... Nov 21 11:58:24.048865253 (291h54m22.600335714s) [noise=true , address=145.79.227.25:4619 , piece_id=5JPN46ZYDROAUDYEYZTEM3W527ZQEDAC5IHOCPYQ2B7J52ZIF2JA, seg_pos={0 0}, upload=257073]: Uploading piece... Nov 21 11:58:24.049423647 (291h54m22.600894108s) [noise=true , address=77.90.31.14:4661 , piece_id=7GNVQHWVEYWHFKFWKOEVPQP73ALNL4CDSOQFTM24MRCKOOK2W72A, seg_pos={0 0}, upload=257073]: Uploading piece... Nov 21 11:58:24.049965448 (291h54m22.601435909s) [noise=true , address=145.79.227.28:4727 , piece_id=IOWSKVNZYX2W47P2ANIZ42JJ3MZ7HZFKZR6GJAZDZL7REZA2GMBA, seg_pos={0 0}, upload=257073]: Uploading piece... Nov 21 11:58:24.050611228 (291h54m22.602081689s) [noise=true , address=45.91.116.9:4263 , piece_id=4JILEFCQOZTXB3QQTA4TKLXPFB7HFA3QM4YCPOXKSHMTIXSYMGXQ, seg_pos={0 0}, upload=257073]: Uploading piece... Nov 21 11:58:24.051249613 (291h54m22.602720074s) [noise=true , address=145.79.227.15:4875 , piece_id=AR26OLW7DWFIEEVWA2IH3Z7I3QHHK6HEMKLFZ7XVXZXH2MF2RDWQ, seg_pos={0 0}, upload=257073]: Uploading piece... Nov 21 11:58:24.051755407 (291h54m22.603225868s) [noise=true , address=45.91.116.13:4154 , piece_id=SH4OYPGES4WWOK6VCXZBG2D3WQWU2PCLJF7U2DNIPRDOQ3CNLQYA, seg_pos={0 0}, upload=257073]: Uploading piece... Nov 21 11:58:24.054542504 (291h54m22.606012965s) [noise=true , address=77.90.31.12:4592 , piece_id=DFW7GNDWCDTXSKSCELBOL2N25DIAOGL5KH5ME5TCV6DJH3VHBVNQ, seg_pos={0 0}, upload=257073]: Uploading piece... Nov 21 11:58:24.055372274 (291h54m22.606842735s) [noise=true , address=77.90.31.10:4520 , piece_id=UGF46ULG4775AETYWQSVRE4MBV2J5VRBTBYGTVQCZ7C725QWJNPA, seg_pos={0 0}, upload=257073]: Uploading piece... Nov 21 11:58:24.056056407 (291h54m22.607526868s) [noise=true , address=145.79.227.21:4473 , piece_id=FKSBVDDTCH5FQBNGQROSUWGVWKUBDQRDDA7CKQCOYSWHS3LS6WOA, seg_pos={0 0}, upload=257073]: Uploading piece... Nov 21 11:58:24.056876177 (291h54m22.608346638s) [noise=true , address=145.79.227.24:4583 , piece_id=3DSY3NGB522AEBZ2VHPO5DTAVPLYHTN3RVB2MCPJH5OULGSPRNDQ, seg_pos={0 0}, upload=257073]: Uploading piece... Nov 21 11:58:25.729310655 (291h54m24.280781116s) [upload=257073]: Next returned nil segment Nov 21 11:58:25.729350490 (291h54m24.280820951s) [upload=257073]: Waiting for error group managing segments... Nov 21 11:58:26.033748111 (291h54m24.585218572s) [noise=true , address=145.79.227.10:4403 , piece_id=CT6D2DWCPLKM3RMZTHRZ5CGWK5XA4Z3WRFZFQQEARFG5FJRZRG2Q, seg_pos={0 0}, upload=257073]: Done uploading piece. err: <nil> Nov 21 11:58:26.040623383 (291h54m24.592093834s) [noise=true , address=145.79.227.17:4971 , piece_id=6IE5YC2ZZPI23AUMEYNZF5S2WKO77EHR57HRLWK5STZUVAMGNUTQ, seg_pos={0 0}, upload=257073]: Done uploading piece. err: <nil> Nov 21 11:58:26.046295113 (291h54m24.597765564s) [noise=true , address=145.79.227.25:4619 , piece_id=5JPN46ZYDROAUDYEYZTEM3W527ZQEDAC5IHOCPYQ2B7J52ZIF2JA, seg_pos={0 0}, upload=257073]: Done uploading piece. err: <nil> Nov 21 11:58:26.050366986 (291h54m24.601837448s) [noise=true , address=145.79.227.14:4838 , piece_id=I2R25G47MYOF4ZGAQP6K74XNHPIOHRRJXT56NVTTDRXXJ7TV7H6Q, seg_pos={0 0}, upload=257073]: Done uploading piece. err: <nil> Nov 21 11:58:26.051064134 (291h54m24.602534595s) [noise=true , address=145.79.227.8:4331 , piece_id=JE53Y3SUDZDUAONS47KOY5R6D6JM3LE4XT7O5HDB34YXKZRS3QZA, seg_pos={0 0}, upload=257073]: Done uploading piece. err: <nil> Nov 21 11:58:26.054474979 (291h54m24.60594544s) [noise=true , address=145.79.227.24:4583 , piece_id=3DSY3NGB522AEBZ2VHPO5DTAVPLYHTN3RVB2MCPJH5OULGSPRNDQ, seg_pos={0 0}, upload=257073]: Done uploading piece. err: <nil> Nov 21 11:58:26.055010749 (291h54m24.6064812s) [noise=true , address=145.79.227.23:4546 , piece_id=27GJOAY3EEJ2KQP3OEOC4DQ3N4AE3XK3266RJMQSNEIMCPSAIKIQ, seg_pos={0 0}, upload=257073]: Done uploading piece. err: <nil> Nov 21 11:58:26.056438778 (291h54m24.607909259s) [noise=true , address=145.79.227.21:4473 , piece_id=FKSBVDDTCH5FQBNGQROSUWGVWKUBDQRDDA7CKQCOYSWHS3LS6WOA, seg_pos={0 0}, upload=257073]: Done uploading piece. err: <nil> Nov 21 11:58:26.062269401 (291h54m24.613739852s) [noise=true , address=145.79.227.28:4727 , piece_id=IOWSKVNZYX2W47P2ANIZ42JJ3MZ7HZFKZR6GJAZDZL7REZA2GMBA, seg_pos={0 0}, upload=257073]: Done uploading piece. err: <nil> Nov 21 11:58:26.073374817 (291h54m24.624845268s) [noise=true , address=145.79.227.15:4875 , piece_id=AR26OLW7DWFIEEVWA2IH3Z7I3QHHK6HEMKLFZ7XVXZXH2MF2RDWQ, seg_pos={0 0}, upload=257073]: Done uploading piece. err: <nil> Nov 21 11:58:26.559914971 (291h54m25.111385432s) [noise=true , address=77.90.31.10:4520 , piece_id=UGF46ULG4775AETYWQSVRE4MBV2J5VRBTBYGTVQCZ7C725QWJNPA, seg_pos={0 0}, upload=257073]: Done uploading piece. err: <nil> Nov 21 11:58:26.590444054 (291h54m25.141914515s) [noise=true , address=77.90.31.14:4661 , piece_id=7GNVQHWVEYWHFKFWKOEVPQP73ALNL4CDSOQFTM24MRCKOOK2W72A, seg_pos={0 0}, upload=257073]: Done uploading piece. err: <nil> Nov 21 11:58:26.592964242 (291h54m25.144434703s) [noise=true , address=45.91.116.7:4010 , piece_id=DJIK5ZX2Y65BTPNB72YDZXZREXRNEWVSI2NQR5HHM54OS5QVLM6Q, seg_pos={0 0}, upload=257073]: Done uploading piece. err: <nil> Nov 21 11:58:26.592978019 (291h54m25.14444847s) [seg_pos={0 0}, upload=257073]: Segment reached optimal threshold of 13 pieces. Nov 21 11:58:26.593165876 (291h54m25.144636327s) [noise=true , address=45.91.116.13:4154 , piece_id=SH4OYPGES4WWOK6VCXZBG2D3WQWU2PCLJF7U2DNIPRDOQ3CNLQYA, seg_pos={0 0}, upload=257073]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: write tcp 188.166.225.170:32964->45.91.116.13:4154: use of closed network connection; write tcp 188.166.225.170:32964->45.91.116.13:4154: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32964->45.91.116.13:4154: use of closed network connection Nov 21 11:58:26.593172589 (291h54m25.14464305s) [noise=true , address=45.91.116.13:4154 , piece_id=SH4OYPGES4WWOK6VCXZBG2D3WQWU2PCLJF7U2DNIPRDOQ3CNLQYA, seg_pos={0 0}, upload=257073]: Long tail cancellation due to optimal threshold Nov 21 11:58:26.593176708 (291h54m25.144647169s) [noise=true , address=45.91.116.9:4263 , piece_id=4JILEFCQOZTXB3QQTA4TKLXPFB7HFA3QM4YCPOXKSHMTIXSYMGXQ, seg_pos={0 0}, upload=257073]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: write tcp 188.166.225.170:44134->45.91.116.9:4263: use of closed network connection; write tcp 188.166.225.170:44134->45.91.116.9:4263: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44134->45.91.116.9:4263: use of closed network connection Nov 21 11:58:26.593188961 (291h54m25.144659422s) [noise=true , address=45.91.116.9:4263 , piece_id=4JILEFCQOZTXB3QQTA4TKLXPFB7HFA3QM4YCPOXKSHMTIXSYMGXQ, seg_pos={0 0}, upload=257073]: Long tail cancellation due to optimal threshold Nov 21 11:58:26.593421203 (291h54m25.144891674s) [noise=true , address=77.90.31.12:4592 , piece_id=DFW7GNDWCDTXSKSCELBOL2N25DIAOGL5KH5ME5TCV6DJH3VHBVNQ, seg_pos={0 0}, upload=257073]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3): protocol: expected piece hash; context canceled; EOF Nov 21 11:58:26.593435059 (291h54m25.14490555s) [noise=true , address=77.90.31.12:4592 , piece_id=DFW7GNDWCDTXSKSCELBOL2N25DIAOGL5KH5ME5TCV6DJH3VHBVNQ, seg_pos={0 0}, upload=257073]: Long tail cancellation due to optimal threshold Nov 21 11:58:26.593459887 (291h54m25.144930348s) [seg_pos={0 0}, upload=257073]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 21 11:58:26.593570738 (291h54m25.145041189s) [upload=257073]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 21 11:58:26.593578963 (291h54m25.145049424s) [upload=257073]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 21 11:58:26.889858055 (291h54m25.441328526s) [upload=257073]: Done uploading segments... Nov 21 11:58:26.889882130 (291h54m25.441352591s) [upload=257073]: Upload finished. err: <nil> Duration: 6.403403694s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 2 out of 60 is failed (threshold=3) [......X.X...................................................] duration: 8.92992406s trace: https://jaeger.dev.storj.io/trace/a7f69ee106efb7f FAIL (2025-11-21 11:59:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== upload=256586]: Uploading piece... Nov 21 11:25:27.210346024 (291h21m25.761816485s) [noise=true , address=89.149.201.147:4008 , piece_id=HCX7I4ICPEHDWTYDFIQMKZ5EE5CGPQLYAIHF7WZMTUUU6CNZPHTA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.211425289 (291h21m25.76289575s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.213103795 (291h21m25.764574246s) [noise=true , address=185.167.204.205:4022 , piece_id=BES3YK6PYQV2AKOWSAGHF7DGGWGWSWPIJU2ICAQ5WBHKZ5VT5L6A, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.213408395 (291h21m25.764878856s) [noise=true , address=185.167.204.205:4027 , piece_id=IVPONKBF2PIHZQAW3L4YA4HOMQV5BQML6SKKCJLIEGT5SZSQ2AOA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.214501747 (291h21m25.765972208s) [noise=true , address=81.171.14.70:4042 , piece_id=XNREZEUZ5L5KZ3L7ZOL6IRK7IHU6DJN4RLEQZRFMDKDH2JWC24CQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.215013752 (291h21m25.766484213s) [noise=true , address=81.171.14.70:4034 , piece_id=ITRLKYVNFSYBLABNGNFO3HRJNMX3MWB345FIMWSIPBNAVZPA47UA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.215087572 (291h21m25.766558044s) [noise=true , address=185.167.204.205:4031 , piece_id=ZEI2YSIQC655R7WS43WADXGFT5RKT5LFSN6IZEQ6SHTDVOV7BZBQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.216432844 (291h21m25.767903305s) [noise=true , address=185.167.204.206:4056 , piece_id=EAMZIK6AVJL7IMFF5VOLW3SMCGYKSKB2VXJTNUNLJCVIT7EKXNMQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.216662281 (291h21m25.768132742s) [noise=true , address=89.149.201.147:4011 , piece_id=ZATNMD5HOK35AWGJJ5V66VDJ4YGUVZQJ2OBERQL32ZV5GW2HBSMQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.218008314 (291h21m25.769478775s) [noise=true , address=81.171.14.70:4032 , piece_id=CFH2RTR3BLBUB5ZIZCFC6UHT3FHT3B2XSCHQMKXNGDJJGPHEMCTQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:28.035150976 (291h21m26.586621437s) [upload=256586]: Next returned nil segment Nov 21 11:25:28.035169622 (291h21m26.586640083s) [upload=256586]: Waiting for error group managing segments... Nov 21 11:25:28.416646836 (291h21m26.968117317s) [noise=true , address=185.167.204.206:4064 , piece_id=SRFIWFQLRAGCNIKYLHMGT4QRUL3WH2QPLLIKAJ7DLFJEQUW5DPPA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.417338854 (291h21m26.968809325s) [noise=true , address=185.167.204.205:4033 , piece_id=FEWQVHPIKN3Q6O5AZ6AGJBLUPZ34PBZFB5X3XNSKGPOGI4AIO7JA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.421089406 (291h21m26.972559867s) [noise=true , address=185.167.204.206:4072 , piece_id=GYCO53OWGOMOHK4FHC5GENN4LTQGSZKZKOI5V534RLV3LHLPKC6Q, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.426252887 (291h21m26.977723348s) [noise=true , address=185.167.204.206:4046 , piece_id=BZ57ZA35ITEOA6CAZJGNODK5MUTXEZWDV7E2OGBZOSMS4OQZYAEA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.431720689 (291h21m26.98319115s) [noise=true , address=185.167.204.205:4027 , piece_id=IVPONKBF2PIHZQAW3L4YA4HOMQV5BQML6SKKCJLIEGT5SZSQ2AOA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.438243951 (291h21m26.989714412s) [noise=true , address=185.167.204.205:4022 , piece_id=BES3YK6PYQV2AKOWSAGHF7DGGWGWSWPIJU2ICAQ5WBHKZ5VT5L6A, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.443364331 (291h21m26.994834792s) [noise=true , address=185.167.204.206:4056 , piece_id=EAMZIK6AVJL7IMFF5VOLW3SMCGYKSKB2VXJTNUNLJCVIT7EKXNMQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.445594728 (291h21m26.997065179s) [noise=true , address=185.167.204.205:4016 , piece_id=EHQS3546UXRXTL2MGFCSRTPDDPAYXEYLMISPP5AWY337MHX7HFLQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.449325822 (291h21m27.000796283s) [noise=true , address=185.167.204.205:4031 , piece_id=ZEI2YSIQC655R7WS43WADXGFT5RKT5LFSN6IZEQ6SHTDVOV7BZBQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.449761211 (291h21m27.001231672s) [noise=true , address=185.167.204.205:4018 , piece_id=7OSLPWJLKFTIJPZEAFMZOAH6JAAMX6HKROI25FODFYBPNJWSMLWA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.450419845 (291h21m27.001890306s) [noise=true , address=185.167.204.206:4062 , piece_id=UF3S6NAVGS2KQJPGA5WXP7YG4RQO44P445SNPCWXKXMONZAYUTNQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.453201252 (291h21m27.004671713s) [noise=true , address=185.167.204.205:4007 , piece_id=MEBGOPZ6MFMQY4UZARS2LKJKEMNN5YGSLKPJGTWSKS45CUNXSRGQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.661209626 (291h21m28.212680087s) [noise=true , address=89.149.201.147:4016 , piece_id=TGRWGGGBKDANC7YKROWPRMIYNAVFEVE3RXOSUOINFI7KTRS2IYUQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.693169182 (291h21m28.244639643s) [noise=true , address=89.149.201.147:4002 , piece_id=UIYJSCMY3LCLRHZUJB6JFJHHXKMYTVKERTZW7F7X44MWJETPY6XA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.721204497 (291h21m28.272674958s) [noise=true , address=89.149.201.147:4008 , piece_id=HCX7I4ICPEHDWTYDFIQMKZ5EE5CGPQLYAIHF7WZMTUUU6CNZPHTA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.764584711 (291h21m28.316055172s) [noise=true , address=89.149.201.147:4001 , piece_id=QUS6NB3NQBLJFB6YCK3EBDPKCTWEWGKAT3NOR24TVI65SQKHWDOQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.768284075 (291h21m28.319754536s) [noise=true , address=89.149.201.147:4011 , piece_id=ZATNMD5HOK35AWGJJ5V66VDJ4YGUVZQJ2OBERQL32ZV5GW2HBSMQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.789681961 (291h21m28.341152422s) [noise=true , address=89.149.201.147:4003 , piece_id=RUAYGN2RGF5JJTWFEGIHWH3BHEUDSS5DN2QJ6YLSHPF5BGUXD3RA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:37.298598682 (291h21m35.850069143s) [noise=true , address=81.171.14.70:4043 , piece_id=T3NF6JR3VJIGDMPNUBLMJ3AOTQP5IHOGJYORMCSS7ZMOENQETZ7A, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:37.497478356 (291h21m36.048948818s) [noise=true , address=81.171.14.70:4032 , piece_id=CFH2RTR3BLBUB5ZIZCFC6UHT3FHT3B2XSCHQMKXNGDJJGPHEMCTQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:40.406166933 (291h21m38.957637394s) [noise=true , address=81.171.14.70:4045 , piece_id=BRWQ7G3E547PJE2RLGSESIB3HMP5C4M5NSKAJNGWSFXUBEO6SYEQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:42.511041157 (291h21m41.062511608s) [noise=true , address=81.171.14.70:4029 , piece_id=DKJM5E7EY7LOSBSEBQOZRC6CI5ZPPE47HFOFQJROPCUMUV567CAA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:12.012682521 (291h22m10.564152982s) [noise=true , address=81.171.14.70:4042 , piece_id=XNREZEUZ5L5KZ3L7ZOL6IRK7IHU6DJN4RLEQZRFMDKDH2JWC24CQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:15.596046283 (291h22m14.147516744s) [noise=true , address=81.171.14.70:4033 , piece_id=JRKPD4EP6FZM5HISOIDJYZY7FAWWQCADAS47FGW2JM5A7RFJR2BQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:18.453059659 (291h22m17.00453012s) [noise=true , address=81.171.14.70:4046 , piece_id=OSONRSF6KJS7F2QJVCC3FPZK4AV7QPG5YSM47CW5LIW7EME44AWA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:23.442583344 (291h22m21.994053805s) [noise=true , address=81.171.14.70:4034 , piece_id=ITRLKYVNFSYBLABNGNFO3HRJNMX3MWB345FIMWSIPBNAVZPA47UA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:23.442621967 (291h22m21.994092428s) [seg_pos={0 0}, upload=256586]: Segment reached optimal threshold of 26 pieces. Nov 21 11:26:23.442836976 (291h22m21.994307437s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MUVGecysnTSFcJoHUL7QMWb3h6dtXgUd7FbA672fQCyEWhePr): protocol: expected piece hash; context canceled; EOF Nov 21 11:26:23.442845934 (291h22m21.994316395s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Long tail cancellation due to optimal threshold Nov 21 11:26:23.443227491 (291h22m21.994697942s) [noise=true , address=81.171.14.70:4027 , piece_id=4FDFCCWF3EIROQAFLOHGS743523OYGSXWQSPQEVD3AMQ6WKDQI7Q, seg_pos={0 0}, upload=256586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122cYvgTxSvC2aEXnBkRk2o8ARTBXj9gqPoa12vW8hogSN6kpA4): protocol: expected piece hash; context canceled; EOF Nov 21 11:26:23.443237770 (291h22m21.994708231s) [noise=true , address=81.171.14.70:4027 , piece_id=4FDFCCWF3EIROQAFLOHGS743523OYGSXWQSPQEVD3AMQ6WKDQI7Q, seg_pos={0 0}, upload=256586]: Long tail cancellation due to optimal threshold Nov 21 11:26:23.443274720 (291h22m21.994745181s) [seg_pos={0 0}, upload=256586]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Nov 21 11:26:23.443378929 (291h22m21.99484939s) [upload=256586]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 21 11:26:23.443387044 (291h22m21.994857495s) [upload=256586]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 21 11:26:23.685139626 (291h22m22.236610077s) [upload=256586]: Done uploading segments... Nov 21 11:26:23.685160676 (291h22m22.236631127s) [upload=256586]: Upload finished. err: <nil> Duration: 8.929944509s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [..X.XX...X.X.X....X............X....X.X.X..X.X..X...X..XXX.X] duration: 30.003520926s trace: https://jaeger.dev.storj.io/trace/58acb9609da86b42 FAIL (2025-11-21 11:58:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== segments... Nov 21 11:58:23.798165294 (291h54m22.349635755s) [upload=257073]: Got next segment. Inline: false Nov 21 11:58:23.798209027 (291h54m22.349679488s) [upload=257073]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 11:58:23.798215149 (291h54m22.34968561s) [upload=257073]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 11:58:24.043750464 (291h54m22.595220925s) [seg_pos={0 0}, upload=257073]: Begin upload segment... Nov 21 11:58:24.043824775 (291h54m22.595295236s) [seg_pos={0 0}, upload=257073]: Done begin upload segment. Nov 21 11:58:24.043886623 (291h54m22.595357084s) [noise=true , address=145.79.227.10:4403 , piece_id=CT6D2DWCPLKM3RMZTHRZ5CGWK5XA4Z3WRFZFQQEARFG5FJRZRG2Q, seg_pos={0 0}, upload=257073]: Uploading piece... Nov 21 11:58:24.043914426 (291h54m22.595384887s) [noise=true , address=145.79.227.14:4838 , piece_id=I2R25G47MYOF4ZGAQP6K74XNHPIOHRRJXT56NVTTDRXXJ7TV7H6Q, seg_pos={0 0}, upload=257073]: Uploading piece... Nov 21 11:58:24.046406581 (291h54m22.597877052s) [noise=true , address=45.91.116.7:4010 , piece_id=DJIK5ZX2Y65BTPNB72YDZXZREXRNEWVSI2NQR5HHM54OS5QVLM6Q, seg_pos={0 0}, upload=257073]: Uploading piece... Nov 21 11:58:24.047221594 (291h54m22.598692045s) [noise=true , address=145.79.227.8:4331 , piece_id=JE53Y3SUDZDUAONS47KOY5R6D6JM3LE4XT7O5HDB34YXKZRS3QZA, seg_pos={0 0}, upload=257073]: Uploading piece... Nov 21 11:58:24.047751973 (291h54m22.599222435s) [noise=true , address=145.79.227.17:4971 , piece_id=6IE5YC2ZZPI23AUMEYNZF5S2WKO77EHR57HRLWK5STZUVAMGNUTQ, seg_pos={0 0}, upload=257073]: Uploading piece... Nov 21 11:58:24.048334221 (291h54m22.599804694s) [noise=true , address=145.79.227.23:4546 , piece_id=27GJOAY3EEJ2KQP3OEOC4DQ3N4AE3XK3266RJMQSNEIMCPSAIKIQ, seg_pos={0 0}, upload=257073]: Uploading piece... Nov 21 11:58:24.048865253 (291h54m22.600335714s) [noise=true , address=145.79.227.25:4619 , piece_id=5JPN46ZYDROAUDYEYZTEM3W527ZQEDAC5IHOCPYQ2B7J52ZIF2JA, seg_pos={0 0}, upload=257073]: Uploading piece... Nov 21 11:58:24.049423647 (291h54m22.600894108s) [noise=true , address=77.90.31.14:4661 , piece_id=7GNVQHWVEYWHFKFWKOEVPQP73ALNL4CDSOQFTM24MRCKOOK2W72A, seg_pos={0 0}, upload=257073]: Uploading piece... Nov 21 11:58:24.049965448 (291h54m22.601435909s) [noise=true , address=145.79.227.28:4727 , piece_id=IOWSKVNZYX2W47P2ANIZ42JJ3MZ7HZFKZR6GJAZDZL7REZA2GMBA, seg_pos={0 0}, upload=257073]: Uploading piece... Nov 21 11:58:24.050611228 (291h54m22.602081689s) [noise=true , address=45.91.116.9:4263 , piece_id=4JILEFCQOZTXB3QQTA4TKLXPFB7HFA3QM4YCPOXKSHMTIXSYMGXQ, seg_pos={0 0}, upload=257073]: Uploading piece... Nov 21 11:58:24.051249613 (291h54m22.602720074s) [noise=true , address=145.79.227.15:4875 , piece_id=AR26OLW7DWFIEEVWA2IH3Z7I3QHHK6HEMKLFZ7XVXZXH2MF2RDWQ, seg_pos={0 0}, upload=257073]: Uploading piece... Nov 21 11:58:24.051755407 (291h54m22.603225868s) [noise=true , address=45.91.116.13:4154 , piece_id=SH4OYPGES4WWOK6VCXZBG2D3WQWU2PCLJF7U2DNIPRDOQ3CNLQYA, seg_pos={0 0}, upload=257073]: Uploading piece... Nov 21 11:58:24.054542504 (291h54m22.606012965s) [noise=true , address=77.90.31.12:4592 , piece_id=DFW7GNDWCDTXSKSCELBOL2N25DIAOGL5KH5ME5TCV6DJH3VHBVNQ, seg_pos={0 0}, upload=257073]: Uploading piece... Nov 21 11:58:24.055372274 (291h54m22.606842735s) [noise=true , address=77.90.31.10:4520 , piece_id=UGF46ULG4775AETYWQSVRE4MBV2J5VRBTBYGTVQCZ7C725QWJNPA, seg_pos={0 0}, upload=257073]: Uploading piece... Nov 21 11:58:24.056056407 (291h54m22.607526868s) [noise=true , address=145.79.227.21:4473 , piece_id=FKSBVDDTCH5FQBNGQROSUWGVWKUBDQRDDA7CKQCOYSWHS3LS6WOA, seg_pos={0 0}, upload=257073]: Uploading piece... Nov 21 11:58:24.056876177 (291h54m22.608346638s) [noise=true , address=145.79.227.24:4583 , piece_id=3DSY3NGB522AEBZ2VHPO5DTAVPLYHTN3RVB2MCPJH5OULGSPRNDQ, seg_pos={0 0}, upload=257073]: Uploading piece... Nov 21 11:58:25.729310655 (291h54m24.280781116s) [upload=257073]: Next returned nil segment Nov 21 11:58:25.729350490 (291h54m24.280820951s) [upload=257073]: Waiting for error group managing segments... Nov 21 11:58:26.033748111 (291h54m24.585218572s) [noise=true , address=145.79.227.10:4403 , piece_id=CT6D2DWCPLKM3RMZTHRZ5CGWK5XA4Z3WRFZFQQEARFG5FJRZRG2Q, seg_pos={0 0}, upload=257073]: Done uploading piece. err: <nil> Nov 21 11:58:26.040623383 (291h54m24.592093834s) [noise=true , address=145.79.227.17:4971 , piece_id=6IE5YC2ZZPI23AUMEYNZF5S2WKO77EHR57HRLWK5STZUVAMGNUTQ, seg_pos={0 0}, upload=257073]: Done uploading piece. err: <nil> Nov 21 11:58:26.046295113 (291h54m24.597765564s) [noise=true , address=145.79.227.25:4619 , piece_id=5JPN46ZYDROAUDYEYZTEM3W527ZQEDAC5IHOCPYQ2B7J52ZIF2JA, seg_pos={0 0}, upload=257073]: Done uploading piece. err: <nil> Nov 21 11:58:26.050366986 (291h54m24.601837448s) [noise=true , address=145.79.227.14:4838 , piece_id=I2R25G47MYOF4ZGAQP6K74XNHPIOHRRJXT56NVTTDRXXJ7TV7H6Q, seg_pos={0 0}, upload=257073]: Done uploading piece. err: <nil> Nov 21 11:58:26.051064134 (291h54m24.602534595s) [noise=true , address=145.79.227.8:4331 , piece_id=JE53Y3SUDZDUAONS47KOY5R6D6JM3LE4XT7O5HDB34YXKZRS3QZA, seg_pos={0 0}, upload=257073]: Done uploading piece. err: <nil> Nov 21 11:58:26.054474979 (291h54m24.60594544s) [noise=true , address=145.79.227.24:4583 , piece_id=3DSY3NGB522AEBZ2VHPO5DTAVPLYHTN3RVB2MCPJH5OULGSPRNDQ, seg_pos={0 0}, upload=257073]: Done uploading piece. err: <nil> Nov 21 11:58:26.055010749 (291h54m24.6064812s) [noise=true , address=145.79.227.23:4546 , piece_id=27GJOAY3EEJ2KQP3OEOC4DQ3N4AE3XK3266RJMQSNEIMCPSAIKIQ, seg_pos={0 0}, upload=257073]: Done uploading piece. err: <nil> Nov 21 11:58:26.056438778 (291h54m24.607909259s) [noise=true , address=145.79.227.21:4473 , piece_id=FKSBVDDTCH5FQBNGQROSUWGVWKUBDQRDDA7CKQCOYSWHS3LS6WOA, seg_pos={0 0}, upload=257073]: Done uploading piece. err: <nil> Nov 21 11:58:26.062269401 (291h54m24.613739852s) [noise=true , address=145.79.227.28:4727 , piece_id=IOWSKVNZYX2W47P2ANIZ42JJ3MZ7HZFKZR6GJAZDZL7REZA2GMBA, seg_pos={0 0}, upload=257073]: Done uploading piece. err: <nil> Nov 21 11:58:26.073374817 (291h54m24.624845268s) [noise=true , address=145.79.227.15:4875 , piece_id=AR26OLW7DWFIEEVWA2IH3Z7I3QHHK6HEMKLFZ7XVXZXH2MF2RDWQ, seg_pos={0 0}, upload=257073]: Done uploading piece. err: <nil> Nov 21 11:58:26.559914971 (291h54m25.111385432s) [noise=true , address=77.90.31.10:4520 , piece_id=UGF46ULG4775AETYWQSVRE4MBV2J5VRBTBYGTVQCZ7C725QWJNPA, seg_pos={0 0}, upload=257073]: Done uploading piece. err: <nil> Nov 21 11:58:26.590444054 (291h54m25.141914515s) [noise=true , address=77.90.31.14:4661 , piece_id=7GNVQHWVEYWHFKFWKOEVPQP73ALNL4CDSOQFTM24MRCKOOK2W72A, seg_pos={0 0}, upload=257073]: Done uploading piece. err: <nil> Nov 21 11:58:26.592964242 (291h54m25.144434703s) [noise=true , address=45.91.116.7:4010 , piece_id=DJIK5ZX2Y65BTPNB72YDZXZREXRNEWVSI2NQR5HHM54OS5QVLM6Q, seg_pos={0 0}, upload=257073]: Done uploading piece. err: <nil> Nov 21 11:58:26.592978019 (291h54m25.14444847s) [seg_pos={0 0}, upload=257073]: Segment reached optimal threshold of 13 pieces. Nov 21 11:58:26.593165876 (291h54m25.144636327s) [noise=true , address=45.91.116.13:4154 , piece_id=SH4OYPGES4WWOK6VCXZBG2D3WQWU2PCLJF7U2DNIPRDOQ3CNLQYA, seg_pos={0 0}, upload=257073]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: write tcp 188.166.225.170:32964->45.91.116.13:4154: use of closed network connection; write tcp 188.166.225.170:32964->45.91.116.13:4154: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32964->45.91.116.13:4154: use of closed network connection Nov 21 11:58:26.593172589 (291h54m25.14464305s) [noise=true , address=45.91.116.13:4154 , piece_id=SH4OYPGES4WWOK6VCXZBG2D3WQWU2PCLJF7U2DNIPRDOQ3CNLQYA, seg_pos={0 0}, upload=257073]: Long tail cancellation due to optimal threshold Nov 21 11:58:26.593176708 (291h54m25.144647169s) [noise=true , address=45.91.116.9:4263 , piece_id=4JILEFCQOZTXB3QQTA4TKLXPFB7HFA3QM4YCPOXKSHMTIXSYMGXQ, seg_pos={0 0}, upload=257073]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: write tcp 188.166.225.170:44134->45.91.116.9:4263: use of closed network connection; write tcp 188.166.225.170:44134->45.91.116.9:4263: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44134->45.91.116.9:4263: use of closed network connection Nov 21 11:58:26.593188961 (291h54m25.144659422s) [noise=true , address=45.91.116.9:4263 , piece_id=4JILEFCQOZTXB3QQTA4TKLXPFB7HFA3QM4YCPOXKSHMTIXSYMGXQ, seg_pos={0 0}, upload=257073]: Long tail cancellation due to optimal threshold Nov 21 11:58:26.593421203 (291h54m25.144891674s) [noise=true , address=77.90.31.12:4592 , piece_id=DFW7GNDWCDTXSKSCELBOL2N25DIAOGL5KH5ME5TCV6DJH3VHBVNQ, seg_pos={0 0}, upload=257073]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3): protocol: expected piece hash; context canceled; EOF Nov 21 11:58:26.593435059 (291h54m25.14490555s) [noise=true , address=77.90.31.12:4592 , piece_id=DFW7GNDWCDTXSKSCELBOL2N25DIAOGL5KH5ME5TCV6DJH3VHBVNQ, seg_pos={0 0}, upload=257073]: Long tail cancellation due to optimal threshold Nov 21 11:58:26.593459887 (291h54m25.144930348s) [seg_pos={0 0}, upload=257073]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 21 11:58:26.593570738 (291h54m25.145041189s) [upload=257073]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 21 11:58:26.593578963 (291h54m25.145049424s) [upload=257073]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 21 11:58:26.889858055 (291h54m25.441328526s) [upload=257073]: Done uploading segments... Nov 21 11:58:26.889882130 (291h54m25.441352591s) [upload=257073]: Upload finished. err: <nil> Duration: 30.003541314s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 2 out of 60 is failed (threshold=3) [.......X.X..................................................] duration: 5.74198459s trace: https://jaeger.dev.storj.io/trace/a7f69ee106efb7f FAIL (2025-11-21 11:58:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== upload=256586]: Uploading piece... Nov 21 11:25:27.210346024 (291h21m25.761816485s) [noise=true , address=89.149.201.147:4008 , piece_id=HCX7I4ICPEHDWTYDFIQMKZ5EE5CGPQLYAIHF7WZMTUUU6CNZPHTA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.211425289 (291h21m25.76289575s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.213103795 (291h21m25.764574246s) [noise=true , address=185.167.204.205:4022 , piece_id=BES3YK6PYQV2AKOWSAGHF7DGGWGWSWPIJU2ICAQ5WBHKZ5VT5L6A, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.213408395 (291h21m25.764878856s) [noise=true , address=185.167.204.205:4027 , piece_id=IVPONKBF2PIHZQAW3L4YA4HOMQV5BQML6SKKCJLIEGT5SZSQ2AOA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.214501747 (291h21m25.765972208s) [noise=true , address=81.171.14.70:4042 , piece_id=XNREZEUZ5L5KZ3L7ZOL6IRK7IHU6DJN4RLEQZRFMDKDH2JWC24CQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.215013752 (291h21m25.766484213s) [noise=true , address=81.171.14.70:4034 , piece_id=ITRLKYVNFSYBLABNGNFO3HRJNMX3MWB345FIMWSIPBNAVZPA47UA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.215087572 (291h21m25.766558044s) [noise=true , address=185.167.204.205:4031 , piece_id=ZEI2YSIQC655R7WS43WADXGFT5RKT5LFSN6IZEQ6SHTDVOV7BZBQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.216432844 (291h21m25.767903305s) [noise=true , address=185.167.204.206:4056 , piece_id=EAMZIK6AVJL7IMFF5VOLW3SMCGYKSKB2VXJTNUNLJCVIT7EKXNMQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.216662281 (291h21m25.768132742s) [noise=true , address=89.149.201.147:4011 , piece_id=ZATNMD5HOK35AWGJJ5V66VDJ4YGUVZQJ2OBERQL32ZV5GW2HBSMQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.218008314 (291h21m25.769478775s) [noise=true , address=81.171.14.70:4032 , piece_id=CFH2RTR3BLBUB5ZIZCFC6UHT3FHT3B2XSCHQMKXNGDJJGPHEMCTQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:28.035150976 (291h21m26.586621437s) [upload=256586]: Next returned nil segment Nov 21 11:25:28.035169622 (291h21m26.586640083s) [upload=256586]: Waiting for error group managing segments... Nov 21 11:25:28.416646836 (291h21m26.968117317s) [noise=true , address=185.167.204.206:4064 , piece_id=SRFIWFQLRAGCNIKYLHMGT4QRUL3WH2QPLLIKAJ7DLFJEQUW5DPPA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.417338854 (291h21m26.968809325s) [noise=true , address=185.167.204.205:4033 , piece_id=FEWQVHPIKN3Q6O5AZ6AGJBLUPZ34PBZFB5X3XNSKGPOGI4AIO7JA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.421089406 (291h21m26.972559867s) [noise=true , address=185.167.204.206:4072 , piece_id=GYCO53OWGOMOHK4FHC5GENN4LTQGSZKZKOI5V534RLV3LHLPKC6Q, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.426252887 (291h21m26.977723348s) [noise=true , address=185.167.204.206:4046 , piece_id=BZ57ZA35ITEOA6CAZJGNODK5MUTXEZWDV7E2OGBZOSMS4OQZYAEA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.431720689 (291h21m26.98319115s) [noise=true , address=185.167.204.205:4027 , piece_id=IVPONKBF2PIHZQAW3L4YA4HOMQV5BQML6SKKCJLIEGT5SZSQ2AOA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.438243951 (291h21m26.989714412s) [noise=true , address=185.167.204.205:4022 , piece_id=BES3YK6PYQV2AKOWSAGHF7DGGWGWSWPIJU2ICAQ5WBHKZ5VT5L6A, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.443364331 (291h21m26.994834792s) [noise=true , address=185.167.204.206:4056 , piece_id=EAMZIK6AVJL7IMFF5VOLW3SMCGYKSKB2VXJTNUNLJCVIT7EKXNMQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.445594728 (291h21m26.997065179s) [noise=true , address=185.167.204.205:4016 , piece_id=EHQS3546UXRXTL2MGFCSRTPDDPAYXEYLMISPP5AWY337MHX7HFLQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.449325822 (291h21m27.000796283s) [noise=true , address=185.167.204.205:4031 , piece_id=ZEI2YSIQC655R7WS43WADXGFT5RKT5LFSN6IZEQ6SHTDVOV7BZBQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.449761211 (291h21m27.001231672s) [noise=true , address=185.167.204.205:4018 , piece_id=7OSLPWJLKFTIJPZEAFMZOAH6JAAMX6HKROI25FODFYBPNJWSMLWA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.450419845 (291h21m27.001890306s) [noise=true , address=185.167.204.206:4062 , piece_id=UF3S6NAVGS2KQJPGA5WXP7YG4RQO44P445SNPCWXKXMONZAYUTNQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.453201252 (291h21m27.004671713s) [noise=true , address=185.167.204.205:4007 , piece_id=MEBGOPZ6MFMQY4UZARS2LKJKEMNN5YGSLKPJGTWSKS45CUNXSRGQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.661209626 (291h21m28.212680087s) [noise=true , address=89.149.201.147:4016 , piece_id=TGRWGGGBKDANC7YKROWPRMIYNAVFEVE3RXOSUOINFI7KTRS2IYUQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.693169182 (291h21m28.244639643s) [noise=true , address=89.149.201.147:4002 , piece_id=UIYJSCMY3LCLRHZUJB6JFJHHXKMYTVKERTZW7F7X44MWJETPY6XA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.721204497 (291h21m28.272674958s) [noise=true , address=89.149.201.147:4008 , piece_id=HCX7I4ICPEHDWTYDFIQMKZ5EE5CGPQLYAIHF7WZMTUUU6CNZPHTA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.764584711 (291h21m28.316055172s) [noise=true , address=89.149.201.147:4001 , piece_id=QUS6NB3NQBLJFB6YCK3EBDPKCTWEWGKAT3NOR24TVI65SQKHWDOQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.768284075 (291h21m28.319754536s) [noise=true , address=89.149.201.147:4011 , piece_id=ZATNMD5HOK35AWGJJ5V66VDJ4YGUVZQJ2OBERQL32ZV5GW2HBSMQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.789681961 (291h21m28.341152422s) [noise=true , address=89.149.201.147:4003 , piece_id=RUAYGN2RGF5JJTWFEGIHWH3BHEUDSS5DN2QJ6YLSHPF5BGUXD3RA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:37.298598682 (291h21m35.850069143s) [noise=true , address=81.171.14.70:4043 , piece_id=T3NF6JR3VJIGDMPNUBLMJ3AOTQP5IHOGJYORMCSS7ZMOENQETZ7A, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:37.497478356 (291h21m36.048948818s) [noise=true , address=81.171.14.70:4032 , piece_id=CFH2RTR3BLBUB5ZIZCFC6UHT3FHT3B2XSCHQMKXNGDJJGPHEMCTQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:40.406166933 (291h21m38.957637394s) [noise=true , address=81.171.14.70:4045 , piece_id=BRWQ7G3E547PJE2RLGSESIB3HMP5C4M5NSKAJNGWSFXUBEO6SYEQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:42.511041157 (291h21m41.062511608s) [noise=true , address=81.171.14.70:4029 , piece_id=DKJM5E7EY7LOSBSEBQOZRC6CI5ZPPE47HFOFQJROPCUMUV567CAA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:12.012682521 (291h22m10.564152982s) [noise=true , address=81.171.14.70:4042 , piece_id=XNREZEUZ5L5KZ3L7ZOL6IRK7IHU6DJN4RLEQZRFMDKDH2JWC24CQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:15.596046283 (291h22m14.147516744s) [noise=true , address=81.171.14.70:4033 , piece_id=JRKPD4EP6FZM5HISOIDJYZY7FAWWQCADAS47FGW2JM5A7RFJR2BQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:18.453059659 (291h22m17.00453012s) [noise=true , address=81.171.14.70:4046 , piece_id=OSONRSF6KJS7F2QJVCC3FPZK4AV7QPG5YSM47CW5LIW7EME44AWA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:23.442583344 (291h22m21.994053805s) [noise=true , address=81.171.14.70:4034 , piece_id=ITRLKYVNFSYBLABNGNFO3HRJNMX3MWB345FIMWSIPBNAVZPA47UA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:23.442621967 (291h22m21.994092428s) [seg_pos={0 0}, upload=256586]: Segment reached optimal threshold of 26 pieces. Nov 21 11:26:23.442836976 (291h22m21.994307437s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MUVGecysnTSFcJoHUL7QMWb3h6dtXgUd7FbA672fQCyEWhePr): protocol: expected piece hash; context canceled; EOF Nov 21 11:26:23.442845934 (291h22m21.994316395s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Long tail cancellation due to optimal threshold Nov 21 11:26:23.443227491 (291h22m21.994697942s) [noise=true , address=81.171.14.70:4027 , piece_id=4FDFCCWF3EIROQAFLOHGS743523OYGSXWQSPQEVD3AMQ6WKDQI7Q, seg_pos={0 0}, upload=256586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122cYvgTxSvC2aEXnBkRk2o8ARTBXj9gqPoa12vW8hogSN6kpA4): protocol: expected piece hash; context canceled; EOF Nov 21 11:26:23.443237770 (291h22m21.994708231s) [noise=true , address=81.171.14.70:4027 , piece_id=4FDFCCWF3EIROQAFLOHGS743523OYGSXWQSPQEVD3AMQ6WKDQI7Q, seg_pos={0 0}, upload=256586]: Long tail cancellation due to optimal threshold Nov 21 11:26:23.443274720 (291h22m21.994745181s) [seg_pos={0 0}, upload=256586]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Nov 21 11:26:23.443378929 (291h22m21.99484939s) [upload=256586]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 21 11:26:23.443387044 (291h22m21.994857495s) [upload=256586]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 21 11:26:23.685139626 (291h22m22.236610077s) [upload=256586]: Done uploading segments... Nov 21 11:26:23.685160676 (291h22m22.236631127s) [upload=256586]: Upload finished. err: <nil> Duration: 5.74201595s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 33 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX...........................] duration: 3.705145897s trace: https://jaeger.dev.storj.io/trace/6af15a3a67b655d5 FAIL (2025-11-21 11:58:40 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 3.705175944s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 2 out of 60 is failed (threshold=3) [........X.X.................................................] duration: 6.74982695s trace: https://jaeger.dev.storj.io/trace/a7f69ee106efb7f FAIL (2025-11-21 11:58:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== upload=256586]: Uploading piece... Nov 21 11:25:27.210346024 (291h21m25.761816485s) [noise=true , address=89.149.201.147:4008 , piece_id=HCX7I4ICPEHDWTYDFIQMKZ5EE5CGPQLYAIHF7WZMTUUU6CNZPHTA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.211425289 (291h21m25.76289575s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.213103795 (291h21m25.764574246s) [noise=true , address=185.167.204.205:4022 , piece_id=BES3YK6PYQV2AKOWSAGHF7DGGWGWSWPIJU2ICAQ5WBHKZ5VT5L6A, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.213408395 (291h21m25.764878856s) [noise=true , address=185.167.204.205:4027 , piece_id=IVPONKBF2PIHZQAW3L4YA4HOMQV5BQML6SKKCJLIEGT5SZSQ2AOA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.214501747 (291h21m25.765972208s) [noise=true , address=81.171.14.70:4042 , piece_id=XNREZEUZ5L5KZ3L7ZOL6IRK7IHU6DJN4RLEQZRFMDKDH2JWC24CQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.215013752 (291h21m25.766484213s) [noise=true , address=81.171.14.70:4034 , piece_id=ITRLKYVNFSYBLABNGNFO3HRJNMX3MWB345FIMWSIPBNAVZPA47UA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.215087572 (291h21m25.766558044s) [noise=true , address=185.167.204.205:4031 , piece_id=ZEI2YSIQC655R7WS43WADXGFT5RKT5LFSN6IZEQ6SHTDVOV7BZBQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.216432844 (291h21m25.767903305s) [noise=true , address=185.167.204.206:4056 , piece_id=EAMZIK6AVJL7IMFF5VOLW3SMCGYKSKB2VXJTNUNLJCVIT7EKXNMQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.216662281 (291h21m25.768132742s) [noise=true , address=89.149.201.147:4011 , piece_id=ZATNMD5HOK35AWGJJ5V66VDJ4YGUVZQJ2OBERQL32ZV5GW2HBSMQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.218008314 (291h21m25.769478775s) [noise=true , address=81.171.14.70:4032 , piece_id=CFH2RTR3BLBUB5ZIZCFC6UHT3FHT3B2XSCHQMKXNGDJJGPHEMCTQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:28.035150976 (291h21m26.586621437s) [upload=256586]: Next returned nil segment Nov 21 11:25:28.035169622 (291h21m26.586640083s) [upload=256586]: Waiting for error group managing segments... Nov 21 11:25:28.416646836 (291h21m26.968117317s) [noise=true , address=185.167.204.206:4064 , piece_id=SRFIWFQLRAGCNIKYLHMGT4QRUL3WH2QPLLIKAJ7DLFJEQUW5DPPA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.417338854 (291h21m26.968809325s) [noise=true , address=185.167.204.205:4033 , piece_id=FEWQVHPIKN3Q6O5AZ6AGJBLUPZ34PBZFB5X3XNSKGPOGI4AIO7JA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.421089406 (291h21m26.972559867s) [noise=true , address=185.167.204.206:4072 , piece_id=GYCO53OWGOMOHK4FHC5GENN4LTQGSZKZKOI5V534RLV3LHLPKC6Q, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.426252887 (291h21m26.977723348s) [noise=true , address=185.167.204.206:4046 , piece_id=BZ57ZA35ITEOA6CAZJGNODK5MUTXEZWDV7E2OGBZOSMS4OQZYAEA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.431720689 (291h21m26.98319115s) [noise=true , address=185.167.204.205:4027 , piece_id=IVPONKBF2PIHZQAW3L4YA4HOMQV5BQML6SKKCJLIEGT5SZSQ2AOA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.438243951 (291h21m26.989714412s) [noise=true , address=185.167.204.205:4022 , piece_id=BES3YK6PYQV2AKOWSAGHF7DGGWGWSWPIJU2ICAQ5WBHKZ5VT5L6A, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.443364331 (291h21m26.994834792s) [noise=true , address=185.167.204.206:4056 , piece_id=EAMZIK6AVJL7IMFF5VOLW3SMCGYKSKB2VXJTNUNLJCVIT7EKXNMQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.445594728 (291h21m26.997065179s) [noise=true , address=185.167.204.205:4016 , piece_id=EHQS3546UXRXTL2MGFCSRTPDDPAYXEYLMISPP5AWY337MHX7HFLQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.449325822 (291h21m27.000796283s) [noise=true , address=185.167.204.205:4031 , piece_id=ZEI2YSIQC655R7WS43WADXGFT5RKT5LFSN6IZEQ6SHTDVOV7BZBQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.449761211 (291h21m27.001231672s) [noise=true , address=185.167.204.205:4018 , piece_id=7OSLPWJLKFTIJPZEAFMZOAH6JAAMX6HKROI25FODFYBPNJWSMLWA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.450419845 (291h21m27.001890306s) [noise=true , address=185.167.204.206:4062 , piece_id=UF3S6NAVGS2KQJPGA5WXP7YG4RQO44P445SNPCWXKXMONZAYUTNQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.453201252 (291h21m27.004671713s) [noise=true , address=185.167.204.205:4007 , piece_id=MEBGOPZ6MFMQY4UZARS2LKJKEMNN5YGSLKPJGTWSKS45CUNXSRGQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.661209626 (291h21m28.212680087s) [noise=true , address=89.149.201.147:4016 , piece_id=TGRWGGGBKDANC7YKROWPRMIYNAVFEVE3RXOSUOINFI7KTRS2IYUQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.693169182 (291h21m28.244639643s) [noise=true , address=89.149.201.147:4002 , piece_id=UIYJSCMY3LCLRHZUJB6JFJHHXKMYTVKERTZW7F7X44MWJETPY6XA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.721204497 (291h21m28.272674958s) [noise=true , address=89.149.201.147:4008 , piece_id=HCX7I4ICPEHDWTYDFIQMKZ5EE5CGPQLYAIHF7WZMTUUU6CNZPHTA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.764584711 (291h21m28.316055172s) [noise=true , address=89.149.201.147:4001 , piece_id=QUS6NB3NQBLJFB6YCK3EBDPKCTWEWGKAT3NOR24TVI65SQKHWDOQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.768284075 (291h21m28.319754536s) [noise=true , address=89.149.201.147:4011 , piece_id=ZATNMD5HOK35AWGJJ5V66VDJ4YGUVZQJ2OBERQL32ZV5GW2HBSMQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.789681961 (291h21m28.341152422s) [noise=true , address=89.149.201.147:4003 , piece_id=RUAYGN2RGF5JJTWFEGIHWH3BHEUDSS5DN2QJ6YLSHPF5BGUXD3RA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:37.298598682 (291h21m35.850069143s) [noise=true , address=81.171.14.70:4043 , piece_id=T3NF6JR3VJIGDMPNUBLMJ3AOTQP5IHOGJYORMCSS7ZMOENQETZ7A, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:37.497478356 (291h21m36.048948818s) [noise=true , address=81.171.14.70:4032 , piece_id=CFH2RTR3BLBUB5ZIZCFC6UHT3FHT3B2XSCHQMKXNGDJJGPHEMCTQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:40.406166933 (291h21m38.957637394s) [noise=true , address=81.171.14.70:4045 , piece_id=BRWQ7G3E547PJE2RLGSESIB3HMP5C4M5NSKAJNGWSFXUBEO6SYEQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:42.511041157 (291h21m41.062511608s) [noise=true , address=81.171.14.70:4029 , piece_id=DKJM5E7EY7LOSBSEBQOZRC6CI5ZPPE47HFOFQJROPCUMUV567CAA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:12.012682521 (291h22m10.564152982s) [noise=true , address=81.171.14.70:4042 , piece_id=XNREZEUZ5L5KZ3L7ZOL6IRK7IHU6DJN4RLEQZRFMDKDH2JWC24CQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:15.596046283 (291h22m14.147516744s) [noise=true , address=81.171.14.70:4033 , piece_id=JRKPD4EP6FZM5HISOIDJYZY7FAWWQCADAS47FGW2JM5A7RFJR2BQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:18.453059659 (291h22m17.00453012s) [noise=true , address=81.171.14.70:4046 , piece_id=OSONRSF6KJS7F2QJVCC3FPZK4AV7QPG5YSM47CW5LIW7EME44AWA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:23.442583344 (291h22m21.994053805s) [noise=true , address=81.171.14.70:4034 , piece_id=ITRLKYVNFSYBLABNGNFO3HRJNMX3MWB345FIMWSIPBNAVZPA47UA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:23.442621967 (291h22m21.994092428s) [seg_pos={0 0}, upload=256586]: Segment reached optimal threshold of 26 pieces. Nov 21 11:26:23.442836976 (291h22m21.994307437s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MUVGecysnTSFcJoHUL7QMWb3h6dtXgUd7FbA672fQCyEWhePr): protocol: expected piece hash; context canceled; EOF Nov 21 11:26:23.442845934 (291h22m21.994316395s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Long tail cancellation due to optimal threshold Nov 21 11:26:23.443227491 (291h22m21.994697942s) [noise=true , address=81.171.14.70:4027 , piece_id=4FDFCCWF3EIROQAFLOHGS743523OYGSXWQSPQEVD3AMQ6WKDQI7Q, seg_pos={0 0}, upload=256586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122cYvgTxSvC2aEXnBkRk2o8ARTBXj9gqPoa12vW8hogSN6kpA4): protocol: expected piece hash; context canceled; EOF Nov 21 11:26:23.443237770 (291h22m21.994708231s) [noise=true , address=81.171.14.70:4027 , piece_id=4FDFCCWF3EIROQAFLOHGS743523OYGSXWQSPQEVD3AMQ6WKDQI7Q, seg_pos={0 0}, upload=256586]: Long tail cancellation due to optimal threshold Nov 21 11:26:23.443274720 (291h22m21.994745181s) [seg_pos={0 0}, upload=256586]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Nov 21 11:26:23.443378929 (291h22m21.99484939s) [upload=256586]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 21 11:26:23.443387044 (291h22m21.994857495s) [upload=256586]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 21 11:26:23.685139626 (291h22m22.236610077s) [upload=256586]: Done uploading segments... Nov 21 11:26:23.685160676 (291h22m22.236631127s) [upload=256586]: Upload finished. err: <nil> Duration: 6.749855766s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 34 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX..........................] duration: 3.583500704s trace: https://jaeger.dev.storj.io/trace/6af15a3a67b655d5 FAIL (2025-11-21 11:58:09 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 3.583528697s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 2 out of 60 is failed (threshold=3) [.........X.X................................................] duration: 5.755210397s trace: https://jaeger.dev.storj.io/trace/a7f69ee106efb7f FAIL (2025-11-21 11:57:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== upload=256586]: Uploading piece... Nov 21 11:25:27.210346024 (291h21m25.761816485s) [noise=true , address=89.149.201.147:4008 , piece_id=HCX7I4ICPEHDWTYDFIQMKZ5EE5CGPQLYAIHF7WZMTUUU6CNZPHTA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.211425289 (291h21m25.76289575s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.213103795 (291h21m25.764574246s) [noise=true , address=185.167.204.205:4022 , piece_id=BES3YK6PYQV2AKOWSAGHF7DGGWGWSWPIJU2ICAQ5WBHKZ5VT5L6A, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.213408395 (291h21m25.764878856s) [noise=true , address=185.167.204.205:4027 , piece_id=IVPONKBF2PIHZQAW3L4YA4HOMQV5BQML6SKKCJLIEGT5SZSQ2AOA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.214501747 (291h21m25.765972208s) [noise=true , address=81.171.14.70:4042 , piece_id=XNREZEUZ5L5KZ3L7ZOL6IRK7IHU6DJN4RLEQZRFMDKDH2JWC24CQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.215013752 (291h21m25.766484213s) [noise=true , address=81.171.14.70:4034 , piece_id=ITRLKYVNFSYBLABNGNFO3HRJNMX3MWB345FIMWSIPBNAVZPA47UA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.215087572 (291h21m25.766558044s) [noise=true , address=185.167.204.205:4031 , piece_id=ZEI2YSIQC655R7WS43WADXGFT5RKT5LFSN6IZEQ6SHTDVOV7BZBQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.216432844 (291h21m25.767903305s) [noise=true , address=185.167.204.206:4056 , piece_id=EAMZIK6AVJL7IMFF5VOLW3SMCGYKSKB2VXJTNUNLJCVIT7EKXNMQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.216662281 (291h21m25.768132742s) [noise=true , address=89.149.201.147:4011 , piece_id=ZATNMD5HOK35AWGJJ5V66VDJ4YGUVZQJ2OBERQL32ZV5GW2HBSMQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.218008314 (291h21m25.769478775s) [noise=true , address=81.171.14.70:4032 , piece_id=CFH2RTR3BLBUB5ZIZCFC6UHT3FHT3B2XSCHQMKXNGDJJGPHEMCTQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:28.035150976 (291h21m26.586621437s) [upload=256586]: Next returned nil segment Nov 21 11:25:28.035169622 (291h21m26.586640083s) [upload=256586]: Waiting for error group managing segments... Nov 21 11:25:28.416646836 (291h21m26.968117317s) [noise=true , address=185.167.204.206:4064 , piece_id=SRFIWFQLRAGCNIKYLHMGT4QRUL3WH2QPLLIKAJ7DLFJEQUW5DPPA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.417338854 (291h21m26.968809325s) [noise=true , address=185.167.204.205:4033 , piece_id=FEWQVHPIKN3Q6O5AZ6AGJBLUPZ34PBZFB5X3XNSKGPOGI4AIO7JA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.421089406 (291h21m26.972559867s) [noise=true , address=185.167.204.206:4072 , piece_id=GYCO53OWGOMOHK4FHC5GENN4LTQGSZKZKOI5V534RLV3LHLPKC6Q, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.426252887 (291h21m26.977723348s) [noise=true , address=185.167.204.206:4046 , piece_id=BZ57ZA35ITEOA6CAZJGNODK5MUTXEZWDV7E2OGBZOSMS4OQZYAEA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.431720689 (291h21m26.98319115s) [noise=true , address=185.167.204.205:4027 , piece_id=IVPONKBF2PIHZQAW3L4YA4HOMQV5BQML6SKKCJLIEGT5SZSQ2AOA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.438243951 (291h21m26.989714412s) [noise=true , address=185.167.204.205:4022 , piece_id=BES3YK6PYQV2AKOWSAGHF7DGGWGWSWPIJU2ICAQ5WBHKZ5VT5L6A, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.443364331 (291h21m26.994834792s) [noise=true , address=185.167.204.206:4056 , piece_id=EAMZIK6AVJL7IMFF5VOLW3SMCGYKSKB2VXJTNUNLJCVIT7EKXNMQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.445594728 (291h21m26.997065179s) [noise=true , address=185.167.204.205:4016 , piece_id=EHQS3546UXRXTL2MGFCSRTPDDPAYXEYLMISPP5AWY337MHX7HFLQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.449325822 (291h21m27.000796283s) [noise=true , address=185.167.204.205:4031 , piece_id=ZEI2YSIQC655R7WS43WADXGFT5RKT5LFSN6IZEQ6SHTDVOV7BZBQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.449761211 (291h21m27.001231672s) [noise=true , address=185.167.204.205:4018 , piece_id=7OSLPWJLKFTIJPZEAFMZOAH6JAAMX6HKROI25FODFYBPNJWSMLWA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.450419845 (291h21m27.001890306s) [noise=true , address=185.167.204.206:4062 , piece_id=UF3S6NAVGS2KQJPGA5WXP7YG4RQO44P445SNPCWXKXMONZAYUTNQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.453201252 (291h21m27.004671713s) [noise=true , address=185.167.204.205:4007 , piece_id=MEBGOPZ6MFMQY4UZARS2LKJKEMNN5YGSLKPJGTWSKS45CUNXSRGQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.661209626 (291h21m28.212680087s) [noise=true , address=89.149.201.147:4016 , piece_id=TGRWGGGBKDANC7YKROWPRMIYNAVFEVE3RXOSUOINFI7KTRS2IYUQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.693169182 (291h21m28.244639643s) [noise=true , address=89.149.201.147:4002 , piece_id=UIYJSCMY3LCLRHZUJB6JFJHHXKMYTVKERTZW7F7X44MWJETPY6XA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.721204497 (291h21m28.272674958s) [noise=true , address=89.149.201.147:4008 , piece_id=HCX7I4ICPEHDWTYDFIQMKZ5EE5CGPQLYAIHF7WZMTUUU6CNZPHTA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.764584711 (291h21m28.316055172s) [noise=true , address=89.149.201.147:4001 , piece_id=QUS6NB3NQBLJFB6YCK3EBDPKCTWEWGKAT3NOR24TVI65SQKHWDOQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.768284075 (291h21m28.319754536s) [noise=true , address=89.149.201.147:4011 , piece_id=ZATNMD5HOK35AWGJJ5V66VDJ4YGUVZQJ2OBERQL32ZV5GW2HBSMQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.789681961 (291h21m28.341152422s) [noise=true , address=89.149.201.147:4003 , piece_id=RUAYGN2RGF5JJTWFEGIHWH3BHEUDSS5DN2QJ6YLSHPF5BGUXD3RA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:37.298598682 (291h21m35.850069143s) [noise=true , address=81.171.14.70:4043 , piece_id=T3NF6JR3VJIGDMPNUBLMJ3AOTQP5IHOGJYORMCSS7ZMOENQETZ7A, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:37.497478356 (291h21m36.048948818s) [noise=true , address=81.171.14.70:4032 , piece_id=CFH2RTR3BLBUB5ZIZCFC6UHT3FHT3B2XSCHQMKXNGDJJGPHEMCTQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:40.406166933 (291h21m38.957637394s) [noise=true , address=81.171.14.70:4045 , piece_id=BRWQ7G3E547PJE2RLGSESIB3HMP5C4M5NSKAJNGWSFXUBEO6SYEQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:42.511041157 (291h21m41.062511608s) [noise=true , address=81.171.14.70:4029 , piece_id=DKJM5E7EY7LOSBSEBQOZRC6CI5ZPPE47HFOFQJROPCUMUV567CAA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:12.012682521 (291h22m10.564152982s) [noise=true , address=81.171.14.70:4042 , piece_id=XNREZEUZ5L5KZ3L7ZOL6IRK7IHU6DJN4RLEQZRFMDKDH2JWC24CQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:15.596046283 (291h22m14.147516744s) [noise=true , address=81.171.14.70:4033 , piece_id=JRKPD4EP6FZM5HISOIDJYZY7FAWWQCADAS47FGW2JM5A7RFJR2BQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:18.453059659 (291h22m17.00453012s) [noise=true , address=81.171.14.70:4046 , piece_id=OSONRSF6KJS7F2QJVCC3FPZK4AV7QPG5YSM47CW5LIW7EME44AWA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:23.442583344 (291h22m21.994053805s) [noise=true , address=81.171.14.70:4034 , piece_id=ITRLKYVNFSYBLABNGNFO3HRJNMX3MWB345FIMWSIPBNAVZPA47UA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:23.442621967 (291h22m21.994092428s) [seg_pos={0 0}, upload=256586]: Segment reached optimal threshold of 26 pieces. Nov 21 11:26:23.442836976 (291h22m21.994307437s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MUVGecysnTSFcJoHUL7QMWb3h6dtXgUd7FbA672fQCyEWhePr): protocol: expected piece hash; context canceled; EOF Nov 21 11:26:23.442845934 (291h22m21.994316395s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Long tail cancellation due to optimal threshold Nov 21 11:26:23.443227491 (291h22m21.994697942s) [noise=true , address=81.171.14.70:4027 , piece_id=4FDFCCWF3EIROQAFLOHGS743523OYGSXWQSPQEVD3AMQ6WKDQI7Q, seg_pos={0 0}, upload=256586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122cYvgTxSvC2aEXnBkRk2o8ARTBXj9gqPoa12vW8hogSN6kpA4): protocol: expected piece hash; context canceled; EOF Nov 21 11:26:23.443237770 (291h22m21.994708231s) [noise=true , address=81.171.14.70:4027 , piece_id=4FDFCCWF3EIROQAFLOHGS743523OYGSXWQSPQEVD3AMQ6WKDQI7Q, seg_pos={0 0}, upload=256586]: Long tail cancellation due to optimal threshold Nov 21 11:26:23.443274720 (291h22m21.994745181s) [seg_pos={0 0}, upload=256586]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Nov 21 11:26:23.443378929 (291h22m21.99484939s) [upload=256586]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 21 11:26:23.443387044 (291h22m21.994857495s) [upload=256586]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 21 11:26:23.685139626 (291h22m22.236610077s) [upload=256586]: Done uploading segments... Nov 21 11:26:23.685160676 (291h22m22.236631127s) [upload=256586]: Upload finished. err: <nil> Duration: 5.755239783s core uplink eu-select-1 eu1-placement-15

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)