Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [..................X..........X.......X.X.......X.X.XXX.X....] duration: 5.900086742s trace: https://jaeger.dev.storj.io/trace/610a5a278586d69b FAIL (2025-12-26 06:40:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: UR4BEWMRCCCM5JBPWIIIYJE24IVP2TZP35NLPPGQRTUBWZ4NLXKQ) context deadline exceeded; piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: NSIQRTRZ6IYC7U6234YFAEN7ZYW6NK63G75IK6YTXJECYZLAC25A) context deadline exceeded ======LOGS====== Dec 26 06:36:25.402042438 (495h7m25.273125473s) [upload=434577]: Starting upload Dec 26 06:36:25.402099898 (495h7m25.273182943s) [upload=434577]: Done starting upload Dec 26 06:36:25.402171925 (495h7m25.27325496s) [upload=434577]: Uploading segments... Dec 26 06:36:25.402316690 (495h7m25.273399736s) [upload=434577]: Got next segment. Inline: false Dec 26 06:36:25.402354793 (495h7m25.273437838s) [upload=434577]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 06:36:25.402360153 (495h7m25.273443198s) [upload=434577]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 06:36:25.658100426 (495h7m25.529183461s) [seg_pos={0 0}, upload=434577]: Begin upload segment... Dec 26 06:36:25.658147776 (495h7m25.529230821s) [seg_pos={0 0}, upload=434577]: Done begin upload segment. Dec 26 06:36:25.658177322 (495h7m25.529260367s) [noise=true , address=145.79.227.35:4482 , piece_id=OFKG22Z35IGBPSS2CGY56DCC7K5QPNUH6MKEVUICGI7GEYFQFALA, seg_pos={0 0}, upload=434577]: Uploading piece... Dec 26 06:36:25.658185919 (495h7m25.529268954s) [noise=true , address=145.79.227.30:4762 , piece_id=32BD4NBDJFFQVNSZV2BDNG7YOBEDUHMJVPKEI7QZ656SGGMVM5DQ, seg_pos={0 0}, upload=434577]: Uploading piece... Dec 26 06:36:25.658280429 (495h7m25.529363464s) [noise=true , address=45.91.116.6:4976 , piece_id=ARM3RYXBKNWUT4RDLPV2TPXW4KSYDJCGFDHSHV2AFD22X2JMUK5Q, seg_pos={0 0}, upload=434577]: Uploading piece... Dec 26 06:36:25.658817742 (495h7m25.529900777s) [noise=true , address=45.91.116.12:4226 , piece_id=MC37EZNWICD2545T6PVZXUNJ5OO4AZMTJDLWDNNBRUTGMZBJXKOA, seg_pos={0 0}, upload=434577]: Uploading piece... Dec 26 06:36:25.659206312 (495h7m25.530289357s) [noise=true , address=145.79.227.44:4193 , piece_id=PA3JZO3SHIJKINT75USL27VISAB3YNBQJNR326RL3LHO3MBKFCTQ, seg_pos={0 0}, upload=434577]: Uploading piece... Dec 26 06:36:25.659551049 (495h7m25.530634094s) [noise=true , address=45.91.116.7:4010 , piece_id=UR4BEWMRCCCM5JBPWIIIYJE24IVP2TZP35NLPPGQRTUBWZ4NLXKQ, seg_pos={0 0}, upload=434577]: Uploading piece... Dec 26 06:36:25.659953835 (495h7m25.53103688s) [noise=true , address=77.90.31.19:4842 , piece_id=BW4BT5IWAVXZYTUYN7DXTNAS3I3NSSGMMWAQKEH64ZF34PDJ7MIA, seg_pos={0 0}, upload=434577]: Uploading piece... Dec 26 06:36:25.660275498 (495h7m25.531358543s) [noise=true , address=145.79.227.29:4942 , piece_id=BQJYKRMRM45P4HIBSMHFXPUL47EDLS6JKEGB42FJ5VB6E46OJHWA, seg_pos={0 0}, upload=434577]: Uploading piece... Dec 26 06:36:25.660688724 (495h7m25.531771769s) [noise=true , address=45.91.116.9:4264 , piece_id=VYLOL2PY6DJ3HEWRGP7QHKQKW7LA36D3WOLPDLZIEU56FKNCMRFA, seg_pos={0 0}, upload=434577]: Uploading piece... Dec 26 06:36:25.661094387 (495h7m25.532177422s) [noise=true , address=145.79.227.25:4617 , piece_id=T3LHNAV74UNQWPWZU3VM774UW6LQ4RADRTC6Y3NSXEGDGHCACIJQ, seg_pos={0 0}, upload=434577]: Uploading piece... Dec 26 06:36:25.662169645 (495h7m25.53325268s) [noise=true , address=45.91.116.4:4940 , piece_id=PX3K6B2D2FNPO4AK2TY4YMCN3C6CFS3OQMI26ZGI6564Q6OUYPIA, seg_pos={0 0}, upload=434577]: Uploading piece... Dec 26 06:36:25.663392453 (495h7m25.534475488s) [noise=true , address=45.91.116.13:4153 , piece_id=NSIQRTRZ6IYC7U6234YFAEN7ZYW6NK63G75IK6YTXJECYZLAC25A, seg_pos={0 0}, upload=434577]: Uploading piece... Dec 26 06:36:25.664717937 (495h7m25.535800972s) [noise=true , address=145.79.227.2:4114 , piece_id=6L35OSYBZHUUJGCYHXRDCIS22GMUNY3WBOSRUWZCHESCO7I7QG4A, seg_pos={0 0}, upload=434577]: Uploading piece... Dec 26 06:36:25.665768417 (495h7m25.536851462s) [noise=true , address=45.91.116.14:4189 , piece_id=7PJYJ2M7J7GBGKBEPMTDV5LBO6ITO53XDFG4BXKRM5ND2ZTQNQZQ, seg_pos={0 0}, upload=434577]: Uploading piece... Dec 26 06:36:25.667267361 (495h7m25.538350426s) [noise=true , address=145.79.227.52:4446 , piece_id=L6QJPT7JLR5O7XO6UGHLVU2LAN2OFSZHLCPO2TBDG2PVZ6FIJGZA, seg_pos={0 0}, upload=434577]: Uploading piece... Dec 26 06:36:25.668498907 (495h7m25.539581952s) [noise=true , address=77.90.31.18:4805 , piece_id=LEJ5ILCYWMFFMRXDP3OGEK5WKUMHFDRKZY4KYIPO5DH64HYMOJXA, seg_pos={0 0}, upload=434577]: Uploading piece... Dec 26 06:36:26.797649813 (495h7m26.668732868s) [noise=true , address=77.90.31.18:4805 , piece_id=LEJ5ILCYWMFFMRXDP3OGEK5WKUMHFDRKZY4KYIPO5DH64HYMOJXA, seg_pos={0 0}, upload=434577]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: storage node overloaded, request limit: 2000 Dec 26 06:36:27.368822683 (495h7m27.239905728s) [upload=434577]: Next returned nil segment Dec 26 06:36:27.368839776 (495h7m27.239922831s) [upload=434577]: Waiting for error group managing segments... Dec 26 06:36:27.678521873 (495h7m27.549604928s) [noise=true , address=145.79.227.35:4482 , piece_id=OFKG22Z35IGBPSS2CGY56DCC7K5QPNUH6MKEVUICGI7GEYFQFALA, seg_pos={0 0}, upload=434577]: Done uploading piece. err: <nil> Dec 26 06:36:27.678952202 (495h7m27.550035237s) [noise=true , address=145.79.227.52:4446 , piece_id=L6QJPT7JLR5O7XO6UGHLVU2LAN2OFSZHLCPO2TBDG2PVZ6FIJGZA, seg_pos={0 0}, upload=434577]: Done uploading piece. err: <nil> Dec 26 06:36:27.679637117 (495h7m27.550720162s) [noise=true , address=145.79.227.2:4114 , piece_id=6L35OSYBZHUUJGCYHXRDCIS22GMUNY3WBOSRUWZCHESCO7I7QG4A, seg_pos={0 0}, upload=434577]: Done uploading piece. err: <nil> Dec 26 06:36:27.682241977 (495h7m27.553325012s) [noise=true , address=145.79.227.29:4942 , piece_id=BQJYKRMRM45P4HIBSMHFXPUL47EDLS6JKEGB42FJ5VB6E46OJHWA, seg_pos={0 0}, upload=434577]: Done uploading piece. err: <nil> Dec 26 06:36:27.690925572 (495h7m27.562008617s) [noise=true , address=145.79.227.25:4617 , piece_id=T3LHNAV74UNQWPWZU3VM774UW6LQ4RADRTC6Y3NSXEGDGHCACIJQ, seg_pos={0 0}, upload=434577]: Done uploading piece. err: <nil> Dec 26 06:36:27.909884327 (495h7m27.780967362s) [noise=true , address=45.91.116.12:4226 , piece_id=MC37EZNWICD2545T6PVZXUNJ5OO4AZMTJDLWDNNBRUTGMZBJXKOA, seg_pos={0 0}, upload=434577]: Done uploading piece. err: <nil> Dec 26 06:36:27.946087294 (495h7m27.817170339s) [noise=true , address=45.91.116.6:4976 , piece_id=ARM3RYXBKNWUT4RDLPV2TPXW4KSYDJCGFDHSHV2AFD22X2JMUK5Q, seg_pos={0 0}, upload=434577]: Done uploading piece. err: <nil> Dec 26 06:36:28.205149221 (495h7m28.076232256s) [noise=true , address=77.90.31.19:4842 , piece_id=BW4BT5IWAVXZYTUYN7DXTNAS3I3NSSGMMWAQKEH64ZF34PDJ7MIA, seg_pos={0 0}, upload=434577]: Done uploading piece. err: <nil> Dec 26 06:36:28.235077159 (495h7m28.106160204s) [noise=true , address=45.91.116.4:4940 , piece_id=PX3K6B2D2FNPO4AK2TY4YMCN3C6CFS3OQMI26ZGI6564Q6OUYPIA, seg_pos={0 0}, upload=434577]: Done uploading piece. err: <nil> Dec 26 06:36:28.636212339 (495h7m28.507295384s) [noise=true , address=45.91.116.14:4189 , piece_id=7PJYJ2M7J7GBGKBEPMTDV5LBO6ITO53XDFG4BXKRM5ND2ZTQNQZQ, seg_pos={0 0}, upload=434577]: Done uploading piece. err: <nil> Dec 26 06:36:28.637196683 (495h7m28.508279718s) [noise=true , address=45.91.116.13:4153 , piece_id=NSIQRTRZ6IYC7U6234YFAEN7ZYW6NK63G75IK6YTXJECYZLAC25A, seg_pos={0 0}, upload=434577]: Done uploading piece. err: <nil> Dec 26 06:36:28.644046045 (495h7m28.51512909s) [noise=true , address=45.91.116.7:4010 , piece_id=UR4BEWMRCCCM5JBPWIIIYJE24IVP2TZP35NLPPGQRTUBWZ4NLXKQ, seg_pos={0 0}, upload=434577]: Done uploading piece. err: <nil> Dec 26 06:36:28.918446120 (495h7m28.789529165s) [noise=true , address=145.79.227.44:4193 , piece_id=PA3JZO3SHIJKINT75USL27VISAB3YNBQJNR326RL3LHO3MBKFCTQ, seg_pos={0 0}, upload=434577]: Done uploading piece. err: <nil> Dec 26 06:36:28.918531934 (495h7m28.789614969s) [seg_pos={0 0}, upload=434577]: Segment reached optimal threshold of 13 pieces. Dec 26 06:36:28.918732516 (495h7m28.789815561s) [noise=true , address=45.91.116.9:4264 , piece_id=VYLOL2PY6DJ3HEWRGP7QHKQKW7LA36D3WOLPDLZIEU56FKNCMRFA, seg_pos={0 0}, upload=434577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 26 06:36:28.918738607 (495h7m28.789821642s) [noise=true , address=45.91.116.9:4264 , piece_id=VYLOL2PY6DJ3HEWRGP7QHKQKW7LA36D3WOLPDLZIEU56FKNCMRFA, seg_pos={0 0}, upload=434577]: Long tail cancellation due to optimal threshold Dec 26 06:36:28.918945321 (495h7m28.790028366s) [noise=true , address=145.79.227.30:4762 , piece_id=32BD4NBDJFFQVNSZV2BDNG7YOBEDUHMJVPKEI7QZ656SGGMVM5DQ, seg_pos={0 0}, upload=434577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 26 06:36:28.918965709 (495h7m28.790048744s) [noise=true , address=145.79.227.30:4762 , piece_id=32BD4NBDJFFQVNSZV2BDNG7YOBEDUHMJVPKEI7QZ656SGGMVM5DQ, seg_pos={0 0}, upload=434577]: Long tail cancellation due to optimal threshold Dec 26 06:36:28.918991459 (495h7m28.790074494s) [seg_pos={0 0}, upload=434577]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 06:36:28.919089966 (495h7m28.790173001s) [upload=434577]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 06:36:28.919097781 (495h7m28.790180816s) [upload=434577]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 06:36:29.222421732 (495h7m29.093504767s) [upload=434577]: Done uploading segments... Dec 26 06:36:29.222439555 (495h7m29.09352259s) [upload=434577]: Upload finished. err: <nil> Duration: 5.900116949s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......................X.....................................] duration: 3.532678693s trace: https://jaeger.dev.storj.io/trace/630094597d401477 FAIL (2025-12-26 06:40:04 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.532709632s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [...................X..........X.......X.X.......X.X.XXX.X...] duration: 27.44114084s trace: https://jaeger.dev.storj.io/trace/610a5a278586d69b FAIL (2025-12-26 06:39:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: UR4BEWMRCCCM5JBPWIIIYJE24IVP2TZP35NLPPGQRTUBWZ4NLXKQ) context deadline exceeded; piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: NSIQRTRZ6IYC7U6234YFAEN7ZYW6NK63G75IK6YTXJECYZLAC25A) context deadline exceeded ======LOGS====== Dec 26 06:36:25.402042438 (495h7m25.273125473s) [upload=434577]: Starting upload Dec 26 06:36:25.402099898 (495h7m25.273182943s) [upload=434577]: Done starting upload Dec 26 06:36:25.402171925 (495h7m25.27325496s) [upload=434577]: Uploading segments... Dec 26 06:36:25.402316690 (495h7m25.273399736s) [upload=434577]: Got next segment. Inline: false Dec 26 06:36:25.402354793 (495h7m25.273437838s) [upload=434577]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 06:36:25.402360153 (495h7m25.273443198s) [upload=434577]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 06:36:25.658100426 (495h7m25.529183461s) [seg_pos={0 0}, upload=434577]: Begin upload segment... Dec 26 06:36:25.658147776 (495h7m25.529230821s) [seg_pos={0 0}, upload=434577]: Done begin upload segment. Dec 26 06:36:25.658177322 (495h7m25.529260367s) [noise=true , address=145.79.227.35:4482 , piece_id=OFKG22Z35IGBPSS2CGY56DCC7K5QPNUH6MKEVUICGI7GEYFQFALA, seg_pos={0 0}, upload=434577]: Uploading piece... Dec 26 06:36:25.658185919 (495h7m25.529268954s) [noise=true , address=145.79.227.30:4762 , piece_id=32BD4NBDJFFQVNSZV2BDNG7YOBEDUHMJVPKEI7QZ656SGGMVM5DQ, seg_pos={0 0}, upload=434577]: Uploading piece... Dec 26 06:36:25.658280429 (495h7m25.529363464s) [noise=true , address=45.91.116.6:4976 , piece_id=ARM3RYXBKNWUT4RDLPV2TPXW4KSYDJCGFDHSHV2AFD22X2JMUK5Q, seg_pos={0 0}, upload=434577]: Uploading piece... Dec 26 06:36:25.658817742 (495h7m25.529900777s) [noise=true , address=45.91.116.12:4226 , piece_id=MC37EZNWICD2545T6PVZXUNJ5OO4AZMTJDLWDNNBRUTGMZBJXKOA, seg_pos={0 0}, upload=434577]: Uploading piece... Dec 26 06:36:25.659206312 (495h7m25.530289357s) [noise=true , address=145.79.227.44:4193 , piece_id=PA3JZO3SHIJKINT75USL27VISAB3YNBQJNR326RL3LHO3MBKFCTQ, seg_pos={0 0}, upload=434577]: Uploading piece... Dec 26 06:36:25.659551049 (495h7m25.530634094s) [noise=true , address=45.91.116.7:4010 , piece_id=UR4BEWMRCCCM5JBPWIIIYJE24IVP2TZP35NLPPGQRTUBWZ4NLXKQ, seg_pos={0 0}, upload=434577]: Uploading piece... Dec 26 06:36:25.659953835 (495h7m25.53103688s) [noise=true , address=77.90.31.19:4842 , piece_id=BW4BT5IWAVXZYTUYN7DXTNAS3I3NSSGMMWAQKEH64ZF34PDJ7MIA, seg_pos={0 0}, upload=434577]: Uploading piece... Dec 26 06:36:25.660275498 (495h7m25.531358543s) [noise=true , address=145.79.227.29:4942 , piece_id=BQJYKRMRM45P4HIBSMHFXPUL47EDLS6JKEGB42FJ5VB6E46OJHWA, seg_pos={0 0}, upload=434577]: Uploading piece... Dec 26 06:36:25.660688724 (495h7m25.531771769s) [noise=true , address=45.91.116.9:4264 , piece_id=VYLOL2PY6DJ3HEWRGP7QHKQKW7LA36D3WOLPDLZIEU56FKNCMRFA, seg_pos={0 0}, upload=434577]: Uploading piece... Dec 26 06:36:25.661094387 (495h7m25.532177422s) [noise=true , address=145.79.227.25:4617 , piece_id=T3LHNAV74UNQWPWZU3VM774UW6LQ4RADRTC6Y3NSXEGDGHCACIJQ, seg_pos={0 0}, upload=434577]: Uploading piece... Dec 26 06:36:25.662169645 (495h7m25.53325268s) [noise=true , address=45.91.116.4:4940 , piece_id=PX3K6B2D2FNPO4AK2TY4YMCN3C6CFS3OQMI26ZGI6564Q6OUYPIA, seg_pos={0 0}, upload=434577]: Uploading piece... Dec 26 06:36:25.663392453 (495h7m25.534475488s) [noise=true , address=45.91.116.13:4153 , piece_id=NSIQRTRZ6IYC7U6234YFAEN7ZYW6NK63G75IK6YTXJECYZLAC25A, seg_pos={0 0}, upload=434577]: Uploading piece... Dec 26 06:36:25.664717937 (495h7m25.535800972s) [noise=true , address=145.79.227.2:4114 , piece_id=6L35OSYBZHUUJGCYHXRDCIS22GMUNY3WBOSRUWZCHESCO7I7QG4A, seg_pos={0 0}, upload=434577]: Uploading piece... Dec 26 06:36:25.665768417 (495h7m25.536851462s) [noise=true , address=45.91.116.14:4189 , piece_id=7PJYJ2M7J7GBGKBEPMTDV5LBO6ITO53XDFG4BXKRM5ND2ZTQNQZQ, seg_pos={0 0}, upload=434577]: Uploading piece... Dec 26 06:36:25.667267361 (495h7m25.538350426s) [noise=true , address=145.79.227.52:4446 , piece_id=L6QJPT7JLR5O7XO6UGHLVU2LAN2OFSZHLCPO2TBDG2PVZ6FIJGZA, seg_pos={0 0}, upload=434577]: Uploading piece... Dec 26 06:36:25.668498907 (495h7m25.539581952s) [noise=true , address=77.90.31.18:4805 , piece_id=LEJ5ILCYWMFFMRXDP3OGEK5WKUMHFDRKZY4KYIPO5DH64HYMOJXA, seg_pos={0 0}, upload=434577]: Uploading piece... Dec 26 06:36:26.797649813 (495h7m26.668732868s) [noise=true , address=77.90.31.18:4805 , piece_id=LEJ5ILCYWMFFMRXDP3OGEK5WKUMHFDRKZY4KYIPO5DH64HYMOJXA, seg_pos={0 0}, upload=434577]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: storage node overloaded, request limit: 2000 Dec 26 06:36:27.368822683 (495h7m27.239905728s) [upload=434577]: Next returned nil segment Dec 26 06:36:27.368839776 (495h7m27.239922831s) [upload=434577]: Waiting for error group managing segments... Dec 26 06:36:27.678521873 (495h7m27.549604928s) [noise=true , address=145.79.227.35:4482 , piece_id=OFKG22Z35IGBPSS2CGY56DCC7K5QPNUH6MKEVUICGI7GEYFQFALA, seg_pos={0 0}, upload=434577]: Done uploading piece. err: <nil> Dec 26 06:36:27.678952202 (495h7m27.550035237s) [noise=true , address=145.79.227.52:4446 , piece_id=L6QJPT7JLR5O7XO6UGHLVU2LAN2OFSZHLCPO2TBDG2PVZ6FIJGZA, seg_pos={0 0}, upload=434577]: Done uploading piece. err: <nil> Dec 26 06:36:27.679637117 (495h7m27.550720162s) [noise=true , address=145.79.227.2:4114 , piece_id=6L35OSYBZHUUJGCYHXRDCIS22GMUNY3WBOSRUWZCHESCO7I7QG4A, seg_pos={0 0}, upload=434577]: Done uploading piece. err: <nil> Dec 26 06:36:27.682241977 (495h7m27.553325012s) [noise=true , address=145.79.227.29:4942 , piece_id=BQJYKRMRM45P4HIBSMHFXPUL47EDLS6JKEGB42FJ5VB6E46OJHWA, seg_pos={0 0}, upload=434577]: Done uploading piece. err: <nil> Dec 26 06:36:27.690925572 (495h7m27.562008617s) [noise=true , address=145.79.227.25:4617 , piece_id=T3LHNAV74UNQWPWZU3VM774UW6LQ4RADRTC6Y3NSXEGDGHCACIJQ, seg_pos={0 0}, upload=434577]: Done uploading piece. err: <nil> Dec 26 06:36:27.909884327 (495h7m27.780967362s) [noise=true , address=45.91.116.12:4226 , piece_id=MC37EZNWICD2545T6PVZXUNJ5OO4AZMTJDLWDNNBRUTGMZBJXKOA, seg_pos={0 0}, upload=434577]: Done uploading piece. err: <nil> Dec 26 06:36:27.946087294 (495h7m27.817170339s) [noise=true , address=45.91.116.6:4976 , piece_id=ARM3RYXBKNWUT4RDLPV2TPXW4KSYDJCGFDHSHV2AFD22X2JMUK5Q, seg_pos={0 0}, upload=434577]: Done uploading piece. err: <nil> Dec 26 06:36:28.205149221 (495h7m28.076232256s) [noise=true , address=77.90.31.19:4842 , piece_id=BW4BT5IWAVXZYTUYN7DXTNAS3I3NSSGMMWAQKEH64ZF34PDJ7MIA, seg_pos={0 0}, upload=434577]: Done uploading piece. err: <nil> Dec 26 06:36:28.235077159 (495h7m28.106160204s) [noise=true , address=45.91.116.4:4940 , piece_id=PX3K6B2D2FNPO4AK2TY4YMCN3C6CFS3OQMI26ZGI6564Q6OUYPIA, seg_pos={0 0}, upload=434577]: Done uploading piece. err: <nil> Dec 26 06:36:28.636212339 (495h7m28.507295384s) [noise=true , address=45.91.116.14:4189 , piece_id=7PJYJ2M7J7GBGKBEPMTDV5LBO6ITO53XDFG4BXKRM5ND2ZTQNQZQ, seg_pos={0 0}, upload=434577]: Done uploading piece. err: <nil> Dec 26 06:36:28.637196683 (495h7m28.508279718s) [noise=true , address=45.91.116.13:4153 , piece_id=NSIQRTRZ6IYC7U6234YFAEN7ZYW6NK63G75IK6YTXJECYZLAC25A, seg_pos={0 0}, upload=434577]: Done uploading piece. err: <nil> Dec 26 06:36:28.644046045 (495h7m28.51512909s) [noise=true , address=45.91.116.7:4010 , piece_id=UR4BEWMRCCCM5JBPWIIIYJE24IVP2TZP35NLPPGQRTUBWZ4NLXKQ, seg_pos={0 0}, upload=434577]: Done uploading piece. err: <nil> Dec 26 06:36:28.918446120 (495h7m28.789529165s) [noise=true , address=145.79.227.44:4193 , piece_id=PA3JZO3SHIJKINT75USL27VISAB3YNBQJNR326RL3LHO3MBKFCTQ, seg_pos={0 0}, upload=434577]: Done uploading piece. err: <nil> Dec 26 06:36:28.918531934 (495h7m28.789614969s) [seg_pos={0 0}, upload=434577]: Segment reached optimal threshold of 13 pieces. Dec 26 06:36:28.918732516 (495h7m28.789815561s) [noise=true , address=45.91.116.9:4264 , piece_id=VYLOL2PY6DJ3HEWRGP7QHKQKW7LA36D3WOLPDLZIEU56FKNCMRFA, seg_pos={0 0}, upload=434577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 26 06:36:28.918738607 (495h7m28.789821642s) [noise=true , address=45.91.116.9:4264 , piece_id=VYLOL2PY6DJ3HEWRGP7QHKQKW7LA36D3WOLPDLZIEU56FKNCMRFA, seg_pos={0 0}, upload=434577]: Long tail cancellation due to optimal threshold Dec 26 06:36:28.918945321 (495h7m28.790028366s) [noise=true , address=145.79.227.30:4762 , piece_id=32BD4NBDJFFQVNSZV2BDNG7YOBEDUHMJVPKEI7QZ656SGGMVM5DQ, seg_pos={0 0}, upload=434577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 26 06:36:28.918965709 (495h7m28.790048744s) [noise=true , address=145.79.227.30:4762 , piece_id=32BD4NBDJFFQVNSZV2BDNG7YOBEDUHMJVPKEI7QZ656SGGMVM5DQ, seg_pos={0 0}, upload=434577]: Long tail cancellation due to optimal threshold Dec 26 06:36:28.918991459 (495h7m28.790074494s) [seg_pos={0 0}, upload=434577]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 06:36:28.919089966 (495h7m28.790173001s) [upload=434577]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 06:36:28.919097781 (495h7m28.790180816s) [upload=434577]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 06:36:29.222421732 (495h7m29.093504767s) [upload=434577]: Done uploading segments... Dec 26 06:36:29.222439555 (495h7m29.09352259s) [upload=434577]: Upload finished. err: <nil> Duration: 27.441179994s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......................X....................................] duration: 3.467179119s trace: https://jaeger.dev.storj.io/trace/630094597d401477 FAIL (2025-12-26 06:39:27 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.467215948s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [....................X..........X.......X.X.......X.X.XXX.X..] duration: 30.001992278s trace: https://jaeger.dev.storj.io/trace/610a5a278586d69b FAIL (2025-12-26 06:38:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: UR4BEWMRCCCM5JBPWIIIYJE24IVP2TZP35NLPPGQRTUBWZ4NLXKQ) context deadline exceeded; piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: NSIQRTRZ6IYC7U6234YFAEN7ZYW6NK63G75IK6YTXJECYZLAC25A) context deadline exceeded ======LOGS====== Dec 26 06:36:25.402042438 (495h7m25.273125473s) [upload=434577]: Starting upload Dec 26 06:36:25.402099898 (495h7m25.273182943s) [upload=434577]: Done starting upload Dec 26 06:36:25.402171925 (495h7m25.27325496s) [upload=434577]: Uploading segments... Dec 26 06:36:25.402316690 (495h7m25.273399736s) [upload=434577]: Got next segment. Inline: false Dec 26 06:36:25.402354793 (495h7m25.273437838s) [upload=434577]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 06:36:25.402360153 (495h7m25.273443198s) [upload=434577]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 06:36:25.658100426 (495h7m25.529183461s) [seg_pos={0 0}, upload=434577]: Begin upload segment... Dec 26 06:36:25.658147776 (495h7m25.529230821s) [seg_pos={0 0}, upload=434577]: Done begin upload segment. Dec 26 06:36:25.658177322 (495h7m25.529260367s) [noise=true , address=145.79.227.35:4482 , piece_id=OFKG22Z35IGBPSS2CGY56DCC7K5QPNUH6MKEVUICGI7GEYFQFALA, seg_pos={0 0}, upload=434577]: Uploading piece... Dec 26 06:36:25.658185919 (495h7m25.529268954s) [noise=true , address=145.79.227.30:4762 , piece_id=32BD4NBDJFFQVNSZV2BDNG7YOBEDUHMJVPKEI7QZ656SGGMVM5DQ, seg_pos={0 0}, upload=434577]: Uploading piece... Dec 26 06:36:25.658280429 (495h7m25.529363464s) [noise=true , address=45.91.116.6:4976 , piece_id=ARM3RYXBKNWUT4RDLPV2TPXW4KSYDJCGFDHSHV2AFD22X2JMUK5Q, seg_pos={0 0}, upload=434577]: Uploading piece... Dec 26 06:36:25.658817742 (495h7m25.529900777s) [noise=true , address=45.91.116.12:4226 , piece_id=MC37EZNWICD2545T6PVZXUNJ5OO4AZMTJDLWDNNBRUTGMZBJXKOA, seg_pos={0 0}, upload=434577]: Uploading piece... Dec 26 06:36:25.659206312 (495h7m25.530289357s) [noise=true , address=145.79.227.44:4193 , piece_id=PA3JZO3SHIJKINT75USL27VISAB3YNBQJNR326RL3LHO3MBKFCTQ, seg_pos={0 0}, upload=434577]: Uploading piece... Dec 26 06:36:25.659551049 (495h7m25.530634094s) [noise=true , address=45.91.116.7:4010 , piece_id=UR4BEWMRCCCM5JBPWIIIYJE24IVP2TZP35NLPPGQRTUBWZ4NLXKQ, seg_pos={0 0}, upload=434577]: Uploading piece... Dec 26 06:36:25.659953835 (495h7m25.53103688s) [noise=true , address=77.90.31.19:4842 , piece_id=BW4BT5IWAVXZYTUYN7DXTNAS3I3NSSGMMWAQKEH64ZF34PDJ7MIA, seg_pos={0 0}, upload=434577]: Uploading piece... Dec 26 06:36:25.660275498 (495h7m25.531358543s) [noise=true , address=145.79.227.29:4942 , piece_id=BQJYKRMRM45P4HIBSMHFXPUL47EDLS6JKEGB42FJ5VB6E46OJHWA, seg_pos={0 0}, upload=434577]: Uploading piece... Dec 26 06:36:25.660688724 (495h7m25.531771769s) [noise=true , address=45.91.116.9:4264 , piece_id=VYLOL2PY6DJ3HEWRGP7QHKQKW7LA36D3WOLPDLZIEU56FKNCMRFA, seg_pos={0 0}, upload=434577]: Uploading piece... Dec 26 06:36:25.661094387 (495h7m25.532177422s) [noise=true , address=145.79.227.25:4617 , piece_id=T3LHNAV74UNQWPWZU3VM774UW6LQ4RADRTC6Y3NSXEGDGHCACIJQ, seg_pos={0 0}, upload=434577]: Uploading piece... Dec 26 06:36:25.662169645 (495h7m25.53325268s) [noise=true , address=45.91.116.4:4940 , piece_id=PX3K6B2D2FNPO4AK2TY4YMCN3C6CFS3OQMI26ZGI6564Q6OUYPIA, seg_pos={0 0}, upload=434577]: Uploading piece... Dec 26 06:36:25.663392453 (495h7m25.534475488s) [noise=true , address=45.91.116.13:4153 , piece_id=NSIQRTRZ6IYC7U6234YFAEN7ZYW6NK63G75IK6YTXJECYZLAC25A, seg_pos={0 0}, upload=434577]: Uploading piece... Dec 26 06:36:25.664717937 (495h7m25.535800972s) [noise=true , address=145.79.227.2:4114 , piece_id=6L35OSYBZHUUJGCYHXRDCIS22GMUNY3WBOSRUWZCHESCO7I7QG4A, seg_pos={0 0}, upload=434577]: Uploading piece... Dec 26 06:36:25.665768417 (495h7m25.536851462s) [noise=true , address=45.91.116.14:4189 , piece_id=7PJYJ2M7J7GBGKBEPMTDV5LBO6ITO53XDFG4BXKRM5ND2ZTQNQZQ, seg_pos={0 0}, upload=434577]: Uploading piece... Dec 26 06:36:25.667267361 (495h7m25.538350426s) [noise=true , address=145.79.227.52:4446 , piece_id=L6QJPT7JLR5O7XO6UGHLVU2LAN2OFSZHLCPO2TBDG2PVZ6FIJGZA, seg_pos={0 0}, upload=434577]: Uploading piece... Dec 26 06:36:25.668498907 (495h7m25.539581952s) [noise=true , address=77.90.31.18:4805 , piece_id=LEJ5ILCYWMFFMRXDP3OGEK5WKUMHFDRKZY4KYIPO5DH64HYMOJXA, seg_pos={0 0}, upload=434577]: Uploading piece... Dec 26 06:36:26.797649813 (495h7m26.668732868s) [noise=true , address=77.90.31.18:4805 , piece_id=LEJ5ILCYWMFFMRXDP3OGEK5WKUMHFDRKZY4KYIPO5DH64HYMOJXA, seg_pos={0 0}, upload=434577]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: storage node overloaded, request limit: 2000 Dec 26 06:36:27.368822683 (495h7m27.239905728s) [upload=434577]: Next returned nil segment Dec 26 06:36:27.368839776 (495h7m27.239922831s) [upload=434577]: Waiting for error group managing segments... Dec 26 06:36:27.678521873 (495h7m27.549604928s) [noise=true , address=145.79.227.35:4482 , piece_id=OFKG22Z35IGBPSS2CGY56DCC7K5QPNUH6MKEVUICGI7GEYFQFALA, seg_pos={0 0}, upload=434577]: Done uploading piece. err: <nil> Dec 26 06:36:27.678952202 (495h7m27.550035237s) [noise=true , address=145.79.227.52:4446 , piece_id=L6QJPT7JLR5O7XO6UGHLVU2LAN2OFSZHLCPO2TBDG2PVZ6FIJGZA, seg_pos={0 0}, upload=434577]: Done uploading piece. err: <nil> Dec 26 06:36:27.679637117 (495h7m27.550720162s) [noise=true , address=145.79.227.2:4114 , piece_id=6L35OSYBZHUUJGCYHXRDCIS22GMUNY3WBOSRUWZCHESCO7I7QG4A, seg_pos={0 0}, upload=434577]: Done uploading piece. err: <nil> Dec 26 06:36:27.682241977 (495h7m27.553325012s) [noise=true , address=145.79.227.29:4942 , piece_id=BQJYKRMRM45P4HIBSMHFXPUL47EDLS6JKEGB42FJ5VB6E46OJHWA, seg_pos={0 0}, upload=434577]: Done uploading piece. err: <nil> Dec 26 06:36:27.690925572 (495h7m27.562008617s) [noise=true , address=145.79.227.25:4617 , piece_id=T3LHNAV74UNQWPWZU3VM774UW6LQ4RADRTC6Y3NSXEGDGHCACIJQ, seg_pos={0 0}, upload=434577]: Done uploading piece. err: <nil> Dec 26 06:36:27.909884327 (495h7m27.780967362s) [noise=true , address=45.91.116.12:4226 , piece_id=MC37EZNWICD2545T6PVZXUNJ5OO4AZMTJDLWDNNBRUTGMZBJXKOA, seg_pos={0 0}, upload=434577]: Done uploading piece. err: <nil> Dec 26 06:36:27.946087294 (495h7m27.817170339s) [noise=true , address=45.91.116.6:4976 , piece_id=ARM3RYXBKNWUT4RDLPV2TPXW4KSYDJCGFDHSHV2AFD22X2JMUK5Q, seg_pos={0 0}, upload=434577]: Done uploading piece. err: <nil> Dec 26 06:36:28.205149221 (495h7m28.076232256s) [noise=true , address=77.90.31.19:4842 , piece_id=BW4BT5IWAVXZYTUYN7DXTNAS3I3NSSGMMWAQKEH64ZF34PDJ7MIA, seg_pos={0 0}, upload=434577]: Done uploading piece. err: <nil> Dec 26 06:36:28.235077159 (495h7m28.106160204s) [noise=true , address=45.91.116.4:4940 , piece_id=PX3K6B2D2FNPO4AK2TY4YMCN3C6CFS3OQMI26ZGI6564Q6OUYPIA, seg_pos={0 0}, upload=434577]: Done uploading piece. err: <nil> Dec 26 06:36:28.636212339 (495h7m28.507295384s) [noise=true , address=45.91.116.14:4189 , piece_id=7PJYJ2M7J7GBGKBEPMTDV5LBO6ITO53XDFG4BXKRM5ND2ZTQNQZQ, seg_pos={0 0}, upload=434577]: Done uploading piece. err: <nil> Dec 26 06:36:28.637196683 (495h7m28.508279718s) [noise=true , address=45.91.116.13:4153 , piece_id=NSIQRTRZ6IYC7U6234YFAEN7ZYW6NK63G75IK6YTXJECYZLAC25A, seg_pos={0 0}, upload=434577]: Done uploading piece. err: <nil> Dec 26 06:36:28.644046045 (495h7m28.51512909s) [noise=true , address=45.91.116.7:4010 , piece_id=UR4BEWMRCCCM5JBPWIIIYJE24IVP2TZP35NLPPGQRTUBWZ4NLXKQ, seg_pos={0 0}, upload=434577]: Done uploading piece. err: <nil> Dec 26 06:36:28.918446120 (495h7m28.789529165s) [noise=true , address=145.79.227.44:4193 , piece_id=PA3JZO3SHIJKINT75USL27VISAB3YNBQJNR326RL3LHO3MBKFCTQ, seg_pos={0 0}, upload=434577]: Done uploading piece. err: <nil> Dec 26 06:36:28.918531934 (495h7m28.789614969s) [seg_pos={0 0}, upload=434577]: Segment reached optimal threshold of 13 pieces. Dec 26 06:36:28.918732516 (495h7m28.789815561s) [noise=true , address=45.91.116.9:4264 , piece_id=VYLOL2PY6DJ3HEWRGP7QHKQKW7LA36D3WOLPDLZIEU56FKNCMRFA, seg_pos={0 0}, upload=434577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 26 06:36:28.918738607 (495h7m28.789821642s) [noise=true , address=45.91.116.9:4264 , piece_id=VYLOL2PY6DJ3HEWRGP7QHKQKW7LA36D3WOLPDLZIEU56FKNCMRFA, seg_pos={0 0}, upload=434577]: Long tail cancellation due to optimal threshold Dec 26 06:36:28.918945321 (495h7m28.790028366s) [noise=true , address=145.79.227.30:4762 , piece_id=32BD4NBDJFFQVNSZV2BDNG7YOBEDUHMJVPKEI7QZ656SGGMVM5DQ, seg_pos={0 0}, upload=434577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 26 06:36:28.918965709 (495h7m28.790048744s) [noise=true , address=145.79.227.30:4762 , piece_id=32BD4NBDJFFQVNSZV2BDNG7YOBEDUHMJVPKEI7QZ656SGGMVM5DQ, seg_pos={0 0}, upload=434577]: Long tail cancellation due to optimal threshold Dec 26 06:36:28.918991459 (495h7m28.790074494s) [seg_pos={0 0}, upload=434577]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 06:36:28.919089966 (495h7m28.790173001s) [upload=434577]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 06:36:28.919097781 (495h7m28.790180816s) [upload=434577]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 06:36:29.222421732 (495h7m29.093504767s) [upload=434577]: Done uploading segments... Dec 26 06:36:29.222439555 (495h7m29.09352259s) [upload=434577]: Upload finished. err: <nil> Duration: 30.002018326s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........................X...................................] duration: 3.547376099s trace: https://jaeger.dev.storj.io/trace/630094597d401477 FAIL (2025-12-26 06:38:24 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.547437395s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.....................X..........X.......X.X.......X.X.XXX.X.] duration: 5.77791783s trace: https://jaeger.dev.storj.io/trace/610a5a278586d69b FAIL (2025-12-26 06:37:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: UR4BEWMRCCCM5JBPWIIIYJE24IVP2TZP35NLPPGQRTUBWZ4NLXKQ) context deadline exceeded; piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: NSIQRTRZ6IYC7U6234YFAEN7ZYW6NK63G75IK6YTXJECYZLAC25A) context deadline exceeded ======LOGS====== Dec 26 06:36:25.402042438 (495h7m25.273125473s) [upload=434577]: Starting upload Dec 26 06:36:25.402099898 (495h7m25.273182943s) [upload=434577]: Done starting upload Dec 26 06:36:25.402171925 (495h7m25.27325496s) [upload=434577]: Uploading segments... Dec 26 06:36:25.402316690 (495h7m25.273399736s) [upload=434577]: Got next segment. Inline: false Dec 26 06:36:25.402354793 (495h7m25.273437838s) [upload=434577]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 06:36:25.402360153 (495h7m25.273443198s) [upload=434577]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 06:36:25.658100426 (495h7m25.529183461s) [seg_pos={0 0}, upload=434577]: Begin upload segment... Dec 26 06:36:25.658147776 (495h7m25.529230821s) [seg_pos={0 0}, upload=434577]: Done begin upload segment. Dec 26 06:36:25.658177322 (495h7m25.529260367s) [noise=true , address=145.79.227.35:4482 , piece_id=OFKG22Z35IGBPSS2CGY56DCC7K5QPNUH6MKEVUICGI7GEYFQFALA, seg_pos={0 0}, upload=434577]: Uploading piece... Dec 26 06:36:25.658185919 (495h7m25.529268954s) [noise=true , address=145.79.227.30:4762 , piece_id=32BD4NBDJFFQVNSZV2BDNG7YOBEDUHMJVPKEI7QZ656SGGMVM5DQ, seg_pos={0 0}, upload=434577]: Uploading piece... Dec 26 06:36:25.658280429 (495h7m25.529363464s) [noise=true , address=45.91.116.6:4976 , piece_id=ARM3RYXBKNWUT4RDLPV2TPXW4KSYDJCGFDHSHV2AFD22X2JMUK5Q, seg_pos={0 0}, upload=434577]: Uploading piece... Dec 26 06:36:25.658817742 (495h7m25.529900777s) [noise=true , address=45.91.116.12:4226 , piece_id=MC37EZNWICD2545T6PVZXUNJ5OO4AZMTJDLWDNNBRUTGMZBJXKOA, seg_pos={0 0}, upload=434577]: Uploading piece... Dec 26 06:36:25.659206312 (495h7m25.530289357s) [noise=true , address=145.79.227.44:4193 , piece_id=PA3JZO3SHIJKINT75USL27VISAB3YNBQJNR326RL3LHO3MBKFCTQ, seg_pos={0 0}, upload=434577]: Uploading piece... Dec 26 06:36:25.659551049 (495h7m25.530634094s) [noise=true , address=45.91.116.7:4010 , piece_id=UR4BEWMRCCCM5JBPWIIIYJE24IVP2TZP35NLPPGQRTUBWZ4NLXKQ, seg_pos={0 0}, upload=434577]: Uploading piece... Dec 26 06:36:25.659953835 (495h7m25.53103688s) [noise=true , address=77.90.31.19:4842 , piece_id=BW4BT5IWAVXZYTUYN7DXTNAS3I3NSSGMMWAQKEH64ZF34PDJ7MIA, seg_pos={0 0}, upload=434577]: Uploading piece... Dec 26 06:36:25.660275498 (495h7m25.531358543s) [noise=true , address=145.79.227.29:4942 , piece_id=BQJYKRMRM45P4HIBSMHFXPUL47EDLS6JKEGB42FJ5VB6E46OJHWA, seg_pos={0 0}, upload=434577]: Uploading piece... Dec 26 06:36:25.660688724 (495h7m25.531771769s) [noise=true , address=45.91.116.9:4264 , piece_id=VYLOL2PY6DJ3HEWRGP7QHKQKW7LA36D3WOLPDLZIEU56FKNCMRFA, seg_pos={0 0}, upload=434577]: Uploading piece... Dec 26 06:36:25.661094387 (495h7m25.532177422s) [noise=true , address=145.79.227.25:4617 , piece_id=T3LHNAV74UNQWPWZU3VM774UW6LQ4RADRTC6Y3NSXEGDGHCACIJQ, seg_pos={0 0}, upload=434577]: Uploading piece... Dec 26 06:36:25.662169645 (495h7m25.53325268s) [noise=true , address=45.91.116.4:4940 , piece_id=PX3K6B2D2FNPO4AK2TY4YMCN3C6CFS3OQMI26ZGI6564Q6OUYPIA, seg_pos={0 0}, upload=434577]: Uploading piece... Dec 26 06:36:25.663392453 (495h7m25.534475488s) [noise=true , address=45.91.116.13:4153 , piece_id=NSIQRTRZ6IYC7U6234YFAEN7ZYW6NK63G75IK6YTXJECYZLAC25A, seg_pos={0 0}, upload=434577]: Uploading piece... Dec 26 06:36:25.664717937 (495h7m25.535800972s) [noise=true , address=145.79.227.2:4114 , piece_id=6L35OSYBZHUUJGCYHXRDCIS22GMUNY3WBOSRUWZCHESCO7I7QG4A, seg_pos={0 0}, upload=434577]: Uploading piece... Dec 26 06:36:25.665768417 (495h7m25.536851462s) [noise=true , address=45.91.116.14:4189 , piece_id=7PJYJ2M7J7GBGKBEPMTDV5LBO6ITO53XDFG4BXKRM5ND2ZTQNQZQ, seg_pos={0 0}, upload=434577]: Uploading piece... Dec 26 06:36:25.667267361 (495h7m25.538350426s) [noise=true , address=145.79.227.52:4446 , piece_id=L6QJPT7JLR5O7XO6UGHLVU2LAN2OFSZHLCPO2TBDG2PVZ6FIJGZA, seg_pos={0 0}, upload=434577]: Uploading piece... Dec 26 06:36:25.668498907 (495h7m25.539581952s) [noise=true , address=77.90.31.18:4805 , piece_id=LEJ5ILCYWMFFMRXDP3OGEK5WKUMHFDRKZY4KYIPO5DH64HYMOJXA, seg_pos={0 0}, upload=434577]: Uploading piece... Dec 26 06:36:26.797649813 (495h7m26.668732868s) [noise=true , address=77.90.31.18:4805 , piece_id=LEJ5ILCYWMFFMRXDP3OGEK5WKUMHFDRKZY4KYIPO5DH64HYMOJXA, seg_pos={0 0}, upload=434577]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: storage node overloaded, request limit: 2000 Dec 26 06:36:27.368822683 (495h7m27.239905728s) [upload=434577]: Next returned nil segment Dec 26 06:36:27.368839776 (495h7m27.239922831s) [upload=434577]: Waiting for error group managing segments... Dec 26 06:36:27.678521873 (495h7m27.549604928s) [noise=true , address=145.79.227.35:4482 , piece_id=OFKG22Z35IGBPSS2CGY56DCC7K5QPNUH6MKEVUICGI7GEYFQFALA, seg_pos={0 0}, upload=434577]: Done uploading piece. err: <nil> Dec 26 06:36:27.678952202 (495h7m27.550035237s) [noise=true , address=145.79.227.52:4446 , piece_id=L6QJPT7JLR5O7XO6UGHLVU2LAN2OFSZHLCPO2TBDG2PVZ6FIJGZA, seg_pos={0 0}, upload=434577]: Done uploading piece. err: <nil> Dec 26 06:36:27.679637117 (495h7m27.550720162s) [noise=true , address=145.79.227.2:4114 , piece_id=6L35OSYBZHUUJGCYHXRDCIS22GMUNY3WBOSRUWZCHESCO7I7QG4A, seg_pos={0 0}, upload=434577]: Done uploading piece. err: <nil> Dec 26 06:36:27.682241977 (495h7m27.553325012s) [noise=true , address=145.79.227.29:4942 , piece_id=BQJYKRMRM45P4HIBSMHFXPUL47EDLS6JKEGB42FJ5VB6E46OJHWA, seg_pos={0 0}, upload=434577]: Done uploading piece. err: <nil> Dec 26 06:36:27.690925572 (495h7m27.562008617s) [noise=true , address=145.79.227.25:4617 , piece_id=T3LHNAV74UNQWPWZU3VM774UW6LQ4RADRTC6Y3NSXEGDGHCACIJQ, seg_pos={0 0}, upload=434577]: Done uploading piece. err: <nil> Dec 26 06:36:27.909884327 (495h7m27.780967362s) [noise=true , address=45.91.116.12:4226 , piece_id=MC37EZNWICD2545T6PVZXUNJ5OO4AZMTJDLWDNNBRUTGMZBJXKOA, seg_pos={0 0}, upload=434577]: Done uploading piece. err: <nil> Dec 26 06:36:27.946087294 (495h7m27.817170339s) [noise=true , address=45.91.116.6:4976 , piece_id=ARM3RYXBKNWUT4RDLPV2TPXW4KSYDJCGFDHSHV2AFD22X2JMUK5Q, seg_pos={0 0}, upload=434577]: Done uploading piece. err: <nil> Dec 26 06:36:28.205149221 (495h7m28.076232256s) [noise=true , address=77.90.31.19:4842 , piece_id=BW4BT5IWAVXZYTUYN7DXTNAS3I3NSSGMMWAQKEH64ZF34PDJ7MIA, seg_pos={0 0}, upload=434577]: Done uploading piece. err: <nil> Dec 26 06:36:28.235077159 (495h7m28.106160204s) [noise=true , address=45.91.116.4:4940 , piece_id=PX3K6B2D2FNPO4AK2TY4YMCN3C6CFS3OQMI26ZGI6564Q6OUYPIA, seg_pos={0 0}, upload=434577]: Done uploading piece. err: <nil> Dec 26 06:36:28.636212339 (495h7m28.507295384s) [noise=true , address=45.91.116.14:4189 , piece_id=7PJYJ2M7J7GBGKBEPMTDV5LBO6ITO53XDFG4BXKRM5ND2ZTQNQZQ, seg_pos={0 0}, upload=434577]: Done uploading piece. err: <nil> Dec 26 06:36:28.637196683 (495h7m28.508279718s) [noise=true , address=45.91.116.13:4153 , piece_id=NSIQRTRZ6IYC7U6234YFAEN7ZYW6NK63G75IK6YTXJECYZLAC25A, seg_pos={0 0}, upload=434577]: Done uploading piece. err: <nil> Dec 26 06:36:28.644046045 (495h7m28.51512909s) [noise=true , address=45.91.116.7:4010 , piece_id=UR4BEWMRCCCM5JBPWIIIYJE24IVP2TZP35NLPPGQRTUBWZ4NLXKQ, seg_pos={0 0}, upload=434577]: Done uploading piece. err: <nil> Dec 26 06:36:28.918446120 (495h7m28.789529165s) [noise=true , address=145.79.227.44:4193 , piece_id=PA3JZO3SHIJKINT75USL27VISAB3YNBQJNR326RL3LHO3MBKFCTQ, seg_pos={0 0}, upload=434577]: Done uploading piece. err: <nil> Dec 26 06:36:28.918531934 (495h7m28.789614969s) [seg_pos={0 0}, upload=434577]: Segment reached optimal threshold of 13 pieces. Dec 26 06:36:28.918732516 (495h7m28.789815561s) [noise=true , address=45.91.116.9:4264 , piece_id=VYLOL2PY6DJ3HEWRGP7QHKQKW7LA36D3WOLPDLZIEU56FKNCMRFA, seg_pos={0 0}, upload=434577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 26 06:36:28.918738607 (495h7m28.789821642s) [noise=true , address=45.91.116.9:4264 , piece_id=VYLOL2PY6DJ3HEWRGP7QHKQKW7LA36D3WOLPDLZIEU56FKNCMRFA, seg_pos={0 0}, upload=434577]: Long tail cancellation due to optimal threshold Dec 26 06:36:28.918945321 (495h7m28.790028366s) [noise=true , address=145.79.227.30:4762 , piece_id=32BD4NBDJFFQVNSZV2BDNG7YOBEDUHMJVPKEI7QZ656SGGMVM5DQ, seg_pos={0 0}, upload=434577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 26 06:36:28.918965709 (495h7m28.790048744s) [noise=true , address=145.79.227.30:4762 , piece_id=32BD4NBDJFFQVNSZV2BDNG7YOBEDUHMJVPKEI7QZ656SGGMVM5DQ, seg_pos={0 0}, upload=434577]: Long tail cancellation due to optimal threshold Dec 26 06:36:28.918991459 (495h7m28.790074494s) [seg_pos={0 0}, upload=434577]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 06:36:28.919089966 (495h7m28.790173001s) [upload=434577]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 06:36:28.919097781 (495h7m28.790180816s) [upload=434577]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 06:36:29.222421732 (495h7m29.093504767s) [upload=434577]: Done uploading segments... Dec 26 06:36:29.222439555 (495h7m29.09352259s) [upload=434577]: Upload finished. err: <nil> Duration: 5.777942918s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........................X..................................] duration: 3.414196792s trace: https://jaeger.dev.storj.io/trace/630094597d401477 FAIL (2025-12-26 06:37:10 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.414223614s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [......................X..........X.......X.X.......X.X.XXX.X] duration: 30.002690765s trace: https://jaeger.dev.storj.io/trace/610a5a278586d69b FAIL (2025-12-26 06:36:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: UR4BEWMRCCCM5JBPWIIIYJE24IVP2TZP35NLPPGQRTUBWZ4NLXKQ) context deadline exceeded; piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: NSIQRTRZ6IYC7U6234YFAEN7ZYW6NK63G75IK6YTXJECYZLAC25A) context deadline exceeded ======LOGS====== Dec 26 06:36:25.402042438 (495h7m25.273125473s) [upload=434577]: Starting upload Dec 26 06:36:25.402099898 (495h7m25.273182943s) [upload=434577]: Done starting upload Dec 26 06:36:25.402171925 (495h7m25.27325496s) [upload=434577]: Uploading segments... Dec 26 06:36:25.402316690 (495h7m25.273399736s) [upload=434577]: Got next segment. Inline: false Dec 26 06:36:25.402354793 (495h7m25.273437838s) [upload=434577]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 06:36:25.402360153 (495h7m25.273443198s) [upload=434577]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 06:36:25.658100426 (495h7m25.529183461s) [seg_pos={0 0}, upload=434577]: Begin upload segment... Dec 26 06:36:25.658147776 (495h7m25.529230821s) [seg_pos={0 0}, upload=434577]: Done begin upload segment. Dec 26 06:36:25.658177322 (495h7m25.529260367s) [noise=true , address=145.79.227.35:4482 , piece_id=OFKG22Z35IGBPSS2CGY56DCC7K5QPNUH6MKEVUICGI7GEYFQFALA, seg_pos={0 0}, upload=434577]: Uploading piece... Dec 26 06:36:25.658185919 (495h7m25.529268954s) [noise=true , address=145.79.227.30:4762 , piece_id=32BD4NBDJFFQVNSZV2BDNG7YOBEDUHMJVPKEI7QZ656SGGMVM5DQ, seg_pos={0 0}, upload=434577]: Uploading piece... Dec 26 06:36:25.658280429 (495h7m25.529363464s) [noise=true , address=45.91.116.6:4976 , piece_id=ARM3RYXBKNWUT4RDLPV2TPXW4KSYDJCGFDHSHV2AFD22X2JMUK5Q, seg_pos={0 0}, upload=434577]: Uploading piece... Dec 26 06:36:25.658817742 (495h7m25.529900777s) [noise=true , address=45.91.116.12:4226 , piece_id=MC37EZNWICD2545T6PVZXUNJ5OO4AZMTJDLWDNNBRUTGMZBJXKOA, seg_pos={0 0}, upload=434577]: Uploading piece... Dec 26 06:36:25.659206312 (495h7m25.530289357s) [noise=true , address=145.79.227.44:4193 , piece_id=PA3JZO3SHIJKINT75USL27VISAB3YNBQJNR326RL3LHO3MBKFCTQ, seg_pos={0 0}, upload=434577]: Uploading piece... Dec 26 06:36:25.659551049 (495h7m25.530634094s) [noise=true , address=45.91.116.7:4010 , piece_id=UR4BEWMRCCCM5JBPWIIIYJE24IVP2TZP35NLPPGQRTUBWZ4NLXKQ, seg_pos={0 0}, upload=434577]: Uploading piece... Dec 26 06:36:25.659953835 (495h7m25.53103688s) [noise=true , address=77.90.31.19:4842 , piece_id=BW4BT5IWAVXZYTUYN7DXTNAS3I3NSSGMMWAQKEH64ZF34PDJ7MIA, seg_pos={0 0}, upload=434577]: Uploading piece... Dec 26 06:36:25.660275498 (495h7m25.531358543s) [noise=true , address=145.79.227.29:4942 , piece_id=BQJYKRMRM45P4HIBSMHFXPUL47EDLS6JKEGB42FJ5VB6E46OJHWA, seg_pos={0 0}, upload=434577]: Uploading piece... Dec 26 06:36:25.660688724 (495h7m25.531771769s) [noise=true , address=45.91.116.9:4264 , piece_id=VYLOL2PY6DJ3HEWRGP7QHKQKW7LA36D3WOLPDLZIEU56FKNCMRFA, seg_pos={0 0}, upload=434577]: Uploading piece... Dec 26 06:36:25.661094387 (495h7m25.532177422s) [noise=true , address=145.79.227.25:4617 , piece_id=T3LHNAV74UNQWPWZU3VM774UW6LQ4RADRTC6Y3NSXEGDGHCACIJQ, seg_pos={0 0}, upload=434577]: Uploading piece... Dec 26 06:36:25.662169645 (495h7m25.53325268s) [noise=true , address=45.91.116.4:4940 , piece_id=PX3K6B2D2FNPO4AK2TY4YMCN3C6CFS3OQMI26ZGI6564Q6OUYPIA, seg_pos={0 0}, upload=434577]: Uploading piece... Dec 26 06:36:25.663392453 (495h7m25.534475488s) [noise=true , address=45.91.116.13:4153 , piece_id=NSIQRTRZ6IYC7U6234YFAEN7ZYW6NK63G75IK6YTXJECYZLAC25A, seg_pos={0 0}, upload=434577]: Uploading piece... Dec 26 06:36:25.664717937 (495h7m25.535800972s) [noise=true , address=145.79.227.2:4114 , piece_id=6L35OSYBZHUUJGCYHXRDCIS22GMUNY3WBOSRUWZCHESCO7I7QG4A, seg_pos={0 0}, upload=434577]: Uploading piece... Dec 26 06:36:25.665768417 (495h7m25.536851462s) [noise=true , address=45.91.116.14:4189 , piece_id=7PJYJ2M7J7GBGKBEPMTDV5LBO6ITO53XDFG4BXKRM5ND2ZTQNQZQ, seg_pos={0 0}, upload=434577]: Uploading piece... Dec 26 06:36:25.667267361 (495h7m25.538350426s) [noise=true , address=145.79.227.52:4446 , piece_id=L6QJPT7JLR5O7XO6UGHLVU2LAN2OFSZHLCPO2TBDG2PVZ6FIJGZA, seg_pos={0 0}, upload=434577]: Uploading piece... Dec 26 06:36:25.668498907 (495h7m25.539581952s) [noise=true , address=77.90.31.18:4805 , piece_id=LEJ5ILCYWMFFMRXDP3OGEK5WKUMHFDRKZY4KYIPO5DH64HYMOJXA, seg_pos={0 0}, upload=434577]: Uploading piece... Dec 26 06:36:26.797649813 (495h7m26.668732868s) [noise=true , address=77.90.31.18:4805 , piece_id=LEJ5ILCYWMFFMRXDP3OGEK5WKUMHFDRKZY4KYIPO5DH64HYMOJXA, seg_pos={0 0}, upload=434577]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: storage node overloaded, request limit: 2000 Dec 26 06:36:27.368822683 (495h7m27.239905728s) [upload=434577]: Next returned nil segment Dec 26 06:36:27.368839776 (495h7m27.239922831s) [upload=434577]: Waiting for error group managing segments... Dec 26 06:36:27.678521873 (495h7m27.549604928s) [noise=true , address=145.79.227.35:4482 , piece_id=OFKG22Z35IGBPSS2CGY56DCC7K5QPNUH6MKEVUICGI7GEYFQFALA, seg_pos={0 0}, upload=434577]: Done uploading piece. err: <nil> Dec 26 06:36:27.678952202 (495h7m27.550035237s) [noise=true , address=145.79.227.52:4446 , piece_id=L6QJPT7JLR5O7XO6UGHLVU2LAN2OFSZHLCPO2TBDG2PVZ6FIJGZA, seg_pos={0 0}, upload=434577]: Done uploading piece. err: <nil> Dec 26 06:36:27.679637117 (495h7m27.550720162s) [noise=true , address=145.79.227.2:4114 , piece_id=6L35OSYBZHUUJGCYHXRDCIS22GMUNY3WBOSRUWZCHESCO7I7QG4A, seg_pos={0 0}, upload=434577]: Done uploading piece. err: <nil> Dec 26 06:36:27.682241977 (495h7m27.553325012s) [noise=true , address=145.79.227.29:4942 , piece_id=BQJYKRMRM45P4HIBSMHFXPUL47EDLS6JKEGB42FJ5VB6E46OJHWA, seg_pos={0 0}, upload=434577]: Done uploading piece. err: <nil> Dec 26 06:36:27.690925572 (495h7m27.562008617s) [noise=true , address=145.79.227.25:4617 , piece_id=T3LHNAV74UNQWPWZU3VM774UW6LQ4RADRTC6Y3NSXEGDGHCACIJQ, seg_pos={0 0}, upload=434577]: Done uploading piece. err: <nil> Dec 26 06:36:27.909884327 (495h7m27.780967362s) [noise=true , address=45.91.116.12:4226 , piece_id=MC37EZNWICD2545T6PVZXUNJ5OO4AZMTJDLWDNNBRUTGMZBJXKOA, seg_pos={0 0}, upload=434577]: Done uploading piece. err: <nil> Dec 26 06:36:27.946087294 (495h7m27.817170339s) [noise=true , address=45.91.116.6:4976 , piece_id=ARM3RYXBKNWUT4RDLPV2TPXW4KSYDJCGFDHSHV2AFD22X2JMUK5Q, seg_pos={0 0}, upload=434577]: Done uploading piece. err: <nil> Dec 26 06:36:28.205149221 (495h7m28.076232256s) [noise=true , address=77.90.31.19:4842 , piece_id=BW4BT5IWAVXZYTUYN7DXTNAS3I3NSSGMMWAQKEH64ZF34PDJ7MIA, seg_pos={0 0}, upload=434577]: Done uploading piece. err: <nil> Dec 26 06:36:28.235077159 (495h7m28.106160204s) [noise=true , address=45.91.116.4:4940 , piece_id=PX3K6B2D2FNPO4AK2TY4YMCN3C6CFS3OQMI26ZGI6564Q6OUYPIA, seg_pos={0 0}, upload=434577]: Done uploading piece. err: <nil> Dec 26 06:36:28.636212339 (495h7m28.507295384s) [noise=true , address=45.91.116.14:4189 , piece_id=7PJYJ2M7J7GBGKBEPMTDV5LBO6ITO53XDFG4BXKRM5ND2ZTQNQZQ, seg_pos={0 0}, upload=434577]: Done uploading piece. err: <nil> Dec 26 06:36:28.637196683 (495h7m28.508279718s) [noise=true , address=45.91.116.13:4153 , piece_id=NSIQRTRZ6IYC7U6234YFAEN7ZYW6NK63G75IK6YTXJECYZLAC25A, seg_pos={0 0}, upload=434577]: Done uploading piece. err: <nil> Dec 26 06:36:28.644046045 (495h7m28.51512909s) [noise=true , address=45.91.116.7:4010 , piece_id=UR4BEWMRCCCM5JBPWIIIYJE24IVP2TZP35NLPPGQRTUBWZ4NLXKQ, seg_pos={0 0}, upload=434577]: Done uploading piece. err: <nil> Dec 26 06:36:28.918446120 (495h7m28.789529165s) [noise=true , address=145.79.227.44:4193 , piece_id=PA3JZO3SHIJKINT75USL27VISAB3YNBQJNR326RL3LHO3MBKFCTQ, seg_pos={0 0}, upload=434577]: Done uploading piece. err: <nil> Dec 26 06:36:28.918531934 (495h7m28.789614969s) [seg_pos={0 0}, upload=434577]: Segment reached optimal threshold of 13 pieces. Dec 26 06:36:28.918732516 (495h7m28.789815561s) [noise=true , address=45.91.116.9:4264 , piece_id=VYLOL2PY6DJ3HEWRGP7QHKQKW7LA36D3WOLPDLZIEU56FKNCMRFA, seg_pos={0 0}, upload=434577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 26 06:36:28.918738607 (495h7m28.789821642s) [noise=true , address=45.91.116.9:4264 , piece_id=VYLOL2PY6DJ3HEWRGP7QHKQKW7LA36D3WOLPDLZIEU56FKNCMRFA, seg_pos={0 0}, upload=434577]: Long tail cancellation due to optimal threshold Dec 26 06:36:28.918945321 (495h7m28.790028366s) [noise=true , address=145.79.227.30:4762 , piece_id=32BD4NBDJFFQVNSZV2BDNG7YOBEDUHMJVPKEI7QZ656SGGMVM5DQ, seg_pos={0 0}, upload=434577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 26 06:36:28.918965709 (495h7m28.790048744s) [noise=true , address=145.79.227.30:4762 , piece_id=32BD4NBDJFFQVNSZV2BDNG7YOBEDUHMJVPKEI7QZ656SGGMVM5DQ, seg_pos={0 0}, upload=434577]: Long tail cancellation due to optimal threshold Dec 26 06:36:28.918991459 (495h7m28.790074494s) [seg_pos={0 0}, upload=434577]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 06:36:28.919089966 (495h7m28.790173001s) [upload=434577]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 06:36:28.919097781 (495h7m28.790180816s) [upload=434577]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 06:36:29.222421732 (495h7m29.093504767s) [upload=434577]: Done uploading segments... Dec 26 06:36:29.222439555 (495h7m29.09352259s) [upload=434577]: Upload finished. err: <nil> Duration: 30.002713298s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........................X.................................] duration: 3.898275652s trace: https://jaeger.dev.storj.io/trace/630094597d401477 FAIL (2025-12-26 06:35:56 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.898303334s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 25.641121815s trace: https://jaeger.dev.storj.io/trace/7e198d4c8fea54d2 FAIL (2025-12-26 06:35:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p, Piece ID: SBCEX7UNWGNFF54SBP4AGRDV5IPG24FQJXJJ7EYQTBSZKOVZWU4Q) context deadline exceeded; piecestore: (Node ID: 1nP8xospdMuDEh3em5K6iBnh8bHXqLhvHy4pHm9m8Lnj4kEH2X, Piece ID: XV7FQGBJNWVRLOAFRUPOWON35JR5ESRJPTCWPUGYWGHJPSDVL4GA) context deadline exceeded; piecestore: (Node ID: 1Mg5mxUjW56quDidSgA788D7ByfAzvsEuWJqwtzY4bLyxRn3mu, Piece ID: ATPIXLMXGONSSLXPPVORSNC6LHPXI2V3UFTHM4VHA2FTY5ZM6H3A) context deadline exceeded; piecestore: (Node ID: 12QmYSEAKnkSLEEVc9E1uxtVigunKMFLk2cdu71Bse8Tqec3HyT, Piece ID: YFKD7GO2FOSPLDXNFPK732QTLCLBQQASWBTMZ2LTLBGSK22PIXIQ) context deadline exceeded; piecestore: (Node ID: 1JsnHuhQhM4rWLTLEuSSHLNE24m7tSW9qMSf48j7yMrAVPWVyV, Piece ID: ARBMDAKX4QBW6JG47KP2PUITCX7HKXCBO75THQPINM4LLHCKAV4Q) context deadline exceeded ======LOGS====== upload=433902]: Uploading piece... Dec 26 05:50:34.986155265 (494h21m34.8572383s) [noise=true , address=64.49.13.140:4509 , piece_id=6TGWTOAVY4YVAKU6VEE2SWVAM264TKU6ZNMGPWH4TNGII3UYXFXQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.986703610 (494h21m34.857786655s) [noise=true , address=77.90.31.10:4522 , piece_id=YFKD7GO2FOSPLDXNFPK732QTLCLBQQASWBTMZ2LTLBGSK22PIXIQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.987145380 (494h21m34.858228425s) [noise=true , address=145.79.227.33:4890 , piece_id=UKNB4ZZTYJGY4MDHOBPYUTHCP3ZDCMJB7O7AJZFYP2HFVIWLUOKQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.987628290 (494h21m34.858711335s) [noise=true , address=145.79.227.40:4052 , piece_id=XRZYL7PT5VHY22N7CCKS3NTZKDDWE36ZU3SORF6ILM7ZMXDND63A, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.988710731 (494h21m34.859793776s) [noise=true , address=77.90.31.13:4653 , piece_id=EY2CWX54ONOWPO3WC33G5QSP5RMP2SUTBQLUOH5JXGDRUEPDHQ2A, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.989894545 (494h21m34.86097759s) [noise=true , address=145.79.227.30:4785 , piece_id=5KMLU4ATBN4YVQDYJKKQY4PEJNKWTO2WNX5GMGJM4F54B6YM2SFA, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.991305081 (494h21m34.862388116s) [noise=true , address=145.79.227.50:4381 , piece_id=H6Q47DIOTOWGN7QH2KWT4IH6KVDSB246S22753Z3UQGB26T6VXPA, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.995796263 (494h21m34.866879359s) [noise=true , address=145.79.227.52:4461 , piece_id=ZAAUDBGY6CC2JNEGWXVPCBAWXARHAJDZ3YPB6Y3NWEWCFDUFGDSQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.996765439 (494h21m34.867848484s) [noise=true , address=77.90.31.16:4755 , piece_id=ARBMDAKX4QBW6JG47KP2PUITCX7HKXCBO75THQPINM4LLHCKAV4Q, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.998107634 (494h21m34.869190679s) [noise=true , address=77.90.31.11:4579 , piece_id=SBCEX7UNWGNFF54SBP4AGRDV5IPG24FQJXJJ7EYQTBSZKOVZWU4Q, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.999340442 (494h21m34.870423477s) [noise=true , address=145.79.227.34:4936 , piece_id=EEA2ZPM44DX3WMI7ZJTDC7T5UIGED6TNT53QXZWGBFRMQUQSCXPQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:35.000540757 (494h21m34.871623812s) [noise=true , address=45.91.116.14:4204 , piece_id=XV7FQGBJNWVRLOAFRUPOWON35JR5ESRJPTCWPUGYWGHJPSDVL4GA, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:36.360077064 (494h21m36.231160099s) [upload=433902]: Next returned nil segment Dec 26 05:50:36.360091642 (494h21m36.231174688s) [upload=433902]: Waiting for error group managing segments... Dec 26 05:50:36.688914299 (494h21m36.559997344s) [noise=true , address=64.49.13.140:4509 , piece_id=6TGWTOAVY4YVAKU6VEE2SWVAM264TKU6ZNMGPWH4TNGII3UYXFXQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.059812634 (494h21m36.930895669s) [noise=true , address=77.90.31.11:4579 , piece_id=SBCEX7UNWGNFF54SBP4AGRDV5IPG24FQJXJJ7EYQTBSZKOVZWU4Q, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.119433874 (494h21m36.990516909s) [noise=true , address=45.91.116.14:4204 , piece_id=XV7FQGBJNWVRLOAFRUPOWON35JR5ESRJPTCWPUGYWGHJPSDVL4GA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.138718204 (494h21m37.00980125s) [noise=true , address=77.90.31.16:4755 , piece_id=ARBMDAKX4QBW6JG47KP2PUITCX7HKXCBO75THQPINM4LLHCKAV4Q, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.146647243 (494h21m37.017730278s) [noise=true , address=77.90.31.13:4653 , piece_id=EY2CWX54ONOWPO3WC33G5QSP5RMP2SUTBQLUOH5JXGDRUEPDHQ2A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.163283722 (494h21m37.034366757s) [noise=true , address=77.90.31.21:4893 , piece_id=GNEEZO2XGOKNODOUPZQ4NGKFE4PCX5LNUWXLV27F6X7GWZVSYNAA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.374956574 (494h21m37.246039609s) [noise=true , address=45.91.116.7:4033 , piece_id=ATPIXLMXGONSSLXPPVORSNC6LHPXI2V3UFTHM4VHA2FTY5ZM6H3A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.777283355 (494h21m37.64836639s) [noise=true , address=77.90.31.12:4622 , piece_id=LTYYPK6ZQ3NXZPBKSV4OMZGCOXXV3N6L5A6MVODNR644D2ZUJX3A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:38.439025909 (494h21m38.310108944s) [noise=true , address=145.79.227.49:4355 , piece_id=IUDOHR73Y6OKFAQSNNSONALWAITRJ4POYRO2QD5LYZKOSTWY4MNA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:38.439599111 (494h21m38.310682166s) [noise=true , address=77.90.31.10:4522 , piece_id=YFKD7GO2FOSPLDXNFPK732QTLCLBQQASWBTMZ2LTLBGSK22PIXIQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:39.259154980 (494h21m39.130238015s) [noise=true , address=145.79.227.34:4936 , piece_id=EEA2ZPM44DX3WMI7ZJTDC7T5UIGED6TNT53QXZWGBFRMQUQSCXPQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:39.430935921 (494h21m39.302018956s) [noise=true , address=145.79.227.44:4212 , piece_id=ABBNLKRE4FM7RHF56V4IAQPKBCCCQLLOMFVMSJEWQUNFGGCI3PVQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:42.941071297 (494h21m42.812154352s) [noise=true , address=77.90.31.14:4675 , piece_id=FKB4O45OYICMELPGXB65QAUD2DRIFTSXZFKIA6TY26VFAP6AVY3A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:51:22.719026850 (494h22m22.590109885s) [noise=true , address=145.79.227.33:4890 , piece_id=UKNB4ZZTYJGY4MDHOBPYUTHCP3ZDCMJB7O7AJZFYP2HFVIWLUOKQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:51:22.719060193 (494h22m22.590143228s) [seg_pos={0 0}, upload=433902]: Segment reached optimal threshold of 14 pieces. Dec 26 05:51:22.719233022 (494h22m22.590316067s) [noise=true , address=145.79.227.50:4381 , piece_id=H6Q47DIOTOWGN7QH2KWT4IH6KVDSB246S22753Z3UQGB26T6VXPA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zU35MWWaaC1npkP5qGcWBXAkfxGsd5UsdkM2nwX9x7wUGCDbY): protocol: write tcp 188.166.225.170:46970->145.79.227.50:4381: use of closed network connection; write tcp 188.166.225.170:46970->145.79.227.50:4381: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46970->145.79.227.50:4381: use of closed network connection Dec 26 05:51:22.719238783 (494h22m22.590321828s) [noise=true , address=145.79.227.50:4381 , piece_id=H6Q47DIOTOWGN7QH2KWT4IH6KVDSB246S22753Z3UQGB26T6VXPA, seg_pos={0 0}, upload=433902]: Long tail cancellation due to optimal threshold Dec 26 05:51:22.719251677 (494h22m22.590334712s) [noise=true , address=145.79.227.30:4785 , piece_id=5KMLU4ATBN4YVQDYJKKQY4PEJNKWTO2WNX5GMGJM4F54B6YM2SFA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gYH83ECV8wai3CZyBSU8NNJM1UsysLYHbQDwBVKZRyjsdnUwB): protocol: write tcp 188.166.225.170:58398->145.79.227.30:4785: use of closed network connection; write tcp 188.166.225.170:58398->145.79.227.30:4785: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58398->145.79.227.30:4785: use of closed network connection Dec 26 05:51:22.719260194 (494h22m22.590343229s) [noise=true , address=145.79.227.30:4785 , piece_id=5KMLU4ATBN4YVQDYJKKQY4PEJNKWTO2WNX5GMGJM4F54B6YM2SFA, seg_pos={0 0}, upload=433902]: Long tail cancellation due to optimal threshold Dec 26 05:51:22.719409408 (494h22m22.590492443s) [noise=true , address=145.79.227.40:4052 , piece_id=XRZYL7PT5VHY22N7CCKS3NTZKDDWE36ZU3SORF6ILM7ZMXDND63A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PViJBYAanwXRWhMq3WHR6k9Stxy8hjuY5FV9J7ErxCy4N1q3V): protocol: write tcp 188.166.225.170:45984->145.79.227.40:4052: use of closed network connection; write tcp 188.166.225.170:45984->145.79.227.40:4052: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45984->145.79.227.40:4052: use of closed network connection Dec 26 05:51:22.719415469 (494h22m22.590498515s) [noise=true , address=145.79.227.40:4052 , piece_id=XRZYL7PT5VHY22N7CCKS3NTZKDDWE36ZU3SORF6ILM7ZMXDND63A, seg_pos={0 0}, upload=433902]: Long tail cancellation due to optimal threshold Dec 26 05:51:22.719467057 (494h22m22.590550092s) [noise=true , address=145.79.227.52:4461 , piece_id=ZAAUDBGY6CC2JNEGWXVPCBAWXARHAJDZ3YPB6Y3NWEWCFDUFGDSQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fNrbSfErEQRnruJCJAVvkoNwQSPWnvNtxunHn6zrQegnNAkxA): protocol: write tcp 188.166.225.170:55966->145.79.227.52:4461: use of closed network connection; write tcp 188.166.225.170:55966->145.79.227.52:4461: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55966->145.79.227.52:4461: use of closed network connection Dec 26 05:51:22.719473931 (494h22m22.590556966s) [noise=true , address=145.79.227.52:4461 , piece_id=ZAAUDBGY6CC2JNEGWXVPCBAWXARHAJDZ3YPB6Y3NWEWCFDUFGDSQ, seg_pos={0 0}, upload=433902]: Long tail cancellation due to optimal threshold Dec 26 05:51:22.719491374 (494h22m22.590574409s) [seg_pos={0 0}, upload=433902]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 05:51:22.719562539 (494h22m22.590645574s) [upload=433902]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:51:22.719567228 (494h22m22.590650263s) [upload=433902]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:51:23.008169117 (494h22m22.879252162s) [upload=433902]: Done uploading segments... Dec 26 05:51:23.008191560 (494h22m22.879274605s) [upload=433902]: Upload finished. err: <nil> Duration: 25.641153345s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [.......................X..........X.......X.X.......X.X.XXX.] duration: 5.978925355s trace: https://jaeger.dev.storj.io/trace/6a0a037a6e3db5ee FAIL (2025-12-26 06:35:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA, Piece ID: 35JRG3N62ZFMLVR6NA7FE5AA6I3MCKA7BHJ57DV6Z44BJ2D2UNFQ) context deadline exceeded; piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: ELZBOPVBQICIY36IKSM2CJDZQRY5KZCVCC6ZAWOHKIZZS4U7NALA) context deadline exceeded ======LOGS====== Dec 26 06:35:04.760439678 (495h6m4.631522723s) [upload=434555]: Starting upload Dec 26 06:35:04.760570547 (495h6m4.631653592s) [upload=434555]: Done starting upload Dec 26 06:35:04.761041794 (495h6m4.63212484s) [upload=434555]: Uploading segments... Dec 26 06:35:04.761144149 (495h6m4.632227225s) [upload=434555]: Got next segment. Inline: false Dec 26 06:35:04.761199875 (495h6m4.63228292s) [upload=434555]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 06:35:04.761208602 (495h6m4.632291647s) [upload=434555]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 06:35:05.021427045 (495h6m4.89251009s) [seg_pos={0 0}, upload=434555]: Begin upload segment... Dec 26 06:35:05.021491608 (495h6m4.892574643s) [seg_pos={0 0}, upload=434555]: Done begin upload segment. Dec 26 06:35:05.021597740 (495h6m4.892680785s) [noise=true , address=45.91.116.10:4084 , piece_id=MBC3L2CXNDOKSB6FB3TFCY7ZFQH4T7X5PFQOFRTUB2UICYIQ23YQ, seg_pos={0 0}, upload=434555]: Uploading piece... Dec 26 06:35:05.021610594 (495h6m4.892693639s) [noise=true , address=145.79.227.10:4403 , piece_id=FD3OQ22YVJAELVEYLEFWBDJM5MJGWM23FUJPO25INVZRY7C6MFTA, seg_pos={0 0}, upload=434555]: Uploading piece... Dec 26 06:35:05.022299917 (495h6m4.893382962s) [noise=true , address=45.91.116.7:4010 , piece_id=G7RMU6UYUWHTUGKL5GWVOX46XC74QMTVKAAH6FDJF5UXNA5NRLHA, seg_pos={0 0}, upload=434555]: Uploading piece... Dec 26 06:35:05.022358308 (495h6m4.893441353s) [noise=true , address=145.79.227.33:4870 , piece_id=LI7COKKQZP3YEOFT6LGVMVPIGR3HCAPGFN6D6ZKESSFWW73GEYYA, seg_pos={0 0}, upload=434555]: Uploading piece... Dec 26 06:35:05.023752794 (495h6m4.894835839s) [noise=true , address=45.91.116.12:4228 , piece_id=35JRG3N62ZFMLVR6NA7FE5AA6I3MCKA7BHJ57DV6Z44BJ2D2UNFQ, seg_pos={0 0}, upload=434555]: Uploading piece... Dec 26 06:35:05.024211778 (495h6m4.895294823s) [noise=true , address=77.90.31.13:4627 , piece_id=ELZBOPVBQICIY36IKSM2CJDZQRY5KZCVCC6ZAWOHKIZZS4U7NALA, seg_pos={0 0}, upload=434555]: Uploading piece... Dec 26 06:35:05.024352175 (495h6m4.89543522s) [noise=true , address=77.90.31.19:4844 , piece_id=73UMM5H6IOPVG6RSOIN7AVX3475T5LBNJFGKXZFTWNOQCNPWGNZA, seg_pos={0 0}, upload=434555]: Uploading piece... Dec 26 06:35:05.024663718 (495h6m4.895746763s) [noise=true , address=45.91.116.13:4156 , piece_id=6HSFXWZSELJYCKKAMPTISTNEOGH2A7MKKT6VGGYN4KWZWEASRILA, seg_pos={0 0}, upload=434555]: Uploading piece... Dec 26 06:35:05.025025226 (495h6m4.896108271s) [noise=true , address=145.79.227.30:4762 , piece_id=5JQMRXUGPTJCPPZ5SCTXKKPPC2CBX6QSVK5DELTHNQ5JTJKWKQBA, seg_pos={0 0}, upload=434555]: Uploading piece... Dec 26 06:35:05.025622353 (495h6m4.896705398s) [noise=true , address=45.91.116.11:4118 , piece_id=PS27RIQORRJ2XJDN4OU524QMULFKDU7W7LPA65QBU77QEGERVTGA, seg_pos={0 0}, upload=434555]: Uploading piece... Dec 26 06:35:05.026734841 (495h6m4.897817886s) [noise=true , address=145.79.227.44:4195 , piece_id=DOWS7DCO22WK7CCCZ2P2SCTTHOLMM3M2JQ2SYK45I6JHHTIBWDXQ, seg_pos={0 0}, upload=434555]: Uploading piece... Dec 26 06:35:05.027852760 (495h6m4.898935815s) [noise=true , address=145.79.227.2:4114 , piece_id=CU4U6AAPWJILLRU2INJVGPSS4EEFDLBCMIQ37TCPTI3PXM77LBPA, seg_pos={0 0}, upload=434555]: Uploading piece... Dec 26 06:35:05.028906527 (495h6m4.899989582s) [noise=true , address=145.79.227.16:4911 , piece_id=CEGNELS2ZJ3W5YZM2THV46HP2STQZVYANTZ5COEUDPMYKNQP2MPA, seg_pos={0 0}, upload=434555]: Uploading piece... Dec 26 06:35:05.029388084 (495h6m4.900471119s) [noise=true , address=45.91.116.4:4940 , piece_id=BD5NTSOEAH2J6VFWBCKIEQN7JDHN3B2KRYE4LPMEVVNZNH5B7MHA, seg_pos={0 0}, upload=434555]: Uploading piece... Dec 26 06:35:05.029900950 (495h6m4.900983995s) [noise=true , address=77.90.31.12:4589 , piece_id=UTACPTTZBVMPNELTPD56JU7JH76UG64FLWX7EHF7JZFBAH6ZG4IA, seg_pos={0 0}, upload=434555]: Uploading piece... Dec 26 06:35:05.031513151 (495h6m4.902596196s) [noise=true , address=45.91.116.14:4191 , piece_id=L5HTLQF6VKZEEOGRYRXPUJA2GMQJISTUEDUVEAMGN2R5CDQIYNMQ, seg_pos={0 0}, upload=434555]: Uploading piece... Dec 26 06:35:06.451450055 (495h6m6.32253313s) [noise=true , address=77.90.31.12:4589 , piece_id=UTACPTTZBVMPNELTPD56JU7JH76UG64FLWX7EHF7JZFBAH6ZG4IA, seg_pos={0 0}, upload=434555]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 26 06:35:06.745433702 (495h6m6.616516737s) [upload=434555]: Next returned nil segment Dec 26 06:35:06.745449311 (495h6m6.616532346s) [upload=434555]: Waiting for error group managing segments... Dec 26 06:35:07.033287466 (495h6m6.904370511s) [noise=true , address=145.79.227.2:4114 , piece_id=CU4U6AAPWJILLRU2INJVGPSS4EEFDLBCMIQ37TCPTI3PXM77LBPA, seg_pos={0 0}, upload=434555]: Done uploading piece. err: <nil> Dec 26 06:35:07.035687747 (495h6m6.906770792s) [noise=true , address=145.79.227.16:4911 , piece_id=CEGNELS2ZJ3W5YZM2THV46HP2STQZVYANTZ5COEUDPMYKNQP2MPA, seg_pos={0 0}, upload=434555]: Done uploading piece. err: <nil> Dec 26 06:35:07.039272383 (495h6m6.910355428s) [noise=true , address=145.79.227.10:4403 , piece_id=FD3OQ22YVJAELVEYLEFWBDJM5MJGWM23FUJPO25INVZRY7C6MFTA, seg_pos={0 0}, upload=434555]: Done uploading piece. err: <nil> Dec 26 06:35:07.059443593 (495h6m6.930526638s) [noise=true , address=145.79.227.33:4870 , piece_id=LI7COKKQZP3YEOFT6LGVMVPIGR3HCAPGFN6D6ZKESSFWW73GEYYA, seg_pos={0 0}, upload=434555]: Done uploading piece. err: <nil> Dec 26 06:35:07.226574209 (495h6m7.097657254s) [noise=true , address=77.90.31.19:4844 , piece_id=73UMM5H6IOPVG6RSOIN7AVX3475T5LBNJFGKXZFTWNOQCNPWGNZA, seg_pos={0 0}, upload=434555]: Done uploading piece. err: <nil> Dec 26 06:35:07.267972800 (495h6m7.139055845s) [noise=true , address=45.91.116.14:4191 , piece_id=L5HTLQF6VKZEEOGRYRXPUJA2GMQJISTUEDUVEAMGN2R5CDQIYNMQ, seg_pos={0 0}, upload=434555]: Done uploading piece. err: <nil> Dec 26 06:35:07.270181756 (495h6m7.141264801s) [noise=true , address=45.91.116.7:4010 , piece_id=G7RMU6UYUWHTUGKL5GWVOX46XC74QMTVKAAH6FDJF5UXNA5NRLHA, seg_pos={0 0}, upload=434555]: Done uploading piece. err: <nil> Dec 26 06:35:07.295349880 (495h6m7.166432925s) [noise=true , address=45.91.116.4:4940 , piece_id=BD5NTSOEAH2J6VFWBCKIEQN7JDHN3B2KRYE4LPMEVVNZNH5B7MHA, seg_pos={0 0}, upload=434555]: Done uploading piece. err: <nil> Dec 26 06:35:07.296061265 (495h6m7.1671443s) [noise=true , address=77.90.31.13:4627 , piece_id=ELZBOPVBQICIY36IKSM2CJDZQRY5KZCVCC6ZAWOHKIZZS4U7NALA, seg_pos={0 0}, upload=434555]: Done uploading piece. err: <nil> Dec 26 06:35:07.304767473 (495h6m7.175850508s) [noise=true , address=45.91.116.13:4156 , piece_id=6HSFXWZSELJYCKKAMPTISTNEOGH2A7MKKT6VGGYN4KWZWEASRILA, seg_pos={0 0}, upload=434555]: Done uploading piece. err: <nil> Dec 26 06:35:07.309903673 (495h6m7.180986718s) [noise=true , address=45.91.116.12:4228 , piece_id=35JRG3N62ZFMLVR6NA7FE5AA6I3MCKA7BHJ57DV6Z44BJ2D2UNFQ, seg_pos={0 0}, upload=434555]: Done uploading piece. err: <nil> Dec 26 06:35:07.710009747 (495h6m7.581092792s) [noise=true , address=45.91.116.10:4084 , piece_id=MBC3L2CXNDOKSB6FB3TFCY7ZFQH4T7X5PFQOFRTUB2UICYIQ23YQ, seg_pos={0 0}, upload=434555]: Done uploading piece. err: <nil> Dec 26 06:35:08.344914229 (495h6m8.215997264s) [noise=true , address=145.79.227.30:4762 , piece_id=5JQMRXUGPTJCPPZ5SCTXKKPPC2CBX6QSVK5DELTHNQ5JTJKWKQBA, seg_pos={0 0}, upload=434555]: Done uploading piece. err: <nil> Dec 26 06:35:08.344953844 (495h6m8.21603689s) [seg_pos={0 0}, upload=434555]: Segment reached optimal threshold of 13 pieces. Dec 26 06:35:08.345232144 (495h6m8.216315199s) [noise=true , address=45.91.116.11:4118 , piece_id=PS27RIQORRJ2XJDN4OU524QMULFKDU7W7LPA65QBU77QEGERVTGA, seg_pos={0 0}, upload=434555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: expected piece hash; context canceled; EOF Dec 26 06:35:08.345243626 (495h6m8.216326671s) [noise=true , address=45.91.116.11:4118 , piece_id=PS27RIQORRJ2XJDN4OU524QMULFKDU7W7LPA65QBU77QEGERVTGA, seg_pos={0 0}, upload=434555]: Long tail cancellation due to optimal threshold Dec 26 06:35:08.345263194 (495h6m8.216346239s) [noise=true , address=145.79.227.44:4195 , piece_id=DOWS7DCO22WK7CCCZ2P2SCTTHOLMM3M2JQ2SYK45I6JHHTIBWDXQ, seg_pos={0 0}, upload=434555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 26 06:35:08.345286137 (495h6m8.216369172s) [noise=true , address=145.79.227.44:4195 , piece_id=DOWS7DCO22WK7CCCZ2P2SCTTHOLMM3M2JQ2SYK45I6JHHTIBWDXQ, seg_pos={0 0}, upload=434555]: Long tail cancellation due to optimal threshold Dec 26 06:35:08.345304432 (495h6m8.216387477s) [seg_pos={0 0}, upload=434555]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 06:35:08.345402459 (495h6m8.216485494s) [upload=434555]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 06:35:08.345407779 (495h6m8.216490824s) [upload=434555]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 06:35:08.636899841 (495h6m8.507982896s) [upload=434555]: Done uploading segments... Dec 26 06:35:08.636924358 (495h6m8.508007403s) [upload=434555]: Upload finished. err: <nil> Duration: 5.978956814s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [X.......................X..........X.......X.X.......X.X.XXX] duration: 30.002575512s trace: https://jaeger.dev.storj.io/trace/6a0a037a6e3db5ee FAIL (2025-12-26 06:35:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA, Piece ID: 35JRG3N62ZFMLVR6NA7FE5AA6I3MCKA7BHJ57DV6Z44BJ2D2UNFQ) context deadline exceeded; piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: ELZBOPVBQICIY36IKSM2CJDZQRY5KZCVCC6ZAWOHKIZZS4U7NALA) context deadline exceeded ======LOGS====== Dec 26 06:35:04.760439678 (495h6m4.631522723s) [upload=434555]: Starting upload Dec 26 06:35:04.760570547 (495h6m4.631653592s) [upload=434555]: Done starting upload Dec 26 06:35:04.761041794 (495h6m4.63212484s) [upload=434555]: Uploading segments... Dec 26 06:35:04.761144149 (495h6m4.632227225s) [upload=434555]: Got next segment. Inline: false Dec 26 06:35:04.761199875 (495h6m4.63228292s) [upload=434555]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 06:35:04.761208602 (495h6m4.632291647s) [upload=434555]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 06:35:05.021427045 (495h6m4.89251009s) [seg_pos={0 0}, upload=434555]: Begin upload segment... Dec 26 06:35:05.021491608 (495h6m4.892574643s) [seg_pos={0 0}, upload=434555]: Done begin upload segment. Dec 26 06:35:05.021597740 (495h6m4.892680785s) [noise=true , address=45.91.116.10:4084 , piece_id=MBC3L2CXNDOKSB6FB3TFCY7ZFQH4T7X5PFQOFRTUB2UICYIQ23YQ, seg_pos={0 0}, upload=434555]: Uploading piece... Dec 26 06:35:05.021610594 (495h6m4.892693639s) [noise=true , address=145.79.227.10:4403 , piece_id=FD3OQ22YVJAELVEYLEFWBDJM5MJGWM23FUJPO25INVZRY7C6MFTA, seg_pos={0 0}, upload=434555]: Uploading piece... Dec 26 06:35:05.022299917 (495h6m4.893382962s) [noise=true , address=45.91.116.7:4010 , piece_id=G7RMU6UYUWHTUGKL5GWVOX46XC74QMTVKAAH6FDJF5UXNA5NRLHA, seg_pos={0 0}, upload=434555]: Uploading piece... Dec 26 06:35:05.022358308 (495h6m4.893441353s) [noise=true , address=145.79.227.33:4870 , piece_id=LI7COKKQZP3YEOFT6LGVMVPIGR3HCAPGFN6D6ZKESSFWW73GEYYA, seg_pos={0 0}, upload=434555]: Uploading piece... Dec 26 06:35:05.023752794 (495h6m4.894835839s) [noise=true , address=45.91.116.12:4228 , piece_id=35JRG3N62ZFMLVR6NA7FE5AA6I3MCKA7BHJ57DV6Z44BJ2D2UNFQ, seg_pos={0 0}, upload=434555]: Uploading piece... Dec 26 06:35:05.024211778 (495h6m4.895294823s) [noise=true , address=77.90.31.13:4627 , piece_id=ELZBOPVBQICIY36IKSM2CJDZQRY5KZCVCC6ZAWOHKIZZS4U7NALA, seg_pos={0 0}, upload=434555]: Uploading piece... Dec 26 06:35:05.024352175 (495h6m4.89543522s) [noise=true , address=77.90.31.19:4844 , piece_id=73UMM5H6IOPVG6RSOIN7AVX3475T5LBNJFGKXZFTWNOQCNPWGNZA, seg_pos={0 0}, upload=434555]: Uploading piece... Dec 26 06:35:05.024663718 (495h6m4.895746763s) [noise=true , address=45.91.116.13:4156 , piece_id=6HSFXWZSELJYCKKAMPTISTNEOGH2A7MKKT6VGGYN4KWZWEASRILA, seg_pos={0 0}, upload=434555]: Uploading piece... Dec 26 06:35:05.025025226 (495h6m4.896108271s) [noise=true , address=145.79.227.30:4762 , piece_id=5JQMRXUGPTJCPPZ5SCTXKKPPC2CBX6QSVK5DELTHNQ5JTJKWKQBA, seg_pos={0 0}, upload=434555]: Uploading piece... Dec 26 06:35:05.025622353 (495h6m4.896705398s) [noise=true , address=45.91.116.11:4118 , piece_id=PS27RIQORRJ2XJDN4OU524QMULFKDU7W7LPA65QBU77QEGERVTGA, seg_pos={0 0}, upload=434555]: Uploading piece... Dec 26 06:35:05.026734841 (495h6m4.897817886s) [noise=true , address=145.79.227.44:4195 , piece_id=DOWS7DCO22WK7CCCZ2P2SCTTHOLMM3M2JQ2SYK45I6JHHTIBWDXQ, seg_pos={0 0}, upload=434555]: Uploading piece... Dec 26 06:35:05.027852760 (495h6m4.898935815s) [noise=true , address=145.79.227.2:4114 , piece_id=CU4U6AAPWJILLRU2INJVGPSS4EEFDLBCMIQ37TCPTI3PXM77LBPA, seg_pos={0 0}, upload=434555]: Uploading piece... Dec 26 06:35:05.028906527 (495h6m4.899989582s) [noise=true , address=145.79.227.16:4911 , piece_id=CEGNELS2ZJ3W5YZM2THV46HP2STQZVYANTZ5COEUDPMYKNQP2MPA, seg_pos={0 0}, upload=434555]: Uploading piece... Dec 26 06:35:05.029388084 (495h6m4.900471119s) [noise=true , address=45.91.116.4:4940 , piece_id=BD5NTSOEAH2J6VFWBCKIEQN7JDHN3B2KRYE4LPMEVVNZNH5B7MHA, seg_pos={0 0}, upload=434555]: Uploading piece... Dec 26 06:35:05.029900950 (495h6m4.900983995s) [noise=true , address=77.90.31.12:4589 , piece_id=UTACPTTZBVMPNELTPD56JU7JH76UG64FLWX7EHF7JZFBAH6ZG4IA, seg_pos={0 0}, upload=434555]: Uploading piece... Dec 26 06:35:05.031513151 (495h6m4.902596196s) [noise=true , address=45.91.116.14:4191 , piece_id=L5HTLQF6VKZEEOGRYRXPUJA2GMQJISTUEDUVEAMGN2R5CDQIYNMQ, seg_pos={0 0}, upload=434555]: Uploading piece... Dec 26 06:35:06.451450055 (495h6m6.32253313s) [noise=true , address=77.90.31.12:4589 , piece_id=UTACPTTZBVMPNELTPD56JU7JH76UG64FLWX7EHF7JZFBAH6ZG4IA, seg_pos={0 0}, upload=434555]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 26 06:35:06.745433702 (495h6m6.616516737s) [upload=434555]: Next returned nil segment Dec 26 06:35:06.745449311 (495h6m6.616532346s) [upload=434555]: Waiting for error group managing segments... Dec 26 06:35:07.033287466 (495h6m6.904370511s) [noise=true , address=145.79.227.2:4114 , piece_id=CU4U6AAPWJILLRU2INJVGPSS4EEFDLBCMIQ37TCPTI3PXM77LBPA, seg_pos={0 0}, upload=434555]: Done uploading piece. err: <nil> Dec 26 06:35:07.035687747 (495h6m6.906770792s) [noise=true , address=145.79.227.16:4911 , piece_id=CEGNELS2ZJ3W5YZM2THV46HP2STQZVYANTZ5COEUDPMYKNQP2MPA, seg_pos={0 0}, upload=434555]: Done uploading piece. err: <nil> Dec 26 06:35:07.039272383 (495h6m6.910355428s) [noise=true , address=145.79.227.10:4403 , piece_id=FD3OQ22YVJAELVEYLEFWBDJM5MJGWM23FUJPO25INVZRY7C6MFTA, seg_pos={0 0}, upload=434555]: Done uploading piece. err: <nil> Dec 26 06:35:07.059443593 (495h6m6.930526638s) [noise=true , address=145.79.227.33:4870 , piece_id=LI7COKKQZP3YEOFT6LGVMVPIGR3HCAPGFN6D6ZKESSFWW73GEYYA, seg_pos={0 0}, upload=434555]: Done uploading piece. err: <nil> Dec 26 06:35:07.226574209 (495h6m7.097657254s) [noise=true , address=77.90.31.19:4844 , piece_id=73UMM5H6IOPVG6RSOIN7AVX3475T5LBNJFGKXZFTWNOQCNPWGNZA, seg_pos={0 0}, upload=434555]: Done uploading piece. err: <nil> Dec 26 06:35:07.267972800 (495h6m7.139055845s) [noise=true , address=45.91.116.14:4191 , piece_id=L5HTLQF6VKZEEOGRYRXPUJA2GMQJISTUEDUVEAMGN2R5CDQIYNMQ, seg_pos={0 0}, upload=434555]: Done uploading piece. err: <nil> Dec 26 06:35:07.270181756 (495h6m7.141264801s) [noise=true , address=45.91.116.7:4010 , piece_id=G7RMU6UYUWHTUGKL5GWVOX46XC74QMTVKAAH6FDJF5UXNA5NRLHA, seg_pos={0 0}, upload=434555]: Done uploading piece. err: <nil> Dec 26 06:35:07.295349880 (495h6m7.166432925s) [noise=true , address=45.91.116.4:4940 , piece_id=BD5NTSOEAH2J6VFWBCKIEQN7JDHN3B2KRYE4LPMEVVNZNH5B7MHA, seg_pos={0 0}, upload=434555]: Done uploading piece. err: <nil> Dec 26 06:35:07.296061265 (495h6m7.1671443s) [noise=true , address=77.90.31.13:4627 , piece_id=ELZBOPVBQICIY36IKSM2CJDZQRY5KZCVCC6ZAWOHKIZZS4U7NALA, seg_pos={0 0}, upload=434555]: Done uploading piece. err: <nil> Dec 26 06:35:07.304767473 (495h6m7.175850508s) [noise=true , address=45.91.116.13:4156 , piece_id=6HSFXWZSELJYCKKAMPTISTNEOGH2A7MKKT6VGGYN4KWZWEASRILA, seg_pos={0 0}, upload=434555]: Done uploading piece. err: <nil> Dec 26 06:35:07.309903673 (495h6m7.180986718s) [noise=true , address=45.91.116.12:4228 , piece_id=35JRG3N62ZFMLVR6NA7FE5AA6I3MCKA7BHJ57DV6Z44BJ2D2UNFQ, seg_pos={0 0}, upload=434555]: Done uploading piece. err: <nil> Dec 26 06:35:07.710009747 (495h6m7.581092792s) [noise=true , address=45.91.116.10:4084 , piece_id=MBC3L2CXNDOKSB6FB3TFCY7ZFQH4T7X5PFQOFRTUB2UICYIQ23YQ, seg_pos={0 0}, upload=434555]: Done uploading piece. err: <nil> Dec 26 06:35:08.344914229 (495h6m8.215997264s) [noise=true , address=145.79.227.30:4762 , piece_id=5JQMRXUGPTJCPPZ5SCTXKKPPC2CBX6QSVK5DELTHNQ5JTJKWKQBA, seg_pos={0 0}, upload=434555]: Done uploading piece. err: <nil> Dec 26 06:35:08.344953844 (495h6m8.21603689s) [seg_pos={0 0}, upload=434555]: Segment reached optimal threshold of 13 pieces. Dec 26 06:35:08.345232144 (495h6m8.216315199s) [noise=true , address=45.91.116.11:4118 , piece_id=PS27RIQORRJ2XJDN4OU524QMULFKDU7W7LPA65QBU77QEGERVTGA, seg_pos={0 0}, upload=434555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: expected piece hash; context canceled; EOF Dec 26 06:35:08.345243626 (495h6m8.216326671s) [noise=true , address=45.91.116.11:4118 , piece_id=PS27RIQORRJ2XJDN4OU524QMULFKDU7W7LPA65QBU77QEGERVTGA, seg_pos={0 0}, upload=434555]: Long tail cancellation due to optimal threshold Dec 26 06:35:08.345263194 (495h6m8.216346239s) [noise=true , address=145.79.227.44:4195 , piece_id=DOWS7DCO22WK7CCCZ2P2SCTTHOLMM3M2JQ2SYK45I6JHHTIBWDXQ, seg_pos={0 0}, upload=434555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 26 06:35:08.345286137 (495h6m8.216369172s) [noise=true , address=145.79.227.44:4195 , piece_id=DOWS7DCO22WK7CCCZ2P2SCTTHOLMM3M2JQ2SYK45I6JHHTIBWDXQ, seg_pos={0 0}, upload=434555]: Long tail cancellation due to optimal threshold Dec 26 06:35:08.345304432 (495h6m8.216387477s) [seg_pos={0 0}, upload=434555]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 06:35:08.345402459 (495h6m8.216485494s) [upload=434555]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 06:35:08.345407779 (495h6m8.216490824s) [upload=434555]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 06:35:08.636899841 (495h6m8.507982896s) [upload=434555]: Done uploading segments... Dec 26 06:35:08.636924358 (495h6m8.508007403s) [upload=434555]: Upload finished. err: <nil> Duration: 30.002630085s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 30.595842882s trace: https://jaeger.dev.storj.io/trace/7e198d4c8fea54d2 FAIL (2025-12-26 06:35:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p, Piece ID: SBCEX7UNWGNFF54SBP4AGRDV5IPG24FQJXJJ7EYQTBSZKOVZWU4Q) context deadline exceeded; piecestore: (Node ID: 1nP8xospdMuDEh3em5K6iBnh8bHXqLhvHy4pHm9m8Lnj4kEH2X, Piece ID: XV7FQGBJNWVRLOAFRUPOWON35JR5ESRJPTCWPUGYWGHJPSDVL4GA) context deadline exceeded; piecestore: (Node ID: 1Mg5mxUjW56quDidSgA788D7ByfAzvsEuWJqwtzY4bLyxRn3mu, Piece ID: ATPIXLMXGONSSLXPPVORSNC6LHPXI2V3UFTHM4VHA2FTY5ZM6H3A) context deadline exceeded; piecestore: (Node ID: 12QmYSEAKnkSLEEVc9E1uxtVigunKMFLk2cdu71Bse8Tqec3HyT, Piece ID: YFKD7GO2FOSPLDXNFPK732QTLCLBQQASWBTMZ2LTLBGSK22PIXIQ) context deadline exceeded; piecestore: (Node ID: 1JsnHuhQhM4rWLTLEuSSHLNE24m7tSW9qMSf48j7yMrAVPWVyV, Piece ID: ARBMDAKX4QBW6JG47KP2PUITCX7HKXCBO75THQPINM4LLHCKAV4Q) context deadline exceeded ======LOGS====== upload=433902]: Uploading piece... Dec 26 05:50:34.986155265 (494h21m34.8572383s) [noise=true , address=64.49.13.140:4509 , piece_id=6TGWTOAVY4YVAKU6VEE2SWVAM264TKU6ZNMGPWH4TNGII3UYXFXQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.986703610 (494h21m34.857786655s) [noise=true , address=77.90.31.10:4522 , piece_id=YFKD7GO2FOSPLDXNFPK732QTLCLBQQASWBTMZ2LTLBGSK22PIXIQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.987145380 (494h21m34.858228425s) [noise=true , address=145.79.227.33:4890 , piece_id=UKNB4ZZTYJGY4MDHOBPYUTHCP3ZDCMJB7O7AJZFYP2HFVIWLUOKQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.987628290 (494h21m34.858711335s) [noise=true , address=145.79.227.40:4052 , piece_id=XRZYL7PT5VHY22N7CCKS3NTZKDDWE36ZU3SORF6ILM7ZMXDND63A, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.988710731 (494h21m34.859793776s) [noise=true , address=77.90.31.13:4653 , piece_id=EY2CWX54ONOWPO3WC33G5QSP5RMP2SUTBQLUOH5JXGDRUEPDHQ2A, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.989894545 (494h21m34.86097759s) [noise=true , address=145.79.227.30:4785 , piece_id=5KMLU4ATBN4YVQDYJKKQY4PEJNKWTO2WNX5GMGJM4F54B6YM2SFA, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.991305081 (494h21m34.862388116s) [noise=true , address=145.79.227.50:4381 , piece_id=H6Q47DIOTOWGN7QH2KWT4IH6KVDSB246S22753Z3UQGB26T6VXPA, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.995796263 (494h21m34.866879359s) [noise=true , address=145.79.227.52:4461 , piece_id=ZAAUDBGY6CC2JNEGWXVPCBAWXARHAJDZ3YPB6Y3NWEWCFDUFGDSQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.996765439 (494h21m34.867848484s) [noise=true , address=77.90.31.16:4755 , piece_id=ARBMDAKX4QBW6JG47KP2PUITCX7HKXCBO75THQPINM4LLHCKAV4Q, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.998107634 (494h21m34.869190679s) [noise=true , address=77.90.31.11:4579 , piece_id=SBCEX7UNWGNFF54SBP4AGRDV5IPG24FQJXJJ7EYQTBSZKOVZWU4Q, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.999340442 (494h21m34.870423477s) [noise=true , address=145.79.227.34:4936 , piece_id=EEA2ZPM44DX3WMI7ZJTDC7T5UIGED6TNT53QXZWGBFRMQUQSCXPQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:35.000540757 (494h21m34.871623812s) [noise=true , address=45.91.116.14:4204 , piece_id=XV7FQGBJNWVRLOAFRUPOWON35JR5ESRJPTCWPUGYWGHJPSDVL4GA, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:36.360077064 (494h21m36.231160099s) [upload=433902]: Next returned nil segment Dec 26 05:50:36.360091642 (494h21m36.231174688s) [upload=433902]: Waiting for error group managing segments... Dec 26 05:50:36.688914299 (494h21m36.559997344s) [noise=true , address=64.49.13.140:4509 , piece_id=6TGWTOAVY4YVAKU6VEE2SWVAM264TKU6ZNMGPWH4TNGII3UYXFXQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.059812634 (494h21m36.930895669s) [noise=true , address=77.90.31.11:4579 , piece_id=SBCEX7UNWGNFF54SBP4AGRDV5IPG24FQJXJJ7EYQTBSZKOVZWU4Q, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.119433874 (494h21m36.990516909s) [noise=true , address=45.91.116.14:4204 , piece_id=XV7FQGBJNWVRLOAFRUPOWON35JR5ESRJPTCWPUGYWGHJPSDVL4GA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.138718204 (494h21m37.00980125s) [noise=true , address=77.90.31.16:4755 , piece_id=ARBMDAKX4QBW6JG47KP2PUITCX7HKXCBO75THQPINM4LLHCKAV4Q, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.146647243 (494h21m37.017730278s) [noise=true , address=77.90.31.13:4653 , piece_id=EY2CWX54ONOWPO3WC33G5QSP5RMP2SUTBQLUOH5JXGDRUEPDHQ2A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.163283722 (494h21m37.034366757s) [noise=true , address=77.90.31.21:4893 , piece_id=GNEEZO2XGOKNODOUPZQ4NGKFE4PCX5LNUWXLV27F6X7GWZVSYNAA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.374956574 (494h21m37.246039609s) [noise=true , address=45.91.116.7:4033 , piece_id=ATPIXLMXGONSSLXPPVORSNC6LHPXI2V3UFTHM4VHA2FTY5ZM6H3A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.777283355 (494h21m37.64836639s) [noise=true , address=77.90.31.12:4622 , piece_id=LTYYPK6ZQ3NXZPBKSV4OMZGCOXXV3N6L5A6MVODNR644D2ZUJX3A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:38.439025909 (494h21m38.310108944s) [noise=true , address=145.79.227.49:4355 , piece_id=IUDOHR73Y6OKFAQSNNSONALWAITRJ4POYRO2QD5LYZKOSTWY4MNA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:38.439599111 (494h21m38.310682166s) [noise=true , address=77.90.31.10:4522 , piece_id=YFKD7GO2FOSPLDXNFPK732QTLCLBQQASWBTMZ2LTLBGSK22PIXIQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:39.259154980 (494h21m39.130238015s) [noise=true , address=145.79.227.34:4936 , piece_id=EEA2ZPM44DX3WMI7ZJTDC7T5UIGED6TNT53QXZWGBFRMQUQSCXPQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:39.430935921 (494h21m39.302018956s) [noise=true , address=145.79.227.44:4212 , piece_id=ABBNLKRE4FM7RHF56V4IAQPKBCCCQLLOMFVMSJEWQUNFGGCI3PVQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:42.941071297 (494h21m42.812154352s) [noise=true , address=77.90.31.14:4675 , piece_id=FKB4O45OYICMELPGXB65QAUD2DRIFTSXZFKIA6TY26VFAP6AVY3A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:51:22.719026850 (494h22m22.590109885s) [noise=true , address=145.79.227.33:4890 , piece_id=UKNB4ZZTYJGY4MDHOBPYUTHCP3ZDCMJB7O7AJZFYP2HFVIWLUOKQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:51:22.719060193 (494h22m22.590143228s) [seg_pos={0 0}, upload=433902]: Segment reached optimal threshold of 14 pieces. Dec 26 05:51:22.719233022 (494h22m22.590316067s) [noise=true , address=145.79.227.50:4381 , piece_id=H6Q47DIOTOWGN7QH2KWT4IH6KVDSB246S22753Z3UQGB26T6VXPA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zU35MWWaaC1npkP5qGcWBXAkfxGsd5UsdkM2nwX9x7wUGCDbY): protocol: write tcp 188.166.225.170:46970->145.79.227.50:4381: use of closed network connection; write tcp 188.166.225.170:46970->145.79.227.50:4381: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46970->145.79.227.50:4381: use of closed network connection Dec 26 05:51:22.719238783 (494h22m22.590321828s) [noise=true , address=145.79.227.50:4381 , piece_id=H6Q47DIOTOWGN7QH2KWT4IH6KVDSB246S22753Z3UQGB26T6VXPA, seg_pos={0 0}, upload=433902]: Long tail cancellation due to optimal threshold Dec 26 05:51:22.719251677 (494h22m22.590334712s) [noise=true , address=145.79.227.30:4785 , piece_id=5KMLU4ATBN4YVQDYJKKQY4PEJNKWTO2WNX5GMGJM4F54B6YM2SFA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gYH83ECV8wai3CZyBSU8NNJM1UsysLYHbQDwBVKZRyjsdnUwB): protocol: write tcp 188.166.225.170:58398->145.79.227.30:4785: use of closed network connection; write tcp 188.166.225.170:58398->145.79.227.30:4785: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58398->145.79.227.30:4785: use of closed network connection Dec 26 05:51:22.719260194 (494h22m22.590343229s) [noise=true , address=145.79.227.30:4785 , piece_id=5KMLU4ATBN4YVQDYJKKQY4PEJNKWTO2WNX5GMGJM4F54B6YM2SFA, seg_pos={0 0}, upload=433902]: Long tail cancellation due to optimal threshold Dec 26 05:51:22.719409408 (494h22m22.590492443s) [noise=true , address=145.79.227.40:4052 , piece_id=XRZYL7PT5VHY22N7CCKS3NTZKDDWE36ZU3SORF6ILM7ZMXDND63A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PViJBYAanwXRWhMq3WHR6k9Stxy8hjuY5FV9J7ErxCy4N1q3V): protocol: write tcp 188.166.225.170:45984->145.79.227.40:4052: use of closed network connection; write tcp 188.166.225.170:45984->145.79.227.40:4052: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45984->145.79.227.40:4052: use of closed network connection Dec 26 05:51:22.719415469 (494h22m22.590498515s) [noise=true , address=145.79.227.40:4052 , piece_id=XRZYL7PT5VHY22N7CCKS3NTZKDDWE36ZU3SORF6ILM7ZMXDND63A, seg_pos={0 0}, upload=433902]: Long tail cancellation due to optimal threshold Dec 26 05:51:22.719467057 (494h22m22.590550092s) [noise=true , address=145.79.227.52:4461 , piece_id=ZAAUDBGY6CC2JNEGWXVPCBAWXARHAJDZ3YPB6Y3NWEWCFDUFGDSQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fNrbSfErEQRnruJCJAVvkoNwQSPWnvNtxunHn6zrQegnNAkxA): protocol: write tcp 188.166.225.170:55966->145.79.227.52:4461: use of closed network connection; write tcp 188.166.225.170:55966->145.79.227.52:4461: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55966->145.79.227.52:4461: use of closed network connection Dec 26 05:51:22.719473931 (494h22m22.590556966s) [noise=true , address=145.79.227.52:4461 , piece_id=ZAAUDBGY6CC2JNEGWXVPCBAWXARHAJDZ3YPB6Y3NWEWCFDUFGDSQ, seg_pos={0 0}, upload=433902]: Long tail cancellation due to optimal threshold Dec 26 05:51:22.719491374 (494h22m22.590574409s) [seg_pos={0 0}, upload=433902]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 05:51:22.719562539 (494h22m22.590645574s) [upload=433902]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:51:22.719567228 (494h22m22.590650263s) [upload=433902]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:51:23.008169117 (494h22m22.879252162s) [upload=433902]: Done uploading segments... Dec 26 05:51:23.008191560 (494h22m22.879274605s) [upload=433902]: Upload finished. err: <nil> Duration: 30.59587918s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........................X................................] duration: 3.455899903s trace: https://jaeger.dev.storj.io/trace/630094597d401477 FAIL (2025-12-26 06:34:43 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.455923458s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 30.783074238s trace: https://jaeger.dev.storj.io/trace/7e198d4c8fea54d2 FAIL (2025-12-26 06:34:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p, Piece ID: SBCEX7UNWGNFF54SBP4AGRDV5IPG24FQJXJJ7EYQTBSZKOVZWU4Q) context deadline exceeded; piecestore: (Node ID: 1nP8xospdMuDEh3em5K6iBnh8bHXqLhvHy4pHm9m8Lnj4kEH2X, Piece ID: XV7FQGBJNWVRLOAFRUPOWON35JR5ESRJPTCWPUGYWGHJPSDVL4GA) context deadline exceeded; piecestore: (Node ID: 1Mg5mxUjW56quDidSgA788D7ByfAzvsEuWJqwtzY4bLyxRn3mu, Piece ID: ATPIXLMXGONSSLXPPVORSNC6LHPXI2V3UFTHM4VHA2FTY5ZM6H3A) context deadline exceeded; piecestore: (Node ID: 12QmYSEAKnkSLEEVc9E1uxtVigunKMFLk2cdu71Bse8Tqec3HyT, Piece ID: YFKD7GO2FOSPLDXNFPK732QTLCLBQQASWBTMZ2LTLBGSK22PIXIQ) context deadline exceeded; piecestore: (Node ID: 1JsnHuhQhM4rWLTLEuSSHLNE24m7tSW9qMSf48j7yMrAVPWVyV, Piece ID: ARBMDAKX4QBW6JG47KP2PUITCX7HKXCBO75THQPINM4LLHCKAV4Q) context deadline exceeded ======LOGS====== upload=433902]: Uploading piece... Dec 26 05:50:34.986155265 (494h21m34.8572383s) [noise=true , address=64.49.13.140:4509 , piece_id=6TGWTOAVY4YVAKU6VEE2SWVAM264TKU6ZNMGPWH4TNGII3UYXFXQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.986703610 (494h21m34.857786655s) [noise=true , address=77.90.31.10:4522 , piece_id=YFKD7GO2FOSPLDXNFPK732QTLCLBQQASWBTMZ2LTLBGSK22PIXIQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.987145380 (494h21m34.858228425s) [noise=true , address=145.79.227.33:4890 , piece_id=UKNB4ZZTYJGY4MDHOBPYUTHCP3ZDCMJB7O7AJZFYP2HFVIWLUOKQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.987628290 (494h21m34.858711335s) [noise=true , address=145.79.227.40:4052 , piece_id=XRZYL7PT5VHY22N7CCKS3NTZKDDWE36ZU3SORF6ILM7ZMXDND63A, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.988710731 (494h21m34.859793776s) [noise=true , address=77.90.31.13:4653 , piece_id=EY2CWX54ONOWPO3WC33G5QSP5RMP2SUTBQLUOH5JXGDRUEPDHQ2A, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.989894545 (494h21m34.86097759s) [noise=true , address=145.79.227.30:4785 , piece_id=5KMLU4ATBN4YVQDYJKKQY4PEJNKWTO2WNX5GMGJM4F54B6YM2SFA, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.991305081 (494h21m34.862388116s) [noise=true , address=145.79.227.50:4381 , piece_id=H6Q47DIOTOWGN7QH2KWT4IH6KVDSB246S22753Z3UQGB26T6VXPA, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.995796263 (494h21m34.866879359s) [noise=true , address=145.79.227.52:4461 , piece_id=ZAAUDBGY6CC2JNEGWXVPCBAWXARHAJDZ3YPB6Y3NWEWCFDUFGDSQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.996765439 (494h21m34.867848484s) [noise=true , address=77.90.31.16:4755 , piece_id=ARBMDAKX4QBW6JG47KP2PUITCX7HKXCBO75THQPINM4LLHCKAV4Q, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.998107634 (494h21m34.869190679s) [noise=true , address=77.90.31.11:4579 , piece_id=SBCEX7UNWGNFF54SBP4AGRDV5IPG24FQJXJJ7EYQTBSZKOVZWU4Q, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.999340442 (494h21m34.870423477s) [noise=true , address=145.79.227.34:4936 , piece_id=EEA2ZPM44DX3WMI7ZJTDC7T5UIGED6TNT53QXZWGBFRMQUQSCXPQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:35.000540757 (494h21m34.871623812s) [noise=true , address=45.91.116.14:4204 , piece_id=XV7FQGBJNWVRLOAFRUPOWON35JR5ESRJPTCWPUGYWGHJPSDVL4GA, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:36.360077064 (494h21m36.231160099s) [upload=433902]: Next returned nil segment Dec 26 05:50:36.360091642 (494h21m36.231174688s) [upload=433902]: Waiting for error group managing segments... Dec 26 05:50:36.688914299 (494h21m36.559997344s) [noise=true , address=64.49.13.140:4509 , piece_id=6TGWTOAVY4YVAKU6VEE2SWVAM264TKU6ZNMGPWH4TNGII3UYXFXQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.059812634 (494h21m36.930895669s) [noise=true , address=77.90.31.11:4579 , piece_id=SBCEX7UNWGNFF54SBP4AGRDV5IPG24FQJXJJ7EYQTBSZKOVZWU4Q, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.119433874 (494h21m36.990516909s) [noise=true , address=45.91.116.14:4204 , piece_id=XV7FQGBJNWVRLOAFRUPOWON35JR5ESRJPTCWPUGYWGHJPSDVL4GA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.138718204 (494h21m37.00980125s) [noise=true , address=77.90.31.16:4755 , piece_id=ARBMDAKX4QBW6JG47KP2PUITCX7HKXCBO75THQPINM4LLHCKAV4Q, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.146647243 (494h21m37.017730278s) [noise=true , address=77.90.31.13:4653 , piece_id=EY2CWX54ONOWPO3WC33G5QSP5RMP2SUTBQLUOH5JXGDRUEPDHQ2A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.163283722 (494h21m37.034366757s) [noise=true , address=77.90.31.21:4893 , piece_id=GNEEZO2XGOKNODOUPZQ4NGKFE4PCX5LNUWXLV27F6X7GWZVSYNAA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.374956574 (494h21m37.246039609s) [noise=true , address=45.91.116.7:4033 , piece_id=ATPIXLMXGONSSLXPPVORSNC6LHPXI2V3UFTHM4VHA2FTY5ZM6H3A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.777283355 (494h21m37.64836639s) [noise=true , address=77.90.31.12:4622 , piece_id=LTYYPK6ZQ3NXZPBKSV4OMZGCOXXV3N6L5A6MVODNR644D2ZUJX3A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:38.439025909 (494h21m38.310108944s) [noise=true , address=145.79.227.49:4355 , piece_id=IUDOHR73Y6OKFAQSNNSONALWAITRJ4POYRO2QD5LYZKOSTWY4MNA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:38.439599111 (494h21m38.310682166s) [noise=true , address=77.90.31.10:4522 , piece_id=YFKD7GO2FOSPLDXNFPK732QTLCLBQQASWBTMZ2LTLBGSK22PIXIQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:39.259154980 (494h21m39.130238015s) [noise=true , address=145.79.227.34:4936 , piece_id=EEA2ZPM44DX3WMI7ZJTDC7T5UIGED6TNT53QXZWGBFRMQUQSCXPQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:39.430935921 (494h21m39.302018956s) [noise=true , address=145.79.227.44:4212 , piece_id=ABBNLKRE4FM7RHF56V4IAQPKBCCCQLLOMFVMSJEWQUNFGGCI3PVQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:42.941071297 (494h21m42.812154352s) [noise=true , address=77.90.31.14:4675 , piece_id=FKB4O45OYICMELPGXB65QAUD2DRIFTSXZFKIA6TY26VFAP6AVY3A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:51:22.719026850 (494h22m22.590109885s) [noise=true , address=145.79.227.33:4890 , piece_id=UKNB4ZZTYJGY4MDHOBPYUTHCP3ZDCMJB7O7AJZFYP2HFVIWLUOKQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:51:22.719060193 (494h22m22.590143228s) [seg_pos={0 0}, upload=433902]: Segment reached optimal threshold of 14 pieces. Dec 26 05:51:22.719233022 (494h22m22.590316067s) [noise=true , address=145.79.227.50:4381 , piece_id=H6Q47DIOTOWGN7QH2KWT4IH6KVDSB246S22753Z3UQGB26T6VXPA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zU35MWWaaC1npkP5qGcWBXAkfxGsd5UsdkM2nwX9x7wUGCDbY): protocol: write tcp 188.166.225.170:46970->145.79.227.50:4381: use of closed network connection; write tcp 188.166.225.170:46970->145.79.227.50:4381: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46970->145.79.227.50:4381: use of closed network connection Dec 26 05:51:22.719238783 (494h22m22.590321828s) [noise=true , address=145.79.227.50:4381 , piece_id=H6Q47DIOTOWGN7QH2KWT4IH6KVDSB246S22753Z3UQGB26T6VXPA, seg_pos={0 0}, upload=433902]: Long tail cancellation due to optimal threshold Dec 26 05:51:22.719251677 (494h22m22.590334712s) [noise=true , address=145.79.227.30:4785 , piece_id=5KMLU4ATBN4YVQDYJKKQY4PEJNKWTO2WNX5GMGJM4F54B6YM2SFA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gYH83ECV8wai3CZyBSU8NNJM1UsysLYHbQDwBVKZRyjsdnUwB): protocol: write tcp 188.166.225.170:58398->145.79.227.30:4785: use of closed network connection; write tcp 188.166.225.170:58398->145.79.227.30:4785: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58398->145.79.227.30:4785: use of closed network connection Dec 26 05:51:22.719260194 (494h22m22.590343229s) [noise=true , address=145.79.227.30:4785 , piece_id=5KMLU4ATBN4YVQDYJKKQY4PEJNKWTO2WNX5GMGJM4F54B6YM2SFA, seg_pos={0 0}, upload=433902]: Long tail cancellation due to optimal threshold Dec 26 05:51:22.719409408 (494h22m22.590492443s) [noise=true , address=145.79.227.40:4052 , piece_id=XRZYL7PT5VHY22N7CCKS3NTZKDDWE36ZU3SORF6ILM7ZMXDND63A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PViJBYAanwXRWhMq3WHR6k9Stxy8hjuY5FV9J7ErxCy4N1q3V): protocol: write tcp 188.166.225.170:45984->145.79.227.40:4052: use of closed network connection; write tcp 188.166.225.170:45984->145.79.227.40:4052: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45984->145.79.227.40:4052: use of closed network connection Dec 26 05:51:22.719415469 (494h22m22.590498515s) [noise=true , address=145.79.227.40:4052 , piece_id=XRZYL7PT5VHY22N7CCKS3NTZKDDWE36ZU3SORF6ILM7ZMXDND63A, seg_pos={0 0}, upload=433902]: Long tail cancellation due to optimal threshold Dec 26 05:51:22.719467057 (494h22m22.590550092s) [noise=true , address=145.79.227.52:4461 , piece_id=ZAAUDBGY6CC2JNEGWXVPCBAWXARHAJDZ3YPB6Y3NWEWCFDUFGDSQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fNrbSfErEQRnruJCJAVvkoNwQSPWnvNtxunHn6zrQegnNAkxA): protocol: write tcp 188.166.225.170:55966->145.79.227.52:4461: use of closed network connection; write tcp 188.166.225.170:55966->145.79.227.52:4461: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55966->145.79.227.52:4461: use of closed network connection Dec 26 05:51:22.719473931 (494h22m22.590556966s) [noise=true , address=145.79.227.52:4461 , piece_id=ZAAUDBGY6CC2JNEGWXVPCBAWXARHAJDZ3YPB6Y3NWEWCFDUFGDSQ, seg_pos={0 0}, upload=433902]: Long tail cancellation due to optimal threshold Dec 26 05:51:22.719491374 (494h22m22.590574409s) [seg_pos={0 0}, upload=433902]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 05:51:22.719562539 (494h22m22.590645574s) [upload=433902]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:51:22.719567228 (494h22m22.590650263s) [upload=433902]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:51:23.008169117 (494h22m22.879252162s) [upload=433902]: Done uploading segments... Dec 26 05:51:23.008191560 (494h22m22.879274605s) [upload=433902]: Upload finished. err: <nil> Duration: 30.783098083s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [.X.......................X..........X.......X.X.......X.X.XX] duration: 30.002132134s trace: https://jaeger.dev.storj.io/trace/55c6e26a832cd8a1 FAIL (2025-12-26 06:34:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: E534ZX4O5DPGROTSSYSO7Y6HMQZBKEC6SUIPG47DGL5ZHHBQOXNA) context deadline exceeded; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: ER4WUOMPIHQPCK2DNEOIL2KS6F4G4AZVHUFXOLZUW35F4VDIOXVQ) context deadline exceeded ======LOGS====== Dec 26 06:33:53.016580725 (495h4m52.88766376s) [upload=434536]: Starting upload Dec 26 06:33:53.016645288 (495h4m52.887728323s) [upload=434536]: Done starting upload Dec 26 06:33:53.016878282 (495h4m52.887961327s) [upload=434536]: Uploading segments... Dec 26 06:33:53.017048286 (495h4m52.888131331s) [upload=434536]: Got next segment. Inline: false Dec 26 06:33:53.017107208 (495h4m52.888190243s) [upload=434536]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 06:33:53.017115543 (495h4m52.888198578s) [upload=434536]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 06:33:53.263867924 (495h4m53.134950969s) [seg_pos={0 0}, upload=434536]: Begin upload segment... Dec 26 06:33:53.263920705 (495h4m53.13500375s) [seg_pos={0 0}, upload=434536]: Done begin upload segment. Dec 26 06:33:53.263950472 (495h4m53.135033517s) [noise=true , address=145.79.227.37:4565 , piece_id=YOFIF6SKF7JYWCYKYKQ2AMGD6PG6D7ZGKXEMAZPYMJQZSHB7TSKQ, seg_pos={0 0}, upload=434536]: Uploading piece... Dec 26 06:33:53.264900060 (495h4m53.135983105s) [noise=true , address=45.91.116.4:4939 , piece_id=2JTAYGGW54AO2SSNGC34M6JBHMNPNPZRSRFE4NXCLZ5XTPRSECWQ, seg_pos={0 0}, upload=434536]: Uploading piece... Dec 26 06:33:53.265321652 (495h4m53.136404697s) [noise=true , address=145.79.227.14:4837 , piece_id=BE5WQ2KEA5NQWKSXZCC4HZVJYFHDZLXF62AJGIY2TTRNKIVJJT2A, seg_pos={0 0}, upload=434536]: Uploading piece... Dec 26 06:33:53.265727586 (495h4m53.136810631s) [noise=true , address=145.79.227.49:4337 , piece_id=MBOL7EEJBFFVFNUZEYNCZZCKOD6PPMOYLIK6LGBOBYLKCTII77FQ, seg_pos={0 0}, upload=434536]: Uploading piece... Dec 26 06:33:53.266143929 (495h4m53.137226964s) [noise=true , address=45.91.116.6:4974 , piece_id=OMHN5SPCWSMDX3MF4HLGLDCQKKKBXLEXGKZVSHOLGTTYSTYU53PQ, seg_pos={0 0}, upload=434536]: Uploading piece... Dec 26 06:33:53.266630715 (495h4m53.137713761s) [noise=true , address=45.91.116.7:4011 , piece_id=7HRWVQJADD5DVU73NLOQYN6ROFZSXPZO3Y532GICQEIIK5B3ISQA, seg_pos={0 0}, upload=434536]: Uploading piece... Dec 26 06:33:53.267087495 (495h4m53.13817054s) [noise=true , address=45.91.116.5:4299 , piece_id=ER4WUOMPIHQPCK2DNEOIL2KS6F4G4AZVHUFXOLZUW35F4VDIOXVQ, seg_pos={0 0}, upload=434536]: Uploading piece... Dec 26 06:33:53.267485302 (495h4m53.138568357s) [noise=true , address=145.79.227.41:4085 , piece_id=CHR44GT7FQEV2GCVGKNOJA5XCRHRAPHACLT74BWXVLMAUK3URZDQ, seg_pos={0 0}, upload=434536]: Uploading piece... Dec 26 06:33:53.267858623 (495h4m53.138941668s) [noise=true , address=45.91.116.14:4189 , piece_id=ZMXS5VN7VUPF5Z2NC6WE4S7NPARKQHFG2ZANOVBGH4VYXX3LBGQA, seg_pos={0 0}, upload=434536]: Uploading piece... Dec 26 06:33:53.268257322 (495h4m53.139340367s) [noise=true , address=145.79.227.32:4835 , piece_id=HY6SZRVYSCAE3KU2SIRJYQHNF66DM6QNTWIVLTBAIQHEIXOKEYRA, seg_pos={0 0}, upload=434536]: Uploading piece... Dec 26 06:33:53.268636644 (495h4m53.13971969s) [noise=true , address=145.79.227.47:4266 , piece_id=5CKGHLDR6IFFUKSPGAOEWBEMQSKWSXO26MT2QLNRZETL4UKBTOYA, seg_pos={0 0}, upload=434536]: Uploading piece... Dec 26 06:33:53.269091911 (495h4m53.140174956s) [noise=true , address=45.91.116.9:4262 , piece_id=WMKK6KZ5XQZF3SUMWXIG4WOYV765G23JHHM2LLX2ACK7RRMGJDLQ, seg_pos={0 0}, upload=434536]: Uploading piece... Dec 26 06:33:53.269563550 (495h4m53.140646605s) [noise=true , address=145.79.227.34:4907 , piece_id=TTSL3SIME3JX3ABNMGSIAI7RFYHXBSPX7GE5AKMUTRXR5C3BXZXQ, seg_pos={0 0}, upload=434536]: Uploading piece... Dec 26 06:33:53.270267911 (495h4m53.141350966s) [noise=true , address=45.91.116.11:4117 , piece_id=E534ZX4O5DPGROTSSYSO7Y6HMQZBKEC6SUIPG47DGL5ZHHBQOXNA, seg_pos={0 0}, upload=434536]: Uploading piece... Dec 26 06:33:53.270821675 (495h4m53.14190471s) [noise=true , address=145.79.227.17:4970 , piece_id=NNATAFYU3UT4JOORYI4V5Y65FBHIAE4LAN6U65VYDKXVFSRA2REQ, seg_pos={0 0}, upload=434536]: Uploading piece... Dec 26 06:33:53.271252806 (495h4m53.142335841s) [noise=true , address=145.79.227.10:4403 , piece_id=AYJ2DYFWG7FIMXA45G37MNVTR24MS3AIJ5KALYOY6FV5YSMNLTVA, seg_pos={0 0}, upload=434536]: Uploading piece... Dec 26 06:33:54.990958485 (495h4m54.86204152s) [upload=434536]: Next returned nil segment Dec 26 06:33:54.990980767 (495h4m54.862063802s) [upload=434536]: Waiting for error group managing segments... Dec 26 06:33:55.278685418 (495h4m55.149768463s) [noise=true , address=145.79.227.14:4837 , piece_id=BE5WQ2KEA5NQWKSXZCC4HZVJYFHDZLXF62AJGIY2TTRNKIVJJT2A, seg_pos={0 0}, upload=434536]: Done uploading piece. err: <nil> Dec 26 06:33:55.278727087 (495h4m55.149810122s) [noise=true , address=145.79.227.41:4085 , piece_id=CHR44GT7FQEV2GCVGKNOJA5XCRHRAPHACLT74BWXVLMAUK3URZDQ, seg_pos={0 0}, upload=434536]: Done uploading piece. err: <nil> Dec 26 06:33:55.280052541 (495h4m55.151135586s) [noise=true , address=145.79.227.37:4565 , piece_id=YOFIF6SKF7JYWCYKYKQ2AMGD6PG6D7ZGKXEMAZPYMJQZSHB7TSKQ, seg_pos={0 0}, upload=434536]: Done uploading piece. err: <nil> Dec 26 06:33:55.280159625 (495h4m55.15124267s) [noise=true , address=145.79.227.17:4970 , piece_id=NNATAFYU3UT4JOORYI4V5Y65FBHIAE4LAN6U65VYDKXVFSRA2REQ, seg_pos={0 0}, upload=434536]: Done uploading piece. err: <nil> Dec 26 06:33:55.286058809 (495h4m55.157141854s) [noise=true , address=145.79.227.10:4403 , piece_id=AYJ2DYFWG7FIMXA45G37MNVTR24MS3AIJ5KALYOY6FV5YSMNLTVA, seg_pos={0 0}, upload=434536]: Done uploading piece. err: <nil> Dec 26 06:33:55.286307522 (495h4m55.157390567s) [noise=true , address=145.79.227.49:4337 , piece_id=MBOL7EEJBFFVFNUZEYNCZZCKOD6PPMOYLIK6LGBOBYLKCTII77FQ, seg_pos={0 0}, upload=434536]: Done uploading piece. err: <nil> Dec 26 06:33:55.286848011 (495h4m55.157931046s) [noise=true , address=145.79.227.32:4835 , piece_id=HY6SZRVYSCAE3KU2SIRJYQHNF66DM6QNTWIVLTBAIQHEIXOKEYRA, seg_pos={0 0}, upload=434536]: Done uploading piece. err: <nil> Dec 26 06:33:55.293410307 (495h4m55.164493362s) [noise=true , address=145.79.227.47:4266 , piece_id=5CKGHLDR6IFFUKSPGAOEWBEMQSKWSXO26MT2QLNRZETL4UKBTOYA, seg_pos={0 0}, upload=434536]: Done uploading piece. err: <nil> Dec 26 06:33:55.305200408 (495h4m55.176283443s) [noise=true , address=145.79.227.34:4907 , piece_id=TTSL3SIME3JX3ABNMGSIAI7RFYHXBSPX7GE5AKMUTRXR5C3BXZXQ, seg_pos={0 0}, upload=434536]: Done uploading piece. err: <nil> Dec 26 06:33:55.480806763 (495h4m55.351889808s) [noise=true , address=45.91.116.9:4262 , piece_id=WMKK6KZ5XQZF3SUMWXIG4WOYV765G23JHHM2LLX2ACK7RRMGJDLQ, seg_pos={0 0}, upload=434536]: Done uploading piece. err: <nil> Dec 26 06:33:55.543437063 (495h4m55.414520098s) [noise=true , address=45.91.116.11:4117 , piece_id=E534ZX4O5DPGROTSSYSO7Y6HMQZBKEC6SUIPG47DGL5ZHHBQOXNA, seg_pos={0 0}, upload=434536]: Done uploading piece. err: <nil> Dec 26 06:33:55.545101342 (495h4m55.416184397s) [noise=true , address=45.91.116.5:4299 , piece_id=ER4WUOMPIHQPCK2DNEOIL2KS6F4G4AZVHUFXOLZUW35F4VDIOXVQ, seg_pos={0 0}, upload=434536]: Done uploading piece. err: <nil> Dec 26 06:33:55.632390887 (495h4m55.503473922s) [noise=true , address=45.91.116.7:4011 , piece_id=7HRWVQJADD5DVU73NLOQYN6ROFZSXPZO3Y532GICQEIIK5B3ISQA, seg_pos={0 0}, upload=434536]: Done uploading piece. err: <nil> Dec 26 06:33:55.632407980 (495h4m55.503491025s) [seg_pos={0 0}, upload=434536]: Segment reached optimal threshold of 13 pieces. Dec 26 06:33:55.632638809 (495h4m55.503721854s) [noise=true , address=45.91.116.6:4974 , piece_id=OMHN5SPCWSMDX3MF4HLGLDCQKKKBXLEXGKZVSHOLGTTYSTYU53PQ, seg_pos={0 0}, upload=434536]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Dec 26 06:33:55.632648167 (495h4m55.503731222s) [noise=true , address=45.91.116.6:4974 , piece_id=OMHN5SPCWSMDX3MF4HLGLDCQKKKBXLEXGKZVSHOLGTTYSTYU53PQ, seg_pos={0 0}, upload=434536]: Long tail cancellation due to optimal threshold Dec 26 06:33:55.632951524 (495h4m55.504034569s) [noise=true , address=45.91.116.14:4189 , piece_id=ZMXS5VN7VUPF5Z2NC6WE4S7NPARKQHFG2ZANOVBGH4VYXX3LBGQA, seg_pos={0 0}, upload=434536]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 26 06:33:55.632960382 (495h4m55.504043447s) [noise=true , address=45.91.116.14:4189 , piece_id=ZMXS5VN7VUPF5Z2NC6WE4S7NPARKQHFG2ZANOVBGH4VYXX3LBGQA, seg_pos={0 0}, upload=434536]: Long tail cancellation due to optimal threshold Dec 26 06:33:55.633131136 (495h4m55.504214191s) [noise=true , address=45.91.116.4:4939 , piece_id=2JTAYGGW54AO2SSNGC34M6JBHMNPNPZRSRFE4NXCLZ5XTPRSECWQ, seg_pos={0 0}, upload=434536]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 26 06:33:55.633139322 (495h4m55.504222387s) [noise=true , address=45.91.116.4:4939 , piece_id=2JTAYGGW54AO2SSNGC34M6JBHMNPNPZRSRFE4NXCLZ5XTPRSECWQ, seg_pos={0 0}, upload=434536]: Long tail cancellation due to optimal threshold Dec 26 06:33:55.633158568 (495h4m55.504241623s) [seg_pos={0 0}, upload=434536]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 06:33:55.633243360 (495h4m55.504326415s) [upload=434536]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 06:33:55.633250073 (495h4m55.504333118s) [upload=434536]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 06:33:55.930018343 (495h4m55.801101378s) [upload=434536]: Done uploading segments... Dec 26 06:33:55.930041076 (495h4m55.801124121s) [upload=434536]: Upload finished. err: <nil> Duration: 30.002145319s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [X.X.......................X..........X.......X.X.......X.X.X] duration: 30.004187788s trace: https://jaeger.dev.storj.io/trace/1de491d2df40f807 FAIL (2025-12-26 06:33:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: OAEFTHJY4XN27DTDHEZBJONNFTYVFIT2YPM7QOZLEDIAUDSNOEOA) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: NI5XHY72LG3DXRDGA272V7THTWRHLJ26WZITMGVYRA63435HCXHQ) context deadline exceeded ======LOGS====== Dec 26 06:33:22.768741754 (495h4m22.639824809s) [upload=434528]: Starting upload Dec 26 06:33:22.768854699 (495h4m22.639937734s) [upload=434528]: Done starting upload Dec 26 06:33:22.768943237 (495h4m22.640026282s) [upload=434528]: Uploading segments... Dec 26 06:33:22.769259761 (495h4m22.640342836s) [upload=434528]: Got next segment. Inline: false Dec 26 06:33:22.769341736 (495h4m22.640424781s) [upload=434528]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 06:33:22.769350272 (495h4m22.640433327s) [upload=434528]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 06:33:23.016278597 (495h4m22.887361662s) [seg_pos={0 0}, upload=434528]: Begin upload segment... Dec 26 06:33:23.016337560 (495h4m22.887420605s) [seg_pos={0 0}, upload=434528]: Done begin upload segment. Dec 26 06:33:23.016370502 (495h4m22.887453537s) [noise=true , address=145.79.227.2:4114 , piece_id=5QI6IKKKEXD5O4QNMGEXODLTVUT75U6MFMUWCLV5LWWU3P6SCKJA, seg_pos={0 0}, upload=434528]: Uploading piece... Dec 26 06:33:23.016499377 (495h4m22.887582422s) [noise=true , address=77.90.31.11:4556 , piece_id=D2SC3D5KONTBNLL5SSFX3LOSJ36PYQCMITOOBQ4NLGA6E444GKGA, seg_pos={0 0}, upload=434528]: Uploading piece... Dec 26 06:33:23.017152952 (495h4m22.888235997s) [noise=true , address=45.91.116.12:4225 , piece_id=OAEFTHJY4XN27DTDHEZBJONNFTYVFIT2YPM7QOZLEDIAUDSNOEOA, seg_pos={0 0}, upload=434528]: Uploading piece... Dec 26 06:33:23.017186676 (495h4m22.888269721s) [noise=true , address=45.91.116.11:4120 , piece_id=J44UHTD6JNP2U72BWSPYZ6UZXTN4ZLQP5TB7PSQ2SJALELKA5J3Q, seg_pos={0 0}, upload=434528]: Uploading piece... Dec 26 06:33:23.017585366 (495h4m22.888668411s) [noise=true , address=145.79.227.33:4869 , piece_id=K47XWKUNBZNHGMC65DCHXUH3G7AJUEAD7O2ENIYPYXW7LZTBC6LA, seg_pos={0 0}, upload=434528]: Uploading piece... Dec 26 06:33:23.017614581 (495h4m22.888697626s) [noise=true , address=145.79.227.19:4079 , piece_id=EJPYP57RF4RKMIWNGE3Q3N3ULC7BWYFV245OYDUK36W2OXWNTY5Q, seg_pos={0 0}, upload=434528]: Uploading piece... Dec 26 06:33:23.018040883 (495h4m22.889123928s) [noise=true , address=45.91.116.6:4973 , piece_id=NI5XHY72LG3DXRDGA272V7THTWRHLJ26WZITMGVYRA63435HCXHQ, seg_pos={0 0}, upload=434528]: Uploading piece... Dec 26 06:33:23.018086269 (495h4m22.889169304s) [noise=true , address=145.79.227.45:4231 , piece_id=2WMB4UBCMOGDZCB6NIIS644QUHDL27SYXKT4O37ZWL3UCZHANYFA, seg_pos={0 0}, upload=434528]: Uploading piece... Dec 26 06:33:23.018532980 (495h4m22.889616025s) [noise=true , address=145.79.227.51:4410 , piece_id=TX3DMZVKZKQQDKCRLSRXE3DDJ6EKKGPC7RGOLKAW3CLHVK2BKI4Q, seg_pos={0 0}, upload=434528]: Uploading piece... Dec 26 06:33:23.018721128 (495h4m22.889804173s) [noise=true , address=145.79.227.6:4259 , piece_id=DGVN7JW6Z7I2R5RWOXD3J7KP646OBL3JKHQNSJRNOUHVRJCYYHCQ, seg_pos={0 0}, upload=434528]: Uploading piece... Dec 26 06:33:23.018939213 (495h4m22.890022258s) [noise=true , address=45.91.116.13:4155 , piece_id=GKTHXIZQM3CSIBNLI5CYBOAZUW25BFD2HTVB23FG3QTHDUTIZWOQ, seg_pos={0 0}, upload=434528]: Uploading piece... Dec 26 06:33:23.019205420 (495h4m22.890288455s) [noise=true , address=45.91.116.14:4190 , piece_id=NC324SPWATSYMJO2GQIUSYVIPMLL55WZJSGT6QKNGMOEQNQT67PA, seg_pos={0 0}, upload=434528]: Uploading piece... Dec 26 06:33:23.019357399 (495h4m22.890440444s) [noise=true , address=45.91.116.7:4011 , piece_id=HXTH4CZCIIDFH6EBCMYN4V2WOCV3CIBXDDF5JMOLXZ6XWZZFMXWQ, seg_pos={0 0}, upload=434528]: Uploading piece... Dec 26 06:33:23.019606204 (495h4m22.890689249s) [noise=true , address=77.90.31.14:4663 , piece_id=BSA54IMJOEW5GHMBTJD55BVZBHVGOQRBOTIKCMKMZQWHQFB2HHMA, seg_pos={0 0}, upload=434528]: Uploading piece... Dec 26 06:33:23.019750108 (495h4m22.890833143s) [noise=true , address=45.91.116.8:4045 , piece_id=S754DS2BZI5USNAZHN4LM6DAV6PUGQN5B76D2UGUFHSVB33U2NEQ, seg_pos={0 0}, upload=434528]: Uploading piece... Dec 26 06:33:23.020060118 (495h4m22.891143163s) [noise=true , address=145.79.227.39:4013 , piece_id=4ILC2PH5YARWONAFXY3PQC2IMFJGUHGQVFMQSXW2U5L7UYKHEPTA, seg_pos={0 0}, upload=434528]: Uploading piece... Dec 26 06:33:24.739655254 (495h4m24.610738289s) [upload=434528]: Next returned nil segment Dec 26 06:33:24.739673548 (495h4m24.610756593s) [upload=434528]: Waiting for error group managing segments... Dec 26 06:33:25.026396559 (495h4m24.897479594s) [noise=true , address=145.79.227.51:4410 , piece_id=TX3DMZVKZKQQDKCRLSRXE3DDJ6EKKGPC7RGOLKAW3CLHVK2BKI4Q, seg_pos={0 0}, upload=434528]: Done uploading piece. err: <nil> Dec 26 06:33:25.028254748 (495h4m24.899337793s) [noise=true , address=145.79.227.2:4114 , piece_id=5QI6IKKKEXD5O4QNMGEXODLTVUT75U6MFMUWCLV5LWWU3P6SCKJA, seg_pos={0 0}, upload=434528]: Done uploading piece. err: <nil> Dec 26 06:33:25.029641289 (495h4m24.900724334s) [noise=true , address=145.79.227.19:4079 , piece_id=EJPYP57RF4RKMIWNGE3Q3N3ULC7BWYFV245OYDUK36W2OXWNTY5Q, seg_pos={0 0}, upload=434528]: Done uploading piece. err: <nil> Dec 26 06:33:25.030902690 (495h4m24.901985735s) [noise=true , address=145.79.227.39:4013 , piece_id=4ILC2PH5YARWONAFXY3PQC2IMFJGUHGQVFMQSXW2U5L7UYKHEPTA, seg_pos={0 0}, upload=434528]: Done uploading piece. err: <nil> Dec 26 06:33:25.048536449 (495h4m24.919619494s) [noise=true , address=145.79.227.45:4231 , piece_id=2WMB4UBCMOGDZCB6NIIS644QUHDL27SYXKT4O37ZWL3UCZHANYFA, seg_pos={0 0}, upload=434528]: Done uploading piece. err: <nil> Dec 26 06:33:25.058889051 (495h4m24.929972096s) [noise=true , address=145.79.227.33:4869 , piece_id=K47XWKUNBZNHGMC65DCHXUH3G7AJUEAD7O2ENIYPYXW7LZTBC6LA, seg_pos={0 0}, upload=434528]: Done uploading piece. err: <nil> Dec 26 06:33:25.234118720 (495h4m25.105201755s) [noise=true , address=77.90.31.11:4556 , piece_id=D2SC3D5KONTBNLL5SSFX3LOSJ36PYQCMITOOBQ4NLGA6E444GKGA, seg_pos={0 0}, upload=434528]: Done uploading piece. err: <nil> Dec 26 06:33:25.309219606 (495h4m25.180302641s) [noise=true , address=45.91.116.13:4155 , piece_id=GKTHXIZQM3CSIBNLI5CYBOAZUW25BFD2HTVB23FG3QTHDUTIZWOQ, seg_pos={0 0}, upload=434528]: Done uploading piece. err: <nil> Dec 26 06:33:25.323837601 (495h4m25.194920636s) [noise=true , address=77.90.31.14:4663 , piece_id=BSA54IMJOEW5GHMBTJD55BVZBHVGOQRBOTIKCMKMZQWHQFB2HHMA, seg_pos={0 0}, upload=434528]: Done uploading piece. err: <nil> Dec 26 06:33:25.370023107 (495h4m25.241106152s) [noise=true , address=45.91.116.12:4225 , piece_id=OAEFTHJY4XN27DTDHEZBJONNFTYVFIT2YPM7QOZLEDIAUDSNOEOA, seg_pos={0 0}, upload=434528]: Done uploading piece. err: <nil> Dec 26 06:33:25.395370583 (495h4m25.266453648s) [noise=true , address=45.91.116.8:4045 , piece_id=S754DS2BZI5USNAZHN4LM6DAV6PUGQN5B76D2UGUFHSVB33U2NEQ, seg_pos={0 0}, upload=434528]: Done uploading piece. err: <nil> Dec 26 06:33:25.893927055 (495h4m25.76501009s) [noise=true , address=45.91.116.14:4190 , piece_id=NC324SPWATSYMJO2GQIUSYVIPMLL55WZJSGT6QKNGMOEQNQT67PA, seg_pos={0 0}, upload=434528]: Done uploading piece. err: <nil> Dec 26 06:33:25.931758572 (495h4m25.802841607s) [noise=true , address=45.91.116.6:4973 , piece_id=NI5XHY72LG3DXRDGA272V7THTWRHLJ26WZITMGVYRA63435HCXHQ, seg_pos={0 0}, upload=434528]: Done uploading piece. err: <nil> Dec 26 06:33:25.931780895 (495h4m25.80286394s) [seg_pos={0 0}, upload=434528]: Segment reached optimal threshold of 13 pieces. Dec 26 06:33:25.931995663 (495h4m25.803078709s) [noise=true , address=145.79.227.6:4259 , piece_id=DGVN7JW6Z7I2R5RWOXD3J7KP646OBL3JKHQNSJRNOUHVRJCYYHCQ, seg_pos={0 0}, upload=434528]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y): protocol: expected piece hash; context canceled; EOF Dec 26 06:33:25.932004942 (495h4m25.803087987s) [noise=true , address=145.79.227.6:4259 , piece_id=DGVN7JW6Z7I2R5RWOXD3J7KP646OBL3JKHQNSJRNOUHVRJCYYHCQ, seg_pos={0 0}, upload=434528]: Long tail cancellation due to optimal threshold Dec 26 06:33:25.932111956 (495h4m25.803194991s) [noise=true , address=45.91.116.7:4011 , piece_id=HXTH4CZCIIDFH6EBCMYN4V2WOCV3CIBXDDF5JMOLXZ6XWZZFMXWQ, seg_pos={0 0}, upload=434528]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 26 06:33:25.932121614 (495h4m25.803204659s) [noise=true , address=45.91.116.7:4011 , piece_id=HXTH4CZCIIDFH6EBCMYN4V2WOCV3CIBXDDF5JMOLXZ6XWZZFMXWQ, seg_pos={0 0}, upload=434528]: Long tail cancellation due to optimal threshold Dec 26 06:33:25.932259306 (495h4m25.803342351s) [noise=true , address=45.91.116.11:4120 , piece_id=J44UHTD6JNP2U72BWSPYZ6UZXTN4ZLQP5TB7PSQ2SJALELKA5J3Q, seg_pos={0 0}, upload=434528]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 26 06:33:25.932266800 (495h4m25.803349845s) [noise=true , address=45.91.116.11:4120 , piece_id=J44UHTD6JNP2U72BWSPYZ6UZXTN4ZLQP5TB7PSQ2SJALELKA5J3Q, seg_pos={0 0}, upload=434528]: Long tail cancellation due to optimal threshold Dec 26 06:33:25.932285246 (495h4m25.803368291s) [seg_pos={0 0}, upload=434528]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 06:33:25.932361330 (495h4m25.803444375s) [upload=434528]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 06:33:25.932368544 (495h4m25.803451579s) [upload=434528]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 06:33:26.241366512 (495h4m26.112449547s) [upload=434528]: Done uploading segments... Dec 26 06:33:26.241388052 (495h4m26.112471097s) [upload=434528]: Upload finished. err: <nil> Duration: 30.004213617s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [............................X...............................] duration: 3.44897796s trace: https://jaeger.dev.storj.io/trace/630094597d401477 FAIL (2025-12-26 06:33:43 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.44900888s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 28.986919251s trace: https://jaeger.dev.storj.io/trace/7e198d4c8fea54d2 FAIL (2025-12-26 06:33:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p, Piece ID: SBCEX7UNWGNFF54SBP4AGRDV5IPG24FQJXJJ7EYQTBSZKOVZWU4Q) context deadline exceeded; piecestore: (Node ID: 1nP8xospdMuDEh3em5K6iBnh8bHXqLhvHy4pHm9m8Lnj4kEH2X, Piece ID: XV7FQGBJNWVRLOAFRUPOWON35JR5ESRJPTCWPUGYWGHJPSDVL4GA) context deadline exceeded; piecestore: (Node ID: 1Mg5mxUjW56quDidSgA788D7ByfAzvsEuWJqwtzY4bLyxRn3mu, Piece ID: ATPIXLMXGONSSLXPPVORSNC6LHPXI2V3UFTHM4VHA2FTY5ZM6H3A) context deadline exceeded; piecestore: (Node ID: 12QmYSEAKnkSLEEVc9E1uxtVigunKMFLk2cdu71Bse8Tqec3HyT, Piece ID: YFKD7GO2FOSPLDXNFPK732QTLCLBQQASWBTMZ2LTLBGSK22PIXIQ) context deadline exceeded; piecestore: (Node ID: 1JsnHuhQhM4rWLTLEuSSHLNE24m7tSW9qMSf48j7yMrAVPWVyV, Piece ID: ARBMDAKX4QBW6JG47KP2PUITCX7HKXCBO75THQPINM4LLHCKAV4Q) context deadline exceeded ======LOGS====== upload=433902]: Uploading piece... Dec 26 05:50:34.986155265 (494h21m34.8572383s) [noise=true , address=64.49.13.140:4509 , piece_id=6TGWTOAVY4YVAKU6VEE2SWVAM264TKU6ZNMGPWH4TNGII3UYXFXQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.986703610 (494h21m34.857786655s) [noise=true , address=77.90.31.10:4522 , piece_id=YFKD7GO2FOSPLDXNFPK732QTLCLBQQASWBTMZ2LTLBGSK22PIXIQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.987145380 (494h21m34.858228425s) [noise=true , address=145.79.227.33:4890 , piece_id=UKNB4ZZTYJGY4MDHOBPYUTHCP3ZDCMJB7O7AJZFYP2HFVIWLUOKQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.987628290 (494h21m34.858711335s) [noise=true , address=145.79.227.40:4052 , piece_id=XRZYL7PT5VHY22N7CCKS3NTZKDDWE36ZU3SORF6ILM7ZMXDND63A, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.988710731 (494h21m34.859793776s) [noise=true , address=77.90.31.13:4653 , piece_id=EY2CWX54ONOWPO3WC33G5QSP5RMP2SUTBQLUOH5JXGDRUEPDHQ2A, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.989894545 (494h21m34.86097759s) [noise=true , address=145.79.227.30:4785 , piece_id=5KMLU4ATBN4YVQDYJKKQY4PEJNKWTO2WNX5GMGJM4F54B6YM2SFA, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.991305081 (494h21m34.862388116s) [noise=true , address=145.79.227.50:4381 , piece_id=H6Q47DIOTOWGN7QH2KWT4IH6KVDSB246S22753Z3UQGB26T6VXPA, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.995796263 (494h21m34.866879359s) [noise=true , address=145.79.227.52:4461 , piece_id=ZAAUDBGY6CC2JNEGWXVPCBAWXARHAJDZ3YPB6Y3NWEWCFDUFGDSQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.996765439 (494h21m34.867848484s) [noise=true , address=77.90.31.16:4755 , piece_id=ARBMDAKX4QBW6JG47KP2PUITCX7HKXCBO75THQPINM4LLHCKAV4Q, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.998107634 (494h21m34.869190679s) [noise=true , address=77.90.31.11:4579 , piece_id=SBCEX7UNWGNFF54SBP4AGRDV5IPG24FQJXJJ7EYQTBSZKOVZWU4Q, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.999340442 (494h21m34.870423477s) [noise=true , address=145.79.227.34:4936 , piece_id=EEA2ZPM44DX3WMI7ZJTDC7T5UIGED6TNT53QXZWGBFRMQUQSCXPQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:35.000540757 (494h21m34.871623812s) [noise=true , address=45.91.116.14:4204 , piece_id=XV7FQGBJNWVRLOAFRUPOWON35JR5ESRJPTCWPUGYWGHJPSDVL4GA, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:36.360077064 (494h21m36.231160099s) [upload=433902]: Next returned nil segment Dec 26 05:50:36.360091642 (494h21m36.231174688s) [upload=433902]: Waiting for error group managing segments... Dec 26 05:50:36.688914299 (494h21m36.559997344s) [noise=true , address=64.49.13.140:4509 , piece_id=6TGWTOAVY4YVAKU6VEE2SWVAM264TKU6ZNMGPWH4TNGII3UYXFXQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.059812634 (494h21m36.930895669s) [noise=true , address=77.90.31.11:4579 , piece_id=SBCEX7UNWGNFF54SBP4AGRDV5IPG24FQJXJJ7EYQTBSZKOVZWU4Q, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.119433874 (494h21m36.990516909s) [noise=true , address=45.91.116.14:4204 , piece_id=XV7FQGBJNWVRLOAFRUPOWON35JR5ESRJPTCWPUGYWGHJPSDVL4GA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.138718204 (494h21m37.00980125s) [noise=true , address=77.90.31.16:4755 , piece_id=ARBMDAKX4QBW6JG47KP2PUITCX7HKXCBO75THQPINM4LLHCKAV4Q, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.146647243 (494h21m37.017730278s) [noise=true , address=77.90.31.13:4653 , piece_id=EY2CWX54ONOWPO3WC33G5QSP5RMP2SUTBQLUOH5JXGDRUEPDHQ2A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.163283722 (494h21m37.034366757s) [noise=true , address=77.90.31.21:4893 , piece_id=GNEEZO2XGOKNODOUPZQ4NGKFE4PCX5LNUWXLV27F6X7GWZVSYNAA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.374956574 (494h21m37.246039609s) [noise=true , address=45.91.116.7:4033 , piece_id=ATPIXLMXGONSSLXPPVORSNC6LHPXI2V3UFTHM4VHA2FTY5ZM6H3A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.777283355 (494h21m37.64836639s) [noise=true , address=77.90.31.12:4622 , piece_id=LTYYPK6ZQ3NXZPBKSV4OMZGCOXXV3N6L5A6MVODNR644D2ZUJX3A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:38.439025909 (494h21m38.310108944s) [noise=true , address=145.79.227.49:4355 , piece_id=IUDOHR73Y6OKFAQSNNSONALWAITRJ4POYRO2QD5LYZKOSTWY4MNA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:38.439599111 (494h21m38.310682166s) [noise=true , address=77.90.31.10:4522 , piece_id=YFKD7GO2FOSPLDXNFPK732QTLCLBQQASWBTMZ2LTLBGSK22PIXIQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:39.259154980 (494h21m39.130238015s) [noise=true , address=145.79.227.34:4936 , piece_id=EEA2ZPM44DX3WMI7ZJTDC7T5UIGED6TNT53QXZWGBFRMQUQSCXPQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:39.430935921 (494h21m39.302018956s) [noise=true , address=145.79.227.44:4212 , piece_id=ABBNLKRE4FM7RHF56V4IAQPKBCCCQLLOMFVMSJEWQUNFGGCI3PVQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:42.941071297 (494h21m42.812154352s) [noise=true , address=77.90.31.14:4675 , piece_id=FKB4O45OYICMELPGXB65QAUD2DRIFTSXZFKIA6TY26VFAP6AVY3A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:51:22.719026850 (494h22m22.590109885s) [noise=true , address=145.79.227.33:4890 , piece_id=UKNB4ZZTYJGY4MDHOBPYUTHCP3ZDCMJB7O7AJZFYP2HFVIWLUOKQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:51:22.719060193 (494h22m22.590143228s) [seg_pos={0 0}, upload=433902]: Segment reached optimal threshold of 14 pieces. Dec 26 05:51:22.719233022 (494h22m22.590316067s) [noise=true , address=145.79.227.50:4381 , piece_id=H6Q47DIOTOWGN7QH2KWT4IH6KVDSB246S22753Z3UQGB26T6VXPA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zU35MWWaaC1npkP5qGcWBXAkfxGsd5UsdkM2nwX9x7wUGCDbY): protocol: write tcp 188.166.225.170:46970->145.79.227.50:4381: use of closed network connection; write tcp 188.166.225.170:46970->145.79.227.50:4381: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46970->145.79.227.50:4381: use of closed network connection Dec 26 05:51:22.719238783 (494h22m22.590321828s) [noise=true , address=145.79.227.50:4381 , piece_id=H6Q47DIOTOWGN7QH2KWT4IH6KVDSB246S22753Z3UQGB26T6VXPA, seg_pos={0 0}, upload=433902]: Long tail cancellation due to optimal threshold Dec 26 05:51:22.719251677 (494h22m22.590334712s) [noise=true , address=145.79.227.30:4785 , piece_id=5KMLU4ATBN4YVQDYJKKQY4PEJNKWTO2WNX5GMGJM4F54B6YM2SFA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gYH83ECV8wai3CZyBSU8NNJM1UsysLYHbQDwBVKZRyjsdnUwB): protocol: write tcp 188.166.225.170:58398->145.79.227.30:4785: use of closed network connection; write tcp 188.166.225.170:58398->145.79.227.30:4785: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58398->145.79.227.30:4785: use of closed network connection Dec 26 05:51:22.719260194 (494h22m22.590343229s) [noise=true , address=145.79.227.30:4785 , piece_id=5KMLU4ATBN4YVQDYJKKQY4PEJNKWTO2WNX5GMGJM4F54B6YM2SFA, seg_pos={0 0}, upload=433902]: Long tail cancellation due to optimal threshold Dec 26 05:51:22.719409408 (494h22m22.590492443s) [noise=true , address=145.79.227.40:4052 , piece_id=XRZYL7PT5VHY22N7CCKS3NTZKDDWE36ZU3SORF6ILM7ZMXDND63A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PViJBYAanwXRWhMq3WHR6k9Stxy8hjuY5FV9J7ErxCy4N1q3V): protocol: write tcp 188.166.225.170:45984->145.79.227.40:4052: use of closed network connection; write tcp 188.166.225.170:45984->145.79.227.40:4052: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45984->145.79.227.40:4052: use of closed network connection Dec 26 05:51:22.719415469 (494h22m22.590498515s) [noise=true , address=145.79.227.40:4052 , piece_id=XRZYL7PT5VHY22N7CCKS3NTZKDDWE36ZU3SORF6ILM7ZMXDND63A, seg_pos={0 0}, upload=433902]: Long tail cancellation due to optimal threshold Dec 26 05:51:22.719467057 (494h22m22.590550092s) [noise=true , address=145.79.227.52:4461 , piece_id=ZAAUDBGY6CC2JNEGWXVPCBAWXARHAJDZ3YPB6Y3NWEWCFDUFGDSQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fNrbSfErEQRnruJCJAVvkoNwQSPWnvNtxunHn6zrQegnNAkxA): protocol: write tcp 188.166.225.170:55966->145.79.227.52:4461: use of closed network connection; write tcp 188.166.225.170:55966->145.79.227.52:4461: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55966->145.79.227.52:4461: use of closed network connection Dec 26 05:51:22.719473931 (494h22m22.590556966s) [noise=true , address=145.79.227.52:4461 , piece_id=ZAAUDBGY6CC2JNEGWXVPCBAWXARHAJDZ3YPB6Y3NWEWCFDUFGDSQ, seg_pos={0 0}, upload=433902]: Long tail cancellation due to optimal threshold Dec 26 05:51:22.719491374 (494h22m22.590574409s) [seg_pos={0 0}, upload=433902]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 05:51:22.719562539 (494h22m22.590645574s) [upload=433902]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:51:22.719567228 (494h22m22.590650263s) [upload=433902]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:51:23.008169117 (494h22m22.879252162s) [upload=433902]: Done uploading segments... Dec 26 05:51:23.008191560 (494h22m22.879274605s) [upload=433902]: Upload finished. err: <nil> Duration: 28.986951912s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.X.X.......................X..........X.......X.X.......X.X.] duration: 5.846760499s trace: https://jaeger.dev.storj.io/trace/1f1d0344ea514d5 FAIL (2025-12-26 06:32:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: BLMLERZPBMLNTU3HZCSNLVH3KUSYJXI2FBEPCR445I3IUFKO7RNQ) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: RZE4W6XP3ZZ33SURGR3ES44EJE5RKDXRQBZS5MHMQ2F7VPATCAJQ) context deadline exceeded; piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: D5ENMNENE6SXNU7HQV2K5LFHUJWZWAEHPKYM2ZNFXMFEZJJBDGBA) context deadline exceeded ======LOGS====== Dec 26 06:31:39.479209150 (495h2m39.350292185s) [upload=434504]: Starting upload Dec 26 06:31:39.479309000 (495h2m39.350392035s) [upload=434504]: Done starting upload Dec 26 06:31:39.479369316 (495h2m39.350452351s) [upload=434504]: Uploading segments... Dec 26 06:31:39.479591258 (495h2m39.350674303s) [upload=434504]: Got next segment. Inline: false Dec 26 06:31:39.479651032 (495h2m39.350734067s) [upload=434504]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 06:31:39.479658185 (495h2m39.35074123s) [upload=434504]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 06:31:39.721689426 (495h2m39.592772461s) [seg_pos={0 0}, upload=434504]: Begin upload segment... Dec 26 06:31:39.721746355 (495h2m39.59282939s) [seg_pos={0 0}, upload=434504]: Done begin upload segment. Dec 26 06:31:39.721792702 (495h2m39.592875748s) [noise=true , address=145.79.227.17:4972 , piece_id=HSDSRT6DKUTLGN5ZX3E7IMIXNKBQ34NRUS7CRUC2NFX2QIHKVV7Q, seg_pos={0 0}, upload=434504]: Uploading piece... Dec 26 06:31:39.721872985 (495h2m39.59295602s) [noise=true , address=45.91.116.7:4009 , piece_id=QHG75INRC3QWMPE445SBWOCOIS5VGG4WL77AMT675W4OPYDXE3PQ, seg_pos={0 0}, upload=434504]: Uploading piece... Dec 26 06:31:39.722386323 (495h2m39.593469368s) [noise=true , address=145.79.227.30:4761 , piece_id=74BWE3S6PKVPRVOBW6PWC4MTSRTM2GLRG4VRLGVSOTMI6IXEYKJA, seg_pos={0 0}, upload=434504]: Uploading piece... Dec 26 06:31:39.723532966 (495h2m39.594616001s) [noise=true , address=45.91.116.12:4227 , piece_id=BR66LHYV7IRD3SCTHVDJOUVOQSGE3S23WEVXULYTYWKPCCOQZW5Q, seg_pos={0 0}, upload=434504]: Uploading piece... Dec 26 06:31:39.724031285 (495h2m39.59511432s) [noise=true , address=45.91.116.8:4045 , piece_id=JSX3ETE67ETTNQOTOP4D7YMVLOGVOYROXGBTZFWZV7NWUGPNHSYA, seg_pos={0 0}, upload=434504]: Uploading piece... Dec 26 06:31:39.724467145 (495h2m39.59555018s) [noise=true , address=145.79.227.39:4013 , piece_id=7QQBE6ZHGMF2XSTIJCE3RGIWEUDFYLGDOHXKPSY3OH72NXDXYHQA, seg_pos={0 0}, upload=434504]: Uploading piece... Dec 26 06:31:39.724887836 (495h2m39.595970881s) [noise=true , address=45.91.116.4:4939 , piece_id=QXV4MGEAZ4H76KR4TNX3OA5RKWNTUZIQKGVJ5KYSRJ6VUO6LOZEA, seg_pos={0 0}, upload=434504]: Uploading piece... Dec 26 06:31:39.726070027 (495h2m39.597153062s) [noise=true , address=45.91.116.6:4975 , piece_id=XJ77ZGFWUWKOTEP5LRICBHY7676OTYSQSBYDWM7MKAZMATNW5POQ, seg_pos={0 0}, upload=434504]: Uploading piece... Dec 26 06:31:39.726395948 (495h2m39.597478983s) [noise=true , address=145.79.227.3:4149 , piece_id=XA6Q7PDQTB47JT3K4B6UK2PPITTCDYXGVWFEBVBLOV7ANM2WXYRA, seg_pos={0 0}, upload=434504]: Uploading piece... Dec 26 06:31:39.727468821 (495h2m39.598551866s) [noise=true , address=145.79.227.48:4303 , piece_id=ANM6TVALEREUIXQJP3FDNCYLCWUPKVQXYEPPVTRX5HDUWASG67OA, seg_pos={0 0}, upload=434504]: Uploading piece... Dec 26 06:31:39.727842472 (495h2m39.598925508s) [noise=true , address=77.90.31.15:4700 , piece_id=BLMLERZPBMLNTU3HZCSNLVH3KUSYJXI2FBEPCR445I3IUFKO7RNQ, seg_pos={0 0}, upload=434504]: Uploading piece... Dec 26 06:31:39.729427731 (495h2m39.600510786s) [noise=true , address=145.79.227.13:4802 , piece_id=NH7IGTZ54LC5CZZWQTP33QK3GMYANXZSIC6MVPIPGWHP3Y6W5B4Q, seg_pos={0 0}, upload=434504]: Uploading piece... Dec 26 06:31:39.730912899 (495h2m39.601995944s) [noise=true , address=145.79.227.51:4410 , piece_id=KDCQ6LZU2BHPOXSCWAAQ37R3PUUORZ24CGETAE6GVTU4JBMZ4AOQ, seg_pos={0 0}, upload=434504]: Uploading piece... Dec 26 06:31:39.732298758 (495h2m39.603381793s) [noise=true , address=45.91.116.13:4155 , piece_id=RZE4W6XP3ZZ33SURGR3ES44EJE5RKDXRQBZS5MHMQ2F7VPATCAJQ, seg_pos={0 0}, upload=434504]: Uploading piece... Dec 26 06:31:39.734737612 (495h2m39.605820657s) [noise=true , address=45.91.116.14:4192 , piece_id=D5ENMNENE6SXNU7HQV2K5LFHUJWZWAEHPKYM2ZNFXMFEZJJBDGBA, seg_pos={0 0}, upload=434504]: Uploading piece... Dec 26 06:31:39.735646462 (495h2m39.606729497s) [noise=true , address=145.79.227.18:4007 , piece_id=6O3FCKQ2GCFHZOGVYOKPFNUCEXE5ZCQAF77P7FABBJQBNAH7KTNQ, seg_pos={0 0}, upload=434504]: Uploading piece... Dec 26 06:31:41.453322335 (495h2m41.32440537s) [upload=434504]: Next returned nil segment Dec 26 06:31:41.453336041 (495h2m41.324419076s) [upload=434504]: Waiting for error group managing segments... Dec 26 06:31:41.740730752 (495h2m41.611813788s) [noise=true , address=145.79.227.51:4410 , piece_id=KDCQ6LZU2BHPOXSCWAAQ37R3PUUORZ24CGETAE6GVTU4JBMZ4AOQ, seg_pos={0 0}, upload=434504]: Done uploading piece. err: <nil> Dec 26 06:31:41.747136009 (495h2m41.618219054s) [noise=true , address=145.79.227.39:4013 , piece_id=7QQBE6ZHGMF2XSTIJCE3RGIWEUDFYLGDOHXKPSY3OH72NXDXYHQA, seg_pos={0 0}, upload=434504]: Done uploading piece. err: <nil> Dec 26 06:31:41.753685651 (495h2m41.624768686s) [noise=true , address=145.79.227.48:4303 , piece_id=ANM6TVALEREUIXQJP3FDNCYLCWUPKVQXYEPPVTRX5HDUWASG67OA, seg_pos={0 0}, upload=434504]: Done uploading piece. err: <nil> Dec 26 06:31:41.756198776 (495h2m41.627281841s) [noise=true , address=145.79.227.17:4972 , piece_id=HSDSRT6DKUTLGN5ZX3E7IMIXNKBQ34NRUS7CRUC2NFX2QIHKVV7Q, seg_pos={0 0}, upload=434504]: Done uploading piece. err: <nil> Dec 26 06:31:41.761657541 (495h2m41.632740586s) [noise=true , address=145.79.227.3:4149 , piece_id=XA6Q7PDQTB47JT3K4B6UK2PPITTCDYXGVWFEBVBLOV7ANM2WXYRA, seg_pos={0 0}, upload=434504]: Done uploading piece. err: <nil> Dec 26 06:31:41.764401165 (495h2m41.63548421s) [noise=true , address=145.79.227.18:4007 , piece_id=6O3FCKQ2GCFHZOGVYOKPFNUCEXE5ZCQAF77P7FABBJQBNAH7KTNQ, seg_pos={0 0}, upload=434504]: Done uploading piece. err: <nil> Dec 26 06:31:41.783526082 (495h2m41.654609117s) [noise=true , address=145.79.227.13:4802 , piece_id=NH7IGTZ54LC5CZZWQTP33QK3GMYANXZSIC6MVPIPGWHP3Y6W5B4Q, seg_pos={0 0}, upload=434504]: Done uploading piece. err: <nil> Dec 26 06:31:41.963940944 (495h2m41.835023989s) [noise=true , address=45.91.116.4:4939 , piece_id=QXV4MGEAZ4H76KR4TNX3OA5RKWNTUZIQKGVJ5KYSRJ6VUO6LOZEA, seg_pos={0 0}, upload=434504]: Done uploading piece. err: <nil> Dec 26 06:31:41.988624225 (495h2m41.85970726s) [noise=true , address=45.91.116.6:4975 , piece_id=XJ77ZGFWUWKOTEP5LRICBHY7676OTYSQSBYDWM7MKAZMATNW5POQ, seg_pos={0 0}, upload=434504]: Done uploading piece. err: <nil> Dec 26 06:31:41.997763819 (495h2m41.868846864s) [noise=true , address=45.91.116.14:4192 , piece_id=D5ENMNENE6SXNU7HQV2K5LFHUJWZWAEHPKYM2ZNFXMFEZJJBDGBA, seg_pos={0 0}, upload=434504]: Done uploading piece. err: <nil> Dec 26 06:31:42.016237286 (495h2m41.887320321s) [noise=true , address=77.90.31.15:4700 , piece_id=BLMLERZPBMLNTU3HZCSNLVH3KUSYJXI2FBEPCR445I3IUFKO7RNQ, seg_pos={0 0}, upload=434504]: Done uploading piece. err: <nil> Dec 26 06:31:42.079360894 (495h2m41.950443939s) [noise=true , address=45.91.116.13:4155 , piece_id=RZE4W6XP3ZZ33SURGR3ES44EJE5RKDXRQBZS5MHMQ2F7VPATCAJQ, seg_pos={0 0}, upload=434504]: Done uploading piece. err: <nil> Dec 26 06:31:42.087559917 (495h2m41.958642962s) [noise=true , address=45.91.116.7:4009 , piece_id=QHG75INRC3QWMPE445SBWOCOIS5VGG4WL77AMT675W4OPYDXE3PQ, seg_pos={0 0}, upload=434504]: Done uploading piece. err: <nil> Dec 26 06:31:42.087584193 (495h2m41.958667238s) [seg_pos={0 0}, upload=434504]: Segment reached optimal threshold of 13 pieces. Dec 26 06:31:42.087869707 (495h2m41.958952762s) [noise=true , address=45.91.116.8:4045 , piece_id=JSX3ETE67ETTNQOTOP4D7YMVLOGVOYROXGBTZFWZV7NWUGPNHSYA, seg_pos={0 0}, upload=434504]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: write tcp 188.166.225.170:57512->45.91.116.8:4045: use of closed network connection; write tcp 188.166.225.170:57512->45.91.116.8:4045: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57512->45.91.116.8:4045: use of closed network connection Dec 26 06:31:42.087878664 (495h2m41.958961709s) [noise=true , address=45.91.116.8:4045 , piece_id=JSX3ETE67ETTNQOTOP4D7YMVLOGVOYROXGBTZFWZV7NWUGPNHSYA, seg_pos={0 0}, upload=434504]: Long tail cancellation due to optimal threshold Dec 26 06:31:42.088178335 (495h2m41.95926138s) [noise=true , address=145.79.227.30:4761 , piece_id=74BWE3S6PKVPRVOBW6PWC4MTSRTM2GLRG4VRLGVSOTMI6IXEYKJA, seg_pos={0 0}, upload=434504]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 26 06:31:42.088185909 (495h2m41.959268954s) [noise=true , address=145.79.227.30:4761 , piece_id=74BWE3S6PKVPRVOBW6PWC4MTSRTM2GLRG4VRLGVSOTMI6IXEYKJA, seg_pos={0 0}, upload=434504]: Long tail cancellation due to optimal threshold Dec 26 06:31:42.088319363 (495h2m41.959402398s) [noise=true , address=45.91.116.12:4227 , piece_id=BR66LHYV7IRD3SCTHVDJOUVOQSGE3S23WEVXULYTYWKPCCOQZW5Q, seg_pos={0 0}, upload=434504]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Dec 26 06:31:42.088326026 (495h2m41.959409071s) [noise=true , address=45.91.116.12:4227 , piece_id=BR66LHYV7IRD3SCTHVDJOUVOQSGE3S23WEVXULYTYWKPCCOQZW5Q, seg_pos={0 0}, upload=434504]: Long tail cancellation due to optimal threshold Dec 26 06:31:42.088343508 (495h2m41.959426553s) [seg_pos={0 0}, upload=434504]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 06:31:42.088420836 (495h2m41.959503891s) [upload=434504]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 06:31:42.088428320 (495h2m41.959511355s) [upload=434504]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 06:31:42.385278857 (495h2m42.256361892s) [upload=434504]: Done uploading segments... Dec 26 06:31:42.385297132 (495h2m42.256380177s) [upload=434504]: Upload finished. err: <nil> Duration: 5.846794292s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 32.415313822s trace: https://jaeger.dev.storj.io/trace/7e198d4c8fea54d2 FAIL (2025-12-26 06:32:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p, Piece ID: SBCEX7UNWGNFF54SBP4AGRDV5IPG24FQJXJJ7EYQTBSZKOVZWU4Q) context deadline exceeded; piecestore: (Node ID: 1nP8xospdMuDEh3em5K6iBnh8bHXqLhvHy4pHm9m8Lnj4kEH2X, Piece ID: XV7FQGBJNWVRLOAFRUPOWON35JR5ESRJPTCWPUGYWGHJPSDVL4GA) context deadline exceeded; piecestore: (Node ID: 1Mg5mxUjW56quDidSgA788D7ByfAzvsEuWJqwtzY4bLyxRn3mu, Piece ID: ATPIXLMXGONSSLXPPVORSNC6LHPXI2V3UFTHM4VHA2FTY5ZM6H3A) context deadline exceeded; piecestore: (Node ID: 12QmYSEAKnkSLEEVc9E1uxtVigunKMFLk2cdu71Bse8Tqec3HyT, Piece ID: YFKD7GO2FOSPLDXNFPK732QTLCLBQQASWBTMZ2LTLBGSK22PIXIQ) context deadline exceeded; piecestore: (Node ID: 1JsnHuhQhM4rWLTLEuSSHLNE24m7tSW9qMSf48j7yMrAVPWVyV, Piece ID: ARBMDAKX4QBW6JG47KP2PUITCX7HKXCBO75THQPINM4LLHCKAV4Q) context deadline exceeded ======LOGS====== upload=433902]: Uploading piece... Dec 26 05:50:34.986155265 (494h21m34.8572383s) [noise=true , address=64.49.13.140:4509 , piece_id=6TGWTOAVY4YVAKU6VEE2SWVAM264TKU6ZNMGPWH4TNGII3UYXFXQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.986703610 (494h21m34.857786655s) [noise=true , address=77.90.31.10:4522 , piece_id=YFKD7GO2FOSPLDXNFPK732QTLCLBQQASWBTMZ2LTLBGSK22PIXIQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.987145380 (494h21m34.858228425s) [noise=true , address=145.79.227.33:4890 , piece_id=UKNB4ZZTYJGY4MDHOBPYUTHCP3ZDCMJB7O7AJZFYP2HFVIWLUOKQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.987628290 (494h21m34.858711335s) [noise=true , address=145.79.227.40:4052 , piece_id=XRZYL7PT5VHY22N7CCKS3NTZKDDWE36ZU3SORF6ILM7ZMXDND63A, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.988710731 (494h21m34.859793776s) [noise=true , address=77.90.31.13:4653 , piece_id=EY2CWX54ONOWPO3WC33G5QSP5RMP2SUTBQLUOH5JXGDRUEPDHQ2A, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.989894545 (494h21m34.86097759s) [noise=true , address=145.79.227.30:4785 , piece_id=5KMLU4ATBN4YVQDYJKKQY4PEJNKWTO2WNX5GMGJM4F54B6YM2SFA, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.991305081 (494h21m34.862388116s) [noise=true , address=145.79.227.50:4381 , piece_id=H6Q47DIOTOWGN7QH2KWT4IH6KVDSB246S22753Z3UQGB26T6VXPA, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.995796263 (494h21m34.866879359s) [noise=true , address=145.79.227.52:4461 , piece_id=ZAAUDBGY6CC2JNEGWXVPCBAWXARHAJDZ3YPB6Y3NWEWCFDUFGDSQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.996765439 (494h21m34.867848484s) [noise=true , address=77.90.31.16:4755 , piece_id=ARBMDAKX4QBW6JG47KP2PUITCX7HKXCBO75THQPINM4LLHCKAV4Q, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.998107634 (494h21m34.869190679s) [noise=true , address=77.90.31.11:4579 , piece_id=SBCEX7UNWGNFF54SBP4AGRDV5IPG24FQJXJJ7EYQTBSZKOVZWU4Q, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.999340442 (494h21m34.870423477s) [noise=true , address=145.79.227.34:4936 , piece_id=EEA2ZPM44DX3WMI7ZJTDC7T5UIGED6TNT53QXZWGBFRMQUQSCXPQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:35.000540757 (494h21m34.871623812s) [noise=true , address=45.91.116.14:4204 , piece_id=XV7FQGBJNWVRLOAFRUPOWON35JR5ESRJPTCWPUGYWGHJPSDVL4GA, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:36.360077064 (494h21m36.231160099s) [upload=433902]: Next returned nil segment Dec 26 05:50:36.360091642 (494h21m36.231174688s) [upload=433902]: Waiting for error group managing segments... Dec 26 05:50:36.688914299 (494h21m36.559997344s) [noise=true , address=64.49.13.140:4509 , piece_id=6TGWTOAVY4YVAKU6VEE2SWVAM264TKU6ZNMGPWH4TNGII3UYXFXQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.059812634 (494h21m36.930895669s) [noise=true , address=77.90.31.11:4579 , piece_id=SBCEX7UNWGNFF54SBP4AGRDV5IPG24FQJXJJ7EYQTBSZKOVZWU4Q, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.119433874 (494h21m36.990516909s) [noise=true , address=45.91.116.14:4204 , piece_id=XV7FQGBJNWVRLOAFRUPOWON35JR5ESRJPTCWPUGYWGHJPSDVL4GA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.138718204 (494h21m37.00980125s) [noise=true , address=77.90.31.16:4755 , piece_id=ARBMDAKX4QBW6JG47KP2PUITCX7HKXCBO75THQPINM4LLHCKAV4Q, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.146647243 (494h21m37.017730278s) [noise=true , address=77.90.31.13:4653 , piece_id=EY2CWX54ONOWPO3WC33G5QSP5RMP2SUTBQLUOH5JXGDRUEPDHQ2A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.163283722 (494h21m37.034366757s) [noise=true , address=77.90.31.21:4893 , piece_id=GNEEZO2XGOKNODOUPZQ4NGKFE4PCX5LNUWXLV27F6X7GWZVSYNAA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.374956574 (494h21m37.246039609s) [noise=true , address=45.91.116.7:4033 , piece_id=ATPIXLMXGONSSLXPPVORSNC6LHPXI2V3UFTHM4VHA2FTY5ZM6H3A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.777283355 (494h21m37.64836639s) [noise=true , address=77.90.31.12:4622 , piece_id=LTYYPK6ZQ3NXZPBKSV4OMZGCOXXV3N6L5A6MVODNR644D2ZUJX3A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:38.439025909 (494h21m38.310108944s) [noise=true , address=145.79.227.49:4355 , piece_id=IUDOHR73Y6OKFAQSNNSONALWAITRJ4POYRO2QD5LYZKOSTWY4MNA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:38.439599111 (494h21m38.310682166s) [noise=true , address=77.90.31.10:4522 , piece_id=YFKD7GO2FOSPLDXNFPK732QTLCLBQQASWBTMZ2LTLBGSK22PIXIQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:39.259154980 (494h21m39.130238015s) [noise=true , address=145.79.227.34:4936 , piece_id=EEA2ZPM44DX3WMI7ZJTDC7T5UIGED6TNT53QXZWGBFRMQUQSCXPQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:39.430935921 (494h21m39.302018956s) [noise=true , address=145.79.227.44:4212 , piece_id=ABBNLKRE4FM7RHF56V4IAQPKBCCCQLLOMFVMSJEWQUNFGGCI3PVQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:42.941071297 (494h21m42.812154352s) [noise=true , address=77.90.31.14:4675 , piece_id=FKB4O45OYICMELPGXB65QAUD2DRIFTSXZFKIA6TY26VFAP6AVY3A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:51:22.719026850 (494h22m22.590109885s) [noise=true , address=145.79.227.33:4890 , piece_id=UKNB4ZZTYJGY4MDHOBPYUTHCP3ZDCMJB7O7AJZFYP2HFVIWLUOKQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:51:22.719060193 (494h22m22.590143228s) [seg_pos={0 0}, upload=433902]: Segment reached optimal threshold of 14 pieces. Dec 26 05:51:22.719233022 (494h22m22.590316067s) [noise=true , address=145.79.227.50:4381 , piece_id=H6Q47DIOTOWGN7QH2KWT4IH6KVDSB246S22753Z3UQGB26T6VXPA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zU35MWWaaC1npkP5qGcWBXAkfxGsd5UsdkM2nwX9x7wUGCDbY): protocol: write tcp 188.166.225.170:46970->145.79.227.50:4381: use of closed network connection; write tcp 188.166.225.170:46970->145.79.227.50:4381: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46970->145.79.227.50:4381: use of closed network connection Dec 26 05:51:22.719238783 (494h22m22.590321828s) [noise=true , address=145.79.227.50:4381 , piece_id=H6Q47DIOTOWGN7QH2KWT4IH6KVDSB246S22753Z3UQGB26T6VXPA, seg_pos={0 0}, upload=433902]: Long tail cancellation due to optimal threshold Dec 26 05:51:22.719251677 (494h22m22.590334712s) [noise=true , address=145.79.227.30:4785 , piece_id=5KMLU4ATBN4YVQDYJKKQY4PEJNKWTO2WNX5GMGJM4F54B6YM2SFA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gYH83ECV8wai3CZyBSU8NNJM1UsysLYHbQDwBVKZRyjsdnUwB): protocol: write tcp 188.166.225.170:58398->145.79.227.30:4785: use of closed network connection; write tcp 188.166.225.170:58398->145.79.227.30:4785: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58398->145.79.227.30:4785: use of closed network connection Dec 26 05:51:22.719260194 (494h22m22.590343229s) [noise=true , address=145.79.227.30:4785 , piece_id=5KMLU4ATBN4YVQDYJKKQY4PEJNKWTO2WNX5GMGJM4F54B6YM2SFA, seg_pos={0 0}, upload=433902]: Long tail cancellation due to optimal threshold Dec 26 05:51:22.719409408 (494h22m22.590492443s) [noise=true , address=145.79.227.40:4052 , piece_id=XRZYL7PT5VHY22N7CCKS3NTZKDDWE36ZU3SORF6ILM7ZMXDND63A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PViJBYAanwXRWhMq3WHR6k9Stxy8hjuY5FV9J7ErxCy4N1q3V): protocol: write tcp 188.166.225.170:45984->145.79.227.40:4052: use of closed network connection; write tcp 188.166.225.170:45984->145.79.227.40:4052: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45984->145.79.227.40:4052: use of closed network connection Dec 26 05:51:22.719415469 (494h22m22.590498515s) [noise=true , address=145.79.227.40:4052 , piece_id=XRZYL7PT5VHY22N7CCKS3NTZKDDWE36ZU3SORF6ILM7ZMXDND63A, seg_pos={0 0}, upload=433902]: Long tail cancellation due to optimal threshold Dec 26 05:51:22.719467057 (494h22m22.590550092s) [noise=true , address=145.79.227.52:4461 , piece_id=ZAAUDBGY6CC2JNEGWXVPCBAWXARHAJDZ3YPB6Y3NWEWCFDUFGDSQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fNrbSfErEQRnruJCJAVvkoNwQSPWnvNtxunHn6zrQegnNAkxA): protocol: write tcp 188.166.225.170:55966->145.79.227.52:4461: use of closed network connection; write tcp 188.166.225.170:55966->145.79.227.52:4461: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55966->145.79.227.52:4461: use of closed network connection Dec 26 05:51:22.719473931 (494h22m22.590556966s) [noise=true , address=145.79.227.52:4461 , piece_id=ZAAUDBGY6CC2JNEGWXVPCBAWXARHAJDZ3YPB6Y3NWEWCFDUFGDSQ, seg_pos={0 0}, upload=433902]: Long tail cancellation due to optimal threshold Dec 26 05:51:22.719491374 (494h22m22.590574409s) [seg_pos={0 0}, upload=433902]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 05:51:22.719562539 (494h22m22.590645574s) [upload=433902]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:51:22.719567228 (494h22m22.590650263s) [upload=433902]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:51:23.008169117 (494h22m22.879252162s) [upload=433902]: Done uploading segments... Dec 26 05:51:23.008191560 (494h22m22.879274605s) [upload=433902]: Upload finished. err: <nil> Duration: 32.415334261s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.............................X..............................] duration: 3.434377401s trace: https://jaeger.dev.storj.io/trace/630094597d401477 FAIL (2025-12-26 06:32:28 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.434406255s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [..X.X.......................X..........X.......X.X.......X.X] duration: 30.002770419s trace: https://jaeger.dev.storj.io/trace/1f1d0344ea514d5 FAIL (2025-12-26 06:32:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: BLMLERZPBMLNTU3HZCSNLVH3KUSYJXI2FBEPCR445I3IUFKO7RNQ) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: RZE4W6XP3ZZ33SURGR3ES44EJE5RKDXRQBZS5MHMQ2F7VPATCAJQ) context deadline exceeded; piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: D5ENMNENE6SXNU7HQV2K5LFHUJWZWAEHPKYM2ZNFXMFEZJJBDGBA) context deadline exceeded ======LOGS====== Dec 26 06:31:39.479209150 (495h2m39.350292185s) [upload=434504]: Starting upload Dec 26 06:31:39.479309000 (495h2m39.350392035s) [upload=434504]: Done starting upload Dec 26 06:31:39.479369316 (495h2m39.350452351s) [upload=434504]: Uploading segments... Dec 26 06:31:39.479591258 (495h2m39.350674303s) [upload=434504]: Got next segment. Inline: false Dec 26 06:31:39.479651032 (495h2m39.350734067s) [upload=434504]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 06:31:39.479658185 (495h2m39.35074123s) [upload=434504]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 06:31:39.721689426 (495h2m39.592772461s) [seg_pos={0 0}, upload=434504]: Begin upload segment... Dec 26 06:31:39.721746355 (495h2m39.59282939s) [seg_pos={0 0}, upload=434504]: Done begin upload segment. Dec 26 06:31:39.721792702 (495h2m39.592875748s) [noise=true , address=145.79.227.17:4972 , piece_id=HSDSRT6DKUTLGN5ZX3E7IMIXNKBQ34NRUS7CRUC2NFX2QIHKVV7Q, seg_pos={0 0}, upload=434504]: Uploading piece... Dec 26 06:31:39.721872985 (495h2m39.59295602s) [noise=true , address=45.91.116.7:4009 , piece_id=QHG75INRC3QWMPE445SBWOCOIS5VGG4WL77AMT675W4OPYDXE3PQ, seg_pos={0 0}, upload=434504]: Uploading piece... Dec 26 06:31:39.722386323 (495h2m39.593469368s) [noise=true , address=145.79.227.30:4761 , piece_id=74BWE3S6PKVPRVOBW6PWC4MTSRTM2GLRG4VRLGVSOTMI6IXEYKJA, seg_pos={0 0}, upload=434504]: Uploading piece... Dec 26 06:31:39.723532966 (495h2m39.594616001s) [noise=true , address=45.91.116.12:4227 , piece_id=BR66LHYV7IRD3SCTHVDJOUVOQSGE3S23WEVXULYTYWKPCCOQZW5Q, seg_pos={0 0}, upload=434504]: Uploading piece... Dec 26 06:31:39.724031285 (495h2m39.59511432s) [noise=true , address=45.91.116.8:4045 , piece_id=JSX3ETE67ETTNQOTOP4D7YMVLOGVOYROXGBTZFWZV7NWUGPNHSYA, seg_pos={0 0}, upload=434504]: Uploading piece... Dec 26 06:31:39.724467145 (495h2m39.59555018s) [noise=true , address=145.79.227.39:4013 , piece_id=7QQBE6ZHGMF2XSTIJCE3RGIWEUDFYLGDOHXKPSY3OH72NXDXYHQA, seg_pos={0 0}, upload=434504]: Uploading piece... Dec 26 06:31:39.724887836 (495h2m39.595970881s) [noise=true , address=45.91.116.4:4939 , piece_id=QXV4MGEAZ4H76KR4TNX3OA5RKWNTUZIQKGVJ5KYSRJ6VUO6LOZEA, seg_pos={0 0}, upload=434504]: Uploading piece... Dec 26 06:31:39.726070027 (495h2m39.597153062s) [noise=true , address=45.91.116.6:4975 , piece_id=XJ77ZGFWUWKOTEP5LRICBHY7676OTYSQSBYDWM7MKAZMATNW5POQ, seg_pos={0 0}, upload=434504]: Uploading piece... Dec 26 06:31:39.726395948 (495h2m39.597478983s) [noise=true , address=145.79.227.3:4149 , piece_id=XA6Q7PDQTB47JT3K4B6UK2PPITTCDYXGVWFEBVBLOV7ANM2WXYRA, seg_pos={0 0}, upload=434504]: Uploading piece... Dec 26 06:31:39.727468821 (495h2m39.598551866s) [noise=true , address=145.79.227.48:4303 , piece_id=ANM6TVALEREUIXQJP3FDNCYLCWUPKVQXYEPPVTRX5HDUWASG67OA, seg_pos={0 0}, upload=434504]: Uploading piece... Dec 26 06:31:39.727842472 (495h2m39.598925508s) [noise=true , address=77.90.31.15:4700 , piece_id=BLMLERZPBMLNTU3HZCSNLVH3KUSYJXI2FBEPCR445I3IUFKO7RNQ, seg_pos={0 0}, upload=434504]: Uploading piece... Dec 26 06:31:39.729427731 (495h2m39.600510786s) [noise=true , address=145.79.227.13:4802 , piece_id=NH7IGTZ54LC5CZZWQTP33QK3GMYANXZSIC6MVPIPGWHP3Y6W5B4Q, seg_pos={0 0}, upload=434504]: Uploading piece... Dec 26 06:31:39.730912899 (495h2m39.601995944s) [noise=true , address=145.79.227.51:4410 , piece_id=KDCQ6LZU2BHPOXSCWAAQ37R3PUUORZ24CGETAE6GVTU4JBMZ4AOQ, seg_pos={0 0}, upload=434504]: Uploading piece... Dec 26 06:31:39.732298758 (495h2m39.603381793s) [noise=true , address=45.91.116.13:4155 , piece_id=RZE4W6XP3ZZ33SURGR3ES44EJE5RKDXRQBZS5MHMQ2F7VPATCAJQ, seg_pos={0 0}, upload=434504]: Uploading piece... Dec 26 06:31:39.734737612 (495h2m39.605820657s) [noise=true , address=45.91.116.14:4192 , piece_id=D5ENMNENE6SXNU7HQV2K5LFHUJWZWAEHPKYM2ZNFXMFEZJJBDGBA, seg_pos={0 0}, upload=434504]: Uploading piece... Dec 26 06:31:39.735646462 (495h2m39.606729497s) [noise=true , address=145.79.227.18:4007 , piece_id=6O3FCKQ2GCFHZOGVYOKPFNUCEXE5ZCQAF77P7FABBJQBNAH7KTNQ, seg_pos={0 0}, upload=434504]: Uploading piece... Dec 26 06:31:41.453322335 (495h2m41.32440537s) [upload=434504]: Next returned nil segment Dec 26 06:31:41.453336041 (495h2m41.324419076s) [upload=434504]: Waiting for error group managing segments... Dec 26 06:31:41.740730752 (495h2m41.611813788s) [noise=true , address=145.79.227.51:4410 , piece_id=KDCQ6LZU2BHPOXSCWAAQ37R3PUUORZ24CGETAE6GVTU4JBMZ4AOQ, seg_pos={0 0}, upload=434504]: Done uploading piece. err: <nil> Dec 26 06:31:41.747136009 (495h2m41.618219054s) [noise=true , address=145.79.227.39:4013 , piece_id=7QQBE6ZHGMF2XSTIJCE3RGIWEUDFYLGDOHXKPSY3OH72NXDXYHQA, seg_pos={0 0}, upload=434504]: Done uploading piece. err: <nil> Dec 26 06:31:41.753685651 (495h2m41.624768686s) [noise=true , address=145.79.227.48:4303 , piece_id=ANM6TVALEREUIXQJP3FDNCYLCWUPKVQXYEPPVTRX5HDUWASG67OA, seg_pos={0 0}, upload=434504]: Done uploading piece. err: <nil> Dec 26 06:31:41.756198776 (495h2m41.627281841s) [noise=true , address=145.79.227.17:4972 , piece_id=HSDSRT6DKUTLGN5ZX3E7IMIXNKBQ34NRUS7CRUC2NFX2QIHKVV7Q, seg_pos={0 0}, upload=434504]: Done uploading piece. err: <nil> Dec 26 06:31:41.761657541 (495h2m41.632740586s) [noise=true , address=145.79.227.3:4149 , piece_id=XA6Q7PDQTB47JT3K4B6UK2PPITTCDYXGVWFEBVBLOV7ANM2WXYRA, seg_pos={0 0}, upload=434504]: Done uploading piece. err: <nil> Dec 26 06:31:41.764401165 (495h2m41.63548421s) [noise=true , address=145.79.227.18:4007 , piece_id=6O3FCKQ2GCFHZOGVYOKPFNUCEXE5ZCQAF77P7FABBJQBNAH7KTNQ, seg_pos={0 0}, upload=434504]: Done uploading piece. err: <nil> Dec 26 06:31:41.783526082 (495h2m41.654609117s) [noise=true , address=145.79.227.13:4802 , piece_id=NH7IGTZ54LC5CZZWQTP33QK3GMYANXZSIC6MVPIPGWHP3Y6W5B4Q, seg_pos={0 0}, upload=434504]: Done uploading piece. err: <nil> Dec 26 06:31:41.963940944 (495h2m41.835023989s) [noise=true , address=45.91.116.4:4939 , piece_id=QXV4MGEAZ4H76KR4TNX3OA5RKWNTUZIQKGVJ5KYSRJ6VUO6LOZEA, seg_pos={0 0}, upload=434504]: Done uploading piece. err: <nil> Dec 26 06:31:41.988624225 (495h2m41.85970726s) [noise=true , address=45.91.116.6:4975 , piece_id=XJ77ZGFWUWKOTEP5LRICBHY7676OTYSQSBYDWM7MKAZMATNW5POQ, seg_pos={0 0}, upload=434504]: Done uploading piece. err: <nil> Dec 26 06:31:41.997763819 (495h2m41.868846864s) [noise=true , address=45.91.116.14:4192 , piece_id=D5ENMNENE6SXNU7HQV2K5LFHUJWZWAEHPKYM2ZNFXMFEZJJBDGBA, seg_pos={0 0}, upload=434504]: Done uploading piece. err: <nil> Dec 26 06:31:42.016237286 (495h2m41.887320321s) [noise=true , address=77.90.31.15:4700 , piece_id=BLMLERZPBMLNTU3HZCSNLVH3KUSYJXI2FBEPCR445I3IUFKO7RNQ, seg_pos={0 0}, upload=434504]: Done uploading piece. err: <nil> Dec 26 06:31:42.079360894 (495h2m41.950443939s) [noise=true , address=45.91.116.13:4155 , piece_id=RZE4W6XP3ZZ33SURGR3ES44EJE5RKDXRQBZS5MHMQ2F7VPATCAJQ, seg_pos={0 0}, upload=434504]: Done uploading piece. err: <nil> Dec 26 06:31:42.087559917 (495h2m41.958642962s) [noise=true , address=45.91.116.7:4009 , piece_id=QHG75INRC3QWMPE445SBWOCOIS5VGG4WL77AMT675W4OPYDXE3PQ, seg_pos={0 0}, upload=434504]: Done uploading piece. err: <nil> Dec 26 06:31:42.087584193 (495h2m41.958667238s) [seg_pos={0 0}, upload=434504]: Segment reached optimal threshold of 13 pieces. Dec 26 06:31:42.087869707 (495h2m41.958952762s) [noise=true , address=45.91.116.8:4045 , piece_id=JSX3ETE67ETTNQOTOP4D7YMVLOGVOYROXGBTZFWZV7NWUGPNHSYA, seg_pos={0 0}, upload=434504]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: write tcp 188.166.225.170:57512->45.91.116.8:4045: use of closed network connection; write tcp 188.166.225.170:57512->45.91.116.8:4045: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57512->45.91.116.8:4045: use of closed network connection Dec 26 06:31:42.087878664 (495h2m41.958961709s) [noise=true , address=45.91.116.8:4045 , piece_id=JSX3ETE67ETTNQOTOP4D7YMVLOGVOYROXGBTZFWZV7NWUGPNHSYA, seg_pos={0 0}, upload=434504]: Long tail cancellation due to optimal threshold Dec 26 06:31:42.088178335 (495h2m41.95926138s) [noise=true , address=145.79.227.30:4761 , piece_id=74BWE3S6PKVPRVOBW6PWC4MTSRTM2GLRG4VRLGVSOTMI6IXEYKJA, seg_pos={0 0}, upload=434504]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 26 06:31:42.088185909 (495h2m41.959268954s) [noise=true , address=145.79.227.30:4761 , piece_id=74BWE3S6PKVPRVOBW6PWC4MTSRTM2GLRG4VRLGVSOTMI6IXEYKJA, seg_pos={0 0}, upload=434504]: Long tail cancellation due to optimal threshold Dec 26 06:31:42.088319363 (495h2m41.959402398s) [noise=true , address=45.91.116.12:4227 , piece_id=BR66LHYV7IRD3SCTHVDJOUVOQSGE3S23WEVXULYTYWKPCCOQZW5Q, seg_pos={0 0}, upload=434504]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Dec 26 06:31:42.088326026 (495h2m41.959409071s) [noise=true , address=45.91.116.12:4227 , piece_id=BR66LHYV7IRD3SCTHVDJOUVOQSGE3S23WEVXULYTYWKPCCOQZW5Q, seg_pos={0 0}, upload=434504]: Long tail cancellation due to optimal threshold Dec 26 06:31:42.088343508 (495h2m41.959426553s) [seg_pos={0 0}, upload=434504]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 06:31:42.088420836 (495h2m41.959503891s) [upload=434504]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 06:31:42.088428320 (495h2m41.959511355s) [upload=434504]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 06:31:42.385278857 (495h2m42.256361892s) [upload=434504]: Done uploading segments... Dec 26 06:31:42.385297132 (495h2m42.256380177s) [upload=434504]: Upload finished. err: <nil> Duration: 30.002799915s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 37.447994578s trace: https://jaeger.dev.storj.io/trace/7e198d4c8fea54d2 FAIL (2025-12-26 06:32:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p, Piece ID: SBCEX7UNWGNFF54SBP4AGRDV5IPG24FQJXJJ7EYQTBSZKOVZWU4Q) context deadline exceeded; piecestore: (Node ID: 1nP8xospdMuDEh3em5K6iBnh8bHXqLhvHy4pHm9m8Lnj4kEH2X, Piece ID: XV7FQGBJNWVRLOAFRUPOWON35JR5ESRJPTCWPUGYWGHJPSDVL4GA) context deadline exceeded; piecestore: (Node ID: 1Mg5mxUjW56quDidSgA788D7ByfAzvsEuWJqwtzY4bLyxRn3mu, Piece ID: ATPIXLMXGONSSLXPPVORSNC6LHPXI2V3UFTHM4VHA2FTY5ZM6H3A) context deadline exceeded; piecestore: (Node ID: 12QmYSEAKnkSLEEVc9E1uxtVigunKMFLk2cdu71Bse8Tqec3HyT, Piece ID: YFKD7GO2FOSPLDXNFPK732QTLCLBQQASWBTMZ2LTLBGSK22PIXIQ) context deadline exceeded; piecestore: (Node ID: 1JsnHuhQhM4rWLTLEuSSHLNE24m7tSW9qMSf48j7yMrAVPWVyV, Piece ID: ARBMDAKX4QBW6JG47KP2PUITCX7HKXCBO75THQPINM4LLHCKAV4Q) context deadline exceeded ======LOGS====== upload=433902]: Uploading piece... Dec 26 05:50:34.986155265 (494h21m34.8572383s) [noise=true , address=64.49.13.140:4509 , piece_id=6TGWTOAVY4YVAKU6VEE2SWVAM264TKU6ZNMGPWH4TNGII3UYXFXQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.986703610 (494h21m34.857786655s) [noise=true , address=77.90.31.10:4522 , piece_id=YFKD7GO2FOSPLDXNFPK732QTLCLBQQASWBTMZ2LTLBGSK22PIXIQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.987145380 (494h21m34.858228425s) [noise=true , address=145.79.227.33:4890 , piece_id=UKNB4ZZTYJGY4MDHOBPYUTHCP3ZDCMJB7O7AJZFYP2HFVIWLUOKQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.987628290 (494h21m34.858711335s) [noise=true , address=145.79.227.40:4052 , piece_id=XRZYL7PT5VHY22N7CCKS3NTZKDDWE36ZU3SORF6ILM7ZMXDND63A, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.988710731 (494h21m34.859793776s) [noise=true , address=77.90.31.13:4653 , piece_id=EY2CWX54ONOWPO3WC33G5QSP5RMP2SUTBQLUOH5JXGDRUEPDHQ2A, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.989894545 (494h21m34.86097759s) [noise=true , address=145.79.227.30:4785 , piece_id=5KMLU4ATBN4YVQDYJKKQY4PEJNKWTO2WNX5GMGJM4F54B6YM2SFA, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.991305081 (494h21m34.862388116s) [noise=true , address=145.79.227.50:4381 , piece_id=H6Q47DIOTOWGN7QH2KWT4IH6KVDSB246S22753Z3UQGB26T6VXPA, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.995796263 (494h21m34.866879359s) [noise=true , address=145.79.227.52:4461 , piece_id=ZAAUDBGY6CC2JNEGWXVPCBAWXARHAJDZ3YPB6Y3NWEWCFDUFGDSQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.996765439 (494h21m34.867848484s) [noise=true , address=77.90.31.16:4755 , piece_id=ARBMDAKX4QBW6JG47KP2PUITCX7HKXCBO75THQPINM4LLHCKAV4Q, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.998107634 (494h21m34.869190679s) [noise=true , address=77.90.31.11:4579 , piece_id=SBCEX7UNWGNFF54SBP4AGRDV5IPG24FQJXJJ7EYQTBSZKOVZWU4Q, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.999340442 (494h21m34.870423477s) [noise=true , address=145.79.227.34:4936 , piece_id=EEA2ZPM44DX3WMI7ZJTDC7T5UIGED6TNT53QXZWGBFRMQUQSCXPQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:35.000540757 (494h21m34.871623812s) [noise=true , address=45.91.116.14:4204 , piece_id=XV7FQGBJNWVRLOAFRUPOWON35JR5ESRJPTCWPUGYWGHJPSDVL4GA, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:36.360077064 (494h21m36.231160099s) [upload=433902]: Next returned nil segment Dec 26 05:50:36.360091642 (494h21m36.231174688s) [upload=433902]: Waiting for error group managing segments... Dec 26 05:50:36.688914299 (494h21m36.559997344s) [noise=true , address=64.49.13.140:4509 , piece_id=6TGWTOAVY4YVAKU6VEE2SWVAM264TKU6ZNMGPWH4TNGII3UYXFXQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.059812634 (494h21m36.930895669s) [noise=true , address=77.90.31.11:4579 , piece_id=SBCEX7UNWGNFF54SBP4AGRDV5IPG24FQJXJJ7EYQTBSZKOVZWU4Q, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.119433874 (494h21m36.990516909s) [noise=true , address=45.91.116.14:4204 , piece_id=XV7FQGBJNWVRLOAFRUPOWON35JR5ESRJPTCWPUGYWGHJPSDVL4GA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.138718204 (494h21m37.00980125s) [noise=true , address=77.90.31.16:4755 , piece_id=ARBMDAKX4QBW6JG47KP2PUITCX7HKXCBO75THQPINM4LLHCKAV4Q, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.146647243 (494h21m37.017730278s) [noise=true , address=77.90.31.13:4653 , piece_id=EY2CWX54ONOWPO3WC33G5QSP5RMP2SUTBQLUOH5JXGDRUEPDHQ2A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.163283722 (494h21m37.034366757s) [noise=true , address=77.90.31.21:4893 , piece_id=GNEEZO2XGOKNODOUPZQ4NGKFE4PCX5LNUWXLV27F6X7GWZVSYNAA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.374956574 (494h21m37.246039609s) [noise=true , address=45.91.116.7:4033 , piece_id=ATPIXLMXGONSSLXPPVORSNC6LHPXI2V3UFTHM4VHA2FTY5ZM6H3A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.777283355 (494h21m37.64836639s) [noise=true , address=77.90.31.12:4622 , piece_id=LTYYPK6ZQ3NXZPBKSV4OMZGCOXXV3N6L5A6MVODNR644D2ZUJX3A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:38.439025909 (494h21m38.310108944s) [noise=true , address=145.79.227.49:4355 , piece_id=IUDOHR73Y6OKFAQSNNSONALWAITRJ4POYRO2QD5LYZKOSTWY4MNA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:38.439599111 (494h21m38.310682166s) [noise=true , address=77.90.31.10:4522 , piece_id=YFKD7GO2FOSPLDXNFPK732QTLCLBQQASWBTMZ2LTLBGSK22PIXIQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:39.259154980 (494h21m39.130238015s) [noise=true , address=145.79.227.34:4936 , piece_id=EEA2ZPM44DX3WMI7ZJTDC7T5UIGED6TNT53QXZWGBFRMQUQSCXPQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:39.430935921 (494h21m39.302018956s) [noise=true , address=145.79.227.44:4212 , piece_id=ABBNLKRE4FM7RHF56V4IAQPKBCCCQLLOMFVMSJEWQUNFGGCI3PVQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:42.941071297 (494h21m42.812154352s) [noise=true , address=77.90.31.14:4675 , piece_id=FKB4O45OYICMELPGXB65QAUD2DRIFTSXZFKIA6TY26VFAP6AVY3A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:51:22.719026850 (494h22m22.590109885s) [noise=true , address=145.79.227.33:4890 , piece_id=UKNB4ZZTYJGY4MDHOBPYUTHCP3ZDCMJB7O7AJZFYP2HFVIWLUOKQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:51:22.719060193 (494h22m22.590143228s) [seg_pos={0 0}, upload=433902]: Segment reached optimal threshold of 14 pieces. Dec 26 05:51:22.719233022 (494h22m22.590316067s) [noise=true , address=145.79.227.50:4381 , piece_id=H6Q47DIOTOWGN7QH2KWT4IH6KVDSB246S22753Z3UQGB26T6VXPA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zU35MWWaaC1npkP5qGcWBXAkfxGsd5UsdkM2nwX9x7wUGCDbY): protocol: write tcp 188.166.225.170:46970->145.79.227.50:4381: use of closed network connection; write tcp 188.166.225.170:46970->145.79.227.50:4381: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46970->145.79.227.50:4381: use of closed network connection Dec 26 05:51:22.719238783 (494h22m22.590321828s) [noise=true , address=145.79.227.50:4381 , piece_id=H6Q47DIOTOWGN7QH2KWT4IH6KVDSB246S22753Z3UQGB26T6VXPA, seg_pos={0 0}, upload=433902]: Long tail cancellation due to optimal threshold Dec 26 05:51:22.719251677 (494h22m22.590334712s) [noise=true , address=145.79.227.30:4785 , piece_id=5KMLU4ATBN4YVQDYJKKQY4PEJNKWTO2WNX5GMGJM4F54B6YM2SFA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gYH83ECV8wai3CZyBSU8NNJM1UsysLYHbQDwBVKZRyjsdnUwB): protocol: write tcp 188.166.225.170:58398->145.79.227.30:4785: use of closed network connection; write tcp 188.166.225.170:58398->145.79.227.30:4785: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58398->145.79.227.30:4785: use of closed network connection Dec 26 05:51:22.719260194 (494h22m22.590343229s) [noise=true , address=145.79.227.30:4785 , piece_id=5KMLU4ATBN4YVQDYJKKQY4PEJNKWTO2WNX5GMGJM4F54B6YM2SFA, seg_pos={0 0}, upload=433902]: Long tail cancellation due to optimal threshold Dec 26 05:51:22.719409408 (494h22m22.590492443s) [noise=true , address=145.79.227.40:4052 , piece_id=XRZYL7PT5VHY22N7CCKS3NTZKDDWE36ZU3SORF6ILM7ZMXDND63A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PViJBYAanwXRWhMq3WHR6k9Stxy8hjuY5FV9J7ErxCy4N1q3V): protocol: write tcp 188.166.225.170:45984->145.79.227.40:4052: use of closed network connection; write tcp 188.166.225.170:45984->145.79.227.40:4052: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45984->145.79.227.40:4052: use of closed network connection Dec 26 05:51:22.719415469 (494h22m22.590498515s) [noise=true , address=145.79.227.40:4052 , piece_id=XRZYL7PT5VHY22N7CCKS3NTZKDDWE36ZU3SORF6ILM7ZMXDND63A, seg_pos={0 0}, upload=433902]: Long tail cancellation due to optimal threshold Dec 26 05:51:22.719467057 (494h22m22.590550092s) [noise=true , address=145.79.227.52:4461 , piece_id=ZAAUDBGY6CC2JNEGWXVPCBAWXARHAJDZ3YPB6Y3NWEWCFDUFGDSQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fNrbSfErEQRnruJCJAVvkoNwQSPWnvNtxunHn6zrQegnNAkxA): protocol: write tcp 188.166.225.170:55966->145.79.227.52:4461: use of closed network connection; write tcp 188.166.225.170:55966->145.79.227.52:4461: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55966->145.79.227.52:4461: use of closed network connection Dec 26 05:51:22.719473931 (494h22m22.590556966s) [noise=true , address=145.79.227.52:4461 , piece_id=ZAAUDBGY6CC2JNEGWXVPCBAWXARHAJDZ3YPB6Y3NWEWCFDUFGDSQ, seg_pos={0 0}, upload=433902]: Long tail cancellation due to optimal threshold Dec 26 05:51:22.719491374 (494h22m22.590574409s) [seg_pos={0 0}, upload=433902]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 05:51:22.719562539 (494h22m22.590645574s) [upload=433902]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:51:22.719567228 (494h22m22.590650263s) [upload=433902]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:51:23.008169117 (494h22m22.879252162s) [upload=433902]: Done uploading segments... Dec 26 05:51:23.008191560 (494h22m22.879274605s) [upload=433902]: Upload finished. err: <nil> Duration: 37.44801105s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..............................X.............................] duration: 3.465624061s trace: https://jaeger.dev.storj.io/trace/630094597d401477 FAIL (2025-12-26 06:31:25 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.465652724s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 25.246937912s trace: https://jaeger.dev.storj.io/trace/7e198d4c8fea54d2 FAIL (2025-12-26 06:31:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p, Piece ID: SBCEX7UNWGNFF54SBP4AGRDV5IPG24FQJXJJ7EYQTBSZKOVZWU4Q) context deadline exceeded; piecestore: (Node ID: 1nP8xospdMuDEh3em5K6iBnh8bHXqLhvHy4pHm9m8Lnj4kEH2X, Piece ID: XV7FQGBJNWVRLOAFRUPOWON35JR5ESRJPTCWPUGYWGHJPSDVL4GA) context deadline exceeded; piecestore: (Node ID: 1Mg5mxUjW56quDidSgA788D7ByfAzvsEuWJqwtzY4bLyxRn3mu, Piece ID: ATPIXLMXGONSSLXPPVORSNC6LHPXI2V3UFTHM4VHA2FTY5ZM6H3A) context deadline exceeded; piecestore: (Node ID: 12QmYSEAKnkSLEEVc9E1uxtVigunKMFLk2cdu71Bse8Tqec3HyT, Piece ID: YFKD7GO2FOSPLDXNFPK732QTLCLBQQASWBTMZ2LTLBGSK22PIXIQ) context deadline exceeded; piecestore: (Node ID: 1JsnHuhQhM4rWLTLEuSSHLNE24m7tSW9qMSf48j7yMrAVPWVyV, Piece ID: ARBMDAKX4QBW6JG47KP2PUITCX7HKXCBO75THQPINM4LLHCKAV4Q) context deadline exceeded ======LOGS====== upload=433902]: Uploading piece... Dec 26 05:50:34.986155265 (494h21m34.8572383s) [noise=true , address=64.49.13.140:4509 , piece_id=6TGWTOAVY4YVAKU6VEE2SWVAM264TKU6ZNMGPWH4TNGII3UYXFXQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.986703610 (494h21m34.857786655s) [noise=true , address=77.90.31.10:4522 , piece_id=YFKD7GO2FOSPLDXNFPK732QTLCLBQQASWBTMZ2LTLBGSK22PIXIQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.987145380 (494h21m34.858228425s) [noise=true , address=145.79.227.33:4890 , piece_id=UKNB4ZZTYJGY4MDHOBPYUTHCP3ZDCMJB7O7AJZFYP2HFVIWLUOKQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.987628290 (494h21m34.858711335s) [noise=true , address=145.79.227.40:4052 , piece_id=XRZYL7PT5VHY22N7CCKS3NTZKDDWE36ZU3SORF6ILM7ZMXDND63A, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.988710731 (494h21m34.859793776s) [noise=true , address=77.90.31.13:4653 , piece_id=EY2CWX54ONOWPO3WC33G5QSP5RMP2SUTBQLUOH5JXGDRUEPDHQ2A, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.989894545 (494h21m34.86097759s) [noise=true , address=145.79.227.30:4785 , piece_id=5KMLU4ATBN4YVQDYJKKQY4PEJNKWTO2WNX5GMGJM4F54B6YM2SFA, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.991305081 (494h21m34.862388116s) [noise=true , address=145.79.227.50:4381 , piece_id=H6Q47DIOTOWGN7QH2KWT4IH6KVDSB246S22753Z3UQGB26T6VXPA, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.995796263 (494h21m34.866879359s) [noise=true , address=145.79.227.52:4461 , piece_id=ZAAUDBGY6CC2JNEGWXVPCBAWXARHAJDZ3YPB6Y3NWEWCFDUFGDSQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.996765439 (494h21m34.867848484s) [noise=true , address=77.90.31.16:4755 , piece_id=ARBMDAKX4QBW6JG47KP2PUITCX7HKXCBO75THQPINM4LLHCKAV4Q, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.998107634 (494h21m34.869190679s) [noise=true , address=77.90.31.11:4579 , piece_id=SBCEX7UNWGNFF54SBP4AGRDV5IPG24FQJXJJ7EYQTBSZKOVZWU4Q, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.999340442 (494h21m34.870423477s) [noise=true , address=145.79.227.34:4936 , piece_id=EEA2ZPM44DX3WMI7ZJTDC7T5UIGED6TNT53QXZWGBFRMQUQSCXPQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:35.000540757 (494h21m34.871623812s) [noise=true , address=45.91.116.14:4204 , piece_id=XV7FQGBJNWVRLOAFRUPOWON35JR5ESRJPTCWPUGYWGHJPSDVL4GA, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:36.360077064 (494h21m36.231160099s) [upload=433902]: Next returned nil segment Dec 26 05:50:36.360091642 (494h21m36.231174688s) [upload=433902]: Waiting for error group managing segments... Dec 26 05:50:36.688914299 (494h21m36.559997344s) [noise=true , address=64.49.13.140:4509 , piece_id=6TGWTOAVY4YVAKU6VEE2SWVAM264TKU6ZNMGPWH4TNGII3UYXFXQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.059812634 (494h21m36.930895669s) [noise=true , address=77.90.31.11:4579 , piece_id=SBCEX7UNWGNFF54SBP4AGRDV5IPG24FQJXJJ7EYQTBSZKOVZWU4Q, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.119433874 (494h21m36.990516909s) [noise=true , address=45.91.116.14:4204 , piece_id=XV7FQGBJNWVRLOAFRUPOWON35JR5ESRJPTCWPUGYWGHJPSDVL4GA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.138718204 (494h21m37.00980125s) [noise=true , address=77.90.31.16:4755 , piece_id=ARBMDAKX4QBW6JG47KP2PUITCX7HKXCBO75THQPINM4LLHCKAV4Q, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.146647243 (494h21m37.017730278s) [noise=true , address=77.90.31.13:4653 , piece_id=EY2CWX54ONOWPO3WC33G5QSP5RMP2SUTBQLUOH5JXGDRUEPDHQ2A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.163283722 (494h21m37.034366757s) [noise=true , address=77.90.31.21:4893 , piece_id=GNEEZO2XGOKNODOUPZQ4NGKFE4PCX5LNUWXLV27F6X7GWZVSYNAA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.374956574 (494h21m37.246039609s) [noise=true , address=45.91.116.7:4033 , piece_id=ATPIXLMXGONSSLXPPVORSNC6LHPXI2V3UFTHM4VHA2FTY5ZM6H3A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.777283355 (494h21m37.64836639s) [noise=true , address=77.90.31.12:4622 , piece_id=LTYYPK6ZQ3NXZPBKSV4OMZGCOXXV3N6L5A6MVODNR644D2ZUJX3A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:38.439025909 (494h21m38.310108944s) [noise=true , address=145.79.227.49:4355 , piece_id=IUDOHR73Y6OKFAQSNNSONALWAITRJ4POYRO2QD5LYZKOSTWY4MNA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:38.439599111 (494h21m38.310682166s) [noise=true , address=77.90.31.10:4522 , piece_id=YFKD7GO2FOSPLDXNFPK732QTLCLBQQASWBTMZ2LTLBGSK22PIXIQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:39.259154980 (494h21m39.130238015s) [noise=true , address=145.79.227.34:4936 , piece_id=EEA2ZPM44DX3WMI7ZJTDC7T5UIGED6TNT53QXZWGBFRMQUQSCXPQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:39.430935921 (494h21m39.302018956s) [noise=true , address=145.79.227.44:4212 , piece_id=ABBNLKRE4FM7RHF56V4IAQPKBCCCQLLOMFVMSJEWQUNFGGCI3PVQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:42.941071297 (494h21m42.812154352s) [noise=true , address=77.90.31.14:4675 , piece_id=FKB4O45OYICMELPGXB65QAUD2DRIFTSXZFKIA6TY26VFAP6AVY3A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:51:22.719026850 (494h22m22.590109885s) [noise=true , address=145.79.227.33:4890 , piece_id=UKNB4ZZTYJGY4MDHOBPYUTHCP3ZDCMJB7O7AJZFYP2HFVIWLUOKQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:51:22.719060193 (494h22m22.590143228s) [seg_pos={0 0}, upload=433902]: Segment reached optimal threshold of 14 pieces. Dec 26 05:51:22.719233022 (494h22m22.590316067s) [noise=true , address=145.79.227.50:4381 , piece_id=H6Q47DIOTOWGN7QH2KWT4IH6KVDSB246S22753Z3UQGB26T6VXPA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zU35MWWaaC1npkP5qGcWBXAkfxGsd5UsdkM2nwX9x7wUGCDbY): protocol: write tcp 188.166.225.170:46970->145.79.227.50:4381: use of closed network connection; write tcp 188.166.225.170:46970->145.79.227.50:4381: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46970->145.79.227.50:4381: use of closed network connection Dec 26 05:51:22.719238783 (494h22m22.590321828s) [noise=true , address=145.79.227.50:4381 , piece_id=H6Q47DIOTOWGN7QH2KWT4IH6KVDSB246S22753Z3UQGB26T6VXPA, seg_pos={0 0}, upload=433902]: Long tail cancellation due to optimal threshold Dec 26 05:51:22.719251677 (494h22m22.590334712s) [noise=true , address=145.79.227.30:4785 , piece_id=5KMLU4ATBN4YVQDYJKKQY4PEJNKWTO2WNX5GMGJM4F54B6YM2SFA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gYH83ECV8wai3CZyBSU8NNJM1UsysLYHbQDwBVKZRyjsdnUwB): protocol: write tcp 188.166.225.170:58398->145.79.227.30:4785: use of closed network connection; write tcp 188.166.225.170:58398->145.79.227.30:4785: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58398->145.79.227.30:4785: use of closed network connection Dec 26 05:51:22.719260194 (494h22m22.590343229s) [noise=true , address=145.79.227.30:4785 , piece_id=5KMLU4ATBN4YVQDYJKKQY4PEJNKWTO2WNX5GMGJM4F54B6YM2SFA, seg_pos={0 0}, upload=433902]: Long tail cancellation due to optimal threshold Dec 26 05:51:22.719409408 (494h22m22.590492443s) [noise=true , address=145.79.227.40:4052 , piece_id=XRZYL7PT5VHY22N7CCKS3NTZKDDWE36ZU3SORF6ILM7ZMXDND63A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PViJBYAanwXRWhMq3WHR6k9Stxy8hjuY5FV9J7ErxCy4N1q3V): protocol: write tcp 188.166.225.170:45984->145.79.227.40:4052: use of closed network connection; write tcp 188.166.225.170:45984->145.79.227.40:4052: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45984->145.79.227.40:4052: use of closed network connection Dec 26 05:51:22.719415469 (494h22m22.590498515s) [noise=true , address=145.79.227.40:4052 , piece_id=XRZYL7PT5VHY22N7CCKS3NTZKDDWE36ZU3SORF6ILM7ZMXDND63A, seg_pos={0 0}, upload=433902]: Long tail cancellation due to optimal threshold Dec 26 05:51:22.719467057 (494h22m22.590550092s) [noise=true , address=145.79.227.52:4461 , piece_id=ZAAUDBGY6CC2JNEGWXVPCBAWXARHAJDZ3YPB6Y3NWEWCFDUFGDSQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fNrbSfErEQRnruJCJAVvkoNwQSPWnvNtxunHn6zrQegnNAkxA): protocol: write tcp 188.166.225.170:55966->145.79.227.52:4461: use of closed network connection; write tcp 188.166.225.170:55966->145.79.227.52:4461: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55966->145.79.227.52:4461: use of closed network connection Dec 26 05:51:22.719473931 (494h22m22.590556966s) [noise=true , address=145.79.227.52:4461 , piece_id=ZAAUDBGY6CC2JNEGWXVPCBAWXARHAJDZ3YPB6Y3NWEWCFDUFGDSQ, seg_pos={0 0}, upload=433902]: Long tail cancellation due to optimal threshold Dec 26 05:51:22.719491374 (494h22m22.590574409s) [seg_pos={0 0}, upload=433902]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 05:51:22.719562539 (494h22m22.590645574s) [upload=433902]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:51:22.719567228 (494h22m22.590650263s) [upload=433902]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:51:23.008169117 (494h22m22.879252162s) [upload=433902]: Done uploading segments... Dec 26 05:51:23.008191560 (494h22m22.879274605s) [upload=433902]: Upload finished. err: <nil> Duration: 25.247126201s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [X..X.X.......................X..........X.......X.X.......X.] duration: 5.767767972s trace: https://jaeger.dev.storj.io/trace/2b84215c812c3049 FAIL (2025-12-26 06:30:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: 6K7V4RON7P3EVFQKAPWF2Y44XIRB45BOZ4M6BRKFHAAMXAZ4TY6A) context deadline exceeded; piecestore: (Node ID: 1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks, Piece ID: RYJ3RJ43RD4L33OITBVNHBLUXQJVWWLILZSOI6K2QL7CGNXJABCA) context deadline exceeded ======LOGS====== 6 06:30:05.652429912 (495h1m5.523512957s) [upload=434481]: Starting upload Dec 26 06:30:05.652501037 (495h1m5.523584082s) [upload=434481]: Done starting upload Dec 26 06:30:05.652753277 (495h1m5.523836322s) [upload=434481]: Uploading segments... Dec 26 06:30:05.652821137 (495h1m5.523904182s) [upload=434481]: Got next segment. Inline: false Dec 26 06:30:05.652868386 (495h1m5.523951431s) [upload=434481]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 06:30:05.652874538 (495h1m5.523957583s) [upload=434481]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 06:30:05.904060291 (495h1m5.775143336s) [seg_pos={0 0}, upload=434481]: Begin upload segment... Dec 26 06:30:05.904128391 (495h1m5.775211436s) [seg_pos={0 0}, upload=434481]: Done begin upload segment. Dec 26 06:30:05.904182603 (495h1m5.775265679s) [noise=true , address=145.79.227.19:4077 , piece_id=DNGDGSHU4KE6N22SFPX6EUVTIOU7DKKKTTBIZCEHBYR6YRZJW3LQ, seg_pos={0 0}, upload=434481]: Uploading piece... Dec 26 06:30:05.904192783 (495h1m5.775275829s) [noise=true , address=45.91.116.10:4084 , piece_id=IZ6YPCBPWW6IMOVTRKKWOBLM7OSORHYL54K3WEK6UVRDD4OKIYOQ, seg_pos={0 0}, upload=434481]: Uploading piece... Dec 26 06:30:05.905323887 (495h1m5.776406932s) [noise=true , address=77.90.31.17:4773 , piece_id=6C4MQKYKSQECLTSJYG4X3Y6VQYRWRWASSIUKTZDJFKASKYJOKX6A, seg_pos={0 0}, upload=434481]: Uploading piece... Dec 26 06:30:05.905472490 (495h1m5.776555555s) [noise=true , address=45.91.116.4:4939 , piece_id=OUN2FPBDDS6T77MSIFQMMA7AGF2UF32UEVNPKWRXDXUEEZGIN4IA, seg_pos={0 0}, upload=434481]: Uploading piece... Dec 26 06:30:05.905760038 (495h1m5.776843083s) [noise=true , address=77.90.31.12:4590 , piece_id=6K7V4RON7P3EVFQKAPWF2Y44XIRB45BOZ4M6BRKFHAAMXAZ4TY6A, seg_pos={0 0}, upload=434481]: Uploading piece... Dec 26 06:30:05.906401048 (495h1m5.777484093s) [noise=true , address=45.91.116.6:4976 , piece_id=HVNG23JZFVJLONY7YINXTQOU7LU7N3MJASUNYTPJXN3VCTCOHZBA, seg_pos={0 0}, upload=434481]: Uploading piece... Dec 26 06:30:05.907466136 (495h1m5.778549172s) [noise=true , address=145.79.227.48:4303 , piece_id=X3KR2KWDNP26SVIZZFDYQZXPL3E6HRHR7CSMGNBY7BMS4OPSUADA, seg_pos={0 0}, upload=434481]: Uploading piece... Dec 26 06:30:05.908013268 (495h1m5.779096313s) [noise=true , address=45.91.116.14:4190 , piece_id=4UEXG5Z6AHXVIHP6GVRAJ4ZPYFAQXJN3Q4YGV6BWTKSACOXVCVBA, seg_pos={0 0}, upload=434481]: Uploading piece... Dec 26 06:30:05.908140751 (495h1m5.779223786s) [noise=true , address=45.91.116.5:4299 , piece_id=6FB4CF4OMEAT5V3CR52CFIZWVHZK6RUQZVRGF4GF7T7FRPMSW7ZQ, seg_pos={0 0}, upload=434481]: Uploading piece... Dec 26 06:30:05.908593403 (495h1m5.779676448s) [noise=true , address=45.91.116.11:4117 , piece_id=NKLNFDDG4BP6PBCIKIHFIKLMJTO3ETO5O6SQOHMQJ4GBOLZYPHEQ, seg_pos={0 0}, upload=434481]: Uploading piece... Dec 26 06:30:05.908742456 (495h1m5.779825501s) [noise=true , address=145.79.227.45:4229 , piece_id=KDC7U3EHMPASW5V6TZOUSQJ2LUOQUUH2ZAIGRRPR2PVNIH7L3TZA, seg_pos={0 0}, upload=434481]: Uploading piece... Dec 26 06:30:05.909056083 (495h1m5.780139128s) [noise=true , address=145.79.227.34:4906 , piece_id=5H35H572VSIMNMNNVBYBKPRQYNLO46C3NREKMLOX42NG6UG3H7MQ, seg_pos={0 0}, upload=434481]: Uploading piece... Dec 26 06:30:05.909299086 (495h1m5.780382131s) [noise=true , address=145.79.227.6:4257 , piece_id=LQS3NRXQIMAANUXSIAHIQQ2PKD2H754CGHVKUNDBVO3MZA7RTRBA, seg_pos={0 0}, upload=434481]: Uploading piece... Dec 26 06:30:05.909506471 (495h1m5.780589516s) [noise=true , address=45.91.116.8:4045 , piece_id=AT3SJHRPDB7EED4J7D745KI7FK2T3JZLKKOZYG6F5YCZKQ6FBQQA, seg_pos={0 0}, upload=434481]: Uploading piece... Dec 26 06:30:05.909984041 (495h1m5.781067086s) [noise=true , address=77.90.31.11:4556 , piece_id=RYJ3RJ43RD4L33OITBVNHBLUXQJVWWLILZSOI6K2QL7CGNXJABCA, seg_pos={0 0}, upload=434481]: Uploading piece... Dec 26 06:30:05.910803762 (495h1m5.781886837s) [noise=true , address=145.79.227.25:4617 , piece_id=PDFSA4MC3KETP6LH4ME2P7KKV34ZHORY4QDBGKTQDT4WDWTI7G5A, seg_pos={0 0}, upload=434481]: Uploading piece... Dec 26 06:30:07.616251981 (495h1m7.487335026s) [upload=434481]: Next returned nil segment Dec 26 06:30:07.616266658 (495h1m7.487349703s) [upload=434481]: Waiting for error group managing segments... Dec 26 06:30:07.913182599 (495h1m7.784265644s) [noise=true , address=145.79.227.25:4617 , piece_id=PDFSA4MC3KETP6LH4ME2P7KKV34ZHORY4QDBGKTQDT4WDWTI7G5A, seg_pos={0 0}, upload=434481]: Done uploading piece. err: <nil> Dec 26 06:30:07.913791419 (495h1m7.784874454s) [noise=true , address=145.79.227.48:4303 , piece_id=X3KR2KWDNP26SVIZZFDYQZXPL3E6HRHR7CSMGNBY7BMS4OPSUADA, seg_pos={0 0}, upload=434481]: Done uploading piece. err: <nil> Dec 26 06:30:07.916007719 (495h1m7.787090765s) [noise=true , address=145.79.227.19:4077 , piece_id=DNGDGSHU4KE6N22SFPX6EUVTIOU7DKKKTTBIZCEHBYR6YRZJW3LQ, seg_pos={0 0}, upload=434481]: Done uploading piece. err: <nil> Dec 26 06:30:07.917603387 (495h1m7.788686432s) [noise=true , address=145.79.227.6:4257 , piece_id=LQS3NRXQIMAANUXSIAHIQQ2PKD2H754CGHVKUNDBVO3MZA7RTRBA, seg_pos={0 0}, upload=434481]: Done uploading piece. err: <nil> Dec 26 06:30:07.930364206 (495h1m7.801447251s) [noise=true , address=145.79.227.45:4229 , piece_id=KDC7U3EHMPASW5V6TZOUSQJ2LUOQUUH2ZAIGRRPR2PVNIH7L3TZA, seg_pos={0 0}, upload=434481]: Done uploading piece. err: <nil> Dec 26 06:30:07.943685473 (495h1m7.814768518s) [noise=true , address=145.79.227.34:4906 , piece_id=5H35H572VSIMNMNNVBYBKPRQYNLO46C3NREKMLOX42NG6UG3H7MQ, seg_pos={0 0}, upload=434481]: Done uploading piece. err: <nil> Dec 26 06:30:08.123531070 (495h1m7.994614105s) [noise=true , address=45.91.116.5:4299 , piece_id=6FB4CF4OMEAT5V3CR52CFIZWVHZK6RUQZVRGF4GF7T7FRPMSW7ZQ, seg_pos={0 0}, upload=434481]: Done uploading piece. err: <nil> Dec 26 06:30:08.123704230 (495h1m7.994787275s) [noise=true , address=77.90.31.11:4556 , piece_id=RYJ3RJ43RD4L33OITBVNHBLUXQJVWWLILZSOI6K2QL7CGNXJABCA, seg_pos={0 0}, upload=434481]: Done uploading piece. err: <nil> Dec 26 06:30:08.225992686 (495h1m8.097075732s) [noise=true , address=45.91.116.8:4045 , piece_id=AT3SJHRPDB7EED4J7D745KI7FK2T3JZLKKOZYG6F5YCZKQ6FBQQA, seg_pos={0 0}, upload=434481]: Done uploading piece. err: <nil> Dec 26 06:30:08.242648572 (495h1m8.113731617s) [noise=true , address=45.91.116.10:4084 , piece_id=IZ6YPCBPWW6IMOVTRKKWOBLM7OSORHYL54K3WEK6UVRDD4OKIYOQ, seg_pos={0 0}, upload=434481]: Done uploading piece. err: <nil> Dec 26 06:30:08.284773425 (495h1m8.15585647s) [noise=true , address=45.91.116.14:4190 , piece_id=4UEXG5Z6AHXVIHP6GVRAJ4ZPYFAQXJN3Q4YGV6BWTKSACOXVCVBA, seg_pos={0 0}, upload=434481]: Done uploading piece. err: <nil> Dec 26 06:30:08.330394266 (495h1m8.201477301s) [noise=true , address=45.91.116.4:4939 , piece_id=OUN2FPBDDS6T77MSIFQMMA7AGF2UF32UEVNPKWRXDXUEEZGIN4IA, seg_pos={0 0}, upload=434481]: Done uploading piece. err: <nil> Dec 26 06:30:08.383522201 (495h1m8.254605246s) [noise=true , address=77.90.31.12:4590 , piece_id=6K7V4RON7P3EVFQKAPWF2Y44XIRB45BOZ4M6BRKFHAAMXAZ4TY6A, seg_pos={0 0}, upload=434481]: Done uploading piece. err: <nil> Dec 26 06:30:08.383545966 (495h1m8.254629011s) [seg_pos={0 0}, upload=434481]: Segment reached optimal threshold of 13 pieces. Dec 26 06:30:08.383760064 (495h1m8.254843109s) [noise=true , address=45.91.116.11:4117 , piece_id=NKLNFDDG4BP6PBCIKIHFIKLMJTO3ETO5O6SQOHMQJ4GBOLZYPHEQ, seg_pos={0 0}, upload=434481]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF Dec 26 06:30:08.383767397 (495h1m8.254850442s) [noise=true , address=45.91.116.11:4117 , piece_id=NKLNFDDG4BP6PBCIKIHFIKLMJTO3ETO5O6SQOHMQJ4GBOLZYPHEQ, seg_pos={0 0}, upload=434481]: Long tail cancellation due to optimal threshold Dec 26 06:30:08.383916571 (495h1m8.254999616s) [noise=true , address=77.90.31.17:4773 , piece_id=6C4MQKYKSQECLTSJYG4X3Y6VQYRWRWASSIUKTZDJFKASKYJOKX6A, seg_pos={0 0}, upload=434481]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: write tcp 188.166.225.170:55372->77.90.31.17:4773: use of closed network connection; write tcp 188.166.225.170:55372->77.90.31.17:4773: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55372->77.90.31.17:4773: use of closed network connection Dec 26 06:30:08.383928224 (495h1m8.255011289s) [noise=true , address=77.90.31.17:4773 , piece_id=6C4MQKYKSQECLTSJYG4X3Y6VQYRWRWASSIUKTZDJFKASKYJOKX6A, seg_pos={0 0}, upload=434481]: Long tail cancellation due to optimal threshold Dec 26 06:30:08.384115039 (495h1m8.255198084s) [noise=true , address=45.91.116.6:4976 , piece_id=HVNG23JZFVJLONY7YINXTQOU7LU7N3MJASUNYTPJXN3VCTCOHZBA, seg_pos={0 0}, upload=434481]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: read tcp 188.166.225.170:57032->45.91.116.6:4976: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 06:30:08.384123355 (495h1m8.2552064s) [noise=true , address=45.91.116.6:4976 , piece_id=HVNG23JZFVJLONY7YINXTQOU7LU7N3MJASUNYTPJXN3VCTCOHZBA, seg_pos={0 0}, upload=434481]: Long tail cancellation due to optimal threshold Dec 26 06:30:08.384203006 (495h1m8.255286041s) [seg_pos={0 0}, upload=434481]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 06:30:08.384306103 (495h1m8.255389148s) [upload=434481]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 06:30:08.384311824 (495h1m8.25539487s) [upload=434481]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 06:30:08.690044113 (495h1m8.561127158s) [upload=434481]: Done uploading segments... Dec 26 06:30:08.690070624 (495h1m8.561153669s) [upload=434481]: Upload finished. err: <nil> Duration: 5.76778215s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...............................X............................] duration: 3.472864486s trace: https://jaeger.dev.storj.io/trace/630094597d401477 FAIL (2025-12-26 06:30:38 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.472889483s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.X..X.X.......................X..........X.......X.X.......X] duration: 30.002138512s trace: https://jaeger.dev.storj.io/trace/2b84215c812c3049 FAIL (2025-12-26 06:30:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: 6K7V4RON7P3EVFQKAPWF2Y44XIRB45BOZ4M6BRKFHAAMXAZ4TY6A) context deadline exceeded; piecestore: (Node ID: 1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks, Piece ID: RYJ3RJ43RD4L33OITBVNHBLUXQJVWWLILZSOI6K2QL7CGNXJABCA) context deadline exceeded ======LOGS====== 6 06:30:05.652429912 (495h1m5.523512957s) [upload=434481]: Starting upload Dec 26 06:30:05.652501037 (495h1m5.523584082s) [upload=434481]: Done starting upload Dec 26 06:30:05.652753277 (495h1m5.523836322s) [upload=434481]: Uploading segments... Dec 26 06:30:05.652821137 (495h1m5.523904182s) [upload=434481]: Got next segment. Inline: false Dec 26 06:30:05.652868386 (495h1m5.523951431s) [upload=434481]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 06:30:05.652874538 (495h1m5.523957583s) [upload=434481]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 06:30:05.904060291 (495h1m5.775143336s) [seg_pos={0 0}, upload=434481]: Begin upload segment... Dec 26 06:30:05.904128391 (495h1m5.775211436s) [seg_pos={0 0}, upload=434481]: Done begin upload segment. Dec 26 06:30:05.904182603 (495h1m5.775265679s) [noise=true , address=145.79.227.19:4077 , piece_id=DNGDGSHU4KE6N22SFPX6EUVTIOU7DKKKTTBIZCEHBYR6YRZJW3LQ, seg_pos={0 0}, upload=434481]: Uploading piece... Dec 26 06:30:05.904192783 (495h1m5.775275829s) [noise=true , address=45.91.116.10:4084 , piece_id=IZ6YPCBPWW6IMOVTRKKWOBLM7OSORHYL54K3WEK6UVRDD4OKIYOQ, seg_pos={0 0}, upload=434481]: Uploading piece... Dec 26 06:30:05.905323887 (495h1m5.776406932s) [noise=true , address=77.90.31.17:4773 , piece_id=6C4MQKYKSQECLTSJYG4X3Y6VQYRWRWASSIUKTZDJFKASKYJOKX6A, seg_pos={0 0}, upload=434481]: Uploading piece... Dec 26 06:30:05.905472490 (495h1m5.776555555s) [noise=true , address=45.91.116.4:4939 , piece_id=OUN2FPBDDS6T77MSIFQMMA7AGF2UF32UEVNPKWRXDXUEEZGIN4IA, seg_pos={0 0}, upload=434481]: Uploading piece... Dec 26 06:30:05.905760038 (495h1m5.776843083s) [noise=true , address=77.90.31.12:4590 , piece_id=6K7V4RON7P3EVFQKAPWF2Y44XIRB45BOZ4M6BRKFHAAMXAZ4TY6A, seg_pos={0 0}, upload=434481]: Uploading piece... Dec 26 06:30:05.906401048 (495h1m5.777484093s) [noise=true , address=45.91.116.6:4976 , piece_id=HVNG23JZFVJLONY7YINXTQOU7LU7N3MJASUNYTPJXN3VCTCOHZBA, seg_pos={0 0}, upload=434481]: Uploading piece... Dec 26 06:30:05.907466136 (495h1m5.778549172s) [noise=true , address=145.79.227.48:4303 , piece_id=X3KR2KWDNP26SVIZZFDYQZXPL3E6HRHR7CSMGNBY7BMS4OPSUADA, seg_pos={0 0}, upload=434481]: Uploading piece... Dec 26 06:30:05.908013268 (495h1m5.779096313s) [noise=true , address=45.91.116.14:4190 , piece_id=4UEXG5Z6AHXVIHP6GVRAJ4ZPYFAQXJN3Q4YGV6BWTKSACOXVCVBA, seg_pos={0 0}, upload=434481]: Uploading piece... Dec 26 06:30:05.908140751 (495h1m5.779223786s) [noise=true , address=45.91.116.5:4299 , piece_id=6FB4CF4OMEAT5V3CR52CFIZWVHZK6RUQZVRGF4GF7T7FRPMSW7ZQ, seg_pos={0 0}, upload=434481]: Uploading piece... Dec 26 06:30:05.908593403 (495h1m5.779676448s) [noise=true , address=45.91.116.11:4117 , piece_id=NKLNFDDG4BP6PBCIKIHFIKLMJTO3ETO5O6SQOHMQJ4GBOLZYPHEQ, seg_pos={0 0}, upload=434481]: Uploading piece... Dec 26 06:30:05.908742456 (495h1m5.779825501s) [noise=true , address=145.79.227.45:4229 , piece_id=KDC7U3EHMPASW5V6TZOUSQJ2LUOQUUH2ZAIGRRPR2PVNIH7L3TZA, seg_pos={0 0}, upload=434481]: Uploading piece... Dec 26 06:30:05.909056083 (495h1m5.780139128s) [noise=true , address=145.79.227.34:4906 , piece_id=5H35H572VSIMNMNNVBYBKPRQYNLO46C3NREKMLOX42NG6UG3H7MQ, seg_pos={0 0}, upload=434481]: Uploading piece... Dec 26 06:30:05.909299086 (495h1m5.780382131s) [noise=true , address=145.79.227.6:4257 , piece_id=LQS3NRXQIMAANUXSIAHIQQ2PKD2H754CGHVKUNDBVO3MZA7RTRBA, seg_pos={0 0}, upload=434481]: Uploading piece... Dec 26 06:30:05.909506471 (495h1m5.780589516s) [noise=true , address=45.91.116.8:4045 , piece_id=AT3SJHRPDB7EED4J7D745KI7FK2T3JZLKKOZYG6F5YCZKQ6FBQQA, seg_pos={0 0}, upload=434481]: Uploading piece... Dec 26 06:30:05.909984041 (495h1m5.781067086s) [noise=true , address=77.90.31.11:4556 , piece_id=RYJ3RJ43RD4L33OITBVNHBLUXQJVWWLILZSOI6K2QL7CGNXJABCA, seg_pos={0 0}, upload=434481]: Uploading piece... Dec 26 06:30:05.910803762 (495h1m5.781886837s) [noise=true , address=145.79.227.25:4617 , piece_id=PDFSA4MC3KETP6LH4ME2P7KKV34ZHORY4QDBGKTQDT4WDWTI7G5A, seg_pos={0 0}, upload=434481]: Uploading piece... Dec 26 06:30:07.616251981 (495h1m7.487335026s) [upload=434481]: Next returned nil segment Dec 26 06:30:07.616266658 (495h1m7.487349703s) [upload=434481]: Waiting for error group managing segments... Dec 26 06:30:07.913182599 (495h1m7.784265644s) [noise=true , address=145.79.227.25:4617 , piece_id=PDFSA4MC3KETP6LH4ME2P7KKV34ZHORY4QDBGKTQDT4WDWTI7G5A, seg_pos={0 0}, upload=434481]: Done uploading piece. err: <nil> Dec 26 06:30:07.913791419 (495h1m7.784874454s) [noise=true , address=145.79.227.48:4303 , piece_id=X3KR2KWDNP26SVIZZFDYQZXPL3E6HRHR7CSMGNBY7BMS4OPSUADA, seg_pos={0 0}, upload=434481]: Done uploading piece. err: <nil> Dec 26 06:30:07.916007719 (495h1m7.787090765s) [noise=true , address=145.79.227.19:4077 , piece_id=DNGDGSHU4KE6N22SFPX6EUVTIOU7DKKKTTBIZCEHBYR6YRZJW3LQ, seg_pos={0 0}, upload=434481]: Done uploading piece. err: <nil> Dec 26 06:30:07.917603387 (495h1m7.788686432s) [noise=true , address=145.79.227.6:4257 , piece_id=LQS3NRXQIMAANUXSIAHIQQ2PKD2H754CGHVKUNDBVO3MZA7RTRBA, seg_pos={0 0}, upload=434481]: Done uploading piece. err: <nil> Dec 26 06:30:07.930364206 (495h1m7.801447251s) [noise=true , address=145.79.227.45:4229 , piece_id=KDC7U3EHMPASW5V6TZOUSQJ2LUOQUUH2ZAIGRRPR2PVNIH7L3TZA, seg_pos={0 0}, upload=434481]: Done uploading piece. err: <nil> Dec 26 06:30:07.943685473 (495h1m7.814768518s) [noise=true , address=145.79.227.34:4906 , piece_id=5H35H572VSIMNMNNVBYBKPRQYNLO46C3NREKMLOX42NG6UG3H7MQ, seg_pos={0 0}, upload=434481]: Done uploading piece. err: <nil> Dec 26 06:30:08.123531070 (495h1m7.994614105s) [noise=true , address=45.91.116.5:4299 , piece_id=6FB4CF4OMEAT5V3CR52CFIZWVHZK6RUQZVRGF4GF7T7FRPMSW7ZQ, seg_pos={0 0}, upload=434481]: Done uploading piece. err: <nil> Dec 26 06:30:08.123704230 (495h1m7.994787275s) [noise=true , address=77.90.31.11:4556 , piece_id=RYJ3RJ43RD4L33OITBVNHBLUXQJVWWLILZSOI6K2QL7CGNXJABCA, seg_pos={0 0}, upload=434481]: Done uploading piece. err: <nil> Dec 26 06:30:08.225992686 (495h1m8.097075732s) [noise=true , address=45.91.116.8:4045 , piece_id=AT3SJHRPDB7EED4J7D745KI7FK2T3JZLKKOZYG6F5YCZKQ6FBQQA, seg_pos={0 0}, upload=434481]: Done uploading piece. err: <nil> Dec 26 06:30:08.242648572 (495h1m8.113731617s) [noise=true , address=45.91.116.10:4084 , piece_id=IZ6YPCBPWW6IMOVTRKKWOBLM7OSORHYL54K3WEK6UVRDD4OKIYOQ, seg_pos={0 0}, upload=434481]: Done uploading piece. err: <nil> Dec 26 06:30:08.284773425 (495h1m8.15585647s) [noise=true , address=45.91.116.14:4190 , piece_id=4UEXG5Z6AHXVIHP6GVRAJ4ZPYFAQXJN3Q4YGV6BWTKSACOXVCVBA, seg_pos={0 0}, upload=434481]: Done uploading piece. err: <nil> Dec 26 06:30:08.330394266 (495h1m8.201477301s) [noise=true , address=45.91.116.4:4939 , piece_id=OUN2FPBDDS6T77MSIFQMMA7AGF2UF32UEVNPKWRXDXUEEZGIN4IA, seg_pos={0 0}, upload=434481]: Done uploading piece. err: <nil> Dec 26 06:30:08.383522201 (495h1m8.254605246s) [noise=true , address=77.90.31.12:4590 , piece_id=6K7V4RON7P3EVFQKAPWF2Y44XIRB45BOZ4M6BRKFHAAMXAZ4TY6A, seg_pos={0 0}, upload=434481]: Done uploading piece. err: <nil> Dec 26 06:30:08.383545966 (495h1m8.254629011s) [seg_pos={0 0}, upload=434481]: Segment reached optimal threshold of 13 pieces. Dec 26 06:30:08.383760064 (495h1m8.254843109s) [noise=true , address=45.91.116.11:4117 , piece_id=NKLNFDDG4BP6PBCIKIHFIKLMJTO3ETO5O6SQOHMQJ4GBOLZYPHEQ, seg_pos={0 0}, upload=434481]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF Dec 26 06:30:08.383767397 (495h1m8.254850442s) [noise=true , address=45.91.116.11:4117 , piece_id=NKLNFDDG4BP6PBCIKIHFIKLMJTO3ETO5O6SQOHMQJ4GBOLZYPHEQ, seg_pos={0 0}, upload=434481]: Long tail cancellation due to optimal threshold Dec 26 06:30:08.383916571 (495h1m8.254999616s) [noise=true , address=77.90.31.17:4773 , piece_id=6C4MQKYKSQECLTSJYG4X3Y6VQYRWRWASSIUKTZDJFKASKYJOKX6A, seg_pos={0 0}, upload=434481]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: write tcp 188.166.225.170:55372->77.90.31.17:4773: use of closed network connection; write tcp 188.166.225.170:55372->77.90.31.17:4773: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55372->77.90.31.17:4773: use of closed network connection Dec 26 06:30:08.383928224 (495h1m8.255011289s) [noise=true , address=77.90.31.17:4773 , piece_id=6C4MQKYKSQECLTSJYG4X3Y6VQYRWRWASSIUKTZDJFKASKYJOKX6A, seg_pos={0 0}, upload=434481]: Long tail cancellation due to optimal threshold Dec 26 06:30:08.384115039 (495h1m8.255198084s) [noise=true , address=45.91.116.6:4976 , piece_id=HVNG23JZFVJLONY7YINXTQOU7LU7N3MJASUNYTPJXN3VCTCOHZBA, seg_pos={0 0}, upload=434481]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: read tcp 188.166.225.170:57032->45.91.116.6:4976: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 06:30:08.384123355 (495h1m8.2552064s) [noise=true , address=45.91.116.6:4976 , piece_id=HVNG23JZFVJLONY7YINXTQOU7LU7N3MJASUNYTPJXN3VCTCOHZBA, seg_pos={0 0}, upload=434481]: Long tail cancellation due to optimal threshold Dec 26 06:30:08.384203006 (495h1m8.255286041s) [seg_pos={0 0}, upload=434481]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 06:30:08.384306103 (495h1m8.255389148s) [upload=434481]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 06:30:08.384311824 (495h1m8.25539487s) [upload=434481]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 06:30:08.690044113 (495h1m8.561127158s) [upload=434481]: Done uploading segments... Dec 26 06:30:08.690070624 (495h1m8.561153669s) [upload=434481]: Upload finished. err: <nil> Duration: 30.002181152s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 29.252187061s trace: https://jaeger.dev.storj.io/trace/7e198d4c8fea54d2 FAIL (2025-12-26 06:30:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p, Piece ID: SBCEX7UNWGNFF54SBP4AGRDV5IPG24FQJXJJ7EYQTBSZKOVZWU4Q) context deadline exceeded; piecestore: (Node ID: 1nP8xospdMuDEh3em5K6iBnh8bHXqLhvHy4pHm9m8Lnj4kEH2X, Piece ID: XV7FQGBJNWVRLOAFRUPOWON35JR5ESRJPTCWPUGYWGHJPSDVL4GA) context deadline exceeded; piecestore: (Node ID: 1Mg5mxUjW56quDidSgA788D7ByfAzvsEuWJqwtzY4bLyxRn3mu, Piece ID: ATPIXLMXGONSSLXPPVORSNC6LHPXI2V3UFTHM4VHA2FTY5ZM6H3A) context deadline exceeded; piecestore: (Node ID: 12QmYSEAKnkSLEEVc9E1uxtVigunKMFLk2cdu71Bse8Tqec3HyT, Piece ID: YFKD7GO2FOSPLDXNFPK732QTLCLBQQASWBTMZ2LTLBGSK22PIXIQ) context deadline exceeded; piecestore: (Node ID: 1JsnHuhQhM4rWLTLEuSSHLNE24m7tSW9qMSf48j7yMrAVPWVyV, Piece ID: ARBMDAKX4QBW6JG47KP2PUITCX7HKXCBO75THQPINM4LLHCKAV4Q) context deadline exceeded ======LOGS====== upload=433902]: Uploading piece... Dec 26 05:50:34.986155265 (494h21m34.8572383s) [noise=true , address=64.49.13.140:4509 , piece_id=6TGWTOAVY4YVAKU6VEE2SWVAM264TKU6ZNMGPWH4TNGII3UYXFXQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.986703610 (494h21m34.857786655s) [noise=true , address=77.90.31.10:4522 , piece_id=YFKD7GO2FOSPLDXNFPK732QTLCLBQQASWBTMZ2LTLBGSK22PIXIQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.987145380 (494h21m34.858228425s) [noise=true , address=145.79.227.33:4890 , piece_id=UKNB4ZZTYJGY4MDHOBPYUTHCP3ZDCMJB7O7AJZFYP2HFVIWLUOKQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.987628290 (494h21m34.858711335s) [noise=true , address=145.79.227.40:4052 , piece_id=XRZYL7PT5VHY22N7CCKS3NTZKDDWE36ZU3SORF6ILM7ZMXDND63A, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.988710731 (494h21m34.859793776s) [noise=true , address=77.90.31.13:4653 , piece_id=EY2CWX54ONOWPO3WC33G5QSP5RMP2SUTBQLUOH5JXGDRUEPDHQ2A, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.989894545 (494h21m34.86097759s) [noise=true , address=145.79.227.30:4785 , piece_id=5KMLU4ATBN4YVQDYJKKQY4PEJNKWTO2WNX5GMGJM4F54B6YM2SFA, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.991305081 (494h21m34.862388116s) [noise=true , address=145.79.227.50:4381 , piece_id=H6Q47DIOTOWGN7QH2KWT4IH6KVDSB246S22753Z3UQGB26T6VXPA, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.995796263 (494h21m34.866879359s) [noise=true , address=145.79.227.52:4461 , piece_id=ZAAUDBGY6CC2JNEGWXVPCBAWXARHAJDZ3YPB6Y3NWEWCFDUFGDSQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.996765439 (494h21m34.867848484s) [noise=true , address=77.90.31.16:4755 , piece_id=ARBMDAKX4QBW6JG47KP2PUITCX7HKXCBO75THQPINM4LLHCKAV4Q, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.998107634 (494h21m34.869190679s) [noise=true , address=77.90.31.11:4579 , piece_id=SBCEX7UNWGNFF54SBP4AGRDV5IPG24FQJXJJ7EYQTBSZKOVZWU4Q, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.999340442 (494h21m34.870423477s) [noise=true , address=145.79.227.34:4936 , piece_id=EEA2ZPM44DX3WMI7ZJTDC7T5UIGED6TNT53QXZWGBFRMQUQSCXPQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:35.000540757 (494h21m34.871623812s) [noise=true , address=45.91.116.14:4204 , piece_id=XV7FQGBJNWVRLOAFRUPOWON35JR5ESRJPTCWPUGYWGHJPSDVL4GA, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:36.360077064 (494h21m36.231160099s) [upload=433902]: Next returned nil segment Dec 26 05:50:36.360091642 (494h21m36.231174688s) [upload=433902]: Waiting for error group managing segments... Dec 26 05:50:36.688914299 (494h21m36.559997344s) [noise=true , address=64.49.13.140:4509 , piece_id=6TGWTOAVY4YVAKU6VEE2SWVAM264TKU6ZNMGPWH4TNGII3UYXFXQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.059812634 (494h21m36.930895669s) [noise=true , address=77.90.31.11:4579 , piece_id=SBCEX7UNWGNFF54SBP4AGRDV5IPG24FQJXJJ7EYQTBSZKOVZWU4Q, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.119433874 (494h21m36.990516909s) [noise=true , address=45.91.116.14:4204 , piece_id=XV7FQGBJNWVRLOAFRUPOWON35JR5ESRJPTCWPUGYWGHJPSDVL4GA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.138718204 (494h21m37.00980125s) [noise=true , address=77.90.31.16:4755 , piece_id=ARBMDAKX4QBW6JG47KP2PUITCX7HKXCBO75THQPINM4LLHCKAV4Q, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.146647243 (494h21m37.017730278s) [noise=true , address=77.90.31.13:4653 , piece_id=EY2CWX54ONOWPO3WC33G5QSP5RMP2SUTBQLUOH5JXGDRUEPDHQ2A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.163283722 (494h21m37.034366757s) [noise=true , address=77.90.31.21:4893 , piece_id=GNEEZO2XGOKNODOUPZQ4NGKFE4PCX5LNUWXLV27F6X7GWZVSYNAA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.374956574 (494h21m37.246039609s) [noise=true , address=45.91.116.7:4033 , piece_id=ATPIXLMXGONSSLXPPVORSNC6LHPXI2V3UFTHM4VHA2FTY5ZM6H3A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.777283355 (494h21m37.64836639s) [noise=true , address=77.90.31.12:4622 , piece_id=LTYYPK6ZQ3NXZPBKSV4OMZGCOXXV3N6L5A6MVODNR644D2ZUJX3A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:38.439025909 (494h21m38.310108944s) [noise=true , address=145.79.227.49:4355 , piece_id=IUDOHR73Y6OKFAQSNNSONALWAITRJ4POYRO2QD5LYZKOSTWY4MNA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:38.439599111 (494h21m38.310682166s) [noise=true , address=77.90.31.10:4522 , piece_id=YFKD7GO2FOSPLDXNFPK732QTLCLBQQASWBTMZ2LTLBGSK22PIXIQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:39.259154980 (494h21m39.130238015s) [noise=true , address=145.79.227.34:4936 , piece_id=EEA2ZPM44DX3WMI7ZJTDC7T5UIGED6TNT53QXZWGBFRMQUQSCXPQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:39.430935921 (494h21m39.302018956s) [noise=true , address=145.79.227.44:4212 , piece_id=ABBNLKRE4FM7RHF56V4IAQPKBCCCQLLOMFVMSJEWQUNFGGCI3PVQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:42.941071297 (494h21m42.812154352s) [noise=true , address=77.90.31.14:4675 , piece_id=FKB4O45OYICMELPGXB65QAUD2DRIFTSXZFKIA6TY26VFAP6AVY3A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:51:22.719026850 (494h22m22.590109885s) [noise=true , address=145.79.227.33:4890 , piece_id=UKNB4ZZTYJGY4MDHOBPYUTHCP3ZDCMJB7O7AJZFYP2HFVIWLUOKQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:51:22.719060193 (494h22m22.590143228s) [seg_pos={0 0}, upload=433902]: Segment reached optimal threshold of 14 pieces. Dec 26 05:51:22.719233022 (494h22m22.590316067s) [noise=true , address=145.79.227.50:4381 , piece_id=H6Q47DIOTOWGN7QH2KWT4IH6KVDSB246S22753Z3UQGB26T6VXPA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zU35MWWaaC1npkP5qGcWBXAkfxGsd5UsdkM2nwX9x7wUGCDbY): protocol: write tcp 188.166.225.170:46970->145.79.227.50:4381: use of closed network connection; write tcp 188.166.225.170:46970->145.79.227.50:4381: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46970->145.79.227.50:4381: use of closed network connection Dec 26 05:51:22.719238783 (494h22m22.590321828s) [noise=true , address=145.79.227.50:4381 , piece_id=H6Q47DIOTOWGN7QH2KWT4IH6KVDSB246S22753Z3UQGB26T6VXPA, seg_pos={0 0}, upload=433902]: Long tail cancellation due to optimal threshold Dec 26 05:51:22.719251677 (494h22m22.590334712s) [noise=true , address=145.79.227.30:4785 , piece_id=5KMLU4ATBN4YVQDYJKKQY4PEJNKWTO2WNX5GMGJM4F54B6YM2SFA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gYH83ECV8wai3CZyBSU8NNJM1UsysLYHbQDwBVKZRyjsdnUwB): protocol: write tcp 188.166.225.170:58398->145.79.227.30:4785: use of closed network connection; write tcp 188.166.225.170:58398->145.79.227.30:4785: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58398->145.79.227.30:4785: use of closed network connection Dec 26 05:51:22.719260194 (494h22m22.590343229s) [noise=true , address=145.79.227.30:4785 , piece_id=5KMLU4ATBN4YVQDYJKKQY4PEJNKWTO2WNX5GMGJM4F54B6YM2SFA, seg_pos={0 0}, upload=433902]: Long tail cancellation due to optimal threshold Dec 26 05:51:22.719409408 (494h22m22.590492443s) [noise=true , address=145.79.227.40:4052 , piece_id=XRZYL7PT5VHY22N7CCKS3NTZKDDWE36ZU3SORF6ILM7ZMXDND63A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PViJBYAanwXRWhMq3WHR6k9Stxy8hjuY5FV9J7ErxCy4N1q3V): protocol: write tcp 188.166.225.170:45984->145.79.227.40:4052: use of closed network connection; write tcp 188.166.225.170:45984->145.79.227.40:4052: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45984->145.79.227.40:4052: use of closed network connection Dec 26 05:51:22.719415469 (494h22m22.590498515s) [noise=true , address=145.79.227.40:4052 , piece_id=XRZYL7PT5VHY22N7CCKS3NTZKDDWE36ZU3SORF6ILM7ZMXDND63A, seg_pos={0 0}, upload=433902]: Long tail cancellation due to optimal threshold Dec 26 05:51:22.719467057 (494h22m22.590550092s) [noise=true , address=145.79.227.52:4461 , piece_id=ZAAUDBGY6CC2JNEGWXVPCBAWXARHAJDZ3YPB6Y3NWEWCFDUFGDSQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fNrbSfErEQRnruJCJAVvkoNwQSPWnvNtxunHn6zrQegnNAkxA): protocol: write tcp 188.166.225.170:55966->145.79.227.52:4461: use of closed network connection; write tcp 188.166.225.170:55966->145.79.227.52:4461: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55966->145.79.227.52:4461: use of closed network connection Dec 26 05:51:22.719473931 (494h22m22.590556966s) [noise=true , address=145.79.227.52:4461 , piece_id=ZAAUDBGY6CC2JNEGWXVPCBAWXARHAJDZ3YPB6Y3NWEWCFDUFGDSQ, seg_pos={0 0}, upload=433902]: Long tail cancellation due to optimal threshold Dec 26 05:51:22.719491374 (494h22m22.590574409s) [seg_pos={0 0}, upload=433902]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 05:51:22.719562539 (494h22m22.590645574s) [upload=433902]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:51:22.719567228 (494h22m22.590650263s) [upload=433902]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:51:23.008169117 (494h22m22.879252162s) [upload=433902]: Done uploading segments... Dec 26 05:51:23.008191560 (494h22m22.879274605s) [upload=433902]: Upload finished. err: <nil> Duration: 29.252268546s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 34.125376229s trace: https://jaeger.dev.storj.io/trace/7e198d4c8fea54d2 FAIL (2025-12-26 06:29:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p, Piece ID: SBCEX7UNWGNFF54SBP4AGRDV5IPG24FQJXJJ7EYQTBSZKOVZWU4Q) context deadline exceeded; piecestore: (Node ID: 1nP8xospdMuDEh3em5K6iBnh8bHXqLhvHy4pHm9m8Lnj4kEH2X, Piece ID: XV7FQGBJNWVRLOAFRUPOWON35JR5ESRJPTCWPUGYWGHJPSDVL4GA) context deadline exceeded; piecestore: (Node ID: 1Mg5mxUjW56quDidSgA788D7ByfAzvsEuWJqwtzY4bLyxRn3mu, Piece ID: ATPIXLMXGONSSLXPPVORSNC6LHPXI2V3UFTHM4VHA2FTY5ZM6H3A) context deadline exceeded; piecestore: (Node ID: 12QmYSEAKnkSLEEVc9E1uxtVigunKMFLk2cdu71Bse8Tqec3HyT, Piece ID: YFKD7GO2FOSPLDXNFPK732QTLCLBQQASWBTMZ2LTLBGSK22PIXIQ) context deadline exceeded; piecestore: (Node ID: 1JsnHuhQhM4rWLTLEuSSHLNE24m7tSW9qMSf48j7yMrAVPWVyV, Piece ID: ARBMDAKX4QBW6JG47KP2PUITCX7HKXCBO75THQPINM4LLHCKAV4Q) context deadline exceeded ======LOGS====== upload=433902]: Uploading piece... Dec 26 05:50:34.986155265 (494h21m34.8572383s) [noise=true , address=64.49.13.140:4509 , piece_id=6TGWTOAVY4YVAKU6VEE2SWVAM264TKU6ZNMGPWH4TNGII3UYXFXQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.986703610 (494h21m34.857786655s) [noise=true , address=77.90.31.10:4522 , piece_id=YFKD7GO2FOSPLDXNFPK732QTLCLBQQASWBTMZ2LTLBGSK22PIXIQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.987145380 (494h21m34.858228425s) [noise=true , address=145.79.227.33:4890 , piece_id=UKNB4ZZTYJGY4MDHOBPYUTHCP3ZDCMJB7O7AJZFYP2HFVIWLUOKQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.987628290 (494h21m34.858711335s) [noise=true , address=145.79.227.40:4052 , piece_id=XRZYL7PT5VHY22N7CCKS3NTZKDDWE36ZU3SORF6ILM7ZMXDND63A, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.988710731 (494h21m34.859793776s) [noise=true , address=77.90.31.13:4653 , piece_id=EY2CWX54ONOWPO3WC33G5QSP5RMP2SUTBQLUOH5JXGDRUEPDHQ2A, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.989894545 (494h21m34.86097759s) [noise=true , address=145.79.227.30:4785 , piece_id=5KMLU4ATBN4YVQDYJKKQY4PEJNKWTO2WNX5GMGJM4F54B6YM2SFA, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.991305081 (494h21m34.862388116s) [noise=true , address=145.79.227.50:4381 , piece_id=H6Q47DIOTOWGN7QH2KWT4IH6KVDSB246S22753Z3UQGB26T6VXPA, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.995796263 (494h21m34.866879359s) [noise=true , address=145.79.227.52:4461 , piece_id=ZAAUDBGY6CC2JNEGWXVPCBAWXARHAJDZ3YPB6Y3NWEWCFDUFGDSQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.996765439 (494h21m34.867848484s) [noise=true , address=77.90.31.16:4755 , piece_id=ARBMDAKX4QBW6JG47KP2PUITCX7HKXCBO75THQPINM4LLHCKAV4Q, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.998107634 (494h21m34.869190679s) [noise=true , address=77.90.31.11:4579 , piece_id=SBCEX7UNWGNFF54SBP4AGRDV5IPG24FQJXJJ7EYQTBSZKOVZWU4Q, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.999340442 (494h21m34.870423477s) [noise=true , address=145.79.227.34:4936 , piece_id=EEA2ZPM44DX3WMI7ZJTDC7T5UIGED6TNT53QXZWGBFRMQUQSCXPQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:35.000540757 (494h21m34.871623812s) [noise=true , address=45.91.116.14:4204 , piece_id=XV7FQGBJNWVRLOAFRUPOWON35JR5ESRJPTCWPUGYWGHJPSDVL4GA, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:36.360077064 (494h21m36.231160099s) [upload=433902]: Next returned nil segment Dec 26 05:50:36.360091642 (494h21m36.231174688s) [upload=433902]: Waiting for error group managing segments... Dec 26 05:50:36.688914299 (494h21m36.559997344s) [noise=true , address=64.49.13.140:4509 , piece_id=6TGWTOAVY4YVAKU6VEE2SWVAM264TKU6ZNMGPWH4TNGII3UYXFXQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.059812634 (494h21m36.930895669s) [noise=true , address=77.90.31.11:4579 , piece_id=SBCEX7UNWGNFF54SBP4AGRDV5IPG24FQJXJJ7EYQTBSZKOVZWU4Q, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.119433874 (494h21m36.990516909s) [noise=true , address=45.91.116.14:4204 , piece_id=XV7FQGBJNWVRLOAFRUPOWON35JR5ESRJPTCWPUGYWGHJPSDVL4GA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.138718204 (494h21m37.00980125s) [noise=true , address=77.90.31.16:4755 , piece_id=ARBMDAKX4QBW6JG47KP2PUITCX7HKXCBO75THQPINM4LLHCKAV4Q, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.146647243 (494h21m37.017730278s) [noise=true , address=77.90.31.13:4653 , piece_id=EY2CWX54ONOWPO3WC33G5QSP5RMP2SUTBQLUOH5JXGDRUEPDHQ2A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.163283722 (494h21m37.034366757s) [noise=true , address=77.90.31.21:4893 , piece_id=GNEEZO2XGOKNODOUPZQ4NGKFE4PCX5LNUWXLV27F6X7GWZVSYNAA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.374956574 (494h21m37.246039609s) [noise=true , address=45.91.116.7:4033 , piece_id=ATPIXLMXGONSSLXPPVORSNC6LHPXI2V3UFTHM4VHA2FTY5ZM6H3A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.777283355 (494h21m37.64836639s) [noise=true , address=77.90.31.12:4622 , piece_id=LTYYPK6ZQ3NXZPBKSV4OMZGCOXXV3N6L5A6MVODNR644D2ZUJX3A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:38.439025909 (494h21m38.310108944s) [noise=true , address=145.79.227.49:4355 , piece_id=IUDOHR73Y6OKFAQSNNSONALWAITRJ4POYRO2QD5LYZKOSTWY4MNA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:38.439599111 (494h21m38.310682166s) [noise=true , address=77.90.31.10:4522 , piece_id=YFKD7GO2FOSPLDXNFPK732QTLCLBQQASWBTMZ2LTLBGSK22PIXIQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:39.259154980 (494h21m39.130238015s) [noise=true , address=145.79.227.34:4936 , piece_id=EEA2ZPM44DX3WMI7ZJTDC7T5UIGED6TNT53QXZWGBFRMQUQSCXPQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:39.430935921 (494h21m39.302018956s) [noise=true , address=145.79.227.44:4212 , piece_id=ABBNLKRE4FM7RHF56V4IAQPKBCCCQLLOMFVMSJEWQUNFGGCI3PVQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:42.941071297 (494h21m42.812154352s) [noise=true , address=77.90.31.14:4675 , piece_id=FKB4O45OYICMELPGXB65QAUD2DRIFTSXZFKIA6TY26VFAP6AVY3A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:51:22.719026850 (494h22m22.590109885s) [noise=true , address=145.79.227.33:4890 , piece_id=UKNB4ZZTYJGY4MDHOBPYUTHCP3ZDCMJB7O7AJZFYP2HFVIWLUOKQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:51:22.719060193 (494h22m22.590143228s) [seg_pos={0 0}, upload=433902]: Segment reached optimal threshold of 14 pieces. Dec 26 05:51:22.719233022 (494h22m22.590316067s) [noise=true , address=145.79.227.50:4381 , piece_id=H6Q47DIOTOWGN7QH2KWT4IH6KVDSB246S22753Z3UQGB26T6VXPA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zU35MWWaaC1npkP5qGcWBXAkfxGsd5UsdkM2nwX9x7wUGCDbY): protocol: write tcp 188.166.225.170:46970->145.79.227.50:4381: use of closed network connection; write tcp 188.166.225.170:46970->145.79.227.50:4381: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46970->145.79.227.50:4381: use of closed network connection Dec 26 05:51:22.719238783 (494h22m22.590321828s) [noise=true , address=145.79.227.50:4381 , piece_id=H6Q47DIOTOWGN7QH2KWT4IH6KVDSB246S22753Z3UQGB26T6VXPA, seg_pos={0 0}, upload=433902]: Long tail cancellation due to optimal threshold Dec 26 05:51:22.719251677 (494h22m22.590334712s) [noise=true , address=145.79.227.30:4785 , piece_id=5KMLU4ATBN4YVQDYJKKQY4PEJNKWTO2WNX5GMGJM4F54B6YM2SFA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gYH83ECV8wai3CZyBSU8NNJM1UsysLYHbQDwBVKZRyjsdnUwB): protocol: write tcp 188.166.225.170:58398->145.79.227.30:4785: use of closed network connection; write tcp 188.166.225.170:58398->145.79.227.30:4785: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58398->145.79.227.30:4785: use of closed network connection Dec 26 05:51:22.719260194 (494h22m22.590343229s) [noise=true , address=145.79.227.30:4785 , piece_id=5KMLU4ATBN4YVQDYJKKQY4PEJNKWTO2WNX5GMGJM4F54B6YM2SFA, seg_pos={0 0}, upload=433902]: Long tail cancellation due to optimal threshold Dec 26 05:51:22.719409408 (494h22m22.590492443s) [noise=true , address=145.79.227.40:4052 , piece_id=XRZYL7PT5VHY22N7CCKS3NTZKDDWE36ZU3SORF6ILM7ZMXDND63A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PViJBYAanwXRWhMq3WHR6k9Stxy8hjuY5FV9J7ErxCy4N1q3V): protocol: write tcp 188.166.225.170:45984->145.79.227.40:4052: use of closed network connection; write tcp 188.166.225.170:45984->145.79.227.40:4052: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45984->145.79.227.40:4052: use of closed network connection Dec 26 05:51:22.719415469 (494h22m22.590498515s) [noise=true , address=145.79.227.40:4052 , piece_id=XRZYL7PT5VHY22N7CCKS3NTZKDDWE36ZU3SORF6ILM7ZMXDND63A, seg_pos={0 0}, upload=433902]: Long tail cancellation due to optimal threshold Dec 26 05:51:22.719467057 (494h22m22.590550092s) [noise=true , address=145.79.227.52:4461 , piece_id=ZAAUDBGY6CC2JNEGWXVPCBAWXARHAJDZ3YPB6Y3NWEWCFDUFGDSQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fNrbSfErEQRnruJCJAVvkoNwQSPWnvNtxunHn6zrQegnNAkxA): protocol: write tcp 188.166.225.170:55966->145.79.227.52:4461: use of closed network connection; write tcp 188.166.225.170:55966->145.79.227.52:4461: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55966->145.79.227.52:4461: use of closed network connection Dec 26 05:51:22.719473931 (494h22m22.590556966s) [noise=true , address=145.79.227.52:4461 , piece_id=ZAAUDBGY6CC2JNEGWXVPCBAWXARHAJDZ3YPB6Y3NWEWCFDUFGDSQ, seg_pos={0 0}, upload=433902]: Long tail cancellation due to optimal threshold Dec 26 05:51:22.719491374 (494h22m22.590574409s) [seg_pos={0 0}, upload=433902]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 05:51:22.719562539 (494h22m22.590645574s) [upload=433902]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:51:22.719567228 (494h22m22.590650263s) [upload=433902]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:51:23.008169117 (494h22m22.879252162s) [upload=433902]: Done uploading segments... Dec 26 05:51:23.008191560 (494h22m22.879274605s) [upload=433902]: Upload finished. err: <nil> Duration: 34.125390486s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [..X..X.X.......................X..........X.......X.X.......] duration: 5.990214831s trace: https://jaeger.dev.storj.io/trace/67a56359499790c8 FAIL (2025-12-26 06:29:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 06:23:47.396606446 (494h54m47.267689491s) [upload=434386]: Starting upload Dec 26 06:23:47.396675247 (494h54m47.267758292s) [upload=434386]: Done starting upload Dec 26 06:23:47.396898202 (494h54m47.267981248s) [upload=434386]: Uploading segments... Dec 26 06:23:47.396953998 (494h54m47.268037073s) [upload=434386]: Got next segment. Inline: false Dec 26 06:23:47.397031125 (494h54m47.26811416s) [upload=434386]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 06:23:47.397037317 (494h54m47.268120362s) [upload=434386]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 06:23:47.643130652 (494h54m47.514213687s) [seg_pos={0 0}, upload=434386]: Begin upload segment... Dec 26 06:23:47.643171870 (494h54m47.514254905s) [seg_pos={0 0}, upload=434386]: Done begin upload segment. Dec 26 06:23:47.643193913 (494h54m47.514276948s) [noise=true , address=145.79.227.38:4977 , piece_id=PKXAJYGJPG6GYOG6DT4M6TBR54TTMBSWBXIIW2RLVMSA4HXBGOUQ, seg_pos={0 0}, upload=434386]: Uploading piece... Dec 26 06:23:47.643278103 (494h54m47.514361138s) [noise=true , address=145.79.227.49:4339 , piece_id=XQ6XNO7FGYMNEWYDGWGIB5IY5IHOAOPK7TUWPVPSEGQRIDA4R6GA, seg_pos={0 0}, upload=434386]: Uploading piece... Dec 26 06:23:47.644040174 (494h54m47.515123219s) [noise=true , address=145.79.227.29:4942 , piece_id=GML4O7CTQ3AR5KAPGOZAMBELR7F6SDGSJSAWPZLO27W53ALPNZ2Q, seg_pos={0 0}, upload=434386]: Uploading piece... Dec 26 06:23:47.644250264 (494h54m47.515333309s) [noise=true , address=145.79.227.3:4149 , piece_id=TOTASJEGQWCGG5KISNGGY3TLAL354QWMFA4VKEKDSCKHLY25PYQQ, seg_pos={0 0}, upload=434386]: Uploading piece... Dec 26 06:23:47.644310018 (494h54m47.515393063s) [noise=true , address=145.79.227.5:4221 , piece_id=U2IAUDBXHO2UXGPSOYT3L2FPIJZFDTJ4GQCHGGXGSFD62KXIV77Q, seg_pos={0 0}, upload=434386]: Uploading piece... Dec 26 06:23:47.644590933 (494h54m47.515673968s) [noise=true , address=77.90.31.21:4880 , piece_id=JLHHQM55ZRSYUHKTKMZOMU4X3KXNQHIXH32KWIFA3CLWADW37W7Q, seg_pos={0 0}, upload=434386]: Uploading piece... Dec 26 06:23:47.644789010 (494h54m47.515872055s) [noise=true , address=45.91.116.13:4153 , piece_id=7PR6IGAFJOAALB3YXBEWFAGCR72DFJ6JWT24WCO7FHUOCSMJNP7A, seg_pos={0 0}, upload=434386]: Uploading piece... Dec 26 06:23:47.644945428 (494h54m47.516028463s) [noise=true , address=45.91.116.14:4191 , piece_id=LPPDT6VMSHKDJWZLB25GN2XLHS73352JSKCRZNEAFRRRWDLA7RNQ, seg_pos={0 0}, upload=434386]: Uploading piece... Dec 26 06:23:47.645273222 (494h54m47.516356257s) [noise=true , address=45.91.116.7:4010 , piece_id=LHU4DWQU7XLA3QTJ2LHPSLGON43MUO3BXHJ5FRZPIRSA5IRH2LNA, seg_pos={0 0}, upload=434386]: Uploading piece... Dec 26 06:23:47.645283942 (494h54m47.516366978s) [noise=true , address=45.91.116.9:4264 , piece_id=N7AODEQP7YXGHYUJCT27BSAFMEJB3ABXUUCEUPSWDUR3F7DUB5TA, seg_pos={0 0}, upload=434386]: Uploading piece... Dec 26 06:23:47.645654749 (494h54m47.516737784s) [noise=true , address=145.79.227.2:4115 , piece_id=D6S76VZDC2FBIFA3EA3TC4NEIFKJSGX6JRD4S26UO2IOLUDUPIDA, seg_pos={0 0}, upload=434386]: Uploading piece... Dec 26 06:23:47.645885748 (494h54m47.516968793s) [noise=true , address=145.79.227.20:4437 , piece_id=P4DNS6VYA2TS43TL7LYUSRUKFDJ37PO2M6D6NP534QQQBH2WIV2A, seg_pos={0 0}, upload=434386]: Uploading piece... Dec 26 06:23:47.646632821 (494h54m47.517715866s) [noise=true , address=77.90.31.15:4700 , piece_id=23B7NZUGOMXA6OMBSKY3HWKZ36VVIVOCQGFBLFZ6EHGC7TD544YA, seg_pos={0 0}, upload=434386]: Uploading piece... Dec 26 06:23:47.647299961 (494h54m47.518382996s) [noise=true , address=145.79.227.39:4013 , piece_id=3EHFPZVBIV6NNDJ46Z5YHVOPBWEMYQLG6NCJ4HT6ZX3IVMGVAFAQ, seg_pos={0 0}, upload=434386]: Uploading piece... Dec 26 06:23:47.647743005 (494h54m47.51882605s) [noise=true , address=45.91.116.11:4118 , piece_id=6BZNBNEDJB4RNNFMAXRMGLKK4GHYTDNKMVTMG5RF3U2VDKMON2KQ, seg_pos={0 0}, upload=434386]: Uploading piece... Dec 26 06:23:47.647943015 (494h54m47.51902606s) [noise=true , address=145.79.227.6:4259 , piece_id=TGQ6V3HNT3X2IY7MQ73RZRNH5NDXENTRDCXWHFHQETQEILSO7FEA, seg_pos={0 0}, upload=434386]: Uploading piece... Dec 26 06:23:49.283219995 (494h54m49.15430303s) [upload=434386]: Next returned nil segment Dec 26 06:23:49.283245634 (494h54m49.154328679s) [upload=434386]: Waiting for error group managing segments... Dec 26 06:23:49.644523232 (494h54m49.515606267s) [noise=true , address=145.79.227.3:4149 , piece_id=TOTASJEGQWCGG5KISNGGY3TLAL354QWMFA4VKEKDSCKHLY25PYQQ, seg_pos={0 0}, upload=434386]: Done uploading piece. err: <nil> Dec 26 06:23:49.648027525 (494h54m49.51911056s) [noise=true , address=145.79.227.49:4339 , piece_id=XQ6XNO7FGYMNEWYDGWGIB5IY5IHOAOPK7TUWPVPSEGQRIDA4R6GA, seg_pos={0 0}, upload=434386]: Done uploading piece. err: <nil> Dec 26 06:23:49.651435925 (494h54m49.52251897s) [noise=true , address=145.79.227.38:4977 , piece_id=PKXAJYGJPG6GYOG6DT4M6TBR54TTMBSWBXIIW2RLVMSA4HXBGOUQ, seg_pos={0 0}, upload=434386]: Done uploading piece. err: <nil> Dec 26 06:23:49.655712879 (494h54m49.526795924s) [noise=true , address=145.79.227.2:4115 , piece_id=D6S76VZDC2FBIFA3EA3TC4NEIFKJSGX6JRD4S26UO2IOLUDUPIDA, seg_pos={0 0}, upload=434386]: Done uploading piece. err: <nil> Dec 26 06:23:49.658634573 (494h54m49.529717618s) [noise=true , address=145.79.227.39:4013 , piece_id=3EHFPZVBIV6NNDJ46Z5YHVOPBWEMYQLG6NCJ4HT6ZX3IVMGVAFAQ, seg_pos={0 0}, upload=434386]: Done uploading piece. err: <nil> Dec 26 06:23:49.670194264 (494h54m49.541277309s) [noise=true , address=145.79.227.29:4942 , piece_id=GML4O7CTQ3AR5KAPGOZAMBELR7F6SDGSJSAWPZLO27W53ALPNZ2Q, seg_pos={0 0}, upload=434386]: Done uploading piece. err: <nil> Dec 26 06:23:49.672063052 (494h54m49.543146097s) [noise=true , address=145.79.227.5:4221 , piece_id=U2IAUDBXHO2UXGPSOYT3L2FPIJZFDTJ4GQCHGGXGSFD62KXIV77Q, seg_pos={0 0}, upload=434386]: Done uploading piece. err: <nil> Dec 26 06:23:49.672693352 (494h54m49.543776387s) [noise=true , address=145.79.227.20:4437 , piece_id=P4DNS6VYA2TS43TL7LYUSRUKFDJ37PO2M6D6NP534QQQBH2WIV2A, seg_pos={0 0}, upload=434386]: Done uploading piece. err: <nil> Dec 26 06:23:49.725648318 (494h54m49.596731363s) [noise=true , address=145.79.227.6:4259 , piece_id=TGQ6V3HNT3X2IY7MQ73RZRNH5NDXENTRDCXWHFHQETQEILSO7FEA, seg_pos={0 0}, upload=434386]: Done uploading piece. err: <nil> Dec 26 06:23:49.861945242 (494h54m49.733028287s) [noise=true , address=77.90.31.21:4880 , piece_id=JLHHQM55ZRSYUHKTKMZOMU4X3KXNQHIXH32KWIFA3CLWADW37W7Q, seg_pos={0 0}, upload=434386]: Done uploading piece. err: <nil> Dec 26 06:23:49.958824257 (494h54m49.829907292s) [noise=true , address=45.91.116.9:4264 , piece_id=N7AODEQP7YXGHYUJCT27BSAFMEJB3ABXUUCEUPSWDUR3F7DUB5TA, seg_pos={0 0}, upload=434386]: Done uploading piece. err: <nil> Dec 26 06:23:49.970075321 (494h54m49.841158366s) [noise=true , address=45.91.116.13:4153 , piece_id=7PR6IGAFJOAALB3YXBEWFAGCR72DFJ6JWT24WCO7FHUOCSMJNP7A, seg_pos={0 0}, upload=434386]: Done uploading piece. err: <nil> Dec 26 06:23:50.140738494 (494h54m50.011821529s) [noise=true , address=77.90.31.15:4700 , piece_id=23B7NZUGOMXA6OMBSKY3HWKZ36VVIVOCQGFBLFZ6EHGC7TD544YA, seg_pos={0 0}, upload=434386]: Done uploading piece. err: <nil> Dec 26 06:23:50.140774813 (494h54m50.011857849s) [seg_pos={0 0}, upload=434386]: Segment reached optimal threshold of 13 pieces. Dec 26 06:23:50.140993680 (494h54m50.012076725s) [noise=true , address=45.91.116.14:4191 , piece_id=LPPDT6VMSHKDJWZLB25GN2XLHS73352JSKCRZNEAFRRRWDLA7RNQ, seg_pos={0 0}, upload=434386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Dec 26 06:23:50.141002648 (494h54m50.012085683s) [noise=true , address=45.91.116.14:4191 , piece_id=LPPDT6VMSHKDJWZLB25GN2XLHS73352JSKCRZNEAFRRRWDLA7RNQ, seg_pos={0 0}, upload=434386]: Long tail cancellation due to optimal threshold Dec 26 06:23:50.141012116 (494h54m50.012095161s) [noise=true , address=45.91.116.7:4010 , piece_id=LHU4DWQU7XLA3QTJ2LHPSLGON43MUO3BXHJ5FRZPIRSA5IRH2LNA, seg_pos={0 0}, upload=434386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 26 06:23:50.141016754 (494h54m50.012099789s) [noise=true , address=45.91.116.7:4010 , piece_id=LHU4DWQU7XLA3QTJ2LHPSLGON43MUO3BXHJ5FRZPIRSA5IRH2LNA, seg_pos={0 0}, upload=434386]: Long tail cancellation due to optimal threshold Dec 26 06:23:50.141145951 (494h54m50.012228986s) [noise=true , address=45.91.116.11:4118 , piece_id=6BZNBNEDJB4RNNFMAXRMGLKK4GHYTDNKMVTMG5RF3U2VDKMON2KQ, seg_pos={0 0}, upload=434386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: expected piece hash; context canceled; EOF Dec 26 06:23:50.141156651 (494h54m50.012239686s) [noise=true , address=45.91.116.11:4118 , piece_id=6BZNBNEDJB4RNNFMAXRMGLKK4GHYTDNKMVTMG5RF3U2VDKMON2KQ, seg_pos={0 0}, upload=434386]: Long tail cancellation due to optimal threshold Dec 26 06:23:50.141172291 (494h54m50.012255336s) [seg_pos={0 0}, upload=434386]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 06:23:50.141260609 (494h54m50.012343644s) [upload=434386]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 06:23:50.141265488 (494h54m50.012348533s) [upload=434386]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 06:23:50.438800618 (494h54m50.309883653s) [upload=434386]: Done uploading segments... Dec 26 06:23:50.438865121 (494h54m50.309948166s) [upload=434386]: Upload finished. err: <nil> Duration: 5.990241643s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [................................X...........................] duration: 3.417201723s trace: https://jaeger.dev.storj.io/trace/630094597d401477 FAIL (2025-12-26 06:29:24 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.417226149s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 30.795164452s trace: https://jaeger.dev.storj.io/trace/7e198d4c8fea54d2 FAIL (2025-12-26 06:29:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p, Piece ID: SBCEX7UNWGNFF54SBP4AGRDV5IPG24FQJXJJ7EYQTBSZKOVZWU4Q) context deadline exceeded; piecestore: (Node ID: 1nP8xospdMuDEh3em5K6iBnh8bHXqLhvHy4pHm9m8Lnj4kEH2X, Piece ID: XV7FQGBJNWVRLOAFRUPOWON35JR5ESRJPTCWPUGYWGHJPSDVL4GA) context deadline exceeded; piecestore: (Node ID: 1Mg5mxUjW56quDidSgA788D7ByfAzvsEuWJqwtzY4bLyxRn3mu, Piece ID: ATPIXLMXGONSSLXPPVORSNC6LHPXI2V3UFTHM4VHA2FTY5ZM6H3A) context deadline exceeded; piecestore: (Node ID: 12QmYSEAKnkSLEEVc9E1uxtVigunKMFLk2cdu71Bse8Tqec3HyT, Piece ID: YFKD7GO2FOSPLDXNFPK732QTLCLBQQASWBTMZ2LTLBGSK22PIXIQ) context deadline exceeded; piecestore: (Node ID: 1JsnHuhQhM4rWLTLEuSSHLNE24m7tSW9qMSf48j7yMrAVPWVyV, Piece ID: ARBMDAKX4QBW6JG47KP2PUITCX7HKXCBO75THQPINM4LLHCKAV4Q) context deadline exceeded ======LOGS====== upload=433902]: Uploading piece... Dec 26 05:50:34.986155265 (494h21m34.8572383s) [noise=true , address=64.49.13.140:4509 , piece_id=6TGWTOAVY4YVAKU6VEE2SWVAM264TKU6ZNMGPWH4TNGII3UYXFXQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.986703610 (494h21m34.857786655s) [noise=true , address=77.90.31.10:4522 , piece_id=YFKD7GO2FOSPLDXNFPK732QTLCLBQQASWBTMZ2LTLBGSK22PIXIQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.987145380 (494h21m34.858228425s) [noise=true , address=145.79.227.33:4890 , piece_id=UKNB4ZZTYJGY4MDHOBPYUTHCP3ZDCMJB7O7AJZFYP2HFVIWLUOKQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.987628290 (494h21m34.858711335s) [noise=true , address=145.79.227.40:4052 , piece_id=XRZYL7PT5VHY22N7CCKS3NTZKDDWE36ZU3SORF6ILM7ZMXDND63A, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.988710731 (494h21m34.859793776s) [noise=true , address=77.90.31.13:4653 , piece_id=EY2CWX54ONOWPO3WC33G5QSP5RMP2SUTBQLUOH5JXGDRUEPDHQ2A, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.989894545 (494h21m34.86097759s) [noise=true , address=145.79.227.30:4785 , piece_id=5KMLU4ATBN4YVQDYJKKQY4PEJNKWTO2WNX5GMGJM4F54B6YM2SFA, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.991305081 (494h21m34.862388116s) [noise=true , address=145.79.227.50:4381 , piece_id=H6Q47DIOTOWGN7QH2KWT4IH6KVDSB246S22753Z3UQGB26T6VXPA, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.995796263 (494h21m34.866879359s) [noise=true , address=145.79.227.52:4461 , piece_id=ZAAUDBGY6CC2JNEGWXVPCBAWXARHAJDZ3YPB6Y3NWEWCFDUFGDSQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.996765439 (494h21m34.867848484s) [noise=true , address=77.90.31.16:4755 , piece_id=ARBMDAKX4QBW6JG47KP2PUITCX7HKXCBO75THQPINM4LLHCKAV4Q, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.998107634 (494h21m34.869190679s) [noise=true , address=77.90.31.11:4579 , piece_id=SBCEX7UNWGNFF54SBP4AGRDV5IPG24FQJXJJ7EYQTBSZKOVZWU4Q, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.999340442 (494h21m34.870423477s) [noise=true , address=145.79.227.34:4936 , piece_id=EEA2ZPM44DX3WMI7ZJTDC7T5UIGED6TNT53QXZWGBFRMQUQSCXPQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:35.000540757 (494h21m34.871623812s) [noise=true , address=45.91.116.14:4204 , piece_id=XV7FQGBJNWVRLOAFRUPOWON35JR5ESRJPTCWPUGYWGHJPSDVL4GA, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:36.360077064 (494h21m36.231160099s) [upload=433902]: Next returned nil segment Dec 26 05:50:36.360091642 (494h21m36.231174688s) [upload=433902]: Waiting for error group managing segments... Dec 26 05:50:36.688914299 (494h21m36.559997344s) [noise=true , address=64.49.13.140:4509 , piece_id=6TGWTOAVY4YVAKU6VEE2SWVAM264TKU6ZNMGPWH4TNGII3UYXFXQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.059812634 (494h21m36.930895669s) [noise=true , address=77.90.31.11:4579 , piece_id=SBCEX7UNWGNFF54SBP4AGRDV5IPG24FQJXJJ7EYQTBSZKOVZWU4Q, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.119433874 (494h21m36.990516909s) [noise=true , address=45.91.116.14:4204 , piece_id=XV7FQGBJNWVRLOAFRUPOWON35JR5ESRJPTCWPUGYWGHJPSDVL4GA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.138718204 (494h21m37.00980125s) [noise=true , address=77.90.31.16:4755 , piece_id=ARBMDAKX4QBW6JG47KP2PUITCX7HKXCBO75THQPINM4LLHCKAV4Q, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.146647243 (494h21m37.017730278s) [noise=true , address=77.90.31.13:4653 , piece_id=EY2CWX54ONOWPO3WC33G5QSP5RMP2SUTBQLUOH5JXGDRUEPDHQ2A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.163283722 (494h21m37.034366757s) [noise=true , address=77.90.31.21:4893 , piece_id=GNEEZO2XGOKNODOUPZQ4NGKFE4PCX5LNUWXLV27F6X7GWZVSYNAA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.374956574 (494h21m37.246039609s) [noise=true , address=45.91.116.7:4033 , piece_id=ATPIXLMXGONSSLXPPVORSNC6LHPXI2V3UFTHM4VHA2FTY5ZM6H3A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.777283355 (494h21m37.64836639s) [noise=true , address=77.90.31.12:4622 , piece_id=LTYYPK6ZQ3NXZPBKSV4OMZGCOXXV3N6L5A6MVODNR644D2ZUJX3A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:38.439025909 (494h21m38.310108944s) [noise=true , address=145.79.227.49:4355 , piece_id=IUDOHR73Y6OKFAQSNNSONALWAITRJ4POYRO2QD5LYZKOSTWY4MNA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:38.439599111 (494h21m38.310682166s) [noise=true , address=77.90.31.10:4522 , piece_id=YFKD7GO2FOSPLDXNFPK732QTLCLBQQASWBTMZ2LTLBGSK22PIXIQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:39.259154980 (494h21m39.130238015s) [noise=true , address=145.79.227.34:4936 , piece_id=EEA2ZPM44DX3WMI7ZJTDC7T5UIGED6TNT53QXZWGBFRMQUQSCXPQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:39.430935921 (494h21m39.302018956s) [noise=true , address=145.79.227.44:4212 , piece_id=ABBNLKRE4FM7RHF56V4IAQPKBCCCQLLOMFVMSJEWQUNFGGCI3PVQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:42.941071297 (494h21m42.812154352s) [noise=true , address=77.90.31.14:4675 , piece_id=FKB4O45OYICMELPGXB65QAUD2DRIFTSXZFKIA6TY26VFAP6AVY3A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:51:22.719026850 (494h22m22.590109885s) [noise=true , address=145.79.227.33:4890 , piece_id=UKNB4ZZTYJGY4MDHOBPYUTHCP3ZDCMJB7O7AJZFYP2HFVIWLUOKQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:51:22.719060193 (494h22m22.590143228s) [seg_pos={0 0}, upload=433902]: Segment reached optimal threshold of 14 pieces. Dec 26 05:51:22.719233022 (494h22m22.590316067s) [noise=true , address=145.79.227.50:4381 , piece_id=H6Q47DIOTOWGN7QH2KWT4IH6KVDSB246S22753Z3UQGB26T6VXPA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zU35MWWaaC1npkP5qGcWBXAkfxGsd5UsdkM2nwX9x7wUGCDbY): protocol: write tcp 188.166.225.170:46970->145.79.227.50:4381: use of closed network connection; write tcp 188.166.225.170:46970->145.79.227.50:4381: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46970->145.79.227.50:4381: use of closed network connection Dec 26 05:51:22.719238783 (494h22m22.590321828s) [noise=true , address=145.79.227.50:4381 , piece_id=H6Q47DIOTOWGN7QH2KWT4IH6KVDSB246S22753Z3UQGB26T6VXPA, seg_pos={0 0}, upload=433902]: Long tail cancellation due to optimal threshold Dec 26 05:51:22.719251677 (494h22m22.590334712s) [noise=true , address=145.79.227.30:4785 , piece_id=5KMLU4ATBN4YVQDYJKKQY4PEJNKWTO2WNX5GMGJM4F54B6YM2SFA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gYH83ECV8wai3CZyBSU8NNJM1UsysLYHbQDwBVKZRyjsdnUwB): protocol: write tcp 188.166.225.170:58398->145.79.227.30:4785: use of closed network connection; write tcp 188.166.225.170:58398->145.79.227.30:4785: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58398->145.79.227.30:4785: use of closed network connection Dec 26 05:51:22.719260194 (494h22m22.590343229s) [noise=true , address=145.79.227.30:4785 , piece_id=5KMLU4ATBN4YVQDYJKKQY4PEJNKWTO2WNX5GMGJM4F54B6YM2SFA, seg_pos={0 0}, upload=433902]: Long tail cancellation due to optimal threshold Dec 26 05:51:22.719409408 (494h22m22.590492443s) [noise=true , address=145.79.227.40:4052 , piece_id=XRZYL7PT5VHY22N7CCKS3NTZKDDWE36ZU3SORF6ILM7ZMXDND63A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PViJBYAanwXRWhMq3WHR6k9Stxy8hjuY5FV9J7ErxCy4N1q3V): protocol: write tcp 188.166.225.170:45984->145.79.227.40:4052: use of closed network connection; write tcp 188.166.225.170:45984->145.79.227.40:4052: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45984->145.79.227.40:4052: use of closed network connection Dec 26 05:51:22.719415469 (494h22m22.590498515s) [noise=true , address=145.79.227.40:4052 , piece_id=XRZYL7PT5VHY22N7CCKS3NTZKDDWE36ZU3SORF6ILM7ZMXDND63A, seg_pos={0 0}, upload=433902]: Long tail cancellation due to optimal threshold Dec 26 05:51:22.719467057 (494h22m22.590550092s) [noise=true , address=145.79.227.52:4461 , piece_id=ZAAUDBGY6CC2JNEGWXVPCBAWXARHAJDZ3YPB6Y3NWEWCFDUFGDSQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fNrbSfErEQRnruJCJAVvkoNwQSPWnvNtxunHn6zrQegnNAkxA): protocol: write tcp 188.166.225.170:55966->145.79.227.52:4461: use of closed network connection; write tcp 188.166.225.170:55966->145.79.227.52:4461: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55966->145.79.227.52:4461: use of closed network connection Dec 26 05:51:22.719473931 (494h22m22.590556966s) [noise=true , address=145.79.227.52:4461 , piece_id=ZAAUDBGY6CC2JNEGWXVPCBAWXARHAJDZ3YPB6Y3NWEWCFDUFGDSQ, seg_pos={0 0}, upload=433902]: Long tail cancellation due to optimal threshold Dec 26 05:51:22.719491374 (494h22m22.590574409s) [seg_pos={0 0}, upload=433902]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 05:51:22.719562539 (494h22m22.590645574s) [upload=433902]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:51:22.719567228 (494h22m22.590650263s) [upload=433902]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:51:23.008169117 (494h22m22.879252162s) [upload=433902]: Done uploading segments... Dec 26 05:51:23.008191560 (494h22m22.879274605s) [upload=433902]: Upload finished. err: <nil> Duration: 30.795193477s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.................................X..........................] duration: 3.479296745s trace: https://jaeger.dev.storj.io/trace/630094597d401477 FAIL (2025-12-26 06:28:38 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.479321071s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [...X..X.X.......................X..........X.......X.X......] duration: 5.903466237s trace: https://jaeger.dev.storj.io/trace/67a56359499790c8 FAIL (2025-12-26 06:28:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 06:23:47.396606446 (494h54m47.267689491s) [upload=434386]: Starting upload Dec 26 06:23:47.396675247 (494h54m47.267758292s) [upload=434386]: Done starting upload Dec 26 06:23:47.396898202 (494h54m47.267981248s) [upload=434386]: Uploading segments... Dec 26 06:23:47.396953998 (494h54m47.268037073s) [upload=434386]: Got next segment. Inline: false Dec 26 06:23:47.397031125 (494h54m47.26811416s) [upload=434386]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 06:23:47.397037317 (494h54m47.268120362s) [upload=434386]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 06:23:47.643130652 (494h54m47.514213687s) [seg_pos={0 0}, upload=434386]: Begin upload segment... Dec 26 06:23:47.643171870 (494h54m47.514254905s) [seg_pos={0 0}, upload=434386]: Done begin upload segment. Dec 26 06:23:47.643193913 (494h54m47.514276948s) [noise=true , address=145.79.227.38:4977 , piece_id=PKXAJYGJPG6GYOG6DT4M6TBR54TTMBSWBXIIW2RLVMSA4HXBGOUQ, seg_pos={0 0}, upload=434386]: Uploading piece... Dec 26 06:23:47.643278103 (494h54m47.514361138s) [noise=true , address=145.79.227.49:4339 , piece_id=XQ6XNO7FGYMNEWYDGWGIB5IY5IHOAOPK7TUWPVPSEGQRIDA4R6GA, seg_pos={0 0}, upload=434386]: Uploading piece... Dec 26 06:23:47.644040174 (494h54m47.515123219s) [noise=true , address=145.79.227.29:4942 , piece_id=GML4O7CTQ3AR5KAPGOZAMBELR7F6SDGSJSAWPZLO27W53ALPNZ2Q, seg_pos={0 0}, upload=434386]: Uploading piece... Dec 26 06:23:47.644250264 (494h54m47.515333309s) [noise=true , address=145.79.227.3:4149 , piece_id=TOTASJEGQWCGG5KISNGGY3TLAL354QWMFA4VKEKDSCKHLY25PYQQ, seg_pos={0 0}, upload=434386]: Uploading piece... Dec 26 06:23:47.644310018 (494h54m47.515393063s) [noise=true , address=145.79.227.5:4221 , piece_id=U2IAUDBXHO2UXGPSOYT3L2FPIJZFDTJ4GQCHGGXGSFD62KXIV77Q, seg_pos={0 0}, upload=434386]: Uploading piece... Dec 26 06:23:47.644590933 (494h54m47.515673968s) [noise=true , address=77.90.31.21:4880 , piece_id=JLHHQM55ZRSYUHKTKMZOMU4X3KXNQHIXH32KWIFA3CLWADW37W7Q, seg_pos={0 0}, upload=434386]: Uploading piece... Dec 26 06:23:47.644789010 (494h54m47.515872055s) [noise=true , address=45.91.116.13:4153 , piece_id=7PR6IGAFJOAALB3YXBEWFAGCR72DFJ6JWT24WCO7FHUOCSMJNP7A, seg_pos={0 0}, upload=434386]: Uploading piece... Dec 26 06:23:47.644945428 (494h54m47.516028463s) [noise=true , address=45.91.116.14:4191 , piece_id=LPPDT6VMSHKDJWZLB25GN2XLHS73352JSKCRZNEAFRRRWDLA7RNQ, seg_pos={0 0}, upload=434386]: Uploading piece... Dec 26 06:23:47.645273222 (494h54m47.516356257s) [noise=true , address=45.91.116.7:4010 , piece_id=LHU4DWQU7XLA3QTJ2LHPSLGON43MUO3BXHJ5FRZPIRSA5IRH2LNA, seg_pos={0 0}, upload=434386]: Uploading piece... Dec 26 06:23:47.645283942 (494h54m47.516366978s) [noise=true , address=45.91.116.9:4264 , piece_id=N7AODEQP7YXGHYUJCT27BSAFMEJB3ABXUUCEUPSWDUR3F7DUB5TA, seg_pos={0 0}, upload=434386]: Uploading piece... Dec 26 06:23:47.645654749 (494h54m47.516737784s) [noise=true , address=145.79.227.2:4115 , piece_id=D6S76VZDC2FBIFA3EA3TC4NEIFKJSGX6JRD4S26UO2IOLUDUPIDA, seg_pos={0 0}, upload=434386]: Uploading piece... Dec 26 06:23:47.645885748 (494h54m47.516968793s) [noise=true , address=145.79.227.20:4437 , piece_id=P4DNS6VYA2TS43TL7LYUSRUKFDJ37PO2M6D6NP534QQQBH2WIV2A, seg_pos={0 0}, upload=434386]: Uploading piece... Dec 26 06:23:47.646632821 (494h54m47.517715866s) [noise=true , address=77.90.31.15:4700 , piece_id=23B7NZUGOMXA6OMBSKY3HWKZ36VVIVOCQGFBLFZ6EHGC7TD544YA, seg_pos={0 0}, upload=434386]: Uploading piece... Dec 26 06:23:47.647299961 (494h54m47.518382996s) [noise=true , address=145.79.227.39:4013 , piece_id=3EHFPZVBIV6NNDJ46Z5YHVOPBWEMYQLG6NCJ4HT6ZX3IVMGVAFAQ, seg_pos={0 0}, upload=434386]: Uploading piece... Dec 26 06:23:47.647743005 (494h54m47.51882605s) [noise=true , address=45.91.116.11:4118 , piece_id=6BZNBNEDJB4RNNFMAXRMGLKK4GHYTDNKMVTMG5RF3U2VDKMON2KQ, seg_pos={0 0}, upload=434386]: Uploading piece... Dec 26 06:23:47.647943015 (494h54m47.51902606s) [noise=true , address=145.79.227.6:4259 , piece_id=TGQ6V3HNT3X2IY7MQ73RZRNH5NDXENTRDCXWHFHQETQEILSO7FEA, seg_pos={0 0}, upload=434386]: Uploading piece... Dec 26 06:23:49.283219995 (494h54m49.15430303s) [upload=434386]: Next returned nil segment Dec 26 06:23:49.283245634 (494h54m49.154328679s) [upload=434386]: Waiting for error group managing segments... Dec 26 06:23:49.644523232 (494h54m49.515606267s) [noise=true , address=145.79.227.3:4149 , piece_id=TOTASJEGQWCGG5KISNGGY3TLAL354QWMFA4VKEKDSCKHLY25PYQQ, seg_pos={0 0}, upload=434386]: Done uploading piece. err: <nil> Dec 26 06:23:49.648027525 (494h54m49.51911056s) [noise=true , address=145.79.227.49:4339 , piece_id=XQ6XNO7FGYMNEWYDGWGIB5IY5IHOAOPK7TUWPVPSEGQRIDA4R6GA, seg_pos={0 0}, upload=434386]: Done uploading piece. err: <nil> Dec 26 06:23:49.651435925 (494h54m49.52251897s) [noise=true , address=145.79.227.38:4977 , piece_id=PKXAJYGJPG6GYOG6DT4M6TBR54TTMBSWBXIIW2RLVMSA4HXBGOUQ, seg_pos={0 0}, upload=434386]: Done uploading piece. err: <nil> Dec 26 06:23:49.655712879 (494h54m49.526795924s) [noise=true , address=145.79.227.2:4115 , piece_id=D6S76VZDC2FBIFA3EA3TC4NEIFKJSGX6JRD4S26UO2IOLUDUPIDA, seg_pos={0 0}, upload=434386]: Done uploading piece. err: <nil> Dec 26 06:23:49.658634573 (494h54m49.529717618s) [noise=true , address=145.79.227.39:4013 , piece_id=3EHFPZVBIV6NNDJ46Z5YHVOPBWEMYQLG6NCJ4HT6ZX3IVMGVAFAQ, seg_pos={0 0}, upload=434386]: Done uploading piece. err: <nil> Dec 26 06:23:49.670194264 (494h54m49.541277309s) [noise=true , address=145.79.227.29:4942 , piece_id=GML4O7CTQ3AR5KAPGOZAMBELR7F6SDGSJSAWPZLO27W53ALPNZ2Q, seg_pos={0 0}, upload=434386]: Done uploading piece. err: <nil> Dec 26 06:23:49.672063052 (494h54m49.543146097s) [noise=true , address=145.79.227.5:4221 , piece_id=U2IAUDBXHO2UXGPSOYT3L2FPIJZFDTJ4GQCHGGXGSFD62KXIV77Q, seg_pos={0 0}, upload=434386]: Done uploading piece. err: <nil> Dec 26 06:23:49.672693352 (494h54m49.543776387s) [noise=true , address=145.79.227.20:4437 , piece_id=P4DNS6VYA2TS43TL7LYUSRUKFDJ37PO2M6D6NP534QQQBH2WIV2A, seg_pos={0 0}, upload=434386]: Done uploading piece. err: <nil> Dec 26 06:23:49.725648318 (494h54m49.596731363s) [noise=true , address=145.79.227.6:4259 , piece_id=TGQ6V3HNT3X2IY7MQ73RZRNH5NDXENTRDCXWHFHQETQEILSO7FEA, seg_pos={0 0}, upload=434386]: Done uploading piece. err: <nil> Dec 26 06:23:49.861945242 (494h54m49.733028287s) [noise=true , address=77.90.31.21:4880 , piece_id=JLHHQM55ZRSYUHKTKMZOMU4X3KXNQHIXH32KWIFA3CLWADW37W7Q, seg_pos={0 0}, upload=434386]: Done uploading piece. err: <nil> Dec 26 06:23:49.958824257 (494h54m49.829907292s) [noise=true , address=45.91.116.9:4264 , piece_id=N7AODEQP7YXGHYUJCT27BSAFMEJB3ABXUUCEUPSWDUR3F7DUB5TA, seg_pos={0 0}, upload=434386]: Done uploading piece. err: <nil> Dec 26 06:23:49.970075321 (494h54m49.841158366s) [noise=true , address=45.91.116.13:4153 , piece_id=7PR6IGAFJOAALB3YXBEWFAGCR72DFJ6JWT24WCO7FHUOCSMJNP7A, seg_pos={0 0}, upload=434386]: Done uploading piece. err: <nil> Dec 26 06:23:50.140738494 (494h54m50.011821529s) [noise=true , address=77.90.31.15:4700 , piece_id=23B7NZUGOMXA6OMBSKY3HWKZ36VVIVOCQGFBLFZ6EHGC7TD544YA, seg_pos={0 0}, upload=434386]: Done uploading piece. err: <nil> Dec 26 06:23:50.140774813 (494h54m50.011857849s) [seg_pos={0 0}, upload=434386]: Segment reached optimal threshold of 13 pieces. Dec 26 06:23:50.140993680 (494h54m50.012076725s) [noise=true , address=45.91.116.14:4191 , piece_id=LPPDT6VMSHKDJWZLB25GN2XLHS73352JSKCRZNEAFRRRWDLA7RNQ, seg_pos={0 0}, upload=434386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Dec 26 06:23:50.141002648 (494h54m50.012085683s) [noise=true , address=45.91.116.14:4191 , piece_id=LPPDT6VMSHKDJWZLB25GN2XLHS73352JSKCRZNEAFRRRWDLA7RNQ, seg_pos={0 0}, upload=434386]: Long tail cancellation due to optimal threshold Dec 26 06:23:50.141012116 (494h54m50.012095161s) [noise=true , address=45.91.116.7:4010 , piece_id=LHU4DWQU7XLA3QTJ2LHPSLGON43MUO3BXHJ5FRZPIRSA5IRH2LNA, seg_pos={0 0}, upload=434386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 26 06:23:50.141016754 (494h54m50.012099789s) [noise=true , address=45.91.116.7:4010 , piece_id=LHU4DWQU7XLA3QTJ2LHPSLGON43MUO3BXHJ5FRZPIRSA5IRH2LNA, seg_pos={0 0}, upload=434386]: Long tail cancellation due to optimal threshold Dec 26 06:23:50.141145951 (494h54m50.012228986s) [noise=true , address=45.91.116.11:4118 , piece_id=6BZNBNEDJB4RNNFMAXRMGLKK4GHYTDNKMVTMG5RF3U2VDKMON2KQ, seg_pos={0 0}, upload=434386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: expected piece hash; context canceled; EOF Dec 26 06:23:50.141156651 (494h54m50.012239686s) [noise=true , address=45.91.116.11:4118 , piece_id=6BZNBNEDJB4RNNFMAXRMGLKK4GHYTDNKMVTMG5RF3U2VDKMON2KQ, seg_pos={0 0}, upload=434386]: Long tail cancellation due to optimal threshold Dec 26 06:23:50.141172291 (494h54m50.012255336s) [seg_pos={0 0}, upload=434386]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 06:23:50.141260609 (494h54m50.012343644s) [upload=434386]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 06:23:50.141265488 (494h54m50.012348533s) [upload=434386]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 06:23:50.438800618 (494h54m50.309883653s) [upload=434386]: Done uploading segments... Dec 26 06:23:50.438865121 (494h54m50.309948166s) [upload=434386]: Upload finished. err: <nil> Duration: 5.903490803s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..................................X.........................] duration: 3.534617943s trace: https://jaeger.dev.storj.io/trace/630094597d401477 FAIL (2025-12-26 06:28:07 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.534646297s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 35.956327304s trace: https://jaeger.dev.storj.io/trace/7e198d4c8fea54d2 FAIL (2025-12-26 06:28:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p, Piece ID: SBCEX7UNWGNFF54SBP4AGRDV5IPG24FQJXJJ7EYQTBSZKOVZWU4Q) context deadline exceeded; piecestore: (Node ID: 1nP8xospdMuDEh3em5K6iBnh8bHXqLhvHy4pHm9m8Lnj4kEH2X, Piece ID: XV7FQGBJNWVRLOAFRUPOWON35JR5ESRJPTCWPUGYWGHJPSDVL4GA) context deadline exceeded; piecestore: (Node ID: 1Mg5mxUjW56quDidSgA788D7ByfAzvsEuWJqwtzY4bLyxRn3mu, Piece ID: ATPIXLMXGONSSLXPPVORSNC6LHPXI2V3UFTHM4VHA2FTY5ZM6H3A) context deadline exceeded; piecestore: (Node ID: 12QmYSEAKnkSLEEVc9E1uxtVigunKMFLk2cdu71Bse8Tqec3HyT, Piece ID: YFKD7GO2FOSPLDXNFPK732QTLCLBQQASWBTMZ2LTLBGSK22PIXIQ) context deadline exceeded; piecestore: (Node ID: 1JsnHuhQhM4rWLTLEuSSHLNE24m7tSW9qMSf48j7yMrAVPWVyV, Piece ID: ARBMDAKX4QBW6JG47KP2PUITCX7HKXCBO75THQPINM4LLHCKAV4Q) context deadline exceeded ======LOGS====== upload=433902]: Uploading piece... Dec 26 05:50:34.986155265 (494h21m34.8572383s) [noise=true , address=64.49.13.140:4509 , piece_id=6TGWTOAVY4YVAKU6VEE2SWVAM264TKU6ZNMGPWH4TNGII3UYXFXQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.986703610 (494h21m34.857786655s) [noise=true , address=77.90.31.10:4522 , piece_id=YFKD7GO2FOSPLDXNFPK732QTLCLBQQASWBTMZ2LTLBGSK22PIXIQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.987145380 (494h21m34.858228425s) [noise=true , address=145.79.227.33:4890 , piece_id=UKNB4ZZTYJGY4MDHOBPYUTHCP3ZDCMJB7O7AJZFYP2HFVIWLUOKQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.987628290 (494h21m34.858711335s) [noise=true , address=145.79.227.40:4052 , piece_id=XRZYL7PT5VHY22N7CCKS3NTZKDDWE36ZU3SORF6ILM7ZMXDND63A, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.988710731 (494h21m34.859793776s) [noise=true , address=77.90.31.13:4653 , piece_id=EY2CWX54ONOWPO3WC33G5QSP5RMP2SUTBQLUOH5JXGDRUEPDHQ2A, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.989894545 (494h21m34.86097759s) [noise=true , address=145.79.227.30:4785 , piece_id=5KMLU4ATBN4YVQDYJKKQY4PEJNKWTO2WNX5GMGJM4F54B6YM2SFA, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.991305081 (494h21m34.862388116s) [noise=true , address=145.79.227.50:4381 , piece_id=H6Q47DIOTOWGN7QH2KWT4IH6KVDSB246S22753Z3UQGB26T6VXPA, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.995796263 (494h21m34.866879359s) [noise=true , address=145.79.227.52:4461 , piece_id=ZAAUDBGY6CC2JNEGWXVPCBAWXARHAJDZ3YPB6Y3NWEWCFDUFGDSQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.996765439 (494h21m34.867848484s) [noise=true , address=77.90.31.16:4755 , piece_id=ARBMDAKX4QBW6JG47KP2PUITCX7HKXCBO75THQPINM4LLHCKAV4Q, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.998107634 (494h21m34.869190679s) [noise=true , address=77.90.31.11:4579 , piece_id=SBCEX7UNWGNFF54SBP4AGRDV5IPG24FQJXJJ7EYQTBSZKOVZWU4Q, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.999340442 (494h21m34.870423477s) [noise=true , address=145.79.227.34:4936 , piece_id=EEA2ZPM44DX3WMI7ZJTDC7T5UIGED6TNT53QXZWGBFRMQUQSCXPQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:35.000540757 (494h21m34.871623812s) [noise=true , address=45.91.116.14:4204 , piece_id=XV7FQGBJNWVRLOAFRUPOWON35JR5ESRJPTCWPUGYWGHJPSDVL4GA, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:36.360077064 (494h21m36.231160099s) [upload=433902]: Next returned nil segment Dec 26 05:50:36.360091642 (494h21m36.231174688s) [upload=433902]: Waiting for error group managing segments... Dec 26 05:50:36.688914299 (494h21m36.559997344s) [noise=true , address=64.49.13.140:4509 , piece_id=6TGWTOAVY4YVAKU6VEE2SWVAM264TKU6ZNMGPWH4TNGII3UYXFXQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.059812634 (494h21m36.930895669s) [noise=true , address=77.90.31.11:4579 , piece_id=SBCEX7UNWGNFF54SBP4AGRDV5IPG24FQJXJJ7EYQTBSZKOVZWU4Q, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.119433874 (494h21m36.990516909s) [noise=true , address=45.91.116.14:4204 , piece_id=XV7FQGBJNWVRLOAFRUPOWON35JR5ESRJPTCWPUGYWGHJPSDVL4GA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.138718204 (494h21m37.00980125s) [noise=true , address=77.90.31.16:4755 , piece_id=ARBMDAKX4QBW6JG47KP2PUITCX7HKXCBO75THQPINM4LLHCKAV4Q, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.146647243 (494h21m37.017730278s) [noise=true , address=77.90.31.13:4653 , piece_id=EY2CWX54ONOWPO3WC33G5QSP5RMP2SUTBQLUOH5JXGDRUEPDHQ2A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.163283722 (494h21m37.034366757s) [noise=true , address=77.90.31.21:4893 , piece_id=GNEEZO2XGOKNODOUPZQ4NGKFE4PCX5LNUWXLV27F6X7GWZVSYNAA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.374956574 (494h21m37.246039609s) [noise=true , address=45.91.116.7:4033 , piece_id=ATPIXLMXGONSSLXPPVORSNC6LHPXI2V3UFTHM4VHA2FTY5ZM6H3A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.777283355 (494h21m37.64836639s) [noise=true , address=77.90.31.12:4622 , piece_id=LTYYPK6ZQ3NXZPBKSV4OMZGCOXXV3N6L5A6MVODNR644D2ZUJX3A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:38.439025909 (494h21m38.310108944s) [noise=true , address=145.79.227.49:4355 , piece_id=IUDOHR73Y6OKFAQSNNSONALWAITRJ4POYRO2QD5LYZKOSTWY4MNA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:38.439599111 (494h21m38.310682166s) [noise=true , address=77.90.31.10:4522 , piece_id=YFKD7GO2FOSPLDXNFPK732QTLCLBQQASWBTMZ2LTLBGSK22PIXIQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:39.259154980 (494h21m39.130238015s) [noise=true , address=145.79.227.34:4936 , piece_id=EEA2ZPM44DX3WMI7ZJTDC7T5UIGED6TNT53QXZWGBFRMQUQSCXPQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:39.430935921 (494h21m39.302018956s) [noise=true , address=145.79.227.44:4212 , piece_id=ABBNLKRE4FM7RHF56V4IAQPKBCCCQLLOMFVMSJEWQUNFGGCI3PVQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:42.941071297 (494h21m42.812154352s) [noise=true , address=77.90.31.14:4675 , piece_id=FKB4O45OYICMELPGXB65QAUD2DRIFTSXZFKIA6TY26VFAP6AVY3A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:51:22.719026850 (494h22m22.590109885s) [noise=true , address=145.79.227.33:4890 , piece_id=UKNB4ZZTYJGY4MDHOBPYUTHCP3ZDCMJB7O7AJZFYP2HFVIWLUOKQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:51:22.719060193 (494h22m22.590143228s) [seg_pos={0 0}, upload=433902]: Segment reached optimal threshold of 14 pieces. Dec 26 05:51:22.719233022 (494h22m22.590316067s) [noise=true , address=145.79.227.50:4381 , piece_id=H6Q47DIOTOWGN7QH2KWT4IH6KVDSB246S22753Z3UQGB26T6VXPA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zU35MWWaaC1npkP5qGcWBXAkfxGsd5UsdkM2nwX9x7wUGCDbY): protocol: write tcp 188.166.225.170:46970->145.79.227.50:4381: use of closed network connection; write tcp 188.166.225.170:46970->145.79.227.50:4381: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46970->145.79.227.50:4381: use of closed network connection Dec 26 05:51:22.719238783 (494h22m22.590321828s) [noise=true , address=145.79.227.50:4381 , piece_id=H6Q47DIOTOWGN7QH2KWT4IH6KVDSB246S22753Z3UQGB26T6VXPA, seg_pos={0 0}, upload=433902]: Long tail cancellation due to optimal threshold Dec 26 05:51:22.719251677 (494h22m22.590334712s) [noise=true , address=145.79.227.30:4785 , piece_id=5KMLU4ATBN4YVQDYJKKQY4PEJNKWTO2WNX5GMGJM4F54B6YM2SFA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gYH83ECV8wai3CZyBSU8NNJM1UsysLYHbQDwBVKZRyjsdnUwB): protocol: write tcp 188.166.225.170:58398->145.79.227.30:4785: use of closed network connection; write tcp 188.166.225.170:58398->145.79.227.30:4785: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58398->145.79.227.30:4785: use of closed network connection Dec 26 05:51:22.719260194 (494h22m22.590343229s) [noise=true , address=145.79.227.30:4785 , piece_id=5KMLU4ATBN4YVQDYJKKQY4PEJNKWTO2WNX5GMGJM4F54B6YM2SFA, seg_pos={0 0}, upload=433902]: Long tail cancellation due to optimal threshold Dec 26 05:51:22.719409408 (494h22m22.590492443s) [noise=true , address=145.79.227.40:4052 , piece_id=XRZYL7PT5VHY22N7CCKS3NTZKDDWE36ZU3SORF6ILM7ZMXDND63A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PViJBYAanwXRWhMq3WHR6k9Stxy8hjuY5FV9J7ErxCy4N1q3V): protocol: write tcp 188.166.225.170:45984->145.79.227.40:4052: use of closed network connection; write tcp 188.166.225.170:45984->145.79.227.40:4052: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45984->145.79.227.40:4052: use of closed network connection Dec 26 05:51:22.719415469 (494h22m22.590498515s) [noise=true , address=145.79.227.40:4052 , piece_id=XRZYL7PT5VHY22N7CCKS3NTZKDDWE36ZU3SORF6ILM7ZMXDND63A, seg_pos={0 0}, upload=433902]: Long tail cancellation due to optimal threshold Dec 26 05:51:22.719467057 (494h22m22.590550092s) [noise=true , address=145.79.227.52:4461 , piece_id=ZAAUDBGY6CC2JNEGWXVPCBAWXARHAJDZ3YPB6Y3NWEWCFDUFGDSQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fNrbSfErEQRnruJCJAVvkoNwQSPWnvNtxunHn6zrQegnNAkxA): protocol: write tcp 188.166.225.170:55966->145.79.227.52:4461: use of closed network connection; write tcp 188.166.225.170:55966->145.79.227.52:4461: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55966->145.79.227.52:4461: use of closed network connection Dec 26 05:51:22.719473931 (494h22m22.590556966s) [noise=true , address=145.79.227.52:4461 , piece_id=ZAAUDBGY6CC2JNEGWXVPCBAWXARHAJDZ3YPB6Y3NWEWCFDUFGDSQ, seg_pos={0 0}, upload=433902]: Long tail cancellation due to optimal threshold Dec 26 05:51:22.719491374 (494h22m22.590574409s) [seg_pos={0 0}, upload=433902]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 05:51:22.719562539 (494h22m22.590645574s) [upload=433902]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:51:22.719567228 (494h22m22.590650263s) [upload=433902]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:51:23.008169117 (494h22m22.879252162s) [upload=433902]: Done uploading segments... Dec 26 05:51:23.008191560 (494h22m22.879274605s) [upload=433902]: Upload finished. err: <nil> Duration: 35.956356418s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [....X..X.X.......................X..........X.......X.X.....] duration: 5.967333782s trace: https://jaeger.dev.storj.io/trace/67a56359499790c8 FAIL (2025-12-26 06:27:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 06:23:47.396606446 (494h54m47.267689491s) [upload=434386]: Starting upload Dec 26 06:23:47.396675247 (494h54m47.267758292s) [upload=434386]: Done starting upload Dec 26 06:23:47.396898202 (494h54m47.267981248s) [upload=434386]: Uploading segments... Dec 26 06:23:47.396953998 (494h54m47.268037073s) [upload=434386]: Got next segment. Inline: false Dec 26 06:23:47.397031125 (494h54m47.26811416s) [upload=434386]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 06:23:47.397037317 (494h54m47.268120362s) [upload=434386]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 06:23:47.643130652 (494h54m47.514213687s) [seg_pos={0 0}, upload=434386]: Begin upload segment... Dec 26 06:23:47.643171870 (494h54m47.514254905s) [seg_pos={0 0}, upload=434386]: Done begin upload segment. Dec 26 06:23:47.643193913 (494h54m47.514276948s) [noise=true , address=145.79.227.38:4977 , piece_id=PKXAJYGJPG6GYOG6DT4M6TBR54TTMBSWBXIIW2RLVMSA4HXBGOUQ, seg_pos={0 0}, upload=434386]: Uploading piece... Dec 26 06:23:47.643278103 (494h54m47.514361138s) [noise=true , address=145.79.227.49:4339 , piece_id=XQ6XNO7FGYMNEWYDGWGIB5IY5IHOAOPK7TUWPVPSEGQRIDA4R6GA, seg_pos={0 0}, upload=434386]: Uploading piece... Dec 26 06:23:47.644040174 (494h54m47.515123219s) [noise=true , address=145.79.227.29:4942 , piece_id=GML4O7CTQ3AR5KAPGOZAMBELR7F6SDGSJSAWPZLO27W53ALPNZ2Q, seg_pos={0 0}, upload=434386]: Uploading piece... Dec 26 06:23:47.644250264 (494h54m47.515333309s) [noise=true , address=145.79.227.3:4149 , piece_id=TOTASJEGQWCGG5KISNGGY3TLAL354QWMFA4VKEKDSCKHLY25PYQQ, seg_pos={0 0}, upload=434386]: Uploading piece... Dec 26 06:23:47.644310018 (494h54m47.515393063s) [noise=true , address=145.79.227.5:4221 , piece_id=U2IAUDBXHO2UXGPSOYT3L2FPIJZFDTJ4GQCHGGXGSFD62KXIV77Q, seg_pos={0 0}, upload=434386]: Uploading piece... Dec 26 06:23:47.644590933 (494h54m47.515673968s) [noise=true , address=77.90.31.21:4880 , piece_id=JLHHQM55ZRSYUHKTKMZOMU4X3KXNQHIXH32KWIFA3CLWADW37W7Q, seg_pos={0 0}, upload=434386]: Uploading piece... Dec 26 06:23:47.644789010 (494h54m47.515872055s) [noise=true , address=45.91.116.13:4153 , piece_id=7PR6IGAFJOAALB3YXBEWFAGCR72DFJ6JWT24WCO7FHUOCSMJNP7A, seg_pos={0 0}, upload=434386]: Uploading piece... Dec 26 06:23:47.644945428 (494h54m47.516028463s) [noise=true , address=45.91.116.14:4191 , piece_id=LPPDT6VMSHKDJWZLB25GN2XLHS73352JSKCRZNEAFRRRWDLA7RNQ, seg_pos={0 0}, upload=434386]: Uploading piece... Dec 26 06:23:47.645273222 (494h54m47.516356257s) [noise=true , address=45.91.116.7:4010 , piece_id=LHU4DWQU7XLA3QTJ2LHPSLGON43MUO3BXHJ5FRZPIRSA5IRH2LNA, seg_pos={0 0}, upload=434386]: Uploading piece... Dec 26 06:23:47.645283942 (494h54m47.516366978s) [noise=true , address=45.91.116.9:4264 , piece_id=N7AODEQP7YXGHYUJCT27BSAFMEJB3ABXUUCEUPSWDUR3F7DUB5TA, seg_pos={0 0}, upload=434386]: Uploading piece... Dec 26 06:23:47.645654749 (494h54m47.516737784s) [noise=true , address=145.79.227.2:4115 , piece_id=D6S76VZDC2FBIFA3EA3TC4NEIFKJSGX6JRD4S26UO2IOLUDUPIDA, seg_pos={0 0}, upload=434386]: Uploading piece... Dec 26 06:23:47.645885748 (494h54m47.516968793s) [noise=true , address=145.79.227.20:4437 , piece_id=P4DNS6VYA2TS43TL7LYUSRUKFDJ37PO2M6D6NP534QQQBH2WIV2A, seg_pos={0 0}, upload=434386]: Uploading piece... Dec 26 06:23:47.646632821 (494h54m47.517715866s) [noise=true , address=77.90.31.15:4700 , piece_id=23B7NZUGOMXA6OMBSKY3HWKZ36VVIVOCQGFBLFZ6EHGC7TD544YA, seg_pos={0 0}, upload=434386]: Uploading piece... Dec 26 06:23:47.647299961 (494h54m47.518382996s) [noise=true , address=145.79.227.39:4013 , piece_id=3EHFPZVBIV6NNDJ46Z5YHVOPBWEMYQLG6NCJ4HT6ZX3IVMGVAFAQ, seg_pos={0 0}, upload=434386]: Uploading piece... Dec 26 06:23:47.647743005 (494h54m47.51882605s) [noise=true , address=45.91.116.11:4118 , piece_id=6BZNBNEDJB4RNNFMAXRMGLKK4GHYTDNKMVTMG5RF3U2VDKMON2KQ, seg_pos={0 0}, upload=434386]: Uploading piece... Dec 26 06:23:47.647943015 (494h54m47.51902606s) [noise=true , address=145.79.227.6:4259 , piece_id=TGQ6V3HNT3X2IY7MQ73RZRNH5NDXENTRDCXWHFHQETQEILSO7FEA, seg_pos={0 0}, upload=434386]: Uploading piece... Dec 26 06:23:49.283219995 (494h54m49.15430303s) [upload=434386]: Next returned nil segment Dec 26 06:23:49.283245634 (494h54m49.154328679s) [upload=434386]: Waiting for error group managing segments... Dec 26 06:23:49.644523232 (494h54m49.515606267s) [noise=true , address=145.79.227.3:4149 , piece_id=TOTASJEGQWCGG5KISNGGY3TLAL354QWMFA4VKEKDSCKHLY25PYQQ, seg_pos={0 0}, upload=434386]: Done uploading piece. err: <nil> Dec 26 06:23:49.648027525 (494h54m49.51911056s) [noise=true , address=145.79.227.49:4339 , piece_id=XQ6XNO7FGYMNEWYDGWGIB5IY5IHOAOPK7TUWPVPSEGQRIDA4R6GA, seg_pos={0 0}, upload=434386]: Done uploading piece. err: <nil> Dec 26 06:23:49.651435925 (494h54m49.52251897s) [noise=true , address=145.79.227.38:4977 , piece_id=PKXAJYGJPG6GYOG6DT4M6TBR54TTMBSWBXIIW2RLVMSA4HXBGOUQ, seg_pos={0 0}, upload=434386]: Done uploading piece. err: <nil> Dec 26 06:23:49.655712879 (494h54m49.526795924s) [noise=true , address=145.79.227.2:4115 , piece_id=D6S76VZDC2FBIFA3EA3TC4NEIFKJSGX6JRD4S26UO2IOLUDUPIDA, seg_pos={0 0}, upload=434386]: Done uploading piece. err: <nil> Dec 26 06:23:49.658634573 (494h54m49.529717618s) [noise=true , address=145.79.227.39:4013 , piece_id=3EHFPZVBIV6NNDJ46Z5YHVOPBWEMYQLG6NCJ4HT6ZX3IVMGVAFAQ, seg_pos={0 0}, upload=434386]: Done uploading piece. err: <nil> Dec 26 06:23:49.670194264 (494h54m49.541277309s) [noise=true , address=145.79.227.29:4942 , piece_id=GML4O7CTQ3AR5KAPGOZAMBELR7F6SDGSJSAWPZLO27W53ALPNZ2Q, seg_pos={0 0}, upload=434386]: Done uploading piece. err: <nil> Dec 26 06:23:49.672063052 (494h54m49.543146097s) [noise=true , address=145.79.227.5:4221 , piece_id=U2IAUDBXHO2UXGPSOYT3L2FPIJZFDTJ4GQCHGGXGSFD62KXIV77Q, seg_pos={0 0}, upload=434386]: Done uploading piece. err: <nil> Dec 26 06:23:49.672693352 (494h54m49.543776387s) [noise=true , address=145.79.227.20:4437 , piece_id=P4DNS6VYA2TS43TL7LYUSRUKFDJ37PO2M6D6NP534QQQBH2WIV2A, seg_pos={0 0}, upload=434386]: Done uploading piece. err: <nil> Dec 26 06:23:49.725648318 (494h54m49.596731363s) [noise=true , address=145.79.227.6:4259 , piece_id=TGQ6V3HNT3X2IY7MQ73RZRNH5NDXENTRDCXWHFHQETQEILSO7FEA, seg_pos={0 0}, upload=434386]: Done uploading piece. err: <nil> Dec 26 06:23:49.861945242 (494h54m49.733028287s) [noise=true , address=77.90.31.21:4880 , piece_id=JLHHQM55ZRSYUHKTKMZOMU4X3KXNQHIXH32KWIFA3CLWADW37W7Q, seg_pos={0 0}, upload=434386]: Done uploading piece. err: <nil> Dec 26 06:23:49.958824257 (494h54m49.829907292s) [noise=true , address=45.91.116.9:4264 , piece_id=N7AODEQP7YXGHYUJCT27BSAFMEJB3ABXUUCEUPSWDUR3F7DUB5TA, seg_pos={0 0}, upload=434386]: Done uploading piece. err: <nil> Dec 26 06:23:49.970075321 (494h54m49.841158366s) [noise=true , address=45.91.116.13:4153 , piece_id=7PR6IGAFJOAALB3YXBEWFAGCR72DFJ6JWT24WCO7FHUOCSMJNP7A, seg_pos={0 0}, upload=434386]: Done uploading piece. err: <nil> Dec 26 06:23:50.140738494 (494h54m50.011821529s) [noise=true , address=77.90.31.15:4700 , piece_id=23B7NZUGOMXA6OMBSKY3HWKZ36VVIVOCQGFBLFZ6EHGC7TD544YA, seg_pos={0 0}, upload=434386]: Done uploading piece. err: <nil> Dec 26 06:23:50.140774813 (494h54m50.011857849s) [seg_pos={0 0}, upload=434386]: Segment reached optimal threshold of 13 pieces. Dec 26 06:23:50.140993680 (494h54m50.012076725s) [noise=true , address=45.91.116.14:4191 , piece_id=LPPDT6VMSHKDJWZLB25GN2XLHS73352JSKCRZNEAFRRRWDLA7RNQ, seg_pos={0 0}, upload=434386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Dec 26 06:23:50.141002648 (494h54m50.012085683s) [noise=true , address=45.91.116.14:4191 , piece_id=LPPDT6VMSHKDJWZLB25GN2XLHS73352JSKCRZNEAFRRRWDLA7RNQ, seg_pos={0 0}, upload=434386]: Long tail cancellation due to optimal threshold Dec 26 06:23:50.141012116 (494h54m50.012095161s) [noise=true , address=45.91.116.7:4010 , piece_id=LHU4DWQU7XLA3QTJ2LHPSLGON43MUO3BXHJ5FRZPIRSA5IRH2LNA, seg_pos={0 0}, upload=434386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 26 06:23:50.141016754 (494h54m50.012099789s) [noise=true , address=45.91.116.7:4010 , piece_id=LHU4DWQU7XLA3QTJ2LHPSLGON43MUO3BXHJ5FRZPIRSA5IRH2LNA, seg_pos={0 0}, upload=434386]: Long tail cancellation due to optimal threshold Dec 26 06:23:50.141145951 (494h54m50.012228986s) [noise=true , address=45.91.116.11:4118 , piece_id=6BZNBNEDJB4RNNFMAXRMGLKK4GHYTDNKMVTMG5RF3U2VDKMON2KQ, seg_pos={0 0}, upload=434386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: expected piece hash; context canceled; EOF Dec 26 06:23:50.141156651 (494h54m50.012239686s) [noise=true , address=45.91.116.11:4118 , piece_id=6BZNBNEDJB4RNNFMAXRMGLKK4GHYTDNKMVTMG5RF3U2VDKMON2KQ, seg_pos={0 0}, upload=434386]: Long tail cancellation due to optimal threshold Dec 26 06:23:50.141172291 (494h54m50.012255336s) [seg_pos={0 0}, upload=434386]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 06:23:50.141260609 (494h54m50.012343644s) [upload=434386]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 06:23:50.141265488 (494h54m50.012348533s) [upload=434386]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 06:23:50.438800618 (494h54m50.309883653s) [upload=434386]: Done uploading segments... Dec 26 06:23:50.438865121 (494h54m50.309948166s) [upload=434386]: Upload finished. err: <nil> Duration: 5.967349481s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...................................X........................] duration: 3.665985061s trace: https://jaeger.dev.storj.io/trace/630094597d401477 FAIL (2025-12-26 06:27:06 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.666018245s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 25.852997716s trace: https://jaeger.dev.storj.io/trace/7e198d4c8fea54d2 FAIL (2025-12-26 06:26:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p, Piece ID: SBCEX7UNWGNFF54SBP4AGRDV5IPG24FQJXJJ7EYQTBSZKOVZWU4Q) context deadline exceeded; piecestore: (Node ID: 1nP8xospdMuDEh3em5K6iBnh8bHXqLhvHy4pHm9m8Lnj4kEH2X, Piece ID: XV7FQGBJNWVRLOAFRUPOWON35JR5ESRJPTCWPUGYWGHJPSDVL4GA) context deadline exceeded; piecestore: (Node ID: 1Mg5mxUjW56quDidSgA788D7ByfAzvsEuWJqwtzY4bLyxRn3mu, Piece ID: ATPIXLMXGONSSLXPPVORSNC6LHPXI2V3UFTHM4VHA2FTY5ZM6H3A) context deadline exceeded; piecestore: (Node ID: 12QmYSEAKnkSLEEVc9E1uxtVigunKMFLk2cdu71Bse8Tqec3HyT, Piece ID: YFKD7GO2FOSPLDXNFPK732QTLCLBQQASWBTMZ2LTLBGSK22PIXIQ) context deadline exceeded; piecestore: (Node ID: 1JsnHuhQhM4rWLTLEuSSHLNE24m7tSW9qMSf48j7yMrAVPWVyV, Piece ID: ARBMDAKX4QBW6JG47KP2PUITCX7HKXCBO75THQPINM4LLHCKAV4Q) context deadline exceeded ======LOGS====== upload=433902]: Uploading piece... Dec 26 05:50:34.986155265 (494h21m34.8572383s) [noise=true , address=64.49.13.140:4509 , piece_id=6TGWTOAVY4YVAKU6VEE2SWVAM264TKU6ZNMGPWH4TNGII3UYXFXQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.986703610 (494h21m34.857786655s) [noise=true , address=77.90.31.10:4522 , piece_id=YFKD7GO2FOSPLDXNFPK732QTLCLBQQASWBTMZ2LTLBGSK22PIXIQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.987145380 (494h21m34.858228425s) [noise=true , address=145.79.227.33:4890 , piece_id=UKNB4ZZTYJGY4MDHOBPYUTHCP3ZDCMJB7O7AJZFYP2HFVIWLUOKQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.987628290 (494h21m34.858711335s) [noise=true , address=145.79.227.40:4052 , piece_id=XRZYL7PT5VHY22N7CCKS3NTZKDDWE36ZU3SORF6ILM7ZMXDND63A, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.988710731 (494h21m34.859793776s) [noise=true , address=77.90.31.13:4653 , piece_id=EY2CWX54ONOWPO3WC33G5QSP5RMP2SUTBQLUOH5JXGDRUEPDHQ2A, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.989894545 (494h21m34.86097759s) [noise=true , address=145.79.227.30:4785 , piece_id=5KMLU4ATBN4YVQDYJKKQY4PEJNKWTO2WNX5GMGJM4F54B6YM2SFA, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.991305081 (494h21m34.862388116s) [noise=true , address=145.79.227.50:4381 , piece_id=H6Q47DIOTOWGN7QH2KWT4IH6KVDSB246S22753Z3UQGB26T6VXPA, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.995796263 (494h21m34.866879359s) [noise=true , address=145.79.227.52:4461 , piece_id=ZAAUDBGY6CC2JNEGWXVPCBAWXARHAJDZ3YPB6Y3NWEWCFDUFGDSQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.996765439 (494h21m34.867848484s) [noise=true , address=77.90.31.16:4755 , piece_id=ARBMDAKX4QBW6JG47KP2PUITCX7HKXCBO75THQPINM4LLHCKAV4Q, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.998107634 (494h21m34.869190679s) [noise=true , address=77.90.31.11:4579 , piece_id=SBCEX7UNWGNFF54SBP4AGRDV5IPG24FQJXJJ7EYQTBSZKOVZWU4Q, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.999340442 (494h21m34.870423477s) [noise=true , address=145.79.227.34:4936 , piece_id=EEA2ZPM44DX3WMI7ZJTDC7T5UIGED6TNT53QXZWGBFRMQUQSCXPQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:35.000540757 (494h21m34.871623812s) [noise=true , address=45.91.116.14:4204 , piece_id=XV7FQGBJNWVRLOAFRUPOWON35JR5ESRJPTCWPUGYWGHJPSDVL4GA, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:36.360077064 (494h21m36.231160099s) [upload=433902]: Next returned nil segment Dec 26 05:50:36.360091642 (494h21m36.231174688s) [upload=433902]: Waiting for error group managing segments... Dec 26 05:50:36.688914299 (494h21m36.559997344s) [noise=true , address=64.49.13.140:4509 , piece_id=6TGWTOAVY4YVAKU6VEE2SWVAM264TKU6ZNMGPWH4TNGII3UYXFXQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.059812634 (494h21m36.930895669s) [noise=true , address=77.90.31.11:4579 , piece_id=SBCEX7UNWGNFF54SBP4AGRDV5IPG24FQJXJJ7EYQTBSZKOVZWU4Q, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.119433874 (494h21m36.990516909s) [noise=true , address=45.91.116.14:4204 , piece_id=XV7FQGBJNWVRLOAFRUPOWON35JR5ESRJPTCWPUGYWGHJPSDVL4GA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.138718204 (494h21m37.00980125s) [noise=true , address=77.90.31.16:4755 , piece_id=ARBMDAKX4QBW6JG47KP2PUITCX7HKXCBO75THQPINM4LLHCKAV4Q, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.146647243 (494h21m37.017730278s) [noise=true , address=77.90.31.13:4653 , piece_id=EY2CWX54ONOWPO3WC33G5QSP5RMP2SUTBQLUOH5JXGDRUEPDHQ2A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.163283722 (494h21m37.034366757s) [noise=true , address=77.90.31.21:4893 , piece_id=GNEEZO2XGOKNODOUPZQ4NGKFE4PCX5LNUWXLV27F6X7GWZVSYNAA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.374956574 (494h21m37.246039609s) [noise=true , address=45.91.116.7:4033 , piece_id=ATPIXLMXGONSSLXPPVORSNC6LHPXI2V3UFTHM4VHA2FTY5ZM6H3A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.777283355 (494h21m37.64836639s) [noise=true , address=77.90.31.12:4622 , piece_id=LTYYPK6ZQ3NXZPBKSV4OMZGCOXXV3N6L5A6MVODNR644D2ZUJX3A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:38.439025909 (494h21m38.310108944s) [noise=true , address=145.79.227.49:4355 , piece_id=IUDOHR73Y6OKFAQSNNSONALWAITRJ4POYRO2QD5LYZKOSTWY4MNA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:38.439599111 (494h21m38.310682166s) [noise=true , address=77.90.31.10:4522 , piece_id=YFKD7GO2FOSPLDXNFPK732QTLCLBQQASWBTMZ2LTLBGSK22PIXIQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:39.259154980 (494h21m39.130238015s) [noise=true , address=145.79.227.34:4936 , piece_id=EEA2ZPM44DX3WMI7ZJTDC7T5UIGED6TNT53QXZWGBFRMQUQSCXPQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:39.430935921 (494h21m39.302018956s) [noise=true , address=145.79.227.44:4212 , piece_id=ABBNLKRE4FM7RHF56V4IAQPKBCCCQLLOMFVMSJEWQUNFGGCI3PVQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:42.941071297 (494h21m42.812154352s) [noise=true , address=77.90.31.14:4675 , piece_id=FKB4O45OYICMELPGXB65QAUD2DRIFTSXZFKIA6TY26VFAP6AVY3A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:51:22.719026850 (494h22m22.590109885s) [noise=true , address=145.79.227.33:4890 , piece_id=UKNB4ZZTYJGY4MDHOBPYUTHCP3ZDCMJB7O7AJZFYP2HFVIWLUOKQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:51:22.719060193 (494h22m22.590143228s) [seg_pos={0 0}, upload=433902]: Segment reached optimal threshold of 14 pieces. Dec 26 05:51:22.719233022 (494h22m22.590316067s) [noise=true , address=145.79.227.50:4381 , piece_id=H6Q47DIOTOWGN7QH2KWT4IH6KVDSB246S22753Z3UQGB26T6VXPA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zU35MWWaaC1npkP5qGcWBXAkfxGsd5UsdkM2nwX9x7wUGCDbY): protocol: write tcp 188.166.225.170:46970->145.79.227.50:4381: use of closed network connection; write tcp 188.166.225.170:46970->145.79.227.50:4381: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46970->145.79.227.50:4381: use of closed network connection Dec 26 05:51:22.719238783 (494h22m22.590321828s) [noise=true , address=145.79.227.50:4381 , piece_id=H6Q47DIOTOWGN7QH2KWT4IH6KVDSB246S22753Z3UQGB26T6VXPA, seg_pos={0 0}, upload=433902]: Long tail cancellation due to optimal threshold Dec 26 05:51:22.719251677 (494h22m22.590334712s) [noise=true , address=145.79.227.30:4785 , piece_id=5KMLU4ATBN4YVQDYJKKQY4PEJNKWTO2WNX5GMGJM4F54B6YM2SFA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gYH83ECV8wai3CZyBSU8NNJM1UsysLYHbQDwBVKZRyjsdnUwB): protocol: write tcp 188.166.225.170:58398->145.79.227.30:4785: use of closed network connection; write tcp 188.166.225.170:58398->145.79.227.30:4785: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58398->145.79.227.30:4785: use of closed network connection Dec 26 05:51:22.719260194 (494h22m22.590343229s) [noise=true , address=145.79.227.30:4785 , piece_id=5KMLU4ATBN4YVQDYJKKQY4PEJNKWTO2WNX5GMGJM4F54B6YM2SFA, seg_pos={0 0}, upload=433902]: Long tail cancellation due to optimal threshold Dec 26 05:51:22.719409408 (494h22m22.590492443s) [noise=true , address=145.79.227.40:4052 , piece_id=XRZYL7PT5VHY22N7CCKS3NTZKDDWE36ZU3SORF6ILM7ZMXDND63A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PViJBYAanwXRWhMq3WHR6k9Stxy8hjuY5FV9J7ErxCy4N1q3V): protocol: write tcp 188.166.225.170:45984->145.79.227.40:4052: use of closed network connection; write tcp 188.166.225.170:45984->145.79.227.40:4052: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45984->145.79.227.40:4052: use of closed network connection Dec 26 05:51:22.719415469 (494h22m22.590498515s) [noise=true , address=145.79.227.40:4052 , piece_id=XRZYL7PT5VHY22N7CCKS3NTZKDDWE36ZU3SORF6ILM7ZMXDND63A, seg_pos={0 0}, upload=433902]: Long tail cancellation due to optimal threshold Dec 26 05:51:22.719467057 (494h22m22.590550092s) [noise=true , address=145.79.227.52:4461 , piece_id=ZAAUDBGY6CC2JNEGWXVPCBAWXARHAJDZ3YPB6Y3NWEWCFDUFGDSQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fNrbSfErEQRnruJCJAVvkoNwQSPWnvNtxunHn6zrQegnNAkxA): protocol: write tcp 188.166.225.170:55966->145.79.227.52:4461: use of closed network connection; write tcp 188.166.225.170:55966->145.79.227.52:4461: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55966->145.79.227.52:4461: use of closed network connection Dec 26 05:51:22.719473931 (494h22m22.590556966s) [noise=true , address=145.79.227.52:4461 , piece_id=ZAAUDBGY6CC2JNEGWXVPCBAWXARHAJDZ3YPB6Y3NWEWCFDUFGDSQ, seg_pos={0 0}, upload=433902]: Long tail cancellation due to optimal threshold Dec 26 05:51:22.719491374 (494h22m22.590574409s) [seg_pos={0 0}, upload=433902]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 05:51:22.719562539 (494h22m22.590645574s) [upload=433902]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:51:22.719567228 (494h22m22.590650263s) [upload=433902]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:51:23.008169117 (494h22m22.879252162s) [upload=433902]: Done uploading segments... Dec 26 05:51:23.008191560 (494h22m22.879274605s) [upload=433902]: Upload finished. err: <nil> Duration: 25.853018176s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.....X..X.X.......................X..........X.......X.X....] duration: 10.043889744s trace: https://jaeger.dev.storj.io/trace/67a56359499790c8 FAIL (2025-12-26 06:26:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 06:23:47.396606446 (494h54m47.267689491s) [upload=434386]: Starting upload Dec 26 06:23:47.396675247 (494h54m47.267758292s) [upload=434386]: Done starting upload Dec 26 06:23:47.396898202 (494h54m47.267981248s) [upload=434386]: Uploading segments... Dec 26 06:23:47.396953998 (494h54m47.268037073s) [upload=434386]: Got next segment. Inline: false Dec 26 06:23:47.397031125 (494h54m47.26811416s) [upload=434386]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 06:23:47.397037317 (494h54m47.268120362s) [upload=434386]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 06:23:47.643130652 (494h54m47.514213687s) [seg_pos={0 0}, upload=434386]: Begin upload segment... Dec 26 06:23:47.643171870 (494h54m47.514254905s) [seg_pos={0 0}, upload=434386]: Done begin upload segment. Dec 26 06:23:47.643193913 (494h54m47.514276948s) [noise=true , address=145.79.227.38:4977 , piece_id=PKXAJYGJPG6GYOG6DT4M6TBR54TTMBSWBXIIW2RLVMSA4HXBGOUQ, seg_pos={0 0}, upload=434386]: Uploading piece... Dec 26 06:23:47.643278103 (494h54m47.514361138s) [noise=true , address=145.79.227.49:4339 , piece_id=XQ6XNO7FGYMNEWYDGWGIB5IY5IHOAOPK7TUWPVPSEGQRIDA4R6GA, seg_pos={0 0}, upload=434386]: Uploading piece... Dec 26 06:23:47.644040174 (494h54m47.515123219s) [noise=true , address=145.79.227.29:4942 , piece_id=GML4O7CTQ3AR5KAPGOZAMBELR7F6SDGSJSAWPZLO27W53ALPNZ2Q, seg_pos={0 0}, upload=434386]: Uploading piece... Dec 26 06:23:47.644250264 (494h54m47.515333309s) [noise=true , address=145.79.227.3:4149 , piece_id=TOTASJEGQWCGG5KISNGGY3TLAL354QWMFA4VKEKDSCKHLY25PYQQ, seg_pos={0 0}, upload=434386]: Uploading piece... Dec 26 06:23:47.644310018 (494h54m47.515393063s) [noise=true , address=145.79.227.5:4221 , piece_id=U2IAUDBXHO2UXGPSOYT3L2FPIJZFDTJ4GQCHGGXGSFD62KXIV77Q, seg_pos={0 0}, upload=434386]: Uploading piece... Dec 26 06:23:47.644590933 (494h54m47.515673968s) [noise=true , address=77.90.31.21:4880 , piece_id=JLHHQM55ZRSYUHKTKMZOMU4X3KXNQHIXH32KWIFA3CLWADW37W7Q, seg_pos={0 0}, upload=434386]: Uploading piece... Dec 26 06:23:47.644789010 (494h54m47.515872055s) [noise=true , address=45.91.116.13:4153 , piece_id=7PR6IGAFJOAALB3YXBEWFAGCR72DFJ6JWT24WCO7FHUOCSMJNP7A, seg_pos={0 0}, upload=434386]: Uploading piece... Dec 26 06:23:47.644945428 (494h54m47.516028463s) [noise=true , address=45.91.116.14:4191 , piece_id=LPPDT6VMSHKDJWZLB25GN2XLHS73352JSKCRZNEAFRRRWDLA7RNQ, seg_pos={0 0}, upload=434386]: Uploading piece... Dec 26 06:23:47.645273222 (494h54m47.516356257s) [noise=true , address=45.91.116.7:4010 , piece_id=LHU4DWQU7XLA3QTJ2LHPSLGON43MUO3BXHJ5FRZPIRSA5IRH2LNA, seg_pos={0 0}, upload=434386]: Uploading piece... Dec 26 06:23:47.645283942 (494h54m47.516366978s) [noise=true , address=45.91.116.9:4264 , piece_id=N7AODEQP7YXGHYUJCT27BSAFMEJB3ABXUUCEUPSWDUR3F7DUB5TA, seg_pos={0 0}, upload=434386]: Uploading piece... Dec 26 06:23:47.645654749 (494h54m47.516737784s) [noise=true , address=145.79.227.2:4115 , piece_id=D6S76VZDC2FBIFA3EA3TC4NEIFKJSGX6JRD4S26UO2IOLUDUPIDA, seg_pos={0 0}, upload=434386]: Uploading piece... Dec 26 06:23:47.645885748 (494h54m47.516968793s) [noise=true , address=145.79.227.20:4437 , piece_id=P4DNS6VYA2TS43TL7LYUSRUKFDJ37PO2M6D6NP534QQQBH2WIV2A, seg_pos={0 0}, upload=434386]: Uploading piece... Dec 26 06:23:47.646632821 (494h54m47.517715866s) [noise=true , address=77.90.31.15:4700 , piece_id=23B7NZUGOMXA6OMBSKY3HWKZ36VVIVOCQGFBLFZ6EHGC7TD544YA, seg_pos={0 0}, upload=434386]: Uploading piece... Dec 26 06:23:47.647299961 (494h54m47.518382996s) [noise=true , address=145.79.227.39:4013 , piece_id=3EHFPZVBIV6NNDJ46Z5YHVOPBWEMYQLG6NCJ4HT6ZX3IVMGVAFAQ, seg_pos={0 0}, upload=434386]: Uploading piece... Dec 26 06:23:47.647743005 (494h54m47.51882605s) [noise=true , address=45.91.116.11:4118 , piece_id=6BZNBNEDJB4RNNFMAXRMGLKK4GHYTDNKMVTMG5RF3U2VDKMON2KQ, seg_pos={0 0}, upload=434386]: Uploading piece... Dec 26 06:23:47.647943015 (494h54m47.51902606s) [noise=true , address=145.79.227.6:4259 , piece_id=TGQ6V3HNT3X2IY7MQ73RZRNH5NDXENTRDCXWHFHQETQEILSO7FEA, seg_pos={0 0}, upload=434386]: Uploading piece... Dec 26 06:23:49.283219995 (494h54m49.15430303s) [upload=434386]: Next returned nil segment Dec 26 06:23:49.283245634 (494h54m49.154328679s) [upload=434386]: Waiting for error group managing segments... Dec 26 06:23:49.644523232 (494h54m49.515606267s) [noise=true , address=145.79.227.3:4149 , piece_id=TOTASJEGQWCGG5KISNGGY3TLAL354QWMFA4VKEKDSCKHLY25PYQQ, seg_pos={0 0}, upload=434386]: Done uploading piece. err: <nil> Dec 26 06:23:49.648027525 (494h54m49.51911056s) [noise=true , address=145.79.227.49:4339 , piece_id=XQ6XNO7FGYMNEWYDGWGIB5IY5IHOAOPK7TUWPVPSEGQRIDA4R6GA, seg_pos={0 0}, upload=434386]: Done uploading piece. err: <nil> Dec 26 06:23:49.651435925 (494h54m49.52251897s) [noise=true , address=145.79.227.38:4977 , piece_id=PKXAJYGJPG6GYOG6DT4M6TBR54TTMBSWBXIIW2RLVMSA4HXBGOUQ, seg_pos={0 0}, upload=434386]: Done uploading piece. err: <nil> Dec 26 06:23:49.655712879 (494h54m49.526795924s) [noise=true , address=145.79.227.2:4115 , piece_id=D6S76VZDC2FBIFA3EA3TC4NEIFKJSGX6JRD4S26UO2IOLUDUPIDA, seg_pos={0 0}, upload=434386]: Done uploading piece. err: <nil> Dec 26 06:23:49.658634573 (494h54m49.529717618s) [noise=true , address=145.79.227.39:4013 , piece_id=3EHFPZVBIV6NNDJ46Z5YHVOPBWEMYQLG6NCJ4HT6ZX3IVMGVAFAQ, seg_pos={0 0}, upload=434386]: Done uploading piece. err: <nil> Dec 26 06:23:49.670194264 (494h54m49.541277309s) [noise=true , address=145.79.227.29:4942 , piece_id=GML4O7CTQ3AR5KAPGOZAMBELR7F6SDGSJSAWPZLO27W53ALPNZ2Q, seg_pos={0 0}, upload=434386]: Done uploading piece. err: <nil> Dec 26 06:23:49.672063052 (494h54m49.543146097s) [noise=true , address=145.79.227.5:4221 , piece_id=U2IAUDBXHO2UXGPSOYT3L2FPIJZFDTJ4GQCHGGXGSFD62KXIV77Q, seg_pos={0 0}, upload=434386]: Done uploading piece. err: <nil> Dec 26 06:23:49.672693352 (494h54m49.543776387s) [noise=true , address=145.79.227.20:4437 , piece_id=P4DNS6VYA2TS43TL7LYUSRUKFDJ37PO2M6D6NP534QQQBH2WIV2A, seg_pos={0 0}, upload=434386]: Done uploading piece. err: <nil> Dec 26 06:23:49.725648318 (494h54m49.596731363s) [noise=true , address=145.79.227.6:4259 , piece_id=TGQ6V3HNT3X2IY7MQ73RZRNH5NDXENTRDCXWHFHQETQEILSO7FEA, seg_pos={0 0}, upload=434386]: Done uploading piece. err: <nil> Dec 26 06:23:49.861945242 (494h54m49.733028287s) [noise=true , address=77.90.31.21:4880 , piece_id=JLHHQM55ZRSYUHKTKMZOMU4X3KXNQHIXH32KWIFA3CLWADW37W7Q, seg_pos={0 0}, upload=434386]: Done uploading piece. err: <nil> Dec 26 06:23:49.958824257 (494h54m49.829907292s) [noise=true , address=45.91.116.9:4264 , piece_id=N7AODEQP7YXGHYUJCT27BSAFMEJB3ABXUUCEUPSWDUR3F7DUB5TA, seg_pos={0 0}, upload=434386]: Done uploading piece. err: <nil> Dec 26 06:23:49.970075321 (494h54m49.841158366s) [noise=true , address=45.91.116.13:4153 , piece_id=7PR6IGAFJOAALB3YXBEWFAGCR72DFJ6JWT24WCO7FHUOCSMJNP7A, seg_pos={0 0}, upload=434386]: Done uploading piece. err: <nil> Dec 26 06:23:50.140738494 (494h54m50.011821529s) [noise=true , address=77.90.31.15:4700 , piece_id=23B7NZUGOMXA6OMBSKY3HWKZ36VVIVOCQGFBLFZ6EHGC7TD544YA, seg_pos={0 0}, upload=434386]: Done uploading piece. err: <nil> Dec 26 06:23:50.140774813 (494h54m50.011857849s) [seg_pos={0 0}, upload=434386]: Segment reached optimal threshold of 13 pieces. Dec 26 06:23:50.140993680 (494h54m50.012076725s) [noise=true , address=45.91.116.14:4191 , piece_id=LPPDT6VMSHKDJWZLB25GN2XLHS73352JSKCRZNEAFRRRWDLA7RNQ, seg_pos={0 0}, upload=434386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Dec 26 06:23:50.141002648 (494h54m50.012085683s) [noise=true , address=45.91.116.14:4191 , piece_id=LPPDT6VMSHKDJWZLB25GN2XLHS73352JSKCRZNEAFRRRWDLA7RNQ, seg_pos={0 0}, upload=434386]: Long tail cancellation due to optimal threshold Dec 26 06:23:50.141012116 (494h54m50.012095161s) [noise=true , address=45.91.116.7:4010 , piece_id=LHU4DWQU7XLA3QTJ2LHPSLGON43MUO3BXHJ5FRZPIRSA5IRH2LNA, seg_pos={0 0}, upload=434386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 26 06:23:50.141016754 (494h54m50.012099789s) [noise=true , address=45.91.116.7:4010 , piece_id=LHU4DWQU7XLA3QTJ2LHPSLGON43MUO3BXHJ5FRZPIRSA5IRH2LNA, seg_pos={0 0}, upload=434386]: Long tail cancellation due to optimal threshold Dec 26 06:23:50.141145951 (494h54m50.012228986s) [noise=true , address=45.91.116.11:4118 , piece_id=6BZNBNEDJB4RNNFMAXRMGLKK4GHYTDNKMVTMG5RF3U2VDKMON2KQ, seg_pos={0 0}, upload=434386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: expected piece hash; context canceled; EOF Dec 26 06:23:50.141156651 (494h54m50.012239686s) [noise=true , address=45.91.116.11:4118 , piece_id=6BZNBNEDJB4RNNFMAXRMGLKK4GHYTDNKMVTMG5RF3U2VDKMON2KQ, seg_pos={0 0}, upload=434386]: Long tail cancellation due to optimal threshold Dec 26 06:23:50.141172291 (494h54m50.012255336s) [seg_pos={0 0}, upload=434386]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 06:23:50.141260609 (494h54m50.012343644s) [upload=434386]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 06:23:50.141265488 (494h54m50.012348533s) [upload=434386]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 06:23:50.438800618 (494h54m50.309883653s) [upload=434386]: Done uploading segments... Dec 26 06:23:50.438865121 (494h54m50.309948166s) [upload=434386]: Upload finished. err: <nil> Duration: 10.043944439s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 27.682653989s trace: https://jaeger.dev.storj.io/trace/7e198d4c8fea54d2 FAIL (2025-12-26 06:26:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p, Piece ID: SBCEX7UNWGNFF54SBP4AGRDV5IPG24FQJXJJ7EYQTBSZKOVZWU4Q) context deadline exceeded; piecestore: (Node ID: 1nP8xospdMuDEh3em5K6iBnh8bHXqLhvHy4pHm9m8Lnj4kEH2X, Piece ID: XV7FQGBJNWVRLOAFRUPOWON35JR5ESRJPTCWPUGYWGHJPSDVL4GA) context deadline exceeded; piecestore: (Node ID: 1Mg5mxUjW56quDidSgA788D7ByfAzvsEuWJqwtzY4bLyxRn3mu, Piece ID: ATPIXLMXGONSSLXPPVORSNC6LHPXI2V3UFTHM4VHA2FTY5ZM6H3A) context deadline exceeded; piecestore: (Node ID: 12QmYSEAKnkSLEEVc9E1uxtVigunKMFLk2cdu71Bse8Tqec3HyT, Piece ID: YFKD7GO2FOSPLDXNFPK732QTLCLBQQASWBTMZ2LTLBGSK22PIXIQ) context deadline exceeded; piecestore: (Node ID: 1JsnHuhQhM4rWLTLEuSSHLNE24m7tSW9qMSf48j7yMrAVPWVyV, Piece ID: ARBMDAKX4QBW6JG47KP2PUITCX7HKXCBO75THQPINM4LLHCKAV4Q) context deadline exceeded ======LOGS====== upload=433902]: Uploading piece... Dec 26 05:50:34.986155265 (494h21m34.8572383s) [noise=true , address=64.49.13.140:4509 , piece_id=6TGWTOAVY4YVAKU6VEE2SWVAM264TKU6ZNMGPWH4TNGII3UYXFXQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.986703610 (494h21m34.857786655s) [noise=true , address=77.90.31.10:4522 , piece_id=YFKD7GO2FOSPLDXNFPK732QTLCLBQQASWBTMZ2LTLBGSK22PIXIQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.987145380 (494h21m34.858228425s) [noise=true , address=145.79.227.33:4890 , piece_id=UKNB4ZZTYJGY4MDHOBPYUTHCP3ZDCMJB7O7AJZFYP2HFVIWLUOKQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.987628290 (494h21m34.858711335s) [noise=true , address=145.79.227.40:4052 , piece_id=XRZYL7PT5VHY22N7CCKS3NTZKDDWE36ZU3SORF6ILM7ZMXDND63A, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.988710731 (494h21m34.859793776s) [noise=true , address=77.90.31.13:4653 , piece_id=EY2CWX54ONOWPO3WC33G5QSP5RMP2SUTBQLUOH5JXGDRUEPDHQ2A, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.989894545 (494h21m34.86097759s) [noise=true , address=145.79.227.30:4785 , piece_id=5KMLU4ATBN4YVQDYJKKQY4PEJNKWTO2WNX5GMGJM4F54B6YM2SFA, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.991305081 (494h21m34.862388116s) [noise=true , address=145.79.227.50:4381 , piece_id=H6Q47DIOTOWGN7QH2KWT4IH6KVDSB246S22753Z3UQGB26T6VXPA, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.995796263 (494h21m34.866879359s) [noise=true , address=145.79.227.52:4461 , piece_id=ZAAUDBGY6CC2JNEGWXVPCBAWXARHAJDZ3YPB6Y3NWEWCFDUFGDSQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.996765439 (494h21m34.867848484s) [noise=true , address=77.90.31.16:4755 , piece_id=ARBMDAKX4QBW6JG47KP2PUITCX7HKXCBO75THQPINM4LLHCKAV4Q, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.998107634 (494h21m34.869190679s) [noise=true , address=77.90.31.11:4579 , piece_id=SBCEX7UNWGNFF54SBP4AGRDV5IPG24FQJXJJ7EYQTBSZKOVZWU4Q, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.999340442 (494h21m34.870423477s) [noise=true , address=145.79.227.34:4936 , piece_id=EEA2ZPM44DX3WMI7ZJTDC7T5UIGED6TNT53QXZWGBFRMQUQSCXPQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:35.000540757 (494h21m34.871623812s) [noise=true , address=45.91.116.14:4204 , piece_id=XV7FQGBJNWVRLOAFRUPOWON35JR5ESRJPTCWPUGYWGHJPSDVL4GA, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:36.360077064 (494h21m36.231160099s) [upload=433902]: Next returned nil segment Dec 26 05:50:36.360091642 (494h21m36.231174688s) [upload=433902]: Waiting for error group managing segments... Dec 26 05:50:36.688914299 (494h21m36.559997344s) [noise=true , address=64.49.13.140:4509 , piece_id=6TGWTOAVY4YVAKU6VEE2SWVAM264TKU6ZNMGPWH4TNGII3UYXFXQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.059812634 (494h21m36.930895669s) [noise=true , address=77.90.31.11:4579 , piece_id=SBCEX7UNWGNFF54SBP4AGRDV5IPG24FQJXJJ7EYQTBSZKOVZWU4Q, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.119433874 (494h21m36.990516909s) [noise=true , address=45.91.116.14:4204 , piece_id=XV7FQGBJNWVRLOAFRUPOWON35JR5ESRJPTCWPUGYWGHJPSDVL4GA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.138718204 (494h21m37.00980125s) [noise=true , address=77.90.31.16:4755 , piece_id=ARBMDAKX4QBW6JG47KP2PUITCX7HKXCBO75THQPINM4LLHCKAV4Q, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.146647243 (494h21m37.017730278s) [noise=true , address=77.90.31.13:4653 , piece_id=EY2CWX54ONOWPO3WC33G5QSP5RMP2SUTBQLUOH5JXGDRUEPDHQ2A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.163283722 (494h21m37.034366757s) [noise=true , address=77.90.31.21:4893 , piece_id=GNEEZO2XGOKNODOUPZQ4NGKFE4PCX5LNUWXLV27F6X7GWZVSYNAA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.374956574 (494h21m37.246039609s) [noise=true , address=45.91.116.7:4033 , piece_id=ATPIXLMXGONSSLXPPVORSNC6LHPXI2V3UFTHM4VHA2FTY5ZM6H3A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.777283355 (494h21m37.64836639s) [noise=true , address=77.90.31.12:4622 , piece_id=LTYYPK6ZQ3NXZPBKSV4OMZGCOXXV3N6L5A6MVODNR644D2ZUJX3A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:38.439025909 (494h21m38.310108944s) [noise=true , address=145.79.227.49:4355 , piece_id=IUDOHR73Y6OKFAQSNNSONALWAITRJ4POYRO2QD5LYZKOSTWY4MNA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:38.439599111 (494h21m38.310682166s) [noise=true , address=77.90.31.10:4522 , piece_id=YFKD7GO2FOSPLDXNFPK732QTLCLBQQASWBTMZ2LTLBGSK22PIXIQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:39.259154980 (494h21m39.130238015s) [noise=true , address=145.79.227.34:4936 , piece_id=EEA2ZPM44DX3WMI7ZJTDC7T5UIGED6TNT53QXZWGBFRMQUQSCXPQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:39.430935921 (494h21m39.302018956s) [noise=true , address=145.79.227.44:4212 , piece_id=ABBNLKRE4FM7RHF56V4IAQPKBCCCQLLOMFVMSJEWQUNFGGCI3PVQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:42.941071297 (494h21m42.812154352s) [noise=true , address=77.90.31.14:4675 , piece_id=FKB4O45OYICMELPGXB65QAUD2DRIFTSXZFKIA6TY26VFAP6AVY3A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:51:22.719026850 (494h22m22.590109885s) [noise=true , address=145.79.227.33:4890 , piece_id=UKNB4ZZTYJGY4MDHOBPYUTHCP3ZDCMJB7O7AJZFYP2HFVIWLUOKQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:51:22.719060193 (494h22m22.590143228s) [seg_pos={0 0}, upload=433902]: Segment reached optimal threshold of 14 pieces. Dec 26 05:51:22.719233022 (494h22m22.590316067s) [noise=true , address=145.79.227.50:4381 , piece_id=H6Q47DIOTOWGN7QH2KWT4IH6KVDSB246S22753Z3UQGB26T6VXPA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zU35MWWaaC1npkP5qGcWBXAkfxGsd5UsdkM2nwX9x7wUGCDbY): protocol: write tcp 188.166.225.170:46970->145.79.227.50:4381: use of closed network connection; write tcp 188.166.225.170:46970->145.79.227.50:4381: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46970->145.79.227.50:4381: use of closed network connection Dec 26 05:51:22.719238783 (494h22m22.590321828s) [noise=true , address=145.79.227.50:4381 , piece_id=H6Q47DIOTOWGN7QH2KWT4IH6KVDSB246S22753Z3UQGB26T6VXPA, seg_pos={0 0}, upload=433902]: Long tail cancellation due to optimal threshold Dec 26 05:51:22.719251677 (494h22m22.590334712s) [noise=true , address=145.79.227.30:4785 , piece_id=5KMLU4ATBN4YVQDYJKKQY4PEJNKWTO2WNX5GMGJM4F54B6YM2SFA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gYH83ECV8wai3CZyBSU8NNJM1UsysLYHbQDwBVKZRyjsdnUwB): protocol: write tcp 188.166.225.170:58398->145.79.227.30:4785: use of closed network connection; write tcp 188.166.225.170:58398->145.79.227.30:4785: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58398->145.79.227.30:4785: use of closed network connection Dec 26 05:51:22.719260194 (494h22m22.590343229s) [noise=true , address=145.79.227.30:4785 , piece_id=5KMLU4ATBN4YVQDYJKKQY4PEJNKWTO2WNX5GMGJM4F54B6YM2SFA, seg_pos={0 0}, upload=433902]: Long tail cancellation due to optimal threshold Dec 26 05:51:22.719409408 (494h22m22.590492443s) [noise=true , address=145.79.227.40:4052 , piece_id=XRZYL7PT5VHY22N7CCKS3NTZKDDWE36ZU3SORF6ILM7ZMXDND63A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PViJBYAanwXRWhMq3WHR6k9Stxy8hjuY5FV9J7ErxCy4N1q3V): protocol: write tcp 188.166.225.170:45984->145.79.227.40:4052: use of closed network connection; write tcp 188.166.225.170:45984->145.79.227.40:4052: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45984->145.79.227.40:4052: use of closed network connection Dec 26 05:51:22.719415469 (494h22m22.590498515s) [noise=true , address=145.79.227.40:4052 , piece_id=XRZYL7PT5VHY22N7CCKS3NTZKDDWE36ZU3SORF6ILM7ZMXDND63A, seg_pos={0 0}, upload=433902]: Long tail cancellation due to optimal threshold Dec 26 05:51:22.719467057 (494h22m22.590550092s) [noise=true , address=145.79.227.52:4461 , piece_id=ZAAUDBGY6CC2JNEGWXVPCBAWXARHAJDZ3YPB6Y3NWEWCFDUFGDSQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fNrbSfErEQRnruJCJAVvkoNwQSPWnvNtxunHn6zrQegnNAkxA): protocol: write tcp 188.166.225.170:55966->145.79.227.52:4461: use of closed network connection; write tcp 188.166.225.170:55966->145.79.227.52:4461: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55966->145.79.227.52:4461: use of closed network connection Dec 26 05:51:22.719473931 (494h22m22.590556966s) [noise=true , address=145.79.227.52:4461 , piece_id=ZAAUDBGY6CC2JNEGWXVPCBAWXARHAJDZ3YPB6Y3NWEWCFDUFGDSQ, seg_pos={0 0}, upload=433902]: Long tail cancellation due to optimal threshold Dec 26 05:51:22.719491374 (494h22m22.590574409s) [seg_pos={0 0}, upload=433902]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 05:51:22.719562539 (494h22m22.590645574s) [upload=433902]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:51:22.719567228 (494h22m22.590650263s) [upload=433902]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:51:23.008169117 (494h22m22.879252162s) [upload=433902]: Done uploading segments... Dec 26 05:51:23.008191560 (494h22m22.879274605s) [upload=433902]: Upload finished. err: <nil> Duration: 27.682689024s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....................................X.......................] duration: 3.552247054s trace: https://jaeger.dev.storj.io/trace/630094597d401477 FAIL (2025-12-26 06:26:29 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.552290107s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [......X..X.X.......................X..........X.......X.X...] duration: 5.762702419s trace: https://jaeger.dev.storj.io/trace/67a56359499790c8 FAIL (2025-12-26 06:26:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 06:23:47.396606446 (494h54m47.267689491s) [upload=434386]: Starting upload Dec 26 06:23:47.396675247 (494h54m47.267758292s) [upload=434386]: Done starting upload Dec 26 06:23:47.396898202 (494h54m47.267981248s) [upload=434386]: Uploading segments... Dec 26 06:23:47.396953998 (494h54m47.268037073s) [upload=434386]: Got next segment. Inline: false Dec 26 06:23:47.397031125 (494h54m47.26811416s) [upload=434386]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 06:23:47.397037317 (494h54m47.268120362s) [upload=434386]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 06:23:47.643130652 (494h54m47.514213687s) [seg_pos={0 0}, upload=434386]: Begin upload segment... Dec 26 06:23:47.643171870 (494h54m47.514254905s) [seg_pos={0 0}, upload=434386]: Done begin upload segment. Dec 26 06:23:47.643193913 (494h54m47.514276948s) [noise=true , address=145.79.227.38:4977 , piece_id=PKXAJYGJPG6GYOG6DT4M6TBR54TTMBSWBXIIW2RLVMSA4HXBGOUQ, seg_pos={0 0}, upload=434386]: Uploading piece... Dec 26 06:23:47.643278103 (494h54m47.514361138s) [noise=true , address=145.79.227.49:4339 , piece_id=XQ6XNO7FGYMNEWYDGWGIB5IY5IHOAOPK7TUWPVPSEGQRIDA4R6GA, seg_pos={0 0}, upload=434386]: Uploading piece... Dec 26 06:23:47.644040174 (494h54m47.515123219s) [noise=true , address=145.79.227.29:4942 , piece_id=GML4O7CTQ3AR5KAPGOZAMBELR7F6SDGSJSAWPZLO27W53ALPNZ2Q, seg_pos={0 0}, upload=434386]: Uploading piece... Dec 26 06:23:47.644250264 (494h54m47.515333309s) [noise=true , address=145.79.227.3:4149 , piece_id=TOTASJEGQWCGG5KISNGGY3TLAL354QWMFA4VKEKDSCKHLY25PYQQ, seg_pos={0 0}, upload=434386]: Uploading piece... Dec 26 06:23:47.644310018 (494h54m47.515393063s) [noise=true , address=145.79.227.5:4221 , piece_id=U2IAUDBXHO2UXGPSOYT3L2FPIJZFDTJ4GQCHGGXGSFD62KXIV77Q, seg_pos={0 0}, upload=434386]: Uploading piece... Dec 26 06:23:47.644590933 (494h54m47.515673968s) [noise=true , address=77.90.31.21:4880 , piece_id=JLHHQM55ZRSYUHKTKMZOMU4X3KXNQHIXH32KWIFA3CLWADW37W7Q, seg_pos={0 0}, upload=434386]: Uploading piece... Dec 26 06:23:47.644789010 (494h54m47.515872055s) [noise=true , address=45.91.116.13:4153 , piece_id=7PR6IGAFJOAALB3YXBEWFAGCR72DFJ6JWT24WCO7FHUOCSMJNP7A, seg_pos={0 0}, upload=434386]: Uploading piece... Dec 26 06:23:47.644945428 (494h54m47.516028463s) [noise=true , address=45.91.116.14:4191 , piece_id=LPPDT6VMSHKDJWZLB25GN2XLHS73352JSKCRZNEAFRRRWDLA7RNQ, seg_pos={0 0}, upload=434386]: Uploading piece... Dec 26 06:23:47.645273222 (494h54m47.516356257s) [noise=true , address=45.91.116.7:4010 , piece_id=LHU4DWQU7XLA3QTJ2LHPSLGON43MUO3BXHJ5FRZPIRSA5IRH2LNA, seg_pos={0 0}, upload=434386]: Uploading piece... Dec 26 06:23:47.645283942 (494h54m47.516366978s) [noise=true , address=45.91.116.9:4264 , piece_id=N7AODEQP7YXGHYUJCT27BSAFMEJB3ABXUUCEUPSWDUR3F7DUB5TA, seg_pos={0 0}, upload=434386]: Uploading piece... Dec 26 06:23:47.645654749 (494h54m47.516737784s) [noise=true , address=145.79.227.2:4115 , piece_id=D6S76VZDC2FBIFA3EA3TC4NEIFKJSGX6JRD4S26UO2IOLUDUPIDA, seg_pos={0 0}, upload=434386]: Uploading piece... Dec 26 06:23:47.645885748 (494h54m47.516968793s) [noise=true , address=145.79.227.20:4437 , piece_id=P4DNS6VYA2TS43TL7LYUSRUKFDJ37PO2M6D6NP534QQQBH2WIV2A, seg_pos={0 0}, upload=434386]: Uploading piece... Dec 26 06:23:47.646632821 (494h54m47.517715866s) [noise=true , address=77.90.31.15:4700 , piece_id=23B7NZUGOMXA6OMBSKY3HWKZ36VVIVOCQGFBLFZ6EHGC7TD544YA, seg_pos={0 0}, upload=434386]: Uploading piece... Dec 26 06:23:47.647299961 (494h54m47.518382996s) [noise=true , address=145.79.227.39:4013 , piece_id=3EHFPZVBIV6NNDJ46Z5YHVOPBWEMYQLG6NCJ4HT6ZX3IVMGVAFAQ, seg_pos={0 0}, upload=434386]: Uploading piece... Dec 26 06:23:47.647743005 (494h54m47.51882605s) [noise=true , address=45.91.116.11:4118 , piece_id=6BZNBNEDJB4RNNFMAXRMGLKK4GHYTDNKMVTMG5RF3U2VDKMON2KQ, seg_pos={0 0}, upload=434386]: Uploading piece... Dec 26 06:23:47.647943015 (494h54m47.51902606s) [noise=true , address=145.79.227.6:4259 , piece_id=TGQ6V3HNT3X2IY7MQ73RZRNH5NDXENTRDCXWHFHQETQEILSO7FEA, seg_pos={0 0}, upload=434386]: Uploading piece... Dec 26 06:23:49.283219995 (494h54m49.15430303s) [upload=434386]: Next returned nil segment Dec 26 06:23:49.283245634 (494h54m49.154328679s) [upload=434386]: Waiting for error group managing segments... Dec 26 06:23:49.644523232 (494h54m49.515606267s) [noise=true , address=145.79.227.3:4149 , piece_id=TOTASJEGQWCGG5KISNGGY3TLAL354QWMFA4VKEKDSCKHLY25PYQQ, seg_pos={0 0}, upload=434386]: Done uploading piece. err: <nil> Dec 26 06:23:49.648027525 (494h54m49.51911056s) [noise=true , address=145.79.227.49:4339 , piece_id=XQ6XNO7FGYMNEWYDGWGIB5IY5IHOAOPK7TUWPVPSEGQRIDA4R6GA, seg_pos={0 0}, upload=434386]: Done uploading piece. err: <nil> Dec 26 06:23:49.651435925 (494h54m49.52251897s) [noise=true , address=145.79.227.38:4977 , piece_id=PKXAJYGJPG6GYOG6DT4M6TBR54TTMBSWBXIIW2RLVMSA4HXBGOUQ, seg_pos={0 0}, upload=434386]: Done uploading piece. err: <nil> Dec 26 06:23:49.655712879 (494h54m49.526795924s) [noise=true , address=145.79.227.2:4115 , piece_id=D6S76VZDC2FBIFA3EA3TC4NEIFKJSGX6JRD4S26UO2IOLUDUPIDA, seg_pos={0 0}, upload=434386]: Done uploading piece. err: <nil> Dec 26 06:23:49.658634573 (494h54m49.529717618s) [noise=true , address=145.79.227.39:4013 , piece_id=3EHFPZVBIV6NNDJ46Z5YHVOPBWEMYQLG6NCJ4HT6ZX3IVMGVAFAQ, seg_pos={0 0}, upload=434386]: Done uploading piece. err: <nil> Dec 26 06:23:49.670194264 (494h54m49.541277309s) [noise=true , address=145.79.227.29:4942 , piece_id=GML4O7CTQ3AR5KAPGOZAMBELR7F6SDGSJSAWPZLO27W53ALPNZ2Q, seg_pos={0 0}, upload=434386]: Done uploading piece. err: <nil> Dec 26 06:23:49.672063052 (494h54m49.543146097s) [noise=true , address=145.79.227.5:4221 , piece_id=U2IAUDBXHO2UXGPSOYT3L2FPIJZFDTJ4GQCHGGXGSFD62KXIV77Q, seg_pos={0 0}, upload=434386]: Done uploading piece. err: <nil> Dec 26 06:23:49.672693352 (494h54m49.543776387s) [noise=true , address=145.79.227.20:4437 , piece_id=P4DNS6VYA2TS43TL7LYUSRUKFDJ37PO2M6D6NP534QQQBH2WIV2A, seg_pos={0 0}, upload=434386]: Done uploading piece. err: <nil> Dec 26 06:23:49.725648318 (494h54m49.596731363s) [noise=true , address=145.79.227.6:4259 , piece_id=TGQ6V3HNT3X2IY7MQ73RZRNH5NDXENTRDCXWHFHQETQEILSO7FEA, seg_pos={0 0}, upload=434386]: Done uploading piece. err: <nil> Dec 26 06:23:49.861945242 (494h54m49.733028287s) [noise=true , address=77.90.31.21:4880 , piece_id=JLHHQM55ZRSYUHKTKMZOMU4X3KXNQHIXH32KWIFA3CLWADW37W7Q, seg_pos={0 0}, upload=434386]: Done uploading piece. err: <nil> Dec 26 06:23:49.958824257 (494h54m49.829907292s) [noise=true , address=45.91.116.9:4264 , piece_id=N7AODEQP7YXGHYUJCT27BSAFMEJB3ABXUUCEUPSWDUR3F7DUB5TA, seg_pos={0 0}, upload=434386]: Done uploading piece. err: <nil> Dec 26 06:23:49.970075321 (494h54m49.841158366s) [noise=true , address=45.91.116.13:4153 , piece_id=7PR6IGAFJOAALB3YXBEWFAGCR72DFJ6JWT24WCO7FHUOCSMJNP7A, seg_pos={0 0}, upload=434386]: Done uploading piece. err: <nil> Dec 26 06:23:50.140738494 (494h54m50.011821529s) [noise=true , address=77.90.31.15:4700 , piece_id=23B7NZUGOMXA6OMBSKY3HWKZ36VVIVOCQGFBLFZ6EHGC7TD544YA, seg_pos={0 0}, upload=434386]: Done uploading piece. err: <nil> Dec 26 06:23:50.140774813 (494h54m50.011857849s) [seg_pos={0 0}, upload=434386]: Segment reached optimal threshold of 13 pieces. Dec 26 06:23:50.140993680 (494h54m50.012076725s) [noise=true , address=45.91.116.14:4191 , piece_id=LPPDT6VMSHKDJWZLB25GN2XLHS73352JSKCRZNEAFRRRWDLA7RNQ, seg_pos={0 0}, upload=434386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Dec 26 06:23:50.141002648 (494h54m50.012085683s) [noise=true , address=45.91.116.14:4191 , piece_id=LPPDT6VMSHKDJWZLB25GN2XLHS73352JSKCRZNEAFRRRWDLA7RNQ, seg_pos={0 0}, upload=434386]: Long tail cancellation due to optimal threshold Dec 26 06:23:50.141012116 (494h54m50.012095161s) [noise=true , address=45.91.116.7:4010 , piece_id=LHU4DWQU7XLA3QTJ2LHPSLGON43MUO3BXHJ5FRZPIRSA5IRH2LNA, seg_pos={0 0}, upload=434386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 26 06:23:50.141016754 (494h54m50.012099789s) [noise=true , address=45.91.116.7:4010 , piece_id=LHU4DWQU7XLA3QTJ2LHPSLGON43MUO3BXHJ5FRZPIRSA5IRH2LNA, seg_pos={0 0}, upload=434386]: Long tail cancellation due to optimal threshold Dec 26 06:23:50.141145951 (494h54m50.012228986s) [noise=true , address=45.91.116.11:4118 , piece_id=6BZNBNEDJB4RNNFMAXRMGLKK4GHYTDNKMVTMG5RF3U2VDKMON2KQ, seg_pos={0 0}, upload=434386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: expected piece hash; context canceled; EOF Dec 26 06:23:50.141156651 (494h54m50.012239686s) [noise=true , address=45.91.116.11:4118 , piece_id=6BZNBNEDJB4RNNFMAXRMGLKK4GHYTDNKMVTMG5RF3U2VDKMON2KQ, seg_pos={0 0}, upload=434386]: Long tail cancellation due to optimal threshold Dec 26 06:23:50.141172291 (494h54m50.012255336s) [seg_pos={0 0}, upload=434386]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 06:23:50.141260609 (494h54m50.012343644s) [upload=434386]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 06:23:50.141265488 (494h54m50.012348533s) [upload=434386]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 06:23:50.438800618 (494h54m50.309883653s) [upload=434386]: Done uploading segments... Dec 26 06:23:50.438865121 (494h54m50.309948166s) [upload=434386]: Upload finished. err: <nil> Duration: 5.76274008s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....................................X......................] duration: 3.341192258s trace: https://jaeger.dev.storj.io/trace/630094597d401477 FAIL (2025-12-26 06:26:07 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.341222426s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.......X..X.X.......................X..........X.......X.X..] duration: 5.903315179s trace: https://jaeger.dev.storj.io/trace/67a56359499790c8 FAIL (2025-12-26 06:25:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 06:23:47.396606446 (494h54m47.267689491s) [upload=434386]: Starting upload Dec 26 06:23:47.396675247 (494h54m47.267758292s) [upload=434386]: Done starting upload Dec 26 06:23:47.396898202 (494h54m47.267981248s) [upload=434386]: Uploading segments... Dec 26 06:23:47.396953998 (494h54m47.268037073s) [upload=434386]: Got next segment. Inline: false Dec 26 06:23:47.397031125 (494h54m47.26811416s) [upload=434386]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 06:23:47.397037317 (494h54m47.268120362s) [upload=434386]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 06:23:47.643130652 (494h54m47.514213687s) [seg_pos={0 0}, upload=434386]: Begin upload segment... Dec 26 06:23:47.643171870 (494h54m47.514254905s) [seg_pos={0 0}, upload=434386]: Done begin upload segment. Dec 26 06:23:47.643193913 (494h54m47.514276948s) [noise=true , address=145.79.227.38:4977 , piece_id=PKXAJYGJPG6GYOG6DT4M6TBR54TTMBSWBXIIW2RLVMSA4HXBGOUQ, seg_pos={0 0}, upload=434386]: Uploading piece... Dec 26 06:23:47.643278103 (494h54m47.514361138s) [noise=true , address=145.79.227.49:4339 , piece_id=XQ6XNO7FGYMNEWYDGWGIB5IY5IHOAOPK7TUWPVPSEGQRIDA4R6GA, seg_pos={0 0}, upload=434386]: Uploading piece... Dec 26 06:23:47.644040174 (494h54m47.515123219s) [noise=true , address=145.79.227.29:4942 , piece_id=GML4O7CTQ3AR5KAPGOZAMBELR7F6SDGSJSAWPZLO27W53ALPNZ2Q, seg_pos={0 0}, upload=434386]: Uploading piece... Dec 26 06:23:47.644250264 (494h54m47.515333309s) [noise=true , address=145.79.227.3:4149 , piece_id=TOTASJEGQWCGG5KISNGGY3TLAL354QWMFA4VKEKDSCKHLY25PYQQ, seg_pos={0 0}, upload=434386]: Uploading piece... Dec 26 06:23:47.644310018 (494h54m47.515393063s) [noise=true , address=145.79.227.5:4221 , piece_id=U2IAUDBXHO2UXGPSOYT3L2FPIJZFDTJ4GQCHGGXGSFD62KXIV77Q, seg_pos={0 0}, upload=434386]: Uploading piece... Dec 26 06:23:47.644590933 (494h54m47.515673968s) [noise=true , address=77.90.31.21:4880 , piece_id=JLHHQM55ZRSYUHKTKMZOMU4X3KXNQHIXH32KWIFA3CLWADW37W7Q, seg_pos={0 0}, upload=434386]: Uploading piece... Dec 26 06:23:47.644789010 (494h54m47.515872055s) [noise=true , address=45.91.116.13:4153 , piece_id=7PR6IGAFJOAALB3YXBEWFAGCR72DFJ6JWT24WCO7FHUOCSMJNP7A, seg_pos={0 0}, upload=434386]: Uploading piece... Dec 26 06:23:47.644945428 (494h54m47.516028463s) [noise=true , address=45.91.116.14:4191 , piece_id=LPPDT6VMSHKDJWZLB25GN2XLHS73352JSKCRZNEAFRRRWDLA7RNQ, seg_pos={0 0}, upload=434386]: Uploading piece... Dec 26 06:23:47.645273222 (494h54m47.516356257s) [noise=true , address=45.91.116.7:4010 , piece_id=LHU4DWQU7XLA3QTJ2LHPSLGON43MUO3BXHJ5FRZPIRSA5IRH2LNA, seg_pos={0 0}, upload=434386]: Uploading piece... Dec 26 06:23:47.645283942 (494h54m47.516366978s) [noise=true , address=45.91.116.9:4264 , piece_id=N7AODEQP7YXGHYUJCT27BSAFMEJB3ABXUUCEUPSWDUR3F7DUB5TA, seg_pos={0 0}, upload=434386]: Uploading piece... Dec 26 06:23:47.645654749 (494h54m47.516737784s) [noise=true , address=145.79.227.2:4115 , piece_id=D6S76VZDC2FBIFA3EA3TC4NEIFKJSGX6JRD4S26UO2IOLUDUPIDA, seg_pos={0 0}, upload=434386]: Uploading piece... Dec 26 06:23:47.645885748 (494h54m47.516968793s) [noise=true , address=145.79.227.20:4437 , piece_id=P4DNS6VYA2TS43TL7LYUSRUKFDJ37PO2M6D6NP534QQQBH2WIV2A, seg_pos={0 0}, upload=434386]: Uploading piece... Dec 26 06:23:47.646632821 (494h54m47.517715866s) [noise=true , address=77.90.31.15:4700 , piece_id=23B7NZUGOMXA6OMBSKY3HWKZ36VVIVOCQGFBLFZ6EHGC7TD544YA, seg_pos={0 0}, upload=434386]: Uploading piece... Dec 26 06:23:47.647299961 (494h54m47.518382996s) [noise=true , address=145.79.227.39:4013 , piece_id=3EHFPZVBIV6NNDJ46Z5YHVOPBWEMYQLG6NCJ4HT6ZX3IVMGVAFAQ, seg_pos={0 0}, upload=434386]: Uploading piece... Dec 26 06:23:47.647743005 (494h54m47.51882605s) [noise=true , address=45.91.116.11:4118 , piece_id=6BZNBNEDJB4RNNFMAXRMGLKK4GHYTDNKMVTMG5RF3U2VDKMON2KQ, seg_pos={0 0}, upload=434386]: Uploading piece... Dec 26 06:23:47.647943015 (494h54m47.51902606s) [noise=true , address=145.79.227.6:4259 , piece_id=TGQ6V3HNT3X2IY7MQ73RZRNH5NDXENTRDCXWHFHQETQEILSO7FEA, seg_pos={0 0}, upload=434386]: Uploading piece... Dec 26 06:23:49.283219995 (494h54m49.15430303s) [upload=434386]: Next returned nil segment Dec 26 06:23:49.283245634 (494h54m49.154328679s) [upload=434386]: Waiting for error group managing segments... Dec 26 06:23:49.644523232 (494h54m49.515606267s) [noise=true , address=145.79.227.3:4149 , piece_id=TOTASJEGQWCGG5KISNGGY3TLAL354QWMFA4VKEKDSCKHLY25PYQQ, seg_pos={0 0}, upload=434386]: Done uploading piece. err: <nil> Dec 26 06:23:49.648027525 (494h54m49.51911056s) [noise=true , address=145.79.227.49:4339 , piece_id=XQ6XNO7FGYMNEWYDGWGIB5IY5IHOAOPK7TUWPVPSEGQRIDA4R6GA, seg_pos={0 0}, upload=434386]: Done uploading piece. err: <nil> Dec 26 06:23:49.651435925 (494h54m49.52251897s) [noise=true , address=145.79.227.38:4977 , piece_id=PKXAJYGJPG6GYOG6DT4M6TBR54TTMBSWBXIIW2RLVMSA4HXBGOUQ, seg_pos={0 0}, upload=434386]: Done uploading piece. err: <nil> Dec 26 06:23:49.655712879 (494h54m49.526795924s) [noise=true , address=145.79.227.2:4115 , piece_id=D6S76VZDC2FBIFA3EA3TC4NEIFKJSGX6JRD4S26UO2IOLUDUPIDA, seg_pos={0 0}, upload=434386]: Done uploading piece. err: <nil> Dec 26 06:23:49.658634573 (494h54m49.529717618s) [noise=true , address=145.79.227.39:4013 , piece_id=3EHFPZVBIV6NNDJ46Z5YHVOPBWEMYQLG6NCJ4HT6ZX3IVMGVAFAQ, seg_pos={0 0}, upload=434386]: Done uploading piece. err: <nil> Dec 26 06:23:49.670194264 (494h54m49.541277309s) [noise=true , address=145.79.227.29:4942 , piece_id=GML4O7CTQ3AR5KAPGOZAMBELR7F6SDGSJSAWPZLO27W53ALPNZ2Q, seg_pos={0 0}, upload=434386]: Done uploading piece. err: <nil> Dec 26 06:23:49.672063052 (494h54m49.543146097s) [noise=true , address=145.79.227.5:4221 , piece_id=U2IAUDBXHO2UXGPSOYT3L2FPIJZFDTJ4GQCHGGXGSFD62KXIV77Q, seg_pos={0 0}, upload=434386]: Done uploading piece. err: <nil> Dec 26 06:23:49.672693352 (494h54m49.543776387s) [noise=true , address=145.79.227.20:4437 , piece_id=P4DNS6VYA2TS43TL7LYUSRUKFDJ37PO2M6D6NP534QQQBH2WIV2A, seg_pos={0 0}, upload=434386]: Done uploading piece. err: <nil> Dec 26 06:23:49.725648318 (494h54m49.596731363s) [noise=true , address=145.79.227.6:4259 , piece_id=TGQ6V3HNT3X2IY7MQ73RZRNH5NDXENTRDCXWHFHQETQEILSO7FEA, seg_pos={0 0}, upload=434386]: Done uploading piece. err: <nil> Dec 26 06:23:49.861945242 (494h54m49.733028287s) [noise=true , address=77.90.31.21:4880 , piece_id=JLHHQM55ZRSYUHKTKMZOMU4X3KXNQHIXH32KWIFA3CLWADW37W7Q, seg_pos={0 0}, upload=434386]: Done uploading piece. err: <nil> Dec 26 06:23:49.958824257 (494h54m49.829907292s) [noise=true , address=45.91.116.9:4264 , piece_id=N7AODEQP7YXGHYUJCT27BSAFMEJB3ABXUUCEUPSWDUR3F7DUB5TA, seg_pos={0 0}, upload=434386]: Done uploading piece. err: <nil> Dec 26 06:23:49.970075321 (494h54m49.841158366s) [noise=true , address=45.91.116.13:4153 , piece_id=7PR6IGAFJOAALB3YXBEWFAGCR72DFJ6JWT24WCO7FHUOCSMJNP7A, seg_pos={0 0}, upload=434386]: Done uploading piece. err: <nil> Dec 26 06:23:50.140738494 (494h54m50.011821529s) [noise=true , address=77.90.31.15:4700 , piece_id=23B7NZUGOMXA6OMBSKY3HWKZ36VVIVOCQGFBLFZ6EHGC7TD544YA, seg_pos={0 0}, upload=434386]: Done uploading piece. err: <nil> Dec 26 06:23:50.140774813 (494h54m50.011857849s) [seg_pos={0 0}, upload=434386]: Segment reached optimal threshold of 13 pieces. Dec 26 06:23:50.140993680 (494h54m50.012076725s) [noise=true , address=45.91.116.14:4191 , piece_id=LPPDT6VMSHKDJWZLB25GN2XLHS73352JSKCRZNEAFRRRWDLA7RNQ, seg_pos={0 0}, upload=434386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Dec 26 06:23:50.141002648 (494h54m50.012085683s) [noise=true , address=45.91.116.14:4191 , piece_id=LPPDT6VMSHKDJWZLB25GN2XLHS73352JSKCRZNEAFRRRWDLA7RNQ, seg_pos={0 0}, upload=434386]: Long tail cancellation due to optimal threshold Dec 26 06:23:50.141012116 (494h54m50.012095161s) [noise=true , address=45.91.116.7:4010 , piece_id=LHU4DWQU7XLA3QTJ2LHPSLGON43MUO3BXHJ5FRZPIRSA5IRH2LNA, seg_pos={0 0}, upload=434386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 26 06:23:50.141016754 (494h54m50.012099789s) [noise=true , address=45.91.116.7:4010 , piece_id=LHU4DWQU7XLA3QTJ2LHPSLGON43MUO3BXHJ5FRZPIRSA5IRH2LNA, seg_pos={0 0}, upload=434386]: Long tail cancellation due to optimal threshold Dec 26 06:23:50.141145951 (494h54m50.012228986s) [noise=true , address=45.91.116.11:4118 , piece_id=6BZNBNEDJB4RNNFMAXRMGLKK4GHYTDNKMVTMG5RF3U2VDKMON2KQ, seg_pos={0 0}, upload=434386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: expected piece hash; context canceled; EOF Dec 26 06:23:50.141156651 (494h54m50.012239686s) [noise=true , address=45.91.116.11:4118 , piece_id=6BZNBNEDJB4RNNFMAXRMGLKK4GHYTDNKMVTMG5RF3U2VDKMON2KQ, seg_pos={0 0}, upload=434386]: Long tail cancellation due to optimal threshold Dec 26 06:23:50.141172291 (494h54m50.012255336s) [seg_pos={0 0}, upload=434386]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 06:23:50.141260609 (494h54m50.012343644s) [upload=434386]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 06:23:50.141265488 (494h54m50.012348533s) [upload=434386]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 06:23:50.438800618 (494h54m50.309883653s) [upload=434386]: Done uploading segments... Dec 26 06:23:50.438865121 (494h54m50.309948166s) [upload=434386]: Upload finished. err: <nil> Duration: 5.903351147s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......................................X.....................] duration: 3.56879767s trace: https://jaeger.dev.storj.io/trace/630094597d401477 FAIL (2025-12-26 06:25:34 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.568819471s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 26.639110066s trace: https://jaeger.dev.storj.io/trace/7e198d4c8fea54d2 FAIL (2025-12-26 06:25:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p, Piece ID: SBCEX7UNWGNFF54SBP4AGRDV5IPG24FQJXJJ7EYQTBSZKOVZWU4Q) context deadline exceeded; piecestore: (Node ID: 1nP8xospdMuDEh3em5K6iBnh8bHXqLhvHy4pHm9m8Lnj4kEH2X, Piece ID: XV7FQGBJNWVRLOAFRUPOWON35JR5ESRJPTCWPUGYWGHJPSDVL4GA) context deadline exceeded; piecestore: (Node ID: 1Mg5mxUjW56quDidSgA788D7ByfAzvsEuWJqwtzY4bLyxRn3mu, Piece ID: ATPIXLMXGONSSLXPPVORSNC6LHPXI2V3UFTHM4VHA2FTY5ZM6H3A) context deadline exceeded; piecestore: (Node ID: 12QmYSEAKnkSLEEVc9E1uxtVigunKMFLk2cdu71Bse8Tqec3HyT, Piece ID: YFKD7GO2FOSPLDXNFPK732QTLCLBQQASWBTMZ2LTLBGSK22PIXIQ) context deadline exceeded; piecestore: (Node ID: 1JsnHuhQhM4rWLTLEuSSHLNE24m7tSW9qMSf48j7yMrAVPWVyV, Piece ID: ARBMDAKX4QBW6JG47KP2PUITCX7HKXCBO75THQPINM4LLHCKAV4Q) context deadline exceeded ======LOGS====== upload=433902]: Uploading piece... Dec 26 05:50:34.986155265 (494h21m34.8572383s) [noise=true , address=64.49.13.140:4509 , piece_id=6TGWTOAVY4YVAKU6VEE2SWVAM264TKU6ZNMGPWH4TNGII3UYXFXQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.986703610 (494h21m34.857786655s) [noise=true , address=77.90.31.10:4522 , piece_id=YFKD7GO2FOSPLDXNFPK732QTLCLBQQASWBTMZ2LTLBGSK22PIXIQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.987145380 (494h21m34.858228425s) [noise=true , address=145.79.227.33:4890 , piece_id=UKNB4ZZTYJGY4MDHOBPYUTHCP3ZDCMJB7O7AJZFYP2HFVIWLUOKQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.987628290 (494h21m34.858711335s) [noise=true , address=145.79.227.40:4052 , piece_id=XRZYL7PT5VHY22N7CCKS3NTZKDDWE36ZU3SORF6ILM7ZMXDND63A, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.988710731 (494h21m34.859793776s) [noise=true , address=77.90.31.13:4653 , piece_id=EY2CWX54ONOWPO3WC33G5QSP5RMP2SUTBQLUOH5JXGDRUEPDHQ2A, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.989894545 (494h21m34.86097759s) [noise=true , address=145.79.227.30:4785 , piece_id=5KMLU4ATBN4YVQDYJKKQY4PEJNKWTO2WNX5GMGJM4F54B6YM2SFA, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.991305081 (494h21m34.862388116s) [noise=true , address=145.79.227.50:4381 , piece_id=H6Q47DIOTOWGN7QH2KWT4IH6KVDSB246S22753Z3UQGB26T6VXPA, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.995796263 (494h21m34.866879359s) [noise=true , address=145.79.227.52:4461 , piece_id=ZAAUDBGY6CC2JNEGWXVPCBAWXARHAJDZ3YPB6Y3NWEWCFDUFGDSQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.996765439 (494h21m34.867848484s) [noise=true , address=77.90.31.16:4755 , piece_id=ARBMDAKX4QBW6JG47KP2PUITCX7HKXCBO75THQPINM4LLHCKAV4Q, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.998107634 (494h21m34.869190679s) [noise=true , address=77.90.31.11:4579 , piece_id=SBCEX7UNWGNFF54SBP4AGRDV5IPG24FQJXJJ7EYQTBSZKOVZWU4Q, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.999340442 (494h21m34.870423477s) [noise=true , address=145.79.227.34:4936 , piece_id=EEA2ZPM44DX3WMI7ZJTDC7T5UIGED6TNT53QXZWGBFRMQUQSCXPQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:35.000540757 (494h21m34.871623812s) [noise=true , address=45.91.116.14:4204 , piece_id=XV7FQGBJNWVRLOAFRUPOWON35JR5ESRJPTCWPUGYWGHJPSDVL4GA, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:36.360077064 (494h21m36.231160099s) [upload=433902]: Next returned nil segment Dec 26 05:50:36.360091642 (494h21m36.231174688s) [upload=433902]: Waiting for error group managing segments... Dec 26 05:50:36.688914299 (494h21m36.559997344s) [noise=true , address=64.49.13.140:4509 , piece_id=6TGWTOAVY4YVAKU6VEE2SWVAM264TKU6ZNMGPWH4TNGII3UYXFXQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.059812634 (494h21m36.930895669s) [noise=true , address=77.90.31.11:4579 , piece_id=SBCEX7UNWGNFF54SBP4AGRDV5IPG24FQJXJJ7EYQTBSZKOVZWU4Q, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.119433874 (494h21m36.990516909s) [noise=true , address=45.91.116.14:4204 , piece_id=XV7FQGBJNWVRLOAFRUPOWON35JR5ESRJPTCWPUGYWGHJPSDVL4GA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.138718204 (494h21m37.00980125s) [noise=true , address=77.90.31.16:4755 , piece_id=ARBMDAKX4QBW6JG47KP2PUITCX7HKXCBO75THQPINM4LLHCKAV4Q, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.146647243 (494h21m37.017730278s) [noise=true , address=77.90.31.13:4653 , piece_id=EY2CWX54ONOWPO3WC33G5QSP5RMP2SUTBQLUOH5JXGDRUEPDHQ2A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.163283722 (494h21m37.034366757s) [noise=true , address=77.90.31.21:4893 , piece_id=GNEEZO2XGOKNODOUPZQ4NGKFE4PCX5LNUWXLV27F6X7GWZVSYNAA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.374956574 (494h21m37.246039609s) [noise=true , address=45.91.116.7:4033 , piece_id=ATPIXLMXGONSSLXPPVORSNC6LHPXI2V3UFTHM4VHA2FTY5ZM6H3A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.777283355 (494h21m37.64836639s) [noise=true , address=77.90.31.12:4622 , piece_id=LTYYPK6ZQ3NXZPBKSV4OMZGCOXXV3N6L5A6MVODNR644D2ZUJX3A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:38.439025909 (494h21m38.310108944s) [noise=true , address=145.79.227.49:4355 , piece_id=IUDOHR73Y6OKFAQSNNSONALWAITRJ4POYRO2QD5LYZKOSTWY4MNA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:38.439599111 (494h21m38.310682166s) [noise=true , address=77.90.31.10:4522 , piece_id=YFKD7GO2FOSPLDXNFPK732QTLCLBQQASWBTMZ2LTLBGSK22PIXIQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:39.259154980 (494h21m39.130238015s) [noise=true , address=145.79.227.34:4936 , piece_id=EEA2ZPM44DX3WMI7ZJTDC7T5UIGED6TNT53QXZWGBFRMQUQSCXPQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:39.430935921 (494h21m39.302018956s) [noise=true , address=145.79.227.44:4212 , piece_id=ABBNLKRE4FM7RHF56V4IAQPKBCCCQLLOMFVMSJEWQUNFGGCI3PVQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:42.941071297 (494h21m42.812154352s) [noise=true , address=77.90.31.14:4675 , piece_id=FKB4O45OYICMELPGXB65QAUD2DRIFTSXZFKIA6TY26VFAP6AVY3A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:51:22.719026850 (494h22m22.590109885s) [noise=true , address=145.79.227.33:4890 , piece_id=UKNB4ZZTYJGY4MDHOBPYUTHCP3ZDCMJB7O7AJZFYP2HFVIWLUOKQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:51:22.719060193 (494h22m22.590143228s) [seg_pos={0 0}, upload=433902]: Segment reached optimal threshold of 14 pieces. Dec 26 05:51:22.719233022 (494h22m22.590316067s) [noise=true , address=145.79.227.50:4381 , piece_id=H6Q47DIOTOWGN7QH2KWT4IH6KVDSB246S22753Z3UQGB26T6VXPA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zU35MWWaaC1npkP5qGcWBXAkfxGsd5UsdkM2nwX9x7wUGCDbY): protocol: write tcp 188.166.225.170:46970->145.79.227.50:4381: use of closed network connection; write tcp 188.166.225.170:46970->145.79.227.50:4381: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46970->145.79.227.50:4381: use of closed network connection Dec 26 05:51:22.719238783 (494h22m22.590321828s) [noise=true , address=145.79.227.50:4381 , piece_id=H6Q47DIOTOWGN7QH2KWT4IH6KVDSB246S22753Z3UQGB26T6VXPA, seg_pos={0 0}, upload=433902]: Long tail cancellation due to optimal threshold Dec 26 05:51:22.719251677 (494h22m22.590334712s) [noise=true , address=145.79.227.30:4785 , piece_id=5KMLU4ATBN4YVQDYJKKQY4PEJNKWTO2WNX5GMGJM4F54B6YM2SFA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gYH83ECV8wai3CZyBSU8NNJM1UsysLYHbQDwBVKZRyjsdnUwB): protocol: write tcp 188.166.225.170:58398->145.79.227.30:4785: use of closed network connection; write tcp 188.166.225.170:58398->145.79.227.30:4785: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58398->145.79.227.30:4785: use of closed network connection Dec 26 05:51:22.719260194 (494h22m22.590343229s) [noise=true , address=145.79.227.30:4785 , piece_id=5KMLU4ATBN4YVQDYJKKQY4PEJNKWTO2WNX5GMGJM4F54B6YM2SFA, seg_pos={0 0}, upload=433902]: Long tail cancellation due to optimal threshold Dec 26 05:51:22.719409408 (494h22m22.590492443s) [noise=true , address=145.79.227.40:4052 , piece_id=XRZYL7PT5VHY22N7CCKS3NTZKDDWE36ZU3SORF6ILM7ZMXDND63A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PViJBYAanwXRWhMq3WHR6k9Stxy8hjuY5FV9J7ErxCy4N1q3V): protocol: write tcp 188.166.225.170:45984->145.79.227.40:4052: use of closed network connection; write tcp 188.166.225.170:45984->145.79.227.40:4052: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45984->145.79.227.40:4052: use of closed network connection Dec 26 05:51:22.719415469 (494h22m22.590498515s) [noise=true , address=145.79.227.40:4052 , piece_id=XRZYL7PT5VHY22N7CCKS3NTZKDDWE36ZU3SORF6ILM7ZMXDND63A, seg_pos={0 0}, upload=433902]: Long tail cancellation due to optimal threshold Dec 26 05:51:22.719467057 (494h22m22.590550092s) [noise=true , address=145.79.227.52:4461 , piece_id=ZAAUDBGY6CC2JNEGWXVPCBAWXARHAJDZ3YPB6Y3NWEWCFDUFGDSQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fNrbSfErEQRnruJCJAVvkoNwQSPWnvNtxunHn6zrQegnNAkxA): protocol: write tcp 188.166.225.170:55966->145.79.227.52:4461: use of closed network connection; write tcp 188.166.225.170:55966->145.79.227.52:4461: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55966->145.79.227.52:4461: use of closed network connection Dec 26 05:51:22.719473931 (494h22m22.590556966s) [noise=true , address=145.79.227.52:4461 , piece_id=ZAAUDBGY6CC2JNEGWXVPCBAWXARHAJDZ3YPB6Y3NWEWCFDUFGDSQ, seg_pos={0 0}, upload=433902]: Long tail cancellation due to optimal threshold Dec 26 05:51:22.719491374 (494h22m22.590574409s) [seg_pos={0 0}, upload=433902]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 05:51:22.719562539 (494h22m22.590645574s) [upload=433902]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:51:22.719567228 (494h22m22.590650263s) [upload=433902]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:51:23.008169117 (494h22m22.879252162s) [upload=433902]: Done uploading segments... Dec 26 05:51:23.008191560 (494h22m22.879274605s) [upload=433902]: Upload finished. err: <nil> Duration: 26.639125265s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)