Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [.X..X.X.XXXX.X..XX.XXXXX.X..................................] duration: 6.081714588s trace: https://jaeger.dev.storj.io/trace/422f33ac8ae6a74a FAIL (2025-11-07 18:38:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ) context deadline exceeded; piecestore: (Node ID: 12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, Piece ID: JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ) context deadline exceeded ======LOGS====== Nov 7 18:12:35.791697270 (198h32m7.487546023s) [upload=174963]: Starting upload Nov 7 18:12:35.791791689 (198h32m7.487640452s) [upload=174963]: Done starting upload Nov 7 18:12:35.791885458 (198h32m7.487734211s) [upload=174963]: Uploading segments... Nov 7 18:12:35.792146455 (198h32m7.487995208s) [upload=174963]: Got next segment. Inline: false Nov 7 18:12:35.792195278 (198h32m7.488044021s) [upload=174963]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 7 18:12:35.792201820 (198h32m7.488050563s) [upload=174963]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 7 18:12:36.063867588 (198h32m7.759716341s) [seg_pos={0 0}, upload=174963]: Begin upload segment... Nov 7 18:12:36.063937832 (198h32m7.759786585s) [seg_pos={0 0}, upload=174963]: Done begin upload segment. Nov 7 18:12:36.063988689 (198h32m7.759837463s) [noise=true , address=145.79.227.27:4690 , piece_id=7CVNJTAEBVYJXRWNSFYFP3E7SKMY5XI4CV2JEZJLSXH4TA2F2JZQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.064083510 (198h32m7.759932263s) [noise=true , address=145.79.227.10:4403 , piece_id=JCMXNJ5PW5ULGOJZ5SGB423VW7VJPC5REYNSNWJZJXHGUQXUS3KA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.064896969 (198h32m7.760745762s) [noise=true , address=145.79.227.28:4726 , piece_id=D6DH5FHFB5ZTPANRPUWZKRYPFYUV2LS6AY5FQL5RBJZIUNM4FKLQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.065443520 (198h32m7.761292273s) [noise=true , address=77.90.31.11:4554 , piece_id=TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.065970974 (198h32m7.761819777s) [noise=true , address=145.79.227.8:4330 , piece_id=32XBCOEGSUTVUCCWVM2D7CB6FKMNLD6N2N4655B3ZYHNAIL4R2QQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.066422053 (198h32m7.762270806s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.067054427 (198h32m7.76290318s) [noise=true , address=77.90.31.10:4519 , piece_id=MIHLKGHCRTUFQSGBIFKIWPNACAJ4CQOEYRDIKA4KJLFP356VMY6A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.067645042 (198h32m7.763493795s) [noise=true , address=45.91.116.6:4973 , piece_id=Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.068379079 (198h32m7.764227842s) [noise=true , address=77.90.31.19:4843 , piece_id=JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.070931099 (198h32m7.766779862s) [noise=true , address=145.79.227.11:4732 , piece_id=VKSPYLTD6C6XBUFXXVVPHIPGFRCRNQK6EVI4SM7Y6ODOMVGGAJOQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.073100420 (198h32m7.768949173s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.075821872 (198h32m7.771670625s) [noise=true , address=45.91.116.4:4937 , piece_id=CWDA2IYNFUYHVDBJ74Y3CJKCZUBN4QYUEJYFKZNIBAUYYAXO3WFQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.078400863 (198h32m7.774249626s) [noise=true , address=45.91.116.10:4082 , piece_id=RW62GCPC56K2IMOFF276GQECXRKW6NY6W44TNAYFNDGQS3UK7EEA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.081230762 (198h32m7.777079515s) [noise=true , address=145.79.227.15:4876 , piece_id=ZPGKGNXRFEWRRJKTS7NKBEVQA7OAWNWYDR3DUTPHT3XVYRJSKWPQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.083298258 (198h32m7.779147012s) [noise=true , address=145.79.227.20:4438 , piece_id=V6WDUVDB7WHTDCTSYCFWS22UKRD5NMXGJ2GHLSFLVVOXZMZWLL4A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.084602091 (198h32m7.780450844s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:37.640211006 (198h32m9.336059759s) [upload=174963]: Next returned nil segment Nov 7 18:12:37.640227878 (198h32m9.336076621s) [upload=174963]: Waiting for error group managing segments... Nov 7 18:12:38.007509819 (198h32m9.703358572s) [noise=true , address=145.79.227.10:4403 , piece_id=JCMXNJ5PW5ULGOJZ5SGB423VW7VJPC5REYNSNWJZJXHGUQXUS3KA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.015182750 (198h32m9.711031503s) [noise=true , address=145.79.227.28:4726 , piece_id=D6DH5FHFB5ZTPANRPUWZKRYPFYUV2LS6AY5FQL5RBJZIUNM4FKLQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.020400886 (198h32m9.71624964s) [noise=true , address=145.79.227.27:4690 , piece_id=7CVNJTAEBVYJXRWNSFYFP3E7SKMY5XI4CV2JEZJLSXH4TA2F2JZQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.062361869 (198h32m9.758210622s) [noise=true , address=145.79.227.15:4876 , piece_id=ZPGKGNXRFEWRRJKTS7NKBEVQA7OAWNWYDR3DUTPHT3XVYRJSKWPQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.096654439 (198h32m9.792503192s) [noise=true , address=145.79.227.20:4438 , piece_id=V6WDUVDB7WHTDCTSYCFWS22UKRD5NMXGJ2GHLSFLVVOXZMZWLL4A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.244365682 (198h32m9.940214435s) [noise=true , address=145.79.227.8:4330 , piece_id=32XBCOEGSUTVUCCWVM2D7CB6FKMNLD6N2N4655B3ZYHNAIL4R2QQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.245114297 (198h32m9.94096305s) [noise=true , address=145.79.227.11:4732 , piece_id=VKSPYLTD6C6XBUFXXVVPHIPGFRCRNQK6EVI4SM7Y6ODOMVGGAJOQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.324356281 (198h32m10.020205034s) [noise=true , address=77.90.31.11:4554 , piece_id=TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.331696878 (198h32m10.027545621s) [noise=true , address=45.91.116.4:4937 , piece_id=CWDA2IYNFUYHVDBJ74Y3CJKCZUBN4QYUEJYFKZNIBAUYYAXO3WFQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.334142676 (198h32m10.029991429s) [noise=true , address=45.91.116.10:4082 , piece_id=RW62GCPC56K2IMOFF276GQECXRKW6NY6W44TNAYFNDGQS3UK7EEA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.346539031 (198h32m10.042387794s) [noise=true , address=77.90.31.10:4519 , piece_id=MIHLKGHCRTUFQSGBIFKIWPNACAJ4CQOEYRDIKA4KJLFP356VMY6A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.419504032 (198h32m10.115352785s) [noise=true , address=77.90.31.19:4843 , piece_id=JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:39.113150499 (198h32m10.808999252s) [noise=true , address=45.91.116.6:4973 , piece_id=Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:39.113177150 (198h32m10.809025903s) [seg_pos={0 0}, upload=174963]: Segment reached optimal threshold of 13 pieces. Nov 7 18:12:39.113381018 (198h32m10.809229772s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113390166 (198h32m10.809238919s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113460199 (198h32m10.809308952s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113470077 (198h32m10.809318821s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113780338 (198h32m10.809629091s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113788243 (198h32m10.809636986s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113858657 (198h32m10.8097074s) [seg_pos={0 0}, upload=174963]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 7 18:12:39.114030194 (198h32m10.809878947s) [upload=174963]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 7 18:12:39.114038290 (198h32m10.809887043s) [upload=174963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 7 18:12:39.451448640 (198h32m11.147297403s) [upload=174963]: Done uploading segments... Nov 7 18:12:39.451473607 (198h32m11.14732236s) [upload=174963]: Upload finished. err: <nil> Duration: 6.081748323s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [X.X..X.X.XXXX.X..XX.XXXXX.X.................................] duration: 6.094572377s trace: https://jaeger.dev.storj.io/trace/422f33ac8ae6a74a FAIL (2025-11-07 18:37:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ) context deadline exceeded; piecestore: (Node ID: 12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, Piece ID: JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ) context deadline exceeded ======LOGS====== Nov 7 18:12:35.791697270 (198h32m7.487546023s) [upload=174963]: Starting upload Nov 7 18:12:35.791791689 (198h32m7.487640452s) [upload=174963]: Done starting upload Nov 7 18:12:35.791885458 (198h32m7.487734211s) [upload=174963]: Uploading segments... Nov 7 18:12:35.792146455 (198h32m7.487995208s) [upload=174963]: Got next segment. Inline: false Nov 7 18:12:35.792195278 (198h32m7.488044021s) [upload=174963]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 7 18:12:35.792201820 (198h32m7.488050563s) [upload=174963]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 7 18:12:36.063867588 (198h32m7.759716341s) [seg_pos={0 0}, upload=174963]: Begin upload segment... Nov 7 18:12:36.063937832 (198h32m7.759786585s) [seg_pos={0 0}, upload=174963]: Done begin upload segment. Nov 7 18:12:36.063988689 (198h32m7.759837463s) [noise=true , address=145.79.227.27:4690 , piece_id=7CVNJTAEBVYJXRWNSFYFP3E7SKMY5XI4CV2JEZJLSXH4TA2F2JZQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.064083510 (198h32m7.759932263s) [noise=true , address=145.79.227.10:4403 , piece_id=JCMXNJ5PW5ULGOJZ5SGB423VW7VJPC5REYNSNWJZJXHGUQXUS3KA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.064896969 (198h32m7.760745762s) [noise=true , address=145.79.227.28:4726 , piece_id=D6DH5FHFB5ZTPANRPUWZKRYPFYUV2LS6AY5FQL5RBJZIUNM4FKLQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.065443520 (198h32m7.761292273s) [noise=true , address=77.90.31.11:4554 , piece_id=TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.065970974 (198h32m7.761819777s) [noise=true , address=145.79.227.8:4330 , piece_id=32XBCOEGSUTVUCCWVM2D7CB6FKMNLD6N2N4655B3ZYHNAIL4R2QQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.066422053 (198h32m7.762270806s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.067054427 (198h32m7.76290318s) [noise=true , address=77.90.31.10:4519 , piece_id=MIHLKGHCRTUFQSGBIFKIWPNACAJ4CQOEYRDIKA4KJLFP356VMY6A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.067645042 (198h32m7.763493795s) [noise=true , address=45.91.116.6:4973 , piece_id=Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.068379079 (198h32m7.764227842s) [noise=true , address=77.90.31.19:4843 , piece_id=JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.070931099 (198h32m7.766779862s) [noise=true , address=145.79.227.11:4732 , piece_id=VKSPYLTD6C6XBUFXXVVPHIPGFRCRNQK6EVI4SM7Y6ODOMVGGAJOQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.073100420 (198h32m7.768949173s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.075821872 (198h32m7.771670625s) [noise=true , address=45.91.116.4:4937 , piece_id=CWDA2IYNFUYHVDBJ74Y3CJKCZUBN4QYUEJYFKZNIBAUYYAXO3WFQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.078400863 (198h32m7.774249626s) [noise=true , address=45.91.116.10:4082 , piece_id=RW62GCPC56K2IMOFF276GQECXRKW6NY6W44TNAYFNDGQS3UK7EEA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.081230762 (198h32m7.777079515s) [noise=true , address=145.79.227.15:4876 , piece_id=ZPGKGNXRFEWRRJKTS7NKBEVQA7OAWNWYDR3DUTPHT3XVYRJSKWPQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.083298258 (198h32m7.779147012s) [noise=true , address=145.79.227.20:4438 , piece_id=V6WDUVDB7WHTDCTSYCFWS22UKRD5NMXGJ2GHLSFLVVOXZMZWLL4A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.084602091 (198h32m7.780450844s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:37.640211006 (198h32m9.336059759s) [upload=174963]: Next returned nil segment Nov 7 18:12:37.640227878 (198h32m9.336076621s) [upload=174963]: Waiting for error group managing segments... Nov 7 18:12:38.007509819 (198h32m9.703358572s) [noise=true , address=145.79.227.10:4403 , piece_id=JCMXNJ5PW5ULGOJZ5SGB423VW7VJPC5REYNSNWJZJXHGUQXUS3KA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.015182750 (198h32m9.711031503s) [noise=true , address=145.79.227.28:4726 , piece_id=D6DH5FHFB5ZTPANRPUWZKRYPFYUV2LS6AY5FQL5RBJZIUNM4FKLQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.020400886 (198h32m9.71624964s) [noise=true , address=145.79.227.27:4690 , piece_id=7CVNJTAEBVYJXRWNSFYFP3E7SKMY5XI4CV2JEZJLSXH4TA2F2JZQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.062361869 (198h32m9.758210622s) [noise=true , address=145.79.227.15:4876 , piece_id=ZPGKGNXRFEWRRJKTS7NKBEVQA7OAWNWYDR3DUTPHT3XVYRJSKWPQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.096654439 (198h32m9.792503192s) [noise=true , address=145.79.227.20:4438 , piece_id=V6WDUVDB7WHTDCTSYCFWS22UKRD5NMXGJ2GHLSFLVVOXZMZWLL4A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.244365682 (198h32m9.940214435s) [noise=true , address=145.79.227.8:4330 , piece_id=32XBCOEGSUTVUCCWVM2D7CB6FKMNLD6N2N4655B3ZYHNAIL4R2QQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.245114297 (198h32m9.94096305s) [noise=true , address=145.79.227.11:4732 , piece_id=VKSPYLTD6C6XBUFXXVVPHIPGFRCRNQK6EVI4SM7Y6ODOMVGGAJOQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.324356281 (198h32m10.020205034s) [noise=true , address=77.90.31.11:4554 , piece_id=TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.331696878 (198h32m10.027545621s) [noise=true , address=45.91.116.4:4937 , piece_id=CWDA2IYNFUYHVDBJ74Y3CJKCZUBN4QYUEJYFKZNIBAUYYAXO3WFQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.334142676 (198h32m10.029991429s) [noise=true , address=45.91.116.10:4082 , piece_id=RW62GCPC56K2IMOFF276GQECXRKW6NY6W44TNAYFNDGQS3UK7EEA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.346539031 (198h32m10.042387794s) [noise=true , address=77.90.31.10:4519 , piece_id=MIHLKGHCRTUFQSGBIFKIWPNACAJ4CQOEYRDIKA4KJLFP356VMY6A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.419504032 (198h32m10.115352785s) [noise=true , address=77.90.31.19:4843 , piece_id=JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:39.113150499 (198h32m10.808999252s) [noise=true , address=45.91.116.6:4973 , piece_id=Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:39.113177150 (198h32m10.809025903s) [seg_pos={0 0}, upload=174963]: Segment reached optimal threshold of 13 pieces. Nov 7 18:12:39.113381018 (198h32m10.809229772s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113390166 (198h32m10.809238919s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113460199 (198h32m10.809308952s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113470077 (198h32m10.809318821s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113780338 (198h32m10.809629091s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113788243 (198h32m10.809636986s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113858657 (198h32m10.8097074s) [seg_pos={0 0}, upload=174963]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 7 18:12:39.114030194 (198h32m10.809878947s) [upload=174963]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 7 18:12:39.114038290 (198h32m10.809887043s) [upload=174963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 7 18:12:39.451448640 (198h32m11.147297403s) [upload=174963]: Done uploading segments... Nov 7 18:12:39.451473607 (198h32m11.14732236s) [upload=174963]: Upload finished. err: <nil> Duration: 6.094609717s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [XX.X..X.X.XXXX.X..XX.XXXXX.X................................] duration: 6.705880638s trace: https://jaeger.dev.storj.io/trace/422f33ac8ae6a74a FAIL (2025-11-07 18:36:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ) context deadline exceeded; piecestore: (Node ID: 12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, Piece ID: JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ) context deadline exceeded ======LOGS====== Nov 7 18:12:35.791697270 (198h32m7.487546023s) [upload=174963]: Starting upload Nov 7 18:12:35.791791689 (198h32m7.487640452s) [upload=174963]: Done starting upload Nov 7 18:12:35.791885458 (198h32m7.487734211s) [upload=174963]: Uploading segments... Nov 7 18:12:35.792146455 (198h32m7.487995208s) [upload=174963]: Got next segment. Inline: false Nov 7 18:12:35.792195278 (198h32m7.488044021s) [upload=174963]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 7 18:12:35.792201820 (198h32m7.488050563s) [upload=174963]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 7 18:12:36.063867588 (198h32m7.759716341s) [seg_pos={0 0}, upload=174963]: Begin upload segment... Nov 7 18:12:36.063937832 (198h32m7.759786585s) [seg_pos={0 0}, upload=174963]: Done begin upload segment. Nov 7 18:12:36.063988689 (198h32m7.759837463s) [noise=true , address=145.79.227.27:4690 , piece_id=7CVNJTAEBVYJXRWNSFYFP3E7SKMY5XI4CV2JEZJLSXH4TA2F2JZQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.064083510 (198h32m7.759932263s) [noise=true , address=145.79.227.10:4403 , piece_id=JCMXNJ5PW5ULGOJZ5SGB423VW7VJPC5REYNSNWJZJXHGUQXUS3KA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.064896969 (198h32m7.760745762s) [noise=true , address=145.79.227.28:4726 , piece_id=D6DH5FHFB5ZTPANRPUWZKRYPFYUV2LS6AY5FQL5RBJZIUNM4FKLQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.065443520 (198h32m7.761292273s) [noise=true , address=77.90.31.11:4554 , piece_id=TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.065970974 (198h32m7.761819777s) [noise=true , address=145.79.227.8:4330 , piece_id=32XBCOEGSUTVUCCWVM2D7CB6FKMNLD6N2N4655B3ZYHNAIL4R2QQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.066422053 (198h32m7.762270806s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.067054427 (198h32m7.76290318s) [noise=true , address=77.90.31.10:4519 , piece_id=MIHLKGHCRTUFQSGBIFKIWPNACAJ4CQOEYRDIKA4KJLFP356VMY6A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.067645042 (198h32m7.763493795s) [noise=true , address=45.91.116.6:4973 , piece_id=Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.068379079 (198h32m7.764227842s) [noise=true , address=77.90.31.19:4843 , piece_id=JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.070931099 (198h32m7.766779862s) [noise=true , address=145.79.227.11:4732 , piece_id=VKSPYLTD6C6XBUFXXVVPHIPGFRCRNQK6EVI4SM7Y6ODOMVGGAJOQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.073100420 (198h32m7.768949173s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.075821872 (198h32m7.771670625s) [noise=true , address=45.91.116.4:4937 , piece_id=CWDA2IYNFUYHVDBJ74Y3CJKCZUBN4QYUEJYFKZNIBAUYYAXO3WFQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.078400863 (198h32m7.774249626s) [noise=true , address=45.91.116.10:4082 , piece_id=RW62GCPC56K2IMOFF276GQECXRKW6NY6W44TNAYFNDGQS3UK7EEA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.081230762 (198h32m7.777079515s) [noise=true , address=145.79.227.15:4876 , piece_id=ZPGKGNXRFEWRRJKTS7NKBEVQA7OAWNWYDR3DUTPHT3XVYRJSKWPQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.083298258 (198h32m7.779147012s) [noise=true , address=145.79.227.20:4438 , piece_id=V6WDUVDB7WHTDCTSYCFWS22UKRD5NMXGJ2GHLSFLVVOXZMZWLL4A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.084602091 (198h32m7.780450844s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:37.640211006 (198h32m9.336059759s) [upload=174963]: Next returned nil segment Nov 7 18:12:37.640227878 (198h32m9.336076621s) [upload=174963]: Waiting for error group managing segments... Nov 7 18:12:38.007509819 (198h32m9.703358572s) [noise=true , address=145.79.227.10:4403 , piece_id=JCMXNJ5PW5ULGOJZ5SGB423VW7VJPC5REYNSNWJZJXHGUQXUS3KA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.015182750 (198h32m9.711031503s) [noise=true , address=145.79.227.28:4726 , piece_id=D6DH5FHFB5ZTPANRPUWZKRYPFYUV2LS6AY5FQL5RBJZIUNM4FKLQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.020400886 (198h32m9.71624964s) [noise=true , address=145.79.227.27:4690 , piece_id=7CVNJTAEBVYJXRWNSFYFP3E7SKMY5XI4CV2JEZJLSXH4TA2F2JZQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.062361869 (198h32m9.758210622s) [noise=true , address=145.79.227.15:4876 , piece_id=ZPGKGNXRFEWRRJKTS7NKBEVQA7OAWNWYDR3DUTPHT3XVYRJSKWPQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.096654439 (198h32m9.792503192s) [noise=true , address=145.79.227.20:4438 , piece_id=V6WDUVDB7WHTDCTSYCFWS22UKRD5NMXGJ2GHLSFLVVOXZMZWLL4A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.244365682 (198h32m9.940214435s) [noise=true , address=145.79.227.8:4330 , piece_id=32XBCOEGSUTVUCCWVM2D7CB6FKMNLD6N2N4655B3ZYHNAIL4R2QQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.245114297 (198h32m9.94096305s) [noise=true , address=145.79.227.11:4732 , piece_id=VKSPYLTD6C6XBUFXXVVPHIPGFRCRNQK6EVI4SM7Y6ODOMVGGAJOQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.324356281 (198h32m10.020205034s) [noise=true , address=77.90.31.11:4554 , piece_id=TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.331696878 (198h32m10.027545621s) [noise=true , address=45.91.116.4:4937 , piece_id=CWDA2IYNFUYHVDBJ74Y3CJKCZUBN4QYUEJYFKZNIBAUYYAXO3WFQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.334142676 (198h32m10.029991429s) [noise=true , address=45.91.116.10:4082 , piece_id=RW62GCPC56K2IMOFF276GQECXRKW6NY6W44TNAYFNDGQS3UK7EEA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.346539031 (198h32m10.042387794s) [noise=true , address=77.90.31.10:4519 , piece_id=MIHLKGHCRTUFQSGBIFKIWPNACAJ4CQOEYRDIKA4KJLFP356VMY6A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.419504032 (198h32m10.115352785s) [noise=true , address=77.90.31.19:4843 , piece_id=JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:39.113150499 (198h32m10.808999252s) [noise=true , address=45.91.116.6:4973 , piece_id=Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:39.113177150 (198h32m10.809025903s) [seg_pos={0 0}, upload=174963]: Segment reached optimal threshold of 13 pieces. Nov 7 18:12:39.113381018 (198h32m10.809229772s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113390166 (198h32m10.809238919s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113460199 (198h32m10.809308952s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113470077 (198h32m10.809318821s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113780338 (198h32m10.809629091s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113788243 (198h32m10.809636986s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113858657 (198h32m10.8097074s) [seg_pos={0 0}, upload=174963]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 7 18:12:39.114030194 (198h32m10.809878947s) [upload=174963]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 7 18:12:39.114038290 (198h32m10.809887043s) [upload=174963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 7 18:12:39.451448640 (198h32m11.147297403s) [upload=174963]: Done uploading segments... Nov 7 18:12:39.451473607 (198h32m11.14732236s) [upload=174963]: Upload finished. err: <nil> Duration: 6.70590884s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [XXX.X..X.X.XXXX.X..XX.XXXXX.X...............................] duration: 6.059421365s trace: https://jaeger.dev.storj.io/trace/422f33ac8ae6a74a FAIL (2025-11-07 18:35:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ) context deadline exceeded; piecestore: (Node ID: 12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, Piece ID: JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ) context deadline exceeded ======LOGS====== Nov 7 18:12:35.791697270 (198h32m7.487546023s) [upload=174963]: Starting upload Nov 7 18:12:35.791791689 (198h32m7.487640452s) [upload=174963]: Done starting upload Nov 7 18:12:35.791885458 (198h32m7.487734211s) [upload=174963]: Uploading segments... Nov 7 18:12:35.792146455 (198h32m7.487995208s) [upload=174963]: Got next segment. Inline: false Nov 7 18:12:35.792195278 (198h32m7.488044021s) [upload=174963]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 7 18:12:35.792201820 (198h32m7.488050563s) [upload=174963]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 7 18:12:36.063867588 (198h32m7.759716341s) [seg_pos={0 0}, upload=174963]: Begin upload segment... Nov 7 18:12:36.063937832 (198h32m7.759786585s) [seg_pos={0 0}, upload=174963]: Done begin upload segment. Nov 7 18:12:36.063988689 (198h32m7.759837463s) [noise=true , address=145.79.227.27:4690 , piece_id=7CVNJTAEBVYJXRWNSFYFP3E7SKMY5XI4CV2JEZJLSXH4TA2F2JZQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.064083510 (198h32m7.759932263s) [noise=true , address=145.79.227.10:4403 , piece_id=JCMXNJ5PW5ULGOJZ5SGB423VW7VJPC5REYNSNWJZJXHGUQXUS3KA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.064896969 (198h32m7.760745762s) [noise=true , address=145.79.227.28:4726 , piece_id=D6DH5FHFB5ZTPANRPUWZKRYPFYUV2LS6AY5FQL5RBJZIUNM4FKLQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.065443520 (198h32m7.761292273s) [noise=true , address=77.90.31.11:4554 , piece_id=TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.065970974 (198h32m7.761819777s) [noise=true , address=145.79.227.8:4330 , piece_id=32XBCOEGSUTVUCCWVM2D7CB6FKMNLD6N2N4655B3ZYHNAIL4R2QQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.066422053 (198h32m7.762270806s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.067054427 (198h32m7.76290318s) [noise=true , address=77.90.31.10:4519 , piece_id=MIHLKGHCRTUFQSGBIFKIWPNACAJ4CQOEYRDIKA4KJLFP356VMY6A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.067645042 (198h32m7.763493795s) [noise=true , address=45.91.116.6:4973 , piece_id=Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.068379079 (198h32m7.764227842s) [noise=true , address=77.90.31.19:4843 , piece_id=JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.070931099 (198h32m7.766779862s) [noise=true , address=145.79.227.11:4732 , piece_id=VKSPYLTD6C6XBUFXXVVPHIPGFRCRNQK6EVI4SM7Y6ODOMVGGAJOQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.073100420 (198h32m7.768949173s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.075821872 (198h32m7.771670625s) [noise=true , address=45.91.116.4:4937 , piece_id=CWDA2IYNFUYHVDBJ74Y3CJKCZUBN4QYUEJYFKZNIBAUYYAXO3WFQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.078400863 (198h32m7.774249626s) [noise=true , address=45.91.116.10:4082 , piece_id=RW62GCPC56K2IMOFF276GQECXRKW6NY6W44TNAYFNDGQS3UK7EEA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.081230762 (198h32m7.777079515s) [noise=true , address=145.79.227.15:4876 , piece_id=ZPGKGNXRFEWRRJKTS7NKBEVQA7OAWNWYDR3DUTPHT3XVYRJSKWPQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.083298258 (198h32m7.779147012s) [noise=true , address=145.79.227.20:4438 , piece_id=V6WDUVDB7WHTDCTSYCFWS22UKRD5NMXGJ2GHLSFLVVOXZMZWLL4A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.084602091 (198h32m7.780450844s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:37.640211006 (198h32m9.336059759s) [upload=174963]: Next returned nil segment Nov 7 18:12:37.640227878 (198h32m9.336076621s) [upload=174963]: Waiting for error group managing segments... Nov 7 18:12:38.007509819 (198h32m9.703358572s) [noise=true , address=145.79.227.10:4403 , piece_id=JCMXNJ5PW5ULGOJZ5SGB423VW7VJPC5REYNSNWJZJXHGUQXUS3KA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.015182750 (198h32m9.711031503s) [noise=true , address=145.79.227.28:4726 , piece_id=D6DH5FHFB5ZTPANRPUWZKRYPFYUV2LS6AY5FQL5RBJZIUNM4FKLQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.020400886 (198h32m9.71624964s) [noise=true , address=145.79.227.27:4690 , piece_id=7CVNJTAEBVYJXRWNSFYFP3E7SKMY5XI4CV2JEZJLSXH4TA2F2JZQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.062361869 (198h32m9.758210622s) [noise=true , address=145.79.227.15:4876 , piece_id=ZPGKGNXRFEWRRJKTS7NKBEVQA7OAWNWYDR3DUTPHT3XVYRJSKWPQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.096654439 (198h32m9.792503192s) [noise=true , address=145.79.227.20:4438 , piece_id=V6WDUVDB7WHTDCTSYCFWS22UKRD5NMXGJ2GHLSFLVVOXZMZWLL4A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.244365682 (198h32m9.940214435s) [noise=true , address=145.79.227.8:4330 , piece_id=32XBCOEGSUTVUCCWVM2D7CB6FKMNLD6N2N4655B3ZYHNAIL4R2QQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.245114297 (198h32m9.94096305s) [noise=true , address=145.79.227.11:4732 , piece_id=VKSPYLTD6C6XBUFXXVVPHIPGFRCRNQK6EVI4SM7Y6ODOMVGGAJOQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.324356281 (198h32m10.020205034s) [noise=true , address=77.90.31.11:4554 , piece_id=TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.331696878 (198h32m10.027545621s) [noise=true , address=45.91.116.4:4937 , piece_id=CWDA2IYNFUYHVDBJ74Y3CJKCZUBN4QYUEJYFKZNIBAUYYAXO3WFQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.334142676 (198h32m10.029991429s) [noise=true , address=45.91.116.10:4082 , piece_id=RW62GCPC56K2IMOFF276GQECXRKW6NY6W44TNAYFNDGQS3UK7EEA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.346539031 (198h32m10.042387794s) [noise=true , address=77.90.31.10:4519 , piece_id=MIHLKGHCRTUFQSGBIFKIWPNACAJ4CQOEYRDIKA4KJLFP356VMY6A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.419504032 (198h32m10.115352785s) [noise=true , address=77.90.31.19:4843 , piece_id=JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:39.113150499 (198h32m10.808999252s) [noise=true , address=45.91.116.6:4973 , piece_id=Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:39.113177150 (198h32m10.809025903s) [seg_pos={0 0}, upload=174963]: Segment reached optimal threshold of 13 pieces. Nov 7 18:12:39.113381018 (198h32m10.809229772s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113390166 (198h32m10.809238919s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113460199 (198h32m10.809308952s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113470077 (198h32m10.809318821s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113780338 (198h32m10.809629091s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113788243 (198h32m10.809636986s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113858657 (198h32m10.8097074s) [seg_pos={0 0}, upload=174963]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 7 18:12:39.114030194 (198h32m10.809878947s) [upload=174963]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 7 18:12:39.114038290 (198h32m10.809887043s) [upload=174963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 7 18:12:39.451448640 (198h32m11.147297403s) [upload=174963]: Done uploading segments... Nov 7 18:12:39.451473607 (198h32m11.14732236s) [upload=174963]: Upload finished. err: <nil> Duration: 6.059453807s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [.XXX.X..X.X.XXXX.X..XX.XXXXX.X..............................] duration: 6.282441897s trace: https://jaeger.dev.storj.io/trace/422f33ac8ae6a74a FAIL (2025-11-07 18:35:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ) context deadline exceeded; piecestore: (Node ID: 12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, Piece ID: JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ) context deadline exceeded ======LOGS====== Nov 7 18:12:35.791697270 (198h32m7.487546023s) [upload=174963]: Starting upload Nov 7 18:12:35.791791689 (198h32m7.487640452s) [upload=174963]: Done starting upload Nov 7 18:12:35.791885458 (198h32m7.487734211s) [upload=174963]: Uploading segments... Nov 7 18:12:35.792146455 (198h32m7.487995208s) [upload=174963]: Got next segment. Inline: false Nov 7 18:12:35.792195278 (198h32m7.488044021s) [upload=174963]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 7 18:12:35.792201820 (198h32m7.488050563s) [upload=174963]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 7 18:12:36.063867588 (198h32m7.759716341s) [seg_pos={0 0}, upload=174963]: Begin upload segment... Nov 7 18:12:36.063937832 (198h32m7.759786585s) [seg_pos={0 0}, upload=174963]: Done begin upload segment. Nov 7 18:12:36.063988689 (198h32m7.759837463s) [noise=true , address=145.79.227.27:4690 , piece_id=7CVNJTAEBVYJXRWNSFYFP3E7SKMY5XI4CV2JEZJLSXH4TA2F2JZQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.064083510 (198h32m7.759932263s) [noise=true , address=145.79.227.10:4403 , piece_id=JCMXNJ5PW5ULGOJZ5SGB423VW7VJPC5REYNSNWJZJXHGUQXUS3KA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.064896969 (198h32m7.760745762s) [noise=true , address=145.79.227.28:4726 , piece_id=D6DH5FHFB5ZTPANRPUWZKRYPFYUV2LS6AY5FQL5RBJZIUNM4FKLQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.065443520 (198h32m7.761292273s) [noise=true , address=77.90.31.11:4554 , piece_id=TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.065970974 (198h32m7.761819777s) [noise=true , address=145.79.227.8:4330 , piece_id=32XBCOEGSUTVUCCWVM2D7CB6FKMNLD6N2N4655B3ZYHNAIL4R2QQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.066422053 (198h32m7.762270806s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.067054427 (198h32m7.76290318s) [noise=true , address=77.90.31.10:4519 , piece_id=MIHLKGHCRTUFQSGBIFKIWPNACAJ4CQOEYRDIKA4KJLFP356VMY6A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.067645042 (198h32m7.763493795s) [noise=true , address=45.91.116.6:4973 , piece_id=Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.068379079 (198h32m7.764227842s) [noise=true , address=77.90.31.19:4843 , piece_id=JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.070931099 (198h32m7.766779862s) [noise=true , address=145.79.227.11:4732 , piece_id=VKSPYLTD6C6XBUFXXVVPHIPGFRCRNQK6EVI4SM7Y6ODOMVGGAJOQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.073100420 (198h32m7.768949173s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.075821872 (198h32m7.771670625s) [noise=true , address=45.91.116.4:4937 , piece_id=CWDA2IYNFUYHVDBJ74Y3CJKCZUBN4QYUEJYFKZNIBAUYYAXO3WFQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.078400863 (198h32m7.774249626s) [noise=true , address=45.91.116.10:4082 , piece_id=RW62GCPC56K2IMOFF276GQECXRKW6NY6W44TNAYFNDGQS3UK7EEA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.081230762 (198h32m7.777079515s) [noise=true , address=145.79.227.15:4876 , piece_id=ZPGKGNXRFEWRRJKTS7NKBEVQA7OAWNWYDR3DUTPHT3XVYRJSKWPQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.083298258 (198h32m7.779147012s) [noise=true , address=145.79.227.20:4438 , piece_id=V6WDUVDB7WHTDCTSYCFWS22UKRD5NMXGJ2GHLSFLVVOXZMZWLL4A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.084602091 (198h32m7.780450844s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:37.640211006 (198h32m9.336059759s) [upload=174963]: Next returned nil segment Nov 7 18:12:37.640227878 (198h32m9.336076621s) [upload=174963]: Waiting for error group managing segments... Nov 7 18:12:38.007509819 (198h32m9.703358572s) [noise=true , address=145.79.227.10:4403 , piece_id=JCMXNJ5PW5ULGOJZ5SGB423VW7VJPC5REYNSNWJZJXHGUQXUS3KA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.015182750 (198h32m9.711031503s) [noise=true , address=145.79.227.28:4726 , piece_id=D6DH5FHFB5ZTPANRPUWZKRYPFYUV2LS6AY5FQL5RBJZIUNM4FKLQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.020400886 (198h32m9.71624964s) [noise=true , address=145.79.227.27:4690 , piece_id=7CVNJTAEBVYJXRWNSFYFP3E7SKMY5XI4CV2JEZJLSXH4TA2F2JZQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.062361869 (198h32m9.758210622s) [noise=true , address=145.79.227.15:4876 , piece_id=ZPGKGNXRFEWRRJKTS7NKBEVQA7OAWNWYDR3DUTPHT3XVYRJSKWPQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.096654439 (198h32m9.792503192s) [noise=true , address=145.79.227.20:4438 , piece_id=V6WDUVDB7WHTDCTSYCFWS22UKRD5NMXGJ2GHLSFLVVOXZMZWLL4A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.244365682 (198h32m9.940214435s) [noise=true , address=145.79.227.8:4330 , piece_id=32XBCOEGSUTVUCCWVM2D7CB6FKMNLD6N2N4655B3ZYHNAIL4R2QQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.245114297 (198h32m9.94096305s) [noise=true , address=145.79.227.11:4732 , piece_id=VKSPYLTD6C6XBUFXXVVPHIPGFRCRNQK6EVI4SM7Y6ODOMVGGAJOQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.324356281 (198h32m10.020205034s) [noise=true , address=77.90.31.11:4554 , piece_id=TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.331696878 (198h32m10.027545621s) [noise=true , address=45.91.116.4:4937 , piece_id=CWDA2IYNFUYHVDBJ74Y3CJKCZUBN4QYUEJYFKZNIBAUYYAXO3WFQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.334142676 (198h32m10.029991429s) [noise=true , address=45.91.116.10:4082 , piece_id=RW62GCPC56K2IMOFF276GQECXRKW6NY6W44TNAYFNDGQS3UK7EEA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.346539031 (198h32m10.042387794s) [noise=true , address=77.90.31.10:4519 , piece_id=MIHLKGHCRTUFQSGBIFKIWPNACAJ4CQOEYRDIKA4KJLFP356VMY6A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.419504032 (198h32m10.115352785s) [noise=true , address=77.90.31.19:4843 , piece_id=JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:39.113150499 (198h32m10.808999252s) [noise=true , address=45.91.116.6:4973 , piece_id=Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:39.113177150 (198h32m10.809025903s) [seg_pos={0 0}, upload=174963]: Segment reached optimal threshold of 13 pieces. Nov 7 18:12:39.113381018 (198h32m10.809229772s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113390166 (198h32m10.809238919s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113460199 (198h32m10.809308952s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113470077 (198h32m10.809318821s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113780338 (198h32m10.809629091s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113788243 (198h32m10.809636986s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113858657 (198h32m10.8097074s) [seg_pos={0 0}, upload=174963]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 7 18:12:39.114030194 (198h32m10.809878947s) [upload=174963]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 7 18:12:39.114038290 (198h32m10.809887043s) [upload=174963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 7 18:12:39.451448640 (198h32m11.147297403s) [upload=174963]: Done uploading segments... Nov 7 18:12:39.451473607 (198h32m11.14732236s) [upload=174963]: Upload finished. err: <nil> Duration: 6.282471072s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [..XXX.X..X.X.XXXX.X..XX.XXXXX.X.............................] duration: 9.185549035s trace: https://jaeger.dev.storj.io/trace/422f33ac8ae6a74a FAIL (2025-11-07 18:34:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ) context deadline exceeded; piecestore: (Node ID: 12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, Piece ID: JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ) context deadline exceeded ======LOGS====== Nov 7 18:12:35.791697270 (198h32m7.487546023s) [upload=174963]: Starting upload Nov 7 18:12:35.791791689 (198h32m7.487640452s) [upload=174963]: Done starting upload Nov 7 18:12:35.791885458 (198h32m7.487734211s) [upload=174963]: Uploading segments... Nov 7 18:12:35.792146455 (198h32m7.487995208s) [upload=174963]: Got next segment. Inline: false Nov 7 18:12:35.792195278 (198h32m7.488044021s) [upload=174963]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 7 18:12:35.792201820 (198h32m7.488050563s) [upload=174963]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 7 18:12:36.063867588 (198h32m7.759716341s) [seg_pos={0 0}, upload=174963]: Begin upload segment... Nov 7 18:12:36.063937832 (198h32m7.759786585s) [seg_pos={0 0}, upload=174963]: Done begin upload segment. Nov 7 18:12:36.063988689 (198h32m7.759837463s) [noise=true , address=145.79.227.27:4690 , piece_id=7CVNJTAEBVYJXRWNSFYFP3E7SKMY5XI4CV2JEZJLSXH4TA2F2JZQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.064083510 (198h32m7.759932263s) [noise=true , address=145.79.227.10:4403 , piece_id=JCMXNJ5PW5ULGOJZ5SGB423VW7VJPC5REYNSNWJZJXHGUQXUS3KA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.064896969 (198h32m7.760745762s) [noise=true , address=145.79.227.28:4726 , piece_id=D6DH5FHFB5ZTPANRPUWZKRYPFYUV2LS6AY5FQL5RBJZIUNM4FKLQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.065443520 (198h32m7.761292273s) [noise=true , address=77.90.31.11:4554 , piece_id=TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.065970974 (198h32m7.761819777s) [noise=true , address=145.79.227.8:4330 , piece_id=32XBCOEGSUTVUCCWVM2D7CB6FKMNLD6N2N4655B3ZYHNAIL4R2QQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.066422053 (198h32m7.762270806s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.067054427 (198h32m7.76290318s) [noise=true , address=77.90.31.10:4519 , piece_id=MIHLKGHCRTUFQSGBIFKIWPNACAJ4CQOEYRDIKA4KJLFP356VMY6A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.067645042 (198h32m7.763493795s) [noise=true , address=45.91.116.6:4973 , piece_id=Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.068379079 (198h32m7.764227842s) [noise=true , address=77.90.31.19:4843 , piece_id=JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.070931099 (198h32m7.766779862s) [noise=true , address=145.79.227.11:4732 , piece_id=VKSPYLTD6C6XBUFXXVVPHIPGFRCRNQK6EVI4SM7Y6ODOMVGGAJOQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.073100420 (198h32m7.768949173s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.075821872 (198h32m7.771670625s) [noise=true , address=45.91.116.4:4937 , piece_id=CWDA2IYNFUYHVDBJ74Y3CJKCZUBN4QYUEJYFKZNIBAUYYAXO3WFQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.078400863 (198h32m7.774249626s) [noise=true , address=45.91.116.10:4082 , piece_id=RW62GCPC56K2IMOFF276GQECXRKW6NY6W44TNAYFNDGQS3UK7EEA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.081230762 (198h32m7.777079515s) [noise=true , address=145.79.227.15:4876 , piece_id=ZPGKGNXRFEWRRJKTS7NKBEVQA7OAWNWYDR3DUTPHT3XVYRJSKWPQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.083298258 (198h32m7.779147012s) [noise=true , address=145.79.227.20:4438 , piece_id=V6WDUVDB7WHTDCTSYCFWS22UKRD5NMXGJ2GHLSFLVVOXZMZWLL4A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.084602091 (198h32m7.780450844s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:37.640211006 (198h32m9.336059759s) [upload=174963]: Next returned nil segment Nov 7 18:12:37.640227878 (198h32m9.336076621s) [upload=174963]: Waiting for error group managing segments... Nov 7 18:12:38.007509819 (198h32m9.703358572s) [noise=true , address=145.79.227.10:4403 , piece_id=JCMXNJ5PW5ULGOJZ5SGB423VW7VJPC5REYNSNWJZJXHGUQXUS3KA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.015182750 (198h32m9.711031503s) [noise=true , address=145.79.227.28:4726 , piece_id=D6DH5FHFB5ZTPANRPUWZKRYPFYUV2LS6AY5FQL5RBJZIUNM4FKLQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.020400886 (198h32m9.71624964s) [noise=true , address=145.79.227.27:4690 , piece_id=7CVNJTAEBVYJXRWNSFYFP3E7SKMY5XI4CV2JEZJLSXH4TA2F2JZQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.062361869 (198h32m9.758210622s) [noise=true , address=145.79.227.15:4876 , piece_id=ZPGKGNXRFEWRRJKTS7NKBEVQA7OAWNWYDR3DUTPHT3XVYRJSKWPQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.096654439 (198h32m9.792503192s) [noise=true , address=145.79.227.20:4438 , piece_id=V6WDUVDB7WHTDCTSYCFWS22UKRD5NMXGJ2GHLSFLVVOXZMZWLL4A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.244365682 (198h32m9.940214435s) [noise=true , address=145.79.227.8:4330 , piece_id=32XBCOEGSUTVUCCWVM2D7CB6FKMNLD6N2N4655B3ZYHNAIL4R2QQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.245114297 (198h32m9.94096305s) [noise=true , address=145.79.227.11:4732 , piece_id=VKSPYLTD6C6XBUFXXVVPHIPGFRCRNQK6EVI4SM7Y6ODOMVGGAJOQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.324356281 (198h32m10.020205034s) [noise=true , address=77.90.31.11:4554 , piece_id=TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.331696878 (198h32m10.027545621s) [noise=true , address=45.91.116.4:4937 , piece_id=CWDA2IYNFUYHVDBJ74Y3CJKCZUBN4QYUEJYFKZNIBAUYYAXO3WFQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.334142676 (198h32m10.029991429s) [noise=true , address=45.91.116.10:4082 , piece_id=RW62GCPC56K2IMOFF276GQECXRKW6NY6W44TNAYFNDGQS3UK7EEA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.346539031 (198h32m10.042387794s) [noise=true , address=77.90.31.10:4519 , piece_id=MIHLKGHCRTUFQSGBIFKIWPNACAJ4CQOEYRDIKA4KJLFP356VMY6A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.419504032 (198h32m10.115352785s) [noise=true , address=77.90.31.19:4843 , piece_id=JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:39.113150499 (198h32m10.808999252s) [noise=true , address=45.91.116.6:4973 , piece_id=Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:39.113177150 (198h32m10.809025903s) [seg_pos={0 0}, upload=174963]: Segment reached optimal threshold of 13 pieces. Nov 7 18:12:39.113381018 (198h32m10.809229772s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113390166 (198h32m10.809238919s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113460199 (198h32m10.809308952s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113470077 (198h32m10.809318821s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113780338 (198h32m10.809629091s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113788243 (198h32m10.809636986s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113858657 (198h32m10.8097074s) [seg_pos={0 0}, upload=174963]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 7 18:12:39.114030194 (198h32m10.809878947s) [upload=174963]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 7 18:12:39.114038290 (198h32m10.809887043s) [upload=174963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 7 18:12:39.451448640 (198h32m11.147297403s) [upload=174963]: Done uploading segments... Nov 7 18:12:39.451473607 (198h32m11.14732236s) [upload=174963]: Upload finished. err: <nil> Duration: 9.185582088s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [X..XXX.X..X.X.XXXX.X..XX.XXXXX.X............................] duration: 6.028437553s trace: https://jaeger.dev.storj.io/trace/422f33ac8ae6a74a FAIL (2025-11-07 18:34:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ) context deadline exceeded; piecestore: (Node ID: 12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, Piece ID: JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ) context deadline exceeded ======LOGS====== Nov 7 18:12:35.791697270 (198h32m7.487546023s) [upload=174963]: Starting upload Nov 7 18:12:35.791791689 (198h32m7.487640452s) [upload=174963]: Done starting upload Nov 7 18:12:35.791885458 (198h32m7.487734211s) [upload=174963]: Uploading segments... Nov 7 18:12:35.792146455 (198h32m7.487995208s) [upload=174963]: Got next segment. Inline: false Nov 7 18:12:35.792195278 (198h32m7.488044021s) [upload=174963]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 7 18:12:35.792201820 (198h32m7.488050563s) [upload=174963]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 7 18:12:36.063867588 (198h32m7.759716341s) [seg_pos={0 0}, upload=174963]: Begin upload segment... Nov 7 18:12:36.063937832 (198h32m7.759786585s) [seg_pos={0 0}, upload=174963]: Done begin upload segment. Nov 7 18:12:36.063988689 (198h32m7.759837463s) [noise=true , address=145.79.227.27:4690 , piece_id=7CVNJTAEBVYJXRWNSFYFP3E7SKMY5XI4CV2JEZJLSXH4TA2F2JZQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.064083510 (198h32m7.759932263s) [noise=true , address=145.79.227.10:4403 , piece_id=JCMXNJ5PW5ULGOJZ5SGB423VW7VJPC5REYNSNWJZJXHGUQXUS3KA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.064896969 (198h32m7.760745762s) [noise=true , address=145.79.227.28:4726 , piece_id=D6DH5FHFB5ZTPANRPUWZKRYPFYUV2LS6AY5FQL5RBJZIUNM4FKLQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.065443520 (198h32m7.761292273s) [noise=true , address=77.90.31.11:4554 , piece_id=TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.065970974 (198h32m7.761819777s) [noise=true , address=145.79.227.8:4330 , piece_id=32XBCOEGSUTVUCCWVM2D7CB6FKMNLD6N2N4655B3ZYHNAIL4R2QQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.066422053 (198h32m7.762270806s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.067054427 (198h32m7.76290318s) [noise=true , address=77.90.31.10:4519 , piece_id=MIHLKGHCRTUFQSGBIFKIWPNACAJ4CQOEYRDIKA4KJLFP356VMY6A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.067645042 (198h32m7.763493795s) [noise=true , address=45.91.116.6:4973 , piece_id=Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.068379079 (198h32m7.764227842s) [noise=true , address=77.90.31.19:4843 , piece_id=JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.070931099 (198h32m7.766779862s) [noise=true , address=145.79.227.11:4732 , piece_id=VKSPYLTD6C6XBUFXXVVPHIPGFRCRNQK6EVI4SM7Y6ODOMVGGAJOQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.073100420 (198h32m7.768949173s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.075821872 (198h32m7.771670625s) [noise=true , address=45.91.116.4:4937 , piece_id=CWDA2IYNFUYHVDBJ74Y3CJKCZUBN4QYUEJYFKZNIBAUYYAXO3WFQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.078400863 (198h32m7.774249626s) [noise=true , address=45.91.116.10:4082 , piece_id=RW62GCPC56K2IMOFF276GQECXRKW6NY6W44TNAYFNDGQS3UK7EEA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.081230762 (198h32m7.777079515s) [noise=true , address=145.79.227.15:4876 , piece_id=ZPGKGNXRFEWRRJKTS7NKBEVQA7OAWNWYDR3DUTPHT3XVYRJSKWPQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.083298258 (198h32m7.779147012s) [noise=true , address=145.79.227.20:4438 , piece_id=V6WDUVDB7WHTDCTSYCFWS22UKRD5NMXGJ2GHLSFLVVOXZMZWLL4A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.084602091 (198h32m7.780450844s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:37.640211006 (198h32m9.336059759s) [upload=174963]: Next returned nil segment Nov 7 18:12:37.640227878 (198h32m9.336076621s) [upload=174963]: Waiting for error group managing segments... Nov 7 18:12:38.007509819 (198h32m9.703358572s) [noise=true , address=145.79.227.10:4403 , piece_id=JCMXNJ5PW5ULGOJZ5SGB423VW7VJPC5REYNSNWJZJXHGUQXUS3KA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.015182750 (198h32m9.711031503s) [noise=true , address=145.79.227.28:4726 , piece_id=D6DH5FHFB5ZTPANRPUWZKRYPFYUV2LS6AY5FQL5RBJZIUNM4FKLQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.020400886 (198h32m9.71624964s) [noise=true , address=145.79.227.27:4690 , piece_id=7CVNJTAEBVYJXRWNSFYFP3E7SKMY5XI4CV2JEZJLSXH4TA2F2JZQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.062361869 (198h32m9.758210622s) [noise=true , address=145.79.227.15:4876 , piece_id=ZPGKGNXRFEWRRJKTS7NKBEVQA7OAWNWYDR3DUTPHT3XVYRJSKWPQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.096654439 (198h32m9.792503192s) [noise=true , address=145.79.227.20:4438 , piece_id=V6WDUVDB7WHTDCTSYCFWS22UKRD5NMXGJ2GHLSFLVVOXZMZWLL4A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.244365682 (198h32m9.940214435s) [noise=true , address=145.79.227.8:4330 , piece_id=32XBCOEGSUTVUCCWVM2D7CB6FKMNLD6N2N4655B3ZYHNAIL4R2QQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.245114297 (198h32m9.94096305s) [noise=true , address=145.79.227.11:4732 , piece_id=VKSPYLTD6C6XBUFXXVVPHIPGFRCRNQK6EVI4SM7Y6ODOMVGGAJOQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.324356281 (198h32m10.020205034s) [noise=true , address=77.90.31.11:4554 , piece_id=TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.331696878 (198h32m10.027545621s) [noise=true , address=45.91.116.4:4937 , piece_id=CWDA2IYNFUYHVDBJ74Y3CJKCZUBN4QYUEJYFKZNIBAUYYAXO3WFQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.334142676 (198h32m10.029991429s) [noise=true , address=45.91.116.10:4082 , piece_id=RW62GCPC56K2IMOFF276GQECXRKW6NY6W44TNAYFNDGQS3UK7EEA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.346539031 (198h32m10.042387794s) [noise=true , address=77.90.31.10:4519 , piece_id=MIHLKGHCRTUFQSGBIFKIWPNACAJ4CQOEYRDIKA4KJLFP356VMY6A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.419504032 (198h32m10.115352785s) [noise=true , address=77.90.31.19:4843 , piece_id=JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:39.113150499 (198h32m10.808999252s) [noise=true , address=45.91.116.6:4973 , piece_id=Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:39.113177150 (198h32m10.809025903s) [seg_pos={0 0}, upload=174963]: Segment reached optimal threshold of 13 pieces. Nov 7 18:12:39.113381018 (198h32m10.809229772s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113390166 (198h32m10.809238919s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113460199 (198h32m10.809308952s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113470077 (198h32m10.809318821s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113780338 (198h32m10.809629091s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113788243 (198h32m10.809636986s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113858657 (198h32m10.8097074s) [seg_pos={0 0}, upload=174963]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 7 18:12:39.114030194 (198h32m10.809878947s) [upload=174963]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 7 18:12:39.114038290 (198h32m10.809887043s) [upload=174963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 7 18:12:39.451448640 (198h32m11.147297403s) [upload=174963]: Done uploading segments... Nov 7 18:12:39.451473607 (198h32m11.14732236s) [upload=174963]: Upload finished. err: <nil> Duration: 6.028470145s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [.X..XXX.X..X.X.XXXX.X..XX.XXXXX.X...........................] duration: 6.130089856s trace: https://jaeger.dev.storj.io/trace/422f33ac8ae6a74a FAIL (2025-11-07 18:33:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ) context deadline exceeded; piecestore: (Node ID: 12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, Piece ID: JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ) context deadline exceeded ======LOGS====== Nov 7 18:12:35.791697270 (198h32m7.487546023s) [upload=174963]: Starting upload Nov 7 18:12:35.791791689 (198h32m7.487640452s) [upload=174963]: Done starting upload Nov 7 18:12:35.791885458 (198h32m7.487734211s) [upload=174963]: Uploading segments... Nov 7 18:12:35.792146455 (198h32m7.487995208s) [upload=174963]: Got next segment. Inline: false Nov 7 18:12:35.792195278 (198h32m7.488044021s) [upload=174963]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 7 18:12:35.792201820 (198h32m7.488050563s) [upload=174963]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 7 18:12:36.063867588 (198h32m7.759716341s) [seg_pos={0 0}, upload=174963]: Begin upload segment... Nov 7 18:12:36.063937832 (198h32m7.759786585s) [seg_pos={0 0}, upload=174963]: Done begin upload segment. Nov 7 18:12:36.063988689 (198h32m7.759837463s) [noise=true , address=145.79.227.27:4690 , piece_id=7CVNJTAEBVYJXRWNSFYFP3E7SKMY5XI4CV2JEZJLSXH4TA2F2JZQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.064083510 (198h32m7.759932263s) [noise=true , address=145.79.227.10:4403 , piece_id=JCMXNJ5PW5ULGOJZ5SGB423VW7VJPC5REYNSNWJZJXHGUQXUS3KA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.064896969 (198h32m7.760745762s) [noise=true , address=145.79.227.28:4726 , piece_id=D6DH5FHFB5ZTPANRPUWZKRYPFYUV2LS6AY5FQL5RBJZIUNM4FKLQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.065443520 (198h32m7.761292273s) [noise=true , address=77.90.31.11:4554 , piece_id=TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.065970974 (198h32m7.761819777s) [noise=true , address=145.79.227.8:4330 , piece_id=32XBCOEGSUTVUCCWVM2D7CB6FKMNLD6N2N4655B3ZYHNAIL4R2QQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.066422053 (198h32m7.762270806s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.067054427 (198h32m7.76290318s) [noise=true , address=77.90.31.10:4519 , piece_id=MIHLKGHCRTUFQSGBIFKIWPNACAJ4CQOEYRDIKA4KJLFP356VMY6A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.067645042 (198h32m7.763493795s) [noise=true , address=45.91.116.6:4973 , piece_id=Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.068379079 (198h32m7.764227842s) [noise=true , address=77.90.31.19:4843 , piece_id=JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.070931099 (198h32m7.766779862s) [noise=true , address=145.79.227.11:4732 , piece_id=VKSPYLTD6C6XBUFXXVVPHIPGFRCRNQK6EVI4SM7Y6ODOMVGGAJOQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.073100420 (198h32m7.768949173s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.075821872 (198h32m7.771670625s) [noise=true , address=45.91.116.4:4937 , piece_id=CWDA2IYNFUYHVDBJ74Y3CJKCZUBN4QYUEJYFKZNIBAUYYAXO3WFQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.078400863 (198h32m7.774249626s) [noise=true , address=45.91.116.10:4082 , piece_id=RW62GCPC56K2IMOFF276GQECXRKW6NY6W44TNAYFNDGQS3UK7EEA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.081230762 (198h32m7.777079515s) [noise=true , address=145.79.227.15:4876 , piece_id=ZPGKGNXRFEWRRJKTS7NKBEVQA7OAWNWYDR3DUTPHT3XVYRJSKWPQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.083298258 (198h32m7.779147012s) [noise=true , address=145.79.227.20:4438 , piece_id=V6WDUVDB7WHTDCTSYCFWS22UKRD5NMXGJ2GHLSFLVVOXZMZWLL4A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.084602091 (198h32m7.780450844s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:37.640211006 (198h32m9.336059759s) [upload=174963]: Next returned nil segment Nov 7 18:12:37.640227878 (198h32m9.336076621s) [upload=174963]: Waiting for error group managing segments... Nov 7 18:12:38.007509819 (198h32m9.703358572s) [noise=true , address=145.79.227.10:4403 , piece_id=JCMXNJ5PW5ULGOJZ5SGB423VW7VJPC5REYNSNWJZJXHGUQXUS3KA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.015182750 (198h32m9.711031503s) [noise=true , address=145.79.227.28:4726 , piece_id=D6DH5FHFB5ZTPANRPUWZKRYPFYUV2LS6AY5FQL5RBJZIUNM4FKLQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.020400886 (198h32m9.71624964s) [noise=true , address=145.79.227.27:4690 , piece_id=7CVNJTAEBVYJXRWNSFYFP3E7SKMY5XI4CV2JEZJLSXH4TA2F2JZQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.062361869 (198h32m9.758210622s) [noise=true , address=145.79.227.15:4876 , piece_id=ZPGKGNXRFEWRRJKTS7NKBEVQA7OAWNWYDR3DUTPHT3XVYRJSKWPQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.096654439 (198h32m9.792503192s) [noise=true , address=145.79.227.20:4438 , piece_id=V6WDUVDB7WHTDCTSYCFWS22UKRD5NMXGJ2GHLSFLVVOXZMZWLL4A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.244365682 (198h32m9.940214435s) [noise=true , address=145.79.227.8:4330 , piece_id=32XBCOEGSUTVUCCWVM2D7CB6FKMNLD6N2N4655B3ZYHNAIL4R2QQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.245114297 (198h32m9.94096305s) [noise=true , address=145.79.227.11:4732 , piece_id=VKSPYLTD6C6XBUFXXVVPHIPGFRCRNQK6EVI4SM7Y6ODOMVGGAJOQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.324356281 (198h32m10.020205034s) [noise=true , address=77.90.31.11:4554 , piece_id=TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.331696878 (198h32m10.027545621s) [noise=true , address=45.91.116.4:4937 , piece_id=CWDA2IYNFUYHVDBJ74Y3CJKCZUBN4QYUEJYFKZNIBAUYYAXO3WFQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.334142676 (198h32m10.029991429s) [noise=true , address=45.91.116.10:4082 , piece_id=RW62GCPC56K2IMOFF276GQECXRKW6NY6W44TNAYFNDGQS3UK7EEA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.346539031 (198h32m10.042387794s) [noise=true , address=77.90.31.10:4519 , piece_id=MIHLKGHCRTUFQSGBIFKIWPNACAJ4CQOEYRDIKA4KJLFP356VMY6A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.419504032 (198h32m10.115352785s) [noise=true , address=77.90.31.19:4843 , piece_id=JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:39.113150499 (198h32m10.808999252s) [noise=true , address=45.91.116.6:4973 , piece_id=Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:39.113177150 (198h32m10.809025903s) [seg_pos={0 0}, upload=174963]: Segment reached optimal threshold of 13 pieces. Nov 7 18:12:39.113381018 (198h32m10.809229772s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113390166 (198h32m10.809238919s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113460199 (198h32m10.809308952s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113470077 (198h32m10.809318821s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113780338 (198h32m10.809629091s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113788243 (198h32m10.809636986s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113858657 (198h32m10.8097074s) [seg_pos={0 0}, upload=174963]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 7 18:12:39.114030194 (198h32m10.809878947s) [upload=174963]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 7 18:12:39.114038290 (198h32m10.809887043s) [upload=174963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 7 18:12:39.451448640 (198h32m11.147297403s) [upload=174963]: Done uploading segments... Nov 7 18:12:39.451473607 (198h32m11.14732236s) [upload=174963]: Upload finished. err: <nil> Duration: 6.130121946s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [X.X..XXX.X..X.X.XXXX.X..XX.XXXXX.X..........................] duration: 6.742829034s trace: https://jaeger.dev.storj.io/trace/422f33ac8ae6a74a FAIL (2025-11-07 18:33:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ) context deadline exceeded; piecestore: (Node ID: 12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, Piece ID: JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ) context deadline exceeded ======LOGS====== Nov 7 18:12:35.791697270 (198h32m7.487546023s) [upload=174963]: Starting upload Nov 7 18:12:35.791791689 (198h32m7.487640452s) [upload=174963]: Done starting upload Nov 7 18:12:35.791885458 (198h32m7.487734211s) [upload=174963]: Uploading segments... Nov 7 18:12:35.792146455 (198h32m7.487995208s) [upload=174963]: Got next segment. Inline: false Nov 7 18:12:35.792195278 (198h32m7.488044021s) [upload=174963]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 7 18:12:35.792201820 (198h32m7.488050563s) [upload=174963]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 7 18:12:36.063867588 (198h32m7.759716341s) [seg_pos={0 0}, upload=174963]: Begin upload segment... Nov 7 18:12:36.063937832 (198h32m7.759786585s) [seg_pos={0 0}, upload=174963]: Done begin upload segment. Nov 7 18:12:36.063988689 (198h32m7.759837463s) [noise=true , address=145.79.227.27:4690 , piece_id=7CVNJTAEBVYJXRWNSFYFP3E7SKMY5XI4CV2JEZJLSXH4TA2F2JZQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.064083510 (198h32m7.759932263s) [noise=true , address=145.79.227.10:4403 , piece_id=JCMXNJ5PW5ULGOJZ5SGB423VW7VJPC5REYNSNWJZJXHGUQXUS3KA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.064896969 (198h32m7.760745762s) [noise=true , address=145.79.227.28:4726 , piece_id=D6DH5FHFB5ZTPANRPUWZKRYPFYUV2LS6AY5FQL5RBJZIUNM4FKLQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.065443520 (198h32m7.761292273s) [noise=true , address=77.90.31.11:4554 , piece_id=TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.065970974 (198h32m7.761819777s) [noise=true , address=145.79.227.8:4330 , piece_id=32XBCOEGSUTVUCCWVM2D7CB6FKMNLD6N2N4655B3ZYHNAIL4R2QQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.066422053 (198h32m7.762270806s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.067054427 (198h32m7.76290318s) [noise=true , address=77.90.31.10:4519 , piece_id=MIHLKGHCRTUFQSGBIFKIWPNACAJ4CQOEYRDIKA4KJLFP356VMY6A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.067645042 (198h32m7.763493795s) [noise=true , address=45.91.116.6:4973 , piece_id=Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.068379079 (198h32m7.764227842s) [noise=true , address=77.90.31.19:4843 , piece_id=JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.070931099 (198h32m7.766779862s) [noise=true , address=145.79.227.11:4732 , piece_id=VKSPYLTD6C6XBUFXXVVPHIPGFRCRNQK6EVI4SM7Y6ODOMVGGAJOQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.073100420 (198h32m7.768949173s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.075821872 (198h32m7.771670625s) [noise=true , address=45.91.116.4:4937 , piece_id=CWDA2IYNFUYHVDBJ74Y3CJKCZUBN4QYUEJYFKZNIBAUYYAXO3WFQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.078400863 (198h32m7.774249626s) [noise=true , address=45.91.116.10:4082 , piece_id=RW62GCPC56K2IMOFF276GQECXRKW6NY6W44TNAYFNDGQS3UK7EEA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.081230762 (198h32m7.777079515s) [noise=true , address=145.79.227.15:4876 , piece_id=ZPGKGNXRFEWRRJKTS7NKBEVQA7OAWNWYDR3DUTPHT3XVYRJSKWPQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.083298258 (198h32m7.779147012s) [noise=true , address=145.79.227.20:4438 , piece_id=V6WDUVDB7WHTDCTSYCFWS22UKRD5NMXGJ2GHLSFLVVOXZMZWLL4A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.084602091 (198h32m7.780450844s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:37.640211006 (198h32m9.336059759s) [upload=174963]: Next returned nil segment Nov 7 18:12:37.640227878 (198h32m9.336076621s) [upload=174963]: Waiting for error group managing segments... Nov 7 18:12:38.007509819 (198h32m9.703358572s) [noise=true , address=145.79.227.10:4403 , piece_id=JCMXNJ5PW5ULGOJZ5SGB423VW7VJPC5REYNSNWJZJXHGUQXUS3KA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.015182750 (198h32m9.711031503s) [noise=true , address=145.79.227.28:4726 , piece_id=D6DH5FHFB5ZTPANRPUWZKRYPFYUV2LS6AY5FQL5RBJZIUNM4FKLQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.020400886 (198h32m9.71624964s) [noise=true , address=145.79.227.27:4690 , piece_id=7CVNJTAEBVYJXRWNSFYFP3E7SKMY5XI4CV2JEZJLSXH4TA2F2JZQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.062361869 (198h32m9.758210622s) [noise=true , address=145.79.227.15:4876 , piece_id=ZPGKGNXRFEWRRJKTS7NKBEVQA7OAWNWYDR3DUTPHT3XVYRJSKWPQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.096654439 (198h32m9.792503192s) [noise=true , address=145.79.227.20:4438 , piece_id=V6WDUVDB7WHTDCTSYCFWS22UKRD5NMXGJ2GHLSFLVVOXZMZWLL4A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.244365682 (198h32m9.940214435s) [noise=true , address=145.79.227.8:4330 , piece_id=32XBCOEGSUTVUCCWVM2D7CB6FKMNLD6N2N4655B3ZYHNAIL4R2QQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.245114297 (198h32m9.94096305s) [noise=true , address=145.79.227.11:4732 , piece_id=VKSPYLTD6C6XBUFXXVVPHIPGFRCRNQK6EVI4SM7Y6ODOMVGGAJOQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.324356281 (198h32m10.020205034s) [noise=true , address=77.90.31.11:4554 , piece_id=TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.331696878 (198h32m10.027545621s) [noise=true , address=45.91.116.4:4937 , piece_id=CWDA2IYNFUYHVDBJ74Y3CJKCZUBN4QYUEJYFKZNIBAUYYAXO3WFQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.334142676 (198h32m10.029991429s) [noise=true , address=45.91.116.10:4082 , piece_id=RW62GCPC56K2IMOFF276GQECXRKW6NY6W44TNAYFNDGQS3UK7EEA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.346539031 (198h32m10.042387794s) [noise=true , address=77.90.31.10:4519 , piece_id=MIHLKGHCRTUFQSGBIFKIWPNACAJ4CQOEYRDIKA4KJLFP356VMY6A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.419504032 (198h32m10.115352785s) [noise=true , address=77.90.31.19:4843 , piece_id=JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:39.113150499 (198h32m10.808999252s) [noise=true , address=45.91.116.6:4973 , piece_id=Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:39.113177150 (198h32m10.809025903s) [seg_pos={0 0}, upload=174963]: Segment reached optimal threshold of 13 pieces. Nov 7 18:12:39.113381018 (198h32m10.809229772s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113390166 (198h32m10.809238919s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113460199 (198h32m10.809308952s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113470077 (198h32m10.809318821s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113780338 (198h32m10.809629091s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113788243 (198h32m10.809636986s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113858657 (198h32m10.8097074s) [seg_pos={0 0}, upload=174963]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 7 18:12:39.114030194 (198h32m10.809878947s) [upload=174963]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 7 18:12:39.114038290 (198h32m10.809887043s) [upload=174963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 7 18:12:39.451448640 (198h32m11.147297403s) [upload=174963]: Done uploading segments... Nov 7 18:12:39.451473607 (198h32m11.14732236s) [upload=174963]: Upload finished. err: <nil> Duration: 6.742872197s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [XX.X..XXX.X..X.X.XXXX.X..XX.XXXXX.X.........................] duration: 6.041569406s trace: https://jaeger.dev.storj.io/trace/422f33ac8ae6a74a FAIL (2025-11-07 18:32:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ) context deadline exceeded; piecestore: (Node ID: 12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, Piece ID: JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ) context deadline exceeded ======LOGS====== Nov 7 18:12:35.791697270 (198h32m7.487546023s) [upload=174963]: Starting upload Nov 7 18:12:35.791791689 (198h32m7.487640452s) [upload=174963]: Done starting upload Nov 7 18:12:35.791885458 (198h32m7.487734211s) [upload=174963]: Uploading segments... Nov 7 18:12:35.792146455 (198h32m7.487995208s) [upload=174963]: Got next segment. Inline: false Nov 7 18:12:35.792195278 (198h32m7.488044021s) [upload=174963]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 7 18:12:35.792201820 (198h32m7.488050563s) [upload=174963]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 7 18:12:36.063867588 (198h32m7.759716341s) [seg_pos={0 0}, upload=174963]: Begin upload segment... Nov 7 18:12:36.063937832 (198h32m7.759786585s) [seg_pos={0 0}, upload=174963]: Done begin upload segment. Nov 7 18:12:36.063988689 (198h32m7.759837463s) [noise=true , address=145.79.227.27:4690 , piece_id=7CVNJTAEBVYJXRWNSFYFP3E7SKMY5XI4CV2JEZJLSXH4TA2F2JZQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.064083510 (198h32m7.759932263s) [noise=true , address=145.79.227.10:4403 , piece_id=JCMXNJ5PW5ULGOJZ5SGB423VW7VJPC5REYNSNWJZJXHGUQXUS3KA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.064896969 (198h32m7.760745762s) [noise=true , address=145.79.227.28:4726 , piece_id=D6DH5FHFB5ZTPANRPUWZKRYPFYUV2LS6AY5FQL5RBJZIUNM4FKLQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.065443520 (198h32m7.761292273s) [noise=true , address=77.90.31.11:4554 , piece_id=TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.065970974 (198h32m7.761819777s) [noise=true , address=145.79.227.8:4330 , piece_id=32XBCOEGSUTVUCCWVM2D7CB6FKMNLD6N2N4655B3ZYHNAIL4R2QQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.066422053 (198h32m7.762270806s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.067054427 (198h32m7.76290318s) [noise=true , address=77.90.31.10:4519 , piece_id=MIHLKGHCRTUFQSGBIFKIWPNACAJ4CQOEYRDIKA4KJLFP356VMY6A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.067645042 (198h32m7.763493795s) [noise=true , address=45.91.116.6:4973 , piece_id=Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.068379079 (198h32m7.764227842s) [noise=true , address=77.90.31.19:4843 , piece_id=JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.070931099 (198h32m7.766779862s) [noise=true , address=145.79.227.11:4732 , piece_id=VKSPYLTD6C6XBUFXXVVPHIPGFRCRNQK6EVI4SM7Y6ODOMVGGAJOQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.073100420 (198h32m7.768949173s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.075821872 (198h32m7.771670625s) [noise=true , address=45.91.116.4:4937 , piece_id=CWDA2IYNFUYHVDBJ74Y3CJKCZUBN4QYUEJYFKZNIBAUYYAXO3WFQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.078400863 (198h32m7.774249626s) [noise=true , address=45.91.116.10:4082 , piece_id=RW62GCPC56K2IMOFF276GQECXRKW6NY6W44TNAYFNDGQS3UK7EEA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.081230762 (198h32m7.777079515s) [noise=true , address=145.79.227.15:4876 , piece_id=ZPGKGNXRFEWRRJKTS7NKBEVQA7OAWNWYDR3DUTPHT3XVYRJSKWPQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.083298258 (198h32m7.779147012s) [noise=true , address=145.79.227.20:4438 , piece_id=V6WDUVDB7WHTDCTSYCFWS22UKRD5NMXGJ2GHLSFLVVOXZMZWLL4A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.084602091 (198h32m7.780450844s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:37.640211006 (198h32m9.336059759s) [upload=174963]: Next returned nil segment Nov 7 18:12:37.640227878 (198h32m9.336076621s) [upload=174963]: Waiting for error group managing segments... Nov 7 18:12:38.007509819 (198h32m9.703358572s) [noise=true , address=145.79.227.10:4403 , piece_id=JCMXNJ5PW5ULGOJZ5SGB423VW7VJPC5REYNSNWJZJXHGUQXUS3KA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.015182750 (198h32m9.711031503s) [noise=true , address=145.79.227.28:4726 , piece_id=D6DH5FHFB5ZTPANRPUWZKRYPFYUV2LS6AY5FQL5RBJZIUNM4FKLQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.020400886 (198h32m9.71624964s) [noise=true , address=145.79.227.27:4690 , piece_id=7CVNJTAEBVYJXRWNSFYFP3E7SKMY5XI4CV2JEZJLSXH4TA2F2JZQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.062361869 (198h32m9.758210622s) [noise=true , address=145.79.227.15:4876 , piece_id=ZPGKGNXRFEWRRJKTS7NKBEVQA7OAWNWYDR3DUTPHT3XVYRJSKWPQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.096654439 (198h32m9.792503192s) [noise=true , address=145.79.227.20:4438 , piece_id=V6WDUVDB7WHTDCTSYCFWS22UKRD5NMXGJ2GHLSFLVVOXZMZWLL4A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.244365682 (198h32m9.940214435s) [noise=true , address=145.79.227.8:4330 , piece_id=32XBCOEGSUTVUCCWVM2D7CB6FKMNLD6N2N4655B3ZYHNAIL4R2QQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.245114297 (198h32m9.94096305s) [noise=true , address=145.79.227.11:4732 , piece_id=VKSPYLTD6C6XBUFXXVVPHIPGFRCRNQK6EVI4SM7Y6ODOMVGGAJOQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.324356281 (198h32m10.020205034s) [noise=true , address=77.90.31.11:4554 , piece_id=TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.331696878 (198h32m10.027545621s) [noise=true , address=45.91.116.4:4937 , piece_id=CWDA2IYNFUYHVDBJ74Y3CJKCZUBN4QYUEJYFKZNIBAUYYAXO3WFQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.334142676 (198h32m10.029991429s) [noise=true , address=45.91.116.10:4082 , piece_id=RW62GCPC56K2IMOFF276GQECXRKW6NY6W44TNAYFNDGQS3UK7EEA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.346539031 (198h32m10.042387794s) [noise=true , address=77.90.31.10:4519 , piece_id=MIHLKGHCRTUFQSGBIFKIWPNACAJ4CQOEYRDIKA4KJLFP356VMY6A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.419504032 (198h32m10.115352785s) [noise=true , address=77.90.31.19:4843 , piece_id=JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:39.113150499 (198h32m10.808999252s) [noise=true , address=45.91.116.6:4973 , piece_id=Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:39.113177150 (198h32m10.809025903s) [seg_pos={0 0}, upload=174963]: Segment reached optimal threshold of 13 pieces. Nov 7 18:12:39.113381018 (198h32m10.809229772s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113390166 (198h32m10.809238919s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113460199 (198h32m10.809308952s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113470077 (198h32m10.809318821s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113780338 (198h32m10.809629091s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113788243 (198h32m10.809636986s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113858657 (198h32m10.8097074s) [seg_pos={0 0}, upload=174963]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 7 18:12:39.114030194 (198h32m10.809878947s) [upload=174963]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 7 18:12:39.114038290 (198h32m10.809887043s) [upload=174963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 7 18:12:39.451448640 (198h32m11.147297403s) [upload=174963]: Done uploading segments... Nov 7 18:12:39.451473607 (198h32m11.14732236s) [upload=174963]: Upload finished. err: <nil> Duration: 6.041598201s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [XXX.X..XXX.X..X.X.XXXX.X..XX.XXXXX.X........................] duration: 6.594752194s trace: https://jaeger.dev.storj.io/trace/422f33ac8ae6a74a FAIL (2025-11-07 18:31:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ) context deadline exceeded; piecestore: (Node ID: 12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, Piece ID: JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ) context deadline exceeded ======LOGS====== Nov 7 18:12:35.791697270 (198h32m7.487546023s) [upload=174963]: Starting upload Nov 7 18:12:35.791791689 (198h32m7.487640452s) [upload=174963]: Done starting upload Nov 7 18:12:35.791885458 (198h32m7.487734211s) [upload=174963]: Uploading segments... Nov 7 18:12:35.792146455 (198h32m7.487995208s) [upload=174963]: Got next segment. Inline: false Nov 7 18:12:35.792195278 (198h32m7.488044021s) [upload=174963]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 7 18:12:35.792201820 (198h32m7.488050563s) [upload=174963]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 7 18:12:36.063867588 (198h32m7.759716341s) [seg_pos={0 0}, upload=174963]: Begin upload segment... Nov 7 18:12:36.063937832 (198h32m7.759786585s) [seg_pos={0 0}, upload=174963]: Done begin upload segment. Nov 7 18:12:36.063988689 (198h32m7.759837463s) [noise=true , address=145.79.227.27:4690 , piece_id=7CVNJTAEBVYJXRWNSFYFP3E7SKMY5XI4CV2JEZJLSXH4TA2F2JZQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.064083510 (198h32m7.759932263s) [noise=true , address=145.79.227.10:4403 , piece_id=JCMXNJ5PW5ULGOJZ5SGB423VW7VJPC5REYNSNWJZJXHGUQXUS3KA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.064896969 (198h32m7.760745762s) [noise=true , address=145.79.227.28:4726 , piece_id=D6DH5FHFB5ZTPANRPUWZKRYPFYUV2LS6AY5FQL5RBJZIUNM4FKLQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.065443520 (198h32m7.761292273s) [noise=true , address=77.90.31.11:4554 , piece_id=TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.065970974 (198h32m7.761819777s) [noise=true , address=145.79.227.8:4330 , piece_id=32XBCOEGSUTVUCCWVM2D7CB6FKMNLD6N2N4655B3ZYHNAIL4R2QQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.066422053 (198h32m7.762270806s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.067054427 (198h32m7.76290318s) [noise=true , address=77.90.31.10:4519 , piece_id=MIHLKGHCRTUFQSGBIFKIWPNACAJ4CQOEYRDIKA4KJLFP356VMY6A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.067645042 (198h32m7.763493795s) [noise=true , address=45.91.116.6:4973 , piece_id=Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.068379079 (198h32m7.764227842s) [noise=true , address=77.90.31.19:4843 , piece_id=JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.070931099 (198h32m7.766779862s) [noise=true , address=145.79.227.11:4732 , piece_id=VKSPYLTD6C6XBUFXXVVPHIPGFRCRNQK6EVI4SM7Y6ODOMVGGAJOQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.073100420 (198h32m7.768949173s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.075821872 (198h32m7.771670625s) [noise=true , address=45.91.116.4:4937 , piece_id=CWDA2IYNFUYHVDBJ74Y3CJKCZUBN4QYUEJYFKZNIBAUYYAXO3WFQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.078400863 (198h32m7.774249626s) [noise=true , address=45.91.116.10:4082 , piece_id=RW62GCPC56K2IMOFF276GQECXRKW6NY6W44TNAYFNDGQS3UK7EEA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.081230762 (198h32m7.777079515s) [noise=true , address=145.79.227.15:4876 , piece_id=ZPGKGNXRFEWRRJKTS7NKBEVQA7OAWNWYDR3DUTPHT3XVYRJSKWPQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.083298258 (198h32m7.779147012s) [noise=true , address=145.79.227.20:4438 , piece_id=V6WDUVDB7WHTDCTSYCFWS22UKRD5NMXGJ2GHLSFLVVOXZMZWLL4A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.084602091 (198h32m7.780450844s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:37.640211006 (198h32m9.336059759s) [upload=174963]: Next returned nil segment Nov 7 18:12:37.640227878 (198h32m9.336076621s) [upload=174963]: Waiting for error group managing segments... Nov 7 18:12:38.007509819 (198h32m9.703358572s) [noise=true , address=145.79.227.10:4403 , piece_id=JCMXNJ5PW5ULGOJZ5SGB423VW7VJPC5REYNSNWJZJXHGUQXUS3KA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.015182750 (198h32m9.711031503s) [noise=true , address=145.79.227.28:4726 , piece_id=D6DH5FHFB5ZTPANRPUWZKRYPFYUV2LS6AY5FQL5RBJZIUNM4FKLQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.020400886 (198h32m9.71624964s) [noise=true , address=145.79.227.27:4690 , piece_id=7CVNJTAEBVYJXRWNSFYFP3E7SKMY5XI4CV2JEZJLSXH4TA2F2JZQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.062361869 (198h32m9.758210622s) [noise=true , address=145.79.227.15:4876 , piece_id=ZPGKGNXRFEWRRJKTS7NKBEVQA7OAWNWYDR3DUTPHT3XVYRJSKWPQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.096654439 (198h32m9.792503192s) [noise=true , address=145.79.227.20:4438 , piece_id=V6WDUVDB7WHTDCTSYCFWS22UKRD5NMXGJ2GHLSFLVVOXZMZWLL4A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.244365682 (198h32m9.940214435s) [noise=true , address=145.79.227.8:4330 , piece_id=32XBCOEGSUTVUCCWVM2D7CB6FKMNLD6N2N4655B3ZYHNAIL4R2QQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.245114297 (198h32m9.94096305s) [noise=true , address=145.79.227.11:4732 , piece_id=VKSPYLTD6C6XBUFXXVVPHIPGFRCRNQK6EVI4SM7Y6ODOMVGGAJOQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.324356281 (198h32m10.020205034s) [noise=true , address=77.90.31.11:4554 , piece_id=TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.331696878 (198h32m10.027545621s) [noise=true , address=45.91.116.4:4937 , piece_id=CWDA2IYNFUYHVDBJ74Y3CJKCZUBN4QYUEJYFKZNIBAUYYAXO3WFQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.334142676 (198h32m10.029991429s) [noise=true , address=45.91.116.10:4082 , piece_id=RW62GCPC56K2IMOFF276GQECXRKW6NY6W44TNAYFNDGQS3UK7EEA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.346539031 (198h32m10.042387794s) [noise=true , address=77.90.31.10:4519 , piece_id=MIHLKGHCRTUFQSGBIFKIWPNACAJ4CQOEYRDIKA4KJLFP356VMY6A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.419504032 (198h32m10.115352785s) [noise=true , address=77.90.31.19:4843 , piece_id=JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:39.113150499 (198h32m10.808999252s) [noise=true , address=45.91.116.6:4973 , piece_id=Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:39.113177150 (198h32m10.809025903s) [seg_pos={0 0}, upload=174963]: Segment reached optimal threshold of 13 pieces. Nov 7 18:12:39.113381018 (198h32m10.809229772s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113390166 (198h32m10.809238919s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113460199 (198h32m10.809308952s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113470077 (198h32m10.809318821s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113780338 (198h32m10.809629091s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113788243 (198h32m10.809636986s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113858657 (198h32m10.8097074s) [seg_pos={0 0}, upload=174963]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 7 18:12:39.114030194 (198h32m10.809878947s) [upload=174963]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 7 18:12:39.114038290 (198h32m10.809887043s) [upload=174963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 7 18:12:39.451448640 (198h32m11.147297403s) [upload=174963]: Done uploading segments... Nov 7 18:12:39.451473607 (198h32m11.14732236s) [upload=174963]: Upload finished. err: <nil> Duration: 6.594783513s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [XXXX.X..XXX.X..X.X.XXXX.X..XX.XXXXX.X.......................] duration: 6.212699652s trace: https://jaeger.dev.storj.io/trace/422f33ac8ae6a74a FAIL (2025-11-07 18:30:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ) context deadline exceeded; piecestore: (Node ID: 12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, Piece ID: JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ) context deadline exceeded ======LOGS====== Nov 7 18:12:35.791697270 (198h32m7.487546023s) [upload=174963]: Starting upload Nov 7 18:12:35.791791689 (198h32m7.487640452s) [upload=174963]: Done starting upload Nov 7 18:12:35.791885458 (198h32m7.487734211s) [upload=174963]: Uploading segments... Nov 7 18:12:35.792146455 (198h32m7.487995208s) [upload=174963]: Got next segment. Inline: false Nov 7 18:12:35.792195278 (198h32m7.488044021s) [upload=174963]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 7 18:12:35.792201820 (198h32m7.488050563s) [upload=174963]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 7 18:12:36.063867588 (198h32m7.759716341s) [seg_pos={0 0}, upload=174963]: Begin upload segment... Nov 7 18:12:36.063937832 (198h32m7.759786585s) [seg_pos={0 0}, upload=174963]: Done begin upload segment. Nov 7 18:12:36.063988689 (198h32m7.759837463s) [noise=true , address=145.79.227.27:4690 , piece_id=7CVNJTAEBVYJXRWNSFYFP3E7SKMY5XI4CV2JEZJLSXH4TA2F2JZQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.064083510 (198h32m7.759932263s) [noise=true , address=145.79.227.10:4403 , piece_id=JCMXNJ5PW5ULGOJZ5SGB423VW7VJPC5REYNSNWJZJXHGUQXUS3KA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.064896969 (198h32m7.760745762s) [noise=true , address=145.79.227.28:4726 , piece_id=D6DH5FHFB5ZTPANRPUWZKRYPFYUV2LS6AY5FQL5RBJZIUNM4FKLQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.065443520 (198h32m7.761292273s) [noise=true , address=77.90.31.11:4554 , piece_id=TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.065970974 (198h32m7.761819777s) [noise=true , address=145.79.227.8:4330 , piece_id=32XBCOEGSUTVUCCWVM2D7CB6FKMNLD6N2N4655B3ZYHNAIL4R2QQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.066422053 (198h32m7.762270806s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.067054427 (198h32m7.76290318s) [noise=true , address=77.90.31.10:4519 , piece_id=MIHLKGHCRTUFQSGBIFKIWPNACAJ4CQOEYRDIKA4KJLFP356VMY6A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.067645042 (198h32m7.763493795s) [noise=true , address=45.91.116.6:4973 , piece_id=Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.068379079 (198h32m7.764227842s) [noise=true , address=77.90.31.19:4843 , piece_id=JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.070931099 (198h32m7.766779862s) [noise=true , address=145.79.227.11:4732 , piece_id=VKSPYLTD6C6XBUFXXVVPHIPGFRCRNQK6EVI4SM7Y6ODOMVGGAJOQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.073100420 (198h32m7.768949173s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.075821872 (198h32m7.771670625s) [noise=true , address=45.91.116.4:4937 , piece_id=CWDA2IYNFUYHVDBJ74Y3CJKCZUBN4QYUEJYFKZNIBAUYYAXO3WFQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.078400863 (198h32m7.774249626s) [noise=true , address=45.91.116.10:4082 , piece_id=RW62GCPC56K2IMOFF276GQECXRKW6NY6W44TNAYFNDGQS3UK7EEA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.081230762 (198h32m7.777079515s) [noise=true , address=145.79.227.15:4876 , piece_id=ZPGKGNXRFEWRRJKTS7NKBEVQA7OAWNWYDR3DUTPHT3XVYRJSKWPQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.083298258 (198h32m7.779147012s) [noise=true , address=145.79.227.20:4438 , piece_id=V6WDUVDB7WHTDCTSYCFWS22UKRD5NMXGJ2GHLSFLVVOXZMZWLL4A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.084602091 (198h32m7.780450844s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:37.640211006 (198h32m9.336059759s) [upload=174963]: Next returned nil segment Nov 7 18:12:37.640227878 (198h32m9.336076621s) [upload=174963]: Waiting for error group managing segments... Nov 7 18:12:38.007509819 (198h32m9.703358572s) [noise=true , address=145.79.227.10:4403 , piece_id=JCMXNJ5PW5ULGOJZ5SGB423VW7VJPC5REYNSNWJZJXHGUQXUS3KA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.015182750 (198h32m9.711031503s) [noise=true , address=145.79.227.28:4726 , piece_id=D6DH5FHFB5ZTPANRPUWZKRYPFYUV2LS6AY5FQL5RBJZIUNM4FKLQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.020400886 (198h32m9.71624964s) [noise=true , address=145.79.227.27:4690 , piece_id=7CVNJTAEBVYJXRWNSFYFP3E7SKMY5XI4CV2JEZJLSXH4TA2F2JZQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.062361869 (198h32m9.758210622s) [noise=true , address=145.79.227.15:4876 , piece_id=ZPGKGNXRFEWRRJKTS7NKBEVQA7OAWNWYDR3DUTPHT3XVYRJSKWPQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.096654439 (198h32m9.792503192s) [noise=true , address=145.79.227.20:4438 , piece_id=V6WDUVDB7WHTDCTSYCFWS22UKRD5NMXGJ2GHLSFLVVOXZMZWLL4A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.244365682 (198h32m9.940214435s) [noise=true , address=145.79.227.8:4330 , piece_id=32XBCOEGSUTVUCCWVM2D7CB6FKMNLD6N2N4655B3ZYHNAIL4R2QQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.245114297 (198h32m9.94096305s) [noise=true , address=145.79.227.11:4732 , piece_id=VKSPYLTD6C6XBUFXXVVPHIPGFRCRNQK6EVI4SM7Y6ODOMVGGAJOQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.324356281 (198h32m10.020205034s) [noise=true , address=77.90.31.11:4554 , piece_id=TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.331696878 (198h32m10.027545621s) [noise=true , address=45.91.116.4:4937 , piece_id=CWDA2IYNFUYHVDBJ74Y3CJKCZUBN4QYUEJYFKZNIBAUYYAXO3WFQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.334142676 (198h32m10.029991429s) [noise=true , address=45.91.116.10:4082 , piece_id=RW62GCPC56K2IMOFF276GQECXRKW6NY6W44TNAYFNDGQS3UK7EEA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.346539031 (198h32m10.042387794s) [noise=true , address=77.90.31.10:4519 , piece_id=MIHLKGHCRTUFQSGBIFKIWPNACAJ4CQOEYRDIKA4KJLFP356VMY6A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.419504032 (198h32m10.115352785s) [noise=true , address=77.90.31.19:4843 , piece_id=JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:39.113150499 (198h32m10.808999252s) [noise=true , address=45.91.116.6:4973 , piece_id=Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:39.113177150 (198h32m10.809025903s) [seg_pos={0 0}, upload=174963]: Segment reached optimal threshold of 13 pieces. Nov 7 18:12:39.113381018 (198h32m10.809229772s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113390166 (198h32m10.809238919s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113460199 (198h32m10.809308952s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113470077 (198h32m10.809318821s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113780338 (198h32m10.809629091s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113788243 (198h32m10.809636986s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113858657 (198h32m10.8097074s) [seg_pos={0 0}, upload=174963]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 7 18:12:39.114030194 (198h32m10.809878947s) [upload=174963]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 7 18:12:39.114038290 (198h32m10.809887043s) [upload=174963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 7 18:12:39.451448640 (198h32m11.147297403s) [upload=174963]: Done uploading segments... Nov 7 18:12:39.451473607 (198h32m11.14732236s) [upload=174963]: Upload finished. err: <nil> Duration: 6.212728497s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [.XXXX.X..XXX.X..X.X.XXXX.X..XX.XXXXX.X......................] duration: 6.575333807s trace: https://jaeger.dev.storj.io/trace/422f33ac8ae6a74a FAIL (2025-11-07 18:30:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ) context deadline exceeded; piecestore: (Node ID: 12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, Piece ID: JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ) context deadline exceeded ======LOGS====== Nov 7 18:12:35.791697270 (198h32m7.487546023s) [upload=174963]: Starting upload Nov 7 18:12:35.791791689 (198h32m7.487640452s) [upload=174963]: Done starting upload Nov 7 18:12:35.791885458 (198h32m7.487734211s) [upload=174963]: Uploading segments... Nov 7 18:12:35.792146455 (198h32m7.487995208s) [upload=174963]: Got next segment. Inline: false Nov 7 18:12:35.792195278 (198h32m7.488044021s) [upload=174963]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 7 18:12:35.792201820 (198h32m7.488050563s) [upload=174963]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 7 18:12:36.063867588 (198h32m7.759716341s) [seg_pos={0 0}, upload=174963]: Begin upload segment... Nov 7 18:12:36.063937832 (198h32m7.759786585s) [seg_pos={0 0}, upload=174963]: Done begin upload segment. Nov 7 18:12:36.063988689 (198h32m7.759837463s) [noise=true , address=145.79.227.27:4690 , piece_id=7CVNJTAEBVYJXRWNSFYFP3E7SKMY5XI4CV2JEZJLSXH4TA2F2JZQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.064083510 (198h32m7.759932263s) [noise=true , address=145.79.227.10:4403 , piece_id=JCMXNJ5PW5ULGOJZ5SGB423VW7VJPC5REYNSNWJZJXHGUQXUS3KA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.064896969 (198h32m7.760745762s) [noise=true , address=145.79.227.28:4726 , piece_id=D6DH5FHFB5ZTPANRPUWZKRYPFYUV2LS6AY5FQL5RBJZIUNM4FKLQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.065443520 (198h32m7.761292273s) [noise=true , address=77.90.31.11:4554 , piece_id=TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.065970974 (198h32m7.761819777s) [noise=true , address=145.79.227.8:4330 , piece_id=32XBCOEGSUTVUCCWVM2D7CB6FKMNLD6N2N4655B3ZYHNAIL4R2QQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.066422053 (198h32m7.762270806s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.067054427 (198h32m7.76290318s) [noise=true , address=77.90.31.10:4519 , piece_id=MIHLKGHCRTUFQSGBIFKIWPNACAJ4CQOEYRDIKA4KJLFP356VMY6A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.067645042 (198h32m7.763493795s) [noise=true , address=45.91.116.6:4973 , piece_id=Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.068379079 (198h32m7.764227842s) [noise=true , address=77.90.31.19:4843 , piece_id=JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.070931099 (198h32m7.766779862s) [noise=true , address=145.79.227.11:4732 , piece_id=VKSPYLTD6C6XBUFXXVVPHIPGFRCRNQK6EVI4SM7Y6ODOMVGGAJOQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.073100420 (198h32m7.768949173s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.075821872 (198h32m7.771670625s) [noise=true , address=45.91.116.4:4937 , piece_id=CWDA2IYNFUYHVDBJ74Y3CJKCZUBN4QYUEJYFKZNIBAUYYAXO3WFQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.078400863 (198h32m7.774249626s) [noise=true , address=45.91.116.10:4082 , piece_id=RW62GCPC56K2IMOFF276GQECXRKW6NY6W44TNAYFNDGQS3UK7EEA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.081230762 (198h32m7.777079515s) [noise=true , address=145.79.227.15:4876 , piece_id=ZPGKGNXRFEWRRJKTS7NKBEVQA7OAWNWYDR3DUTPHT3XVYRJSKWPQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.083298258 (198h32m7.779147012s) [noise=true , address=145.79.227.20:4438 , piece_id=V6WDUVDB7WHTDCTSYCFWS22UKRD5NMXGJ2GHLSFLVVOXZMZWLL4A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.084602091 (198h32m7.780450844s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:37.640211006 (198h32m9.336059759s) [upload=174963]: Next returned nil segment Nov 7 18:12:37.640227878 (198h32m9.336076621s) [upload=174963]: Waiting for error group managing segments... Nov 7 18:12:38.007509819 (198h32m9.703358572s) [noise=true , address=145.79.227.10:4403 , piece_id=JCMXNJ5PW5ULGOJZ5SGB423VW7VJPC5REYNSNWJZJXHGUQXUS3KA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.015182750 (198h32m9.711031503s) [noise=true , address=145.79.227.28:4726 , piece_id=D6DH5FHFB5ZTPANRPUWZKRYPFYUV2LS6AY5FQL5RBJZIUNM4FKLQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.020400886 (198h32m9.71624964s) [noise=true , address=145.79.227.27:4690 , piece_id=7CVNJTAEBVYJXRWNSFYFP3E7SKMY5XI4CV2JEZJLSXH4TA2F2JZQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.062361869 (198h32m9.758210622s) [noise=true , address=145.79.227.15:4876 , piece_id=ZPGKGNXRFEWRRJKTS7NKBEVQA7OAWNWYDR3DUTPHT3XVYRJSKWPQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.096654439 (198h32m9.792503192s) [noise=true , address=145.79.227.20:4438 , piece_id=V6WDUVDB7WHTDCTSYCFWS22UKRD5NMXGJ2GHLSFLVVOXZMZWLL4A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.244365682 (198h32m9.940214435s) [noise=true , address=145.79.227.8:4330 , piece_id=32XBCOEGSUTVUCCWVM2D7CB6FKMNLD6N2N4655B3ZYHNAIL4R2QQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.245114297 (198h32m9.94096305s) [noise=true , address=145.79.227.11:4732 , piece_id=VKSPYLTD6C6XBUFXXVVPHIPGFRCRNQK6EVI4SM7Y6ODOMVGGAJOQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.324356281 (198h32m10.020205034s) [noise=true , address=77.90.31.11:4554 , piece_id=TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.331696878 (198h32m10.027545621s) [noise=true , address=45.91.116.4:4937 , piece_id=CWDA2IYNFUYHVDBJ74Y3CJKCZUBN4QYUEJYFKZNIBAUYYAXO3WFQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.334142676 (198h32m10.029991429s) [noise=true , address=45.91.116.10:4082 , piece_id=RW62GCPC56K2IMOFF276GQECXRKW6NY6W44TNAYFNDGQS3UK7EEA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.346539031 (198h32m10.042387794s) [noise=true , address=77.90.31.10:4519 , piece_id=MIHLKGHCRTUFQSGBIFKIWPNACAJ4CQOEYRDIKA4KJLFP356VMY6A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.419504032 (198h32m10.115352785s) [noise=true , address=77.90.31.19:4843 , piece_id=JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:39.113150499 (198h32m10.808999252s) [noise=true , address=45.91.116.6:4973 , piece_id=Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:39.113177150 (198h32m10.809025903s) [seg_pos={0 0}, upload=174963]: Segment reached optimal threshold of 13 pieces. Nov 7 18:12:39.113381018 (198h32m10.809229772s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113390166 (198h32m10.809238919s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113460199 (198h32m10.809308952s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113470077 (198h32m10.809318821s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113780338 (198h32m10.809629091s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113788243 (198h32m10.809636986s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113858657 (198h32m10.8097074s) [seg_pos={0 0}, upload=174963]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 7 18:12:39.114030194 (198h32m10.809878947s) [upload=174963]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 7 18:12:39.114038290 (198h32m10.809887043s) [upload=174963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 7 18:12:39.451448640 (198h32m11.147297403s) [upload=174963]: Done uploading segments... Nov 7 18:12:39.451473607 (198h32m11.14732236s) [upload=174963]: Upload finished. err: <nil> Duration: 6.575369325s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [..XXXX.X..XXX.X..X.X.XXXX.X..XX.XXXXX.X.....................] duration: 6.198038018s trace: https://jaeger.dev.storj.io/trace/422f33ac8ae6a74a FAIL (2025-11-07 18:29:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ) context deadline exceeded; piecestore: (Node ID: 12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, Piece ID: JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ) context deadline exceeded ======LOGS====== Nov 7 18:12:35.791697270 (198h32m7.487546023s) [upload=174963]: Starting upload Nov 7 18:12:35.791791689 (198h32m7.487640452s) [upload=174963]: Done starting upload Nov 7 18:12:35.791885458 (198h32m7.487734211s) [upload=174963]: Uploading segments... Nov 7 18:12:35.792146455 (198h32m7.487995208s) [upload=174963]: Got next segment. Inline: false Nov 7 18:12:35.792195278 (198h32m7.488044021s) [upload=174963]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 7 18:12:35.792201820 (198h32m7.488050563s) [upload=174963]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 7 18:12:36.063867588 (198h32m7.759716341s) [seg_pos={0 0}, upload=174963]: Begin upload segment... Nov 7 18:12:36.063937832 (198h32m7.759786585s) [seg_pos={0 0}, upload=174963]: Done begin upload segment. Nov 7 18:12:36.063988689 (198h32m7.759837463s) [noise=true , address=145.79.227.27:4690 , piece_id=7CVNJTAEBVYJXRWNSFYFP3E7SKMY5XI4CV2JEZJLSXH4TA2F2JZQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.064083510 (198h32m7.759932263s) [noise=true , address=145.79.227.10:4403 , piece_id=JCMXNJ5PW5ULGOJZ5SGB423VW7VJPC5REYNSNWJZJXHGUQXUS3KA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.064896969 (198h32m7.760745762s) [noise=true , address=145.79.227.28:4726 , piece_id=D6DH5FHFB5ZTPANRPUWZKRYPFYUV2LS6AY5FQL5RBJZIUNM4FKLQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.065443520 (198h32m7.761292273s) [noise=true , address=77.90.31.11:4554 , piece_id=TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.065970974 (198h32m7.761819777s) [noise=true , address=145.79.227.8:4330 , piece_id=32XBCOEGSUTVUCCWVM2D7CB6FKMNLD6N2N4655B3ZYHNAIL4R2QQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.066422053 (198h32m7.762270806s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.067054427 (198h32m7.76290318s) [noise=true , address=77.90.31.10:4519 , piece_id=MIHLKGHCRTUFQSGBIFKIWPNACAJ4CQOEYRDIKA4KJLFP356VMY6A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.067645042 (198h32m7.763493795s) [noise=true , address=45.91.116.6:4973 , piece_id=Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.068379079 (198h32m7.764227842s) [noise=true , address=77.90.31.19:4843 , piece_id=JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.070931099 (198h32m7.766779862s) [noise=true , address=145.79.227.11:4732 , piece_id=VKSPYLTD6C6XBUFXXVVPHIPGFRCRNQK6EVI4SM7Y6ODOMVGGAJOQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.073100420 (198h32m7.768949173s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.075821872 (198h32m7.771670625s) [noise=true , address=45.91.116.4:4937 , piece_id=CWDA2IYNFUYHVDBJ74Y3CJKCZUBN4QYUEJYFKZNIBAUYYAXO3WFQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.078400863 (198h32m7.774249626s) [noise=true , address=45.91.116.10:4082 , piece_id=RW62GCPC56K2IMOFF276GQECXRKW6NY6W44TNAYFNDGQS3UK7EEA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.081230762 (198h32m7.777079515s) [noise=true , address=145.79.227.15:4876 , piece_id=ZPGKGNXRFEWRRJKTS7NKBEVQA7OAWNWYDR3DUTPHT3XVYRJSKWPQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.083298258 (198h32m7.779147012s) [noise=true , address=145.79.227.20:4438 , piece_id=V6WDUVDB7WHTDCTSYCFWS22UKRD5NMXGJ2GHLSFLVVOXZMZWLL4A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.084602091 (198h32m7.780450844s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:37.640211006 (198h32m9.336059759s) [upload=174963]: Next returned nil segment Nov 7 18:12:37.640227878 (198h32m9.336076621s) [upload=174963]: Waiting for error group managing segments... Nov 7 18:12:38.007509819 (198h32m9.703358572s) [noise=true , address=145.79.227.10:4403 , piece_id=JCMXNJ5PW5ULGOJZ5SGB423VW7VJPC5REYNSNWJZJXHGUQXUS3KA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.015182750 (198h32m9.711031503s) [noise=true , address=145.79.227.28:4726 , piece_id=D6DH5FHFB5ZTPANRPUWZKRYPFYUV2LS6AY5FQL5RBJZIUNM4FKLQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.020400886 (198h32m9.71624964s) [noise=true , address=145.79.227.27:4690 , piece_id=7CVNJTAEBVYJXRWNSFYFP3E7SKMY5XI4CV2JEZJLSXH4TA2F2JZQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.062361869 (198h32m9.758210622s) [noise=true , address=145.79.227.15:4876 , piece_id=ZPGKGNXRFEWRRJKTS7NKBEVQA7OAWNWYDR3DUTPHT3XVYRJSKWPQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.096654439 (198h32m9.792503192s) [noise=true , address=145.79.227.20:4438 , piece_id=V6WDUVDB7WHTDCTSYCFWS22UKRD5NMXGJ2GHLSFLVVOXZMZWLL4A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.244365682 (198h32m9.940214435s) [noise=true , address=145.79.227.8:4330 , piece_id=32XBCOEGSUTVUCCWVM2D7CB6FKMNLD6N2N4655B3ZYHNAIL4R2QQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.245114297 (198h32m9.94096305s) [noise=true , address=145.79.227.11:4732 , piece_id=VKSPYLTD6C6XBUFXXVVPHIPGFRCRNQK6EVI4SM7Y6ODOMVGGAJOQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.324356281 (198h32m10.020205034s) [noise=true , address=77.90.31.11:4554 , piece_id=TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.331696878 (198h32m10.027545621s) [noise=true , address=45.91.116.4:4937 , piece_id=CWDA2IYNFUYHVDBJ74Y3CJKCZUBN4QYUEJYFKZNIBAUYYAXO3WFQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.334142676 (198h32m10.029991429s) [noise=true , address=45.91.116.10:4082 , piece_id=RW62GCPC56K2IMOFF276GQECXRKW6NY6W44TNAYFNDGQS3UK7EEA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.346539031 (198h32m10.042387794s) [noise=true , address=77.90.31.10:4519 , piece_id=MIHLKGHCRTUFQSGBIFKIWPNACAJ4CQOEYRDIKA4KJLFP356VMY6A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.419504032 (198h32m10.115352785s) [noise=true , address=77.90.31.19:4843 , piece_id=JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:39.113150499 (198h32m10.808999252s) [noise=true , address=45.91.116.6:4973 , piece_id=Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:39.113177150 (198h32m10.809025903s) [seg_pos={0 0}, upload=174963]: Segment reached optimal threshold of 13 pieces. Nov 7 18:12:39.113381018 (198h32m10.809229772s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113390166 (198h32m10.809238919s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113460199 (198h32m10.809308952s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113470077 (198h32m10.809318821s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113780338 (198h32m10.809629091s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113788243 (198h32m10.809636986s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113858657 (198h32m10.8097074s) [seg_pos={0 0}, upload=174963]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 7 18:12:39.114030194 (198h32m10.809878947s) [upload=174963]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 7 18:12:39.114038290 (198h32m10.809887043s) [upload=174963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 7 18:12:39.451448640 (198h32m11.147297403s) [upload=174963]: Done uploading segments... Nov 7 18:12:39.451473607 (198h32m11.14732236s) [upload=174963]: Upload finished. err: <nil> Duration: 6.198073615s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [X..XXXX.X..XXX.X..X.X.XXXX.X..XX.XXXXX.X....................] duration: 6.041811596s trace: https://jaeger.dev.storj.io/trace/422f33ac8ae6a74a FAIL (2025-11-07 18:28:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ) context deadline exceeded; piecestore: (Node ID: 12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, Piece ID: JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ) context deadline exceeded ======LOGS====== Nov 7 18:12:35.791697270 (198h32m7.487546023s) [upload=174963]: Starting upload Nov 7 18:12:35.791791689 (198h32m7.487640452s) [upload=174963]: Done starting upload Nov 7 18:12:35.791885458 (198h32m7.487734211s) [upload=174963]: Uploading segments... Nov 7 18:12:35.792146455 (198h32m7.487995208s) [upload=174963]: Got next segment. Inline: false Nov 7 18:12:35.792195278 (198h32m7.488044021s) [upload=174963]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 7 18:12:35.792201820 (198h32m7.488050563s) [upload=174963]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 7 18:12:36.063867588 (198h32m7.759716341s) [seg_pos={0 0}, upload=174963]: Begin upload segment... Nov 7 18:12:36.063937832 (198h32m7.759786585s) [seg_pos={0 0}, upload=174963]: Done begin upload segment. Nov 7 18:12:36.063988689 (198h32m7.759837463s) [noise=true , address=145.79.227.27:4690 , piece_id=7CVNJTAEBVYJXRWNSFYFP3E7SKMY5XI4CV2JEZJLSXH4TA2F2JZQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.064083510 (198h32m7.759932263s) [noise=true , address=145.79.227.10:4403 , piece_id=JCMXNJ5PW5ULGOJZ5SGB423VW7VJPC5REYNSNWJZJXHGUQXUS3KA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.064896969 (198h32m7.760745762s) [noise=true , address=145.79.227.28:4726 , piece_id=D6DH5FHFB5ZTPANRPUWZKRYPFYUV2LS6AY5FQL5RBJZIUNM4FKLQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.065443520 (198h32m7.761292273s) [noise=true , address=77.90.31.11:4554 , piece_id=TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.065970974 (198h32m7.761819777s) [noise=true , address=145.79.227.8:4330 , piece_id=32XBCOEGSUTVUCCWVM2D7CB6FKMNLD6N2N4655B3ZYHNAIL4R2QQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.066422053 (198h32m7.762270806s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.067054427 (198h32m7.76290318s) [noise=true , address=77.90.31.10:4519 , piece_id=MIHLKGHCRTUFQSGBIFKIWPNACAJ4CQOEYRDIKA4KJLFP356VMY6A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.067645042 (198h32m7.763493795s) [noise=true , address=45.91.116.6:4973 , piece_id=Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.068379079 (198h32m7.764227842s) [noise=true , address=77.90.31.19:4843 , piece_id=JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.070931099 (198h32m7.766779862s) [noise=true , address=145.79.227.11:4732 , piece_id=VKSPYLTD6C6XBUFXXVVPHIPGFRCRNQK6EVI4SM7Y6ODOMVGGAJOQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.073100420 (198h32m7.768949173s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.075821872 (198h32m7.771670625s) [noise=true , address=45.91.116.4:4937 , piece_id=CWDA2IYNFUYHVDBJ74Y3CJKCZUBN4QYUEJYFKZNIBAUYYAXO3WFQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.078400863 (198h32m7.774249626s) [noise=true , address=45.91.116.10:4082 , piece_id=RW62GCPC56K2IMOFF276GQECXRKW6NY6W44TNAYFNDGQS3UK7EEA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.081230762 (198h32m7.777079515s) [noise=true , address=145.79.227.15:4876 , piece_id=ZPGKGNXRFEWRRJKTS7NKBEVQA7OAWNWYDR3DUTPHT3XVYRJSKWPQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.083298258 (198h32m7.779147012s) [noise=true , address=145.79.227.20:4438 , piece_id=V6WDUVDB7WHTDCTSYCFWS22UKRD5NMXGJ2GHLSFLVVOXZMZWLL4A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.084602091 (198h32m7.780450844s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:37.640211006 (198h32m9.336059759s) [upload=174963]: Next returned nil segment Nov 7 18:12:37.640227878 (198h32m9.336076621s) [upload=174963]: Waiting for error group managing segments... Nov 7 18:12:38.007509819 (198h32m9.703358572s) [noise=true , address=145.79.227.10:4403 , piece_id=JCMXNJ5PW5ULGOJZ5SGB423VW7VJPC5REYNSNWJZJXHGUQXUS3KA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.015182750 (198h32m9.711031503s) [noise=true , address=145.79.227.28:4726 , piece_id=D6DH5FHFB5ZTPANRPUWZKRYPFYUV2LS6AY5FQL5RBJZIUNM4FKLQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.020400886 (198h32m9.71624964s) [noise=true , address=145.79.227.27:4690 , piece_id=7CVNJTAEBVYJXRWNSFYFP3E7SKMY5XI4CV2JEZJLSXH4TA2F2JZQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.062361869 (198h32m9.758210622s) [noise=true , address=145.79.227.15:4876 , piece_id=ZPGKGNXRFEWRRJKTS7NKBEVQA7OAWNWYDR3DUTPHT3XVYRJSKWPQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.096654439 (198h32m9.792503192s) [noise=true , address=145.79.227.20:4438 , piece_id=V6WDUVDB7WHTDCTSYCFWS22UKRD5NMXGJ2GHLSFLVVOXZMZWLL4A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.244365682 (198h32m9.940214435s) [noise=true , address=145.79.227.8:4330 , piece_id=32XBCOEGSUTVUCCWVM2D7CB6FKMNLD6N2N4655B3ZYHNAIL4R2QQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.245114297 (198h32m9.94096305s) [noise=true , address=145.79.227.11:4732 , piece_id=VKSPYLTD6C6XBUFXXVVPHIPGFRCRNQK6EVI4SM7Y6ODOMVGGAJOQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.324356281 (198h32m10.020205034s) [noise=true , address=77.90.31.11:4554 , piece_id=TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.331696878 (198h32m10.027545621s) [noise=true , address=45.91.116.4:4937 , piece_id=CWDA2IYNFUYHVDBJ74Y3CJKCZUBN4QYUEJYFKZNIBAUYYAXO3WFQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.334142676 (198h32m10.029991429s) [noise=true , address=45.91.116.10:4082 , piece_id=RW62GCPC56K2IMOFF276GQECXRKW6NY6W44TNAYFNDGQS3UK7EEA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.346539031 (198h32m10.042387794s) [noise=true , address=77.90.31.10:4519 , piece_id=MIHLKGHCRTUFQSGBIFKIWPNACAJ4CQOEYRDIKA4KJLFP356VMY6A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.419504032 (198h32m10.115352785s) [noise=true , address=77.90.31.19:4843 , piece_id=JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:39.113150499 (198h32m10.808999252s) [noise=true , address=45.91.116.6:4973 , piece_id=Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:39.113177150 (198h32m10.809025903s) [seg_pos={0 0}, upload=174963]: Segment reached optimal threshold of 13 pieces. Nov 7 18:12:39.113381018 (198h32m10.809229772s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113390166 (198h32m10.809238919s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113460199 (198h32m10.809308952s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113470077 (198h32m10.809318821s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113780338 (198h32m10.809629091s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113788243 (198h32m10.809636986s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113858657 (198h32m10.8097074s) [seg_pos={0 0}, upload=174963]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 7 18:12:39.114030194 (198h32m10.809878947s) [upload=174963]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 7 18:12:39.114038290 (198h32m10.809887043s) [upload=174963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 7 18:12:39.451448640 (198h32m11.147297403s) [upload=174963]: Done uploading segments... Nov 7 18:12:39.451473607 (198h32m11.14732236s) [upload=174963]: Upload finished. err: <nil> Duration: 6.041844709s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [XX..XXXX.X..XXX.X..X.X.XXXX.X..XX.XXXXX.X...................] duration: 5.983942261s trace: https://jaeger.dev.storj.io/trace/422f33ac8ae6a74a FAIL (2025-11-07 18:28:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ) context deadline exceeded; piecestore: (Node ID: 12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, Piece ID: JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ) context deadline exceeded ======LOGS====== Nov 7 18:12:35.791697270 (198h32m7.487546023s) [upload=174963]: Starting upload Nov 7 18:12:35.791791689 (198h32m7.487640452s) [upload=174963]: Done starting upload Nov 7 18:12:35.791885458 (198h32m7.487734211s) [upload=174963]: Uploading segments... Nov 7 18:12:35.792146455 (198h32m7.487995208s) [upload=174963]: Got next segment. Inline: false Nov 7 18:12:35.792195278 (198h32m7.488044021s) [upload=174963]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 7 18:12:35.792201820 (198h32m7.488050563s) [upload=174963]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 7 18:12:36.063867588 (198h32m7.759716341s) [seg_pos={0 0}, upload=174963]: Begin upload segment... Nov 7 18:12:36.063937832 (198h32m7.759786585s) [seg_pos={0 0}, upload=174963]: Done begin upload segment. Nov 7 18:12:36.063988689 (198h32m7.759837463s) [noise=true , address=145.79.227.27:4690 , piece_id=7CVNJTAEBVYJXRWNSFYFP3E7SKMY5XI4CV2JEZJLSXH4TA2F2JZQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.064083510 (198h32m7.759932263s) [noise=true , address=145.79.227.10:4403 , piece_id=JCMXNJ5PW5ULGOJZ5SGB423VW7VJPC5REYNSNWJZJXHGUQXUS3KA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.064896969 (198h32m7.760745762s) [noise=true , address=145.79.227.28:4726 , piece_id=D6DH5FHFB5ZTPANRPUWZKRYPFYUV2LS6AY5FQL5RBJZIUNM4FKLQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.065443520 (198h32m7.761292273s) [noise=true , address=77.90.31.11:4554 , piece_id=TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.065970974 (198h32m7.761819777s) [noise=true , address=145.79.227.8:4330 , piece_id=32XBCOEGSUTVUCCWVM2D7CB6FKMNLD6N2N4655B3ZYHNAIL4R2QQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.066422053 (198h32m7.762270806s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.067054427 (198h32m7.76290318s) [noise=true , address=77.90.31.10:4519 , piece_id=MIHLKGHCRTUFQSGBIFKIWPNACAJ4CQOEYRDIKA4KJLFP356VMY6A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.067645042 (198h32m7.763493795s) [noise=true , address=45.91.116.6:4973 , piece_id=Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.068379079 (198h32m7.764227842s) [noise=true , address=77.90.31.19:4843 , piece_id=JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.070931099 (198h32m7.766779862s) [noise=true , address=145.79.227.11:4732 , piece_id=VKSPYLTD6C6XBUFXXVVPHIPGFRCRNQK6EVI4SM7Y6ODOMVGGAJOQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.073100420 (198h32m7.768949173s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.075821872 (198h32m7.771670625s) [noise=true , address=45.91.116.4:4937 , piece_id=CWDA2IYNFUYHVDBJ74Y3CJKCZUBN4QYUEJYFKZNIBAUYYAXO3WFQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.078400863 (198h32m7.774249626s) [noise=true , address=45.91.116.10:4082 , piece_id=RW62GCPC56K2IMOFF276GQECXRKW6NY6W44TNAYFNDGQS3UK7EEA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.081230762 (198h32m7.777079515s) [noise=true , address=145.79.227.15:4876 , piece_id=ZPGKGNXRFEWRRJKTS7NKBEVQA7OAWNWYDR3DUTPHT3XVYRJSKWPQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.083298258 (198h32m7.779147012s) [noise=true , address=145.79.227.20:4438 , piece_id=V6WDUVDB7WHTDCTSYCFWS22UKRD5NMXGJ2GHLSFLVVOXZMZWLL4A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.084602091 (198h32m7.780450844s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:37.640211006 (198h32m9.336059759s) [upload=174963]: Next returned nil segment Nov 7 18:12:37.640227878 (198h32m9.336076621s) [upload=174963]: Waiting for error group managing segments... Nov 7 18:12:38.007509819 (198h32m9.703358572s) [noise=true , address=145.79.227.10:4403 , piece_id=JCMXNJ5PW5ULGOJZ5SGB423VW7VJPC5REYNSNWJZJXHGUQXUS3KA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.015182750 (198h32m9.711031503s) [noise=true , address=145.79.227.28:4726 , piece_id=D6DH5FHFB5ZTPANRPUWZKRYPFYUV2LS6AY5FQL5RBJZIUNM4FKLQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.020400886 (198h32m9.71624964s) [noise=true , address=145.79.227.27:4690 , piece_id=7CVNJTAEBVYJXRWNSFYFP3E7SKMY5XI4CV2JEZJLSXH4TA2F2JZQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.062361869 (198h32m9.758210622s) [noise=true , address=145.79.227.15:4876 , piece_id=ZPGKGNXRFEWRRJKTS7NKBEVQA7OAWNWYDR3DUTPHT3XVYRJSKWPQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.096654439 (198h32m9.792503192s) [noise=true , address=145.79.227.20:4438 , piece_id=V6WDUVDB7WHTDCTSYCFWS22UKRD5NMXGJ2GHLSFLVVOXZMZWLL4A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.244365682 (198h32m9.940214435s) [noise=true , address=145.79.227.8:4330 , piece_id=32XBCOEGSUTVUCCWVM2D7CB6FKMNLD6N2N4655B3ZYHNAIL4R2QQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.245114297 (198h32m9.94096305s) [noise=true , address=145.79.227.11:4732 , piece_id=VKSPYLTD6C6XBUFXXVVPHIPGFRCRNQK6EVI4SM7Y6ODOMVGGAJOQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.324356281 (198h32m10.020205034s) [noise=true , address=77.90.31.11:4554 , piece_id=TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.331696878 (198h32m10.027545621s) [noise=true , address=45.91.116.4:4937 , piece_id=CWDA2IYNFUYHVDBJ74Y3CJKCZUBN4QYUEJYFKZNIBAUYYAXO3WFQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.334142676 (198h32m10.029991429s) [noise=true , address=45.91.116.10:4082 , piece_id=RW62GCPC56K2IMOFF276GQECXRKW6NY6W44TNAYFNDGQS3UK7EEA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.346539031 (198h32m10.042387794s) [noise=true , address=77.90.31.10:4519 , piece_id=MIHLKGHCRTUFQSGBIFKIWPNACAJ4CQOEYRDIKA4KJLFP356VMY6A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.419504032 (198h32m10.115352785s) [noise=true , address=77.90.31.19:4843 , piece_id=JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:39.113150499 (198h32m10.808999252s) [noise=true , address=45.91.116.6:4973 , piece_id=Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:39.113177150 (198h32m10.809025903s) [seg_pos={0 0}, upload=174963]: Segment reached optimal threshold of 13 pieces. Nov 7 18:12:39.113381018 (198h32m10.809229772s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113390166 (198h32m10.809238919s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113460199 (198h32m10.809308952s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113470077 (198h32m10.809318821s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113780338 (198h32m10.809629091s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113788243 (198h32m10.809636986s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113858657 (198h32m10.8097074s) [seg_pos={0 0}, upload=174963]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 7 18:12:39.114030194 (198h32m10.809878947s) [upload=174963]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 7 18:12:39.114038290 (198h32m10.809887043s) [upload=174963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 7 18:12:39.451448640 (198h32m11.147297403s) [upload=174963]: Done uploading segments... Nov 7 18:12:39.451473607 (198h32m11.14732236s) [upload=174963]: Upload finished. err: <nil> Duration: 5.983977297s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 27 out of 60 is failed (threshold=3) [XXX..XXXX.X..XXX.X..X.X.XXXX.X..XX.XXXXX.X..................] duration: 6.13048334s trace: https://jaeger.dev.storj.io/trace/422f33ac8ae6a74a FAIL (2025-11-07 18:28:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ) context deadline exceeded; piecestore: (Node ID: 12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, Piece ID: JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ) context deadline exceeded ======LOGS====== Nov 7 18:12:35.791697270 (198h32m7.487546023s) [upload=174963]: Starting upload Nov 7 18:12:35.791791689 (198h32m7.487640452s) [upload=174963]: Done starting upload Nov 7 18:12:35.791885458 (198h32m7.487734211s) [upload=174963]: Uploading segments... Nov 7 18:12:35.792146455 (198h32m7.487995208s) [upload=174963]: Got next segment. Inline: false Nov 7 18:12:35.792195278 (198h32m7.488044021s) [upload=174963]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 7 18:12:35.792201820 (198h32m7.488050563s) [upload=174963]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 7 18:12:36.063867588 (198h32m7.759716341s) [seg_pos={0 0}, upload=174963]: Begin upload segment... Nov 7 18:12:36.063937832 (198h32m7.759786585s) [seg_pos={0 0}, upload=174963]: Done begin upload segment. Nov 7 18:12:36.063988689 (198h32m7.759837463s) [noise=true , address=145.79.227.27:4690 , piece_id=7CVNJTAEBVYJXRWNSFYFP3E7SKMY5XI4CV2JEZJLSXH4TA2F2JZQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.064083510 (198h32m7.759932263s) [noise=true , address=145.79.227.10:4403 , piece_id=JCMXNJ5PW5ULGOJZ5SGB423VW7VJPC5REYNSNWJZJXHGUQXUS3KA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.064896969 (198h32m7.760745762s) [noise=true , address=145.79.227.28:4726 , piece_id=D6DH5FHFB5ZTPANRPUWZKRYPFYUV2LS6AY5FQL5RBJZIUNM4FKLQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.065443520 (198h32m7.761292273s) [noise=true , address=77.90.31.11:4554 , piece_id=TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.065970974 (198h32m7.761819777s) [noise=true , address=145.79.227.8:4330 , piece_id=32XBCOEGSUTVUCCWVM2D7CB6FKMNLD6N2N4655B3ZYHNAIL4R2QQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.066422053 (198h32m7.762270806s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.067054427 (198h32m7.76290318s) [noise=true , address=77.90.31.10:4519 , piece_id=MIHLKGHCRTUFQSGBIFKIWPNACAJ4CQOEYRDIKA4KJLFP356VMY6A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.067645042 (198h32m7.763493795s) [noise=true , address=45.91.116.6:4973 , piece_id=Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.068379079 (198h32m7.764227842s) [noise=true , address=77.90.31.19:4843 , piece_id=JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.070931099 (198h32m7.766779862s) [noise=true , address=145.79.227.11:4732 , piece_id=VKSPYLTD6C6XBUFXXVVPHIPGFRCRNQK6EVI4SM7Y6ODOMVGGAJOQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.073100420 (198h32m7.768949173s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.075821872 (198h32m7.771670625s) [noise=true , address=45.91.116.4:4937 , piece_id=CWDA2IYNFUYHVDBJ74Y3CJKCZUBN4QYUEJYFKZNIBAUYYAXO3WFQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.078400863 (198h32m7.774249626s) [noise=true , address=45.91.116.10:4082 , piece_id=RW62GCPC56K2IMOFF276GQECXRKW6NY6W44TNAYFNDGQS3UK7EEA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.081230762 (198h32m7.777079515s) [noise=true , address=145.79.227.15:4876 , piece_id=ZPGKGNXRFEWRRJKTS7NKBEVQA7OAWNWYDR3DUTPHT3XVYRJSKWPQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.083298258 (198h32m7.779147012s) [noise=true , address=145.79.227.20:4438 , piece_id=V6WDUVDB7WHTDCTSYCFWS22UKRD5NMXGJ2GHLSFLVVOXZMZWLL4A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.084602091 (198h32m7.780450844s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:37.640211006 (198h32m9.336059759s) [upload=174963]: Next returned nil segment Nov 7 18:12:37.640227878 (198h32m9.336076621s) [upload=174963]: Waiting for error group managing segments... Nov 7 18:12:38.007509819 (198h32m9.703358572s) [noise=true , address=145.79.227.10:4403 , piece_id=JCMXNJ5PW5ULGOJZ5SGB423VW7VJPC5REYNSNWJZJXHGUQXUS3KA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.015182750 (198h32m9.711031503s) [noise=true , address=145.79.227.28:4726 , piece_id=D6DH5FHFB5ZTPANRPUWZKRYPFYUV2LS6AY5FQL5RBJZIUNM4FKLQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.020400886 (198h32m9.71624964s) [noise=true , address=145.79.227.27:4690 , piece_id=7CVNJTAEBVYJXRWNSFYFP3E7SKMY5XI4CV2JEZJLSXH4TA2F2JZQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.062361869 (198h32m9.758210622s) [noise=true , address=145.79.227.15:4876 , piece_id=ZPGKGNXRFEWRRJKTS7NKBEVQA7OAWNWYDR3DUTPHT3XVYRJSKWPQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.096654439 (198h32m9.792503192s) [noise=true , address=145.79.227.20:4438 , piece_id=V6WDUVDB7WHTDCTSYCFWS22UKRD5NMXGJ2GHLSFLVVOXZMZWLL4A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.244365682 (198h32m9.940214435s) [noise=true , address=145.79.227.8:4330 , piece_id=32XBCOEGSUTVUCCWVM2D7CB6FKMNLD6N2N4655B3ZYHNAIL4R2QQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.245114297 (198h32m9.94096305s) [noise=true , address=145.79.227.11:4732 , piece_id=VKSPYLTD6C6XBUFXXVVPHIPGFRCRNQK6EVI4SM7Y6ODOMVGGAJOQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.324356281 (198h32m10.020205034s) [noise=true , address=77.90.31.11:4554 , piece_id=TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.331696878 (198h32m10.027545621s) [noise=true , address=45.91.116.4:4937 , piece_id=CWDA2IYNFUYHVDBJ74Y3CJKCZUBN4QYUEJYFKZNIBAUYYAXO3WFQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.334142676 (198h32m10.029991429s) [noise=true , address=45.91.116.10:4082 , piece_id=RW62GCPC56K2IMOFF276GQECXRKW6NY6W44TNAYFNDGQS3UK7EEA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.346539031 (198h32m10.042387794s) [noise=true , address=77.90.31.10:4519 , piece_id=MIHLKGHCRTUFQSGBIFKIWPNACAJ4CQOEYRDIKA4KJLFP356VMY6A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.419504032 (198h32m10.115352785s) [noise=true , address=77.90.31.19:4843 , piece_id=JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:39.113150499 (198h32m10.808999252s) [noise=true , address=45.91.116.6:4973 , piece_id=Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:39.113177150 (198h32m10.809025903s) [seg_pos={0 0}, upload=174963]: Segment reached optimal threshold of 13 pieces. Nov 7 18:12:39.113381018 (198h32m10.809229772s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113390166 (198h32m10.809238919s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113460199 (198h32m10.809308952s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113470077 (198h32m10.809318821s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113780338 (198h32m10.809629091s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113788243 (198h32m10.809636986s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113858657 (198h32m10.8097074s) [seg_pos={0 0}, upload=174963]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 7 18:12:39.114030194 (198h32m10.809878947s) [upload=174963]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 7 18:12:39.114038290 (198h32m10.809887043s) [upload=174963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 7 18:12:39.451448640 (198h32m11.147297403s) [upload=174963]: Done uploading segments... Nov 7 18:12:39.451473607 (198h32m11.14732236s) [upload=174963]: Upload finished. err: <nil> Duration: 6.130516612s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 28 out of 60 is failed (threshold=3) [XXXX..XXXX.X..XXX.X..X.X.XXXX.X..XX.XXXXX.X.................] duration: 6.06247127s trace: https://jaeger.dev.storj.io/trace/422f33ac8ae6a74a FAIL (2025-11-07 18:26:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ) context deadline exceeded; piecestore: (Node ID: 12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, Piece ID: JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ) context deadline exceeded ======LOGS====== Nov 7 18:12:35.791697270 (198h32m7.487546023s) [upload=174963]: Starting upload Nov 7 18:12:35.791791689 (198h32m7.487640452s) [upload=174963]: Done starting upload Nov 7 18:12:35.791885458 (198h32m7.487734211s) [upload=174963]: Uploading segments... Nov 7 18:12:35.792146455 (198h32m7.487995208s) [upload=174963]: Got next segment. Inline: false Nov 7 18:12:35.792195278 (198h32m7.488044021s) [upload=174963]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 7 18:12:35.792201820 (198h32m7.488050563s) [upload=174963]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 7 18:12:36.063867588 (198h32m7.759716341s) [seg_pos={0 0}, upload=174963]: Begin upload segment... Nov 7 18:12:36.063937832 (198h32m7.759786585s) [seg_pos={0 0}, upload=174963]: Done begin upload segment. Nov 7 18:12:36.063988689 (198h32m7.759837463s) [noise=true , address=145.79.227.27:4690 , piece_id=7CVNJTAEBVYJXRWNSFYFP3E7SKMY5XI4CV2JEZJLSXH4TA2F2JZQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.064083510 (198h32m7.759932263s) [noise=true , address=145.79.227.10:4403 , piece_id=JCMXNJ5PW5ULGOJZ5SGB423VW7VJPC5REYNSNWJZJXHGUQXUS3KA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.064896969 (198h32m7.760745762s) [noise=true , address=145.79.227.28:4726 , piece_id=D6DH5FHFB5ZTPANRPUWZKRYPFYUV2LS6AY5FQL5RBJZIUNM4FKLQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.065443520 (198h32m7.761292273s) [noise=true , address=77.90.31.11:4554 , piece_id=TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.065970974 (198h32m7.761819777s) [noise=true , address=145.79.227.8:4330 , piece_id=32XBCOEGSUTVUCCWVM2D7CB6FKMNLD6N2N4655B3ZYHNAIL4R2QQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.066422053 (198h32m7.762270806s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.067054427 (198h32m7.76290318s) [noise=true , address=77.90.31.10:4519 , piece_id=MIHLKGHCRTUFQSGBIFKIWPNACAJ4CQOEYRDIKA4KJLFP356VMY6A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.067645042 (198h32m7.763493795s) [noise=true , address=45.91.116.6:4973 , piece_id=Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.068379079 (198h32m7.764227842s) [noise=true , address=77.90.31.19:4843 , piece_id=JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.070931099 (198h32m7.766779862s) [noise=true , address=145.79.227.11:4732 , piece_id=VKSPYLTD6C6XBUFXXVVPHIPGFRCRNQK6EVI4SM7Y6ODOMVGGAJOQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.073100420 (198h32m7.768949173s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.075821872 (198h32m7.771670625s) [noise=true , address=45.91.116.4:4937 , piece_id=CWDA2IYNFUYHVDBJ74Y3CJKCZUBN4QYUEJYFKZNIBAUYYAXO3WFQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.078400863 (198h32m7.774249626s) [noise=true , address=45.91.116.10:4082 , piece_id=RW62GCPC56K2IMOFF276GQECXRKW6NY6W44TNAYFNDGQS3UK7EEA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.081230762 (198h32m7.777079515s) [noise=true , address=145.79.227.15:4876 , piece_id=ZPGKGNXRFEWRRJKTS7NKBEVQA7OAWNWYDR3DUTPHT3XVYRJSKWPQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.083298258 (198h32m7.779147012s) [noise=true , address=145.79.227.20:4438 , piece_id=V6WDUVDB7WHTDCTSYCFWS22UKRD5NMXGJ2GHLSFLVVOXZMZWLL4A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.084602091 (198h32m7.780450844s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:37.640211006 (198h32m9.336059759s) [upload=174963]: Next returned nil segment Nov 7 18:12:37.640227878 (198h32m9.336076621s) [upload=174963]: Waiting for error group managing segments... Nov 7 18:12:38.007509819 (198h32m9.703358572s) [noise=true , address=145.79.227.10:4403 , piece_id=JCMXNJ5PW5ULGOJZ5SGB423VW7VJPC5REYNSNWJZJXHGUQXUS3KA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.015182750 (198h32m9.711031503s) [noise=true , address=145.79.227.28:4726 , piece_id=D6DH5FHFB5ZTPANRPUWZKRYPFYUV2LS6AY5FQL5RBJZIUNM4FKLQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.020400886 (198h32m9.71624964s) [noise=true , address=145.79.227.27:4690 , piece_id=7CVNJTAEBVYJXRWNSFYFP3E7SKMY5XI4CV2JEZJLSXH4TA2F2JZQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.062361869 (198h32m9.758210622s) [noise=true , address=145.79.227.15:4876 , piece_id=ZPGKGNXRFEWRRJKTS7NKBEVQA7OAWNWYDR3DUTPHT3XVYRJSKWPQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.096654439 (198h32m9.792503192s) [noise=true , address=145.79.227.20:4438 , piece_id=V6WDUVDB7WHTDCTSYCFWS22UKRD5NMXGJ2GHLSFLVVOXZMZWLL4A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.244365682 (198h32m9.940214435s) [noise=true , address=145.79.227.8:4330 , piece_id=32XBCOEGSUTVUCCWVM2D7CB6FKMNLD6N2N4655B3ZYHNAIL4R2QQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.245114297 (198h32m9.94096305s) [noise=true , address=145.79.227.11:4732 , piece_id=VKSPYLTD6C6XBUFXXVVPHIPGFRCRNQK6EVI4SM7Y6ODOMVGGAJOQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.324356281 (198h32m10.020205034s) [noise=true , address=77.90.31.11:4554 , piece_id=TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.331696878 (198h32m10.027545621s) [noise=true , address=45.91.116.4:4937 , piece_id=CWDA2IYNFUYHVDBJ74Y3CJKCZUBN4QYUEJYFKZNIBAUYYAXO3WFQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.334142676 (198h32m10.029991429s) [noise=true , address=45.91.116.10:4082 , piece_id=RW62GCPC56K2IMOFF276GQECXRKW6NY6W44TNAYFNDGQS3UK7EEA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.346539031 (198h32m10.042387794s) [noise=true , address=77.90.31.10:4519 , piece_id=MIHLKGHCRTUFQSGBIFKIWPNACAJ4CQOEYRDIKA4KJLFP356VMY6A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.419504032 (198h32m10.115352785s) [noise=true , address=77.90.31.19:4843 , piece_id=JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:39.113150499 (198h32m10.808999252s) [noise=true , address=45.91.116.6:4973 , piece_id=Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:39.113177150 (198h32m10.809025903s) [seg_pos={0 0}, upload=174963]: Segment reached optimal threshold of 13 pieces. Nov 7 18:12:39.113381018 (198h32m10.809229772s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113390166 (198h32m10.809238919s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113460199 (198h32m10.809308952s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113470077 (198h32m10.809318821s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113780338 (198h32m10.809629091s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113788243 (198h32m10.809636986s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113858657 (198h32m10.8097074s) [seg_pos={0 0}, upload=174963]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 7 18:12:39.114030194 (198h32m10.809878947s) [upload=174963]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 7 18:12:39.114038290 (198h32m10.809887043s) [upload=174963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 7 18:12:39.451448640 (198h32m11.147297403s) [upload=174963]: Done uploading segments... Nov 7 18:12:39.451473607 (198h32m11.14732236s) [upload=174963]: Upload finished. err: <nil> Duration: 6.062498453s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 29 out of 60 is failed (threshold=3) [XXXXX..XXXX.X..XXX.X..X.X.XXXX.X..XX.XXXXX.X................] duration: 6.029354717s trace: https://jaeger.dev.storj.io/trace/422f33ac8ae6a74a FAIL (2025-11-07 18:25:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ) context deadline exceeded; piecestore: (Node ID: 12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, Piece ID: JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ) context deadline exceeded ======LOGS====== Nov 7 18:12:35.791697270 (198h32m7.487546023s) [upload=174963]: Starting upload Nov 7 18:12:35.791791689 (198h32m7.487640452s) [upload=174963]: Done starting upload Nov 7 18:12:35.791885458 (198h32m7.487734211s) [upload=174963]: Uploading segments... Nov 7 18:12:35.792146455 (198h32m7.487995208s) [upload=174963]: Got next segment. Inline: false Nov 7 18:12:35.792195278 (198h32m7.488044021s) [upload=174963]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 7 18:12:35.792201820 (198h32m7.488050563s) [upload=174963]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 7 18:12:36.063867588 (198h32m7.759716341s) [seg_pos={0 0}, upload=174963]: Begin upload segment... Nov 7 18:12:36.063937832 (198h32m7.759786585s) [seg_pos={0 0}, upload=174963]: Done begin upload segment. Nov 7 18:12:36.063988689 (198h32m7.759837463s) [noise=true , address=145.79.227.27:4690 , piece_id=7CVNJTAEBVYJXRWNSFYFP3E7SKMY5XI4CV2JEZJLSXH4TA2F2JZQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.064083510 (198h32m7.759932263s) [noise=true , address=145.79.227.10:4403 , piece_id=JCMXNJ5PW5ULGOJZ5SGB423VW7VJPC5REYNSNWJZJXHGUQXUS3KA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.064896969 (198h32m7.760745762s) [noise=true , address=145.79.227.28:4726 , piece_id=D6DH5FHFB5ZTPANRPUWZKRYPFYUV2LS6AY5FQL5RBJZIUNM4FKLQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.065443520 (198h32m7.761292273s) [noise=true , address=77.90.31.11:4554 , piece_id=TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.065970974 (198h32m7.761819777s) [noise=true , address=145.79.227.8:4330 , piece_id=32XBCOEGSUTVUCCWVM2D7CB6FKMNLD6N2N4655B3ZYHNAIL4R2QQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.066422053 (198h32m7.762270806s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.067054427 (198h32m7.76290318s) [noise=true , address=77.90.31.10:4519 , piece_id=MIHLKGHCRTUFQSGBIFKIWPNACAJ4CQOEYRDIKA4KJLFP356VMY6A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.067645042 (198h32m7.763493795s) [noise=true , address=45.91.116.6:4973 , piece_id=Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.068379079 (198h32m7.764227842s) [noise=true , address=77.90.31.19:4843 , piece_id=JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.070931099 (198h32m7.766779862s) [noise=true , address=145.79.227.11:4732 , piece_id=VKSPYLTD6C6XBUFXXVVPHIPGFRCRNQK6EVI4SM7Y6ODOMVGGAJOQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.073100420 (198h32m7.768949173s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.075821872 (198h32m7.771670625s) [noise=true , address=45.91.116.4:4937 , piece_id=CWDA2IYNFUYHVDBJ74Y3CJKCZUBN4QYUEJYFKZNIBAUYYAXO3WFQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.078400863 (198h32m7.774249626s) [noise=true , address=45.91.116.10:4082 , piece_id=RW62GCPC56K2IMOFF276GQECXRKW6NY6W44TNAYFNDGQS3UK7EEA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.081230762 (198h32m7.777079515s) [noise=true , address=145.79.227.15:4876 , piece_id=ZPGKGNXRFEWRRJKTS7NKBEVQA7OAWNWYDR3DUTPHT3XVYRJSKWPQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.083298258 (198h32m7.779147012s) [noise=true , address=145.79.227.20:4438 , piece_id=V6WDUVDB7WHTDCTSYCFWS22UKRD5NMXGJ2GHLSFLVVOXZMZWLL4A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.084602091 (198h32m7.780450844s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:37.640211006 (198h32m9.336059759s) [upload=174963]: Next returned nil segment Nov 7 18:12:37.640227878 (198h32m9.336076621s) [upload=174963]: Waiting for error group managing segments... Nov 7 18:12:38.007509819 (198h32m9.703358572s) [noise=true , address=145.79.227.10:4403 , piece_id=JCMXNJ5PW5ULGOJZ5SGB423VW7VJPC5REYNSNWJZJXHGUQXUS3KA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.015182750 (198h32m9.711031503s) [noise=true , address=145.79.227.28:4726 , piece_id=D6DH5FHFB5ZTPANRPUWZKRYPFYUV2LS6AY5FQL5RBJZIUNM4FKLQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.020400886 (198h32m9.71624964s) [noise=true , address=145.79.227.27:4690 , piece_id=7CVNJTAEBVYJXRWNSFYFP3E7SKMY5XI4CV2JEZJLSXH4TA2F2JZQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.062361869 (198h32m9.758210622s) [noise=true , address=145.79.227.15:4876 , piece_id=ZPGKGNXRFEWRRJKTS7NKBEVQA7OAWNWYDR3DUTPHT3XVYRJSKWPQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.096654439 (198h32m9.792503192s) [noise=true , address=145.79.227.20:4438 , piece_id=V6WDUVDB7WHTDCTSYCFWS22UKRD5NMXGJ2GHLSFLVVOXZMZWLL4A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.244365682 (198h32m9.940214435s) [noise=true , address=145.79.227.8:4330 , piece_id=32XBCOEGSUTVUCCWVM2D7CB6FKMNLD6N2N4655B3ZYHNAIL4R2QQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.245114297 (198h32m9.94096305s) [noise=true , address=145.79.227.11:4732 , piece_id=VKSPYLTD6C6XBUFXXVVPHIPGFRCRNQK6EVI4SM7Y6ODOMVGGAJOQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.324356281 (198h32m10.020205034s) [noise=true , address=77.90.31.11:4554 , piece_id=TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.331696878 (198h32m10.027545621s) [noise=true , address=45.91.116.4:4937 , piece_id=CWDA2IYNFUYHVDBJ74Y3CJKCZUBN4QYUEJYFKZNIBAUYYAXO3WFQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.334142676 (198h32m10.029991429s) [noise=true , address=45.91.116.10:4082 , piece_id=RW62GCPC56K2IMOFF276GQECXRKW6NY6W44TNAYFNDGQS3UK7EEA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.346539031 (198h32m10.042387794s) [noise=true , address=77.90.31.10:4519 , piece_id=MIHLKGHCRTUFQSGBIFKIWPNACAJ4CQOEYRDIKA4KJLFP356VMY6A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.419504032 (198h32m10.115352785s) [noise=true , address=77.90.31.19:4843 , piece_id=JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:39.113150499 (198h32m10.808999252s) [noise=true , address=45.91.116.6:4973 , piece_id=Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:39.113177150 (198h32m10.809025903s) [seg_pos={0 0}, upload=174963]: Segment reached optimal threshold of 13 pieces. Nov 7 18:12:39.113381018 (198h32m10.809229772s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113390166 (198h32m10.809238919s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113460199 (198h32m10.809308952s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113470077 (198h32m10.809318821s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113780338 (198h32m10.809629091s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113788243 (198h32m10.809636986s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113858657 (198h32m10.8097074s) [seg_pos={0 0}, upload=174963]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 7 18:12:39.114030194 (198h32m10.809878947s) [upload=174963]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 7 18:12:39.114038290 (198h32m10.809887043s) [upload=174963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 7 18:12:39.451448640 (198h32m11.147297403s) [upload=174963]: Done uploading segments... Nov 7 18:12:39.451473607 (198h32m11.14732236s) [upload=174963]: Upload finished. err: <nil> Duration: 6.029391107s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 30 out of 60 is failed (threshold=3) [XXXXXX..XXXX.X..XXX.X..X.X.XXXX.X..XX.XXXXX.X...............] duration: 6.190495391s trace: https://jaeger.dev.storj.io/trace/422f33ac8ae6a74a FAIL (2025-11-07 18:24:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ) context deadline exceeded; piecestore: (Node ID: 12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, Piece ID: JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ) context deadline exceeded ======LOGS====== Nov 7 18:12:35.791697270 (198h32m7.487546023s) [upload=174963]: Starting upload Nov 7 18:12:35.791791689 (198h32m7.487640452s) [upload=174963]: Done starting upload Nov 7 18:12:35.791885458 (198h32m7.487734211s) [upload=174963]: Uploading segments... Nov 7 18:12:35.792146455 (198h32m7.487995208s) [upload=174963]: Got next segment. Inline: false Nov 7 18:12:35.792195278 (198h32m7.488044021s) [upload=174963]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 7 18:12:35.792201820 (198h32m7.488050563s) [upload=174963]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 7 18:12:36.063867588 (198h32m7.759716341s) [seg_pos={0 0}, upload=174963]: Begin upload segment... Nov 7 18:12:36.063937832 (198h32m7.759786585s) [seg_pos={0 0}, upload=174963]: Done begin upload segment. Nov 7 18:12:36.063988689 (198h32m7.759837463s) [noise=true , address=145.79.227.27:4690 , piece_id=7CVNJTAEBVYJXRWNSFYFP3E7SKMY5XI4CV2JEZJLSXH4TA2F2JZQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.064083510 (198h32m7.759932263s) [noise=true , address=145.79.227.10:4403 , piece_id=JCMXNJ5PW5ULGOJZ5SGB423VW7VJPC5REYNSNWJZJXHGUQXUS3KA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.064896969 (198h32m7.760745762s) [noise=true , address=145.79.227.28:4726 , piece_id=D6DH5FHFB5ZTPANRPUWZKRYPFYUV2LS6AY5FQL5RBJZIUNM4FKLQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.065443520 (198h32m7.761292273s) [noise=true , address=77.90.31.11:4554 , piece_id=TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.065970974 (198h32m7.761819777s) [noise=true , address=145.79.227.8:4330 , piece_id=32XBCOEGSUTVUCCWVM2D7CB6FKMNLD6N2N4655B3ZYHNAIL4R2QQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.066422053 (198h32m7.762270806s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.067054427 (198h32m7.76290318s) [noise=true , address=77.90.31.10:4519 , piece_id=MIHLKGHCRTUFQSGBIFKIWPNACAJ4CQOEYRDIKA4KJLFP356VMY6A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.067645042 (198h32m7.763493795s) [noise=true , address=45.91.116.6:4973 , piece_id=Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.068379079 (198h32m7.764227842s) [noise=true , address=77.90.31.19:4843 , piece_id=JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.070931099 (198h32m7.766779862s) [noise=true , address=145.79.227.11:4732 , piece_id=VKSPYLTD6C6XBUFXXVVPHIPGFRCRNQK6EVI4SM7Y6ODOMVGGAJOQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.073100420 (198h32m7.768949173s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.075821872 (198h32m7.771670625s) [noise=true , address=45.91.116.4:4937 , piece_id=CWDA2IYNFUYHVDBJ74Y3CJKCZUBN4QYUEJYFKZNIBAUYYAXO3WFQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.078400863 (198h32m7.774249626s) [noise=true , address=45.91.116.10:4082 , piece_id=RW62GCPC56K2IMOFF276GQECXRKW6NY6W44TNAYFNDGQS3UK7EEA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.081230762 (198h32m7.777079515s) [noise=true , address=145.79.227.15:4876 , piece_id=ZPGKGNXRFEWRRJKTS7NKBEVQA7OAWNWYDR3DUTPHT3XVYRJSKWPQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.083298258 (198h32m7.779147012s) [noise=true , address=145.79.227.20:4438 , piece_id=V6WDUVDB7WHTDCTSYCFWS22UKRD5NMXGJ2GHLSFLVVOXZMZWLL4A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.084602091 (198h32m7.780450844s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:37.640211006 (198h32m9.336059759s) [upload=174963]: Next returned nil segment Nov 7 18:12:37.640227878 (198h32m9.336076621s) [upload=174963]: Waiting for error group managing segments... Nov 7 18:12:38.007509819 (198h32m9.703358572s) [noise=true , address=145.79.227.10:4403 , piece_id=JCMXNJ5PW5ULGOJZ5SGB423VW7VJPC5REYNSNWJZJXHGUQXUS3KA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.015182750 (198h32m9.711031503s) [noise=true , address=145.79.227.28:4726 , piece_id=D6DH5FHFB5ZTPANRPUWZKRYPFYUV2LS6AY5FQL5RBJZIUNM4FKLQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.020400886 (198h32m9.71624964s) [noise=true , address=145.79.227.27:4690 , piece_id=7CVNJTAEBVYJXRWNSFYFP3E7SKMY5XI4CV2JEZJLSXH4TA2F2JZQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.062361869 (198h32m9.758210622s) [noise=true , address=145.79.227.15:4876 , piece_id=ZPGKGNXRFEWRRJKTS7NKBEVQA7OAWNWYDR3DUTPHT3XVYRJSKWPQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.096654439 (198h32m9.792503192s) [noise=true , address=145.79.227.20:4438 , piece_id=V6WDUVDB7WHTDCTSYCFWS22UKRD5NMXGJ2GHLSFLVVOXZMZWLL4A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.244365682 (198h32m9.940214435s) [noise=true , address=145.79.227.8:4330 , piece_id=32XBCOEGSUTVUCCWVM2D7CB6FKMNLD6N2N4655B3ZYHNAIL4R2QQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.245114297 (198h32m9.94096305s) [noise=true , address=145.79.227.11:4732 , piece_id=VKSPYLTD6C6XBUFXXVVPHIPGFRCRNQK6EVI4SM7Y6ODOMVGGAJOQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.324356281 (198h32m10.020205034s) [noise=true , address=77.90.31.11:4554 , piece_id=TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.331696878 (198h32m10.027545621s) [noise=true , address=45.91.116.4:4937 , piece_id=CWDA2IYNFUYHVDBJ74Y3CJKCZUBN4QYUEJYFKZNIBAUYYAXO3WFQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.334142676 (198h32m10.029991429s) [noise=true , address=45.91.116.10:4082 , piece_id=RW62GCPC56K2IMOFF276GQECXRKW6NY6W44TNAYFNDGQS3UK7EEA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.346539031 (198h32m10.042387794s) [noise=true , address=77.90.31.10:4519 , piece_id=MIHLKGHCRTUFQSGBIFKIWPNACAJ4CQOEYRDIKA4KJLFP356VMY6A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.419504032 (198h32m10.115352785s) [noise=true , address=77.90.31.19:4843 , piece_id=JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:39.113150499 (198h32m10.808999252s) [noise=true , address=45.91.116.6:4973 , piece_id=Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:39.113177150 (198h32m10.809025903s) [seg_pos={0 0}, upload=174963]: Segment reached optimal threshold of 13 pieces. Nov 7 18:12:39.113381018 (198h32m10.809229772s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113390166 (198h32m10.809238919s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113460199 (198h32m10.809308952s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113470077 (198h32m10.809318821s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113780338 (198h32m10.809629091s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113788243 (198h32m10.809636986s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113858657 (198h32m10.8097074s) [seg_pos={0 0}, upload=174963]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 7 18:12:39.114030194 (198h32m10.809878947s) [upload=174963]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 7 18:12:39.114038290 (198h32m10.809887043s) [upload=174963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 7 18:12:39.451448640 (198h32m11.147297403s) [upload=174963]: Done uploading segments... Nov 7 18:12:39.451473607 (198h32m11.14732236s) [upload=174963]: Upload finished. err: <nil> Duration: 6.19052188s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 31 out of 60 is failed (threshold=3) [XXXXXXX..XXXX.X..XXX.X..X.X.XXXX.X..XX.XXXXX.X..............] duration: 6.172785624s trace: https://jaeger.dev.storj.io/trace/422f33ac8ae6a74a FAIL (2025-11-07 18:23:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ) context deadline exceeded; piecestore: (Node ID: 12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, Piece ID: JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ) context deadline exceeded ======LOGS====== Nov 7 18:12:35.791697270 (198h32m7.487546023s) [upload=174963]: Starting upload Nov 7 18:12:35.791791689 (198h32m7.487640452s) [upload=174963]: Done starting upload Nov 7 18:12:35.791885458 (198h32m7.487734211s) [upload=174963]: Uploading segments... Nov 7 18:12:35.792146455 (198h32m7.487995208s) [upload=174963]: Got next segment. Inline: false Nov 7 18:12:35.792195278 (198h32m7.488044021s) [upload=174963]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 7 18:12:35.792201820 (198h32m7.488050563s) [upload=174963]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 7 18:12:36.063867588 (198h32m7.759716341s) [seg_pos={0 0}, upload=174963]: Begin upload segment... Nov 7 18:12:36.063937832 (198h32m7.759786585s) [seg_pos={0 0}, upload=174963]: Done begin upload segment. Nov 7 18:12:36.063988689 (198h32m7.759837463s) [noise=true , address=145.79.227.27:4690 , piece_id=7CVNJTAEBVYJXRWNSFYFP3E7SKMY5XI4CV2JEZJLSXH4TA2F2JZQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.064083510 (198h32m7.759932263s) [noise=true , address=145.79.227.10:4403 , piece_id=JCMXNJ5PW5ULGOJZ5SGB423VW7VJPC5REYNSNWJZJXHGUQXUS3KA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.064896969 (198h32m7.760745762s) [noise=true , address=145.79.227.28:4726 , piece_id=D6DH5FHFB5ZTPANRPUWZKRYPFYUV2LS6AY5FQL5RBJZIUNM4FKLQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.065443520 (198h32m7.761292273s) [noise=true , address=77.90.31.11:4554 , piece_id=TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.065970974 (198h32m7.761819777s) [noise=true , address=145.79.227.8:4330 , piece_id=32XBCOEGSUTVUCCWVM2D7CB6FKMNLD6N2N4655B3ZYHNAIL4R2QQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.066422053 (198h32m7.762270806s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.067054427 (198h32m7.76290318s) [noise=true , address=77.90.31.10:4519 , piece_id=MIHLKGHCRTUFQSGBIFKIWPNACAJ4CQOEYRDIKA4KJLFP356VMY6A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.067645042 (198h32m7.763493795s) [noise=true , address=45.91.116.6:4973 , piece_id=Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.068379079 (198h32m7.764227842s) [noise=true , address=77.90.31.19:4843 , piece_id=JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.070931099 (198h32m7.766779862s) [noise=true , address=145.79.227.11:4732 , piece_id=VKSPYLTD6C6XBUFXXVVPHIPGFRCRNQK6EVI4SM7Y6ODOMVGGAJOQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.073100420 (198h32m7.768949173s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.075821872 (198h32m7.771670625s) [noise=true , address=45.91.116.4:4937 , piece_id=CWDA2IYNFUYHVDBJ74Y3CJKCZUBN4QYUEJYFKZNIBAUYYAXO3WFQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.078400863 (198h32m7.774249626s) [noise=true , address=45.91.116.10:4082 , piece_id=RW62GCPC56K2IMOFF276GQECXRKW6NY6W44TNAYFNDGQS3UK7EEA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.081230762 (198h32m7.777079515s) [noise=true , address=145.79.227.15:4876 , piece_id=ZPGKGNXRFEWRRJKTS7NKBEVQA7OAWNWYDR3DUTPHT3XVYRJSKWPQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.083298258 (198h32m7.779147012s) [noise=true , address=145.79.227.20:4438 , piece_id=V6WDUVDB7WHTDCTSYCFWS22UKRD5NMXGJ2GHLSFLVVOXZMZWLL4A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.084602091 (198h32m7.780450844s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:37.640211006 (198h32m9.336059759s) [upload=174963]: Next returned nil segment Nov 7 18:12:37.640227878 (198h32m9.336076621s) [upload=174963]: Waiting for error group managing segments... Nov 7 18:12:38.007509819 (198h32m9.703358572s) [noise=true , address=145.79.227.10:4403 , piece_id=JCMXNJ5PW5ULGOJZ5SGB423VW7VJPC5REYNSNWJZJXHGUQXUS3KA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.015182750 (198h32m9.711031503s) [noise=true , address=145.79.227.28:4726 , piece_id=D6DH5FHFB5ZTPANRPUWZKRYPFYUV2LS6AY5FQL5RBJZIUNM4FKLQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.020400886 (198h32m9.71624964s) [noise=true , address=145.79.227.27:4690 , piece_id=7CVNJTAEBVYJXRWNSFYFP3E7SKMY5XI4CV2JEZJLSXH4TA2F2JZQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.062361869 (198h32m9.758210622s) [noise=true , address=145.79.227.15:4876 , piece_id=ZPGKGNXRFEWRRJKTS7NKBEVQA7OAWNWYDR3DUTPHT3XVYRJSKWPQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.096654439 (198h32m9.792503192s) [noise=true , address=145.79.227.20:4438 , piece_id=V6WDUVDB7WHTDCTSYCFWS22UKRD5NMXGJ2GHLSFLVVOXZMZWLL4A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.244365682 (198h32m9.940214435s) [noise=true , address=145.79.227.8:4330 , piece_id=32XBCOEGSUTVUCCWVM2D7CB6FKMNLD6N2N4655B3ZYHNAIL4R2QQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.245114297 (198h32m9.94096305s) [noise=true , address=145.79.227.11:4732 , piece_id=VKSPYLTD6C6XBUFXXVVPHIPGFRCRNQK6EVI4SM7Y6ODOMVGGAJOQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.324356281 (198h32m10.020205034s) [noise=true , address=77.90.31.11:4554 , piece_id=TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.331696878 (198h32m10.027545621s) [noise=true , address=45.91.116.4:4937 , piece_id=CWDA2IYNFUYHVDBJ74Y3CJKCZUBN4QYUEJYFKZNIBAUYYAXO3WFQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.334142676 (198h32m10.029991429s) [noise=true , address=45.91.116.10:4082 , piece_id=RW62GCPC56K2IMOFF276GQECXRKW6NY6W44TNAYFNDGQS3UK7EEA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.346539031 (198h32m10.042387794s) [noise=true , address=77.90.31.10:4519 , piece_id=MIHLKGHCRTUFQSGBIFKIWPNACAJ4CQOEYRDIKA4KJLFP356VMY6A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.419504032 (198h32m10.115352785s) [noise=true , address=77.90.31.19:4843 , piece_id=JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:39.113150499 (198h32m10.808999252s) [noise=true , address=45.91.116.6:4973 , piece_id=Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:39.113177150 (198h32m10.809025903s) [seg_pos={0 0}, upload=174963]: Segment reached optimal threshold of 13 pieces. Nov 7 18:12:39.113381018 (198h32m10.809229772s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113390166 (198h32m10.809238919s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113460199 (198h32m10.809308952s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113470077 (198h32m10.809318821s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113780338 (198h32m10.809629091s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113788243 (198h32m10.809636986s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113858657 (198h32m10.8097074s) [seg_pos={0 0}, upload=174963]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 7 18:12:39.114030194 (198h32m10.809878947s) [upload=174963]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 7 18:12:39.114038290 (198h32m10.809887043s) [upload=174963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 7 18:12:39.451448640 (198h32m11.147297403s) [upload=174963]: Done uploading segments... Nov 7 18:12:39.451473607 (198h32m11.14732236s) [upload=174963]: Upload finished. err: <nil> Duration: 6.172816834s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 32 out of 60 is failed (threshold=3) [XXXXXXXX..XXXX.X..XXX.X..X.X.XXXX.X..XX.XXXXX.X.............] duration: 6.17201635s trace: https://jaeger.dev.storj.io/trace/422f33ac8ae6a74a FAIL (2025-11-07 18:22:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ) context deadline exceeded; piecestore: (Node ID: 12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, Piece ID: JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ) context deadline exceeded ======LOGS====== Nov 7 18:12:35.791697270 (198h32m7.487546023s) [upload=174963]: Starting upload Nov 7 18:12:35.791791689 (198h32m7.487640452s) [upload=174963]: Done starting upload Nov 7 18:12:35.791885458 (198h32m7.487734211s) [upload=174963]: Uploading segments... Nov 7 18:12:35.792146455 (198h32m7.487995208s) [upload=174963]: Got next segment. Inline: false Nov 7 18:12:35.792195278 (198h32m7.488044021s) [upload=174963]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 7 18:12:35.792201820 (198h32m7.488050563s) [upload=174963]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 7 18:12:36.063867588 (198h32m7.759716341s) [seg_pos={0 0}, upload=174963]: Begin upload segment... Nov 7 18:12:36.063937832 (198h32m7.759786585s) [seg_pos={0 0}, upload=174963]: Done begin upload segment. Nov 7 18:12:36.063988689 (198h32m7.759837463s) [noise=true , address=145.79.227.27:4690 , piece_id=7CVNJTAEBVYJXRWNSFYFP3E7SKMY5XI4CV2JEZJLSXH4TA2F2JZQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.064083510 (198h32m7.759932263s) [noise=true , address=145.79.227.10:4403 , piece_id=JCMXNJ5PW5ULGOJZ5SGB423VW7VJPC5REYNSNWJZJXHGUQXUS3KA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.064896969 (198h32m7.760745762s) [noise=true , address=145.79.227.28:4726 , piece_id=D6DH5FHFB5ZTPANRPUWZKRYPFYUV2LS6AY5FQL5RBJZIUNM4FKLQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.065443520 (198h32m7.761292273s) [noise=true , address=77.90.31.11:4554 , piece_id=TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.065970974 (198h32m7.761819777s) [noise=true , address=145.79.227.8:4330 , piece_id=32XBCOEGSUTVUCCWVM2D7CB6FKMNLD6N2N4655B3ZYHNAIL4R2QQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.066422053 (198h32m7.762270806s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.067054427 (198h32m7.76290318s) [noise=true , address=77.90.31.10:4519 , piece_id=MIHLKGHCRTUFQSGBIFKIWPNACAJ4CQOEYRDIKA4KJLFP356VMY6A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.067645042 (198h32m7.763493795s) [noise=true , address=45.91.116.6:4973 , piece_id=Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.068379079 (198h32m7.764227842s) [noise=true , address=77.90.31.19:4843 , piece_id=JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.070931099 (198h32m7.766779862s) [noise=true , address=145.79.227.11:4732 , piece_id=VKSPYLTD6C6XBUFXXVVPHIPGFRCRNQK6EVI4SM7Y6ODOMVGGAJOQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.073100420 (198h32m7.768949173s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.075821872 (198h32m7.771670625s) [noise=true , address=45.91.116.4:4937 , piece_id=CWDA2IYNFUYHVDBJ74Y3CJKCZUBN4QYUEJYFKZNIBAUYYAXO3WFQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.078400863 (198h32m7.774249626s) [noise=true , address=45.91.116.10:4082 , piece_id=RW62GCPC56K2IMOFF276GQECXRKW6NY6W44TNAYFNDGQS3UK7EEA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.081230762 (198h32m7.777079515s) [noise=true , address=145.79.227.15:4876 , piece_id=ZPGKGNXRFEWRRJKTS7NKBEVQA7OAWNWYDR3DUTPHT3XVYRJSKWPQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.083298258 (198h32m7.779147012s) [noise=true , address=145.79.227.20:4438 , piece_id=V6WDUVDB7WHTDCTSYCFWS22UKRD5NMXGJ2GHLSFLVVOXZMZWLL4A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.084602091 (198h32m7.780450844s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:37.640211006 (198h32m9.336059759s) [upload=174963]: Next returned nil segment Nov 7 18:12:37.640227878 (198h32m9.336076621s) [upload=174963]: Waiting for error group managing segments... Nov 7 18:12:38.007509819 (198h32m9.703358572s) [noise=true , address=145.79.227.10:4403 , piece_id=JCMXNJ5PW5ULGOJZ5SGB423VW7VJPC5REYNSNWJZJXHGUQXUS3KA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.015182750 (198h32m9.711031503s) [noise=true , address=145.79.227.28:4726 , piece_id=D6DH5FHFB5ZTPANRPUWZKRYPFYUV2LS6AY5FQL5RBJZIUNM4FKLQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.020400886 (198h32m9.71624964s) [noise=true , address=145.79.227.27:4690 , piece_id=7CVNJTAEBVYJXRWNSFYFP3E7SKMY5XI4CV2JEZJLSXH4TA2F2JZQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.062361869 (198h32m9.758210622s) [noise=true , address=145.79.227.15:4876 , piece_id=ZPGKGNXRFEWRRJKTS7NKBEVQA7OAWNWYDR3DUTPHT3XVYRJSKWPQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.096654439 (198h32m9.792503192s) [noise=true , address=145.79.227.20:4438 , piece_id=V6WDUVDB7WHTDCTSYCFWS22UKRD5NMXGJ2GHLSFLVVOXZMZWLL4A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.244365682 (198h32m9.940214435s) [noise=true , address=145.79.227.8:4330 , piece_id=32XBCOEGSUTVUCCWVM2D7CB6FKMNLD6N2N4655B3ZYHNAIL4R2QQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.245114297 (198h32m9.94096305s) [noise=true , address=145.79.227.11:4732 , piece_id=VKSPYLTD6C6XBUFXXVVPHIPGFRCRNQK6EVI4SM7Y6ODOMVGGAJOQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.324356281 (198h32m10.020205034s) [noise=true , address=77.90.31.11:4554 , piece_id=TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.331696878 (198h32m10.027545621s) [noise=true , address=45.91.116.4:4937 , piece_id=CWDA2IYNFUYHVDBJ74Y3CJKCZUBN4QYUEJYFKZNIBAUYYAXO3WFQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.334142676 (198h32m10.029991429s) [noise=true , address=45.91.116.10:4082 , piece_id=RW62GCPC56K2IMOFF276GQECXRKW6NY6W44TNAYFNDGQS3UK7EEA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.346539031 (198h32m10.042387794s) [noise=true , address=77.90.31.10:4519 , piece_id=MIHLKGHCRTUFQSGBIFKIWPNACAJ4CQOEYRDIKA4KJLFP356VMY6A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.419504032 (198h32m10.115352785s) [noise=true , address=77.90.31.19:4843 , piece_id=JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:39.113150499 (198h32m10.808999252s) [noise=true , address=45.91.116.6:4973 , piece_id=Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:39.113177150 (198h32m10.809025903s) [seg_pos={0 0}, upload=174963]: Segment reached optimal threshold of 13 pieces. Nov 7 18:12:39.113381018 (198h32m10.809229772s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113390166 (198h32m10.809238919s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113460199 (198h32m10.809308952s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113470077 (198h32m10.809318821s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113780338 (198h32m10.809629091s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113788243 (198h32m10.809636986s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113858657 (198h32m10.8097074s) [seg_pos={0 0}, upload=174963]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 7 18:12:39.114030194 (198h32m10.809878947s) [upload=174963]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 7 18:12:39.114038290 (198h32m10.809887043s) [upload=174963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 7 18:12:39.451448640 (198h32m11.147297403s) [upload=174963]: Done uploading segments... Nov 7 18:12:39.451473607 (198h32m11.14732236s) [upload=174963]: Upload finished. err: <nil> Duration: 6.17204803s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 33 out of 60 is failed (threshold=3) [XXXXXXXXX..XXXX.X..XXX.X..X.X.XXXX.X..XX.XXXXX.X............] duration: 6.216974325s trace: https://jaeger.dev.storj.io/trace/422f33ac8ae6a74a FAIL (2025-11-07 18:21:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ) context deadline exceeded; piecestore: (Node ID: 12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, Piece ID: JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ) context deadline exceeded ======LOGS====== Nov 7 18:12:35.791697270 (198h32m7.487546023s) [upload=174963]: Starting upload Nov 7 18:12:35.791791689 (198h32m7.487640452s) [upload=174963]: Done starting upload Nov 7 18:12:35.791885458 (198h32m7.487734211s) [upload=174963]: Uploading segments... Nov 7 18:12:35.792146455 (198h32m7.487995208s) [upload=174963]: Got next segment. Inline: false Nov 7 18:12:35.792195278 (198h32m7.488044021s) [upload=174963]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 7 18:12:35.792201820 (198h32m7.488050563s) [upload=174963]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 7 18:12:36.063867588 (198h32m7.759716341s) [seg_pos={0 0}, upload=174963]: Begin upload segment... Nov 7 18:12:36.063937832 (198h32m7.759786585s) [seg_pos={0 0}, upload=174963]: Done begin upload segment. Nov 7 18:12:36.063988689 (198h32m7.759837463s) [noise=true , address=145.79.227.27:4690 , piece_id=7CVNJTAEBVYJXRWNSFYFP3E7SKMY5XI4CV2JEZJLSXH4TA2F2JZQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.064083510 (198h32m7.759932263s) [noise=true , address=145.79.227.10:4403 , piece_id=JCMXNJ5PW5ULGOJZ5SGB423VW7VJPC5REYNSNWJZJXHGUQXUS3KA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.064896969 (198h32m7.760745762s) [noise=true , address=145.79.227.28:4726 , piece_id=D6DH5FHFB5ZTPANRPUWZKRYPFYUV2LS6AY5FQL5RBJZIUNM4FKLQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.065443520 (198h32m7.761292273s) [noise=true , address=77.90.31.11:4554 , piece_id=TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.065970974 (198h32m7.761819777s) [noise=true , address=145.79.227.8:4330 , piece_id=32XBCOEGSUTVUCCWVM2D7CB6FKMNLD6N2N4655B3ZYHNAIL4R2QQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.066422053 (198h32m7.762270806s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.067054427 (198h32m7.76290318s) [noise=true , address=77.90.31.10:4519 , piece_id=MIHLKGHCRTUFQSGBIFKIWPNACAJ4CQOEYRDIKA4KJLFP356VMY6A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.067645042 (198h32m7.763493795s) [noise=true , address=45.91.116.6:4973 , piece_id=Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.068379079 (198h32m7.764227842s) [noise=true , address=77.90.31.19:4843 , piece_id=JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.070931099 (198h32m7.766779862s) [noise=true , address=145.79.227.11:4732 , piece_id=VKSPYLTD6C6XBUFXXVVPHIPGFRCRNQK6EVI4SM7Y6ODOMVGGAJOQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.073100420 (198h32m7.768949173s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.075821872 (198h32m7.771670625s) [noise=true , address=45.91.116.4:4937 , piece_id=CWDA2IYNFUYHVDBJ74Y3CJKCZUBN4QYUEJYFKZNIBAUYYAXO3WFQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.078400863 (198h32m7.774249626s) [noise=true , address=45.91.116.10:4082 , piece_id=RW62GCPC56K2IMOFF276GQECXRKW6NY6W44TNAYFNDGQS3UK7EEA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.081230762 (198h32m7.777079515s) [noise=true , address=145.79.227.15:4876 , piece_id=ZPGKGNXRFEWRRJKTS7NKBEVQA7OAWNWYDR3DUTPHT3XVYRJSKWPQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.083298258 (198h32m7.779147012s) [noise=true , address=145.79.227.20:4438 , piece_id=V6WDUVDB7WHTDCTSYCFWS22UKRD5NMXGJ2GHLSFLVVOXZMZWLL4A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.084602091 (198h32m7.780450844s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:37.640211006 (198h32m9.336059759s) [upload=174963]: Next returned nil segment Nov 7 18:12:37.640227878 (198h32m9.336076621s) [upload=174963]: Waiting for error group managing segments... Nov 7 18:12:38.007509819 (198h32m9.703358572s) [noise=true , address=145.79.227.10:4403 , piece_id=JCMXNJ5PW5ULGOJZ5SGB423VW7VJPC5REYNSNWJZJXHGUQXUS3KA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.015182750 (198h32m9.711031503s) [noise=true , address=145.79.227.28:4726 , piece_id=D6DH5FHFB5ZTPANRPUWZKRYPFYUV2LS6AY5FQL5RBJZIUNM4FKLQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.020400886 (198h32m9.71624964s) [noise=true , address=145.79.227.27:4690 , piece_id=7CVNJTAEBVYJXRWNSFYFP3E7SKMY5XI4CV2JEZJLSXH4TA2F2JZQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.062361869 (198h32m9.758210622s) [noise=true , address=145.79.227.15:4876 , piece_id=ZPGKGNXRFEWRRJKTS7NKBEVQA7OAWNWYDR3DUTPHT3XVYRJSKWPQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.096654439 (198h32m9.792503192s) [noise=true , address=145.79.227.20:4438 , piece_id=V6WDUVDB7WHTDCTSYCFWS22UKRD5NMXGJ2GHLSFLVVOXZMZWLL4A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.244365682 (198h32m9.940214435s) [noise=true , address=145.79.227.8:4330 , piece_id=32XBCOEGSUTVUCCWVM2D7CB6FKMNLD6N2N4655B3ZYHNAIL4R2QQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.245114297 (198h32m9.94096305s) [noise=true , address=145.79.227.11:4732 , piece_id=VKSPYLTD6C6XBUFXXVVPHIPGFRCRNQK6EVI4SM7Y6ODOMVGGAJOQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.324356281 (198h32m10.020205034s) [noise=true , address=77.90.31.11:4554 , piece_id=TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.331696878 (198h32m10.027545621s) [noise=true , address=45.91.116.4:4937 , piece_id=CWDA2IYNFUYHVDBJ74Y3CJKCZUBN4QYUEJYFKZNIBAUYYAXO3WFQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.334142676 (198h32m10.029991429s) [noise=true , address=45.91.116.10:4082 , piece_id=RW62GCPC56K2IMOFF276GQECXRKW6NY6W44TNAYFNDGQS3UK7EEA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.346539031 (198h32m10.042387794s) [noise=true , address=77.90.31.10:4519 , piece_id=MIHLKGHCRTUFQSGBIFKIWPNACAJ4CQOEYRDIKA4KJLFP356VMY6A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.419504032 (198h32m10.115352785s) [noise=true , address=77.90.31.19:4843 , piece_id=JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:39.113150499 (198h32m10.808999252s) [noise=true , address=45.91.116.6:4973 , piece_id=Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:39.113177150 (198h32m10.809025903s) [seg_pos={0 0}, upload=174963]: Segment reached optimal threshold of 13 pieces. Nov 7 18:12:39.113381018 (198h32m10.809229772s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113390166 (198h32m10.809238919s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113460199 (198h32m10.809308952s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113470077 (198h32m10.809318821s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113780338 (198h32m10.809629091s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113788243 (198h32m10.809636986s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113858657 (198h32m10.8097074s) [seg_pos={0 0}, upload=174963]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 7 18:12:39.114030194 (198h32m10.809878947s) [upload=174963]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 7 18:12:39.114038290 (198h32m10.809887043s) [upload=174963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 7 18:12:39.451448640 (198h32m11.147297403s) [upload=174963]: Done uploading segments... Nov 7 18:12:39.451473607 (198h32m11.14732236s) [upload=174963]: Upload finished. err: <nil> Duration: 6.217034941s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 34 out of 60 is failed (threshold=3) [XXXXXXXXXX..XXXX.X..XXX.X..X.X.XXXX.X..XX.XXXXX.X...........] duration: 6.126090085s trace: https://jaeger.dev.storj.io/trace/422f33ac8ae6a74a FAIL (2025-11-07 18:21:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ) context deadline exceeded; piecestore: (Node ID: 12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, Piece ID: JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ) context deadline exceeded ======LOGS====== Nov 7 18:12:35.791697270 (198h32m7.487546023s) [upload=174963]: Starting upload Nov 7 18:12:35.791791689 (198h32m7.487640452s) [upload=174963]: Done starting upload Nov 7 18:12:35.791885458 (198h32m7.487734211s) [upload=174963]: Uploading segments... Nov 7 18:12:35.792146455 (198h32m7.487995208s) [upload=174963]: Got next segment. Inline: false Nov 7 18:12:35.792195278 (198h32m7.488044021s) [upload=174963]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 7 18:12:35.792201820 (198h32m7.488050563s) [upload=174963]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 7 18:12:36.063867588 (198h32m7.759716341s) [seg_pos={0 0}, upload=174963]: Begin upload segment... Nov 7 18:12:36.063937832 (198h32m7.759786585s) [seg_pos={0 0}, upload=174963]: Done begin upload segment. Nov 7 18:12:36.063988689 (198h32m7.759837463s) [noise=true , address=145.79.227.27:4690 , piece_id=7CVNJTAEBVYJXRWNSFYFP3E7SKMY5XI4CV2JEZJLSXH4TA2F2JZQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.064083510 (198h32m7.759932263s) [noise=true , address=145.79.227.10:4403 , piece_id=JCMXNJ5PW5ULGOJZ5SGB423VW7VJPC5REYNSNWJZJXHGUQXUS3KA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.064896969 (198h32m7.760745762s) [noise=true , address=145.79.227.28:4726 , piece_id=D6DH5FHFB5ZTPANRPUWZKRYPFYUV2LS6AY5FQL5RBJZIUNM4FKLQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.065443520 (198h32m7.761292273s) [noise=true , address=77.90.31.11:4554 , piece_id=TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.065970974 (198h32m7.761819777s) [noise=true , address=145.79.227.8:4330 , piece_id=32XBCOEGSUTVUCCWVM2D7CB6FKMNLD6N2N4655B3ZYHNAIL4R2QQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.066422053 (198h32m7.762270806s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.067054427 (198h32m7.76290318s) [noise=true , address=77.90.31.10:4519 , piece_id=MIHLKGHCRTUFQSGBIFKIWPNACAJ4CQOEYRDIKA4KJLFP356VMY6A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.067645042 (198h32m7.763493795s) [noise=true , address=45.91.116.6:4973 , piece_id=Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.068379079 (198h32m7.764227842s) [noise=true , address=77.90.31.19:4843 , piece_id=JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.070931099 (198h32m7.766779862s) [noise=true , address=145.79.227.11:4732 , piece_id=VKSPYLTD6C6XBUFXXVVPHIPGFRCRNQK6EVI4SM7Y6ODOMVGGAJOQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.073100420 (198h32m7.768949173s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.075821872 (198h32m7.771670625s) [noise=true , address=45.91.116.4:4937 , piece_id=CWDA2IYNFUYHVDBJ74Y3CJKCZUBN4QYUEJYFKZNIBAUYYAXO3WFQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.078400863 (198h32m7.774249626s) [noise=true , address=45.91.116.10:4082 , piece_id=RW62GCPC56K2IMOFF276GQECXRKW6NY6W44TNAYFNDGQS3UK7EEA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.081230762 (198h32m7.777079515s) [noise=true , address=145.79.227.15:4876 , piece_id=ZPGKGNXRFEWRRJKTS7NKBEVQA7OAWNWYDR3DUTPHT3XVYRJSKWPQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.083298258 (198h32m7.779147012s) [noise=true , address=145.79.227.20:4438 , piece_id=V6WDUVDB7WHTDCTSYCFWS22UKRD5NMXGJ2GHLSFLVVOXZMZWLL4A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.084602091 (198h32m7.780450844s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:37.640211006 (198h32m9.336059759s) [upload=174963]: Next returned nil segment Nov 7 18:12:37.640227878 (198h32m9.336076621s) [upload=174963]: Waiting for error group managing segments... Nov 7 18:12:38.007509819 (198h32m9.703358572s) [noise=true , address=145.79.227.10:4403 , piece_id=JCMXNJ5PW5ULGOJZ5SGB423VW7VJPC5REYNSNWJZJXHGUQXUS3KA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.015182750 (198h32m9.711031503s) [noise=true , address=145.79.227.28:4726 , piece_id=D6DH5FHFB5ZTPANRPUWZKRYPFYUV2LS6AY5FQL5RBJZIUNM4FKLQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.020400886 (198h32m9.71624964s) [noise=true , address=145.79.227.27:4690 , piece_id=7CVNJTAEBVYJXRWNSFYFP3E7SKMY5XI4CV2JEZJLSXH4TA2F2JZQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.062361869 (198h32m9.758210622s) [noise=true , address=145.79.227.15:4876 , piece_id=ZPGKGNXRFEWRRJKTS7NKBEVQA7OAWNWYDR3DUTPHT3XVYRJSKWPQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.096654439 (198h32m9.792503192s) [noise=true , address=145.79.227.20:4438 , piece_id=V6WDUVDB7WHTDCTSYCFWS22UKRD5NMXGJ2GHLSFLVVOXZMZWLL4A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.244365682 (198h32m9.940214435s) [noise=true , address=145.79.227.8:4330 , piece_id=32XBCOEGSUTVUCCWVM2D7CB6FKMNLD6N2N4655B3ZYHNAIL4R2QQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.245114297 (198h32m9.94096305s) [noise=true , address=145.79.227.11:4732 , piece_id=VKSPYLTD6C6XBUFXXVVPHIPGFRCRNQK6EVI4SM7Y6ODOMVGGAJOQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.324356281 (198h32m10.020205034s) [noise=true , address=77.90.31.11:4554 , piece_id=TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.331696878 (198h32m10.027545621s) [noise=true , address=45.91.116.4:4937 , piece_id=CWDA2IYNFUYHVDBJ74Y3CJKCZUBN4QYUEJYFKZNIBAUYYAXO3WFQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.334142676 (198h32m10.029991429s) [noise=true , address=45.91.116.10:4082 , piece_id=RW62GCPC56K2IMOFF276GQECXRKW6NY6W44TNAYFNDGQS3UK7EEA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.346539031 (198h32m10.042387794s) [noise=true , address=77.90.31.10:4519 , piece_id=MIHLKGHCRTUFQSGBIFKIWPNACAJ4CQOEYRDIKA4KJLFP356VMY6A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.419504032 (198h32m10.115352785s) [noise=true , address=77.90.31.19:4843 , piece_id=JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:39.113150499 (198h32m10.808999252s) [noise=true , address=45.91.116.6:4973 , piece_id=Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:39.113177150 (198h32m10.809025903s) [seg_pos={0 0}, upload=174963]: Segment reached optimal threshold of 13 pieces. Nov 7 18:12:39.113381018 (198h32m10.809229772s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113390166 (198h32m10.809238919s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113460199 (198h32m10.809308952s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113470077 (198h32m10.809318821s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113780338 (198h32m10.809629091s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113788243 (198h32m10.809636986s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113858657 (198h32m10.8097074s) [seg_pos={0 0}, upload=174963]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 7 18:12:39.114030194 (198h32m10.809878947s) [upload=174963]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 7 18:12:39.114038290 (198h32m10.809887043s) [upload=174963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 7 18:12:39.451448640 (198h32m11.147297403s) [upload=174963]: Done uploading segments... Nov 7 18:12:39.451473607 (198h32m11.14732236s) [upload=174963]: Upload finished. err: <nil> Duration: 6.126111266s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 34 out of 60 is failed (threshold=3) [.XXXXXXXXXX..XXXX.X..XXX.X..X.X.XXXX.X..XX.XXXXX.X..........] duration: 25.737247288s trace: https://jaeger.dev.storj.io/trace/422f33ac8ae6a74a FAIL (2025-11-07 18:21:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ) context deadline exceeded; piecestore: (Node ID: 12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, Piece ID: JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ) context deadline exceeded ======LOGS====== Nov 7 18:12:35.791697270 (198h32m7.487546023s) [upload=174963]: Starting upload Nov 7 18:12:35.791791689 (198h32m7.487640452s) [upload=174963]: Done starting upload Nov 7 18:12:35.791885458 (198h32m7.487734211s) [upload=174963]: Uploading segments... Nov 7 18:12:35.792146455 (198h32m7.487995208s) [upload=174963]: Got next segment. Inline: false Nov 7 18:12:35.792195278 (198h32m7.488044021s) [upload=174963]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 7 18:12:35.792201820 (198h32m7.488050563s) [upload=174963]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 7 18:12:36.063867588 (198h32m7.759716341s) [seg_pos={0 0}, upload=174963]: Begin upload segment... Nov 7 18:12:36.063937832 (198h32m7.759786585s) [seg_pos={0 0}, upload=174963]: Done begin upload segment. Nov 7 18:12:36.063988689 (198h32m7.759837463s) [noise=true , address=145.79.227.27:4690 , piece_id=7CVNJTAEBVYJXRWNSFYFP3E7SKMY5XI4CV2JEZJLSXH4TA2F2JZQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.064083510 (198h32m7.759932263s) [noise=true , address=145.79.227.10:4403 , piece_id=JCMXNJ5PW5ULGOJZ5SGB423VW7VJPC5REYNSNWJZJXHGUQXUS3KA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.064896969 (198h32m7.760745762s) [noise=true , address=145.79.227.28:4726 , piece_id=D6DH5FHFB5ZTPANRPUWZKRYPFYUV2LS6AY5FQL5RBJZIUNM4FKLQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.065443520 (198h32m7.761292273s) [noise=true , address=77.90.31.11:4554 , piece_id=TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.065970974 (198h32m7.761819777s) [noise=true , address=145.79.227.8:4330 , piece_id=32XBCOEGSUTVUCCWVM2D7CB6FKMNLD6N2N4655B3ZYHNAIL4R2QQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.066422053 (198h32m7.762270806s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.067054427 (198h32m7.76290318s) [noise=true , address=77.90.31.10:4519 , piece_id=MIHLKGHCRTUFQSGBIFKIWPNACAJ4CQOEYRDIKA4KJLFP356VMY6A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.067645042 (198h32m7.763493795s) [noise=true , address=45.91.116.6:4973 , piece_id=Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.068379079 (198h32m7.764227842s) [noise=true , address=77.90.31.19:4843 , piece_id=JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.070931099 (198h32m7.766779862s) [noise=true , address=145.79.227.11:4732 , piece_id=VKSPYLTD6C6XBUFXXVVPHIPGFRCRNQK6EVI4SM7Y6ODOMVGGAJOQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.073100420 (198h32m7.768949173s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.075821872 (198h32m7.771670625s) [noise=true , address=45.91.116.4:4937 , piece_id=CWDA2IYNFUYHVDBJ74Y3CJKCZUBN4QYUEJYFKZNIBAUYYAXO3WFQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.078400863 (198h32m7.774249626s) [noise=true , address=45.91.116.10:4082 , piece_id=RW62GCPC56K2IMOFF276GQECXRKW6NY6W44TNAYFNDGQS3UK7EEA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.081230762 (198h32m7.777079515s) [noise=true , address=145.79.227.15:4876 , piece_id=ZPGKGNXRFEWRRJKTS7NKBEVQA7OAWNWYDR3DUTPHT3XVYRJSKWPQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.083298258 (198h32m7.779147012s) [noise=true , address=145.79.227.20:4438 , piece_id=V6WDUVDB7WHTDCTSYCFWS22UKRD5NMXGJ2GHLSFLVVOXZMZWLL4A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.084602091 (198h32m7.780450844s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:37.640211006 (198h32m9.336059759s) [upload=174963]: Next returned nil segment Nov 7 18:12:37.640227878 (198h32m9.336076621s) [upload=174963]: Waiting for error group managing segments... Nov 7 18:12:38.007509819 (198h32m9.703358572s) [noise=true , address=145.79.227.10:4403 , piece_id=JCMXNJ5PW5ULGOJZ5SGB423VW7VJPC5REYNSNWJZJXHGUQXUS3KA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.015182750 (198h32m9.711031503s) [noise=true , address=145.79.227.28:4726 , piece_id=D6DH5FHFB5ZTPANRPUWZKRYPFYUV2LS6AY5FQL5RBJZIUNM4FKLQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.020400886 (198h32m9.71624964s) [noise=true , address=145.79.227.27:4690 , piece_id=7CVNJTAEBVYJXRWNSFYFP3E7SKMY5XI4CV2JEZJLSXH4TA2F2JZQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.062361869 (198h32m9.758210622s) [noise=true , address=145.79.227.15:4876 , piece_id=ZPGKGNXRFEWRRJKTS7NKBEVQA7OAWNWYDR3DUTPHT3XVYRJSKWPQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.096654439 (198h32m9.792503192s) [noise=true , address=145.79.227.20:4438 , piece_id=V6WDUVDB7WHTDCTSYCFWS22UKRD5NMXGJ2GHLSFLVVOXZMZWLL4A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.244365682 (198h32m9.940214435s) [noise=true , address=145.79.227.8:4330 , piece_id=32XBCOEGSUTVUCCWVM2D7CB6FKMNLD6N2N4655B3ZYHNAIL4R2QQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.245114297 (198h32m9.94096305s) [noise=true , address=145.79.227.11:4732 , piece_id=VKSPYLTD6C6XBUFXXVVPHIPGFRCRNQK6EVI4SM7Y6ODOMVGGAJOQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.324356281 (198h32m10.020205034s) [noise=true , address=77.90.31.11:4554 , piece_id=TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.331696878 (198h32m10.027545621s) [noise=true , address=45.91.116.4:4937 , piece_id=CWDA2IYNFUYHVDBJ74Y3CJKCZUBN4QYUEJYFKZNIBAUYYAXO3WFQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.334142676 (198h32m10.029991429s) [noise=true , address=45.91.116.10:4082 , piece_id=RW62GCPC56K2IMOFF276GQECXRKW6NY6W44TNAYFNDGQS3UK7EEA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.346539031 (198h32m10.042387794s) [noise=true , address=77.90.31.10:4519 , piece_id=MIHLKGHCRTUFQSGBIFKIWPNACAJ4CQOEYRDIKA4KJLFP356VMY6A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.419504032 (198h32m10.115352785s) [noise=true , address=77.90.31.19:4843 , piece_id=JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:39.113150499 (198h32m10.808999252s) [noise=true , address=45.91.116.6:4973 , piece_id=Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:39.113177150 (198h32m10.809025903s) [seg_pos={0 0}, upload=174963]: Segment reached optimal threshold of 13 pieces. Nov 7 18:12:39.113381018 (198h32m10.809229772s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113390166 (198h32m10.809238919s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113460199 (198h32m10.809308952s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113470077 (198h32m10.809318821s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113780338 (198h32m10.809629091s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113788243 (198h32m10.809636986s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113858657 (198h32m10.8097074s) [seg_pos={0 0}, upload=174963]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 7 18:12:39.114030194 (198h32m10.809878947s) [upload=174963]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 7 18:12:39.114038290 (198h32m10.809887043s) [upload=174963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 7 18:12:39.451448640 (198h32m11.147297403s) [upload=174963]: Done uploading segments... Nov 7 18:12:39.451473607 (198h32m11.14732236s) [upload=174963]: Upload finished. err: <nil> Duration: 25.737282364s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 35 out of 60 is failed (threshold=3) [X.XXXXXXXXXX..XXXX.X..XXX.X..X.X.XXXX.X..XX.XXXXX.X.........] duration: 6.036551742s trace: https://jaeger.dev.storj.io/trace/422f33ac8ae6a74a FAIL (2025-11-07 18:19:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ) context deadline exceeded; piecestore: (Node ID: 12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, Piece ID: JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ) context deadline exceeded ======LOGS====== Nov 7 18:12:35.791697270 (198h32m7.487546023s) [upload=174963]: Starting upload Nov 7 18:12:35.791791689 (198h32m7.487640452s) [upload=174963]: Done starting upload Nov 7 18:12:35.791885458 (198h32m7.487734211s) [upload=174963]: Uploading segments... Nov 7 18:12:35.792146455 (198h32m7.487995208s) [upload=174963]: Got next segment. Inline: false Nov 7 18:12:35.792195278 (198h32m7.488044021s) [upload=174963]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 7 18:12:35.792201820 (198h32m7.488050563s) [upload=174963]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 7 18:12:36.063867588 (198h32m7.759716341s) [seg_pos={0 0}, upload=174963]: Begin upload segment... Nov 7 18:12:36.063937832 (198h32m7.759786585s) [seg_pos={0 0}, upload=174963]: Done begin upload segment. Nov 7 18:12:36.063988689 (198h32m7.759837463s) [noise=true , address=145.79.227.27:4690 , piece_id=7CVNJTAEBVYJXRWNSFYFP3E7SKMY5XI4CV2JEZJLSXH4TA2F2JZQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.064083510 (198h32m7.759932263s) [noise=true , address=145.79.227.10:4403 , piece_id=JCMXNJ5PW5ULGOJZ5SGB423VW7VJPC5REYNSNWJZJXHGUQXUS3KA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.064896969 (198h32m7.760745762s) [noise=true , address=145.79.227.28:4726 , piece_id=D6DH5FHFB5ZTPANRPUWZKRYPFYUV2LS6AY5FQL5RBJZIUNM4FKLQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.065443520 (198h32m7.761292273s) [noise=true , address=77.90.31.11:4554 , piece_id=TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.065970974 (198h32m7.761819777s) [noise=true , address=145.79.227.8:4330 , piece_id=32XBCOEGSUTVUCCWVM2D7CB6FKMNLD6N2N4655B3ZYHNAIL4R2QQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.066422053 (198h32m7.762270806s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.067054427 (198h32m7.76290318s) [noise=true , address=77.90.31.10:4519 , piece_id=MIHLKGHCRTUFQSGBIFKIWPNACAJ4CQOEYRDIKA4KJLFP356VMY6A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.067645042 (198h32m7.763493795s) [noise=true , address=45.91.116.6:4973 , piece_id=Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.068379079 (198h32m7.764227842s) [noise=true , address=77.90.31.19:4843 , piece_id=JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.070931099 (198h32m7.766779862s) [noise=true , address=145.79.227.11:4732 , piece_id=VKSPYLTD6C6XBUFXXVVPHIPGFRCRNQK6EVI4SM7Y6ODOMVGGAJOQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.073100420 (198h32m7.768949173s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.075821872 (198h32m7.771670625s) [noise=true , address=45.91.116.4:4937 , piece_id=CWDA2IYNFUYHVDBJ74Y3CJKCZUBN4QYUEJYFKZNIBAUYYAXO3WFQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.078400863 (198h32m7.774249626s) [noise=true , address=45.91.116.10:4082 , piece_id=RW62GCPC56K2IMOFF276GQECXRKW6NY6W44TNAYFNDGQS3UK7EEA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.081230762 (198h32m7.777079515s) [noise=true , address=145.79.227.15:4876 , piece_id=ZPGKGNXRFEWRRJKTS7NKBEVQA7OAWNWYDR3DUTPHT3XVYRJSKWPQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.083298258 (198h32m7.779147012s) [noise=true , address=145.79.227.20:4438 , piece_id=V6WDUVDB7WHTDCTSYCFWS22UKRD5NMXGJ2GHLSFLVVOXZMZWLL4A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.084602091 (198h32m7.780450844s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:37.640211006 (198h32m9.336059759s) [upload=174963]: Next returned nil segment Nov 7 18:12:37.640227878 (198h32m9.336076621s) [upload=174963]: Waiting for error group managing segments... Nov 7 18:12:38.007509819 (198h32m9.703358572s) [noise=true , address=145.79.227.10:4403 , piece_id=JCMXNJ5PW5ULGOJZ5SGB423VW7VJPC5REYNSNWJZJXHGUQXUS3KA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.015182750 (198h32m9.711031503s) [noise=true , address=145.79.227.28:4726 , piece_id=D6DH5FHFB5ZTPANRPUWZKRYPFYUV2LS6AY5FQL5RBJZIUNM4FKLQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.020400886 (198h32m9.71624964s) [noise=true , address=145.79.227.27:4690 , piece_id=7CVNJTAEBVYJXRWNSFYFP3E7SKMY5XI4CV2JEZJLSXH4TA2F2JZQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.062361869 (198h32m9.758210622s) [noise=true , address=145.79.227.15:4876 , piece_id=ZPGKGNXRFEWRRJKTS7NKBEVQA7OAWNWYDR3DUTPHT3XVYRJSKWPQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.096654439 (198h32m9.792503192s) [noise=true , address=145.79.227.20:4438 , piece_id=V6WDUVDB7WHTDCTSYCFWS22UKRD5NMXGJ2GHLSFLVVOXZMZWLL4A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.244365682 (198h32m9.940214435s) [noise=true , address=145.79.227.8:4330 , piece_id=32XBCOEGSUTVUCCWVM2D7CB6FKMNLD6N2N4655B3ZYHNAIL4R2QQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.245114297 (198h32m9.94096305s) [noise=true , address=145.79.227.11:4732 , piece_id=VKSPYLTD6C6XBUFXXVVPHIPGFRCRNQK6EVI4SM7Y6ODOMVGGAJOQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.324356281 (198h32m10.020205034s) [noise=true , address=77.90.31.11:4554 , piece_id=TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.331696878 (198h32m10.027545621s) [noise=true , address=45.91.116.4:4937 , piece_id=CWDA2IYNFUYHVDBJ74Y3CJKCZUBN4QYUEJYFKZNIBAUYYAXO3WFQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.334142676 (198h32m10.029991429s) [noise=true , address=45.91.116.10:4082 , piece_id=RW62GCPC56K2IMOFF276GQECXRKW6NY6W44TNAYFNDGQS3UK7EEA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.346539031 (198h32m10.042387794s) [noise=true , address=77.90.31.10:4519 , piece_id=MIHLKGHCRTUFQSGBIFKIWPNACAJ4CQOEYRDIKA4KJLFP356VMY6A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.419504032 (198h32m10.115352785s) [noise=true , address=77.90.31.19:4843 , piece_id=JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:39.113150499 (198h32m10.808999252s) [noise=true , address=45.91.116.6:4973 , piece_id=Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:39.113177150 (198h32m10.809025903s) [seg_pos={0 0}, upload=174963]: Segment reached optimal threshold of 13 pieces. Nov 7 18:12:39.113381018 (198h32m10.809229772s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113390166 (198h32m10.809238919s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113460199 (198h32m10.809308952s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113470077 (198h32m10.809318821s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113780338 (198h32m10.809629091s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113788243 (198h32m10.809636986s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113858657 (198h32m10.8097074s) [seg_pos={0 0}, upload=174963]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 7 18:12:39.114030194 (198h32m10.809878947s) [upload=174963]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 7 18:12:39.114038290 (198h32m10.809887043s) [upload=174963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 7 18:12:39.451448640 (198h32m11.147297403s) [upload=174963]: Done uploading segments... Nov 7 18:12:39.451473607 (198h32m11.14732236s) [upload=174963]: Upload finished. err: <nil> Duration: 6.036584835s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 35 out of 60 is failed (threshold=3) [.X.XXXXXXXXXX..XXXX.X..XXX.X..X.X.XXXX.X..XX.XXXXX.X........] duration: 6.161384343s trace: https://jaeger.dev.storj.io/trace/422f33ac8ae6a74a FAIL (2025-11-07 18:19:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ) context deadline exceeded; piecestore: (Node ID: 12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, Piece ID: JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ) context deadline exceeded ======LOGS====== Nov 7 18:12:35.791697270 (198h32m7.487546023s) [upload=174963]: Starting upload Nov 7 18:12:35.791791689 (198h32m7.487640452s) [upload=174963]: Done starting upload Nov 7 18:12:35.791885458 (198h32m7.487734211s) [upload=174963]: Uploading segments... Nov 7 18:12:35.792146455 (198h32m7.487995208s) [upload=174963]: Got next segment. Inline: false Nov 7 18:12:35.792195278 (198h32m7.488044021s) [upload=174963]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 7 18:12:35.792201820 (198h32m7.488050563s) [upload=174963]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 7 18:12:36.063867588 (198h32m7.759716341s) [seg_pos={0 0}, upload=174963]: Begin upload segment... Nov 7 18:12:36.063937832 (198h32m7.759786585s) [seg_pos={0 0}, upload=174963]: Done begin upload segment. Nov 7 18:12:36.063988689 (198h32m7.759837463s) [noise=true , address=145.79.227.27:4690 , piece_id=7CVNJTAEBVYJXRWNSFYFP3E7SKMY5XI4CV2JEZJLSXH4TA2F2JZQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.064083510 (198h32m7.759932263s) [noise=true , address=145.79.227.10:4403 , piece_id=JCMXNJ5PW5ULGOJZ5SGB423VW7VJPC5REYNSNWJZJXHGUQXUS3KA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.064896969 (198h32m7.760745762s) [noise=true , address=145.79.227.28:4726 , piece_id=D6DH5FHFB5ZTPANRPUWZKRYPFYUV2LS6AY5FQL5RBJZIUNM4FKLQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.065443520 (198h32m7.761292273s) [noise=true , address=77.90.31.11:4554 , piece_id=TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.065970974 (198h32m7.761819777s) [noise=true , address=145.79.227.8:4330 , piece_id=32XBCOEGSUTVUCCWVM2D7CB6FKMNLD6N2N4655B3ZYHNAIL4R2QQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.066422053 (198h32m7.762270806s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.067054427 (198h32m7.76290318s) [noise=true , address=77.90.31.10:4519 , piece_id=MIHLKGHCRTUFQSGBIFKIWPNACAJ4CQOEYRDIKA4KJLFP356VMY6A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.067645042 (198h32m7.763493795s) [noise=true , address=45.91.116.6:4973 , piece_id=Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.068379079 (198h32m7.764227842s) [noise=true , address=77.90.31.19:4843 , piece_id=JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.070931099 (198h32m7.766779862s) [noise=true , address=145.79.227.11:4732 , piece_id=VKSPYLTD6C6XBUFXXVVPHIPGFRCRNQK6EVI4SM7Y6ODOMVGGAJOQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.073100420 (198h32m7.768949173s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.075821872 (198h32m7.771670625s) [noise=true , address=45.91.116.4:4937 , piece_id=CWDA2IYNFUYHVDBJ74Y3CJKCZUBN4QYUEJYFKZNIBAUYYAXO3WFQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.078400863 (198h32m7.774249626s) [noise=true , address=45.91.116.10:4082 , piece_id=RW62GCPC56K2IMOFF276GQECXRKW6NY6W44TNAYFNDGQS3UK7EEA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.081230762 (198h32m7.777079515s) [noise=true , address=145.79.227.15:4876 , piece_id=ZPGKGNXRFEWRRJKTS7NKBEVQA7OAWNWYDR3DUTPHT3XVYRJSKWPQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.083298258 (198h32m7.779147012s) [noise=true , address=145.79.227.20:4438 , piece_id=V6WDUVDB7WHTDCTSYCFWS22UKRD5NMXGJ2GHLSFLVVOXZMZWLL4A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.084602091 (198h32m7.780450844s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:37.640211006 (198h32m9.336059759s) [upload=174963]: Next returned nil segment Nov 7 18:12:37.640227878 (198h32m9.336076621s) [upload=174963]: Waiting for error group managing segments... Nov 7 18:12:38.007509819 (198h32m9.703358572s) [noise=true , address=145.79.227.10:4403 , piece_id=JCMXNJ5PW5ULGOJZ5SGB423VW7VJPC5REYNSNWJZJXHGUQXUS3KA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.015182750 (198h32m9.711031503s) [noise=true , address=145.79.227.28:4726 , piece_id=D6DH5FHFB5ZTPANRPUWZKRYPFYUV2LS6AY5FQL5RBJZIUNM4FKLQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.020400886 (198h32m9.71624964s) [noise=true , address=145.79.227.27:4690 , piece_id=7CVNJTAEBVYJXRWNSFYFP3E7SKMY5XI4CV2JEZJLSXH4TA2F2JZQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.062361869 (198h32m9.758210622s) [noise=true , address=145.79.227.15:4876 , piece_id=ZPGKGNXRFEWRRJKTS7NKBEVQA7OAWNWYDR3DUTPHT3XVYRJSKWPQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.096654439 (198h32m9.792503192s) [noise=true , address=145.79.227.20:4438 , piece_id=V6WDUVDB7WHTDCTSYCFWS22UKRD5NMXGJ2GHLSFLVVOXZMZWLL4A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.244365682 (198h32m9.940214435s) [noise=true , address=145.79.227.8:4330 , piece_id=32XBCOEGSUTVUCCWVM2D7CB6FKMNLD6N2N4655B3ZYHNAIL4R2QQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.245114297 (198h32m9.94096305s) [noise=true , address=145.79.227.11:4732 , piece_id=VKSPYLTD6C6XBUFXXVVPHIPGFRCRNQK6EVI4SM7Y6ODOMVGGAJOQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.324356281 (198h32m10.020205034s) [noise=true , address=77.90.31.11:4554 , piece_id=TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.331696878 (198h32m10.027545621s) [noise=true , address=45.91.116.4:4937 , piece_id=CWDA2IYNFUYHVDBJ74Y3CJKCZUBN4QYUEJYFKZNIBAUYYAXO3WFQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.334142676 (198h32m10.029991429s) [noise=true , address=45.91.116.10:4082 , piece_id=RW62GCPC56K2IMOFF276GQECXRKW6NY6W44TNAYFNDGQS3UK7EEA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.346539031 (198h32m10.042387794s) [noise=true , address=77.90.31.10:4519 , piece_id=MIHLKGHCRTUFQSGBIFKIWPNACAJ4CQOEYRDIKA4KJLFP356VMY6A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.419504032 (198h32m10.115352785s) [noise=true , address=77.90.31.19:4843 , piece_id=JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:39.113150499 (198h32m10.808999252s) [noise=true , address=45.91.116.6:4973 , piece_id=Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:39.113177150 (198h32m10.809025903s) [seg_pos={0 0}, upload=174963]: Segment reached optimal threshold of 13 pieces. Nov 7 18:12:39.113381018 (198h32m10.809229772s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113390166 (198h32m10.809238919s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113460199 (198h32m10.809308952s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113470077 (198h32m10.809318821s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113780338 (198h32m10.809629091s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113788243 (198h32m10.809636986s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113858657 (198h32m10.8097074s) [seg_pos={0 0}, upload=174963]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 7 18:12:39.114030194 (198h32m10.809878947s) [upload=174963]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 7 18:12:39.114038290 (198h32m10.809887043s) [upload=174963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 7 18:12:39.451448640 (198h32m11.147297403s) [upload=174963]: Done uploading segments... Nov 7 18:12:39.451473607 (198h32m11.14732236s) [upload=174963]: Upload finished. err: <nil> Duration: 6.161448185s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 36 out of 60 is failed (threshold=3) [X.X.XXXXXXXXXX..XXXX.X..XXX.X..X.X.XXXX.X..XX.XXXXX.X.......] duration: 6.334833536s trace: https://jaeger.dev.storj.io/trace/422f33ac8ae6a74a FAIL (2025-11-07 18:18:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ) context deadline exceeded; piecestore: (Node ID: 12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, Piece ID: JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ) context deadline exceeded ======LOGS====== Nov 7 18:12:35.791697270 (198h32m7.487546023s) [upload=174963]: Starting upload Nov 7 18:12:35.791791689 (198h32m7.487640452s) [upload=174963]: Done starting upload Nov 7 18:12:35.791885458 (198h32m7.487734211s) [upload=174963]: Uploading segments... Nov 7 18:12:35.792146455 (198h32m7.487995208s) [upload=174963]: Got next segment. Inline: false Nov 7 18:12:35.792195278 (198h32m7.488044021s) [upload=174963]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 7 18:12:35.792201820 (198h32m7.488050563s) [upload=174963]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 7 18:12:36.063867588 (198h32m7.759716341s) [seg_pos={0 0}, upload=174963]: Begin upload segment... Nov 7 18:12:36.063937832 (198h32m7.759786585s) [seg_pos={0 0}, upload=174963]: Done begin upload segment. Nov 7 18:12:36.063988689 (198h32m7.759837463s) [noise=true , address=145.79.227.27:4690 , piece_id=7CVNJTAEBVYJXRWNSFYFP3E7SKMY5XI4CV2JEZJLSXH4TA2F2JZQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.064083510 (198h32m7.759932263s) [noise=true , address=145.79.227.10:4403 , piece_id=JCMXNJ5PW5ULGOJZ5SGB423VW7VJPC5REYNSNWJZJXHGUQXUS3KA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.064896969 (198h32m7.760745762s) [noise=true , address=145.79.227.28:4726 , piece_id=D6DH5FHFB5ZTPANRPUWZKRYPFYUV2LS6AY5FQL5RBJZIUNM4FKLQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.065443520 (198h32m7.761292273s) [noise=true , address=77.90.31.11:4554 , piece_id=TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.065970974 (198h32m7.761819777s) [noise=true , address=145.79.227.8:4330 , piece_id=32XBCOEGSUTVUCCWVM2D7CB6FKMNLD6N2N4655B3ZYHNAIL4R2QQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.066422053 (198h32m7.762270806s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.067054427 (198h32m7.76290318s) [noise=true , address=77.90.31.10:4519 , piece_id=MIHLKGHCRTUFQSGBIFKIWPNACAJ4CQOEYRDIKA4KJLFP356VMY6A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.067645042 (198h32m7.763493795s) [noise=true , address=45.91.116.6:4973 , piece_id=Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.068379079 (198h32m7.764227842s) [noise=true , address=77.90.31.19:4843 , piece_id=JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.070931099 (198h32m7.766779862s) [noise=true , address=145.79.227.11:4732 , piece_id=VKSPYLTD6C6XBUFXXVVPHIPGFRCRNQK6EVI4SM7Y6ODOMVGGAJOQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.073100420 (198h32m7.768949173s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.075821872 (198h32m7.771670625s) [noise=true , address=45.91.116.4:4937 , piece_id=CWDA2IYNFUYHVDBJ74Y3CJKCZUBN4QYUEJYFKZNIBAUYYAXO3WFQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.078400863 (198h32m7.774249626s) [noise=true , address=45.91.116.10:4082 , piece_id=RW62GCPC56K2IMOFF276GQECXRKW6NY6W44TNAYFNDGQS3UK7EEA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.081230762 (198h32m7.777079515s) [noise=true , address=145.79.227.15:4876 , piece_id=ZPGKGNXRFEWRRJKTS7NKBEVQA7OAWNWYDR3DUTPHT3XVYRJSKWPQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.083298258 (198h32m7.779147012s) [noise=true , address=145.79.227.20:4438 , piece_id=V6WDUVDB7WHTDCTSYCFWS22UKRD5NMXGJ2GHLSFLVVOXZMZWLL4A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.084602091 (198h32m7.780450844s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:37.640211006 (198h32m9.336059759s) [upload=174963]: Next returned nil segment Nov 7 18:12:37.640227878 (198h32m9.336076621s) [upload=174963]: Waiting for error group managing segments... Nov 7 18:12:38.007509819 (198h32m9.703358572s) [noise=true , address=145.79.227.10:4403 , piece_id=JCMXNJ5PW5ULGOJZ5SGB423VW7VJPC5REYNSNWJZJXHGUQXUS3KA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.015182750 (198h32m9.711031503s) [noise=true , address=145.79.227.28:4726 , piece_id=D6DH5FHFB5ZTPANRPUWZKRYPFYUV2LS6AY5FQL5RBJZIUNM4FKLQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.020400886 (198h32m9.71624964s) [noise=true , address=145.79.227.27:4690 , piece_id=7CVNJTAEBVYJXRWNSFYFP3E7SKMY5XI4CV2JEZJLSXH4TA2F2JZQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.062361869 (198h32m9.758210622s) [noise=true , address=145.79.227.15:4876 , piece_id=ZPGKGNXRFEWRRJKTS7NKBEVQA7OAWNWYDR3DUTPHT3XVYRJSKWPQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.096654439 (198h32m9.792503192s) [noise=true , address=145.79.227.20:4438 , piece_id=V6WDUVDB7WHTDCTSYCFWS22UKRD5NMXGJ2GHLSFLVVOXZMZWLL4A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.244365682 (198h32m9.940214435s) [noise=true , address=145.79.227.8:4330 , piece_id=32XBCOEGSUTVUCCWVM2D7CB6FKMNLD6N2N4655B3ZYHNAIL4R2QQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.245114297 (198h32m9.94096305s) [noise=true , address=145.79.227.11:4732 , piece_id=VKSPYLTD6C6XBUFXXVVPHIPGFRCRNQK6EVI4SM7Y6ODOMVGGAJOQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.324356281 (198h32m10.020205034s) [noise=true , address=77.90.31.11:4554 , piece_id=TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.331696878 (198h32m10.027545621s) [noise=true , address=45.91.116.4:4937 , piece_id=CWDA2IYNFUYHVDBJ74Y3CJKCZUBN4QYUEJYFKZNIBAUYYAXO3WFQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.334142676 (198h32m10.029991429s) [noise=true , address=45.91.116.10:4082 , piece_id=RW62GCPC56K2IMOFF276GQECXRKW6NY6W44TNAYFNDGQS3UK7EEA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.346539031 (198h32m10.042387794s) [noise=true , address=77.90.31.10:4519 , piece_id=MIHLKGHCRTUFQSGBIFKIWPNACAJ4CQOEYRDIKA4KJLFP356VMY6A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.419504032 (198h32m10.115352785s) [noise=true , address=77.90.31.19:4843 , piece_id=JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:39.113150499 (198h32m10.808999252s) [noise=true , address=45.91.116.6:4973 , piece_id=Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:39.113177150 (198h32m10.809025903s) [seg_pos={0 0}, upload=174963]: Segment reached optimal threshold of 13 pieces. Nov 7 18:12:39.113381018 (198h32m10.809229772s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113390166 (198h32m10.809238919s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113460199 (198h32m10.809308952s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113470077 (198h32m10.809318821s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113780338 (198h32m10.809629091s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113788243 (198h32m10.809636986s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113858657 (198h32m10.8097074s) [seg_pos={0 0}, upload=174963]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 7 18:12:39.114030194 (198h32m10.809878947s) [upload=174963]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 7 18:12:39.114038290 (198h32m10.809887043s) [upload=174963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 7 18:12:39.451448640 (198h32m11.147297403s) [upload=174963]: Done uploading segments... Nov 7 18:12:39.451473607 (198h32m11.14732236s) [upload=174963]: Upload finished. err: <nil> Duration: 6.334860346s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [X...........................................................] duration: 3.435448582s trace: https://jaeger.dev.storj.io/trace/66eac9b92124ab25 FAIL (2025-11-07 18:18:28 UTC) orbiter warning: context deadline exceeded Duration: 3.435476955s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 37 out of 60 is failed (threshold=3) [XX.X.XXXXXXXXXX..XXXX.X..XXX.X..X.X.XXXX.X..XX.XXXXX.X......] duration: 6.194690007s trace: https://jaeger.dev.storj.io/trace/422f33ac8ae6a74a FAIL (2025-11-07 18:17:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ) context deadline exceeded; piecestore: (Node ID: 12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, Piece ID: JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ) context deadline exceeded ======LOGS====== Nov 7 18:12:35.791697270 (198h32m7.487546023s) [upload=174963]: Starting upload Nov 7 18:12:35.791791689 (198h32m7.487640452s) [upload=174963]: Done starting upload Nov 7 18:12:35.791885458 (198h32m7.487734211s) [upload=174963]: Uploading segments... Nov 7 18:12:35.792146455 (198h32m7.487995208s) [upload=174963]: Got next segment. Inline: false Nov 7 18:12:35.792195278 (198h32m7.488044021s) [upload=174963]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 7 18:12:35.792201820 (198h32m7.488050563s) [upload=174963]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 7 18:12:36.063867588 (198h32m7.759716341s) [seg_pos={0 0}, upload=174963]: Begin upload segment... Nov 7 18:12:36.063937832 (198h32m7.759786585s) [seg_pos={0 0}, upload=174963]: Done begin upload segment. Nov 7 18:12:36.063988689 (198h32m7.759837463s) [noise=true , address=145.79.227.27:4690 , piece_id=7CVNJTAEBVYJXRWNSFYFP3E7SKMY5XI4CV2JEZJLSXH4TA2F2JZQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.064083510 (198h32m7.759932263s) [noise=true , address=145.79.227.10:4403 , piece_id=JCMXNJ5PW5ULGOJZ5SGB423VW7VJPC5REYNSNWJZJXHGUQXUS3KA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.064896969 (198h32m7.760745762s) [noise=true , address=145.79.227.28:4726 , piece_id=D6DH5FHFB5ZTPANRPUWZKRYPFYUV2LS6AY5FQL5RBJZIUNM4FKLQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.065443520 (198h32m7.761292273s) [noise=true , address=77.90.31.11:4554 , piece_id=TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.065970974 (198h32m7.761819777s) [noise=true , address=145.79.227.8:4330 , piece_id=32XBCOEGSUTVUCCWVM2D7CB6FKMNLD6N2N4655B3ZYHNAIL4R2QQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.066422053 (198h32m7.762270806s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.067054427 (198h32m7.76290318s) [noise=true , address=77.90.31.10:4519 , piece_id=MIHLKGHCRTUFQSGBIFKIWPNACAJ4CQOEYRDIKA4KJLFP356VMY6A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.067645042 (198h32m7.763493795s) [noise=true , address=45.91.116.6:4973 , piece_id=Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.068379079 (198h32m7.764227842s) [noise=true , address=77.90.31.19:4843 , piece_id=JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.070931099 (198h32m7.766779862s) [noise=true , address=145.79.227.11:4732 , piece_id=VKSPYLTD6C6XBUFXXVVPHIPGFRCRNQK6EVI4SM7Y6ODOMVGGAJOQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.073100420 (198h32m7.768949173s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.075821872 (198h32m7.771670625s) [noise=true , address=45.91.116.4:4937 , piece_id=CWDA2IYNFUYHVDBJ74Y3CJKCZUBN4QYUEJYFKZNIBAUYYAXO3WFQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.078400863 (198h32m7.774249626s) [noise=true , address=45.91.116.10:4082 , piece_id=RW62GCPC56K2IMOFF276GQECXRKW6NY6W44TNAYFNDGQS3UK7EEA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.081230762 (198h32m7.777079515s) [noise=true , address=145.79.227.15:4876 , piece_id=ZPGKGNXRFEWRRJKTS7NKBEVQA7OAWNWYDR3DUTPHT3XVYRJSKWPQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.083298258 (198h32m7.779147012s) [noise=true , address=145.79.227.20:4438 , piece_id=V6WDUVDB7WHTDCTSYCFWS22UKRD5NMXGJ2GHLSFLVVOXZMZWLL4A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.084602091 (198h32m7.780450844s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:37.640211006 (198h32m9.336059759s) [upload=174963]: Next returned nil segment Nov 7 18:12:37.640227878 (198h32m9.336076621s) [upload=174963]: Waiting for error group managing segments... Nov 7 18:12:38.007509819 (198h32m9.703358572s) [noise=true , address=145.79.227.10:4403 , piece_id=JCMXNJ5PW5ULGOJZ5SGB423VW7VJPC5REYNSNWJZJXHGUQXUS3KA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.015182750 (198h32m9.711031503s) [noise=true , address=145.79.227.28:4726 , piece_id=D6DH5FHFB5ZTPANRPUWZKRYPFYUV2LS6AY5FQL5RBJZIUNM4FKLQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.020400886 (198h32m9.71624964s) [noise=true , address=145.79.227.27:4690 , piece_id=7CVNJTAEBVYJXRWNSFYFP3E7SKMY5XI4CV2JEZJLSXH4TA2F2JZQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.062361869 (198h32m9.758210622s) [noise=true , address=145.79.227.15:4876 , piece_id=ZPGKGNXRFEWRRJKTS7NKBEVQA7OAWNWYDR3DUTPHT3XVYRJSKWPQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.096654439 (198h32m9.792503192s) [noise=true , address=145.79.227.20:4438 , piece_id=V6WDUVDB7WHTDCTSYCFWS22UKRD5NMXGJ2GHLSFLVVOXZMZWLL4A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.244365682 (198h32m9.940214435s) [noise=true , address=145.79.227.8:4330 , piece_id=32XBCOEGSUTVUCCWVM2D7CB6FKMNLD6N2N4655B3ZYHNAIL4R2QQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.245114297 (198h32m9.94096305s) [noise=true , address=145.79.227.11:4732 , piece_id=VKSPYLTD6C6XBUFXXVVPHIPGFRCRNQK6EVI4SM7Y6ODOMVGGAJOQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.324356281 (198h32m10.020205034s) [noise=true , address=77.90.31.11:4554 , piece_id=TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.331696878 (198h32m10.027545621s) [noise=true , address=45.91.116.4:4937 , piece_id=CWDA2IYNFUYHVDBJ74Y3CJKCZUBN4QYUEJYFKZNIBAUYYAXO3WFQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.334142676 (198h32m10.029991429s) [noise=true , address=45.91.116.10:4082 , piece_id=RW62GCPC56K2IMOFF276GQECXRKW6NY6W44TNAYFNDGQS3UK7EEA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.346539031 (198h32m10.042387794s) [noise=true , address=77.90.31.10:4519 , piece_id=MIHLKGHCRTUFQSGBIFKIWPNACAJ4CQOEYRDIKA4KJLFP356VMY6A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.419504032 (198h32m10.115352785s) [noise=true , address=77.90.31.19:4843 , piece_id=JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:39.113150499 (198h32m10.808999252s) [noise=true , address=45.91.116.6:4973 , piece_id=Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:39.113177150 (198h32m10.809025903s) [seg_pos={0 0}, upload=174963]: Segment reached optimal threshold of 13 pieces. Nov 7 18:12:39.113381018 (198h32m10.809229772s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113390166 (198h32m10.809238919s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113460199 (198h32m10.809308952s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113470077 (198h32m10.809318821s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113780338 (198h32m10.809629091s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113788243 (198h32m10.809636986s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113858657 (198h32m10.8097074s) [seg_pos={0 0}, upload=174963]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 7 18:12:39.114030194 (198h32m10.809878947s) [upload=174963]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 7 18:12:39.114038290 (198h32m10.809887043s) [upload=174963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 7 18:12:39.451448640 (198h32m11.147297403s) [upload=174963]: Done uploading segments... Nov 7 18:12:39.451473607 (198h32m11.14732236s) [upload=174963]: Upload finished. err: <nil> Duration: 6.194740363s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.X..........................................................] duration: 3.381182598s trace: https://jaeger.dev.storj.io/trace/66eac9b92124ab25 FAIL (2025-11-07 18:17:27 UTC) orbiter warning: context deadline exceeded Duration: 3.381210661s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 37 out of 60 is failed (threshold=3) [.XX.X.XXXXXXXXXX..XXXX.X..XXX.X..X.X.XXXX.X..XX.XXXXX.X.....] duration: 6.395493594s trace: https://jaeger.dev.storj.io/trace/422f33ac8ae6a74a FAIL (2025-11-07 18:16:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ) context deadline exceeded; piecestore: (Node ID: 12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, Piece ID: JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ) context deadline exceeded ======LOGS====== Nov 7 18:12:35.791697270 (198h32m7.487546023s) [upload=174963]: Starting upload Nov 7 18:12:35.791791689 (198h32m7.487640452s) [upload=174963]: Done starting upload Nov 7 18:12:35.791885458 (198h32m7.487734211s) [upload=174963]: Uploading segments... Nov 7 18:12:35.792146455 (198h32m7.487995208s) [upload=174963]: Got next segment. Inline: false Nov 7 18:12:35.792195278 (198h32m7.488044021s) [upload=174963]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 7 18:12:35.792201820 (198h32m7.488050563s) [upload=174963]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 7 18:12:36.063867588 (198h32m7.759716341s) [seg_pos={0 0}, upload=174963]: Begin upload segment... Nov 7 18:12:36.063937832 (198h32m7.759786585s) [seg_pos={0 0}, upload=174963]: Done begin upload segment. Nov 7 18:12:36.063988689 (198h32m7.759837463s) [noise=true , address=145.79.227.27:4690 , piece_id=7CVNJTAEBVYJXRWNSFYFP3E7SKMY5XI4CV2JEZJLSXH4TA2F2JZQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.064083510 (198h32m7.759932263s) [noise=true , address=145.79.227.10:4403 , piece_id=JCMXNJ5PW5ULGOJZ5SGB423VW7VJPC5REYNSNWJZJXHGUQXUS3KA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.064896969 (198h32m7.760745762s) [noise=true , address=145.79.227.28:4726 , piece_id=D6DH5FHFB5ZTPANRPUWZKRYPFYUV2LS6AY5FQL5RBJZIUNM4FKLQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.065443520 (198h32m7.761292273s) [noise=true , address=77.90.31.11:4554 , piece_id=TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.065970974 (198h32m7.761819777s) [noise=true , address=145.79.227.8:4330 , piece_id=32XBCOEGSUTVUCCWVM2D7CB6FKMNLD6N2N4655B3ZYHNAIL4R2QQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.066422053 (198h32m7.762270806s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.067054427 (198h32m7.76290318s) [noise=true , address=77.90.31.10:4519 , piece_id=MIHLKGHCRTUFQSGBIFKIWPNACAJ4CQOEYRDIKA4KJLFP356VMY6A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.067645042 (198h32m7.763493795s) [noise=true , address=45.91.116.6:4973 , piece_id=Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.068379079 (198h32m7.764227842s) [noise=true , address=77.90.31.19:4843 , piece_id=JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.070931099 (198h32m7.766779862s) [noise=true , address=145.79.227.11:4732 , piece_id=VKSPYLTD6C6XBUFXXVVPHIPGFRCRNQK6EVI4SM7Y6ODOMVGGAJOQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.073100420 (198h32m7.768949173s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.075821872 (198h32m7.771670625s) [noise=true , address=45.91.116.4:4937 , piece_id=CWDA2IYNFUYHVDBJ74Y3CJKCZUBN4QYUEJYFKZNIBAUYYAXO3WFQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.078400863 (198h32m7.774249626s) [noise=true , address=45.91.116.10:4082 , piece_id=RW62GCPC56K2IMOFF276GQECXRKW6NY6W44TNAYFNDGQS3UK7EEA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.081230762 (198h32m7.777079515s) [noise=true , address=145.79.227.15:4876 , piece_id=ZPGKGNXRFEWRRJKTS7NKBEVQA7OAWNWYDR3DUTPHT3XVYRJSKWPQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.083298258 (198h32m7.779147012s) [noise=true , address=145.79.227.20:4438 , piece_id=V6WDUVDB7WHTDCTSYCFWS22UKRD5NMXGJ2GHLSFLVVOXZMZWLL4A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.084602091 (198h32m7.780450844s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:37.640211006 (198h32m9.336059759s) [upload=174963]: Next returned nil segment Nov 7 18:12:37.640227878 (198h32m9.336076621s) [upload=174963]: Waiting for error group managing segments... Nov 7 18:12:38.007509819 (198h32m9.703358572s) [noise=true , address=145.79.227.10:4403 , piece_id=JCMXNJ5PW5ULGOJZ5SGB423VW7VJPC5REYNSNWJZJXHGUQXUS3KA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.015182750 (198h32m9.711031503s) [noise=true , address=145.79.227.28:4726 , piece_id=D6DH5FHFB5ZTPANRPUWZKRYPFYUV2LS6AY5FQL5RBJZIUNM4FKLQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.020400886 (198h32m9.71624964s) [noise=true , address=145.79.227.27:4690 , piece_id=7CVNJTAEBVYJXRWNSFYFP3E7SKMY5XI4CV2JEZJLSXH4TA2F2JZQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.062361869 (198h32m9.758210622s) [noise=true , address=145.79.227.15:4876 , piece_id=ZPGKGNXRFEWRRJKTS7NKBEVQA7OAWNWYDR3DUTPHT3XVYRJSKWPQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.096654439 (198h32m9.792503192s) [noise=true , address=145.79.227.20:4438 , piece_id=V6WDUVDB7WHTDCTSYCFWS22UKRD5NMXGJ2GHLSFLVVOXZMZWLL4A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.244365682 (198h32m9.940214435s) [noise=true , address=145.79.227.8:4330 , piece_id=32XBCOEGSUTVUCCWVM2D7CB6FKMNLD6N2N4655B3ZYHNAIL4R2QQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.245114297 (198h32m9.94096305s) [noise=true , address=145.79.227.11:4732 , piece_id=VKSPYLTD6C6XBUFXXVVPHIPGFRCRNQK6EVI4SM7Y6ODOMVGGAJOQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.324356281 (198h32m10.020205034s) [noise=true , address=77.90.31.11:4554 , piece_id=TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.331696878 (198h32m10.027545621s) [noise=true , address=45.91.116.4:4937 , piece_id=CWDA2IYNFUYHVDBJ74Y3CJKCZUBN4QYUEJYFKZNIBAUYYAXO3WFQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.334142676 (198h32m10.029991429s) [noise=true , address=45.91.116.10:4082 , piece_id=RW62GCPC56K2IMOFF276GQECXRKW6NY6W44TNAYFNDGQS3UK7EEA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.346539031 (198h32m10.042387794s) [noise=true , address=77.90.31.10:4519 , piece_id=MIHLKGHCRTUFQSGBIFKIWPNACAJ4CQOEYRDIKA4KJLFP356VMY6A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.419504032 (198h32m10.115352785s) [noise=true , address=77.90.31.19:4843 , piece_id=JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:39.113150499 (198h32m10.808999252s) [noise=true , address=45.91.116.6:4973 , piece_id=Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:39.113177150 (198h32m10.809025903s) [seg_pos={0 0}, upload=174963]: Segment reached optimal threshold of 13 pieces. Nov 7 18:12:39.113381018 (198h32m10.809229772s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113390166 (198h32m10.809238919s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113460199 (198h32m10.809308952s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113470077 (198h32m10.809318821s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113780338 (198h32m10.809629091s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113788243 (198h32m10.809636986s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113858657 (198h32m10.8097074s) [seg_pos={0 0}, upload=174963]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 7 18:12:39.114030194 (198h32m10.809878947s) [upload=174963]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 7 18:12:39.114038290 (198h32m10.809887043s) [upload=174963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 7 18:12:39.451448640 (198h32m11.147297403s) [upload=174963]: Done uploading segments... Nov 7 18:12:39.451473607 (198h32m11.14732236s) [upload=174963]: Upload finished. err: <nil> Duration: 6.395527227s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..X.........................................................] duration: 3.566578821s trace: https://jaeger.dev.storj.io/trace/66eac9b92124ab25 FAIL (2025-11-07 18:16:33 UTC) orbiter warning: context deadline exceeded Duration: 3.566611944s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...X........................................................] duration: 3.634628804s trace: https://jaeger.dev.storj.io/trace/66eac9b92124ab25 FAIL (2025-11-07 18:16:04 UTC) orbiter warning: context deadline exceeded Duration: 3.634682626s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 38 out of 60 is failed (threshold=3) [X.XX.X.XXXXXXXXXX..XXXX.X..XXX.X..X.X.XXXX.X..XX.XXXXX.X....] duration: 6.493169907s trace: https://jaeger.dev.storj.io/trace/422f33ac8ae6a74a FAIL (2025-11-07 18:15:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ) context deadline exceeded; piecestore: (Node ID: 12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, Piece ID: JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ) context deadline exceeded ======LOGS====== Nov 7 18:12:35.791697270 (198h32m7.487546023s) [upload=174963]: Starting upload Nov 7 18:12:35.791791689 (198h32m7.487640452s) [upload=174963]: Done starting upload Nov 7 18:12:35.791885458 (198h32m7.487734211s) [upload=174963]: Uploading segments... Nov 7 18:12:35.792146455 (198h32m7.487995208s) [upload=174963]: Got next segment. Inline: false Nov 7 18:12:35.792195278 (198h32m7.488044021s) [upload=174963]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 7 18:12:35.792201820 (198h32m7.488050563s) [upload=174963]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 7 18:12:36.063867588 (198h32m7.759716341s) [seg_pos={0 0}, upload=174963]: Begin upload segment... Nov 7 18:12:36.063937832 (198h32m7.759786585s) [seg_pos={0 0}, upload=174963]: Done begin upload segment. Nov 7 18:12:36.063988689 (198h32m7.759837463s) [noise=true , address=145.79.227.27:4690 , piece_id=7CVNJTAEBVYJXRWNSFYFP3E7SKMY5XI4CV2JEZJLSXH4TA2F2JZQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.064083510 (198h32m7.759932263s) [noise=true , address=145.79.227.10:4403 , piece_id=JCMXNJ5PW5ULGOJZ5SGB423VW7VJPC5REYNSNWJZJXHGUQXUS3KA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.064896969 (198h32m7.760745762s) [noise=true , address=145.79.227.28:4726 , piece_id=D6DH5FHFB5ZTPANRPUWZKRYPFYUV2LS6AY5FQL5RBJZIUNM4FKLQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.065443520 (198h32m7.761292273s) [noise=true , address=77.90.31.11:4554 , piece_id=TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.065970974 (198h32m7.761819777s) [noise=true , address=145.79.227.8:4330 , piece_id=32XBCOEGSUTVUCCWVM2D7CB6FKMNLD6N2N4655B3ZYHNAIL4R2QQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.066422053 (198h32m7.762270806s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.067054427 (198h32m7.76290318s) [noise=true , address=77.90.31.10:4519 , piece_id=MIHLKGHCRTUFQSGBIFKIWPNACAJ4CQOEYRDIKA4KJLFP356VMY6A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.067645042 (198h32m7.763493795s) [noise=true , address=45.91.116.6:4973 , piece_id=Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.068379079 (198h32m7.764227842s) [noise=true , address=77.90.31.19:4843 , piece_id=JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.070931099 (198h32m7.766779862s) [noise=true , address=145.79.227.11:4732 , piece_id=VKSPYLTD6C6XBUFXXVVPHIPGFRCRNQK6EVI4SM7Y6ODOMVGGAJOQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.073100420 (198h32m7.768949173s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.075821872 (198h32m7.771670625s) [noise=true , address=45.91.116.4:4937 , piece_id=CWDA2IYNFUYHVDBJ74Y3CJKCZUBN4QYUEJYFKZNIBAUYYAXO3WFQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.078400863 (198h32m7.774249626s) [noise=true , address=45.91.116.10:4082 , piece_id=RW62GCPC56K2IMOFF276GQECXRKW6NY6W44TNAYFNDGQS3UK7EEA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.081230762 (198h32m7.777079515s) [noise=true , address=145.79.227.15:4876 , piece_id=ZPGKGNXRFEWRRJKTS7NKBEVQA7OAWNWYDR3DUTPHT3XVYRJSKWPQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.083298258 (198h32m7.779147012s) [noise=true , address=145.79.227.20:4438 , piece_id=V6WDUVDB7WHTDCTSYCFWS22UKRD5NMXGJ2GHLSFLVVOXZMZWLL4A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.084602091 (198h32m7.780450844s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:37.640211006 (198h32m9.336059759s) [upload=174963]: Next returned nil segment Nov 7 18:12:37.640227878 (198h32m9.336076621s) [upload=174963]: Waiting for error group managing segments... Nov 7 18:12:38.007509819 (198h32m9.703358572s) [noise=true , address=145.79.227.10:4403 , piece_id=JCMXNJ5PW5ULGOJZ5SGB423VW7VJPC5REYNSNWJZJXHGUQXUS3KA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.015182750 (198h32m9.711031503s) [noise=true , address=145.79.227.28:4726 , piece_id=D6DH5FHFB5ZTPANRPUWZKRYPFYUV2LS6AY5FQL5RBJZIUNM4FKLQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.020400886 (198h32m9.71624964s) [noise=true , address=145.79.227.27:4690 , piece_id=7CVNJTAEBVYJXRWNSFYFP3E7SKMY5XI4CV2JEZJLSXH4TA2F2JZQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.062361869 (198h32m9.758210622s) [noise=true , address=145.79.227.15:4876 , piece_id=ZPGKGNXRFEWRRJKTS7NKBEVQA7OAWNWYDR3DUTPHT3XVYRJSKWPQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.096654439 (198h32m9.792503192s) [noise=true , address=145.79.227.20:4438 , piece_id=V6WDUVDB7WHTDCTSYCFWS22UKRD5NMXGJ2GHLSFLVVOXZMZWLL4A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.244365682 (198h32m9.940214435s) [noise=true , address=145.79.227.8:4330 , piece_id=32XBCOEGSUTVUCCWVM2D7CB6FKMNLD6N2N4655B3ZYHNAIL4R2QQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.245114297 (198h32m9.94096305s) [noise=true , address=145.79.227.11:4732 , piece_id=VKSPYLTD6C6XBUFXXVVPHIPGFRCRNQK6EVI4SM7Y6ODOMVGGAJOQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.324356281 (198h32m10.020205034s) [noise=true , address=77.90.31.11:4554 , piece_id=TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.331696878 (198h32m10.027545621s) [noise=true , address=45.91.116.4:4937 , piece_id=CWDA2IYNFUYHVDBJ74Y3CJKCZUBN4QYUEJYFKZNIBAUYYAXO3WFQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.334142676 (198h32m10.029991429s) [noise=true , address=45.91.116.10:4082 , piece_id=RW62GCPC56K2IMOFF276GQECXRKW6NY6W44TNAYFNDGQS3UK7EEA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.346539031 (198h32m10.042387794s) [noise=true , address=77.90.31.10:4519 , piece_id=MIHLKGHCRTUFQSGBIFKIWPNACAJ4CQOEYRDIKA4KJLFP356VMY6A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.419504032 (198h32m10.115352785s) [noise=true , address=77.90.31.19:4843 , piece_id=JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:39.113150499 (198h32m10.808999252s) [noise=true , address=45.91.116.6:4973 , piece_id=Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:39.113177150 (198h32m10.809025903s) [seg_pos={0 0}, upload=174963]: Segment reached optimal threshold of 13 pieces. Nov 7 18:12:39.113381018 (198h32m10.809229772s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113390166 (198h32m10.809238919s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113460199 (198h32m10.809308952s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113470077 (198h32m10.809318821s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113780338 (198h32m10.809629091s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113788243 (198h32m10.809636986s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113858657 (198h32m10.8097074s) [seg_pos={0 0}, upload=174963]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 7 18:12:39.114030194 (198h32m10.809878947s) [upload=174963]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 7 18:12:39.114038290 (198h32m10.809887043s) [upload=174963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 7 18:12:39.451448640 (198h32m11.147297403s) [upload=174963]: Done uploading segments... Nov 7 18:12:39.451473607 (198h32m11.14732236s) [upload=174963]: Upload finished. err: <nil> Duration: 6.493192259s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....X.......................................................] duration: 3.500266528s trace: https://jaeger.dev.storj.io/trace/66eac9b92124ab25 FAIL (2025-11-07 18:14:50 UTC) orbiter warning: context deadline exceeded Duration: 3.500302447s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 39 out of 60 is failed (threshold=3) [XX.XX.X.XXXXXXXXXX..XXXX.X..XXX.X..X.X.XXXX.X..XX.XXXXX.X...] duration: 6.66688395s trace: https://jaeger.dev.storj.io/trace/422f33ac8ae6a74a FAIL (2025-11-07 18:14:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ) context deadline exceeded; piecestore: (Node ID: 12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, Piece ID: JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ) context deadline exceeded ======LOGS====== Nov 7 18:12:35.791697270 (198h32m7.487546023s) [upload=174963]: Starting upload Nov 7 18:12:35.791791689 (198h32m7.487640452s) [upload=174963]: Done starting upload Nov 7 18:12:35.791885458 (198h32m7.487734211s) [upload=174963]: Uploading segments... Nov 7 18:12:35.792146455 (198h32m7.487995208s) [upload=174963]: Got next segment. Inline: false Nov 7 18:12:35.792195278 (198h32m7.488044021s) [upload=174963]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 7 18:12:35.792201820 (198h32m7.488050563s) [upload=174963]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 7 18:12:36.063867588 (198h32m7.759716341s) [seg_pos={0 0}, upload=174963]: Begin upload segment... Nov 7 18:12:36.063937832 (198h32m7.759786585s) [seg_pos={0 0}, upload=174963]: Done begin upload segment. Nov 7 18:12:36.063988689 (198h32m7.759837463s) [noise=true , address=145.79.227.27:4690 , piece_id=7CVNJTAEBVYJXRWNSFYFP3E7SKMY5XI4CV2JEZJLSXH4TA2F2JZQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.064083510 (198h32m7.759932263s) [noise=true , address=145.79.227.10:4403 , piece_id=JCMXNJ5PW5ULGOJZ5SGB423VW7VJPC5REYNSNWJZJXHGUQXUS3KA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.064896969 (198h32m7.760745762s) [noise=true , address=145.79.227.28:4726 , piece_id=D6DH5FHFB5ZTPANRPUWZKRYPFYUV2LS6AY5FQL5RBJZIUNM4FKLQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.065443520 (198h32m7.761292273s) [noise=true , address=77.90.31.11:4554 , piece_id=TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.065970974 (198h32m7.761819777s) [noise=true , address=145.79.227.8:4330 , piece_id=32XBCOEGSUTVUCCWVM2D7CB6FKMNLD6N2N4655B3ZYHNAIL4R2QQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.066422053 (198h32m7.762270806s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.067054427 (198h32m7.76290318s) [noise=true , address=77.90.31.10:4519 , piece_id=MIHLKGHCRTUFQSGBIFKIWPNACAJ4CQOEYRDIKA4KJLFP356VMY6A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.067645042 (198h32m7.763493795s) [noise=true , address=45.91.116.6:4973 , piece_id=Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.068379079 (198h32m7.764227842s) [noise=true , address=77.90.31.19:4843 , piece_id=JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.070931099 (198h32m7.766779862s) [noise=true , address=145.79.227.11:4732 , piece_id=VKSPYLTD6C6XBUFXXVVPHIPGFRCRNQK6EVI4SM7Y6ODOMVGGAJOQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.073100420 (198h32m7.768949173s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.075821872 (198h32m7.771670625s) [noise=true , address=45.91.116.4:4937 , piece_id=CWDA2IYNFUYHVDBJ74Y3CJKCZUBN4QYUEJYFKZNIBAUYYAXO3WFQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.078400863 (198h32m7.774249626s) [noise=true , address=45.91.116.10:4082 , piece_id=RW62GCPC56K2IMOFF276GQECXRKW6NY6W44TNAYFNDGQS3UK7EEA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.081230762 (198h32m7.777079515s) [noise=true , address=145.79.227.15:4876 , piece_id=ZPGKGNXRFEWRRJKTS7NKBEVQA7OAWNWYDR3DUTPHT3XVYRJSKWPQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.083298258 (198h32m7.779147012s) [noise=true , address=145.79.227.20:4438 , piece_id=V6WDUVDB7WHTDCTSYCFWS22UKRD5NMXGJ2GHLSFLVVOXZMZWLL4A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.084602091 (198h32m7.780450844s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:37.640211006 (198h32m9.336059759s) [upload=174963]: Next returned nil segment Nov 7 18:12:37.640227878 (198h32m9.336076621s) [upload=174963]: Waiting for error group managing segments... Nov 7 18:12:38.007509819 (198h32m9.703358572s) [noise=true , address=145.79.227.10:4403 , piece_id=JCMXNJ5PW5ULGOJZ5SGB423VW7VJPC5REYNSNWJZJXHGUQXUS3KA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.015182750 (198h32m9.711031503s) [noise=true , address=145.79.227.28:4726 , piece_id=D6DH5FHFB5ZTPANRPUWZKRYPFYUV2LS6AY5FQL5RBJZIUNM4FKLQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.020400886 (198h32m9.71624964s) [noise=true , address=145.79.227.27:4690 , piece_id=7CVNJTAEBVYJXRWNSFYFP3E7SKMY5XI4CV2JEZJLSXH4TA2F2JZQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.062361869 (198h32m9.758210622s) [noise=true , address=145.79.227.15:4876 , piece_id=ZPGKGNXRFEWRRJKTS7NKBEVQA7OAWNWYDR3DUTPHT3XVYRJSKWPQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.096654439 (198h32m9.792503192s) [noise=true , address=145.79.227.20:4438 , piece_id=V6WDUVDB7WHTDCTSYCFWS22UKRD5NMXGJ2GHLSFLVVOXZMZWLL4A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.244365682 (198h32m9.940214435s) [noise=true , address=145.79.227.8:4330 , piece_id=32XBCOEGSUTVUCCWVM2D7CB6FKMNLD6N2N4655B3ZYHNAIL4R2QQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.245114297 (198h32m9.94096305s) [noise=true , address=145.79.227.11:4732 , piece_id=VKSPYLTD6C6XBUFXXVVPHIPGFRCRNQK6EVI4SM7Y6ODOMVGGAJOQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.324356281 (198h32m10.020205034s) [noise=true , address=77.90.31.11:4554 , piece_id=TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.331696878 (198h32m10.027545621s) [noise=true , address=45.91.116.4:4937 , piece_id=CWDA2IYNFUYHVDBJ74Y3CJKCZUBN4QYUEJYFKZNIBAUYYAXO3WFQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.334142676 (198h32m10.029991429s) [noise=true , address=45.91.116.10:4082 , piece_id=RW62GCPC56K2IMOFF276GQECXRKW6NY6W44TNAYFNDGQS3UK7EEA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.346539031 (198h32m10.042387794s) [noise=true , address=77.90.31.10:4519 , piece_id=MIHLKGHCRTUFQSGBIFKIWPNACAJ4CQOEYRDIKA4KJLFP356VMY6A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.419504032 (198h32m10.115352785s) [noise=true , address=77.90.31.19:4843 , piece_id=JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:39.113150499 (198h32m10.808999252s) [noise=true , address=45.91.116.6:4973 , piece_id=Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:39.113177150 (198h32m10.809025903s) [seg_pos={0 0}, upload=174963]: Segment reached optimal threshold of 13 pieces. Nov 7 18:12:39.113381018 (198h32m10.809229772s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113390166 (198h32m10.809238919s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113460199 (198h32m10.809308952s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113470077 (198h32m10.809318821s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113780338 (198h32m10.809629091s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113788243 (198h32m10.809636986s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113858657 (198h32m10.8097074s) [seg_pos={0 0}, upload=174963]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 7 18:12:39.114030194 (198h32m10.809878947s) [upload=174963]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 7 18:12:39.114038290 (198h32m10.809887043s) [upload=174963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 7 18:12:39.451448640 (198h32m11.147297403s) [upload=174963]: Done uploading segments... Nov 7 18:12:39.451473607 (198h32m11.14732236s) [upload=174963]: Upload finished. err: <nil> Duration: 6.666913405s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....X......................................................] duration: 3.449253349s trace: https://jaeger.dev.storj.io/trace/66eac9b92124ab25 FAIL (2025-11-07 18:14:26 UTC) orbiter warning: context deadline exceeded Duration: 3.449280991s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 40 out of 60 is failed (threshold=3) [XXX.XX.X.XXXXXXXXXX..XXXX.X..XXX.X..X.X.XXXX.X..XX.XXXXX.X..] duration: 7.735264599s trace: https://jaeger.dev.storj.io/trace/422f33ac8ae6a74a FAIL (2025-11-07 18:14:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ) context deadline exceeded; piecestore: (Node ID: 12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, Piece ID: JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ) context deadline exceeded ======LOGS====== Nov 7 18:12:35.791697270 (198h32m7.487546023s) [upload=174963]: Starting upload Nov 7 18:12:35.791791689 (198h32m7.487640452s) [upload=174963]: Done starting upload Nov 7 18:12:35.791885458 (198h32m7.487734211s) [upload=174963]: Uploading segments... Nov 7 18:12:35.792146455 (198h32m7.487995208s) [upload=174963]: Got next segment. Inline: false Nov 7 18:12:35.792195278 (198h32m7.488044021s) [upload=174963]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 7 18:12:35.792201820 (198h32m7.488050563s) [upload=174963]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 7 18:12:36.063867588 (198h32m7.759716341s) [seg_pos={0 0}, upload=174963]: Begin upload segment... Nov 7 18:12:36.063937832 (198h32m7.759786585s) [seg_pos={0 0}, upload=174963]: Done begin upload segment. Nov 7 18:12:36.063988689 (198h32m7.759837463s) [noise=true , address=145.79.227.27:4690 , piece_id=7CVNJTAEBVYJXRWNSFYFP3E7SKMY5XI4CV2JEZJLSXH4TA2F2JZQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.064083510 (198h32m7.759932263s) [noise=true , address=145.79.227.10:4403 , piece_id=JCMXNJ5PW5ULGOJZ5SGB423VW7VJPC5REYNSNWJZJXHGUQXUS3KA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.064896969 (198h32m7.760745762s) [noise=true , address=145.79.227.28:4726 , piece_id=D6DH5FHFB5ZTPANRPUWZKRYPFYUV2LS6AY5FQL5RBJZIUNM4FKLQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.065443520 (198h32m7.761292273s) [noise=true , address=77.90.31.11:4554 , piece_id=TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.065970974 (198h32m7.761819777s) [noise=true , address=145.79.227.8:4330 , piece_id=32XBCOEGSUTVUCCWVM2D7CB6FKMNLD6N2N4655B3ZYHNAIL4R2QQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.066422053 (198h32m7.762270806s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.067054427 (198h32m7.76290318s) [noise=true , address=77.90.31.10:4519 , piece_id=MIHLKGHCRTUFQSGBIFKIWPNACAJ4CQOEYRDIKA4KJLFP356VMY6A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.067645042 (198h32m7.763493795s) [noise=true , address=45.91.116.6:4973 , piece_id=Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.068379079 (198h32m7.764227842s) [noise=true , address=77.90.31.19:4843 , piece_id=JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.070931099 (198h32m7.766779862s) [noise=true , address=145.79.227.11:4732 , piece_id=VKSPYLTD6C6XBUFXXVVPHIPGFRCRNQK6EVI4SM7Y6ODOMVGGAJOQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.073100420 (198h32m7.768949173s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.075821872 (198h32m7.771670625s) [noise=true , address=45.91.116.4:4937 , piece_id=CWDA2IYNFUYHVDBJ74Y3CJKCZUBN4QYUEJYFKZNIBAUYYAXO3WFQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.078400863 (198h32m7.774249626s) [noise=true , address=45.91.116.10:4082 , piece_id=RW62GCPC56K2IMOFF276GQECXRKW6NY6W44TNAYFNDGQS3UK7EEA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.081230762 (198h32m7.777079515s) [noise=true , address=145.79.227.15:4876 , piece_id=ZPGKGNXRFEWRRJKTS7NKBEVQA7OAWNWYDR3DUTPHT3XVYRJSKWPQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.083298258 (198h32m7.779147012s) [noise=true , address=145.79.227.20:4438 , piece_id=V6WDUVDB7WHTDCTSYCFWS22UKRD5NMXGJ2GHLSFLVVOXZMZWLL4A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.084602091 (198h32m7.780450844s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:37.640211006 (198h32m9.336059759s) [upload=174963]: Next returned nil segment Nov 7 18:12:37.640227878 (198h32m9.336076621s) [upload=174963]: Waiting for error group managing segments... Nov 7 18:12:38.007509819 (198h32m9.703358572s) [noise=true , address=145.79.227.10:4403 , piece_id=JCMXNJ5PW5ULGOJZ5SGB423VW7VJPC5REYNSNWJZJXHGUQXUS3KA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.015182750 (198h32m9.711031503s) [noise=true , address=145.79.227.28:4726 , piece_id=D6DH5FHFB5ZTPANRPUWZKRYPFYUV2LS6AY5FQL5RBJZIUNM4FKLQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.020400886 (198h32m9.71624964s) [noise=true , address=145.79.227.27:4690 , piece_id=7CVNJTAEBVYJXRWNSFYFP3E7SKMY5XI4CV2JEZJLSXH4TA2F2JZQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.062361869 (198h32m9.758210622s) [noise=true , address=145.79.227.15:4876 , piece_id=ZPGKGNXRFEWRRJKTS7NKBEVQA7OAWNWYDR3DUTPHT3XVYRJSKWPQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.096654439 (198h32m9.792503192s) [noise=true , address=145.79.227.20:4438 , piece_id=V6WDUVDB7WHTDCTSYCFWS22UKRD5NMXGJ2GHLSFLVVOXZMZWLL4A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.244365682 (198h32m9.940214435s) [noise=true , address=145.79.227.8:4330 , piece_id=32XBCOEGSUTVUCCWVM2D7CB6FKMNLD6N2N4655B3ZYHNAIL4R2QQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.245114297 (198h32m9.94096305s) [noise=true , address=145.79.227.11:4732 , piece_id=VKSPYLTD6C6XBUFXXVVPHIPGFRCRNQK6EVI4SM7Y6ODOMVGGAJOQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.324356281 (198h32m10.020205034s) [noise=true , address=77.90.31.11:4554 , piece_id=TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.331696878 (198h32m10.027545621s) [noise=true , address=45.91.116.4:4937 , piece_id=CWDA2IYNFUYHVDBJ74Y3CJKCZUBN4QYUEJYFKZNIBAUYYAXO3WFQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.334142676 (198h32m10.029991429s) [noise=true , address=45.91.116.10:4082 , piece_id=RW62GCPC56K2IMOFF276GQECXRKW6NY6W44TNAYFNDGQS3UK7EEA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.346539031 (198h32m10.042387794s) [noise=true , address=77.90.31.10:4519 , piece_id=MIHLKGHCRTUFQSGBIFKIWPNACAJ4CQOEYRDIKA4KJLFP356VMY6A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.419504032 (198h32m10.115352785s) [noise=true , address=77.90.31.19:4843 , piece_id=JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:39.113150499 (198h32m10.808999252s) [noise=true , address=45.91.116.6:4973 , piece_id=Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:39.113177150 (198h32m10.809025903s) [seg_pos={0 0}, upload=174963]: Segment reached optimal threshold of 13 pieces. Nov 7 18:12:39.113381018 (198h32m10.809229772s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113390166 (198h32m10.809238919s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113460199 (198h32m10.809308952s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113470077 (198h32m10.809318821s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113780338 (198h32m10.809629091s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113788243 (198h32m10.809636986s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113858657 (198h32m10.8097074s) [seg_pos={0 0}, upload=174963]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 7 18:12:39.114030194 (198h32m10.809878947s) [upload=174963]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 7 18:12:39.114038290 (198h32m10.809887043s) [upload=174963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 7 18:12:39.451448640 (198h32m11.147297403s) [upload=174963]: Done uploading segments... Nov 7 18:12:39.451473607 (198h32m11.14732236s) [upload=174963]: Upload finished. err: <nil> Duration: 7.73529623s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......X.....................................................] duration: 3.654584052s trace: https://jaeger.dev.storj.io/trace/66eac9b92124ab25 FAIL (2025-11-07 18:13:32 UTC) orbiter warning: context deadline exceeded Duration: 3.65460963s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 41 out of 60 is failed (threshold=3) [XXXX.XX.X.XXXXXXXXXX..XXXX.X..XXX.X..X.X.XXXX.X..XX.XXXXX.X.] duration: 12.884392313s trace: https://jaeger.dev.storj.io/trace/422f33ac8ae6a74a FAIL (2025-11-07 18:13:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ) context deadline exceeded; piecestore: (Node ID: 12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, Piece ID: JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ) context deadline exceeded ======LOGS====== Nov 7 18:12:35.791697270 (198h32m7.487546023s) [upload=174963]: Starting upload Nov 7 18:12:35.791791689 (198h32m7.487640452s) [upload=174963]: Done starting upload Nov 7 18:12:35.791885458 (198h32m7.487734211s) [upload=174963]: Uploading segments... Nov 7 18:12:35.792146455 (198h32m7.487995208s) [upload=174963]: Got next segment. Inline: false Nov 7 18:12:35.792195278 (198h32m7.488044021s) [upload=174963]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 7 18:12:35.792201820 (198h32m7.488050563s) [upload=174963]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 7 18:12:36.063867588 (198h32m7.759716341s) [seg_pos={0 0}, upload=174963]: Begin upload segment... Nov 7 18:12:36.063937832 (198h32m7.759786585s) [seg_pos={0 0}, upload=174963]: Done begin upload segment. Nov 7 18:12:36.063988689 (198h32m7.759837463s) [noise=true , address=145.79.227.27:4690 , piece_id=7CVNJTAEBVYJXRWNSFYFP3E7SKMY5XI4CV2JEZJLSXH4TA2F2JZQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.064083510 (198h32m7.759932263s) [noise=true , address=145.79.227.10:4403 , piece_id=JCMXNJ5PW5ULGOJZ5SGB423VW7VJPC5REYNSNWJZJXHGUQXUS3KA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.064896969 (198h32m7.760745762s) [noise=true , address=145.79.227.28:4726 , piece_id=D6DH5FHFB5ZTPANRPUWZKRYPFYUV2LS6AY5FQL5RBJZIUNM4FKLQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.065443520 (198h32m7.761292273s) [noise=true , address=77.90.31.11:4554 , piece_id=TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.065970974 (198h32m7.761819777s) [noise=true , address=145.79.227.8:4330 , piece_id=32XBCOEGSUTVUCCWVM2D7CB6FKMNLD6N2N4655B3ZYHNAIL4R2QQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.066422053 (198h32m7.762270806s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.067054427 (198h32m7.76290318s) [noise=true , address=77.90.31.10:4519 , piece_id=MIHLKGHCRTUFQSGBIFKIWPNACAJ4CQOEYRDIKA4KJLFP356VMY6A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.067645042 (198h32m7.763493795s) [noise=true , address=45.91.116.6:4973 , piece_id=Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.068379079 (198h32m7.764227842s) [noise=true , address=77.90.31.19:4843 , piece_id=JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.070931099 (198h32m7.766779862s) [noise=true , address=145.79.227.11:4732 , piece_id=VKSPYLTD6C6XBUFXXVVPHIPGFRCRNQK6EVI4SM7Y6ODOMVGGAJOQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.073100420 (198h32m7.768949173s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.075821872 (198h32m7.771670625s) [noise=true , address=45.91.116.4:4937 , piece_id=CWDA2IYNFUYHVDBJ74Y3CJKCZUBN4QYUEJYFKZNIBAUYYAXO3WFQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.078400863 (198h32m7.774249626s) [noise=true , address=45.91.116.10:4082 , piece_id=RW62GCPC56K2IMOFF276GQECXRKW6NY6W44TNAYFNDGQS3UK7EEA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.081230762 (198h32m7.777079515s) [noise=true , address=145.79.227.15:4876 , piece_id=ZPGKGNXRFEWRRJKTS7NKBEVQA7OAWNWYDR3DUTPHT3XVYRJSKWPQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.083298258 (198h32m7.779147012s) [noise=true , address=145.79.227.20:4438 , piece_id=V6WDUVDB7WHTDCTSYCFWS22UKRD5NMXGJ2GHLSFLVVOXZMZWLL4A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.084602091 (198h32m7.780450844s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:37.640211006 (198h32m9.336059759s) [upload=174963]: Next returned nil segment Nov 7 18:12:37.640227878 (198h32m9.336076621s) [upload=174963]: Waiting for error group managing segments... Nov 7 18:12:38.007509819 (198h32m9.703358572s) [noise=true , address=145.79.227.10:4403 , piece_id=JCMXNJ5PW5ULGOJZ5SGB423VW7VJPC5REYNSNWJZJXHGUQXUS3KA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.015182750 (198h32m9.711031503s) [noise=true , address=145.79.227.28:4726 , piece_id=D6DH5FHFB5ZTPANRPUWZKRYPFYUV2LS6AY5FQL5RBJZIUNM4FKLQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.020400886 (198h32m9.71624964s) [noise=true , address=145.79.227.27:4690 , piece_id=7CVNJTAEBVYJXRWNSFYFP3E7SKMY5XI4CV2JEZJLSXH4TA2F2JZQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.062361869 (198h32m9.758210622s) [noise=true , address=145.79.227.15:4876 , piece_id=ZPGKGNXRFEWRRJKTS7NKBEVQA7OAWNWYDR3DUTPHT3XVYRJSKWPQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.096654439 (198h32m9.792503192s) [noise=true , address=145.79.227.20:4438 , piece_id=V6WDUVDB7WHTDCTSYCFWS22UKRD5NMXGJ2GHLSFLVVOXZMZWLL4A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.244365682 (198h32m9.940214435s) [noise=true , address=145.79.227.8:4330 , piece_id=32XBCOEGSUTVUCCWVM2D7CB6FKMNLD6N2N4655B3ZYHNAIL4R2QQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.245114297 (198h32m9.94096305s) [noise=true , address=145.79.227.11:4732 , piece_id=VKSPYLTD6C6XBUFXXVVPHIPGFRCRNQK6EVI4SM7Y6ODOMVGGAJOQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.324356281 (198h32m10.020205034s) [noise=true , address=77.90.31.11:4554 , piece_id=TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.331696878 (198h32m10.027545621s) [noise=true , address=45.91.116.4:4937 , piece_id=CWDA2IYNFUYHVDBJ74Y3CJKCZUBN4QYUEJYFKZNIBAUYYAXO3WFQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.334142676 (198h32m10.029991429s) [noise=true , address=45.91.116.10:4082 , piece_id=RW62GCPC56K2IMOFF276GQECXRKW6NY6W44TNAYFNDGQS3UK7EEA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.346539031 (198h32m10.042387794s) [noise=true , address=77.90.31.10:4519 , piece_id=MIHLKGHCRTUFQSGBIFKIWPNACAJ4CQOEYRDIKA4KJLFP356VMY6A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.419504032 (198h32m10.115352785s) [noise=true , address=77.90.31.19:4843 , piece_id=JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:39.113150499 (198h32m10.808999252s) [noise=true , address=45.91.116.6:4973 , piece_id=Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:39.113177150 (198h32m10.809025903s) [seg_pos={0 0}, upload=174963]: Segment reached optimal threshold of 13 pieces. Nov 7 18:12:39.113381018 (198h32m10.809229772s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113390166 (198h32m10.809238919s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113460199 (198h32m10.809308952s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113470077 (198h32m10.809318821s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113780338 (198h32m10.809629091s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113788243 (198h32m10.809636986s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113858657 (198h32m10.8097074s) [seg_pos={0 0}, upload=174963]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 7 18:12:39.114030194 (198h32m10.809878947s) [upload=174963]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 7 18:12:39.114038290 (198h32m10.809887043s) [upload=174963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 7 18:12:39.451448640 (198h32m11.147297403s) [upload=174963]: Done uploading segments... Nov 7 18:12:39.451473607 (198h32m11.14732236s) [upload=174963]: Upload finished. err: <nil> Duration: 12.884413314s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 42 out of 60 is failed (threshold=3) [XXXXX.XX.X.XXXXXXXXXX..XXXX.X..XXX.X..X.X.XXXX.X..XX.XXXXX.X] duration: 30.00457307s trace: https://jaeger.dev.storj.io/trace/422f33ac8ae6a74a FAIL (2025-11-07 18:13:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ) context deadline exceeded; piecestore: (Node ID: 12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, Piece ID: JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ) context deadline exceeded ======LOGS====== Nov 7 18:12:35.791697270 (198h32m7.487546023s) [upload=174963]: Starting upload Nov 7 18:12:35.791791689 (198h32m7.487640452s) [upload=174963]: Done starting upload Nov 7 18:12:35.791885458 (198h32m7.487734211s) [upload=174963]: Uploading segments... Nov 7 18:12:35.792146455 (198h32m7.487995208s) [upload=174963]: Got next segment. Inline: false Nov 7 18:12:35.792195278 (198h32m7.488044021s) [upload=174963]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 7 18:12:35.792201820 (198h32m7.488050563s) [upload=174963]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 7 18:12:36.063867588 (198h32m7.759716341s) [seg_pos={0 0}, upload=174963]: Begin upload segment... Nov 7 18:12:36.063937832 (198h32m7.759786585s) [seg_pos={0 0}, upload=174963]: Done begin upload segment. Nov 7 18:12:36.063988689 (198h32m7.759837463s) [noise=true , address=145.79.227.27:4690 , piece_id=7CVNJTAEBVYJXRWNSFYFP3E7SKMY5XI4CV2JEZJLSXH4TA2F2JZQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.064083510 (198h32m7.759932263s) [noise=true , address=145.79.227.10:4403 , piece_id=JCMXNJ5PW5ULGOJZ5SGB423VW7VJPC5REYNSNWJZJXHGUQXUS3KA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.064896969 (198h32m7.760745762s) [noise=true , address=145.79.227.28:4726 , piece_id=D6DH5FHFB5ZTPANRPUWZKRYPFYUV2LS6AY5FQL5RBJZIUNM4FKLQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.065443520 (198h32m7.761292273s) [noise=true , address=77.90.31.11:4554 , piece_id=TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.065970974 (198h32m7.761819777s) [noise=true , address=145.79.227.8:4330 , piece_id=32XBCOEGSUTVUCCWVM2D7CB6FKMNLD6N2N4655B3ZYHNAIL4R2QQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.066422053 (198h32m7.762270806s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.067054427 (198h32m7.76290318s) [noise=true , address=77.90.31.10:4519 , piece_id=MIHLKGHCRTUFQSGBIFKIWPNACAJ4CQOEYRDIKA4KJLFP356VMY6A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.067645042 (198h32m7.763493795s) [noise=true , address=45.91.116.6:4973 , piece_id=Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.068379079 (198h32m7.764227842s) [noise=true , address=77.90.31.19:4843 , piece_id=JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.070931099 (198h32m7.766779862s) [noise=true , address=145.79.227.11:4732 , piece_id=VKSPYLTD6C6XBUFXXVVPHIPGFRCRNQK6EVI4SM7Y6ODOMVGGAJOQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.073100420 (198h32m7.768949173s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.075821872 (198h32m7.771670625s) [noise=true , address=45.91.116.4:4937 , piece_id=CWDA2IYNFUYHVDBJ74Y3CJKCZUBN4QYUEJYFKZNIBAUYYAXO3WFQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.078400863 (198h32m7.774249626s) [noise=true , address=45.91.116.10:4082 , piece_id=RW62GCPC56K2IMOFF276GQECXRKW6NY6W44TNAYFNDGQS3UK7EEA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.081230762 (198h32m7.777079515s) [noise=true , address=145.79.227.15:4876 , piece_id=ZPGKGNXRFEWRRJKTS7NKBEVQA7OAWNWYDR3DUTPHT3XVYRJSKWPQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.083298258 (198h32m7.779147012s) [noise=true , address=145.79.227.20:4438 , piece_id=V6WDUVDB7WHTDCTSYCFWS22UKRD5NMXGJ2GHLSFLVVOXZMZWLL4A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.084602091 (198h32m7.780450844s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:37.640211006 (198h32m9.336059759s) [upload=174963]: Next returned nil segment Nov 7 18:12:37.640227878 (198h32m9.336076621s) [upload=174963]: Waiting for error group managing segments... Nov 7 18:12:38.007509819 (198h32m9.703358572s) [noise=true , address=145.79.227.10:4403 , piece_id=JCMXNJ5PW5ULGOJZ5SGB423VW7VJPC5REYNSNWJZJXHGUQXUS3KA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.015182750 (198h32m9.711031503s) [noise=true , address=145.79.227.28:4726 , piece_id=D6DH5FHFB5ZTPANRPUWZKRYPFYUV2LS6AY5FQL5RBJZIUNM4FKLQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.020400886 (198h32m9.71624964s) [noise=true , address=145.79.227.27:4690 , piece_id=7CVNJTAEBVYJXRWNSFYFP3E7SKMY5XI4CV2JEZJLSXH4TA2F2JZQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.062361869 (198h32m9.758210622s) [noise=true , address=145.79.227.15:4876 , piece_id=ZPGKGNXRFEWRRJKTS7NKBEVQA7OAWNWYDR3DUTPHT3XVYRJSKWPQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.096654439 (198h32m9.792503192s) [noise=true , address=145.79.227.20:4438 , piece_id=V6WDUVDB7WHTDCTSYCFWS22UKRD5NMXGJ2GHLSFLVVOXZMZWLL4A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.244365682 (198h32m9.940214435s) [noise=true , address=145.79.227.8:4330 , piece_id=32XBCOEGSUTVUCCWVM2D7CB6FKMNLD6N2N4655B3ZYHNAIL4R2QQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.245114297 (198h32m9.94096305s) [noise=true , address=145.79.227.11:4732 , piece_id=VKSPYLTD6C6XBUFXXVVPHIPGFRCRNQK6EVI4SM7Y6ODOMVGGAJOQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.324356281 (198h32m10.020205034s) [noise=true , address=77.90.31.11:4554 , piece_id=TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.331696878 (198h32m10.027545621s) [noise=true , address=45.91.116.4:4937 , piece_id=CWDA2IYNFUYHVDBJ74Y3CJKCZUBN4QYUEJYFKZNIBAUYYAXO3WFQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.334142676 (198h32m10.029991429s) [noise=true , address=45.91.116.10:4082 , piece_id=RW62GCPC56K2IMOFF276GQECXRKW6NY6W44TNAYFNDGQS3UK7EEA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.346539031 (198h32m10.042387794s) [noise=true , address=77.90.31.10:4519 , piece_id=MIHLKGHCRTUFQSGBIFKIWPNACAJ4CQOEYRDIKA4KJLFP356VMY6A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.419504032 (198h32m10.115352785s) [noise=true , address=77.90.31.19:4843 , piece_id=JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:39.113150499 (198h32m10.808999252s) [noise=true , address=45.91.116.6:4973 , piece_id=Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:39.113177150 (198h32m10.809025903s) [seg_pos={0 0}, upload=174963]: Segment reached optimal threshold of 13 pieces. Nov 7 18:12:39.113381018 (198h32m10.809229772s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113390166 (198h32m10.809238919s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113460199 (198h32m10.809308952s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113470077 (198h32m10.809318821s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113780338 (198h32m10.809629091s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113788243 (198h32m10.809636986s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113858657 (198h32m10.8097074s) [seg_pos={0 0}, upload=174963]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 7 18:12:39.114030194 (198h32m10.809878947s) [upload=174963]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 7 18:12:39.114038290 (198h32m10.809887043s) [upload=174963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 7 18:12:39.451448640 (198h32m11.147297403s) [upload=174963]: Done uploading segments... Nov 7 18:12:39.451473607 (198h32m11.14732236s) [upload=174963]: Upload finished. err: <nil> Duration: 30.004597055s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......X....................................................] duration: 3.528943605s trace: https://jaeger.dev.storj.io/trace/66eac9b92124ab25 FAIL (2025-11-07 18:12:32 UTC) orbiter warning: context deadline exceeded Duration: 3.529084554s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 42 out of 60 is failed (threshold=3) [XXXXXX.XX.X.XXXXXXXXXX..XXXX.X..XXX.X..X.X.XXXX.X..XX.XXXXX.] duration: 9.028528483s trace: https://jaeger.dev.storj.io/trace/6762ceeb4e2d3cba FAIL (2025-11-07 18:11:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== gments... Nov 7 18:10:44.556559190 (198h30m16.252407943s) [upload=174936]: Got next segment. Inline: false Nov 7 18:10:44.556591612 (198h30m16.252440365s) [upload=174936]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 7 18:10:44.556597022 (198h30m16.252445775s) [upload=174936]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 7 18:10:44.803088385 (198h30m16.498937148s) [seg_pos={0 0}, upload=174936]: Begin upload segment... Nov 7 18:10:44.803157287 (198h30m16.49900604s) [seg_pos={0 0}, upload=174936]: Done begin upload segment. Nov 7 18:10:44.803200188 (198h30m16.499048941s) [noise=true , address=145.79.227.15:4876 , piece_id=VU6NTQUDEIJT7EPBDILHY6TGSQSP22RQ33DP4BIZS5MAOYT5L7EQ, seg_pos={0 0}, upload=174936]: Uploading piece... Nov 7 18:10:44.803296341 (198h30m16.499145104s) [noise=true , address=77.90.31.16:4734 , piece_id=XYGIPGX3PX2I4ELWTZ5FSR5DPOEPFOBN4IFDFY4RZOUWTRKJ6VMA, seg_pos={0 0}, upload=174936]: Uploading piece... Nov 7 18:10:44.804010691 (198h30m16.499859434s) [noise=true , address=145.79.227.18:4005 , piece_id=SDA64VTQVZPIF27HVB2ZOXUJ6GDJMPVZFNVXKZJPHIP2VG6NOTRA, seg_pos={0 0}, upload=174936]: Uploading piece... Nov 7 18:10:44.804529820 (198h30m16.500378573s) [noise=true , address=45.91.116.14:4189 , piece_id=L3B3A3OPTYNWFHSJRICW6HSNXY74E4ORDUOHEE4OLXT6NSFCR3DA, seg_pos={0 0}, upload=174936]: Uploading piece... Nov 7 18:10:44.804968596 (198h30m16.500817349s) [noise=true , address=145.79.227.20:4438 , piece_id=MDKC3IRFLX2KKDP7SBXRFIHCVU4364XRVQBWI53WUEICZ5BZUFNQ, seg_pos={0 0}, upload=174936]: Uploading piece... Nov 7 18:10:44.805499246 (198h30m16.501347999s) [noise=true , address=77.90.31.12:4592 , piece_id=LQAUL2JWUPDLGAUYTZMXW7LUKIT3GWAVT7JUBJNEYP5GUWNVYMDA, seg_pos={0 0}, upload=174936]: Uploading piece... Nov 7 18:10:44.805966305 (198h30m16.501815058s) [noise=true , address=77.90.31.13:4628 , piece_id=GNDLNHW5SQRDLSLGEBZ2JITUPWJFDEQ7E2UNZLYMXELVCTIZBO7A, seg_pos={0 0}, upload=174936]: Uploading piece... Nov 7 18:10:44.806107724 (198h30m16.501956477s) [noise=true , address=145.79.227.9:4366 , piece_id=V3ADPOP5LG2H53GUYSIRYZYJIQTL33NCZM5G43GP3AM2K425BIXA, seg_pos={0 0}, upload=174936]: Uploading piece... Nov 7 18:10:44.806654846 (198h30m16.502503599s) [noise=true , address=45.91.116.5:4297 , piece_id=G5WE647S556ED7SBWYEWLDHW7CJ4Z6GBQEL7262UKYTAZQ7MUV6A, seg_pos={0 0}, upload=174936]: Uploading piece... Nov 7 18:10:44.807354188 (198h30m16.503202941s) [noise=true , address=45.91.116.7:4009 , piece_id=HMWWI7RIOTIQI2ARH72YMQGSBMYMXU2YWZIAJX2NKGZ5BNP5ZEPA, seg_pos={0 0}, upload=174936]: Uploading piece... Nov 7 18:10:44.807690308 (198h30m16.503539051s) [noise=true , address=145.79.227.17:4972 , piece_id=7RND2Z4SKKR6TUHV5ZTPOITC5CQALRLZ2DS6D7UI4FFM3SL7FGWA, seg_pos={0 0}, upload=174936]: Uploading piece... Nov 7 18:10:44.807805657 (198h30m16.50365441s) [noise=true , address=45.91.116.6:4974 , piece_id=XVNUTZ5AZ7IZBP2ADJOTJWGALWQ6HZJ4LMUEAHLCCJCEWCEJLAVQ, seg_pos={0 0}, upload=174936]: Uploading piece... Nov 7 18:10:44.808218494 (198h30m16.504067237s) [noise=true , address=145.79.227.23:4547 , piece_id=ZB3R37HAMAEKGEIJJBWTUAVAOQTC6O3WGNIM4IWLL22HK5AJWBFQ, seg_pos={0 0}, upload=174936]: Uploading piece... Nov 7 18:10:44.808277266 (198h30m16.504126009s) [noise=true , address=145.79.227.25:4619 , piece_id=UWWCC3C3QL6EUU2OO47VMFS3NMVADPAU2C2UK7ZOANOOVLECGM6A, seg_pos={0 0}, upload=174936]: Uploading piece... Nov 7 18:10:44.808731761 (198h30m16.504580514s) [noise=true , address=145.79.227.10:4402 , piece_id=I7JJMDGK762KPDUMDTSFPTMYDYOS74NMHU2C4OTURW4QTVTDAT5A, seg_pos={0 0}, upload=174936]: Uploading piece... Nov 7 18:10:44.809175286 (198h30m16.505024039s) [noise=true , address=145.79.227.27:4689 , piece_id=BPK7ZVYIHHFE7T4ATNOERLCPQPZRSGSZZR5V2NANKSKKAXOTDHMQ, seg_pos={0 0}, upload=174936]: Uploading piece... Nov 7 18:10:46.475905841 (198h30m18.171754584s) [upload=174936]: Next returned nil segment Nov 7 18:10:46.475920959 (198h30m18.171769712s) [upload=174936]: Waiting for error group managing segments... Nov 7 18:10:46.756953564 (198h30m18.452802317s) [noise=true , address=145.79.227.18:4005 , piece_id=SDA64VTQVZPIF27HVB2ZOXUJ6GDJMPVZFNVXKZJPHIP2VG6NOTRA, seg_pos={0 0}, upload=174936]: Done uploading piece. err: <nil> Nov 7 18:10:46.757094672 (198h30m18.452943425s) [noise=true , address=145.79.227.20:4438 , piece_id=MDKC3IRFLX2KKDP7SBXRFIHCVU4364XRVQBWI53WUEICZ5BZUFNQ, seg_pos={0 0}, upload=174936]: Done uploading piece. err: <nil> Nov 7 18:10:46.757955552 (198h30m18.453804305s) [noise=true , address=145.79.227.15:4876 , piece_id=VU6NTQUDEIJT7EPBDILHY6TGSQSP22RQ33DP4BIZS5MAOYT5L7EQ, seg_pos={0 0}, upload=174936]: Done uploading piece. err: <nil> Nov 7 18:10:46.758019474 (198h30m18.453868227s) [noise=true , address=145.79.227.9:4366 , piece_id=V3ADPOP5LG2H53GUYSIRYZYJIQTL33NCZM5G43GP3AM2K425BIXA, seg_pos={0 0}, upload=174936]: Done uploading piece. err: <nil> Nov 7 18:10:46.796667859 (198h30m18.492516602s) [noise=true , address=145.79.227.17:4972 , piece_id=7RND2Z4SKKR6TUHV5ZTPOITC5CQALRLZ2DS6D7UI4FFM3SL7FGWA, seg_pos={0 0}, upload=174936]: Done uploading piece. err: <nil> Nov 7 18:10:46.807827419 (198h30m18.503676172s) [noise=true , address=145.79.227.23:4547 , piece_id=ZB3R37HAMAEKGEIJJBWTUAVAOQTC6O3WGNIM4IWLL22HK5AJWBFQ, seg_pos={0 0}, upload=174936]: Done uploading piece. err: <nil> Nov 7 18:10:46.830442662 (198h30m18.526291405s) [noise=true , address=145.79.227.10:4402 , piece_id=I7JJMDGK762KPDUMDTSFPTMYDYOS74NMHU2C4OTURW4QTVTDAT5A, seg_pos={0 0}, upload=174936]: Done uploading piece. err: <nil> Nov 7 18:10:46.836881693 (198h30m18.532730446s) [noise=true , address=145.79.227.25:4619 , piece_id=UWWCC3C3QL6EUU2OO47VMFS3NMVADPAU2C2UK7ZOANOOVLECGM6A, seg_pos={0 0}, upload=174936]: Done uploading piece. err: <nil> Nov 7 18:10:46.844652741 (198h30m18.540501494s) [noise=true , address=145.79.227.27:4689 , piece_id=BPK7ZVYIHHFE7T4ATNOERLCPQPZRSGSZZR5V2NANKSKKAXOTDHMQ, seg_pos={0 0}, upload=174936]: Done uploading piece. err: <nil> Nov 7 18:10:47.030225661 (198h30m18.726074414s) [noise=true , address=45.91.116.14:4189 , piece_id=L3B3A3OPTYNWFHSJRICW6HSNXY74E4ORDUOHEE4OLXT6NSFCR3DA, seg_pos={0 0}, upload=174936]: Done uploading piece. err: <nil> Nov 7 18:10:47.037841463 (198h30m18.733690216s) [noise=true , address=77.90.31.16:4734 , piece_id=XYGIPGX3PX2I4ELWTZ5FSR5DPOEPFOBN4IFDFY4RZOUWTRKJ6VMA, seg_pos={0 0}, upload=174936]: Done uploading piece. err: <nil> Nov 7 18:10:47.064199594 (198h30m18.760048367s) [noise=true , address=77.90.31.13:4628 , piece_id=GNDLNHW5SQRDLSLGEBZ2JITUPWJFDEQ7E2UNZLYMXELVCTIZBO7A, seg_pos={0 0}, upload=174936]: Done uploading piece. err: <nil> Nov 7 18:10:47.090965923 (198h30m18.786814666s) [noise=true , address=45.91.116.5:4297 , piece_id=G5WE647S556ED7SBWYEWLDHW7CJ4Z6GBQEL7262UKYTAZQ7MUV6A, seg_pos={0 0}, upload=174936]: Done uploading piece. err: <nil> Nov 7 18:10:47.090995199 (198h30m18.786843952s) [seg_pos={0 0}, upload=174936]: Segment reached optimal threshold of 13 pieces. Nov 7 18:10:47.091239924 (198h30m18.787088678s) [noise=true , address=77.90.31.12:4592 , piece_id=LQAUL2JWUPDLGAUYTZMXW7LUKIT3GWAVT7JUBJNEYP5GUWNVYMDA, seg_pos={0 0}, upload=174936]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3): protocol: write tcp 188.166.225.170:55518->77.90.31.12:4592: use of closed network connection; write tcp 188.166.225.170:55518->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55518->77.90.31.12:4592: use of closed network connection Nov 7 18:10:47.091248751 (198h30m18.787097504s) [noise=true , address=77.90.31.12:4592 , piece_id=LQAUL2JWUPDLGAUYTZMXW7LUKIT3GWAVT7JUBJNEYP5GUWNVYMDA, seg_pos={0 0}, upload=174936]: Long tail cancellation due to optimal threshold Nov 7 18:10:47.091425998 (198h30m18.787274752s) [noise=true , address=45.91.116.7:4009 , piece_id=HMWWI7RIOTIQI2ARH72YMQGSBMYMXU2YWZIAJX2NKGZ5BNP5ZEPA, seg_pos={0 0}, upload=174936]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: write tcp 188.166.225.170:32852->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:32852->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32852->45.91.116.7:4009: use of closed network connection Nov 7 18:10:47.091440365 (198h30m18.787289118s) [noise=true , address=45.91.116.7:4009 , piece_id=HMWWI7RIOTIQI2ARH72YMQGSBMYMXU2YWZIAJX2NKGZ5BNP5ZEPA, seg_pos={0 0}, upload=174936]: Long tail cancellation due to optimal threshold Nov 7 18:10:47.091508155 (198h30m18.787356908s) [noise=true , address=45.91.116.6:4974 , piece_id=XVNUTZ5AZ7IZBP2ADJOTJWGALWQ6HZJ4LMUEAHLCCJCEWCEJLAVQ, seg_pos={0 0}, upload=174936]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Nov 7 18:10:47.091517723 (198h30m18.787366477s) [noise=true , address=45.91.116.6:4974 , piece_id=XVNUTZ5AZ7IZBP2ADJOTJWGALWQ6HZJ4LMUEAHLCCJCEWCEJLAVQ, seg_pos={0 0}, upload=174936]: Long tail cancellation due to optimal threshold Nov 7 18:10:47.091566065 (198h30m18.787414818s) [seg_pos={0 0}, upload=174936]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 7 18:10:47.091657440 (198h30m18.787506183s) [upload=174936]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 7 18:10:47.091664022 (198h30m18.787512775s) [upload=174936]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 7 18:10:47.411537288 (198h30m19.107386031s) [upload=174936]: Done uploading segments... Nov 7 18:10:47.411560922 (198h30m19.107409675s) [upload=174936]: Upload finished. err: <nil> Duration: 9.02855845s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........X...................................................] duration: 3.567133537s trace: https://jaeger.dev.storj.io/trace/66eac9b92124ab25 FAIL (2025-11-07 18:11:17 UTC) orbiter warning: context deadline exceeded Duration: 3.567149498s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 43 out of 60 is failed (threshold=3) [XXXXXXX.XX.X.XXXXXXXXXX..XXXX.X..XXX.X..X.X.XXXX.X..XX.XXXXX] duration: 30.004705417s trace: https://jaeger.dev.storj.io/trace/6762ceeb4e2d3cba FAIL (2025-11-07 18:11:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== gments... Nov 7 18:10:44.556559190 (198h30m16.252407943s) [upload=174936]: Got next segment. Inline: false Nov 7 18:10:44.556591612 (198h30m16.252440365s) [upload=174936]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 7 18:10:44.556597022 (198h30m16.252445775s) [upload=174936]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 7 18:10:44.803088385 (198h30m16.498937148s) [seg_pos={0 0}, upload=174936]: Begin upload segment... Nov 7 18:10:44.803157287 (198h30m16.49900604s) [seg_pos={0 0}, upload=174936]: Done begin upload segment. Nov 7 18:10:44.803200188 (198h30m16.499048941s) [noise=true , address=145.79.227.15:4876 , piece_id=VU6NTQUDEIJT7EPBDILHY6TGSQSP22RQ33DP4BIZS5MAOYT5L7EQ, seg_pos={0 0}, upload=174936]: Uploading piece... Nov 7 18:10:44.803296341 (198h30m16.499145104s) [noise=true , address=77.90.31.16:4734 , piece_id=XYGIPGX3PX2I4ELWTZ5FSR5DPOEPFOBN4IFDFY4RZOUWTRKJ6VMA, seg_pos={0 0}, upload=174936]: Uploading piece... Nov 7 18:10:44.804010691 (198h30m16.499859434s) [noise=true , address=145.79.227.18:4005 , piece_id=SDA64VTQVZPIF27HVB2ZOXUJ6GDJMPVZFNVXKZJPHIP2VG6NOTRA, seg_pos={0 0}, upload=174936]: Uploading piece... Nov 7 18:10:44.804529820 (198h30m16.500378573s) [noise=true , address=45.91.116.14:4189 , piece_id=L3B3A3OPTYNWFHSJRICW6HSNXY74E4ORDUOHEE4OLXT6NSFCR3DA, seg_pos={0 0}, upload=174936]: Uploading piece... Nov 7 18:10:44.804968596 (198h30m16.500817349s) [noise=true , address=145.79.227.20:4438 , piece_id=MDKC3IRFLX2KKDP7SBXRFIHCVU4364XRVQBWI53WUEICZ5BZUFNQ, seg_pos={0 0}, upload=174936]: Uploading piece... Nov 7 18:10:44.805499246 (198h30m16.501347999s) [noise=true , address=77.90.31.12:4592 , piece_id=LQAUL2JWUPDLGAUYTZMXW7LUKIT3GWAVT7JUBJNEYP5GUWNVYMDA, seg_pos={0 0}, upload=174936]: Uploading piece... Nov 7 18:10:44.805966305 (198h30m16.501815058s) [noise=true , address=77.90.31.13:4628 , piece_id=GNDLNHW5SQRDLSLGEBZ2JITUPWJFDEQ7E2UNZLYMXELVCTIZBO7A, seg_pos={0 0}, upload=174936]: Uploading piece... Nov 7 18:10:44.806107724 (198h30m16.501956477s) [noise=true , address=145.79.227.9:4366 , piece_id=V3ADPOP5LG2H53GUYSIRYZYJIQTL33NCZM5G43GP3AM2K425BIXA, seg_pos={0 0}, upload=174936]: Uploading piece... Nov 7 18:10:44.806654846 (198h30m16.502503599s) [noise=true , address=45.91.116.5:4297 , piece_id=G5WE647S556ED7SBWYEWLDHW7CJ4Z6GBQEL7262UKYTAZQ7MUV6A, seg_pos={0 0}, upload=174936]: Uploading piece... Nov 7 18:10:44.807354188 (198h30m16.503202941s) [noise=true , address=45.91.116.7:4009 , piece_id=HMWWI7RIOTIQI2ARH72YMQGSBMYMXU2YWZIAJX2NKGZ5BNP5ZEPA, seg_pos={0 0}, upload=174936]: Uploading piece... Nov 7 18:10:44.807690308 (198h30m16.503539051s) [noise=true , address=145.79.227.17:4972 , piece_id=7RND2Z4SKKR6TUHV5ZTPOITC5CQALRLZ2DS6D7UI4FFM3SL7FGWA, seg_pos={0 0}, upload=174936]: Uploading piece... Nov 7 18:10:44.807805657 (198h30m16.50365441s) [noise=true , address=45.91.116.6:4974 , piece_id=XVNUTZ5AZ7IZBP2ADJOTJWGALWQ6HZJ4LMUEAHLCCJCEWCEJLAVQ, seg_pos={0 0}, upload=174936]: Uploading piece... Nov 7 18:10:44.808218494 (198h30m16.504067237s) [noise=true , address=145.79.227.23:4547 , piece_id=ZB3R37HAMAEKGEIJJBWTUAVAOQTC6O3WGNIM4IWLL22HK5AJWBFQ, seg_pos={0 0}, upload=174936]: Uploading piece... Nov 7 18:10:44.808277266 (198h30m16.504126009s) [noise=true , address=145.79.227.25:4619 , piece_id=UWWCC3C3QL6EUU2OO47VMFS3NMVADPAU2C2UK7ZOANOOVLECGM6A, seg_pos={0 0}, upload=174936]: Uploading piece... Nov 7 18:10:44.808731761 (198h30m16.504580514s) [noise=true , address=145.79.227.10:4402 , piece_id=I7JJMDGK762KPDUMDTSFPTMYDYOS74NMHU2C4OTURW4QTVTDAT5A, seg_pos={0 0}, upload=174936]: Uploading piece... Nov 7 18:10:44.809175286 (198h30m16.505024039s) [noise=true , address=145.79.227.27:4689 , piece_id=BPK7ZVYIHHFE7T4ATNOERLCPQPZRSGSZZR5V2NANKSKKAXOTDHMQ, seg_pos={0 0}, upload=174936]: Uploading piece... Nov 7 18:10:46.475905841 (198h30m18.171754584s) [upload=174936]: Next returned nil segment Nov 7 18:10:46.475920959 (198h30m18.171769712s) [upload=174936]: Waiting for error group managing segments... Nov 7 18:10:46.756953564 (198h30m18.452802317s) [noise=true , address=145.79.227.18:4005 , piece_id=SDA64VTQVZPIF27HVB2ZOXUJ6GDJMPVZFNVXKZJPHIP2VG6NOTRA, seg_pos={0 0}, upload=174936]: Done uploading piece. err: <nil> Nov 7 18:10:46.757094672 (198h30m18.452943425s) [noise=true , address=145.79.227.20:4438 , piece_id=MDKC3IRFLX2KKDP7SBXRFIHCVU4364XRVQBWI53WUEICZ5BZUFNQ, seg_pos={0 0}, upload=174936]: Done uploading piece. err: <nil> Nov 7 18:10:46.757955552 (198h30m18.453804305s) [noise=true , address=145.79.227.15:4876 , piece_id=VU6NTQUDEIJT7EPBDILHY6TGSQSP22RQ33DP4BIZS5MAOYT5L7EQ, seg_pos={0 0}, upload=174936]: Done uploading piece. err: <nil> Nov 7 18:10:46.758019474 (198h30m18.453868227s) [noise=true , address=145.79.227.9:4366 , piece_id=V3ADPOP5LG2H53GUYSIRYZYJIQTL33NCZM5G43GP3AM2K425BIXA, seg_pos={0 0}, upload=174936]: Done uploading piece. err: <nil> Nov 7 18:10:46.796667859 (198h30m18.492516602s) [noise=true , address=145.79.227.17:4972 , piece_id=7RND2Z4SKKR6TUHV5ZTPOITC5CQALRLZ2DS6D7UI4FFM3SL7FGWA, seg_pos={0 0}, upload=174936]: Done uploading piece. err: <nil> Nov 7 18:10:46.807827419 (198h30m18.503676172s) [noise=true , address=145.79.227.23:4547 , piece_id=ZB3R37HAMAEKGEIJJBWTUAVAOQTC6O3WGNIM4IWLL22HK5AJWBFQ, seg_pos={0 0}, upload=174936]: Done uploading piece. err: <nil> Nov 7 18:10:46.830442662 (198h30m18.526291405s) [noise=true , address=145.79.227.10:4402 , piece_id=I7JJMDGK762KPDUMDTSFPTMYDYOS74NMHU2C4OTURW4QTVTDAT5A, seg_pos={0 0}, upload=174936]: Done uploading piece. err: <nil> Nov 7 18:10:46.836881693 (198h30m18.532730446s) [noise=true , address=145.79.227.25:4619 , piece_id=UWWCC3C3QL6EUU2OO47VMFS3NMVADPAU2C2UK7ZOANOOVLECGM6A, seg_pos={0 0}, upload=174936]: Done uploading piece. err: <nil> Nov 7 18:10:46.844652741 (198h30m18.540501494s) [noise=true , address=145.79.227.27:4689 , piece_id=BPK7ZVYIHHFE7T4ATNOERLCPQPZRSGSZZR5V2NANKSKKAXOTDHMQ, seg_pos={0 0}, upload=174936]: Done uploading piece. err: <nil> Nov 7 18:10:47.030225661 (198h30m18.726074414s) [noise=true , address=45.91.116.14:4189 , piece_id=L3B3A3OPTYNWFHSJRICW6HSNXY74E4ORDUOHEE4OLXT6NSFCR3DA, seg_pos={0 0}, upload=174936]: Done uploading piece. err: <nil> Nov 7 18:10:47.037841463 (198h30m18.733690216s) [noise=true , address=77.90.31.16:4734 , piece_id=XYGIPGX3PX2I4ELWTZ5FSR5DPOEPFOBN4IFDFY4RZOUWTRKJ6VMA, seg_pos={0 0}, upload=174936]: Done uploading piece. err: <nil> Nov 7 18:10:47.064199594 (198h30m18.760048367s) [noise=true , address=77.90.31.13:4628 , piece_id=GNDLNHW5SQRDLSLGEBZ2JITUPWJFDEQ7E2UNZLYMXELVCTIZBO7A, seg_pos={0 0}, upload=174936]: Done uploading piece. err: <nil> Nov 7 18:10:47.090965923 (198h30m18.786814666s) [noise=true , address=45.91.116.5:4297 , piece_id=G5WE647S556ED7SBWYEWLDHW7CJ4Z6GBQEL7262UKYTAZQ7MUV6A, seg_pos={0 0}, upload=174936]: Done uploading piece. err: <nil> Nov 7 18:10:47.090995199 (198h30m18.786843952s) [seg_pos={0 0}, upload=174936]: Segment reached optimal threshold of 13 pieces. Nov 7 18:10:47.091239924 (198h30m18.787088678s) [noise=true , address=77.90.31.12:4592 , piece_id=LQAUL2JWUPDLGAUYTZMXW7LUKIT3GWAVT7JUBJNEYP5GUWNVYMDA, seg_pos={0 0}, upload=174936]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3): protocol: write tcp 188.166.225.170:55518->77.90.31.12:4592: use of closed network connection; write tcp 188.166.225.170:55518->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55518->77.90.31.12:4592: use of closed network connection Nov 7 18:10:47.091248751 (198h30m18.787097504s) [noise=true , address=77.90.31.12:4592 , piece_id=LQAUL2JWUPDLGAUYTZMXW7LUKIT3GWAVT7JUBJNEYP5GUWNVYMDA, seg_pos={0 0}, upload=174936]: Long tail cancellation due to optimal threshold Nov 7 18:10:47.091425998 (198h30m18.787274752s) [noise=true , address=45.91.116.7:4009 , piece_id=HMWWI7RIOTIQI2ARH72YMQGSBMYMXU2YWZIAJX2NKGZ5BNP5ZEPA, seg_pos={0 0}, upload=174936]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: write tcp 188.166.225.170:32852->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:32852->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32852->45.91.116.7:4009: use of closed network connection Nov 7 18:10:47.091440365 (198h30m18.787289118s) [noise=true , address=45.91.116.7:4009 , piece_id=HMWWI7RIOTIQI2ARH72YMQGSBMYMXU2YWZIAJX2NKGZ5BNP5ZEPA, seg_pos={0 0}, upload=174936]: Long tail cancellation due to optimal threshold Nov 7 18:10:47.091508155 (198h30m18.787356908s) [noise=true , address=45.91.116.6:4974 , piece_id=XVNUTZ5AZ7IZBP2ADJOTJWGALWQ6HZJ4LMUEAHLCCJCEWCEJLAVQ, seg_pos={0 0}, upload=174936]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Nov 7 18:10:47.091517723 (198h30m18.787366477s) [noise=true , address=45.91.116.6:4974 , piece_id=XVNUTZ5AZ7IZBP2ADJOTJWGALWQ6HZJ4LMUEAHLCCJCEWCEJLAVQ, seg_pos={0 0}, upload=174936]: Long tail cancellation due to optimal threshold Nov 7 18:10:47.091566065 (198h30m18.787414818s) [seg_pos={0 0}, upload=174936]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 7 18:10:47.091657440 (198h30m18.787506183s) [upload=174936]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 7 18:10:47.091664022 (198h30m18.787512775s) [upload=174936]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 7 18:10:47.411537288 (198h30m19.107386031s) [upload=174936]: Done uploading segments... Nov 7 18:10:47.411560922 (198h30m19.107409675s) [upload=174936]: Upload finished. err: <nil> Duration: 30.004736436s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 43 out of 60 is failed (threshold=3) [XXXXXXXX.XX.X.XXXXXXXXXX..XXXX.X..XXX.X..X.X.XXXX.X..XX.XXXX] duration: 30.004100386s trace: https://jaeger.dev.storj.io/trace/34d89a0075253366 FAIL (2025-11-07 18:10:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: PZDVPSNYMKMBPXOJERQSS4SY4H5QIA55ZFXPFUJKXBEALKUNU5HA) context deadline exceeded; piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: IO2BTQ52CIAG5LDWK4ORDUNJMGSRIUN22OMRDVPQUPL2Z5LK5ORQ) context deadline exceeded ======LOGS====== Nov 7 18:10:05.744253838 (198h29m37.440102601s) [upload=174927]: Starting upload Nov 7 18:10:05.744360551 (198h29m37.440209314s) [upload=174927]: Done starting upload Nov 7 18:10:05.744566864 (198h29m37.440415627s) [upload=174927]: Uploading segments... Nov 7 18:10:05.745084530 (198h29m37.440933293s) [upload=174927]: Got next segment. Inline: false Nov 7 18:10:05.745135998 (198h29m37.440984751s) [upload=174927]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 7 18:10:05.745142680 (198h29m37.440991433s) [upload=174927]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 7 18:10:05.996902544 (198h29m37.692751307s) [seg_pos={0 0}, upload=174927]: Begin upload segment... Nov 7 18:10:05.997040677 (198h29m37.69288943s) [seg_pos={0 0}, upload=174927]: Done begin upload segment. Nov 7 18:10:05.997081585 (198h29m37.692930338s) [noise=true , address=77.90.31.15:4697 , piece_id=LXIQO7WS773RZBULGGMNNS72EHWANHB25K5I6C4ITZ5LRTLETUPA, seg_pos={0 0}, upload=174927]: Uploading piece... Nov 7 18:10:06.007491417 (198h29m37.70334017s) [noise=true , address=45.91.116.12:4227 , piece_id=PZDVPSNYMKMBPXOJERQSS4SY4H5QIA55ZFXPFUJKXBEALKUNU5HA, seg_pos={0 0}, upload=174927]: Uploading piece... Nov 7 18:10:06.009787048 (198h29m37.705635801s) [noise=true , address=145.79.227.23:4547 , piece_id=PIXIVREAUTYMUBMUH5DLWB5GSCDMHB64W5DIFVF26CABBH3YZ3VA, seg_pos={0 0}, upload=174927]: Uploading piece... Nov 7 18:10:06.011888460 (198h29m37.707737253s) [noise=true , address=145.79.227.30:4761 , piece_id=Y7BUFJNHVEPO3FHKGXU2X3HBPM5FAR2DPQJFCWWHY7HTTQDKJBVQ, seg_pos={0 0}, upload=174927]: Uploading piece... Nov 7 18:10:06.013676024 (198h29m37.709524777s) [noise=true , address=145.79.227.10:4402 , piece_id=HFBLJIPZHADP2C3SOFQQA5BWIAQHUBZYYNYSQXOF2VVSXKJQH5KQ, seg_pos={0 0}, upload=174927]: Uploading piece... Nov 7 18:10:06.013980834 (198h29m37.709829588s) [noise=true , address=145.79.227.17:4972 , piece_id=5PP6QIUDWK3CYPUAQD5MHHXOXA735DG7UGWLW6HBSWTZRYXNADNA, seg_pos={0 0}, upload=174927]: Uploading piece... Nov 7 18:10:06.015727059 (198h29m37.711575812s) [noise=true , address=45.91.116.13:4154 , piece_id=IO2BTQ52CIAG5LDWK4ORDUNJMGSRIUN22OMRDVPQUPL2Z5LK5ORQ, seg_pos={0 0}, upload=174927]: Uploading piece... Nov 7 18:10:06.017508922 (198h29m37.713357675s) [noise=true , address=145.79.227.8:4330 , piece_id=5CQSJDWMS7B4ODJTLHP4EY5WDAF4R4FADATARHDAPO7A5POJ6YRA, seg_pos={0 0}, upload=174927]: Uploading piece... Nov 7 18:10:06.021690374 (198h29m37.717539127s) [noise=true , address=145.79.227.15:4876 , piece_id=G7OIL6ASP6AK4IJSKE3VPDLOHL2ORNEPRMUHNCHNFA3HSUS2FNIA, seg_pos={0 0}, upload=174927]: Uploading piece... Nov 7 18:10:06.023837253 (198h29m37.719686006s) [noise=true , address=77.90.31.14:4661 , piece_id=AEHXF2CRYRKGQ2TF6OOF6LAZCWVJWMNCUBVXXTXG57IN5TXSMVKQ, seg_pos={0 0}, upload=174927]: Uploading piece... Nov 7 18:10:06.050548467 (198h29m37.74639722s) [noise=true , address=45.91.116.14:4189 , piece_id=TYGAH7PNZHMHXOB6Q3U2FIVXMMII5L6CWI4OAKBSVQ4X7532VRJA, seg_pos={0 0}, upload=174927]: Uploading piece... Nov 7 18:10:06.052824331 (198h29m37.748673084s) [noise=true , address=145.79.227.28:4727 , piece_id=2DSC5IZOJEKENM4OW3GYXGELXIXUNO67O2XGZ5L6JFLJO2OSX5GA, seg_pos={0 0}, upload=174927]: Uploading piece... Nov 7 18:10:06.054868723 (198h29m37.750717507s) [noise=true , address=145.79.227.11:4732 , piece_id=VEUKQ5PO5D6FWSXIUFK2ONW4NNOVEM6YDPHMIBWLM5GXFXOCM4AA, seg_pos={0 0}, upload=174927]: Uploading piece... Nov 7 18:10:06.057464447 (198h29m37.75331326s) [noise=true , address=145.79.227.24:4581 , piece_id=U52WWJLDDXOKICXCV7QMK7GNQH5IQTXVNHL5IATJ7KSCD5O5WBLQ, seg_pos={0 0}, upload=174927]: Uploading piece... Nov 7 18:10:06.060258667 (198h29m37.75610743s) [noise=true , address=145.79.227.14:4840 , piece_id=5FTTYVGIBP2K2GUUBMV2PBL6FQO5JM6HVKS4PBYSH4FCNUDKQ7WQ, seg_pos={0 0}, upload=174927]: Uploading piece... Nov 7 18:10:06.062157894 (198h29m37.758006647s) [noise=true , address=145.79.227.21:4474 , piece_id=KRPKJLCHATR7XNQXDK4VJWWP3SJCIQCQVYNG4L2CO35CBLDBLIQA, seg_pos={0 0}, upload=174927]: Uploading piece... Nov 7 18:10:07.727683511 (198h29m39.423532264s) [upload=174927]: Next returned nil segment Nov 7 18:10:07.727702707 (198h29m39.42355146s) [upload=174927]: Waiting for error group managing segments... Nov 7 18:10:08.012137239 (198h29m39.707985993s) [noise=true , address=145.79.227.28:4727 , piece_id=2DSC5IZOJEKENM4OW3GYXGELXIXUNO67O2XGZ5L6JFLJO2OSX5GA, seg_pos={0 0}, upload=174927]: Done uploading piece. err: <nil> Nov 7 18:10:08.020478573 (198h29m39.716327326s) [noise=true , address=145.79.227.15:4876 , piece_id=G7OIL6ASP6AK4IJSKE3VPDLOHL2ORNEPRMUHNCHNFA3HSUS2FNIA, seg_pos={0 0}, upload=174927]: Done uploading piece. err: <nil> Nov 7 18:10:08.091355648 (198h29m39.787204411s) [noise=true , address=145.79.227.30:4761 , piece_id=Y7BUFJNHVEPO3FHKGXU2X3HBPM5FAR2DPQJFCWWHY7HTTQDKJBVQ, seg_pos={0 0}, upload=174927]: Done uploading piece. err: <nil> Nov 7 18:10:08.186815393 (198h29m39.882664146s) [noise=true , address=145.79.227.14:4840 , piece_id=5FTTYVGIBP2K2GUUBMV2PBL6FQO5JM6HVKS4PBYSH4FCNUDKQ7WQ, seg_pos={0 0}, upload=174927]: Done uploading piece. err: <nil> Nov 7 18:10:08.227396918 (198h29m39.923245671s) [noise=true , address=145.79.227.21:4474 , piece_id=KRPKJLCHATR7XNQXDK4VJWWP3SJCIQCQVYNG4L2CO35CBLDBLIQA, seg_pos={0 0}, upload=174927]: Done uploading piece. err: <nil> Nov 7 18:10:08.249512492 (198h29m39.945361245s) [noise=true , address=145.79.227.10:4402 , piece_id=HFBLJIPZHADP2C3SOFQQA5BWIAQHUBZYYNYSQXOF2VVSXKJQH5KQ, seg_pos={0 0}, upload=174927]: Done uploading piece. err: <nil> Nov 7 18:10:08.266019183 (198h29m39.961867926s) [noise=true , address=45.91.116.12:4227 , piece_id=PZDVPSNYMKMBPXOJERQSS4SY4H5QIA55ZFXPFUJKXBEALKUNU5HA, seg_pos={0 0}, upload=174927]: Done uploading piece. err: <nil> Nov 7 18:10:08.271236408 (198h29m39.967085171s) [noise=true , address=77.90.31.14:4661 , piece_id=AEHXF2CRYRKGQ2TF6OOF6LAZCWVJWMNCUBVXXTXG57IN5TXSMVKQ, seg_pos={0 0}, upload=174927]: Done uploading piece. err: <nil> Nov 7 18:10:08.275270701 (198h29m39.971119454s) [noise=true , address=45.91.116.14:4189 , piece_id=TYGAH7PNZHMHXOB6Q3U2FIVXMMII5L6CWI4OAKBSVQ4X7532VRJA, seg_pos={0 0}, upload=174927]: Done uploading piece. err: <nil> Nov 7 18:10:08.302117252 (198h29m39.997966015s) [noise=true , address=77.90.31.15:4697 , piece_id=LXIQO7WS773RZBULGGMNNS72EHWANHB25K5I6C4ITZ5LRTLETUPA, seg_pos={0 0}, upload=174927]: Done uploading piece. err: <nil> Nov 7 18:10:08.333843996 (198h29m40.029692749s) [noise=true , address=145.79.227.11:4732 , piece_id=VEUKQ5PO5D6FWSXIUFK2ONW4NNOVEM6YDPHMIBWLM5GXFXOCM4AA, seg_pos={0 0}, upload=174927]: Done uploading piece. err: <nil> Nov 7 18:10:08.434127608 (198h29m40.129976361s) [noise=true , address=45.91.116.13:4154 , piece_id=IO2BTQ52CIAG5LDWK4ORDUNJMGSRIUN22OMRDVPQUPL2Z5LK5ORQ, seg_pos={0 0}, upload=174927]: Done uploading piece. err: <nil> Nov 7 18:10:08.497341410 (198h29m40.193190163s) [noise=true , address=145.79.227.24:4581 , piece_id=U52WWJLDDXOKICXCV7QMK7GNQH5IQTXVNHL5IATJ7KSCD5O5WBLQ, seg_pos={0 0}, upload=174927]: Done uploading piece. err: <nil> Nov 7 18:10:08.497365436 (198h29m40.193214179s) [seg_pos={0 0}, upload=174927]: Segment reached optimal threshold of 13 pieces. Nov 7 18:10:08.497650288 (198h29m40.193499051s) [noise=true , address=145.79.227.8:4330 , piece_id=5CQSJDWMS7B4ODJTLHP4EY5WDAF4R4FADATARHDAPO7A5POJ6YRA, seg_pos={0 0}, upload=174927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW): protocol: expected piece hash; context canceled; EOF Nov 7 18:10:08.497660858 (198h29m40.193509611s) [noise=true , address=145.79.227.8:4330 , piece_id=5CQSJDWMS7B4ODJTLHP4EY5WDAF4R4FADATARHDAPO7A5POJ6YRA, seg_pos={0 0}, upload=174927]: Long tail cancellation due to optimal threshold Nov 7 18:10:08.497768804 (198h29m40.193617557s) [noise=true , address=145.79.227.23:4547 , piece_id=PIXIVREAUTYMUBMUH5DLWB5GSCDMHB64W5DIFVF26CABBH3YZ3VA, seg_pos={0 0}, upload=174927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW): protocol: expected piece hash; context canceled; EOF Nov 7 18:10:08.497837194 (198h29m40.193685948s) [noise=true , address=145.79.227.23:4547 , piece_id=PIXIVREAUTYMUBMUH5DLWB5GSCDMHB64W5DIFVF26CABBH3YZ3VA, seg_pos={0 0}, upload=174927]: Long tail cancellation due to optimal threshold Nov 7 18:10:08.497893271 (198h29m40.193742024s) [noise=true , address=145.79.227.17:4972 , piece_id=5PP6QIUDWK3CYPUAQD5MHHXOXA735DG7UGWLW6HBSWTZRYXNADNA, seg_pos={0 0}, upload=174927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: expected piece hash; context canceled; EOF Nov 7 18:10:08.497902788 (198h29m40.193751542s) [noise=true , address=145.79.227.17:4972 , piece_id=5PP6QIUDWK3CYPUAQD5MHHXOXA735DG7UGWLW6HBSWTZRYXNADNA, seg_pos={0 0}, upload=174927]: Long tail cancellation due to optimal threshold Nov 7 18:10:08.497927406 (198h29m40.193776159s) [seg_pos={0 0}, upload=174927]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 7 18:10:08.498040180 (198h29m40.193888943s) [upload=174927]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 7 18:10:08.498048487 (198h29m40.19389724s) [upload=174927]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 7 18:10:08.791936040 (198h29m40.487784833s) [upload=174927]: Done uploading segments... Nov 7 18:10:08.791967149 (198h29m40.487815902s) [upload=174927]: Upload finished. err: <nil> Duration: 30.004128199s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........X..................................................] duration: 3.54690111s trace: https://jaeger.dev.storj.io/trace/66eac9b92124ab25 FAIL (2025-11-07 18:10:03 UTC) orbiter warning: context deadline exceeded Duration: 3.546928661s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 42 out of 60 is failed (threshold=3) [.XXXXXXXX.XX.X.XXXXXXXXXX..XXXX.X..XXX.X..X.X.XXXX.X..XX.XXX] duration: 30.003858667s trace: https://jaeger.dev.storj.io/trace/5e8ae5e5264b98c6 FAIL (2025-11-07 18:09:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 7 18:09:11.898996245 (198h28m43.594844988s) [upload=174916]: Starting upload Nov 7 18:09:11.899071499 (198h28m43.594920252s) [upload=174916]: Done starting upload Nov 7 18:09:11.899299773 (198h28m43.595148526s) [upload=174916]: Uploading segments... Nov 7 18:09:11.900453710 (198h28m43.596302463s) [upload=174916]: Got next segment. Inline: false Nov 7 18:09:11.900630066 (198h28m43.596478819s) [upload=174916]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 7 18:09:11.900637842 (198h28m43.596486595s) [upload=174916]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 7 18:09:12.151351593 (198h28m43.847200346s) [seg_pos={0 0}, upload=174916]: Begin upload segment... Nov 7 18:09:12.151439960 (198h28m43.847288713s) [seg_pos={0 0}, upload=174916]: Done begin upload segment. Nov 7 18:09:12.151520464 (198h28m43.847369207s) [noise=true , address=145.79.227.27:4690 , piece_id=FCSUXHRG2HKLYF423FG3W7ZJOM7S6OK4D2263IIVW256ZPFRPOEQ, seg_pos={0 0}, upload=174916]: Uploading piece... Nov 7 18:09:12.152339784 (198h28m43.848188537s) [noise=true , address=145.79.227.16:4910 , piece_id=HGOAAGYKHQG43LVML4XBEAOQXFGLGVYRKMYVHM3MKXJQPU5MVGVQ, seg_pos={0 0}, upload=174916]: Uploading piece... Nov 7 18:09:12.153267421 (198h28m43.849116174s) [noise=true , address=145.79.227.20:4439 , piece_id=7SXPLI6BIGQLAGPBNZ65ZMDVVHQTFC2NNGKYGDKQU2B6AW3UZHWQ, seg_pos={0 0}, upload=174916]: Uploading piece... Nov 7 18:09:12.153356279 (198h28m43.849205032s) [noise=true , address=77.90.31.12:4592 , piece_id=I36CLKY2NE3KFK35XVF6LLQ7O4F773QY4Q5B4MZUMRLUJK34BQ7A, seg_pos={0 0}, upload=174916]: Uploading piece... Nov 7 18:09:12.153726675 (198h28m43.849575418s) [noise=true , address=145.79.227.17:4972 , piece_id=J5OIIAXAGY7ALSJY6MGAKTEOWTJZX4P5RLJXGFWMFY7DGKXR7LSA, seg_pos={0 0}, upload=174916]: Uploading piece... Nov 7 18:09:12.154083113 (198h28m43.849931867s) [noise=true , address=77.90.31.15:4698 , piece_id=UR7ZUAXICZ6WC7JIW2G5EKNMV4EKMFXM546SFF6QTXYD74RBONIA, seg_pos={0 0}, upload=174916]: Uploading piece... Nov 7 18:09:12.154101589 (198h28m43.849950342s) [noise=true , address=45.91.116.9:4262 , piece_id=GWPWKPBPSDUPPQCGP54PSS33QTOUIXET4QKZUAECLSZIPBX4HUZA, seg_pos={0 0}, upload=174916]: Uploading piece... Nov 7 18:09:12.154602232 (198h28m43.850450995s) [noise=true , address=145.79.227.8:4330 , piece_id=UZZTP7LWY63K3ZQB6AV4ORVZ6E7FRP4LQKJS6ZQTTGGJOVJ2RDMQ, seg_pos={0 0}, upload=174916]: Uploading piece... Nov 7 18:09:12.154847810 (198h28m43.850696563s) [noise=true , address=145.79.227.14:4839 , piece_id=IXIAFSDZZN2NAFD5G635QCNASASP3SXC772RLKQLVUESNG7WEC6Q, seg_pos={0 0}, upload=174916]: Uploading piece... Nov 7 18:09:12.155834088 (198h28m43.851682841s) [noise=true , address=77.90.31.18:4805 , piece_id=OWC5DFEH3PMM5KXMZRFKLC7GH7NLGPLSZX3KZLLZ7OIRSDF54QDA, seg_pos={0 0}, upload=174916]: Uploading piece... Nov 7 18:09:12.156285959 (198h28m43.852134712s) [noise=true , address=145.79.227.32:4834 , piece_id=33TAXPFRRX56TZNEIETUWEVBRFM6IO4IJJWHQVQIF4JGN42XQVRQ, seg_pos={0 0}, upload=174916]: Uploading piece... Nov 7 18:09:12.156695768 (198h28m43.852544521s) [noise=true , address=45.91.116.9:4263 , piece_id=OPZWRB4WGDS7B5YYYPOW55YK3RYDUBUM3T6XUYIIJYNB6SMJUHMQ, seg_pos={0 0}, upload=174916]: Uploading piece... Nov 7 18:09:12.157092114 (198h28m43.852940867s) [noise=true , address=145.79.227.22:4511 , piece_id=UZSJI2EXVYOTZIRD4NHJQJDO6BWYHPOCJM743YKF4UYQLP2AZ2UA, seg_pos={0 0}, upload=174916]: Uploading piece... Nov 7 18:09:12.157422342 (198h28m43.853271095s) [noise=true , address=145.79.227.24:4582 , piece_id=MZIJANPGWTWWLUACHYGB4IEJKC5TB7KZHHH6SG5U4H7RJPSLJU2A, seg_pos={0 0}, upload=174916]: Uploading piece... Nov 7 18:09:12.158581911 (198h28m43.854430664s) [noise=true , address=145.79.227.13:4803 , piece_id=36OXMBBJ3K6S3334CW46HVMKYSWRIYB7ZOSVXP6IULMW3SLLIMRA, seg_pos={0 0}, upload=174916]: Uploading piece... Nov 7 18:09:12.161014954 (198h28m43.856863707s) [noise=true , address=77.90.31.13:4628 , piece_id=OYINMPDQHOBYVSWM4APTN5HKC64HESGLU7E2SJLKIWUBDCNDDQEQ, seg_pos={0 0}, upload=174916]: Uploading piece... Nov 7 18:09:14.017691283 (198h28m45.713540036s) [upload=174916]: Next returned nil segment Nov 7 18:09:14.017710378 (198h28m45.713559131s) [upload=174916]: Waiting for error group managing segments... Nov 7 18:09:14.311965823 (198h28m46.007814586s) [noise=true , address=145.79.227.32:4834 , piece_id=33TAXPFRRX56TZNEIETUWEVBRFM6IO4IJJWHQVQIF4JGN42XQVRQ, seg_pos={0 0}, upload=174916]: Done uploading piece. err: <nil> Nov 7 18:09:14.378354470 (198h28m46.074203233s) [noise=true , address=145.79.227.8:4330 , piece_id=UZZTP7LWY63K3ZQB6AV4ORVZ6E7FRP4LQKJS6ZQTTGGJOVJ2RDMQ, seg_pos={0 0}, upload=174916]: Done uploading piece. err: <nil> Nov 7 18:09:14.380859269 (198h28m46.076708023s) [noise=true , address=145.79.227.17:4972 , piece_id=J5OIIAXAGY7ALSJY6MGAKTEOWTJZX4P5RLJXGFWMFY7DGKXR7LSA, seg_pos={0 0}, upload=174916]: Done uploading piece. err: <nil> Nov 7 18:09:14.382331754 (198h28m46.078180507s) [noise=true , address=145.79.227.22:4511 , piece_id=UZSJI2EXVYOTZIRD4NHJQJDO6BWYHPOCJM743YKF4UYQLP2AZ2UA, seg_pos={0 0}, upload=174916]: Done uploading piece. err: <nil> Nov 7 18:09:14.383845115 (198h28m46.079693868s) [noise=true , address=77.90.31.12:4592 , piece_id=I36CLKY2NE3KFK35XVF6LLQ7O4F773QY4Q5B4MZUMRLUJK34BQ7A, seg_pos={0 0}, upload=174916]: Done uploading piece. err: <nil> Nov 7 18:09:14.390244822 (198h28m46.086093565s) [noise=true , address=45.91.116.9:4263 , piece_id=OPZWRB4WGDS7B5YYYPOW55YK3RYDUBUM3T6XUYIIJYNB6SMJUHMQ, seg_pos={0 0}, upload=174916]: Done uploading piece. err: <nil> Nov 7 18:09:14.456167923 (198h28m46.152016676s) [noise=true , address=145.79.227.24:4582 , piece_id=MZIJANPGWTWWLUACHYGB4IEJKC5TB7KZHHH6SG5U4H7RJPSLJU2A, seg_pos={0 0}, upload=174916]: Done uploading piece. err: <nil> Nov 7 18:09:14.456769249 (198h28m46.152618002s) [noise=true , address=145.79.227.27:4690 , piece_id=FCSUXHRG2HKLYF423FG3W7ZJOM7S6OK4D2263IIVW256ZPFRPOEQ, seg_pos={0 0}, upload=174916]: Done uploading piece. err: <nil> Nov 7 18:09:14.465768796 (198h28m46.161617549s) [noise=true , address=145.79.227.13:4803 , piece_id=36OXMBBJ3K6S3334CW46HVMKYSWRIYB7ZOSVXP6IULMW3SLLIMRA, seg_pos={0 0}, upload=174916]: Done uploading piece. err: <nil> Nov 7 18:09:14.473323431 (198h28m46.169172184s) [noise=true , address=45.91.116.9:4262 , piece_id=GWPWKPBPSDUPPQCGP54PSS33QTOUIXET4QKZUAECLSZIPBX4HUZA, seg_pos={0 0}, upload=174916]: Done uploading piece. err: <nil> Nov 7 18:09:14.511445763 (198h28m46.207294526s) [noise=true , address=77.90.31.13:4628 , piece_id=OYINMPDQHOBYVSWM4APTN5HKC64HESGLU7E2SJLKIWUBDCNDDQEQ, seg_pos={0 0}, upload=174916]: Done uploading piece. err: <nil> Nov 7 18:09:14.520953549 (198h28m46.216802312s) [noise=true , address=145.79.227.20:4439 , piece_id=7SXPLI6BIGQLAGPBNZ65ZMDVVHQTFC2NNGKYGDKQU2B6AW3UZHWQ, seg_pos={0 0}, upload=174916]: Done uploading piece. err: <nil> Nov 7 18:09:14.539041211 (198h28m46.234889964s) [noise=true , address=145.79.227.16:4910 , piece_id=HGOAAGYKHQG43LVML4XBEAOQXFGLGVYRKMYVHM3MKXJQPU5MVGVQ, seg_pos={0 0}, upload=174916]: Done uploading piece. err: <nil> Nov 7 18:09:14.539067972 (198h28m46.234916735s) [seg_pos={0 0}, upload=174916]: Segment reached optimal threshold of 13 pieces. Nov 7 18:09:14.539289844 (198h28m46.235138597s) [noise=true , address=77.90.31.18:4805 , piece_id=OWC5DFEH3PMM5KXMZRFKLC7GH7NLGPLSZX3KZLLZ7OIRSDF54QDA, seg_pos={0 0}, upload=174916]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: expected piece hash; context canceled; EOF Nov 7 18:09:14.539299262 (198h28m46.235148015s) [noise=true , address=77.90.31.18:4805 , piece_id=OWC5DFEH3PMM5KXMZRFKLC7GH7NLGPLSZX3KZLLZ7OIRSDF54QDA, seg_pos={0 0}, upload=174916]: Long tail cancellation due to optimal threshold Nov 7 18:09:14.540095759 (198h28m46.235944522s) [noise=true , address=77.90.31.15:4698 , piece_id=UR7ZUAXICZ6WC7JIW2G5EKNMV4EKMFXM546SFF6QTXYD74RBONIA, seg_pos={0 0}, upload=174916]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj): protocol: write tcp 188.166.225.170:59006->77.90.31.15:4698: use of closed network connection; write tcp 188.166.225.170:59006->77.90.31.15:4698: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59006->77.90.31.15:4698: use of closed network connection Nov 7 18:09:14.540106801 (198h28m46.235955564s) [noise=true , address=77.90.31.15:4698 , piece_id=UR7ZUAXICZ6WC7JIW2G5EKNMV4EKMFXM546SFF6QTXYD74RBONIA, seg_pos={0 0}, upload=174916]: Long tail cancellation due to optimal threshold Nov 7 18:09:14.540319886 (198h28m46.236168629s) [noise=true , address=145.79.227.14:4839 , piece_id=IXIAFSDZZN2NAFD5G635QCNASASP3SXC772RLKQLVUESNG7WEC6Q, seg_pos={0 0}, upload=174916]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv): protocol: expected piece hash; context canceled; EOF Nov 7 18:09:14.540329474 (198h28m46.236178227s) [noise=true , address=145.79.227.14:4839 , piece_id=IXIAFSDZZN2NAFD5G635QCNASASP3SXC772RLKQLVUESNG7WEC6Q, seg_pos={0 0}, upload=174916]: Long tail cancellation due to optimal threshold Nov 7 18:09:14.540351306 (198h28m46.236200049s) [seg_pos={0 0}, upload=174916]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 7 18:09:14.540451247 (198h28m46.2363s) [upload=174916]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 7 18:09:14.540458290 (198h28m46.236307043s) [upload=174916]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 7 18:09:14.832145784 (198h28m46.527994527s) [upload=174916]: Done uploading segments... Nov 7 18:09:14.832169990 (198h28m46.528018743s) [upload=174916]: Upload finished. err: <nil> Duration: 30.003878135s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........X.................................................] duration: 3.571196484s trace: https://jaeger.dev.storj.io/trace/66eac9b92124ab25 FAIL (2025-11-07 18:09:37 UTC) orbiter warning: context deadline exceeded Duration: 3.571232433s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)