Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 5.918044319s trace: https://jaeger.dev.storj.io/trace/422f33ac8ae6a74a FAIL (2025-11-07 18:58:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ) context deadline exceeded; piecestore: (Node ID: 12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, Piece ID: JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ) context deadline exceeded ======LOGS====== Nov 7 18:12:35.791697270 (198h32m7.487546023s) [upload=174963]: Starting upload Nov 7 18:12:35.791791689 (198h32m7.487640452s) [upload=174963]: Done starting upload Nov 7 18:12:35.791885458 (198h32m7.487734211s) [upload=174963]: Uploading segments... Nov 7 18:12:35.792146455 (198h32m7.487995208s) [upload=174963]: Got next segment. Inline: false Nov 7 18:12:35.792195278 (198h32m7.488044021s) [upload=174963]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 7 18:12:35.792201820 (198h32m7.488050563s) [upload=174963]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 7 18:12:36.063867588 (198h32m7.759716341s) [seg_pos={0 0}, upload=174963]: Begin upload segment... Nov 7 18:12:36.063937832 (198h32m7.759786585s) [seg_pos={0 0}, upload=174963]: Done begin upload segment. Nov 7 18:12:36.063988689 (198h32m7.759837463s) [noise=true , address=145.79.227.27:4690 , piece_id=7CVNJTAEBVYJXRWNSFYFP3E7SKMY5XI4CV2JEZJLSXH4TA2F2JZQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.064083510 (198h32m7.759932263s) [noise=true , address=145.79.227.10:4403 , piece_id=JCMXNJ5PW5ULGOJZ5SGB423VW7VJPC5REYNSNWJZJXHGUQXUS3KA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.064896969 (198h32m7.760745762s) [noise=true , address=145.79.227.28:4726 , piece_id=D6DH5FHFB5ZTPANRPUWZKRYPFYUV2LS6AY5FQL5RBJZIUNM4FKLQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.065443520 (198h32m7.761292273s) [noise=true , address=77.90.31.11:4554 , piece_id=TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.065970974 (198h32m7.761819777s) [noise=true , address=145.79.227.8:4330 , piece_id=32XBCOEGSUTVUCCWVM2D7CB6FKMNLD6N2N4655B3ZYHNAIL4R2QQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.066422053 (198h32m7.762270806s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.067054427 (198h32m7.76290318s) [noise=true , address=77.90.31.10:4519 , piece_id=MIHLKGHCRTUFQSGBIFKIWPNACAJ4CQOEYRDIKA4KJLFP356VMY6A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.067645042 (198h32m7.763493795s) [noise=true , address=45.91.116.6:4973 , piece_id=Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.068379079 (198h32m7.764227842s) [noise=true , address=77.90.31.19:4843 , piece_id=JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.070931099 (198h32m7.766779862s) [noise=true , address=145.79.227.11:4732 , piece_id=VKSPYLTD6C6XBUFXXVVPHIPGFRCRNQK6EVI4SM7Y6ODOMVGGAJOQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.073100420 (198h32m7.768949173s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.075821872 (198h32m7.771670625s) [noise=true , address=45.91.116.4:4937 , piece_id=CWDA2IYNFUYHVDBJ74Y3CJKCZUBN4QYUEJYFKZNIBAUYYAXO3WFQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.078400863 (198h32m7.774249626s) [noise=true , address=45.91.116.10:4082 , piece_id=RW62GCPC56K2IMOFF276GQECXRKW6NY6W44TNAYFNDGQS3UK7EEA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.081230762 (198h32m7.777079515s) [noise=true , address=145.79.227.15:4876 , piece_id=ZPGKGNXRFEWRRJKTS7NKBEVQA7OAWNWYDR3DUTPHT3XVYRJSKWPQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.083298258 (198h32m7.779147012s) [noise=true , address=145.79.227.20:4438 , piece_id=V6WDUVDB7WHTDCTSYCFWS22UKRD5NMXGJ2GHLSFLVVOXZMZWLL4A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.084602091 (198h32m7.780450844s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:37.640211006 (198h32m9.336059759s) [upload=174963]: Next returned nil segment Nov 7 18:12:37.640227878 (198h32m9.336076621s) [upload=174963]: Waiting for error group managing segments... Nov 7 18:12:38.007509819 (198h32m9.703358572s) [noise=true , address=145.79.227.10:4403 , piece_id=JCMXNJ5PW5ULGOJZ5SGB423VW7VJPC5REYNSNWJZJXHGUQXUS3KA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.015182750 (198h32m9.711031503s) [noise=true , address=145.79.227.28:4726 , piece_id=D6DH5FHFB5ZTPANRPUWZKRYPFYUV2LS6AY5FQL5RBJZIUNM4FKLQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.020400886 (198h32m9.71624964s) [noise=true , address=145.79.227.27:4690 , piece_id=7CVNJTAEBVYJXRWNSFYFP3E7SKMY5XI4CV2JEZJLSXH4TA2F2JZQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.062361869 (198h32m9.758210622s) [noise=true , address=145.79.227.15:4876 , piece_id=ZPGKGNXRFEWRRJKTS7NKBEVQA7OAWNWYDR3DUTPHT3XVYRJSKWPQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.096654439 (198h32m9.792503192s) [noise=true , address=145.79.227.20:4438 , piece_id=V6WDUVDB7WHTDCTSYCFWS22UKRD5NMXGJ2GHLSFLVVOXZMZWLL4A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.244365682 (198h32m9.940214435s) [noise=true , address=145.79.227.8:4330 , piece_id=32XBCOEGSUTVUCCWVM2D7CB6FKMNLD6N2N4655B3ZYHNAIL4R2QQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.245114297 (198h32m9.94096305s) [noise=true , address=145.79.227.11:4732 , piece_id=VKSPYLTD6C6XBUFXXVVPHIPGFRCRNQK6EVI4SM7Y6ODOMVGGAJOQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.324356281 (198h32m10.020205034s) [noise=true , address=77.90.31.11:4554 , piece_id=TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.331696878 (198h32m10.027545621s) [noise=true , address=45.91.116.4:4937 , piece_id=CWDA2IYNFUYHVDBJ74Y3CJKCZUBN4QYUEJYFKZNIBAUYYAXO3WFQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.334142676 (198h32m10.029991429s) [noise=true , address=45.91.116.10:4082 , piece_id=RW62GCPC56K2IMOFF276GQECXRKW6NY6W44TNAYFNDGQS3UK7EEA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.346539031 (198h32m10.042387794s) [noise=true , address=77.90.31.10:4519 , piece_id=MIHLKGHCRTUFQSGBIFKIWPNACAJ4CQOEYRDIKA4KJLFP356VMY6A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.419504032 (198h32m10.115352785s) [noise=true , address=77.90.31.19:4843 , piece_id=JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:39.113150499 (198h32m10.808999252s) [noise=true , address=45.91.116.6:4973 , piece_id=Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:39.113177150 (198h32m10.809025903s) [seg_pos={0 0}, upload=174963]: Segment reached optimal threshold of 13 pieces. Nov 7 18:12:39.113381018 (198h32m10.809229772s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113390166 (198h32m10.809238919s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113460199 (198h32m10.809308952s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113470077 (198h32m10.809318821s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113780338 (198h32m10.809629091s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113788243 (198h32m10.809636986s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113858657 (198h32m10.8097074s) [seg_pos={0 0}, upload=174963]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 7 18:12:39.114030194 (198h32m10.809878947s) [upload=174963]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 7 18:12:39.114038290 (198h32m10.809887043s) [upload=174963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 7 18:12:39.451448640 (198h32m11.147297403s) [upload=174963]: Done uploading segments... Nov 7 18:12:39.451473607 (198h32m11.14732236s) [upload=174963]: Upload finished. err: <nil> Duration: 5.918107829s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......................................X.....................] duration: 3.487850864s trace: https://jaeger.dev.storj.io/trace/272952ae802c1916 FAIL (2025-11-07 18:57:11 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.487883477s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [X.X.........................................................] duration: 5.973920335s trace: https://jaeger.dev.storj.io/trace/422f33ac8ae6a74a FAIL (2025-11-07 18:57:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ) context deadline exceeded; piecestore: (Node ID: 12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, Piece ID: JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ) context deadline exceeded ======LOGS====== Nov 7 18:12:35.791697270 (198h32m7.487546023s) [upload=174963]: Starting upload Nov 7 18:12:35.791791689 (198h32m7.487640452s) [upload=174963]: Done starting upload Nov 7 18:12:35.791885458 (198h32m7.487734211s) [upload=174963]: Uploading segments... Nov 7 18:12:35.792146455 (198h32m7.487995208s) [upload=174963]: Got next segment. Inline: false Nov 7 18:12:35.792195278 (198h32m7.488044021s) [upload=174963]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 7 18:12:35.792201820 (198h32m7.488050563s) [upload=174963]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 7 18:12:36.063867588 (198h32m7.759716341s) [seg_pos={0 0}, upload=174963]: Begin upload segment... Nov 7 18:12:36.063937832 (198h32m7.759786585s) [seg_pos={0 0}, upload=174963]: Done begin upload segment. Nov 7 18:12:36.063988689 (198h32m7.759837463s) [noise=true , address=145.79.227.27:4690 , piece_id=7CVNJTAEBVYJXRWNSFYFP3E7SKMY5XI4CV2JEZJLSXH4TA2F2JZQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.064083510 (198h32m7.759932263s) [noise=true , address=145.79.227.10:4403 , piece_id=JCMXNJ5PW5ULGOJZ5SGB423VW7VJPC5REYNSNWJZJXHGUQXUS3KA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.064896969 (198h32m7.760745762s) [noise=true , address=145.79.227.28:4726 , piece_id=D6DH5FHFB5ZTPANRPUWZKRYPFYUV2LS6AY5FQL5RBJZIUNM4FKLQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.065443520 (198h32m7.761292273s) [noise=true , address=77.90.31.11:4554 , piece_id=TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.065970974 (198h32m7.761819777s) [noise=true , address=145.79.227.8:4330 , piece_id=32XBCOEGSUTVUCCWVM2D7CB6FKMNLD6N2N4655B3ZYHNAIL4R2QQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.066422053 (198h32m7.762270806s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.067054427 (198h32m7.76290318s) [noise=true , address=77.90.31.10:4519 , piece_id=MIHLKGHCRTUFQSGBIFKIWPNACAJ4CQOEYRDIKA4KJLFP356VMY6A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.067645042 (198h32m7.763493795s) [noise=true , address=45.91.116.6:4973 , piece_id=Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.068379079 (198h32m7.764227842s) [noise=true , address=77.90.31.19:4843 , piece_id=JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.070931099 (198h32m7.766779862s) [noise=true , address=145.79.227.11:4732 , piece_id=VKSPYLTD6C6XBUFXXVVPHIPGFRCRNQK6EVI4SM7Y6ODOMVGGAJOQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.073100420 (198h32m7.768949173s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.075821872 (198h32m7.771670625s) [noise=true , address=45.91.116.4:4937 , piece_id=CWDA2IYNFUYHVDBJ74Y3CJKCZUBN4QYUEJYFKZNIBAUYYAXO3WFQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.078400863 (198h32m7.774249626s) [noise=true , address=45.91.116.10:4082 , piece_id=RW62GCPC56K2IMOFF276GQECXRKW6NY6W44TNAYFNDGQS3UK7EEA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.081230762 (198h32m7.777079515s) [noise=true , address=145.79.227.15:4876 , piece_id=ZPGKGNXRFEWRRJKTS7NKBEVQA7OAWNWYDR3DUTPHT3XVYRJSKWPQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.083298258 (198h32m7.779147012s) [noise=true , address=145.79.227.20:4438 , piece_id=V6WDUVDB7WHTDCTSYCFWS22UKRD5NMXGJ2GHLSFLVVOXZMZWLL4A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.084602091 (198h32m7.780450844s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:37.640211006 (198h32m9.336059759s) [upload=174963]: Next returned nil segment Nov 7 18:12:37.640227878 (198h32m9.336076621s) [upload=174963]: Waiting for error group managing segments... Nov 7 18:12:38.007509819 (198h32m9.703358572s) [noise=true , address=145.79.227.10:4403 , piece_id=JCMXNJ5PW5ULGOJZ5SGB423VW7VJPC5REYNSNWJZJXHGUQXUS3KA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.015182750 (198h32m9.711031503s) [noise=true , address=145.79.227.28:4726 , piece_id=D6DH5FHFB5ZTPANRPUWZKRYPFYUV2LS6AY5FQL5RBJZIUNM4FKLQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.020400886 (198h32m9.71624964s) [noise=true , address=145.79.227.27:4690 , piece_id=7CVNJTAEBVYJXRWNSFYFP3E7SKMY5XI4CV2JEZJLSXH4TA2F2JZQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.062361869 (198h32m9.758210622s) [noise=true , address=145.79.227.15:4876 , piece_id=ZPGKGNXRFEWRRJKTS7NKBEVQA7OAWNWYDR3DUTPHT3XVYRJSKWPQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.096654439 (198h32m9.792503192s) [noise=true , address=145.79.227.20:4438 , piece_id=V6WDUVDB7WHTDCTSYCFWS22UKRD5NMXGJ2GHLSFLVVOXZMZWLL4A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.244365682 (198h32m9.940214435s) [noise=true , address=145.79.227.8:4330 , piece_id=32XBCOEGSUTVUCCWVM2D7CB6FKMNLD6N2N4655B3ZYHNAIL4R2QQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.245114297 (198h32m9.94096305s) [noise=true , address=145.79.227.11:4732 , piece_id=VKSPYLTD6C6XBUFXXVVPHIPGFRCRNQK6EVI4SM7Y6ODOMVGGAJOQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.324356281 (198h32m10.020205034s) [noise=true , address=77.90.31.11:4554 , piece_id=TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.331696878 (198h32m10.027545621s) [noise=true , address=45.91.116.4:4937 , piece_id=CWDA2IYNFUYHVDBJ74Y3CJKCZUBN4QYUEJYFKZNIBAUYYAXO3WFQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.334142676 (198h32m10.029991429s) [noise=true , address=45.91.116.10:4082 , piece_id=RW62GCPC56K2IMOFF276GQECXRKW6NY6W44TNAYFNDGQS3UK7EEA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.346539031 (198h32m10.042387794s) [noise=true , address=77.90.31.10:4519 , piece_id=MIHLKGHCRTUFQSGBIFKIWPNACAJ4CQOEYRDIKA4KJLFP356VMY6A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.419504032 (198h32m10.115352785s) [noise=true , address=77.90.31.19:4843 , piece_id=JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:39.113150499 (198h32m10.808999252s) [noise=true , address=45.91.116.6:4973 , piece_id=Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:39.113177150 (198h32m10.809025903s) [seg_pos={0 0}, upload=174963]: Segment reached optimal threshold of 13 pieces. Nov 7 18:12:39.113381018 (198h32m10.809229772s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113390166 (198h32m10.809238919s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113460199 (198h32m10.809308952s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113470077 (198h32m10.809318821s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113780338 (198h32m10.809629091s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113788243 (198h32m10.809636986s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113858657 (198h32m10.8097074s) [seg_pos={0 0}, upload=174963]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 7 18:12:39.114030194 (198h32m10.809878947s) [upload=174963]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 7 18:12:39.114038290 (198h32m10.809887043s) [upload=174963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 7 18:12:39.451448640 (198h32m11.147297403s) [upload=174963]: Done uploading segments... Nov 7 18:12:39.451473607 (198h32m11.14732236s) [upload=174963]: Upload finished. err: <nil> Duration: 5.973951606s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [XX.X........................................................] duration: 5.921243449s trace: https://jaeger.dev.storj.io/trace/422f33ac8ae6a74a FAIL (2025-11-07 18:56:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ) context deadline exceeded; piecestore: (Node ID: 12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, Piece ID: JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ) context deadline exceeded ======LOGS====== Nov 7 18:12:35.791697270 (198h32m7.487546023s) [upload=174963]: Starting upload Nov 7 18:12:35.791791689 (198h32m7.487640452s) [upload=174963]: Done starting upload Nov 7 18:12:35.791885458 (198h32m7.487734211s) [upload=174963]: Uploading segments... Nov 7 18:12:35.792146455 (198h32m7.487995208s) [upload=174963]: Got next segment. Inline: false Nov 7 18:12:35.792195278 (198h32m7.488044021s) [upload=174963]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 7 18:12:35.792201820 (198h32m7.488050563s) [upload=174963]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 7 18:12:36.063867588 (198h32m7.759716341s) [seg_pos={0 0}, upload=174963]: Begin upload segment... Nov 7 18:12:36.063937832 (198h32m7.759786585s) [seg_pos={0 0}, upload=174963]: Done begin upload segment. Nov 7 18:12:36.063988689 (198h32m7.759837463s) [noise=true , address=145.79.227.27:4690 , piece_id=7CVNJTAEBVYJXRWNSFYFP3E7SKMY5XI4CV2JEZJLSXH4TA2F2JZQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.064083510 (198h32m7.759932263s) [noise=true , address=145.79.227.10:4403 , piece_id=JCMXNJ5PW5ULGOJZ5SGB423VW7VJPC5REYNSNWJZJXHGUQXUS3KA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.064896969 (198h32m7.760745762s) [noise=true , address=145.79.227.28:4726 , piece_id=D6DH5FHFB5ZTPANRPUWZKRYPFYUV2LS6AY5FQL5RBJZIUNM4FKLQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.065443520 (198h32m7.761292273s) [noise=true , address=77.90.31.11:4554 , piece_id=TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.065970974 (198h32m7.761819777s) [noise=true , address=145.79.227.8:4330 , piece_id=32XBCOEGSUTVUCCWVM2D7CB6FKMNLD6N2N4655B3ZYHNAIL4R2QQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.066422053 (198h32m7.762270806s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.067054427 (198h32m7.76290318s) [noise=true , address=77.90.31.10:4519 , piece_id=MIHLKGHCRTUFQSGBIFKIWPNACAJ4CQOEYRDIKA4KJLFP356VMY6A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.067645042 (198h32m7.763493795s) [noise=true , address=45.91.116.6:4973 , piece_id=Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.068379079 (198h32m7.764227842s) [noise=true , address=77.90.31.19:4843 , piece_id=JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.070931099 (198h32m7.766779862s) [noise=true , address=145.79.227.11:4732 , piece_id=VKSPYLTD6C6XBUFXXVVPHIPGFRCRNQK6EVI4SM7Y6ODOMVGGAJOQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.073100420 (198h32m7.768949173s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.075821872 (198h32m7.771670625s) [noise=true , address=45.91.116.4:4937 , piece_id=CWDA2IYNFUYHVDBJ74Y3CJKCZUBN4QYUEJYFKZNIBAUYYAXO3WFQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.078400863 (198h32m7.774249626s) [noise=true , address=45.91.116.10:4082 , piece_id=RW62GCPC56K2IMOFF276GQECXRKW6NY6W44TNAYFNDGQS3UK7EEA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.081230762 (198h32m7.777079515s) [noise=true , address=145.79.227.15:4876 , piece_id=ZPGKGNXRFEWRRJKTS7NKBEVQA7OAWNWYDR3DUTPHT3XVYRJSKWPQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.083298258 (198h32m7.779147012s) [noise=true , address=145.79.227.20:4438 , piece_id=V6WDUVDB7WHTDCTSYCFWS22UKRD5NMXGJ2GHLSFLVVOXZMZWLL4A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.084602091 (198h32m7.780450844s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:37.640211006 (198h32m9.336059759s) [upload=174963]: Next returned nil segment Nov 7 18:12:37.640227878 (198h32m9.336076621s) [upload=174963]: Waiting for error group managing segments... Nov 7 18:12:38.007509819 (198h32m9.703358572s) [noise=true , address=145.79.227.10:4403 , piece_id=JCMXNJ5PW5ULGOJZ5SGB423VW7VJPC5REYNSNWJZJXHGUQXUS3KA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.015182750 (198h32m9.711031503s) [noise=true , address=145.79.227.28:4726 , piece_id=D6DH5FHFB5ZTPANRPUWZKRYPFYUV2LS6AY5FQL5RBJZIUNM4FKLQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.020400886 (198h32m9.71624964s) [noise=true , address=145.79.227.27:4690 , piece_id=7CVNJTAEBVYJXRWNSFYFP3E7SKMY5XI4CV2JEZJLSXH4TA2F2JZQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.062361869 (198h32m9.758210622s) [noise=true , address=145.79.227.15:4876 , piece_id=ZPGKGNXRFEWRRJKTS7NKBEVQA7OAWNWYDR3DUTPHT3XVYRJSKWPQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.096654439 (198h32m9.792503192s) [noise=true , address=145.79.227.20:4438 , piece_id=V6WDUVDB7WHTDCTSYCFWS22UKRD5NMXGJ2GHLSFLVVOXZMZWLL4A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.244365682 (198h32m9.940214435s) [noise=true , address=145.79.227.8:4330 , piece_id=32XBCOEGSUTVUCCWVM2D7CB6FKMNLD6N2N4655B3ZYHNAIL4R2QQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.245114297 (198h32m9.94096305s) [noise=true , address=145.79.227.11:4732 , piece_id=VKSPYLTD6C6XBUFXXVVPHIPGFRCRNQK6EVI4SM7Y6ODOMVGGAJOQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.324356281 (198h32m10.020205034s) [noise=true , address=77.90.31.11:4554 , piece_id=TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.331696878 (198h32m10.027545621s) [noise=true , address=45.91.116.4:4937 , piece_id=CWDA2IYNFUYHVDBJ74Y3CJKCZUBN4QYUEJYFKZNIBAUYYAXO3WFQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.334142676 (198h32m10.029991429s) [noise=true , address=45.91.116.10:4082 , piece_id=RW62GCPC56K2IMOFF276GQECXRKW6NY6W44TNAYFNDGQS3UK7EEA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.346539031 (198h32m10.042387794s) [noise=true , address=77.90.31.10:4519 , piece_id=MIHLKGHCRTUFQSGBIFKIWPNACAJ4CQOEYRDIKA4KJLFP356VMY6A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.419504032 (198h32m10.115352785s) [noise=true , address=77.90.31.19:4843 , piece_id=JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:39.113150499 (198h32m10.808999252s) [noise=true , address=45.91.116.6:4973 , piece_id=Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:39.113177150 (198h32m10.809025903s) [seg_pos={0 0}, upload=174963]: Segment reached optimal threshold of 13 pieces. Nov 7 18:12:39.113381018 (198h32m10.809229772s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113390166 (198h32m10.809238919s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113460199 (198h32m10.809308952s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113470077 (198h32m10.809318821s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113780338 (198h32m10.809629091s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113788243 (198h32m10.809636986s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113858657 (198h32m10.8097074s) [seg_pos={0 0}, upload=174963]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 7 18:12:39.114030194 (198h32m10.809878947s) [upload=174963]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 7 18:12:39.114038290 (198h32m10.809887043s) [upload=174963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 7 18:12:39.451448640 (198h32m11.147297403s) [upload=174963]: Done uploading segments... Nov 7 18:12:39.451473607 (198h32m11.14732236s) [upload=174963]: Upload finished. err: <nil> Duration: 5.921273778s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......................................X....................] duration: 3.450150901s trace: https://jaeger.dev.storj.io/trace/272952ae802c1916 FAIL (2025-11-07 18:56:22 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.450262244s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [XXX.X.......................................................] duration: 6.039307632s trace: https://jaeger.dev.storj.io/trace/422f33ac8ae6a74a FAIL (2025-11-07 18:55:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ) context deadline exceeded; piecestore: (Node ID: 12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, Piece ID: JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ) context deadline exceeded ======LOGS====== Nov 7 18:12:35.791697270 (198h32m7.487546023s) [upload=174963]: Starting upload Nov 7 18:12:35.791791689 (198h32m7.487640452s) [upload=174963]: Done starting upload Nov 7 18:12:35.791885458 (198h32m7.487734211s) [upload=174963]: Uploading segments... Nov 7 18:12:35.792146455 (198h32m7.487995208s) [upload=174963]: Got next segment. Inline: false Nov 7 18:12:35.792195278 (198h32m7.488044021s) [upload=174963]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 7 18:12:35.792201820 (198h32m7.488050563s) [upload=174963]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 7 18:12:36.063867588 (198h32m7.759716341s) [seg_pos={0 0}, upload=174963]: Begin upload segment... Nov 7 18:12:36.063937832 (198h32m7.759786585s) [seg_pos={0 0}, upload=174963]: Done begin upload segment. Nov 7 18:12:36.063988689 (198h32m7.759837463s) [noise=true , address=145.79.227.27:4690 , piece_id=7CVNJTAEBVYJXRWNSFYFP3E7SKMY5XI4CV2JEZJLSXH4TA2F2JZQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.064083510 (198h32m7.759932263s) [noise=true , address=145.79.227.10:4403 , piece_id=JCMXNJ5PW5ULGOJZ5SGB423VW7VJPC5REYNSNWJZJXHGUQXUS3KA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.064896969 (198h32m7.760745762s) [noise=true , address=145.79.227.28:4726 , piece_id=D6DH5FHFB5ZTPANRPUWZKRYPFYUV2LS6AY5FQL5RBJZIUNM4FKLQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.065443520 (198h32m7.761292273s) [noise=true , address=77.90.31.11:4554 , piece_id=TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.065970974 (198h32m7.761819777s) [noise=true , address=145.79.227.8:4330 , piece_id=32XBCOEGSUTVUCCWVM2D7CB6FKMNLD6N2N4655B3ZYHNAIL4R2QQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.066422053 (198h32m7.762270806s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.067054427 (198h32m7.76290318s) [noise=true , address=77.90.31.10:4519 , piece_id=MIHLKGHCRTUFQSGBIFKIWPNACAJ4CQOEYRDIKA4KJLFP356VMY6A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.067645042 (198h32m7.763493795s) [noise=true , address=45.91.116.6:4973 , piece_id=Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.068379079 (198h32m7.764227842s) [noise=true , address=77.90.31.19:4843 , piece_id=JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.070931099 (198h32m7.766779862s) [noise=true , address=145.79.227.11:4732 , piece_id=VKSPYLTD6C6XBUFXXVVPHIPGFRCRNQK6EVI4SM7Y6ODOMVGGAJOQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.073100420 (198h32m7.768949173s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.075821872 (198h32m7.771670625s) [noise=true , address=45.91.116.4:4937 , piece_id=CWDA2IYNFUYHVDBJ74Y3CJKCZUBN4QYUEJYFKZNIBAUYYAXO3WFQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.078400863 (198h32m7.774249626s) [noise=true , address=45.91.116.10:4082 , piece_id=RW62GCPC56K2IMOFF276GQECXRKW6NY6W44TNAYFNDGQS3UK7EEA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.081230762 (198h32m7.777079515s) [noise=true , address=145.79.227.15:4876 , piece_id=ZPGKGNXRFEWRRJKTS7NKBEVQA7OAWNWYDR3DUTPHT3XVYRJSKWPQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.083298258 (198h32m7.779147012s) [noise=true , address=145.79.227.20:4438 , piece_id=V6WDUVDB7WHTDCTSYCFWS22UKRD5NMXGJ2GHLSFLVVOXZMZWLL4A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.084602091 (198h32m7.780450844s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:37.640211006 (198h32m9.336059759s) [upload=174963]: Next returned nil segment Nov 7 18:12:37.640227878 (198h32m9.336076621s) [upload=174963]: Waiting for error group managing segments... Nov 7 18:12:38.007509819 (198h32m9.703358572s) [noise=true , address=145.79.227.10:4403 , piece_id=JCMXNJ5PW5ULGOJZ5SGB423VW7VJPC5REYNSNWJZJXHGUQXUS3KA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.015182750 (198h32m9.711031503s) [noise=true , address=145.79.227.28:4726 , piece_id=D6DH5FHFB5ZTPANRPUWZKRYPFYUV2LS6AY5FQL5RBJZIUNM4FKLQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.020400886 (198h32m9.71624964s) [noise=true , address=145.79.227.27:4690 , piece_id=7CVNJTAEBVYJXRWNSFYFP3E7SKMY5XI4CV2JEZJLSXH4TA2F2JZQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.062361869 (198h32m9.758210622s) [noise=true , address=145.79.227.15:4876 , piece_id=ZPGKGNXRFEWRRJKTS7NKBEVQA7OAWNWYDR3DUTPHT3XVYRJSKWPQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.096654439 (198h32m9.792503192s) [noise=true , address=145.79.227.20:4438 , piece_id=V6WDUVDB7WHTDCTSYCFWS22UKRD5NMXGJ2GHLSFLVVOXZMZWLL4A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.244365682 (198h32m9.940214435s) [noise=true , address=145.79.227.8:4330 , piece_id=32XBCOEGSUTVUCCWVM2D7CB6FKMNLD6N2N4655B3ZYHNAIL4R2QQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.245114297 (198h32m9.94096305s) [noise=true , address=145.79.227.11:4732 , piece_id=VKSPYLTD6C6XBUFXXVVPHIPGFRCRNQK6EVI4SM7Y6ODOMVGGAJOQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.324356281 (198h32m10.020205034s) [noise=true , address=77.90.31.11:4554 , piece_id=TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.331696878 (198h32m10.027545621s) [noise=true , address=45.91.116.4:4937 , piece_id=CWDA2IYNFUYHVDBJ74Y3CJKCZUBN4QYUEJYFKZNIBAUYYAXO3WFQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.334142676 (198h32m10.029991429s) [noise=true , address=45.91.116.10:4082 , piece_id=RW62GCPC56K2IMOFF276GQECXRKW6NY6W44TNAYFNDGQS3UK7EEA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.346539031 (198h32m10.042387794s) [noise=true , address=77.90.31.10:4519 , piece_id=MIHLKGHCRTUFQSGBIFKIWPNACAJ4CQOEYRDIKA4KJLFP356VMY6A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.419504032 (198h32m10.115352785s) [noise=true , address=77.90.31.19:4843 , piece_id=JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:39.113150499 (198h32m10.808999252s) [noise=true , address=45.91.116.6:4973 , piece_id=Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:39.113177150 (198h32m10.809025903s) [seg_pos={0 0}, upload=174963]: Segment reached optimal threshold of 13 pieces. Nov 7 18:12:39.113381018 (198h32m10.809229772s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113390166 (198h32m10.809238919s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113460199 (198h32m10.809308952s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113470077 (198h32m10.809318821s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113780338 (198h32m10.809629091s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113788243 (198h32m10.809636986s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113858657 (198h32m10.8097074s) [seg_pos={0 0}, upload=174963]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 7 18:12:39.114030194 (198h32m10.809878947s) [upload=174963]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 7 18:12:39.114038290 (198h32m10.809887043s) [upload=174963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 7 18:12:39.451448640 (198h32m11.147297403s) [upload=174963]: Done uploading segments... Nov 7 18:12:39.451473607 (198h32m11.14732236s) [upload=174963]: Upload finished. err: <nil> Duration: 6.039325946s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........................................X...................] duration: 3.349928406s trace: https://jaeger.dev.storj.io/trace/272952ae802c1916 FAIL (2025-11-07 18:55:27 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.350192699s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [XXXX.X......................................................] duration: 6.204858928s trace: https://jaeger.dev.storj.io/trace/422f33ac8ae6a74a FAIL (2025-11-07 18:55:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ) context deadline exceeded; piecestore: (Node ID: 12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, Piece ID: JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ) context deadline exceeded ======LOGS====== Nov 7 18:12:35.791697270 (198h32m7.487546023s) [upload=174963]: Starting upload Nov 7 18:12:35.791791689 (198h32m7.487640452s) [upload=174963]: Done starting upload Nov 7 18:12:35.791885458 (198h32m7.487734211s) [upload=174963]: Uploading segments... Nov 7 18:12:35.792146455 (198h32m7.487995208s) [upload=174963]: Got next segment. Inline: false Nov 7 18:12:35.792195278 (198h32m7.488044021s) [upload=174963]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 7 18:12:35.792201820 (198h32m7.488050563s) [upload=174963]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 7 18:12:36.063867588 (198h32m7.759716341s) [seg_pos={0 0}, upload=174963]: Begin upload segment... Nov 7 18:12:36.063937832 (198h32m7.759786585s) [seg_pos={0 0}, upload=174963]: Done begin upload segment. Nov 7 18:12:36.063988689 (198h32m7.759837463s) [noise=true , address=145.79.227.27:4690 , piece_id=7CVNJTAEBVYJXRWNSFYFP3E7SKMY5XI4CV2JEZJLSXH4TA2F2JZQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.064083510 (198h32m7.759932263s) [noise=true , address=145.79.227.10:4403 , piece_id=JCMXNJ5PW5ULGOJZ5SGB423VW7VJPC5REYNSNWJZJXHGUQXUS3KA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.064896969 (198h32m7.760745762s) [noise=true , address=145.79.227.28:4726 , piece_id=D6DH5FHFB5ZTPANRPUWZKRYPFYUV2LS6AY5FQL5RBJZIUNM4FKLQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.065443520 (198h32m7.761292273s) [noise=true , address=77.90.31.11:4554 , piece_id=TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.065970974 (198h32m7.761819777s) [noise=true , address=145.79.227.8:4330 , piece_id=32XBCOEGSUTVUCCWVM2D7CB6FKMNLD6N2N4655B3ZYHNAIL4R2QQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.066422053 (198h32m7.762270806s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.067054427 (198h32m7.76290318s) [noise=true , address=77.90.31.10:4519 , piece_id=MIHLKGHCRTUFQSGBIFKIWPNACAJ4CQOEYRDIKA4KJLFP356VMY6A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.067645042 (198h32m7.763493795s) [noise=true , address=45.91.116.6:4973 , piece_id=Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.068379079 (198h32m7.764227842s) [noise=true , address=77.90.31.19:4843 , piece_id=JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.070931099 (198h32m7.766779862s) [noise=true , address=145.79.227.11:4732 , piece_id=VKSPYLTD6C6XBUFXXVVPHIPGFRCRNQK6EVI4SM7Y6ODOMVGGAJOQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.073100420 (198h32m7.768949173s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.075821872 (198h32m7.771670625s) [noise=true , address=45.91.116.4:4937 , piece_id=CWDA2IYNFUYHVDBJ74Y3CJKCZUBN4QYUEJYFKZNIBAUYYAXO3WFQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.078400863 (198h32m7.774249626s) [noise=true , address=45.91.116.10:4082 , piece_id=RW62GCPC56K2IMOFF276GQECXRKW6NY6W44TNAYFNDGQS3UK7EEA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.081230762 (198h32m7.777079515s) [noise=true , address=145.79.227.15:4876 , piece_id=ZPGKGNXRFEWRRJKTS7NKBEVQA7OAWNWYDR3DUTPHT3XVYRJSKWPQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.083298258 (198h32m7.779147012s) [noise=true , address=145.79.227.20:4438 , piece_id=V6WDUVDB7WHTDCTSYCFWS22UKRD5NMXGJ2GHLSFLVVOXZMZWLL4A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.084602091 (198h32m7.780450844s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:37.640211006 (198h32m9.336059759s) [upload=174963]: Next returned nil segment Nov 7 18:12:37.640227878 (198h32m9.336076621s) [upload=174963]: Waiting for error group managing segments... Nov 7 18:12:38.007509819 (198h32m9.703358572s) [noise=true , address=145.79.227.10:4403 , piece_id=JCMXNJ5PW5ULGOJZ5SGB423VW7VJPC5REYNSNWJZJXHGUQXUS3KA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.015182750 (198h32m9.711031503s) [noise=true , address=145.79.227.28:4726 , piece_id=D6DH5FHFB5ZTPANRPUWZKRYPFYUV2LS6AY5FQL5RBJZIUNM4FKLQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.020400886 (198h32m9.71624964s) [noise=true , address=145.79.227.27:4690 , piece_id=7CVNJTAEBVYJXRWNSFYFP3E7SKMY5XI4CV2JEZJLSXH4TA2F2JZQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.062361869 (198h32m9.758210622s) [noise=true , address=145.79.227.15:4876 , piece_id=ZPGKGNXRFEWRRJKTS7NKBEVQA7OAWNWYDR3DUTPHT3XVYRJSKWPQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.096654439 (198h32m9.792503192s) [noise=true , address=145.79.227.20:4438 , piece_id=V6WDUVDB7WHTDCTSYCFWS22UKRD5NMXGJ2GHLSFLVVOXZMZWLL4A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.244365682 (198h32m9.940214435s) [noise=true , address=145.79.227.8:4330 , piece_id=32XBCOEGSUTVUCCWVM2D7CB6FKMNLD6N2N4655B3ZYHNAIL4R2QQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.245114297 (198h32m9.94096305s) [noise=true , address=145.79.227.11:4732 , piece_id=VKSPYLTD6C6XBUFXXVVPHIPGFRCRNQK6EVI4SM7Y6ODOMVGGAJOQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.324356281 (198h32m10.020205034s) [noise=true , address=77.90.31.11:4554 , piece_id=TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.331696878 (198h32m10.027545621s) [noise=true , address=45.91.116.4:4937 , piece_id=CWDA2IYNFUYHVDBJ74Y3CJKCZUBN4QYUEJYFKZNIBAUYYAXO3WFQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.334142676 (198h32m10.029991429s) [noise=true , address=45.91.116.10:4082 , piece_id=RW62GCPC56K2IMOFF276GQECXRKW6NY6W44TNAYFNDGQS3UK7EEA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.346539031 (198h32m10.042387794s) [noise=true , address=77.90.31.10:4519 , piece_id=MIHLKGHCRTUFQSGBIFKIWPNACAJ4CQOEYRDIKA4KJLFP356VMY6A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.419504032 (198h32m10.115352785s) [noise=true , address=77.90.31.19:4843 , piece_id=JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:39.113150499 (198h32m10.808999252s) [noise=true , address=45.91.116.6:4973 , piece_id=Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:39.113177150 (198h32m10.809025903s) [seg_pos={0 0}, upload=174963]: Segment reached optimal threshold of 13 pieces. Nov 7 18:12:39.113381018 (198h32m10.809229772s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113390166 (198h32m10.809238919s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113460199 (198h32m10.809308952s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113470077 (198h32m10.809318821s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113780338 (198h32m10.809629091s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113788243 (198h32m10.809636986s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113858657 (198h32m10.8097074s) [seg_pos={0 0}, upload=174963]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 7 18:12:39.114030194 (198h32m10.809878947s) [upload=174963]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 7 18:12:39.114038290 (198h32m10.809887043s) [upload=174963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 7 18:12:39.451448640 (198h32m11.147297403s) [upload=174963]: Done uploading segments... Nov 7 18:12:39.451473607 (198h32m11.14732236s) [upload=174963]: Upload finished. err: <nil> Duration: 6.204898583s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........................................X..................] duration: 3.573295917s trace: https://jaeger.dev.storj.io/trace/272952ae802c1916 FAIL (2025-11-07 18:55:02 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.573317517s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........................................X.................] duration: 3.423983854s trace: https://jaeger.dev.storj.io/trace/272952ae802c1916 FAIL (2025-11-07 18:54:31 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.42401385s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [XXXXX.X.....................................................] duration: 5.989234019s trace: https://jaeger.dev.storj.io/trace/422f33ac8ae6a74a FAIL (2025-11-07 18:53:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ) context deadline exceeded; piecestore: (Node ID: 12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, Piece ID: JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ) context deadline exceeded ======LOGS====== Nov 7 18:12:35.791697270 (198h32m7.487546023s) [upload=174963]: Starting upload Nov 7 18:12:35.791791689 (198h32m7.487640452s) [upload=174963]: Done starting upload Nov 7 18:12:35.791885458 (198h32m7.487734211s) [upload=174963]: Uploading segments... Nov 7 18:12:35.792146455 (198h32m7.487995208s) [upload=174963]: Got next segment. Inline: false Nov 7 18:12:35.792195278 (198h32m7.488044021s) [upload=174963]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 7 18:12:35.792201820 (198h32m7.488050563s) [upload=174963]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 7 18:12:36.063867588 (198h32m7.759716341s) [seg_pos={0 0}, upload=174963]: Begin upload segment... Nov 7 18:12:36.063937832 (198h32m7.759786585s) [seg_pos={0 0}, upload=174963]: Done begin upload segment. Nov 7 18:12:36.063988689 (198h32m7.759837463s) [noise=true , address=145.79.227.27:4690 , piece_id=7CVNJTAEBVYJXRWNSFYFP3E7SKMY5XI4CV2JEZJLSXH4TA2F2JZQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.064083510 (198h32m7.759932263s) [noise=true , address=145.79.227.10:4403 , piece_id=JCMXNJ5PW5ULGOJZ5SGB423VW7VJPC5REYNSNWJZJXHGUQXUS3KA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.064896969 (198h32m7.760745762s) [noise=true , address=145.79.227.28:4726 , piece_id=D6DH5FHFB5ZTPANRPUWZKRYPFYUV2LS6AY5FQL5RBJZIUNM4FKLQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.065443520 (198h32m7.761292273s) [noise=true , address=77.90.31.11:4554 , piece_id=TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.065970974 (198h32m7.761819777s) [noise=true , address=145.79.227.8:4330 , piece_id=32XBCOEGSUTVUCCWVM2D7CB6FKMNLD6N2N4655B3ZYHNAIL4R2QQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.066422053 (198h32m7.762270806s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.067054427 (198h32m7.76290318s) [noise=true , address=77.90.31.10:4519 , piece_id=MIHLKGHCRTUFQSGBIFKIWPNACAJ4CQOEYRDIKA4KJLFP356VMY6A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.067645042 (198h32m7.763493795s) [noise=true , address=45.91.116.6:4973 , piece_id=Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.068379079 (198h32m7.764227842s) [noise=true , address=77.90.31.19:4843 , piece_id=JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.070931099 (198h32m7.766779862s) [noise=true , address=145.79.227.11:4732 , piece_id=VKSPYLTD6C6XBUFXXVVPHIPGFRCRNQK6EVI4SM7Y6ODOMVGGAJOQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.073100420 (198h32m7.768949173s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.075821872 (198h32m7.771670625s) [noise=true , address=45.91.116.4:4937 , piece_id=CWDA2IYNFUYHVDBJ74Y3CJKCZUBN4QYUEJYFKZNIBAUYYAXO3WFQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.078400863 (198h32m7.774249626s) [noise=true , address=45.91.116.10:4082 , piece_id=RW62GCPC56K2IMOFF276GQECXRKW6NY6W44TNAYFNDGQS3UK7EEA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.081230762 (198h32m7.777079515s) [noise=true , address=145.79.227.15:4876 , piece_id=ZPGKGNXRFEWRRJKTS7NKBEVQA7OAWNWYDR3DUTPHT3XVYRJSKWPQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.083298258 (198h32m7.779147012s) [noise=true , address=145.79.227.20:4438 , piece_id=V6WDUVDB7WHTDCTSYCFWS22UKRD5NMXGJ2GHLSFLVVOXZMZWLL4A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.084602091 (198h32m7.780450844s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:37.640211006 (198h32m9.336059759s) [upload=174963]: Next returned nil segment Nov 7 18:12:37.640227878 (198h32m9.336076621s) [upload=174963]: Waiting for error group managing segments... Nov 7 18:12:38.007509819 (198h32m9.703358572s) [noise=true , address=145.79.227.10:4403 , piece_id=JCMXNJ5PW5ULGOJZ5SGB423VW7VJPC5REYNSNWJZJXHGUQXUS3KA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.015182750 (198h32m9.711031503s) [noise=true , address=145.79.227.28:4726 , piece_id=D6DH5FHFB5ZTPANRPUWZKRYPFYUV2LS6AY5FQL5RBJZIUNM4FKLQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.020400886 (198h32m9.71624964s) [noise=true , address=145.79.227.27:4690 , piece_id=7CVNJTAEBVYJXRWNSFYFP3E7SKMY5XI4CV2JEZJLSXH4TA2F2JZQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.062361869 (198h32m9.758210622s) [noise=true , address=145.79.227.15:4876 , piece_id=ZPGKGNXRFEWRRJKTS7NKBEVQA7OAWNWYDR3DUTPHT3XVYRJSKWPQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.096654439 (198h32m9.792503192s) [noise=true , address=145.79.227.20:4438 , piece_id=V6WDUVDB7WHTDCTSYCFWS22UKRD5NMXGJ2GHLSFLVVOXZMZWLL4A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.244365682 (198h32m9.940214435s) [noise=true , address=145.79.227.8:4330 , piece_id=32XBCOEGSUTVUCCWVM2D7CB6FKMNLD6N2N4655B3ZYHNAIL4R2QQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.245114297 (198h32m9.94096305s) [noise=true , address=145.79.227.11:4732 , piece_id=VKSPYLTD6C6XBUFXXVVPHIPGFRCRNQK6EVI4SM7Y6ODOMVGGAJOQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.324356281 (198h32m10.020205034s) [noise=true , address=77.90.31.11:4554 , piece_id=TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.331696878 (198h32m10.027545621s) [noise=true , address=45.91.116.4:4937 , piece_id=CWDA2IYNFUYHVDBJ74Y3CJKCZUBN4QYUEJYFKZNIBAUYYAXO3WFQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.334142676 (198h32m10.029991429s) [noise=true , address=45.91.116.10:4082 , piece_id=RW62GCPC56K2IMOFF276GQECXRKW6NY6W44TNAYFNDGQS3UK7EEA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.346539031 (198h32m10.042387794s) [noise=true , address=77.90.31.10:4519 , piece_id=MIHLKGHCRTUFQSGBIFKIWPNACAJ4CQOEYRDIKA4KJLFP356VMY6A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.419504032 (198h32m10.115352785s) [noise=true , address=77.90.31.19:4843 , piece_id=JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:39.113150499 (198h32m10.808999252s) [noise=true , address=45.91.116.6:4973 , piece_id=Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:39.113177150 (198h32m10.809025903s) [seg_pos={0 0}, upload=174963]: Segment reached optimal threshold of 13 pieces. Nov 7 18:12:39.113381018 (198h32m10.809229772s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113390166 (198h32m10.809238919s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113460199 (198h32m10.809308952s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113470077 (198h32m10.809318821s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113780338 (198h32m10.809629091s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113788243 (198h32m10.809636986s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113858657 (198h32m10.8097074s) [seg_pos={0 0}, upload=174963]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 7 18:12:39.114030194 (198h32m10.809878947s) [upload=174963]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 7 18:12:39.114038290 (198h32m10.809887043s) [upload=174963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 7 18:12:39.451448640 (198h32m11.147297403s) [upload=174963]: Done uploading segments... Nov 7 18:12:39.451473607 (198h32m11.14732236s) [upload=174963]: Upload finished. err: <nil> Duration: 5.98926531s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........................................X................] duration: 3.516568733s trace: https://jaeger.dev.storj.io/trace/272952ae802c1916 FAIL (2025-11-07 18:53:26 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.516631051s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.XXXXX.X....................................................] duration: 6.088994524s trace: https://jaeger.dev.storj.io/trace/422f33ac8ae6a74a FAIL (2025-11-07 18:53:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ) context deadline exceeded; piecestore: (Node ID: 12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, Piece ID: JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ) context deadline exceeded ======LOGS====== Nov 7 18:12:35.791697270 (198h32m7.487546023s) [upload=174963]: Starting upload Nov 7 18:12:35.791791689 (198h32m7.487640452s) [upload=174963]: Done starting upload Nov 7 18:12:35.791885458 (198h32m7.487734211s) [upload=174963]: Uploading segments... Nov 7 18:12:35.792146455 (198h32m7.487995208s) [upload=174963]: Got next segment. Inline: false Nov 7 18:12:35.792195278 (198h32m7.488044021s) [upload=174963]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 7 18:12:35.792201820 (198h32m7.488050563s) [upload=174963]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 7 18:12:36.063867588 (198h32m7.759716341s) [seg_pos={0 0}, upload=174963]: Begin upload segment... Nov 7 18:12:36.063937832 (198h32m7.759786585s) [seg_pos={0 0}, upload=174963]: Done begin upload segment. Nov 7 18:12:36.063988689 (198h32m7.759837463s) [noise=true , address=145.79.227.27:4690 , piece_id=7CVNJTAEBVYJXRWNSFYFP3E7SKMY5XI4CV2JEZJLSXH4TA2F2JZQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.064083510 (198h32m7.759932263s) [noise=true , address=145.79.227.10:4403 , piece_id=JCMXNJ5PW5ULGOJZ5SGB423VW7VJPC5REYNSNWJZJXHGUQXUS3KA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.064896969 (198h32m7.760745762s) [noise=true , address=145.79.227.28:4726 , piece_id=D6DH5FHFB5ZTPANRPUWZKRYPFYUV2LS6AY5FQL5RBJZIUNM4FKLQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.065443520 (198h32m7.761292273s) [noise=true , address=77.90.31.11:4554 , piece_id=TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.065970974 (198h32m7.761819777s) [noise=true , address=145.79.227.8:4330 , piece_id=32XBCOEGSUTVUCCWVM2D7CB6FKMNLD6N2N4655B3ZYHNAIL4R2QQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.066422053 (198h32m7.762270806s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.067054427 (198h32m7.76290318s) [noise=true , address=77.90.31.10:4519 , piece_id=MIHLKGHCRTUFQSGBIFKIWPNACAJ4CQOEYRDIKA4KJLFP356VMY6A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.067645042 (198h32m7.763493795s) [noise=true , address=45.91.116.6:4973 , piece_id=Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.068379079 (198h32m7.764227842s) [noise=true , address=77.90.31.19:4843 , piece_id=JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.070931099 (198h32m7.766779862s) [noise=true , address=145.79.227.11:4732 , piece_id=VKSPYLTD6C6XBUFXXVVPHIPGFRCRNQK6EVI4SM7Y6ODOMVGGAJOQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.073100420 (198h32m7.768949173s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.075821872 (198h32m7.771670625s) [noise=true , address=45.91.116.4:4937 , piece_id=CWDA2IYNFUYHVDBJ74Y3CJKCZUBN4QYUEJYFKZNIBAUYYAXO3WFQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.078400863 (198h32m7.774249626s) [noise=true , address=45.91.116.10:4082 , piece_id=RW62GCPC56K2IMOFF276GQECXRKW6NY6W44TNAYFNDGQS3UK7EEA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.081230762 (198h32m7.777079515s) [noise=true , address=145.79.227.15:4876 , piece_id=ZPGKGNXRFEWRRJKTS7NKBEVQA7OAWNWYDR3DUTPHT3XVYRJSKWPQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.083298258 (198h32m7.779147012s) [noise=true , address=145.79.227.20:4438 , piece_id=V6WDUVDB7WHTDCTSYCFWS22UKRD5NMXGJ2GHLSFLVVOXZMZWLL4A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.084602091 (198h32m7.780450844s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:37.640211006 (198h32m9.336059759s) [upload=174963]: Next returned nil segment Nov 7 18:12:37.640227878 (198h32m9.336076621s) [upload=174963]: Waiting for error group managing segments... Nov 7 18:12:38.007509819 (198h32m9.703358572s) [noise=true , address=145.79.227.10:4403 , piece_id=JCMXNJ5PW5ULGOJZ5SGB423VW7VJPC5REYNSNWJZJXHGUQXUS3KA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.015182750 (198h32m9.711031503s) [noise=true , address=145.79.227.28:4726 , piece_id=D6DH5FHFB5ZTPANRPUWZKRYPFYUV2LS6AY5FQL5RBJZIUNM4FKLQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.020400886 (198h32m9.71624964s) [noise=true , address=145.79.227.27:4690 , piece_id=7CVNJTAEBVYJXRWNSFYFP3E7SKMY5XI4CV2JEZJLSXH4TA2F2JZQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.062361869 (198h32m9.758210622s) [noise=true , address=145.79.227.15:4876 , piece_id=ZPGKGNXRFEWRRJKTS7NKBEVQA7OAWNWYDR3DUTPHT3XVYRJSKWPQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.096654439 (198h32m9.792503192s) [noise=true , address=145.79.227.20:4438 , piece_id=V6WDUVDB7WHTDCTSYCFWS22UKRD5NMXGJ2GHLSFLVVOXZMZWLL4A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.244365682 (198h32m9.940214435s) [noise=true , address=145.79.227.8:4330 , piece_id=32XBCOEGSUTVUCCWVM2D7CB6FKMNLD6N2N4655B3ZYHNAIL4R2QQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.245114297 (198h32m9.94096305s) [noise=true , address=145.79.227.11:4732 , piece_id=VKSPYLTD6C6XBUFXXVVPHIPGFRCRNQK6EVI4SM7Y6ODOMVGGAJOQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.324356281 (198h32m10.020205034s) [noise=true , address=77.90.31.11:4554 , piece_id=TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.331696878 (198h32m10.027545621s) [noise=true , address=45.91.116.4:4937 , piece_id=CWDA2IYNFUYHVDBJ74Y3CJKCZUBN4QYUEJYFKZNIBAUYYAXO3WFQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.334142676 (198h32m10.029991429s) [noise=true , address=45.91.116.10:4082 , piece_id=RW62GCPC56K2IMOFF276GQECXRKW6NY6W44TNAYFNDGQS3UK7EEA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.346539031 (198h32m10.042387794s) [noise=true , address=77.90.31.10:4519 , piece_id=MIHLKGHCRTUFQSGBIFKIWPNACAJ4CQOEYRDIKA4KJLFP356VMY6A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.419504032 (198h32m10.115352785s) [noise=true , address=77.90.31.19:4843 , piece_id=JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:39.113150499 (198h32m10.808999252s) [noise=true , address=45.91.116.6:4973 , piece_id=Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:39.113177150 (198h32m10.809025903s) [seg_pos={0 0}, upload=174963]: Segment reached optimal threshold of 13 pieces. Nov 7 18:12:39.113381018 (198h32m10.809229772s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113390166 (198h32m10.809238919s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113460199 (198h32m10.809308952s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113470077 (198h32m10.809318821s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113780338 (198h32m10.809629091s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113788243 (198h32m10.809636986s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113858657 (198h32m10.8097074s) [seg_pos={0 0}, upload=174963]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 7 18:12:39.114030194 (198h32m10.809878947s) [upload=174963]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 7 18:12:39.114038290 (198h32m10.809887043s) [upload=174963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 7 18:12:39.451448640 (198h32m11.147297403s) [upload=174963]: Done uploading segments... Nov 7 18:12:39.451473607 (198h32m11.14732236s) [upload=174963]: Upload finished. err: <nil> Duration: 6.089058716s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [X.XXXXX.X...................................................] duration: 6.074324657s trace: https://jaeger.dev.storj.io/trace/422f33ac8ae6a74a FAIL (2025-11-07 18:52:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ) context deadline exceeded; piecestore: (Node ID: 12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, Piece ID: JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ) context deadline exceeded ======LOGS====== Nov 7 18:12:35.791697270 (198h32m7.487546023s) [upload=174963]: Starting upload Nov 7 18:12:35.791791689 (198h32m7.487640452s) [upload=174963]: Done starting upload Nov 7 18:12:35.791885458 (198h32m7.487734211s) [upload=174963]: Uploading segments... Nov 7 18:12:35.792146455 (198h32m7.487995208s) [upload=174963]: Got next segment. Inline: false Nov 7 18:12:35.792195278 (198h32m7.488044021s) [upload=174963]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 7 18:12:35.792201820 (198h32m7.488050563s) [upload=174963]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 7 18:12:36.063867588 (198h32m7.759716341s) [seg_pos={0 0}, upload=174963]: Begin upload segment... Nov 7 18:12:36.063937832 (198h32m7.759786585s) [seg_pos={0 0}, upload=174963]: Done begin upload segment. Nov 7 18:12:36.063988689 (198h32m7.759837463s) [noise=true , address=145.79.227.27:4690 , piece_id=7CVNJTAEBVYJXRWNSFYFP3E7SKMY5XI4CV2JEZJLSXH4TA2F2JZQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.064083510 (198h32m7.759932263s) [noise=true , address=145.79.227.10:4403 , piece_id=JCMXNJ5PW5ULGOJZ5SGB423VW7VJPC5REYNSNWJZJXHGUQXUS3KA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.064896969 (198h32m7.760745762s) [noise=true , address=145.79.227.28:4726 , piece_id=D6DH5FHFB5ZTPANRPUWZKRYPFYUV2LS6AY5FQL5RBJZIUNM4FKLQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.065443520 (198h32m7.761292273s) [noise=true , address=77.90.31.11:4554 , piece_id=TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.065970974 (198h32m7.761819777s) [noise=true , address=145.79.227.8:4330 , piece_id=32XBCOEGSUTVUCCWVM2D7CB6FKMNLD6N2N4655B3ZYHNAIL4R2QQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.066422053 (198h32m7.762270806s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.067054427 (198h32m7.76290318s) [noise=true , address=77.90.31.10:4519 , piece_id=MIHLKGHCRTUFQSGBIFKIWPNACAJ4CQOEYRDIKA4KJLFP356VMY6A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.067645042 (198h32m7.763493795s) [noise=true , address=45.91.116.6:4973 , piece_id=Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.068379079 (198h32m7.764227842s) [noise=true , address=77.90.31.19:4843 , piece_id=JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.070931099 (198h32m7.766779862s) [noise=true , address=145.79.227.11:4732 , piece_id=VKSPYLTD6C6XBUFXXVVPHIPGFRCRNQK6EVI4SM7Y6ODOMVGGAJOQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.073100420 (198h32m7.768949173s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.075821872 (198h32m7.771670625s) [noise=true , address=45.91.116.4:4937 , piece_id=CWDA2IYNFUYHVDBJ74Y3CJKCZUBN4QYUEJYFKZNIBAUYYAXO3WFQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.078400863 (198h32m7.774249626s) [noise=true , address=45.91.116.10:4082 , piece_id=RW62GCPC56K2IMOFF276GQECXRKW6NY6W44TNAYFNDGQS3UK7EEA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.081230762 (198h32m7.777079515s) [noise=true , address=145.79.227.15:4876 , piece_id=ZPGKGNXRFEWRRJKTS7NKBEVQA7OAWNWYDR3DUTPHT3XVYRJSKWPQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.083298258 (198h32m7.779147012s) [noise=true , address=145.79.227.20:4438 , piece_id=V6WDUVDB7WHTDCTSYCFWS22UKRD5NMXGJ2GHLSFLVVOXZMZWLL4A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.084602091 (198h32m7.780450844s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:37.640211006 (198h32m9.336059759s) [upload=174963]: Next returned nil segment Nov 7 18:12:37.640227878 (198h32m9.336076621s) [upload=174963]: Waiting for error group managing segments... Nov 7 18:12:38.007509819 (198h32m9.703358572s) [noise=true , address=145.79.227.10:4403 , piece_id=JCMXNJ5PW5ULGOJZ5SGB423VW7VJPC5REYNSNWJZJXHGUQXUS3KA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.015182750 (198h32m9.711031503s) [noise=true , address=145.79.227.28:4726 , piece_id=D6DH5FHFB5ZTPANRPUWZKRYPFYUV2LS6AY5FQL5RBJZIUNM4FKLQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.020400886 (198h32m9.71624964s) [noise=true , address=145.79.227.27:4690 , piece_id=7CVNJTAEBVYJXRWNSFYFP3E7SKMY5XI4CV2JEZJLSXH4TA2F2JZQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.062361869 (198h32m9.758210622s) [noise=true , address=145.79.227.15:4876 , piece_id=ZPGKGNXRFEWRRJKTS7NKBEVQA7OAWNWYDR3DUTPHT3XVYRJSKWPQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.096654439 (198h32m9.792503192s) [noise=true , address=145.79.227.20:4438 , piece_id=V6WDUVDB7WHTDCTSYCFWS22UKRD5NMXGJ2GHLSFLVVOXZMZWLL4A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.244365682 (198h32m9.940214435s) [noise=true , address=145.79.227.8:4330 , piece_id=32XBCOEGSUTVUCCWVM2D7CB6FKMNLD6N2N4655B3ZYHNAIL4R2QQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.245114297 (198h32m9.94096305s) [noise=true , address=145.79.227.11:4732 , piece_id=VKSPYLTD6C6XBUFXXVVPHIPGFRCRNQK6EVI4SM7Y6ODOMVGGAJOQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.324356281 (198h32m10.020205034s) [noise=true , address=77.90.31.11:4554 , piece_id=TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.331696878 (198h32m10.027545621s) [noise=true , address=45.91.116.4:4937 , piece_id=CWDA2IYNFUYHVDBJ74Y3CJKCZUBN4QYUEJYFKZNIBAUYYAXO3WFQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.334142676 (198h32m10.029991429s) [noise=true , address=45.91.116.10:4082 , piece_id=RW62GCPC56K2IMOFF276GQECXRKW6NY6W44TNAYFNDGQS3UK7EEA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.346539031 (198h32m10.042387794s) [noise=true , address=77.90.31.10:4519 , piece_id=MIHLKGHCRTUFQSGBIFKIWPNACAJ4CQOEYRDIKA4KJLFP356VMY6A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.419504032 (198h32m10.115352785s) [noise=true , address=77.90.31.19:4843 , piece_id=JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:39.113150499 (198h32m10.808999252s) [noise=true , address=45.91.116.6:4973 , piece_id=Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:39.113177150 (198h32m10.809025903s) [seg_pos={0 0}, upload=174963]: Segment reached optimal threshold of 13 pieces. Nov 7 18:12:39.113381018 (198h32m10.809229772s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113390166 (198h32m10.809238919s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113460199 (198h32m10.809308952s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113470077 (198h32m10.809318821s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113780338 (198h32m10.809629091s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113788243 (198h32m10.809636986s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113858657 (198h32m10.8097074s) [seg_pos={0 0}, upload=174963]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 7 18:12:39.114030194 (198h32m10.809878947s) [upload=174963]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 7 18:12:39.114038290 (198h32m10.809887043s) [upload=174963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 7 18:12:39.451448640 (198h32m11.147297403s) [upload=174963]: Done uploading segments... Nov 7 18:12:39.451473607 (198h32m11.14732236s) [upload=174963]: Upload finished. err: <nil> Duration: 6.074356207s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [............................................X...............] duration: 3.549021865s trace: https://jaeger.dev.storj.io/trace/272952ae802c1916 FAIL (2025-11-07 18:52:36 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.549070336s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.............................................X..............] duration: 3.444059471s trace: https://jaeger.dev.storj.io/trace/272952ae802c1916 FAIL (2025-11-07 18:51:58 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.444084168s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [XX.XXXXX.X..................................................] duration: 6.280154512s trace: https://jaeger.dev.storj.io/trace/422f33ac8ae6a74a FAIL (2025-11-07 18:51:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ) context deadline exceeded; piecestore: (Node ID: 12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, Piece ID: JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ) context deadline exceeded ======LOGS====== Nov 7 18:12:35.791697270 (198h32m7.487546023s) [upload=174963]: Starting upload Nov 7 18:12:35.791791689 (198h32m7.487640452s) [upload=174963]: Done starting upload Nov 7 18:12:35.791885458 (198h32m7.487734211s) [upload=174963]: Uploading segments... Nov 7 18:12:35.792146455 (198h32m7.487995208s) [upload=174963]: Got next segment. Inline: false Nov 7 18:12:35.792195278 (198h32m7.488044021s) [upload=174963]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 7 18:12:35.792201820 (198h32m7.488050563s) [upload=174963]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 7 18:12:36.063867588 (198h32m7.759716341s) [seg_pos={0 0}, upload=174963]: Begin upload segment... Nov 7 18:12:36.063937832 (198h32m7.759786585s) [seg_pos={0 0}, upload=174963]: Done begin upload segment. Nov 7 18:12:36.063988689 (198h32m7.759837463s) [noise=true , address=145.79.227.27:4690 , piece_id=7CVNJTAEBVYJXRWNSFYFP3E7SKMY5XI4CV2JEZJLSXH4TA2F2JZQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.064083510 (198h32m7.759932263s) [noise=true , address=145.79.227.10:4403 , piece_id=JCMXNJ5PW5ULGOJZ5SGB423VW7VJPC5REYNSNWJZJXHGUQXUS3KA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.064896969 (198h32m7.760745762s) [noise=true , address=145.79.227.28:4726 , piece_id=D6DH5FHFB5ZTPANRPUWZKRYPFYUV2LS6AY5FQL5RBJZIUNM4FKLQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.065443520 (198h32m7.761292273s) [noise=true , address=77.90.31.11:4554 , piece_id=TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.065970974 (198h32m7.761819777s) [noise=true , address=145.79.227.8:4330 , piece_id=32XBCOEGSUTVUCCWVM2D7CB6FKMNLD6N2N4655B3ZYHNAIL4R2QQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.066422053 (198h32m7.762270806s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.067054427 (198h32m7.76290318s) [noise=true , address=77.90.31.10:4519 , piece_id=MIHLKGHCRTUFQSGBIFKIWPNACAJ4CQOEYRDIKA4KJLFP356VMY6A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.067645042 (198h32m7.763493795s) [noise=true , address=45.91.116.6:4973 , piece_id=Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.068379079 (198h32m7.764227842s) [noise=true , address=77.90.31.19:4843 , piece_id=JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.070931099 (198h32m7.766779862s) [noise=true , address=145.79.227.11:4732 , piece_id=VKSPYLTD6C6XBUFXXVVPHIPGFRCRNQK6EVI4SM7Y6ODOMVGGAJOQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.073100420 (198h32m7.768949173s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.075821872 (198h32m7.771670625s) [noise=true , address=45.91.116.4:4937 , piece_id=CWDA2IYNFUYHVDBJ74Y3CJKCZUBN4QYUEJYFKZNIBAUYYAXO3WFQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.078400863 (198h32m7.774249626s) [noise=true , address=45.91.116.10:4082 , piece_id=RW62GCPC56K2IMOFF276GQECXRKW6NY6W44TNAYFNDGQS3UK7EEA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.081230762 (198h32m7.777079515s) [noise=true , address=145.79.227.15:4876 , piece_id=ZPGKGNXRFEWRRJKTS7NKBEVQA7OAWNWYDR3DUTPHT3XVYRJSKWPQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.083298258 (198h32m7.779147012s) [noise=true , address=145.79.227.20:4438 , piece_id=V6WDUVDB7WHTDCTSYCFWS22UKRD5NMXGJ2GHLSFLVVOXZMZWLL4A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.084602091 (198h32m7.780450844s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:37.640211006 (198h32m9.336059759s) [upload=174963]: Next returned nil segment Nov 7 18:12:37.640227878 (198h32m9.336076621s) [upload=174963]: Waiting for error group managing segments... Nov 7 18:12:38.007509819 (198h32m9.703358572s) [noise=true , address=145.79.227.10:4403 , piece_id=JCMXNJ5PW5ULGOJZ5SGB423VW7VJPC5REYNSNWJZJXHGUQXUS3KA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.015182750 (198h32m9.711031503s) [noise=true , address=145.79.227.28:4726 , piece_id=D6DH5FHFB5ZTPANRPUWZKRYPFYUV2LS6AY5FQL5RBJZIUNM4FKLQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.020400886 (198h32m9.71624964s) [noise=true , address=145.79.227.27:4690 , piece_id=7CVNJTAEBVYJXRWNSFYFP3E7SKMY5XI4CV2JEZJLSXH4TA2F2JZQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.062361869 (198h32m9.758210622s) [noise=true , address=145.79.227.15:4876 , piece_id=ZPGKGNXRFEWRRJKTS7NKBEVQA7OAWNWYDR3DUTPHT3XVYRJSKWPQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.096654439 (198h32m9.792503192s) [noise=true , address=145.79.227.20:4438 , piece_id=V6WDUVDB7WHTDCTSYCFWS22UKRD5NMXGJ2GHLSFLVVOXZMZWLL4A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.244365682 (198h32m9.940214435s) [noise=true , address=145.79.227.8:4330 , piece_id=32XBCOEGSUTVUCCWVM2D7CB6FKMNLD6N2N4655B3ZYHNAIL4R2QQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.245114297 (198h32m9.94096305s) [noise=true , address=145.79.227.11:4732 , piece_id=VKSPYLTD6C6XBUFXXVVPHIPGFRCRNQK6EVI4SM7Y6ODOMVGGAJOQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.324356281 (198h32m10.020205034s) [noise=true , address=77.90.31.11:4554 , piece_id=TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.331696878 (198h32m10.027545621s) [noise=true , address=45.91.116.4:4937 , piece_id=CWDA2IYNFUYHVDBJ74Y3CJKCZUBN4QYUEJYFKZNIBAUYYAXO3WFQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.334142676 (198h32m10.029991429s) [noise=true , address=45.91.116.10:4082 , piece_id=RW62GCPC56K2IMOFF276GQECXRKW6NY6W44TNAYFNDGQS3UK7EEA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.346539031 (198h32m10.042387794s) [noise=true , address=77.90.31.10:4519 , piece_id=MIHLKGHCRTUFQSGBIFKIWPNACAJ4CQOEYRDIKA4KJLFP356VMY6A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.419504032 (198h32m10.115352785s) [noise=true , address=77.90.31.19:4843 , piece_id=JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:39.113150499 (198h32m10.808999252s) [noise=true , address=45.91.116.6:4973 , piece_id=Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:39.113177150 (198h32m10.809025903s) [seg_pos={0 0}, upload=174963]: Segment reached optimal threshold of 13 pieces. Nov 7 18:12:39.113381018 (198h32m10.809229772s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113390166 (198h32m10.809238919s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113460199 (198h32m10.809308952s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113470077 (198h32m10.809318821s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113780338 (198h32m10.809629091s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113788243 (198h32m10.809636986s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113858657 (198h32m10.8097074s) [seg_pos={0 0}, upload=174963]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 7 18:12:39.114030194 (198h32m10.809878947s) [upload=174963]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 7 18:12:39.114038290 (198h32m10.809887043s) [upload=174963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 7 18:12:39.451448640 (198h32m11.147297403s) [upload=174963]: Done uploading segments... Nov 7 18:12:39.451473607 (198h32m11.14732236s) [upload=174963]: Upload finished. err: <nil> Duration: 6.280185089s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.XX.XXXXX.X.................................................] duration: 6.015401057s trace: https://jaeger.dev.storj.io/trace/422f33ac8ae6a74a FAIL (2025-11-07 18:51:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ) context deadline exceeded; piecestore: (Node ID: 12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, Piece ID: JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ) context deadline exceeded ======LOGS====== Nov 7 18:12:35.791697270 (198h32m7.487546023s) [upload=174963]: Starting upload Nov 7 18:12:35.791791689 (198h32m7.487640452s) [upload=174963]: Done starting upload Nov 7 18:12:35.791885458 (198h32m7.487734211s) [upload=174963]: Uploading segments... Nov 7 18:12:35.792146455 (198h32m7.487995208s) [upload=174963]: Got next segment. Inline: false Nov 7 18:12:35.792195278 (198h32m7.488044021s) [upload=174963]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 7 18:12:35.792201820 (198h32m7.488050563s) [upload=174963]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 7 18:12:36.063867588 (198h32m7.759716341s) [seg_pos={0 0}, upload=174963]: Begin upload segment... Nov 7 18:12:36.063937832 (198h32m7.759786585s) [seg_pos={0 0}, upload=174963]: Done begin upload segment. Nov 7 18:12:36.063988689 (198h32m7.759837463s) [noise=true , address=145.79.227.27:4690 , piece_id=7CVNJTAEBVYJXRWNSFYFP3E7SKMY5XI4CV2JEZJLSXH4TA2F2JZQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.064083510 (198h32m7.759932263s) [noise=true , address=145.79.227.10:4403 , piece_id=JCMXNJ5PW5ULGOJZ5SGB423VW7VJPC5REYNSNWJZJXHGUQXUS3KA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.064896969 (198h32m7.760745762s) [noise=true , address=145.79.227.28:4726 , piece_id=D6DH5FHFB5ZTPANRPUWZKRYPFYUV2LS6AY5FQL5RBJZIUNM4FKLQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.065443520 (198h32m7.761292273s) [noise=true , address=77.90.31.11:4554 , piece_id=TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.065970974 (198h32m7.761819777s) [noise=true , address=145.79.227.8:4330 , piece_id=32XBCOEGSUTVUCCWVM2D7CB6FKMNLD6N2N4655B3ZYHNAIL4R2QQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.066422053 (198h32m7.762270806s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.067054427 (198h32m7.76290318s) [noise=true , address=77.90.31.10:4519 , piece_id=MIHLKGHCRTUFQSGBIFKIWPNACAJ4CQOEYRDIKA4KJLFP356VMY6A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.067645042 (198h32m7.763493795s) [noise=true , address=45.91.116.6:4973 , piece_id=Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.068379079 (198h32m7.764227842s) [noise=true , address=77.90.31.19:4843 , piece_id=JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.070931099 (198h32m7.766779862s) [noise=true , address=145.79.227.11:4732 , piece_id=VKSPYLTD6C6XBUFXXVVPHIPGFRCRNQK6EVI4SM7Y6ODOMVGGAJOQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.073100420 (198h32m7.768949173s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.075821872 (198h32m7.771670625s) [noise=true , address=45.91.116.4:4937 , piece_id=CWDA2IYNFUYHVDBJ74Y3CJKCZUBN4QYUEJYFKZNIBAUYYAXO3WFQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.078400863 (198h32m7.774249626s) [noise=true , address=45.91.116.10:4082 , piece_id=RW62GCPC56K2IMOFF276GQECXRKW6NY6W44TNAYFNDGQS3UK7EEA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.081230762 (198h32m7.777079515s) [noise=true , address=145.79.227.15:4876 , piece_id=ZPGKGNXRFEWRRJKTS7NKBEVQA7OAWNWYDR3DUTPHT3XVYRJSKWPQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.083298258 (198h32m7.779147012s) [noise=true , address=145.79.227.20:4438 , piece_id=V6WDUVDB7WHTDCTSYCFWS22UKRD5NMXGJ2GHLSFLVVOXZMZWLL4A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.084602091 (198h32m7.780450844s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:37.640211006 (198h32m9.336059759s) [upload=174963]: Next returned nil segment Nov 7 18:12:37.640227878 (198h32m9.336076621s) [upload=174963]: Waiting for error group managing segments... Nov 7 18:12:38.007509819 (198h32m9.703358572s) [noise=true , address=145.79.227.10:4403 , piece_id=JCMXNJ5PW5ULGOJZ5SGB423VW7VJPC5REYNSNWJZJXHGUQXUS3KA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.015182750 (198h32m9.711031503s) [noise=true , address=145.79.227.28:4726 , piece_id=D6DH5FHFB5ZTPANRPUWZKRYPFYUV2LS6AY5FQL5RBJZIUNM4FKLQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.020400886 (198h32m9.71624964s) [noise=true , address=145.79.227.27:4690 , piece_id=7CVNJTAEBVYJXRWNSFYFP3E7SKMY5XI4CV2JEZJLSXH4TA2F2JZQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.062361869 (198h32m9.758210622s) [noise=true , address=145.79.227.15:4876 , piece_id=ZPGKGNXRFEWRRJKTS7NKBEVQA7OAWNWYDR3DUTPHT3XVYRJSKWPQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.096654439 (198h32m9.792503192s) [noise=true , address=145.79.227.20:4438 , piece_id=V6WDUVDB7WHTDCTSYCFWS22UKRD5NMXGJ2GHLSFLVVOXZMZWLL4A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.244365682 (198h32m9.940214435s) [noise=true , address=145.79.227.8:4330 , piece_id=32XBCOEGSUTVUCCWVM2D7CB6FKMNLD6N2N4655B3ZYHNAIL4R2QQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.245114297 (198h32m9.94096305s) [noise=true , address=145.79.227.11:4732 , piece_id=VKSPYLTD6C6XBUFXXVVPHIPGFRCRNQK6EVI4SM7Y6ODOMVGGAJOQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.324356281 (198h32m10.020205034s) [noise=true , address=77.90.31.11:4554 , piece_id=TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.331696878 (198h32m10.027545621s) [noise=true , address=45.91.116.4:4937 , piece_id=CWDA2IYNFUYHVDBJ74Y3CJKCZUBN4QYUEJYFKZNIBAUYYAXO3WFQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.334142676 (198h32m10.029991429s) [noise=true , address=45.91.116.10:4082 , piece_id=RW62GCPC56K2IMOFF276GQECXRKW6NY6W44TNAYFNDGQS3UK7EEA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.346539031 (198h32m10.042387794s) [noise=true , address=77.90.31.10:4519 , piece_id=MIHLKGHCRTUFQSGBIFKIWPNACAJ4CQOEYRDIKA4KJLFP356VMY6A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.419504032 (198h32m10.115352785s) [noise=true , address=77.90.31.19:4843 , piece_id=JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:39.113150499 (198h32m10.808999252s) [noise=true , address=45.91.116.6:4973 , piece_id=Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:39.113177150 (198h32m10.809025903s) [seg_pos={0 0}, upload=174963]: Segment reached optimal threshold of 13 pieces. Nov 7 18:12:39.113381018 (198h32m10.809229772s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113390166 (198h32m10.809238919s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113460199 (198h32m10.809308952s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113470077 (198h32m10.809318821s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113780338 (198h32m10.809629091s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113788243 (198h32m10.809636986s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113858657 (198h32m10.8097074s) [seg_pos={0 0}, upload=174963]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 7 18:12:39.114030194 (198h32m10.809878947s) [upload=174963]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 7 18:12:39.114038290 (198h32m10.809887043s) [upload=174963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 7 18:12:39.451448640 (198h32m11.147297403s) [upload=174963]: Done uploading segments... Nov 7 18:12:39.451473607 (198h32m11.14732236s) [upload=174963]: Upload finished. err: <nil> Duration: 6.015431966s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..............................................X.............] duration: 3.529224834s trace: https://jaeger.dev.storj.io/trace/272952ae802c1916 FAIL (2025-11-07 18:50:47 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.52925453s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [..XX.XXXXX.X................................................] duration: 5.954434078s trace: https://jaeger.dev.storj.io/trace/422f33ac8ae6a74a FAIL (2025-11-07 18:50:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ) context deadline exceeded; piecestore: (Node ID: 12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, Piece ID: JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ) context deadline exceeded ======LOGS====== Nov 7 18:12:35.791697270 (198h32m7.487546023s) [upload=174963]: Starting upload Nov 7 18:12:35.791791689 (198h32m7.487640452s) [upload=174963]: Done starting upload Nov 7 18:12:35.791885458 (198h32m7.487734211s) [upload=174963]: Uploading segments... Nov 7 18:12:35.792146455 (198h32m7.487995208s) [upload=174963]: Got next segment. Inline: false Nov 7 18:12:35.792195278 (198h32m7.488044021s) [upload=174963]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 7 18:12:35.792201820 (198h32m7.488050563s) [upload=174963]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 7 18:12:36.063867588 (198h32m7.759716341s) [seg_pos={0 0}, upload=174963]: Begin upload segment... Nov 7 18:12:36.063937832 (198h32m7.759786585s) [seg_pos={0 0}, upload=174963]: Done begin upload segment. Nov 7 18:12:36.063988689 (198h32m7.759837463s) [noise=true , address=145.79.227.27:4690 , piece_id=7CVNJTAEBVYJXRWNSFYFP3E7SKMY5XI4CV2JEZJLSXH4TA2F2JZQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.064083510 (198h32m7.759932263s) [noise=true , address=145.79.227.10:4403 , piece_id=JCMXNJ5PW5ULGOJZ5SGB423VW7VJPC5REYNSNWJZJXHGUQXUS3KA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.064896969 (198h32m7.760745762s) [noise=true , address=145.79.227.28:4726 , piece_id=D6DH5FHFB5ZTPANRPUWZKRYPFYUV2LS6AY5FQL5RBJZIUNM4FKLQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.065443520 (198h32m7.761292273s) [noise=true , address=77.90.31.11:4554 , piece_id=TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.065970974 (198h32m7.761819777s) [noise=true , address=145.79.227.8:4330 , piece_id=32XBCOEGSUTVUCCWVM2D7CB6FKMNLD6N2N4655B3ZYHNAIL4R2QQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.066422053 (198h32m7.762270806s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.067054427 (198h32m7.76290318s) [noise=true , address=77.90.31.10:4519 , piece_id=MIHLKGHCRTUFQSGBIFKIWPNACAJ4CQOEYRDIKA4KJLFP356VMY6A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.067645042 (198h32m7.763493795s) [noise=true , address=45.91.116.6:4973 , piece_id=Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.068379079 (198h32m7.764227842s) [noise=true , address=77.90.31.19:4843 , piece_id=JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.070931099 (198h32m7.766779862s) [noise=true , address=145.79.227.11:4732 , piece_id=VKSPYLTD6C6XBUFXXVVPHIPGFRCRNQK6EVI4SM7Y6ODOMVGGAJOQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.073100420 (198h32m7.768949173s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.075821872 (198h32m7.771670625s) [noise=true , address=45.91.116.4:4937 , piece_id=CWDA2IYNFUYHVDBJ74Y3CJKCZUBN4QYUEJYFKZNIBAUYYAXO3WFQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.078400863 (198h32m7.774249626s) [noise=true , address=45.91.116.10:4082 , piece_id=RW62GCPC56K2IMOFF276GQECXRKW6NY6W44TNAYFNDGQS3UK7EEA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.081230762 (198h32m7.777079515s) [noise=true , address=145.79.227.15:4876 , piece_id=ZPGKGNXRFEWRRJKTS7NKBEVQA7OAWNWYDR3DUTPHT3XVYRJSKWPQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.083298258 (198h32m7.779147012s) [noise=true , address=145.79.227.20:4438 , piece_id=V6WDUVDB7WHTDCTSYCFWS22UKRD5NMXGJ2GHLSFLVVOXZMZWLL4A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.084602091 (198h32m7.780450844s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:37.640211006 (198h32m9.336059759s) [upload=174963]: Next returned nil segment Nov 7 18:12:37.640227878 (198h32m9.336076621s) [upload=174963]: Waiting for error group managing segments... Nov 7 18:12:38.007509819 (198h32m9.703358572s) [noise=true , address=145.79.227.10:4403 , piece_id=JCMXNJ5PW5ULGOJZ5SGB423VW7VJPC5REYNSNWJZJXHGUQXUS3KA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.015182750 (198h32m9.711031503s) [noise=true , address=145.79.227.28:4726 , piece_id=D6DH5FHFB5ZTPANRPUWZKRYPFYUV2LS6AY5FQL5RBJZIUNM4FKLQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.020400886 (198h32m9.71624964s) [noise=true , address=145.79.227.27:4690 , piece_id=7CVNJTAEBVYJXRWNSFYFP3E7SKMY5XI4CV2JEZJLSXH4TA2F2JZQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.062361869 (198h32m9.758210622s) [noise=true , address=145.79.227.15:4876 , piece_id=ZPGKGNXRFEWRRJKTS7NKBEVQA7OAWNWYDR3DUTPHT3XVYRJSKWPQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.096654439 (198h32m9.792503192s) [noise=true , address=145.79.227.20:4438 , piece_id=V6WDUVDB7WHTDCTSYCFWS22UKRD5NMXGJ2GHLSFLVVOXZMZWLL4A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.244365682 (198h32m9.940214435s) [noise=true , address=145.79.227.8:4330 , piece_id=32XBCOEGSUTVUCCWVM2D7CB6FKMNLD6N2N4655B3ZYHNAIL4R2QQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.245114297 (198h32m9.94096305s) [noise=true , address=145.79.227.11:4732 , piece_id=VKSPYLTD6C6XBUFXXVVPHIPGFRCRNQK6EVI4SM7Y6ODOMVGGAJOQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.324356281 (198h32m10.020205034s) [noise=true , address=77.90.31.11:4554 , piece_id=TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.331696878 (198h32m10.027545621s) [noise=true , address=45.91.116.4:4937 , piece_id=CWDA2IYNFUYHVDBJ74Y3CJKCZUBN4QYUEJYFKZNIBAUYYAXO3WFQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.334142676 (198h32m10.029991429s) [noise=true , address=45.91.116.10:4082 , piece_id=RW62GCPC56K2IMOFF276GQECXRKW6NY6W44TNAYFNDGQS3UK7EEA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.346539031 (198h32m10.042387794s) [noise=true , address=77.90.31.10:4519 , piece_id=MIHLKGHCRTUFQSGBIFKIWPNACAJ4CQOEYRDIKA4KJLFP356VMY6A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.419504032 (198h32m10.115352785s) [noise=true , address=77.90.31.19:4843 , piece_id=JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:39.113150499 (198h32m10.808999252s) [noise=true , address=45.91.116.6:4973 , piece_id=Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:39.113177150 (198h32m10.809025903s) [seg_pos={0 0}, upload=174963]: Segment reached optimal threshold of 13 pieces. Nov 7 18:12:39.113381018 (198h32m10.809229772s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113390166 (198h32m10.809238919s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113460199 (198h32m10.809308952s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113470077 (198h32m10.809318821s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113780338 (198h32m10.809629091s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113788243 (198h32m10.809636986s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113858657 (198h32m10.8097074s) [seg_pos={0 0}, upload=174963]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 7 18:12:39.114030194 (198h32m10.809878947s) [upload=174963]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 7 18:12:39.114038290 (198h32m10.809887043s) [upload=174963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 7 18:12:39.451448640 (198h32m11.147297403s) [upload=174963]: Done uploading segments... Nov 7 18:12:39.451473607 (198h32m11.14732236s) [upload=174963]: Upload finished. err: <nil> Duration: 5.954463315s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [X..XX.XXXXX.X...............................................] duration: 5.928833747s trace: https://jaeger.dev.storj.io/trace/422f33ac8ae6a74a FAIL (2025-11-07 18:49:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ) context deadline exceeded; piecestore: (Node ID: 12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, Piece ID: JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ) context deadline exceeded ======LOGS====== Nov 7 18:12:35.791697270 (198h32m7.487546023s) [upload=174963]: Starting upload Nov 7 18:12:35.791791689 (198h32m7.487640452s) [upload=174963]: Done starting upload Nov 7 18:12:35.791885458 (198h32m7.487734211s) [upload=174963]: Uploading segments... Nov 7 18:12:35.792146455 (198h32m7.487995208s) [upload=174963]: Got next segment. Inline: false Nov 7 18:12:35.792195278 (198h32m7.488044021s) [upload=174963]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 7 18:12:35.792201820 (198h32m7.488050563s) [upload=174963]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 7 18:12:36.063867588 (198h32m7.759716341s) [seg_pos={0 0}, upload=174963]: Begin upload segment... Nov 7 18:12:36.063937832 (198h32m7.759786585s) [seg_pos={0 0}, upload=174963]: Done begin upload segment. Nov 7 18:12:36.063988689 (198h32m7.759837463s) [noise=true , address=145.79.227.27:4690 , piece_id=7CVNJTAEBVYJXRWNSFYFP3E7SKMY5XI4CV2JEZJLSXH4TA2F2JZQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.064083510 (198h32m7.759932263s) [noise=true , address=145.79.227.10:4403 , piece_id=JCMXNJ5PW5ULGOJZ5SGB423VW7VJPC5REYNSNWJZJXHGUQXUS3KA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.064896969 (198h32m7.760745762s) [noise=true , address=145.79.227.28:4726 , piece_id=D6DH5FHFB5ZTPANRPUWZKRYPFYUV2LS6AY5FQL5RBJZIUNM4FKLQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.065443520 (198h32m7.761292273s) [noise=true , address=77.90.31.11:4554 , piece_id=TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.065970974 (198h32m7.761819777s) [noise=true , address=145.79.227.8:4330 , piece_id=32XBCOEGSUTVUCCWVM2D7CB6FKMNLD6N2N4655B3ZYHNAIL4R2QQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.066422053 (198h32m7.762270806s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.067054427 (198h32m7.76290318s) [noise=true , address=77.90.31.10:4519 , piece_id=MIHLKGHCRTUFQSGBIFKIWPNACAJ4CQOEYRDIKA4KJLFP356VMY6A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.067645042 (198h32m7.763493795s) [noise=true , address=45.91.116.6:4973 , piece_id=Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.068379079 (198h32m7.764227842s) [noise=true , address=77.90.31.19:4843 , piece_id=JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.070931099 (198h32m7.766779862s) [noise=true , address=145.79.227.11:4732 , piece_id=VKSPYLTD6C6XBUFXXVVPHIPGFRCRNQK6EVI4SM7Y6ODOMVGGAJOQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.073100420 (198h32m7.768949173s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.075821872 (198h32m7.771670625s) [noise=true , address=45.91.116.4:4937 , piece_id=CWDA2IYNFUYHVDBJ74Y3CJKCZUBN4QYUEJYFKZNIBAUYYAXO3WFQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.078400863 (198h32m7.774249626s) [noise=true , address=45.91.116.10:4082 , piece_id=RW62GCPC56K2IMOFF276GQECXRKW6NY6W44TNAYFNDGQS3UK7EEA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.081230762 (198h32m7.777079515s) [noise=true , address=145.79.227.15:4876 , piece_id=ZPGKGNXRFEWRRJKTS7NKBEVQA7OAWNWYDR3DUTPHT3XVYRJSKWPQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.083298258 (198h32m7.779147012s) [noise=true , address=145.79.227.20:4438 , piece_id=V6WDUVDB7WHTDCTSYCFWS22UKRD5NMXGJ2GHLSFLVVOXZMZWLL4A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.084602091 (198h32m7.780450844s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:37.640211006 (198h32m9.336059759s) [upload=174963]: Next returned nil segment Nov 7 18:12:37.640227878 (198h32m9.336076621s) [upload=174963]: Waiting for error group managing segments... Nov 7 18:12:38.007509819 (198h32m9.703358572s) [noise=true , address=145.79.227.10:4403 , piece_id=JCMXNJ5PW5ULGOJZ5SGB423VW7VJPC5REYNSNWJZJXHGUQXUS3KA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.015182750 (198h32m9.711031503s) [noise=true , address=145.79.227.28:4726 , piece_id=D6DH5FHFB5ZTPANRPUWZKRYPFYUV2LS6AY5FQL5RBJZIUNM4FKLQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.020400886 (198h32m9.71624964s) [noise=true , address=145.79.227.27:4690 , piece_id=7CVNJTAEBVYJXRWNSFYFP3E7SKMY5XI4CV2JEZJLSXH4TA2F2JZQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.062361869 (198h32m9.758210622s) [noise=true , address=145.79.227.15:4876 , piece_id=ZPGKGNXRFEWRRJKTS7NKBEVQA7OAWNWYDR3DUTPHT3XVYRJSKWPQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.096654439 (198h32m9.792503192s) [noise=true , address=145.79.227.20:4438 , piece_id=V6WDUVDB7WHTDCTSYCFWS22UKRD5NMXGJ2GHLSFLVVOXZMZWLL4A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.244365682 (198h32m9.940214435s) [noise=true , address=145.79.227.8:4330 , piece_id=32XBCOEGSUTVUCCWVM2D7CB6FKMNLD6N2N4655B3ZYHNAIL4R2QQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.245114297 (198h32m9.94096305s) [noise=true , address=145.79.227.11:4732 , piece_id=VKSPYLTD6C6XBUFXXVVPHIPGFRCRNQK6EVI4SM7Y6ODOMVGGAJOQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.324356281 (198h32m10.020205034s) [noise=true , address=77.90.31.11:4554 , piece_id=TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.331696878 (198h32m10.027545621s) [noise=true , address=45.91.116.4:4937 , piece_id=CWDA2IYNFUYHVDBJ74Y3CJKCZUBN4QYUEJYFKZNIBAUYYAXO3WFQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.334142676 (198h32m10.029991429s) [noise=true , address=45.91.116.10:4082 , piece_id=RW62GCPC56K2IMOFF276GQECXRKW6NY6W44TNAYFNDGQS3UK7EEA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.346539031 (198h32m10.042387794s) [noise=true , address=77.90.31.10:4519 , piece_id=MIHLKGHCRTUFQSGBIFKIWPNACAJ4CQOEYRDIKA4KJLFP356VMY6A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.419504032 (198h32m10.115352785s) [noise=true , address=77.90.31.19:4843 , piece_id=JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:39.113150499 (198h32m10.808999252s) [noise=true , address=45.91.116.6:4973 , piece_id=Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:39.113177150 (198h32m10.809025903s) [seg_pos={0 0}, upload=174963]: Segment reached optimal threshold of 13 pieces. Nov 7 18:12:39.113381018 (198h32m10.809229772s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113390166 (198h32m10.809238919s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113460199 (198h32m10.809308952s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113470077 (198h32m10.809318821s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113780338 (198h32m10.809629091s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113788243 (198h32m10.809636986s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113858657 (198h32m10.8097074s) [seg_pos={0 0}, upload=174963]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 7 18:12:39.114030194 (198h32m10.809878947s) [upload=174963]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 7 18:12:39.114038290 (198h32m10.809887043s) [upload=174963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 7 18:12:39.451448640 (198h32m11.147297403s) [upload=174963]: Done uploading segments... Nov 7 18:12:39.451473607 (198h32m11.14732236s) [upload=174963]: Upload finished. err: <nil> Duration: 5.928862512s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...............................................X............] duration: 3.427956136s trace: https://jaeger.dev.storj.io/trace/272952ae802c1916 FAIL (2025-11-07 18:49:48 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.428017875s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [................................................X...........] duration: 3.547826767s trace: https://jaeger.dev.storj.io/trace/272952ae802c1916 FAIL (2025-11-07 18:49:08 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.548030916s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [.X..XX.XXXXX.X..............................................] duration: 6.366080572s trace: https://jaeger.dev.storj.io/trace/422f33ac8ae6a74a FAIL (2025-11-07 18:49:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ) context deadline exceeded; piecestore: (Node ID: 12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, Piece ID: JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ) context deadline exceeded ======LOGS====== Nov 7 18:12:35.791697270 (198h32m7.487546023s) [upload=174963]: Starting upload Nov 7 18:12:35.791791689 (198h32m7.487640452s) [upload=174963]: Done starting upload Nov 7 18:12:35.791885458 (198h32m7.487734211s) [upload=174963]: Uploading segments... Nov 7 18:12:35.792146455 (198h32m7.487995208s) [upload=174963]: Got next segment. Inline: false Nov 7 18:12:35.792195278 (198h32m7.488044021s) [upload=174963]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 7 18:12:35.792201820 (198h32m7.488050563s) [upload=174963]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 7 18:12:36.063867588 (198h32m7.759716341s) [seg_pos={0 0}, upload=174963]: Begin upload segment... Nov 7 18:12:36.063937832 (198h32m7.759786585s) [seg_pos={0 0}, upload=174963]: Done begin upload segment. Nov 7 18:12:36.063988689 (198h32m7.759837463s) [noise=true , address=145.79.227.27:4690 , piece_id=7CVNJTAEBVYJXRWNSFYFP3E7SKMY5XI4CV2JEZJLSXH4TA2F2JZQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.064083510 (198h32m7.759932263s) [noise=true , address=145.79.227.10:4403 , piece_id=JCMXNJ5PW5ULGOJZ5SGB423VW7VJPC5REYNSNWJZJXHGUQXUS3KA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.064896969 (198h32m7.760745762s) [noise=true , address=145.79.227.28:4726 , piece_id=D6DH5FHFB5ZTPANRPUWZKRYPFYUV2LS6AY5FQL5RBJZIUNM4FKLQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.065443520 (198h32m7.761292273s) [noise=true , address=77.90.31.11:4554 , piece_id=TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.065970974 (198h32m7.761819777s) [noise=true , address=145.79.227.8:4330 , piece_id=32XBCOEGSUTVUCCWVM2D7CB6FKMNLD6N2N4655B3ZYHNAIL4R2QQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.066422053 (198h32m7.762270806s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.067054427 (198h32m7.76290318s) [noise=true , address=77.90.31.10:4519 , piece_id=MIHLKGHCRTUFQSGBIFKIWPNACAJ4CQOEYRDIKA4KJLFP356VMY6A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.067645042 (198h32m7.763493795s) [noise=true , address=45.91.116.6:4973 , piece_id=Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.068379079 (198h32m7.764227842s) [noise=true , address=77.90.31.19:4843 , piece_id=JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.070931099 (198h32m7.766779862s) [noise=true , address=145.79.227.11:4732 , piece_id=VKSPYLTD6C6XBUFXXVVPHIPGFRCRNQK6EVI4SM7Y6ODOMVGGAJOQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.073100420 (198h32m7.768949173s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.075821872 (198h32m7.771670625s) [noise=true , address=45.91.116.4:4937 , piece_id=CWDA2IYNFUYHVDBJ74Y3CJKCZUBN4QYUEJYFKZNIBAUYYAXO3WFQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.078400863 (198h32m7.774249626s) [noise=true , address=45.91.116.10:4082 , piece_id=RW62GCPC56K2IMOFF276GQECXRKW6NY6W44TNAYFNDGQS3UK7EEA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.081230762 (198h32m7.777079515s) [noise=true , address=145.79.227.15:4876 , piece_id=ZPGKGNXRFEWRRJKTS7NKBEVQA7OAWNWYDR3DUTPHT3XVYRJSKWPQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.083298258 (198h32m7.779147012s) [noise=true , address=145.79.227.20:4438 , piece_id=V6WDUVDB7WHTDCTSYCFWS22UKRD5NMXGJ2GHLSFLVVOXZMZWLL4A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.084602091 (198h32m7.780450844s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:37.640211006 (198h32m9.336059759s) [upload=174963]: Next returned nil segment Nov 7 18:12:37.640227878 (198h32m9.336076621s) [upload=174963]: Waiting for error group managing segments... Nov 7 18:12:38.007509819 (198h32m9.703358572s) [noise=true , address=145.79.227.10:4403 , piece_id=JCMXNJ5PW5ULGOJZ5SGB423VW7VJPC5REYNSNWJZJXHGUQXUS3KA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.015182750 (198h32m9.711031503s) [noise=true , address=145.79.227.28:4726 , piece_id=D6DH5FHFB5ZTPANRPUWZKRYPFYUV2LS6AY5FQL5RBJZIUNM4FKLQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.020400886 (198h32m9.71624964s) [noise=true , address=145.79.227.27:4690 , piece_id=7CVNJTAEBVYJXRWNSFYFP3E7SKMY5XI4CV2JEZJLSXH4TA2F2JZQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.062361869 (198h32m9.758210622s) [noise=true , address=145.79.227.15:4876 , piece_id=ZPGKGNXRFEWRRJKTS7NKBEVQA7OAWNWYDR3DUTPHT3XVYRJSKWPQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.096654439 (198h32m9.792503192s) [noise=true , address=145.79.227.20:4438 , piece_id=V6WDUVDB7WHTDCTSYCFWS22UKRD5NMXGJ2GHLSFLVVOXZMZWLL4A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.244365682 (198h32m9.940214435s) [noise=true , address=145.79.227.8:4330 , piece_id=32XBCOEGSUTVUCCWVM2D7CB6FKMNLD6N2N4655B3ZYHNAIL4R2QQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.245114297 (198h32m9.94096305s) [noise=true , address=145.79.227.11:4732 , piece_id=VKSPYLTD6C6XBUFXXVVPHIPGFRCRNQK6EVI4SM7Y6ODOMVGGAJOQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.324356281 (198h32m10.020205034s) [noise=true , address=77.90.31.11:4554 , piece_id=TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.331696878 (198h32m10.027545621s) [noise=true , address=45.91.116.4:4937 , piece_id=CWDA2IYNFUYHVDBJ74Y3CJKCZUBN4QYUEJYFKZNIBAUYYAXO3WFQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.334142676 (198h32m10.029991429s) [noise=true , address=45.91.116.10:4082 , piece_id=RW62GCPC56K2IMOFF276GQECXRKW6NY6W44TNAYFNDGQS3UK7EEA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.346539031 (198h32m10.042387794s) [noise=true , address=77.90.31.10:4519 , piece_id=MIHLKGHCRTUFQSGBIFKIWPNACAJ4CQOEYRDIKA4KJLFP356VMY6A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.419504032 (198h32m10.115352785s) [noise=true , address=77.90.31.19:4843 , piece_id=JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:39.113150499 (198h32m10.808999252s) [noise=true , address=45.91.116.6:4973 , piece_id=Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:39.113177150 (198h32m10.809025903s) [seg_pos={0 0}, upload=174963]: Segment reached optimal threshold of 13 pieces. Nov 7 18:12:39.113381018 (198h32m10.809229772s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113390166 (198h32m10.809238919s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113460199 (198h32m10.809308952s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113470077 (198h32m10.809318821s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113780338 (198h32m10.809629091s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113788243 (198h32m10.809636986s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113858657 (198h32m10.8097074s) [seg_pos={0 0}, upload=174963]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 7 18:12:39.114030194 (198h32m10.809878947s) [upload=174963]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 7 18:12:39.114038290 (198h32m10.809887043s) [upload=174963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 7 18:12:39.451448640 (198h32m11.147297403s) [upload=174963]: Done uploading segments... Nov 7 18:12:39.451473607 (198h32m11.14732236s) [upload=174963]: Upload finished. err: <nil> Duration: 6.366106802s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [X.X..XX.XXXXX.X.............................................] duration: 6.024422124s trace: https://jaeger.dev.storj.io/trace/422f33ac8ae6a74a FAIL (2025-11-07 18:48:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ) context deadline exceeded; piecestore: (Node ID: 12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, Piece ID: JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ) context deadline exceeded ======LOGS====== Nov 7 18:12:35.791697270 (198h32m7.487546023s) [upload=174963]: Starting upload Nov 7 18:12:35.791791689 (198h32m7.487640452s) [upload=174963]: Done starting upload Nov 7 18:12:35.791885458 (198h32m7.487734211s) [upload=174963]: Uploading segments... Nov 7 18:12:35.792146455 (198h32m7.487995208s) [upload=174963]: Got next segment. Inline: false Nov 7 18:12:35.792195278 (198h32m7.488044021s) [upload=174963]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 7 18:12:35.792201820 (198h32m7.488050563s) [upload=174963]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 7 18:12:36.063867588 (198h32m7.759716341s) [seg_pos={0 0}, upload=174963]: Begin upload segment... Nov 7 18:12:36.063937832 (198h32m7.759786585s) [seg_pos={0 0}, upload=174963]: Done begin upload segment. Nov 7 18:12:36.063988689 (198h32m7.759837463s) [noise=true , address=145.79.227.27:4690 , piece_id=7CVNJTAEBVYJXRWNSFYFP3E7SKMY5XI4CV2JEZJLSXH4TA2F2JZQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.064083510 (198h32m7.759932263s) [noise=true , address=145.79.227.10:4403 , piece_id=JCMXNJ5PW5ULGOJZ5SGB423VW7VJPC5REYNSNWJZJXHGUQXUS3KA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.064896969 (198h32m7.760745762s) [noise=true , address=145.79.227.28:4726 , piece_id=D6DH5FHFB5ZTPANRPUWZKRYPFYUV2LS6AY5FQL5RBJZIUNM4FKLQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.065443520 (198h32m7.761292273s) [noise=true , address=77.90.31.11:4554 , piece_id=TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.065970974 (198h32m7.761819777s) [noise=true , address=145.79.227.8:4330 , piece_id=32XBCOEGSUTVUCCWVM2D7CB6FKMNLD6N2N4655B3ZYHNAIL4R2QQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.066422053 (198h32m7.762270806s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.067054427 (198h32m7.76290318s) [noise=true , address=77.90.31.10:4519 , piece_id=MIHLKGHCRTUFQSGBIFKIWPNACAJ4CQOEYRDIKA4KJLFP356VMY6A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.067645042 (198h32m7.763493795s) [noise=true , address=45.91.116.6:4973 , piece_id=Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.068379079 (198h32m7.764227842s) [noise=true , address=77.90.31.19:4843 , piece_id=JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.070931099 (198h32m7.766779862s) [noise=true , address=145.79.227.11:4732 , piece_id=VKSPYLTD6C6XBUFXXVVPHIPGFRCRNQK6EVI4SM7Y6ODOMVGGAJOQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.073100420 (198h32m7.768949173s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.075821872 (198h32m7.771670625s) [noise=true , address=45.91.116.4:4937 , piece_id=CWDA2IYNFUYHVDBJ74Y3CJKCZUBN4QYUEJYFKZNIBAUYYAXO3WFQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.078400863 (198h32m7.774249626s) [noise=true , address=45.91.116.10:4082 , piece_id=RW62GCPC56K2IMOFF276GQECXRKW6NY6W44TNAYFNDGQS3UK7EEA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.081230762 (198h32m7.777079515s) [noise=true , address=145.79.227.15:4876 , piece_id=ZPGKGNXRFEWRRJKTS7NKBEVQA7OAWNWYDR3DUTPHT3XVYRJSKWPQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.083298258 (198h32m7.779147012s) [noise=true , address=145.79.227.20:4438 , piece_id=V6WDUVDB7WHTDCTSYCFWS22UKRD5NMXGJ2GHLSFLVVOXZMZWLL4A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.084602091 (198h32m7.780450844s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:37.640211006 (198h32m9.336059759s) [upload=174963]: Next returned nil segment Nov 7 18:12:37.640227878 (198h32m9.336076621s) [upload=174963]: Waiting for error group managing segments... Nov 7 18:12:38.007509819 (198h32m9.703358572s) [noise=true , address=145.79.227.10:4403 , piece_id=JCMXNJ5PW5ULGOJZ5SGB423VW7VJPC5REYNSNWJZJXHGUQXUS3KA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.015182750 (198h32m9.711031503s) [noise=true , address=145.79.227.28:4726 , piece_id=D6DH5FHFB5ZTPANRPUWZKRYPFYUV2LS6AY5FQL5RBJZIUNM4FKLQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.020400886 (198h32m9.71624964s) [noise=true , address=145.79.227.27:4690 , piece_id=7CVNJTAEBVYJXRWNSFYFP3E7SKMY5XI4CV2JEZJLSXH4TA2F2JZQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.062361869 (198h32m9.758210622s) [noise=true , address=145.79.227.15:4876 , piece_id=ZPGKGNXRFEWRRJKTS7NKBEVQA7OAWNWYDR3DUTPHT3XVYRJSKWPQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.096654439 (198h32m9.792503192s) [noise=true , address=145.79.227.20:4438 , piece_id=V6WDUVDB7WHTDCTSYCFWS22UKRD5NMXGJ2GHLSFLVVOXZMZWLL4A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.244365682 (198h32m9.940214435s) [noise=true , address=145.79.227.8:4330 , piece_id=32XBCOEGSUTVUCCWVM2D7CB6FKMNLD6N2N4655B3ZYHNAIL4R2QQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.245114297 (198h32m9.94096305s) [noise=true , address=145.79.227.11:4732 , piece_id=VKSPYLTD6C6XBUFXXVVPHIPGFRCRNQK6EVI4SM7Y6ODOMVGGAJOQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.324356281 (198h32m10.020205034s) [noise=true , address=77.90.31.11:4554 , piece_id=TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.331696878 (198h32m10.027545621s) [noise=true , address=45.91.116.4:4937 , piece_id=CWDA2IYNFUYHVDBJ74Y3CJKCZUBN4QYUEJYFKZNIBAUYYAXO3WFQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.334142676 (198h32m10.029991429s) [noise=true , address=45.91.116.10:4082 , piece_id=RW62GCPC56K2IMOFF276GQECXRKW6NY6W44TNAYFNDGQS3UK7EEA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.346539031 (198h32m10.042387794s) [noise=true , address=77.90.31.10:4519 , piece_id=MIHLKGHCRTUFQSGBIFKIWPNACAJ4CQOEYRDIKA4KJLFP356VMY6A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.419504032 (198h32m10.115352785s) [noise=true , address=77.90.31.19:4843 , piece_id=JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:39.113150499 (198h32m10.808999252s) [noise=true , address=45.91.116.6:4973 , piece_id=Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:39.113177150 (198h32m10.809025903s) [seg_pos={0 0}, upload=174963]: Segment reached optimal threshold of 13 pieces. Nov 7 18:12:39.113381018 (198h32m10.809229772s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113390166 (198h32m10.809238919s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113460199 (198h32m10.809308952s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113470077 (198h32m10.809318821s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113780338 (198h32m10.809629091s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113788243 (198h32m10.809636986s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113858657 (198h32m10.8097074s) [seg_pos={0 0}, upload=174963]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 7 18:12:39.114030194 (198h32m10.809878947s) [upload=174963]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 7 18:12:39.114038290 (198h32m10.809887043s) [upload=174963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 7 18:12:39.451448640 (198h32m11.147297403s) [upload=174963]: Done uploading segments... Nov 7 18:12:39.451473607 (198h32m11.14732236s) [upload=174963]: Upload finished. err: <nil> Duration: 6.024440559s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.................................................X..........] duration: 3.562519488s trace: https://jaeger.dev.storj.io/trace/272952ae802c1916 FAIL (2025-11-07 18:48:04 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.562551228s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [XX.X..XX.XXXXX.X............................................] duration: 6.068428704s trace: https://jaeger.dev.storj.io/trace/422f33ac8ae6a74a FAIL (2025-11-07 18:47:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ) context deadline exceeded; piecestore: (Node ID: 12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, Piece ID: JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ) context deadline exceeded ======LOGS====== Nov 7 18:12:35.791697270 (198h32m7.487546023s) [upload=174963]: Starting upload Nov 7 18:12:35.791791689 (198h32m7.487640452s) [upload=174963]: Done starting upload Nov 7 18:12:35.791885458 (198h32m7.487734211s) [upload=174963]: Uploading segments... Nov 7 18:12:35.792146455 (198h32m7.487995208s) [upload=174963]: Got next segment. Inline: false Nov 7 18:12:35.792195278 (198h32m7.488044021s) [upload=174963]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 7 18:12:35.792201820 (198h32m7.488050563s) [upload=174963]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 7 18:12:36.063867588 (198h32m7.759716341s) [seg_pos={0 0}, upload=174963]: Begin upload segment... Nov 7 18:12:36.063937832 (198h32m7.759786585s) [seg_pos={0 0}, upload=174963]: Done begin upload segment. Nov 7 18:12:36.063988689 (198h32m7.759837463s) [noise=true , address=145.79.227.27:4690 , piece_id=7CVNJTAEBVYJXRWNSFYFP3E7SKMY5XI4CV2JEZJLSXH4TA2F2JZQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.064083510 (198h32m7.759932263s) [noise=true , address=145.79.227.10:4403 , piece_id=JCMXNJ5PW5ULGOJZ5SGB423VW7VJPC5REYNSNWJZJXHGUQXUS3KA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.064896969 (198h32m7.760745762s) [noise=true , address=145.79.227.28:4726 , piece_id=D6DH5FHFB5ZTPANRPUWZKRYPFYUV2LS6AY5FQL5RBJZIUNM4FKLQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.065443520 (198h32m7.761292273s) [noise=true , address=77.90.31.11:4554 , piece_id=TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.065970974 (198h32m7.761819777s) [noise=true , address=145.79.227.8:4330 , piece_id=32XBCOEGSUTVUCCWVM2D7CB6FKMNLD6N2N4655B3ZYHNAIL4R2QQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.066422053 (198h32m7.762270806s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.067054427 (198h32m7.76290318s) [noise=true , address=77.90.31.10:4519 , piece_id=MIHLKGHCRTUFQSGBIFKIWPNACAJ4CQOEYRDIKA4KJLFP356VMY6A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.067645042 (198h32m7.763493795s) [noise=true , address=45.91.116.6:4973 , piece_id=Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.068379079 (198h32m7.764227842s) [noise=true , address=77.90.31.19:4843 , piece_id=JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.070931099 (198h32m7.766779862s) [noise=true , address=145.79.227.11:4732 , piece_id=VKSPYLTD6C6XBUFXXVVPHIPGFRCRNQK6EVI4SM7Y6ODOMVGGAJOQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.073100420 (198h32m7.768949173s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.075821872 (198h32m7.771670625s) [noise=true , address=45.91.116.4:4937 , piece_id=CWDA2IYNFUYHVDBJ74Y3CJKCZUBN4QYUEJYFKZNIBAUYYAXO3WFQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.078400863 (198h32m7.774249626s) [noise=true , address=45.91.116.10:4082 , piece_id=RW62GCPC56K2IMOFF276GQECXRKW6NY6W44TNAYFNDGQS3UK7EEA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.081230762 (198h32m7.777079515s) [noise=true , address=145.79.227.15:4876 , piece_id=ZPGKGNXRFEWRRJKTS7NKBEVQA7OAWNWYDR3DUTPHT3XVYRJSKWPQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.083298258 (198h32m7.779147012s) [noise=true , address=145.79.227.20:4438 , piece_id=V6WDUVDB7WHTDCTSYCFWS22UKRD5NMXGJ2GHLSFLVVOXZMZWLL4A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.084602091 (198h32m7.780450844s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:37.640211006 (198h32m9.336059759s) [upload=174963]: Next returned nil segment Nov 7 18:12:37.640227878 (198h32m9.336076621s) [upload=174963]: Waiting for error group managing segments... Nov 7 18:12:38.007509819 (198h32m9.703358572s) [noise=true , address=145.79.227.10:4403 , piece_id=JCMXNJ5PW5ULGOJZ5SGB423VW7VJPC5REYNSNWJZJXHGUQXUS3KA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.015182750 (198h32m9.711031503s) [noise=true , address=145.79.227.28:4726 , piece_id=D6DH5FHFB5ZTPANRPUWZKRYPFYUV2LS6AY5FQL5RBJZIUNM4FKLQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.020400886 (198h32m9.71624964s) [noise=true , address=145.79.227.27:4690 , piece_id=7CVNJTAEBVYJXRWNSFYFP3E7SKMY5XI4CV2JEZJLSXH4TA2F2JZQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.062361869 (198h32m9.758210622s) [noise=true , address=145.79.227.15:4876 , piece_id=ZPGKGNXRFEWRRJKTS7NKBEVQA7OAWNWYDR3DUTPHT3XVYRJSKWPQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.096654439 (198h32m9.792503192s) [noise=true , address=145.79.227.20:4438 , piece_id=V6WDUVDB7WHTDCTSYCFWS22UKRD5NMXGJ2GHLSFLVVOXZMZWLL4A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.244365682 (198h32m9.940214435s) [noise=true , address=145.79.227.8:4330 , piece_id=32XBCOEGSUTVUCCWVM2D7CB6FKMNLD6N2N4655B3ZYHNAIL4R2QQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.245114297 (198h32m9.94096305s) [noise=true , address=145.79.227.11:4732 , piece_id=VKSPYLTD6C6XBUFXXVVPHIPGFRCRNQK6EVI4SM7Y6ODOMVGGAJOQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.324356281 (198h32m10.020205034s) [noise=true , address=77.90.31.11:4554 , piece_id=TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.331696878 (198h32m10.027545621s) [noise=true , address=45.91.116.4:4937 , piece_id=CWDA2IYNFUYHVDBJ74Y3CJKCZUBN4QYUEJYFKZNIBAUYYAXO3WFQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.334142676 (198h32m10.029991429s) [noise=true , address=45.91.116.10:4082 , piece_id=RW62GCPC56K2IMOFF276GQECXRKW6NY6W44TNAYFNDGQS3UK7EEA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.346539031 (198h32m10.042387794s) [noise=true , address=77.90.31.10:4519 , piece_id=MIHLKGHCRTUFQSGBIFKIWPNACAJ4CQOEYRDIKA4KJLFP356VMY6A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.419504032 (198h32m10.115352785s) [noise=true , address=77.90.31.19:4843 , piece_id=JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:39.113150499 (198h32m10.808999252s) [noise=true , address=45.91.116.6:4973 , piece_id=Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:39.113177150 (198h32m10.809025903s) [seg_pos={0 0}, upload=174963]: Segment reached optimal threshold of 13 pieces. Nov 7 18:12:39.113381018 (198h32m10.809229772s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113390166 (198h32m10.809238919s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113460199 (198h32m10.809308952s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113470077 (198h32m10.809318821s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113780338 (198h32m10.809629091s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113788243 (198h32m10.809636986s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113858657 (198h32m10.8097074s) [seg_pos={0 0}, upload=174963]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 7 18:12:39.114030194 (198h32m10.809878947s) [upload=174963]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 7 18:12:39.114038290 (198h32m10.809887043s) [upload=174963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 7 18:12:39.451448640 (198h32m11.147297403s) [upload=174963]: Done uploading segments... Nov 7 18:12:39.451473607 (198h32m11.14732236s) [upload=174963]: Upload finished. err: <nil> Duration: 6.068465645s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..................................................X.........] duration: 3.53747548s trace: https://jaeger.dev.storj.io/trace/272952ae802c1916 FAIL (2025-11-07 18:47:12 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.537529171s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...................................................X........] duration: 3.403428532s trace: https://jaeger.dev.storj.io/trace/272952ae802c1916 FAIL (2025-11-07 18:46:07 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.403507222s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [XXX.X..XX.XXXXX.X...........................................] duration: 6.147313886s trace: https://jaeger.dev.storj.io/trace/422f33ac8ae6a74a FAIL (2025-11-07 18:46:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ) context deadline exceeded; piecestore: (Node ID: 12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, Piece ID: JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ) context deadline exceeded ======LOGS====== Nov 7 18:12:35.791697270 (198h32m7.487546023s) [upload=174963]: Starting upload Nov 7 18:12:35.791791689 (198h32m7.487640452s) [upload=174963]: Done starting upload Nov 7 18:12:35.791885458 (198h32m7.487734211s) [upload=174963]: Uploading segments... Nov 7 18:12:35.792146455 (198h32m7.487995208s) [upload=174963]: Got next segment. Inline: false Nov 7 18:12:35.792195278 (198h32m7.488044021s) [upload=174963]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 7 18:12:35.792201820 (198h32m7.488050563s) [upload=174963]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 7 18:12:36.063867588 (198h32m7.759716341s) [seg_pos={0 0}, upload=174963]: Begin upload segment... Nov 7 18:12:36.063937832 (198h32m7.759786585s) [seg_pos={0 0}, upload=174963]: Done begin upload segment. Nov 7 18:12:36.063988689 (198h32m7.759837463s) [noise=true , address=145.79.227.27:4690 , piece_id=7CVNJTAEBVYJXRWNSFYFP3E7SKMY5XI4CV2JEZJLSXH4TA2F2JZQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.064083510 (198h32m7.759932263s) [noise=true , address=145.79.227.10:4403 , piece_id=JCMXNJ5PW5ULGOJZ5SGB423VW7VJPC5REYNSNWJZJXHGUQXUS3KA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.064896969 (198h32m7.760745762s) [noise=true , address=145.79.227.28:4726 , piece_id=D6DH5FHFB5ZTPANRPUWZKRYPFYUV2LS6AY5FQL5RBJZIUNM4FKLQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.065443520 (198h32m7.761292273s) [noise=true , address=77.90.31.11:4554 , piece_id=TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.065970974 (198h32m7.761819777s) [noise=true , address=145.79.227.8:4330 , piece_id=32XBCOEGSUTVUCCWVM2D7CB6FKMNLD6N2N4655B3ZYHNAIL4R2QQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.066422053 (198h32m7.762270806s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.067054427 (198h32m7.76290318s) [noise=true , address=77.90.31.10:4519 , piece_id=MIHLKGHCRTUFQSGBIFKIWPNACAJ4CQOEYRDIKA4KJLFP356VMY6A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.067645042 (198h32m7.763493795s) [noise=true , address=45.91.116.6:4973 , piece_id=Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.068379079 (198h32m7.764227842s) [noise=true , address=77.90.31.19:4843 , piece_id=JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.070931099 (198h32m7.766779862s) [noise=true , address=145.79.227.11:4732 , piece_id=VKSPYLTD6C6XBUFXXVVPHIPGFRCRNQK6EVI4SM7Y6ODOMVGGAJOQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.073100420 (198h32m7.768949173s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.075821872 (198h32m7.771670625s) [noise=true , address=45.91.116.4:4937 , piece_id=CWDA2IYNFUYHVDBJ74Y3CJKCZUBN4QYUEJYFKZNIBAUYYAXO3WFQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.078400863 (198h32m7.774249626s) [noise=true , address=45.91.116.10:4082 , piece_id=RW62GCPC56K2IMOFF276GQECXRKW6NY6W44TNAYFNDGQS3UK7EEA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.081230762 (198h32m7.777079515s) [noise=true , address=145.79.227.15:4876 , piece_id=ZPGKGNXRFEWRRJKTS7NKBEVQA7OAWNWYDR3DUTPHT3XVYRJSKWPQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.083298258 (198h32m7.779147012s) [noise=true , address=145.79.227.20:4438 , piece_id=V6WDUVDB7WHTDCTSYCFWS22UKRD5NMXGJ2GHLSFLVVOXZMZWLL4A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.084602091 (198h32m7.780450844s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:37.640211006 (198h32m9.336059759s) [upload=174963]: Next returned nil segment Nov 7 18:12:37.640227878 (198h32m9.336076621s) [upload=174963]: Waiting for error group managing segments... Nov 7 18:12:38.007509819 (198h32m9.703358572s) [noise=true , address=145.79.227.10:4403 , piece_id=JCMXNJ5PW5ULGOJZ5SGB423VW7VJPC5REYNSNWJZJXHGUQXUS3KA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.015182750 (198h32m9.711031503s) [noise=true , address=145.79.227.28:4726 , piece_id=D6DH5FHFB5ZTPANRPUWZKRYPFYUV2LS6AY5FQL5RBJZIUNM4FKLQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.020400886 (198h32m9.71624964s) [noise=true , address=145.79.227.27:4690 , piece_id=7CVNJTAEBVYJXRWNSFYFP3E7SKMY5XI4CV2JEZJLSXH4TA2F2JZQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.062361869 (198h32m9.758210622s) [noise=true , address=145.79.227.15:4876 , piece_id=ZPGKGNXRFEWRRJKTS7NKBEVQA7OAWNWYDR3DUTPHT3XVYRJSKWPQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.096654439 (198h32m9.792503192s) [noise=true , address=145.79.227.20:4438 , piece_id=V6WDUVDB7WHTDCTSYCFWS22UKRD5NMXGJ2GHLSFLVVOXZMZWLL4A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.244365682 (198h32m9.940214435s) [noise=true , address=145.79.227.8:4330 , piece_id=32XBCOEGSUTVUCCWVM2D7CB6FKMNLD6N2N4655B3ZYHNAIL4R2QQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.245114297 (198h32m9.94096305s) [noise=true , address=145.79.227.11:4732 , piece_id=VKSPYLTD6C6XBUFXXVVPHIPGFRCRNQK6EVI4SM7Y6ODOMVGGAJOQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.324356281 (198h32m10.020205034s) [noise=true , address=77.90.31.11:4554 , piece_id=TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.331696878 (198h32m10.027545621s) [noise=true , address=45.91.116.4:4937 , piece_id=CWDA2IYNFUYHVDBJ74Y3CJKCZUBN4QYUEJYFKZNIBAUYYAXO3WFQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.334142676 (198h32m10.029991429s) [noise=true , address=45.91.116.10:4082 , piece_id=RW62GCPC56K2IMOFF276GQECXRKW6NY6W44TNAYFNDGQS3UK7EEA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.346539031 (198h32m10.042387794s) [noise=true , address=77.90.31.10:4519 , piece_id=MIHLKGHCRTUFQSGBIFKIWPNACAJ4CQOEYRDIKA4KJLFP356VMY6A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.419504032 (198h32m10.115352785s) [noise=true , address=77.90.31.19:4843 , piece_id=JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:39.113150499 (198h32m10.808999252s) [noise=true , address=45.91.116.6:4973 , piece_id=Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:39.113177150 (198h32m10.809025903s) [seg_pos={0 0}, upload=174963]: Segment reached optimal threshold of 13 pieces. Nov 7 18:12:39.113381018 (198h32m10.809229772s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113390166 (198h32m10.809238919s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113460199 (198h32m10.809308952s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113470077 (198h32m10.809318821s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113780338 (198h32m10.809629091s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113788243 (198h32m10.809636986s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113858657 (198h32m10.8097074s) [seg_pos={0 0}, upload=174963]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 7 18:12:39.114030194 (198h32m10.809878947s) [upload=174963]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 7 18:12:39.114038290 (198h32m10.809887043s) [upload=174963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 7 18:12:39.451448640 (198h32m11.147297403s) [upload=174963]: Done uploading segments... Nov 7 18:12:39.451473607 (198h32m11.14732236s) [upload=174963]: Upload finished. err: <nil> Duration: 6.147348552s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....................................................X.......] duration: 3.536274263s trace: https://jaeger.dev.storj.io/trace/272952ae802c1916 FAIL (2025-11-07 18:45:33 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.536306936s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [XXXX.X..XX.XXXXX.X..........................................] duration: 6.041194412s trace: https://jaeger.dev.storj.io/trace/422f33ac8ae6a74a FAIL (2025-11-07 18:44:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ) context deadline exceeded; piecestore: (Node ID: 12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, Piece ID: JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ) context deadline exceeded ======LOGS====== Nov 7 18:12:35.791697270 (198h32m7.487546023s) [upload=174963]: Starting upload Nov 7 18:12:35.791791689 (198h32m7.487640452s) [upload=174963]: Done starting upload Nov 7 18:12:35.791885458 (198h32m7.487734211s) [upload=174963]: Uploading segments... Nov 7 18:12:35.792146455 (198h32m7.487995208s) [upload=174963]: Got next segment. Inline: false Nov 7 18:12:35.792195278 (198h32m7.488044021s) [upload=174963]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 7 18:12:35.792201820 (198h32m7.488050563s) [upload=174963]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 7 18:12:36.063867588 (198h32m7.759716341s) [seg_pos={0 0}, upload=174963]: Begin upload segment... Nov 7 18:12:36.063937832 (198h32m7.759786585s) [seg_pos={0 0}, upload=174963]: Done begin upload segment. Nov 7 18:12:36.063988689 (198h32m7.759837463s) [noise=true , address=145.79.227.27:4690 , piece_id=7CVNJTAEBVYJXRWNSFYFP3E7SKMY5XI4CV2JEZJLSXH4TA2F2JZQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.064083510 (198h32m7.759932263s) [noise=true , address=145.79.227.10:4403 , piece_id=JCMXNJ5PW5ULGOJZ5SGB423VW7VJPC5REYNSNWJZJXHGUQXUS3KA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.064896969 (198h32m7.760745762s) [noise=true , address=145.79.227.28:4726 , piece_id=D6DH5FHFB5ZTPANRPUWZKRYPFYUV2LS6AY5FQL5RBJZIUNM4FKLQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.065443520 (198h32m7.761292273s) [noise=true , address=77.90.31.11:4554 , piece_id=TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.065970974 (198h32m7.761819777s) [noise=true , address=145.79.227.8:4330 , piece_id=32XBCOEGSUTVUCCWVM2D7CB6FKMNLD6N2N4655B3ZYHNAIL4R2QQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.066422053 (198h32m7.762270806s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.067054427 (198h32m7.76290318s) [noise=true , address=77.90.31.10:4519 , piece_id=MIHLKGHCRTUFQSGBIFKIWPNACAJ4CQOEYRDIKA4KJLFP356VMY6A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.067645042 (198h32m7.763493795s) [noise=true , address=45.91.116.6:4973 , piece_id=Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.068379079 (198h32m7.764227842s) [noise=true , address=77.90.31.19:4843 , piece_id=JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.070931099 (198h32m7.766779862s) [noise=true , address=145.79.227.11:4732 , piece_id=VKSPYLTD6C6XBUFXXVVPHIPGFRCRNQK6EVI4SM7Y6ODOMVGGAJOQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.073100420 (198h32m7.768949173s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.075821872 (198h32m7.771670625s) [noise=true , address=45.91.116.4:4937 , piece_id=CWDA2IYNFUYHVDBJ74Y3CJKCZUBN4QYUEJYFKZNIBAUYYAXO3WFQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.078400863 (198h32m7.774249626s) [noise=true , address=45.91.116.10:4082 , piece_id=RW62GCPC56K2IMOFF276GQECXRKW6NY6W44TNAYFNDGQS3UK7EEA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.081230762 (198h32m7.777079515s) [noise=true , address=145.79.227.15:4876 , piece_id=ZPGKGNXRFEWRRJKTS7NKBEVQA7OAWNWYDR3DUTPHT3XVYRJSKWPQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.083298258 (198h32m7.779147012s) [noise=true , address=145.79.227.20:4438 , piece_id=V6WDUVDB7WHTDCTSYCFWS22UKRD5NMXGJ2GHLSFLVVOXZMZWLL4A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.084602091 (198h32m7.780450844s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:37.640211006 (198h32m9.336059759s) [upload=174963]: Next returned nil segment Nov 7 18:12:37.640227878 (198h32m9.336076621s) [upload=174963]: Waiting for error group managing segments... Nov 7 18:12:38.007509819 (198h32m9.703358572s) [noise=true , address=145.79.227.10:4403 , piece_id=JCMXNJ5PW5ULGOJZ5SGB423VW7VJPC5REYNSNWJZJXHGUQXUS3KA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.015182750 (198h32m9.711031503s) [noise=true , address=145.79.227.28:4726 , piece_id=D6DH5FHFB5ZTPANRPUWZKRYPFYUV2LS6AY5FQL5RBJZIUNM4FKLQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.020400886 (198h32m9.71624964s) [noise=true , address=145.79.227.27:4690 , piece_id=7CVNJTAEBVYJXRWNSFYFP3E7SKMY5XI4CV2JEZJLSXH4TA2F2JZQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.062361869 (198h32m9.758210622s) [noise=true , address=145.79.227.15:4876 , piece_id=ZPGKGNXRFEWRRJKTS7NKBEVQA7OAWNWYDR3DUTPHT3XVYRJSKWPQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.096654439 (198h32m9.792503192s) [noise=true , address=145.79.227.20:4438 , piece_id=V6WDUVDB7WHTDCTSYCFWS22UKRD5NMXGJ2GHLSFLVVOXZMZWLL4A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.244365682 (198h32m9.940214435s) [noise=true , address=145.79.227.8:4330 , piece_id=32XBCOEGSUTVUCCWVM2D7CB6FKMNLD6N2N4655B3ZYHNAIL4R2QQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.245114297 (198h32m9.94096305s) [noise=true , address=145.79.227.11:4732 , piece_id=VKSPYLTD6C6XBUFXXVVPHIPGFRCRNQK6EVI4SM7Y6ODOMVGGAJOQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.324356281 (198h32m10.020205034s) [noise=true , address=77.90.31.11:4554 , piece_id=TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.331696878 (198h32m10.027545621s) [noise=true , address=45.91.116.4:4937 , piece_id=CWDA2IYNFUYHVDBJ74Y3CJKCZUBN4QYUEJYFKZNIBAUYYAXO3WFQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.334142676 (198h32m10.029991429s) [noise=true , address=45.91.116.10:4082 , piece_id=RW62GCPC56K2IMOFF276GQECXRKW6NY6W44TNAYFNDGQS3UK7EEA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.346539031 (198h32m10.042387794s) [noise=true , address=77.90.31.10:4519 , piece_id=MIHLKGHCRTUFQSGBIFKIWPNACAJ4CQOEYRDIKA4KJLFP356VMY6A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.419504032 (198h32m10.115352785s) [noise=true , address=77.90.31.19:4843 , piece_id=JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:39.113150499 (198h32m10.808999252s) [noise=true , address=45.91.116.6:4973 , piece_id=Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:39.113177150 (198h32m10.809025903s) [seg_pos={0 0}, upload=174963]: Segment reached optimal threshold of 13 pieces. Nov 7 18:12:39.113381018 (198h32m10.809229772s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113390166 (198h32m10.809238919s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113460199 (198h32m10.809308952s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113470077 (198h32m10.809318821s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113780338 (198h32m10.809629091s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113788243 (198h32m10.809636986s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113858657 (198h32m10.8097074s) [seg_pos={0 0}, upload=174963]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 7 18:12:39.114030194 (198h32m10.809878947s) [upload=174963]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 7 18:12:39.114038290 (198h32m10.809887043s) [upload=174963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 7 18:12:39.451448640 (198h32m11.147297403s) [upload=174963]: Done uploading segments... Nov 7 18:12:39.451473607 (198h32m11.14732236s) [upload=174963]: Upload finished. err: <nil> Duration: 6.041221525s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....................................................X......] duration: 3.486535575s trace: https://jaeger.dev.storj.io/trace/272952ae802c1916 FAIL (2025-11-07 18:44:35 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.486566343s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.XXXX.X..XX.XXXXX.X.........................................] duration: 5.9542646s trace: https://jaeger.dev.storj.io/trace/422f33ac8ae6a74a FAIL (2025-11-07 18:44:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ) context deadline exceeded; piecestore: (Node ID: 12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, Piece ID: JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ) context deadline exceeded ======LOGS====== Nov 7 18:12:35.791697270 (198h32m7.487546023s) [upload=174963]: Starting upload Nov 7 18:12:35.791791689 (198h32m7.487640452s) [upload=174963]: Done starting upload Nov 7 18:12:35.791885458 (198h32m7.487734211s) [upload=174963]: Uploading segments... Nov 7 18:12:35.792146455 (198h32m7.487995208s) [upload=174963]: Got next segment. Inline: false Nov 7 18:12:35.792195278 (198h32m7.488044021s) [upload=174963]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 7 18:12:35.792201820 (198h32m7.488050563s) [upload=174963]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 7 18:12:36.063867588 (198h32m7.759716341s) [seg_pos={0 0}, upload=174963]: Begin upload segment... Nov 7 18:12:36.063937832 (198h32m7.759786585s) [seg_pos={0 0}, upload=174963]: Done begin upload segment. Nov 7 18:12:36.063988689 (198h32m7.759837463s) [noise=true , address=145.79.227.27:4690 , piece_id=7CVNJTAEBVYJXRWNSFYFP3E7SKMY5XI4CV2JEZJLSXH4TA2F2JZQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.064083510 (198h32m7.759932263s) [noise=true , address=145.79.227.10:4403 , piece_id=JCMXNJ5PW5ULGOJZ5SGB423VW7VJPC5REYNSNWJZJXHGUQXUS3KA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.064896969 (198h32m7.760745762s) [noise=true , address=145.79.227.28:4726 , piece_id=D6DH5FHFB5ZTPANRPUWZKRYPFYUV2LS6AY5FQL5RBJZIUNM4FKLQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.065443520 (198h32m7.761292273s) [noise=true , address=77.90.31.11:4554 , piece_id=TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.065970974 (198h32m7.761819777s) [noise=true , address=145.79.227.8:4330 , piece_id=32XBCOEGSUTVUCCWVM2D7CB6FKMNLD6N2N4655B3ZYHNAIL4R2QQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.066422053 (198h32m7.762270806s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.067054427 (198h32m7.76290318s) [noise=true , address=77.90.31.10:4519 , piece_id=MIHLKGHCRTUFQSGBIFKIWPNACAJ4CQOEYRDIKA4KJLFP356VMY6A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.067645042 (198h32m7.763493795s) [noise=true , address=45.91.116.6:4973 , piece_id=Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.068379079 (198h32m7.764227842s) [noise=true , address=77.90.31.19:4843 , piece_id=JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.070931099 (198h32m7.766779862s) [noise=true , address=145.79.227.11:4732 , piece_id=VKSPYLTD6C6XBUFXXVVPHIPGFRCRNQK6EVI4SM7Y6ODOMVGGAJOQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.073100420 (198h32m7.768949173s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.075821872 (198h32m7.771670625s) [noise=true , address=45.91.116.4:4937 , piece_id=CWDA2IYNFUYHVDBJ74Y3CJKCZUBN4QYUEJYFKZNIBAUYYAXO3WFQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.078400863 (198h32m7.774249626s) [noise=true , address=45.91.116.10:4082 , piece_id=RW62GCPC56K2IMOFF276GQECXRKW6NY6W44TNAYFNDGQS3UK7EEA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.081230762 (198h32m7.777079515s) [noise=true , address=145.79.227.15:4876 , piece_id=ZPGKGNXRFEWRRJKTS7NKBEVQA7OAWNWYDR3DUTPHT3XVYRJSKWPQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.083298258 (198h32m7.779147012s) [noise=true , address=145.79.227.20:4438 , piece_id=V6WDUVDB7WHTDCTSYCFWS22UKRD5NMXGJ2GHLSFLVVOXZMZWLL4A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.084602091 (198h32m7.780450844s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:37.640211006 (198h32m9.336059759s) [upload=174963]: Next returned nil segment Nov 7 18:12:37.640227878 (198h32m9.336076621s) [upload=174963]: Waiting for error group managing segments... Nov 7 18:12:38.007509819 (198h32m9.703358572s) [noise=true , address=145.79.227.10:4403 , piece_id=JCMXNJ5PW5ULGOJZ5SGB423VW7VJPC5REYNSNWJZJXHGUQXUS3KA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.015182750 (198h32m9.711031503s) [noise=true , address=145.79.227.28:4726 , piece_id=D6DH5FHFB5ZTPANRPUWZKRYPFYUV2LS6AY5FQL5RBJZIUNM4FKLQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.020400886 (198h32m9.71624964s) [noise=true , address=145.79.227.27:4690 , piece_id=7CVNJTAEBVYJXRWNSFYFP3E7SKMY5XI4CV2JEZJLSXH4TA2F2JZQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.062361869 (198h32m9.758210622s) [noise=true , address=145.79.227.15:4876 , piece_id=ZPGKGNXRFEWRRJKTS7NKBEVQA7OAWNWYDR3DUTPHT3XVYRJSKWPQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.096654439 (198h32m9.792503192s) [noise=true , address=145.79.227.20:4438 , piece_id=V6WDUVDB7WHTDCTSYCFWS22UKRD5NMXGJ2GHLSFLVVOXZMZWLL4A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.244365682 (198h32m9.940214435s) [noise=true , address=145.79.227.8:4330 , piece_id=32XBCOEGSUTVUCCWVM2D7CB6FKMNLD6N2N4655B3ZYHNAIL4R2QQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.245114297 (198h32m9.94096305s) [noise=true , address=145.79.227.11:4732 , piece_id=VKSPYLTD6C6XBUFXXVVPHIPGFRCRNQK6EVI4SM7Y6ODOMVGGAJOQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.324356281 (198h32m10.020205034s) [noise=true , address=77.90.31.11:4554 , piece_id=TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.331696878 (198h32m10.027545621s) [noise=true , address=45.91.116.4:4937 , piece_id=CWDA2IYNFUYHVDBJ74Y3CJKCZUBN4QYUEJYFKZNIBAUYYAXO3WFQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.334142676 (198h32m10.029991429s) [noise=true , address=45.91.116.10:4082 , piece_id=RW62GCPC56K2IMOFF276GQECXRKW6NY6W44TNAYFNDGQS3UK7EEA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.346539031 (198h32m10.042387794s) [noise=true , address=77.90.31.10:4519 , piece_id=MIHLKGHCRTUFQSGBIFKIWPNACAJ4CQOEYRDIKA4KJLFP356VMY6A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.419504032 (198h32m10.115352785s) [noise=true , address=77.90.31.19:4843 , piece_id=JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:39.113150499 (198h32m10.808999252s) [noise=true , address=45.91.116.6:4973 , piece_id=Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:39.113177150 (198h32m10.809025903s) [seg_pos={0 0}, upload=174963]: Segment reached optimal threshold of 13 pieces. Nov 7 18:12:39.113381018 (198h32m10.809229772s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113390166 (198h32m10.809238919s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113460199 (198h32m10.809308952s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113470077 (198h32m10.809318821s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113780338 (198h32m10.809629091s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113788243 (198h32m10.809636986s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113858657 (198h32m10.8097074s) [seg_pos={0 0}, upload=174963]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 7 18:12:39.114030194 (198h32m10.809878947s) [upload=174963]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 7 18:12:39.114038290 (198h32m10.809887043s) [upload=174963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 7 18:12:39.451448640 (198h32m11.147297403s) [upload=174963]: Done uploading segments... Nov 7 18:12:39.451473607 (198h32m11.14732236s) [upload=174963]: Upload finished. err: <nil> Duration: 5.954320376s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......................................................X.....] duration: 3.749419232s trace: https://jaeger.dev.storj.io/trace/272952ae802c1916 FAIL (2025-11-07 18:43:38 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.749471351s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [X.XXXX.X..XX.XXXXX.X........................................] duration: 5.947022805s trace: https://jaeger.dev.storj.io/trace/422f33ac8ae6a74a FAIL (2025-11-07 18:43:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ) context deadline exceeded; piecestore: (Node ID: 12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, Piece ID: JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ) context deadline exceeded ======LOGS====== Nov 7 18:12:35.791697270 (198h32m7.487546023s) [upload=174963]: Starting upload Nov 7 18:12:35.791791689 (198h32m7.487640452s) [upload=174963]: Done starting upload Nov 7 18:12:35.791885458 (198h32m7.487734211s) [upload=174963]: Uploading segments... Nov 7 18:12:35.792146455 (198h32m7.487995208s) [upload=174963]: Got next segment. Inline: false Nov 7 18:12:35.792195278 (198h32m7.488044021s) [upload=174963]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 7 18:12:35.792201820 (198h32m7.488050563s) [upload=174963]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 7 18:12:36.063867588 (198h32m7.759716341s) [seg_pos={0 0}, upload=174963]: Begin upload segment... Nov 7 18:12:36.063937832 (198h32m7.759786585s) [seg_pos={0 0}, upload=174963]: Done begin upload segment. Nov 7 18:12:36.063988689 (198h32m7.759837463s) [noise=true , address=145.79.227.27:4690 , piece_id=7CVNJTAEBVYJXRWNSFYFP3E7SKMY5XI4CV2JEZJLSXH4TA2F2JZQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.064083510 (198h32m7.759932263s) [noise=true , address=145.79.227.10:4403 , piece_id=JCMXNJ5PW5ULGOJZ5SGB423VW7VJPC5REYNSNWJZJXHGUQXUS3KA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.064896969 (198h32m7.760745762s) [noise=true , address=145.79.227.28:4726 , piece_id=D6DH5FHFB5ZTPANRPUWZKRYPFYUV2LS6AY5FQL5RBJZIUNM4FKLQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.065443520 (198h32m7.761292273s) [noise=true , address=77.90.31.11:4554 , piece_id=TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.065970974 (198h32m7.761819777s) [noise=true , address=145.79.227.8:4330 , piece_id=32XBCOEGSUTVUCCWVM2D7CB6FKMNLD6N2N4655B3ZYHNAIL4R2QQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.066422053 (198h32m7.762270806s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.067054427 (198h32m7.76290318s) [noise=true , address=77.90.31.10:4519 , piece_id=MIHLKGHCRTUFQSGBIFKIWPNACAJ4CQOEYRDIKA4KJLFP356VMY6A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.067645042 (198h32m7.763493795s) [noise=true , address=45.91.116.6:4973 , piece_id=Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.068379079 (198h32m7.764227842s) [noise=true , address=77.90.31.19:4843 , piece_id=JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.070931099 (198h32m7.766779862s) [noise=true , address=145.79.227.11:4732 , piece_id=VKSPYLTD6C6XBUFXXVVPHIPGFRCRNQK6EVI4SM7Y6ODOMVGGAJOQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.073100420 (198h32m7.768949173s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.075821872 (198h32m7.771670625s) [noise=true , address=45.91.116.4:4937 , piece_id=CWDA2IYNFUYHVDBJ74Y3CJKCZUBN4QYUEJYFKZNIBAUYYAXO3WFQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.078400863 (198h32m7.774249626s) [noise=true , address=45.91.116.10:4082 , piece_id=RW62GCPC56K2IMOFF276GQECXRKW6NY6W44TNAYFNDGQS3UK7EEA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.081230762 (198h32m7.777079515s) [noise=true , address=145.79.227.15:4876 , piece_id=ZPGKGNXRFEWRRJKTS7NKBEVQA7OAWNWYDR3DUTPHT3XVYRJSKWPQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.083298258 (198h32m7.779147012s) [noise=true , address=145.79.227.20:4438 , piece_id=V6WDUVDB7WHTDCTSYCFWS22UKRD5NMXGJ2GHLSFLVVOXZMZWLL4A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.084602091 (198h32m7.780450844s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:37.640211006 (198h32m9.336059759s) [upload=174963]: Next returned nil segment Nov 7 18:12:37.640227878 (198h32m9.336076621s) [upload=174963]: Waiting for error group managing segments... Nov 7 18:12:38.007509819 (198h32m9.703358572s) [noise=true , address=145.79.227.10:4403 , piece_id=JCMXNJ5PW5ULGOJZ5SGB423VW7VJPC5REYNSNWJZJXHGUQXUS3KA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.015182750 (198h32m9.711031503s) [noise=true , address=145.79.227.28:4726 , piece_id=D6DH5FHFB5ZTPANRPUWZKRYPFYUV2LS6AY5FQL5RBJZIUNM4FKLQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.020400886 (198h32m9.71624964s) [noise=true , address=145.79.227.27:4690 , piece_id=7CVNJTAEBVYJXRWNSFYFP3E7SKMY5XI4CV2JEZJLSXH4TA2F2JZQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.062361869 (198h32m9.758210622s) [noise=true , address=145.79.227.15:4876 , piece_id=ZPGKGNXRFEWRRJKTS7NKBEVQA7OAWNWYDR3DUTPHT3XVYRJSKWPQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.096654439 (198h32m9.792503192s) [noise=true , address=145.79.227.20:4438 , piece_id=V6WDUVDB7WHTDCTSYCFWS22UKRD5NMXGJ2GHLSFLVVOXZMZWLL4A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.244365682 (198h32m9.940214435s) [noise=true , address=145.79.227.8:4330 , piece_id=32XBCOEGSUTVUCCWVM2D7CB6FKMNLD6N2N4655B3ZYHNAIL4R2QQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.245114297 (198h32m9.94096305s) [noise=true , address=145.79.227.11:4732 , piece_id=VKSPYLTD6C6XBUFXXVVPHIPGFRCRNQK6EVI4SM7Y6ODOMVGGAJOQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.324356281 (198h32m10.020205034s) [noise=true , address=77.90.31.11:4554 , piece_id=TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.331696878 (198h32m10.027545621s) [noise=true , address=45.91.116.4:4937 , piece_id=CWDA2IYNFUYHVDBJ74Y3CJKCZUBN4QYUEJYFKZNIBAUYYAXO3WFQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.334142676 (198h32m10.029991429s) [noise=true , address=45.91.116.10:4082 , piece_id=RW62GCPC56K2IMOFF276GQECXRKW6NY6W44TNAYFNDGQS3UK7EEA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.346539031 (198h32m10.042387794s) [noise=true , address=77.90.31.10:4519 , piece_id=MIHLKGHCRTUFQSGBIFKIWPNACAJ4CQOEYRDIKA4KJLFP356VMY6A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.419504032 (198h32m10.115352785s) [noise=true , address=77.90.31.19:4843 , piece_id=JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:39.113150499 (198h32m10.808999252s) [noise=true , address=45.91.116.6:4973 , piece_id=Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:39.113177150 (198h32m10.809025903s) [seg_pos={0 0}, upload=174963]: Segment reached optimal threshold of 13 pieces. Nov 7 18:12:39.113381018 (198h32m10.809229772s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113390166 (198h32m10.809238919s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113460199 (198h32m10.809308952s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113470077 (198h32m10.809318821s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113780338 (198h32m10.809629091s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113788243 (198h32m10.809636986s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113858657 (198h32m10.8097074s) [seg_pos={0 0}, upload=174963]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 7 18:12:39.114030194 (198h32m10.809878947s) [upload=174963]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 7 18:12:39.114038290 (198h32m10.809887043s) [upload=174963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 7 18:12:39.451448640 (198h32m11.147297403s) [upload=174963]: Done uploading segments... Nov 7 18:12:39.451473607 (198h32m11.14732236s) [upload=174963]: Upload finished. err: <nil> Duration: 5.947040949s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......................................................X....] duration: 3.520089741s trace: https://jaeger.dev.storj.io/trace/272952ae802c1916 FAIL (2025-11-07 18:42:34 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.520185783s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [.X.XXXX.X..XX.XXXXX.X.......................................] duration: 6.021445294s trace: https://jaeger.dev.storj.io/trace/422f33ac8ae6a74a FAIL (2025-11-07 18:42:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ) context deadline exceeded; piecestore: (Node ID: 12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, Piece ID: JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ) context deadline exceeded ======LOGS====== Nov 7 18:12:35.791697270 (198h32m7.487546023s) [upload=174963]: Starting upload Nov 7 18:12:35.791791689 (198h32m7.487640452s) [upload=174963]: Done starting upload Nov 7 18:12:35.791885458 (198h32m7.487734211s) [upload=174963]: Uploading segments... Nov 7 18:12:35.792146455 (198h32m7.487995208s) [upload=174963]: Got next segment. Inline: false Nov 7 18:12:35.792195278 (198h32m7.488044021s) [upload=174963]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 7 18:12:35.792201820 (198h32m7.488050563s) [upload=174963]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 7 18:12:36.063867588 (198h32m7.759716341s) [seg_pos={0 0}, upload=174963]: Begin upload segment... Nov 7 18:12:36.063937832 (198h32m7.759786585s) [seg_pos={0 0}, upload=174963]: Done begin upload segment. Nov 7 18:12:36.063988689 (198h32m7.759837463s) [noise=true , address=145.79.227.27:4690 , piece_id=7CVNJTAEBVYJXRWNSFYFP3E7SKMY5XI4CV2JEZJLSXH4TA2F2JZQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.064083510 (198h32m7.759932263s) [noise=true , address=145.79.227.10:4403 , piece_id=JCMXNJ5PW5ULGOJZ5SGB423VW7VJPC5REYNSNWJZJXHGUQXUS3KA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.064896969 (198h32m7.760745762s) [noise=true , address=145.79.227.28:4726 , piece_id=D6DH5FHFB5ZTPANRPUWZKRYPFYUV2LS6AY5FQL5RBJZIUNM4FKLQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.065443520 (198h32m7.761292273s) [noise=true , address=77.90.31.11:4554 , piece_id=TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.065970974 (198h32m7.761819777s) [noise=true , address=145.79.227.8:4330 , piece_id=32XBCOEGSUTVUCCWVM2D7CB6FKMNLD6N2N4655B3ZYHNAIL4R2QQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.066422053 (198h32m7.762270806s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.067054427 (198h32m7.76290318s) [noise=true , address=77.90.31.10:4519 , piece_id=MIHLKGHCRTUFQSGBIFKIWPNACAJ4CQOEYRDIKA4KJLFP356VMY6A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.067645042 (198h32m7.763493795s) [noise=true , address=45.91.116.6:4973 , piece_id=Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.068379079 (198h32m7.764227842s) [noise=true , address=77.90.31.19:4843 , piece_id=JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.070931099 (198h32m7.766779862s) [noise=true , address=145.79.227.11:4732 , piece_id=VKSPYLTD6C6XBUFXXVVPHIPGFRCRNQK6EVI4SM7Y6ODOMVGGAJOQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.073100420 (198h32m7.768949173s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.075821872 (198h32m7.771670625s) [noise=true , address=45.91.116.4:4937 , piece_id=CWDA2IYNFUYHVDBJ74Y3CJKCZUBN4QYUEJYFKZNIBAUYYAXO3WFQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.078400863 (198h32m7.774249626s) [noise=true , address=45.91.116.10:4082 , piece_id=RW62GCPC56K2IMOFF276GQECXRKW6NY6W44TNAYFNDGQS3UK7EEA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.081230762 (198h32m7.777079515s) [noise=true , address=145.79.227.15:4876 , piece_id=ZPGKGNXRFEWRRJKTS7NKBEVQA7OAWNWYDR3DUTPHT3XVYRJSKWPQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.083298258 (198h32m7.779147012s) [noise=true , address=145.79.227.20:4438 , piece_id=V6WDUVDB7WHTDCTSYCFWS22UKRD5NMXGJ2GHLSFLVVOXZMZWLL4A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.084602091 (198h32m7.780450844s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:37.640211006 (198h32m9.336059759s) [upload=174963]: Next returned nil segment Nov 7 18:12:37.640227878 (198h32m9.336076621s) [upload=174963]: Waiting for error group managing segments... Nov 7 18:12:38.007509819 (198h32m9.703358572s) [noise=true , address=145.79.227.10:4403 , piece_id=JCMXNJ5PW5ULGOJZ5SGB423VW7VJPC5REYNSNWJZJXHGUQXUS3KA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.015182750 (198h32m9.711031503s) [noise=true , address=145.79.227.28:4726 , piece_id=D6DH5FHFB5ZTPANRPUWZKRYPFYUV2LS6AY5FQL5RBJZIUNM4FKLQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.020400886 (198h32m9.71624964s) [noise=true , address=145.79.227.27:4690 , piece_id=7CVNJTAEBVYJXRWNSFYFP3E7SKMY5XI4CV2JEZJLSXH4TA2F2JZQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.062361869 (198h32m9.758210622s) [noise=true , address=145.79.227.15:4876 , piece_id=ZPGKGNXRFEWRRJKTS7NKBEVQA7OAWNWYDR3DUTPHT3XVYRJSKWPQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.096654439 (198h32m9.792503192s) [noise=true , address=145.79.227.20:4438 , piece_id=V6WDUVDB7WHTDCTSYCFWS22UKRD5NMXGJ2GHLSFLVVOXZMZWLL4A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.244365682 (198h32m9.940214435s) [noise=true , address=145.79.227.8:4330 , piece_id=32XBCOEGSUTVUCCWVM2D7CB6FKMNLD6N2N4655B3ZYHNAIL4R2QQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.245114297 (198h32m9.94096305s) [noise=true , address=145.79.227.11:4732 , piece_id=VKSPYLTD6C6XBUFXXVVPHIPGFRCRNQK6EVI4SM7Y6ODOMVGGAJOQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.324356281 (198h32m10.020205034s) [noise=true , address=77.90.31.11:4554 , piece_id=TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.331696878 (198h32m10.027545621s) [noise=true , address=45.91.116.4:4937 , piece_id=CWDA2IYNFUYHVDBJ74Y3CJKCZUBN4QYUEJYFKZNIBAUYYAXO3WFQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.334142676 (198h32m10.029991429s) [noise=true , address=45.91.116.10:4082 , piece_id=RW62GCPC56K2IMOFF276GQECXRKW6NY6W44TNAYFNDGQS3UK7EEA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.346539031 (198h32m10.042387794s) [noise=true , address=77.90.31.10:4519 , piece_id=MIHLKGHCRTUFQSGBIFKIWPNACAJ4CQOEYRDIKA4KJLFP356VMY6A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.419504032 (198h32m10.115352785s) [noise=true , address=77.90.31.19:4843 , piece_id=JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:39.113150499 (198h32m10.808999252s) [noise=true , address=45.91.116.6:4973 , piece_id=Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:39.113177150 (198h32m10.809025903s) [seg_pos={0 0}, upload=174963]: Segment reached optimal threshold of 13 pieces. Nov 7 18:12:39.113381018 (198h32m10.809229772s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113390166 (198h32m10.809238919s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113460199 (198h32m10.809308952s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113470077 (198h32m10.809318821s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113780338 (198h32m10.809629091s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113788243 (198h32m10.809636986s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113858657 (198h32m10.8097074s) [seg_pos={0 0}, upload=174963]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 7 18:12:39.114030194 (198h32m10.809878947s) [upload=174963]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 7 18:12:39.114038290 (198h32m10.809887043s) [upload=174963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 7 18:12:39.451448640 (198h32m11.147297403s) [upload=174963]: Done uploading segments... Nov 7 18:12:39.451473607 (198h32m11.14732236s) [upload=174963]: Upload finished. err: <nil> Duration: 6.021478038s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........................................................X...] duration: 3.460601714s trace: https://jaeger.dev.storj.io/trace/272952ae802c1916 FAIL (2025-11-07 18:41:51 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.460629387s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [X.X.XXXX.X..XX.XXXXX.X......................................] duration: 6.746767442s trace: https://jaeger.dev.storj.io/trace/422f33ac8ae6a74a FAIL (2025-11-07 18:41:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ) context deadline exceeded; piecestore: (Node ID: 12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, Piece ID: JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ) context deadline exceeded ======LOGS====== Nov 7 18:12:35.791697270 (198h32m7.487546023s) [upload=174963]: Starting upload Nov 7 18:12:35.791791689 (198h32m7.487640452s) [upload=174963]: Done starting upload Nov 7 18:12:35.791885458 (198h32m7.487734211s) [upload=174963]: Uploading segments... Nov 7 18:12:35.792146455 (198h32m7.487995208s) [upload=174963]: Got next segment. Inline: false Nov 7 18:12:35.792195278 (198h32m7.488044021s) [upload=174963]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 7 18:12:35.792201820 (198h32m7.488050563s) [upload=174963]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 7 18:12:36.063867588 (198h32m7.759716341s) [seg_pos={0 0}, upload=174963]: Begin upload segment... Nov 7 18:12:36.063937832 (198h32m7.759786585s) [seg_pos={0 0}, upload=174963]: Done begin upload segment. Nov 7 18:12:36.063988689 (198h32m7.759837463s) [noise=true , address=145.79.227.27:4690 , piece_id=7CVNJTAEBVYJXRWNSFYFP3E7SKMY5XI4CV2JEZJLSXH4TA2F2JZQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.064083510 (198h32m7.759932263s) [noise=true , address=145.79.227.10:4403 , piece_id=JCMXNJ5PW5ULGOJZ5SGB423VW7VJPC5REYNSNWJZJXHGUQXUS3KA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.064896969 (198h32m7.760745762s) [noise=true , address=145.79.227.28:4726 , piece_id=D6DH5FHFB5ZTPANRPUWZKRYPFYUV2LS6AY5FQL5RBJZIUNM4FKLQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.065443520 (198h32m7.761292273s) [noise=true , address=77.90.31.11:4554 , piece_id=TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.065970974 (198h32m7.761819777s) [noise=true , address=145.79.227.8:4330 , piece_id=32XBCOEGSUTVUCCWVM2D7CB6FKMNLD6N2N4655B3ZYHNAIL4R2QQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.066422053 (198h32m7.762270806s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.067054427 (198h32m7.76290318s) [noise=true , address=77.90.31.10:4519 , piece_id=MIHLKGHCRTUFQSGBIFKIWPNACAJ4CQOEYRDIKA4KJLFP356VMY6A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.067645042 (198h32m7.763493795s) [noise=true , address=45.91.116.6:4973 , piece_id=Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.068379079 (198h32m7.764227842s) [noise=true , address=77.90.31.19:4843 , piece_id=JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.070931099 (198h32m7.766779862s) [noise=true , address=145.79.227.11:4732 , piece_id=VKSPYLTD6C6XBUFXXVVPHIPGFRCRNQK6EVI4SM7Y6ODOMVGGAJOQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.073100420 (198h32m7.768949173s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.075821872 (198h32m7.771670625s) [noise=true , address=45.91.116.4:4937 , piece_id=CWDA2IYNFUYHVDBJ74Y3CJKCZUBN4QYUEJYFKZNIBAUYYAXO3WFQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.078400863 (198h32m7.774249626s) [noise=true , address=45.91.116.10:4082 , piece_id=RW62GCPC56K2IMOFF276GQECXRKW6NY6W44TNAYFNDGQS3UK7EEA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.081230762 (198h32m7.777079515s) [noise=true , address=145.79.227.15:4876 , piece_id=ZPGKGNXRFEWRRJKTS7NKBEVQA7OAWNWYDR3DUTPHT3XVYRJSKWPQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.083298258 (198h32m7.779147012s) [noise=true , address=145.79.227.20:4438 , piece_id=V6WDUVDB7WHTDCTSYCFWS22UKRD5NMXGJ2GHLSFLVVOXZMZWLL4A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.084602091 (198h32m7.780450844s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:37.640211006 (198h32m9.336059759s) [upload=174963]: Next returned nil segment Nov 7 18:12:37.640227878 (198h32m9.336076621s) [upload=174963]: Waiting for error group managing segments... Nov 7 18:12:38.007509819 (198h32m9.703358572s) [noise=true , address=145.79.227.10:4403 , piece_id=JCMXNJ5PW5ULGOJZ5SGB423VW7VJPC5REYNSNWJZJXHGUQXUS3KA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.015182750 (198h32m9.711031503s) [noise=true , address=145.79.227.28:4726 , piece_id=D6DH5FHFB5ZTPANRPUWZKRYPFYUV2LS6AY5FQL5RBJZIUNM4FKLQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.020400886 (198h32m9.71624964s) [noise=true , address=145.79.227.27:4690 , piece_id=7CVNJTAEBVYJXRWNSFYFP3E7SKMY5XI4CV2JEZJLSXH4TA2F2JZQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.062361869 (198h32m9.758210622s) [noise=true , address=145.79.227.15:4876 , piece_id=ZPGKGNXRFEWRRJKTS7NKBEVQA7OAWNWYDR3DUTPHT3XVYRJSKWPQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.096654439 (198h32m9.792503192s) [noise=true , address=145.79.227.20:4438 , piece_id=V6WDUVDB7WHTDCTSYCFWS22UKRD5NMXGJ2GHLSFLVVOXZMZWLL4A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.244365682 (198h32m9.940214435s) [noise=true , address=145.79.227.8:4330 , piece_id=32XBCOEGSUTVUCCWVM2D7CB6FKMNLD6N2N4655B3ZYHNAIL4R2QQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.245114297 (198h32m9.94096305s) [noise=true , address=145.79.227.11:4732 , piece_id=VKSPYLTD6C6XBUFXXVVPHIPGFRCRNQK6EVI4SM7Y6ODOMVGGAJOQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.324356281 (198h32m10.020205034s) [noise=true , address=77.90.31.11:4554 , piece_id=TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.331696878 (198h32m10.027545621s) [noise=true , address=45.91.116.4:4937 , piece_id=CWDA2IYNFUYHVDBJ74Y3CJKCZUBN4QYUEJYFKZNIBAUYYAXO3WFQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.334142676 (198h32m10.029991429s) [noise=true , address=45.91.116.10:4082 , piece_id=RW62GCPC56K2IMOFF276GQECXRKW6NY6W44TNAYFNDGQS3UK7EEA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.346539031 (198h32m10.042387794s) [noise=true , address=77.90.31.10:4519 , piece_id=MIHLKGHCRTUFQSGBIFKIWPNACAJ4CQOEYRDIKA4KJLFP356VMY6A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.419504032 (198h32m10.115352785s) [noise=true , address=77.90.31.19:4843 , piece_id=JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:39.113150499 (198h32m10.808999252s) [noise=true , address=45.91.116.6:4973 , piece_id=Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:39.113177150 (198h32m10.809025903s) [seg_pos={0 0}, upload=174963]: Segment reached optimal threshold of 13 pieces. Nov 7 18:12:39.113381018 (198h32m10.809229772s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113390166 (198h32m10.809238919s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113460199 (198h32m10.809308952s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113470077 (198h32m10.809318821s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113780338 (198h32m10.809629091s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113788243 (198h32m10.809636986s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113858657 (198h32m10.8097074s) [seg_pos={0 0}, upload=174963]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 7 18:12:39.114030194 (198h32m10.809878947s) [upload=174963]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 7 18:12:39.114038290 (198h32m10.809887043s) [upload=174963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 7 18:12:39.451448640 (198h32m11.147297403s) [upload=174963]: Done uploading segments... Nov 7 18:12:39.451473607 (198h32m11.14732236s) [upload=174963]: Upload finished. err: <nil> Duration: 6.746793242s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.X.X.XXXX.X..XX.XXXXX.X.....................................] duration: 6.159643523s trace: https://jaeger.dev.storj.io/trace/422f33ac8ae6a74a FAIL (2025-11-07 18:40:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ) context deadline exceeded; piecestore: (Node ID: 12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, Piece ID: JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ) context deadline exceeded ======LOGS====== Nov 7 18:12:35.791697270 (198h32m7.487546023s) [upload=174963]: Starting upload Nov 7 18:12:35.791791689 (198h32m7.487640452s) [upload=174963]: Done starting upload Nov 7 18:12:35.791885458 (198h32m7.487734211s) [upload=174963]: Uploading segments... Nov 7 18:12:35.792146455 (198h32m7.487995208s) [upload=174963]: Got next segment. Inline: false Nov 7 18:12:35.792195278 (198h32m7.488044021s) [upload=174963]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 7 18:12:35.792201820 (198h32m7.488050563s) [upload=174963]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 7 18:12:36.063867588 (198h32m7.759716341s) [seg_pos={0 0}, upload=174963]: Begin upload segment... Nov 7 18:12:36.063937832 (198h32m7.759786585s) [seg_pos={0 0}, upload=174963]: Done begin upload segment. Nov 7 18:12:36.063988689 (198h32m7.759837463s) [noise=true , address=145.79.227.27:4690 , piece_id=7CVNJTAEBVYJXRWNSFYFP3E7SKMY5XI4CV2JEZJLSXH4TA2F2JZQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.064083510 (198h32m7.759932263s) [noise=true , address=145.79.227.10:4403 , piece_id=JCMXNJ5PW5ULGOJZ5SGB423VW7VJPC5REYNSNWJZJXHGUQXUS3KA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.064896969 (198h32m7.760745762s) [noise=true , address=145.79.227.28:4726 , piece_id=D6DH5FHFB5ZTPANRPUWZKRYPFYUV2LS6AY5FQL5RBJZIUNM4FKLQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.065443520 (198h32m7.761292273s) [noise=true , address=77.90.31.11:4554 , piece_id=TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.065970974 (198h32m7.761819777s) [noise=true , address=145.79.227.8:4330 , piece_id=32XBCOEGSUTVUCCWVM2D7CB6FKMNLD6N2N4655B3ZYHNAIL4R2QQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.066422053 (198h32m7.762270806s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.067054427 (198h32m7.76290318s) [noise=true , address=77.90.31.10:4519 , piece_id=MIHLKGHCRTUFQSGBIFKIWPNACAJ4CQOEYRDIKA4KJLFP356VMY6A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.067645042 (198h32m7.763493795s) [noise=true , address=45.91.116.6:4973 , piece_id=Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.068379079 (198h32m7.764227842s) [noise=true , address=77.90.31.19:4843 , piece_id=JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.070931099 (198h32m7.766779862s) [noise=true , address=145.79.227.11:4732 , piece_id=VKSPYLTD6C6XBUFXXVVPHIPGFRCRNQK6EVI4SM7Y6ODOMVGGAJOQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.073100420 (198h32m7.768949173s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.075821872 (198h32m7.771670625s) [noise=true , address=45.91.116.4:4937 , piece_id=CWDA2IYNFUYHVDBJ74Y3CJKCZUBN4QYUEJYFKZNIBAUYYAXO3WFQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.078400863 (198h32m7.774249626s) [noise=true , address=45.91.116.10:4082 , piece_id=RW62GCPC56K2IMOFF276GQECXRKW6NY6W44TNAYFNDGQS3UK7EEA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.081230762 (198h32m7.777079515s) [noise=true , address=145.79.227.15:4876 , piece_id=ZPGKGNXRFEWRRJKTS7NKBEVQA7OAWNWYDR3DUTPHT3XVYRJSKWPQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.083298258 (198h32m7.779147012s) [noise=true , address=145.79.227.20:4438 , piece_id=V6WDUVDB7WHTDCTSYCFWS22UKRD5NMXGJ2GHLSFLVVOXZMZWLL4A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.084602091 (198h32m7.780450844s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:37.640211006 (198h32m9.336059759s) [upload=174963]: Next returned nil segment Nov 7 18:12:37.640227878 (198h32m9.336076621s) [upload=174963]: Waiting for error group managing segments... Nov 7 18:12:38.007509819 (198h32m9.703358572s) [noise=true , address=145.79.227.10:4403 , piece_id=JCMXNJ5PW5ULGOJZ5SGB423VW7VJPC5REYNSNWJZJXHGUQXUS3KA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.015182750 (198h32m9.711031503s) [noise=true , address=145.79.227.28:4726 , piece_id=D6DH5FHFB5ZTPANRPUWZKRYPFYUV2LS6AY5FQL5RBJZIUNM4FKLQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.020400886 (198h32m9.71624964s) [noise=true , address=145.79.227.27:4690 , piece_id=7CVNJTAEBVYJXRWNSFYFP3E7SKMY5XI4CV2JEZJLSXH4TA2F2JZQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.062361869 (198h32m9.758210622s) [noise=true , address=145.79.227.15:4876 , piece_id=ZPGKGNXRFEWRRJKTS7NKBEVQA7OAWNWYDR3DUTPHT3XVYRJSKWPQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.096654439 (198h32m9.792503192s) [noise=true , address=145.79.227.20:4438 , piece_id=V6WDUVDB7WHTDCTSYCFWS22UKRD5NMXGJ2GHLSFLVVOXZMZWLL4A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.244365682 (198h32m9.940214435s) [noise=true , address=145.79.227.8:4330 , piece_id=32XBCOEGSUTVUCCWVM2D7CB6FKMNLD6N2N4655B3ZYHNAIL4R2QQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.245114297 (198h32m9.94096305s) [noise=true , address=145.79.227.11:4732 , piece_id=VKSPYLTD6C6XBUFXXVVPHIPGFRCRNQK6EVI4SM7Y6ODOMVGGAJOQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.324356281 (198h32m10.020205034s) [noise=true , address=77.90.31.11:4554 , piece_id=TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.331696878 (198h32m10.027545621s) [noise=true , address=45.91.116.4:4937 , piece_id=CWDA2IYNFUYHVDBJ74Y3CJKCZUBN4QYUEJYFKZNIBAUYYAXO3WFQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.334142676 (198h32m10.029991429s) [noise=true , address=45.91.116.10:4082 , piece_id=RW62GCPC56K2IMOFF276GQECXRKW6NY6W44TNAYFNDGQS3UK7EEA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.346539031 (198h32m10.042387794s) [noise=true , address=77.90.31.10:4519 , piece_id=MIHLKGHCRTUFQSGBIFKIWPNACAJ4CQOEYRDIKA4KJLFP356VMY6A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.419504032 (198h32m10.115352785s) [noise=true , address=77.90.31.19:4843 , piece_id=JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:39.113150499 (198h32m10.808999252s) [noise=true , address=45.91.116.6:4973 , piece_id=Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:39.113177150 (198h32m10.809025903s) [seg_pos={0 0}, upload=174963]: Segment reached optimal threshold of 13 pieces. Nov 7 18:12:39.113381018 (198h32m10.809229772s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113390166 (198h32m10.809238919s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113460199 (198h32m10.809308952s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113470077 (198h32m10.809318821s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113780338 (198h32m10.809629091s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113788243 (198h32m10.809636986s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113858657 (198h32m10.8097074s) [seg_pos={0 0}, upload=174963]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 7 18:12:39.114030194 (198h32m10.809878947s) [upload=174963]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 7 18:12:39.114038290 (198h32m10.809887043s) [upload=174963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 7 18:12:39.451448640 (198h32m11.147297403s) [upload=174963]: Done uploading segments... Nov 7 18:12:39.451473607 (198h32m11.14732236s) [upload=174963]: Upload finished. err: <nil> Duration: 6.159676416s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........................................................X..] duration: 3.494833351s trace: https://jaeger.dev.storj.io/trace/272952ae802c1916 FAIL (2025-11-07 18:40:56 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.494860433s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........................................................X.] duration: 3.43421055s trace: https://jaeger.dev.storj.io/trace/272952ae802c1916 FAIL (2025-11-07 18:40:20 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.434248492s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [..X.X.XXXX.X..XX.XXXXX.X....................................] duration: 7.344146931s trace: https://jaeger.dev.storj.io/trace/422f33ac8ae6a74a FAIL (2025-11-07 18:39:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ) context deadline exceeded; piecestore: (Node ID: 12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, Piece ID: JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ) context deadline exceeded ======LOGS====== Nov 7 18:12:35.791697270 (198h32m7.487546023s) [upload=174963]: Starting upload Nov 7 18:12:35.791791689 (198h32m7.487640452s) [upload=174963]: Done starting upload Nov 7 18:12:35.791885458 (198h32m7.487734211s) [upload=174963]: Uploading segments... Nov 7 18:12:35.792146455 (198h32m7.487995208s) [upload=174963]: Got next segment. Inline: false Nov 7 18:12:35.792195278 (198h32m7.488044021s) [upload=174963]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 7 18:12:35.792201820 (198h32m7.488050563s) [upload=174963]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 7 18:12:36.063867588 (198h32m7.759716341s) [seg_pos={0 0}, upload=174963]: Begin upload segment... Nov 7 18:12:36.063937832 (198h32m7.759786585s) [seg_pos={0 0}, upload=174963]: Done begin upload segment. Nov 7 18:12:36.063988689 (198h32m7.759837463s) [noise=true , address=145.79.227.27:4690 , piece_id=7CVNJTAEBVYJXRWNSFYFP3E7SKMY5XI4CV2JEZJLSXH4TA2F2JZQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.064083510 (198h32m7.759932263s) [noise=true , address=145.79.227.10:4403 , piece_id=JCMXNJ5PW5ULGOJZ5SGB423VW7VJPC5REYNSNWJZJXHGUQXUS3KA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.064896969 (198h32m7.760745762s) [noise=true , address=145.79.227.28:4726 , piece_id=D6DH5FHFB5ZTPANRPUWZKRYPFYUV2LS6AY5FQL5RBJZIUNM4FKLQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.065443520 (198h32m7.761292273s) [noise=true , address=77.90.31.11:4554 , piece_id=TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.065970974 (198h32m7.761819777s) [noise=true , address=145.79.227.8:4330 , piece_id=32XBCOEGSUTVUCCWVM2D7CB6FKMNLD6N2N4655B3ZYHNAIL4R2QQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.066422053 (198h32m7.762270806s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.067054427 (198h32m7.76290318s) [noise=true , address=77.90.31.10:4519 , piece_id=MIHLKGHCRTUFQSGBIFKIWPNACAJ4CQOEYRDIKA4KJLFP356VMY6A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.067645042 (198h32m7.763493795s) [noise=true , address=45.91.116.6:4973 , piece_id=Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.068379079 (198h32m7.764227842s) [noise=true , address=77.90.31.19:4843 , piece_id=JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.070931099 (198h32m7.766779862s) [noise=true , address=145.79.227.11:4732 , piece_id=VKSPYLTD6C6XBUFXXVVPHIPGFRCRNQK6EVI4SM7Y6ODOMVGGAJOQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.073100420 (198h32m7.768949173s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.075821872 (198h32m7.771670625s) [noise=true , address=45.91.116.4:4937 , piece_id=CWDA2IYNFUYHVDBJ74Y3CJKCZUBN4QYUEJYFKZNIBAUYYAXO3WFQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.078400863 (198h32m7.774249626s) [noise=true , address=45.91.116.10:4082 , piece_id=RW62GCPC56K2IMOFF276GQECXRKW6NY6W44TNAYFNDGQS3UK7EEA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.081230762 (198h32m7.777079515s) [noise=true , address=145.79.227.15:4876 , piece_id=ZPGKGNXRFEWRRJKTS7NKBEVQA7OAWNWYDR3DUTPHT3XVYRJSKWPQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.083298258 (198h32m7.779147012s) [noise=true , address=145.79.227.20:4438 , piece_id=V6WDUVDB7WHTDCTSYCFWS22UKRD5NMXGJ2GHLSFLVVOXZMZWLL4A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.084602091 (198h32m7.780450844s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:37.640211006 (198h32m9.336059759s) [upload=174963]: Next returned nil segment Nov 7 18:12:37.640227878 (198h32m9.336076621s) [upload=174963]: Waiting for error group managing segments... Nov 7 18:12:38.007509819 (198h32m9.703358572s) [noise=true , address=145.79.227.10:4403 , piece_id=JCMXNJ5PW5ULGOJZ5SGB423VW7VJPC5REYNSNWJZJXHGUQXUS3KA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.015182750 (198h32m9.711031503s) [noise=true , address=145.79.227.28:4726 , piece_id=D6DH5FHFB5ZTPANRPUWZKRYPFYUV2LS6AY5FQL5RBJZIUNM4FKLQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.020400886 (198h32m9.71624964s) [noise=true , address=145.79.227.27:4690 , piece_id=7CVNJTAEBVYJXRWNSFYFP3E7SKMY5XI4CV2JEZJLSXH4TA2F2JZQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.062361869 (198h32m9.758210622s) [noise=true , address=145.79.227.15:4876 , piece_id=ZPGKGNXRFEWRRJKTS7NKBEVQA7OAWNWYDR3DUTPHT3XVYRJSKWPQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.096654439 (198h32m9.792503192s) [noise=true , address=145.79.227.20:4438 , piece_id=V6WDUVDB7WHTDCTSYCFWS22UKRD5NMXGJ2GHLSFLVVOXZMZWLL4A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.244365682 (198h32m9.940214435s) [noise=true , address=145.79.227.8:4330 , piece_id=32XBCOEGSUTVUCCWVM2D7CB6FKMNLD6N2N4655B3ZYHNAIL4R2QQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.245114297 (198h32m9.94096305s) [noise=true , address=145.79.227.11:4732 , piece_id=VKSPYLTD6C6XBUFXXVVPHIPGFRCRNQK6EVI4SM7Y6ODOMVGGAJOQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.324356281 (198h32m10.020205034s) [noise=true , address=77.90.31.11:4554 , piece_id=TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.331696878 (198h32m10.027545621s) [noise=true , address=45.91.116.4:4937 , piece_id=CWDA2IYNFUYHVDBJ74Y3CJKCZUBN4QYUEJYFKZNIBAUYYAXO3WFQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.334142676 (198h32m10.029991429s) [noise=true , address=45.91.116.10:4082 , piece_id=RW62GCPC56K2IMOFF276GQECXRKW6NY6W44TNAYFNDGQS3UK7EEA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.346539031 (198h32m10.042387794s) [noise=true , address=77.90.31.10:4519 , piece_id=MIHLKGHCRTUFQSGBIFKIWPNACAJ4CQOEYRDIKA4KJLFP356VMY6A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.419504032 (198h32m10.115352785s) [noise=true , address=77.90.31.19:4843 , piece_id=JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:39.113150499 (198h32m10.808999252s) [noise=true , address=45.91.116.6:4973 , piece_id=Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:39.113177150 (198h32m10.809025903s) [seg_pos={0 0}, upload=174963]: Segment reached optimal threshold of 13 pieces. Nov 7 18:12:39.113381018 (198h32m10.809229772s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113390166 (198h32m10.809238919s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113460199 (198h32m10.809308952s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113470077 (198h32m10.809318821s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113780338 (198h32m10.809629091s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113788243 (198h32m10.809636986s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113858657 (198h32m10.8097074s) [seg_pos={0 0}, upload=174963]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 7 18:12:39.114030194 (198h32m10.809878947s) [upload=174963]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 7 18:12:39.114038290 (198h32m10.809887043s) [upload=174963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 7 18:12:39.451448640 (198h32m11.147297403s) [upload=174963]: Done uploading segments... Nov 7 18:12:39.451473607 (198h32m11.14732236s) [upload=174963]: Upload finished. err: <nil> Duration: 7.344174403s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........................................................X] duration: 5.001511825s trace: https://jaeger.dev.storj.io/trace/272952ae802c1916 FAIL (2025-11-07 18:39:20 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001534659s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [X..X.X.XXXX.X..XX.XXXXX.X...................................] duration: 6.595121006s trace: https://jaeger.dev.storj.io/trace/422f33ac8ae6a74a FAIL (2025-11-07 18:39:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ) context deadline exceeded; piecestore: (Node ID: 12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, Piece ID: JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ) context deadline exceeded ======LOGS====== Nov 7 18:12:35.791697270 (198h32m7.487546023s) [upload=174963]: Starting upload Nov 7 18:12:35.791791689 (198h32m7.487640452s) [upload=174963]: Done starting upload Nov 7 18:12:35.791885458 (198h32m7.487734211s) [upload=174963]: Uploading segments... Nov 7 18:12:35.792146455 (198h32m7.487995208s) [upload=174963]: Got next segment. Inline: false Nov 7 18:12:35.792195278 (198h32m7.488044021s) [upload=174963]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 7 18:12:35.792201820 (198h32m7.488050563s) [upload=174963]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 7 18:12:36.063867588 (198h32m7.759716341s) [seg_pos={0 0}, upload=174963]: Begin upload segment... Nov 7 18:12:36.063937832 (198h32m7.759786585s) [seg_pos={0 0}, upload=174963]: Done begin upload segment. Nov 7 18:12:36.063988689 (198h32m7.759837463s) [noise=true , address=145.79.227.27:4690 , piece_id=7CVNJTAEBVYJXRWNSFYFP3E7SKMY5XI4CV2JEZJLSXH4TA2F2JZQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.064083510 (198h32m7.759932263s) [noise=true , address=145.79.227.10:4403 , piece_id=JCMXNJ5PW5ULGOJZ5SGB423VW7VJPC5REYNSNWJZJXHGUQXUS3KA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.064896969 (198h32m7.760745762s) [noise=true , address=145.79.227.28:4726 , piece_id=D6DH5FHFB5ZTPANRPUWZKRYPFYUV2LS6AY5FQL5RBJZIUNM4FKLQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.065443520 (198h32m7.761292273s) [noise=true , address=77.90.31.11:4554 , piece_id=TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.065970974 (198h32m7.761819777s) [noise=true , address=145.79.227.8:4330 , piece_id=32XBCOEGSUTVUCCWVM2D7CB6FKMNLD6N2N4655B3ZYHNAIL4R2QQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.066422053 (198h32m7.762270806s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.067054427 (198h32m7.76290318s) [noise=true , address=77.90.31.10:4519 , piece_id=MIHLKGHCRTUFQSGBIFKIWPNACAJ4CQOEYRDIKA4KJLFP356VMY6A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.067645042 (198h32m7.763493795s) [noise=true , address=45.91.116.6:4973 , piece_id=Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.068379079 (198h32m7.764227842s) [noise=true , address=77.90.31.19:4843 , piece_id=JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.070931099 (198h32m7.766779862s) [noise=true , address=145.79.227.11:4732 , piece_id=VKSPYLTD6C6XBUFXXVVPHIPGFRCRNQK6EVI4SM7Y6ODOMVGGAJOQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.073100420 (198h32m7.768949173s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.075821872 (198h32m7.771670625s) [noise=true , address=45.91.116.4:4937 , piece_id=CWDA2IYNFUYHVDBJ74Y3CJKCZUBN4QYUEJYFKZNIBAUYYAXO3WFQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.078400863 (198h32m7.774249626s) [noise=true , address=45.91.116.10:4082 , piece_id=RW62GCPC56K2IMOFF276GQECXRKW6NY6W44TNAYFNDGQS3UK7EEA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.081230762 (198h32m7.777079515s) [noise=true , address=145.79.227.15:4876 , piece_id=ZPGKGNXRFEWRRJKTS7NKBEVQA7OAWNWYDR3DUTPHT3XVYRJSKWPQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.083298258 (198h32m7.779147012s) [noise=true , address=145.79.227.20:4438 , piece_id=V6WDUVDB7WHTDCTSYCFWS22UKRD5NMXGJ2GHLSFLVVOXZMZWLL4A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.084602091 (198h32m7.780450844s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:37.640211006 (198h32m9.336059759s) [upload=174963]: Next returned nil segment Nov 7 18:12:37.640227878 (198h32m9.336076621s) [upload=174963]: Waiting for error group managing segments... Nov 7 18:12:38.007509819 (198h32m9.703358572s) [noise=true , address=145.79.227.10:4403 , piece_id=JCMXNJ5PW5ULGOJZ5SGB423VW7VJPC5REYNSNWJZJXHGUQXUS3KA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.015182750 (198h32m9.711031503s) [noise=true , address=145.79.227.28:4726 , piece_id=D6DH5FHFB5ZTPANRPUWZKRYPFYUV2LS6AY5FQL5RBJZIUNM4FKLQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.020400886 (198h32m9.71624964s) [noise=true , address=145.79.227.27:4690 , piece_id=7CVNJTAEBVYJXRWNSFYFP3E7SKMY5XI4CV2JEZJLSXH4TA2F2JZQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.062361869 (198h32m9.758210622s) [noise=true , address=145.79.227.15:4876 , piece_id=ZPGKGNXRFEWRRJKTS7NKBEVQA7OAWNWYDR3DUTPHT3XVYRJSKWPQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.096654439 (198h32m9.792503192s) [noise=true , address=145.79.227.20:4438 , piece_id=V6WDUVDB7WHTDCTSYCFWS22UKRD5NMXGJ2GHLSFLVVOXZMZWLL4A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.244365682 (198h32m9.940214435s) [noise=true , address=145.79.227.8:4330 , piece_id=32XBCOEGSUTVUCCWVM2D7CB6FKMNLD6N2N4655B3ZYHNAIL4R2QQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.245114297 (198h32m9.94096305s) [noise=true , address=145.79.227.11:4732 , piece_id=VKSPYLTD6C6XBUFXXVVPHIPGFRCRNQK6EVI4SM7Y6ODOMVGGAJOQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.324356281 (198h32m10.020205034s) [noise=true , address=77.90.31.11:4554 , piece_id=TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.331696878 (198h32m10.027545621s) [noise=true , address=45.91.116.4:4937 , piece_id=CWDA2IYNFUYHVDBJ74Y3CJKCZUBN4QYUEJYFKZNIBAUYYAXO3WFQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.334142676 (198h32m10.029991429s) [noise=true , address=45.91.116.10:4082 , piece_id=RW62GCPC56K2IMOFF276GQECXRKW6NY6W44TNAYFNDGQS3UK7EEA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.346539031 (198h32m10.042387794s) [noise=true , address=77.90.31.10:4519 , piece_id=MIHLKGHCRTUFQSGBIFKIWPNACAJ4CQOEYRDIKA4KJLFP356VMY6A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.419504032 (198h32m10.115352785s) [noise=true , address=77.90.31.19:4843 , piece_id=JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:39.113150499 (198h32m10.808999252s) [noise=true , address=45.91.116.6:4973 , piece_id=Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:39.113177150 (198h32m10.809025903s) [seg_pos={0 0}, upload=174963]: Segment reached optimal threshold of 13 pieces. Nov 7 18:12:39.113381018 (198h32m10.809229772s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113390166 (198h32m10.809238919s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113460199 (198h32m10.809308952s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113470077 (198h32m10.809318821s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113780338 (198h32m10.809629091s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113788243 (198h32m10.809636986s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113858657 (198h32m10.8097074s) [seg_pos={0 0}, upload=174963]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 7 18:12:39.114030194 (198h32m10.809878947s) [upload=174963]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 7 18:12:39.114038290 (198h32m10.809887043s) [upload=174963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 7 18:12:39.451448640 (198h32m11.147297403s) [upload=174963]: Done uploading segments... Nov 7 18:12:39.451473607 (198h32m11.14732236s) [upload=174963]: Upload finished. err: <nil> Duration: 6.595155911s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [.X..X.X.XXXX.X..XX.XXXXX.X..................................] duration: 6.081714588s trace: https://jaeger.dev.storj.io/trace/422f33ac8ae6a74a FAIL (2025-11-07 18:38:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ) context deadline exceeded; piecestore: (Node ID: 12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, Piece ID: JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ) context deadline exceeded ======LOGS====== Nov 7 18:12:35.791697270 (198h32m7.487546023s) [upload=174963]: Starting upload Nov 7 18:12:35.791791689 (198h32m7.487640452s) [upload=174963]: Done starting upload Nov 7 18:12:35.791885458 (198h32m7.487734211s) [upload=174963]: Uploading segments... Nov 7 18:12:35.792146455 (198h32m7.487995208s) [upload=174963]: Got next segment. Inline: false Nov 7 18:12:35.792195278 (198h32m7.488044021s) [upload=174963]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 7 18:12:35.792201820 (198h32m7.488050563s) [upload=174963]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 7 18:12:36.063867588 (198h32m7.759716341s) [seg_pos={0 0}, upload=174963]: Begin upload segment... Nov 7 18:12:36.063937832 (198h32m7.759786585s) [seg_pos={0 0}, upload=174963]: Done begin upload segment. Nov 7 18:12:36.063988689 (198h32m7.759837463s) [noise=true , address=145.79.227.27:4690 , piece_id=7CVNJTAEBVYJXRWNSFYFP3E7SKMY5XI4CV2JEZJLSXH4TA2F2JZQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.064083510 (198h32m7.759932263s) [noise=true , address=145.79.227.10:4403 , piece_id=JCMXNJ5PW5ULGOJZ5SGB423VW7VJPC5REYNSNWJZJXHGUQXUS3KA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.064896969 (198h32m7.760745762s) [noise=true , address=145.79.227.28:4726 , piece_id=D6DH5FHFB5ZTPANRPUWZKRYPFYUV2LS6AY5FQL5RBJZIUNM4FKLQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.065443520 (198h32m7.761292273s) [noise=true , address=77.90.31.11:4554 , piece_id=TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.065970974 (198h32m7.761819777s) [noise=true , address=145.79.227.8:4330 , piece_id=32XBCOEGSUTVUCCWVM2D7CB6FKMNLD6N2N4655B3ZYHNAIL4R2QQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.066422053 (198h32m7.762270806s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.067054427 (198h32m7.76290318s) [noise=true , address=77.90.31.10:4519 , piece_id=MIHLKGHCRTUFQSGBIFKIWPNACAJ4CQOEYRDIKA4KJLFP356VMY6A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.067645042 (198h32m7.763493795s) [noise=true , address=45.91.116.6:4973 , piece_id=Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.068379079 (198h32m7.764227842s) [noise=true , address=77.90.31.19:4843 , piece_id=JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.070931099 (198h32m7.766779862s) [noise=true , address=145.79.227.11:4732 , piece_id=VKSPYLTD6C6XBUFXXVVPHIPGFRCRNQK6EVI4SM7Y6ODOMVGGAJOQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.073100420 (198h32m7.768949173s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.075821872 (198h32m7.771670625s) [noise=true , address=45.91.116.4:4937 , piece_id=CWDA2IYNFUYHVDBJ74Y3CJKCZUBN4QYUEJYFKZNIBAUYYAXO3WFQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.078400863 (198h32m7.774249626s) [noise=true , address=45.91.116.10:4082 , piece_id=RW62GCPC56K2IMOFF276GQECXRKW6NY6W44TNAYFNDGQS3UK7EEA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.081230762 (198h32m7.777079515s) [noise=true , address=145.79.227.15:4876 , piece_id=ZPGKGNXRFEWRRJKTS7NKBEVQA7OAWNWYDR3DUTPHT3XVYRJSKWPQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.083298258 (198h32m7.779147012s) [noise=true , address=145.79.227.20:4438 , piece_id=V6WDUVDB7WHTDCTSYCFWS22UKRD5NMXGJ2GHLSFLVVOXZMZWLL4A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.084602091 (198h32m7.780450844s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:37.640211006 (198h32m9.336059759s) [upload=174963]: Next returned nil segment Nov 7 18:12:37.640227878 (198h32m9.336076621s) [upload=174963]: Waiting for error group managing segments... Nov 7 18:12:38.007509819 (198h32m9.703358572s) [noise=true , address=145.79.227.10:4403 , piece_id=JCMXNJ5PW5ULGOJZ5SGB423VW7VJPC5REYNSNWJZJXHGUQXUS3KA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.015182750 (198h32m9.711031503s) [noise=true , address=145.79.227.28:4726 , piece_id=D6DH5FHFB5ZTPANRPUWZKRYPFYUV2LS6AY5FQL5RBJZIUNM4FKLQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.020400886 (198h32m9.71624964s) [noise=true , address=145.79.227.27:4690 , piece_id=7CVNJTAEBVYJXRWNSFYFP3E7SKMY5XI4CV2JEZJLSXH4TA2F2JZQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.062361869 (198h32m9.758210622s) [noise=true , address=145.79.227.15:4876 , piece_id=ZPGKGNXRFEWRRJKTS7NKBEVQA7OAWNWYDR3DUTPHT3XVYRJSKWPQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.096654439 (198h32m9.792503192s) [noise=true , address=145.79.227.20:4438 , piece_id=V6WDUVDB7WHTDCTSYCFWS22UKRD5NMXGJ2GHLSFLVVOXZMZWLL4A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.244365682 (198h32m9.940214435s) [noise=true , address=145.79.227.8:4330 , piece_id=32XBCOEGSUTVUCCWVM2D7CB6FKMNLD6N2N4655B3ZYHNAIL4R2QQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.245114297 (198h32m9.94096305s) [noise=true , address=145.79.227.11:4732 , piece_id=VKSPYLTD6C6XBUFXXVVPHIPGFRCRNQK6EVI4SM7Y6ODOMVGGAJOQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.324356281 (198h32m10.020205034s) [noise=true , address=77.90.31.11:4554 , piece_id=TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.331696878 (198h32m10.027545621s) [noise=true , address=45.91.116.4:4937 , piece_id=CWDA2IYNFUYHVDBJ74Y3CJKCZUBN4QYUEJYFKZNIBAUYYAXO3WFQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.334142676 (198h32m10.029991429s) [noise=true , address=45.91.116.10:4082 , piece_id=RW62GCPC56K2IMOFF276GQECXRKW6NY6W44TNAYFNDGQS3UK7EEA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.346539031 (198h32m10.042387794s) [noise=true , address=77.90.31.10:4519 , piece_id=MIHLKGHCRTUFQSGBIFKIWPNACAJ4CQOEYRDIKA4KJLFP356VMY6A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.419504032 (198h32m10.115352785s) [noise=true , address=77.90.31.19:4843 , piece_id=JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:39.113150499 (198h32m10.808999252s) [noise=true , address=45.91.116.6:4973 , piece_id=Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:39.113177150 (198h32m10.809025903s) [seg_pos={0 0}, upload=174963]: Segment reached optimal threshold of 13 pieces. Nov 7 18:12:39.113381018 (198h32m10.809229772s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113390166 (198h32m10.809238919s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113460199 (198h32m10.809308952s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113470077 (198h32m10.809318821s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113780338 (198h32m10.809629091s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113788243 (198h32m10.809636986s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113858657 (198h32m10.8097074s) [seg_pos={0 0}, upload=174963]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 7 18:12:39.114030194 (198h32m10.809878947s) [upload=174963]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 7 18:12:39.114038290 (198h32m10.809887043s) [upload=174963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 7 18:12:39.451448640 (198h32m11.147297403s) [upload=174963]: Done uploading segments... Nov 7 18:12:39.451473607 (198h32m11.14732236s) [upload=174963]: Upload finished. err: <nil> Duration: 6.081748323s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [X.X..X.X.XXXX.X..XX.XXXXX.X.................................] duration: 6.094572377s trace: https://jaeger.dev.storj.io/trace/422f33ac8ae6a74a FAIL (2025-11-07 18:37:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ) context deadline exceeded; piecestore: (Node ID: 12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, Piece ID: JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ) context deadline exceeded ======LOGS====== Nov 7 18:12:35.791697270 (198h32m7.487546023s) [upload=174963]: Starting upload Nov 7 18:12:35.791791689 (198h32m7.487640452s) [upload=174963]: Done starting upload Nov 7 18:12:35.791885458 (198h32m7.487734211s) [upload=174963]: Uploading segments... Nov 7 18:12:35.792146455 (198h32m7.487995208s) [upload=174963]: Got next segment. Inline: false Nov 7 18:12:35.792195278 (198h32m7.488044021s) [upload=174963]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 7 18:12:35.792201820 (198h32m7.488050563s) [upload=174963]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 7 18:12:36.063867588 (198h32m7.759716341s) [seg_pos={0 0}, upload=174963]: Begin upload segment... Nov 7 18:12:36.063937832 (198h32m7.759786585s) [seg_pos={0 0}, upload=174963]: Done begin upload segment. Nov 7 18:12:36.063988689 (198h32m7.759837463s) [noise=true , address=145.79.227.27:4690 , piece_id=7CVNJTAEBVYJXRWNSFYFP3E7SKMY5XI4CV2JEZJLSXH4TA2F2JZQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.064083510 (198h32m7.759932263s) [noise=true , address=145.79.227.10:4403 , piece_id=JCMXNJ5PW5ULGOJZ5SGB423VW7VJPC5REYNSNWJZJXHGUQXUS3KA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.064896969 (198h32m7.760745762s) [noise=true , address=145.79.227.28:4726 , piece_id=D6DH5FHFB5ZTPANRPUWZKRYPFYUV2LS6AY5FQL5RBJZIUNM4FKLQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.065443520 (198h32m7.761292273s) [noise=true , address=77.90.31.11:4554 , piece_id=TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.065970974 (198h32m7.761819777s) [noise=true , address=145.79.227.8:4330 , piece_id=32XBCOEGSUTVUCCWVM2D7CB6FKMNLD6N2N4655B3ZYHNAIL4R2QQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.066422053 (198h32m7.762270806s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.067054427 (198h32m7.76290318s) [noise=true , address=77.90.31.10:4519 , piece_id=MIHLKGHCRTUFQSGBIFKIWPNACAJ4CQOEYRDIKA4KJLFP356VMY6A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.067645042 (198h32m7.763493795s) [noise=true , address=45.91.116.6:4973 , piece_id=Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.068379079 (198h32m7.764227842s) [noise=true , address=77.90.31.19:4843 , piece_id=JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.070931099 (198h32m7.766779862s) [noise=true , address=145.79.227.11:4732 , piece_id=VKSPYLTD6C6XBUFXXVVPHIPGFRCRNQK6EVI4SM7Y6ODOMVGGAJOQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.073100420 (198h32m7.768949173s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.075821872 (198h32m7.771670625s) [noise=true , address=45.91.116.4:4937 , piece_id=CWDA2IYNFUYHVDBJ74Y3CJKCZUBN4QYUEJYFKZNIBAUYYAXO3WFQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.078400863 (198h32m7.774249626s) [noise=true , address=45.91.116.10:4082 , piece_id=RW62GCPC56K2IMOFF276GQECXRKW6NY6W44TNAYFNDGQS3UK7EEA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.081230762 (198h32m7.777079515s) [noise=true , address=145.79.227.15:4876 , piece_id=ZPGKGNXRFEWRRJKTS7NKBEVQA7OAWNWYDR3DUTPHT3XVYRJSKWPQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.083298258 (198h32m7.779147012s) [noise=true , address=145.79.227.20:4438 , piece_id=V6WDUVDB7WHTDCTSYCFWS22UKRD5NMXGJ2GHLSFLVVOXZMZWLL4A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.084602091 (198h32m7.780450844s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:37.640211006 (198h32m9.336059759s) [upload=174963]: Next returned nil segment Nov 7 18:12:37.640227878 (198h32m9.336076621s) [upload=174963]: Waiting for error group managing segments... Nov 7 18:12:38.007509819 (198h32m9.703358572s) [noise=true , address=145.79.227.10:4403 , piece_id=JCMXNJ5PW5ULGOJZ5SGB423VW7VJPC5REYNSNWJZJXHGUQXUS3KA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.015182750 (198h32m9.711031503s) [noise=true , address=145.79.227.28:4726 , piece_id=D6DH5FHFB5ZTPANRPUWZKRYPFYUV2LS6AY5FQL5RBJZIUNM4FKLQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.020400886 (198h32m9.71624964s) [noise=true , address=145.79.227.27:4690 , piece_id=7CVNJTAEBVYJXRWNSFYFP3E7SKMY5XI4CV2JEZJLSXH4TA2F2JZQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.062361869 (198h32m9.758210622s) [noise=true , address=145.79.227.15:4876 , piece_id=ZPGKGNXRFEWRRJKTS7NKBEVQA7OAWNWYDR3DUTPHT3XVYRJSKWPQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.096654439 (198h32m9.792503192s) [noise=true , address=145.79.227.20:4438 , piece_id=V6WDUVDB7WHTDCTSYCFWS22UKRD5NMXGJ2GHLSFLVVOXZMZWLL4A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.244365682 (198h32m9.940214435s) [noise=true , address=145.79.227.8:4330 , piece_id=32XBCOEGSUTVUCCWVM2D7CB6FKMNLD6N2N4655B3ZYHNAIL4R2QQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.245114297 (198h32m9.94096305s) [noise=true , address=145.79.227.11:4732 , piece_id=VKSPYLTD6C6XBUFXXVVPHIPGFRCRNQK6EVI4SM7Y6ODOMVGGAJOQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.324356281 (198h32m10.020205034s) [noise=true , address=77.90.31.11:4554 , piece_id=TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.331696878 (198h32m10.027545621s) [noise=true , address=45.91.116.4:4937 , piece_id=CWDA2IYNFUYHVDBJ74Y3CJKCZUBN4QYUEJYFKZNIBAUYYAXO3WFQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.334142676 (198h32m10.029991429s) [noise=true , address=45.91.116.10:4082 , piece_id=RW62GCPC56K2IMOFF276GQECXRKW6NY6W44TNAYFNDGQS3UK7EEA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.346539031 (198h32m10.042387794s) [noise=true , address=77.90.31.10:4519 , piece_id=MIHLKGHCRTUFQSGBIFKIWPNACAJ4CQOEYRDIKA4KJLFP356VMY6A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.419504032 (198h32m10.115352785s) [noise=true , address=77.90.31.19:4843 , piece_id=JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:39.113150499 (198h32m10.808999252s) [noise=true , address=45.91.116.6:4973 , piece_id=Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:39.113177150 (198h32m10.809025903s) [seg_pos={0 0}, upload=174963]: Segment reached optimal threshold of 13 pieces. Nov 7 18:12:39.113381018 (198h32m10.809229772s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113390166 (198h32m10.809238919s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113460199 (198h32m10.809308952s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113470077 (198h32m10.809318821s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113780338 (198h32m10.809629091s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113788243 (198h32m10.809636986s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113858657 (198h32m10.8097074s) [seg_pos={0 0}, upload=174963]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 7 18:12:39.114030194 (198h32m10.809878947s) [upload=174963]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 7 18:12:39.114038290 (198h32m10.809887043s) [upload=174963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 7 18:12:39.451448640 (198h32m11.147297403s) [upload=174963]: Done uploading segments... Nov 7 18:12:39.451473607 (198h32m11.14732236s) [upload=174963]: Upload finished. err: <nil> Duration: 6.094609717s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [XX.X..X.X.XXXX.X..XX.XXXXX.X................................] duration: 6.705880638s trace: https://jaeger.dev.storj.io/trace/422f33ac8ae6a74a FAIL (2025-11-07 18:36:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ) context deadline exceeded; piecestore: (Node ID: 12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, Piece ID: JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ) context deadline exceeded ======LOGS====== Nov 7 18:12:35.791697270 (198h32m7.487546023s) [upload=174963]: Starting upload Nov 7 18:12:35.791791689 (198h32m7.487640452s) [upload=174963]: Done starting upload Nov 7 18:12:35.791885458 (198h32m7.487734211s) [upload=174963]: Uploading segments... Nov 7 18:12:35.792146455 (198h32m7.487995208s) [upload=174963]: Got next segment. Inline: false Nov 7 18:12:35.792195278 (198h32m7.488044021s) [upload=174963]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 7 18:12:35.792201820 (198h32m7.488050563s) [upload=174963]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 7 18:12:36.063867588 (198h32m7.759716341s) [seg_pos={0 0}, upload=174963]: Begin upload segment... Nov 7 18:12:36.063937832 (198h32m7.759786585s) [seg_pos={0 0}, upload=174963]: Done begin upload segment. Nov 7 18:12:36.063988689 (198h32m7.759837463s) [noise=true , address=145.79.227.27:4690 , piece_id=7CVNJTAEBVYJXRWNSFYFP3E7SKMY5XI4CV2JEZJLSXH4TA2F2JZQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.064083510 (198h32m7.759932263s) [noise=true , address=145.79.227.10:4403 , piece_id=JCMXNJ5PW5ULGOJZ5SGB423VW7VJPC5REYNSNWJZJXHGUQXUS3KA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.064896969 (198h32m7.760745762s) [noise=true , address=145.79.227.28:4726 , piece_id=D6DH5FHFB5ZTPANRPUWZKRYPFYUV2LS6AY5FQL5RBJZIUNM4FKLQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.065443520 (198h32m7.761292273s) [noise=true , address=77.90.31.11:4554 , piece_id=TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.065970974 (198h32m7.761819777s) [noise=true , address=145.79.227.8:4330 , piece_id=32XBCOEGSUTVUCCWVM2D7CB6FKMNLD6N2N4655B3ZYHNAIL4R2QQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.066422053 (198h32m7.762270806s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.067054427 (198h32m7.76290318s) [noise=true , address=77.90.31.10:4519 , piece_id=MIHLKGHCRTUFQSGBIFKIWPNACAJ4CQOEYRDIKA4KJLFP356VMY6A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.067645042 (198h32m7.763493795s) [noise=true , address=45.91.116.6:4973 , piece_id=Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.068379079 (198h32m7.764227842s) [noise=true , address=77.90.31.19:4843 , piece_id=JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.070931099 (198h32m7.766779862s) [noise=true , address=145.79.227.11:4732 , piece_id=VKSPYLTD6C6XBUFXXVVPHIPGFRCRNQK6EVI4SM7Y6ODOMVGGAJOQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.073100420 (198h32m7.768949173s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.075821872 (198h32m7.771670625s) [noise=true , address=45.91.116.4:4937 , piece_id=CWDA2IYNFUYHVDBJ74Y3CJKCZUBN4QYUEJYFKZNIBAUYYAXO3WFQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.078400863 (198h32m7.774249626s) [noise=true , address=45.91.116.10:4082 , piece_id=RW62GCPC56K2IMOFF276GQECXRKW6NY6W44TNAYFNDGQS3UK7EEA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.081230762 (198h32m7.777079515s) [noise=true , address=145.79.227.15:4876 , piece_id=ZPGKGNXRFEWRRJKTS7NKBEVQA7OAWNWYDR3DUTPHT3XVYRJSKWPQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.083298258 (198h32m7.779147012s) [noise=true , address=145.79.227.20:4438 , piece_id=V6WDUVDB7WHTDCTSYCFWS22UKRD5NMXGJ2GHLSFLVVOXZMZWLL4A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.084602091 (198h32m7.780450844s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:37.640211006 (198h32m9.336059759s) [upload=174963]: Next returned nil segment Nov 7 18:12:37.640227878 (198h32m9.336076621s) [upload=174963]: Waiting for error group managing segments... Nov 7 18:12:38.007509819 (198h32m9.703358572s) [noise=true , address=145.79.227.10:4403 , piece_id=JCMXNJ5PW5ULGOJZ5SGB423VW7VJPC5REYNSNWJZJXHGUQXUS3KA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.015182750 (198h32m9.711031503s) [noise=true , address=145.79.227.28:4726 , piece_id=D6DH5FHFB5ZTPANRPUWZKRYPFYUV2LS6AY5FQL5RBJZIUNM4FKLQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.020400886 (198h32m9.71624964s) [noise=true , address=145.79.227.27:4690 , piece_id=7CVNJTAEBVYJXRWNSFYFP3E7SKMY5XI4CV2JEZJLSXH4TA2F2JZQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.062361869 (198h32m9.758210622s) [noise=true , address=145.79.227.15:4876 , piece_id=ZPGKGNXRFEWRRJKTS7NKBEVQA7OAWNWYDR3DUTPHT3XVYRJSKWPQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.096654439 (198h32m9.792503192s) [noise=true , address=145.79.227.20:4438 , piece_id=V6WDUVDB7WHTDCTSYCFWS22UKRD5NMXGJ2GHLSFLVVOXZMZWLL4A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.244365682 (198h32m9.940214435s) [noise=true , address=145.79.227.8:4330 , piece_id=32XBCOEGSUTVUCCWVM2D7CB6FKMNLD6N2N4655B3ZYHNAIL4R2QQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.245114297 (198h32m9.94096305s) [noise=true , address=145.79.227.11:4732 , piece_id=VKSPYLTD6C6XBUFXXVVPHIPGFRCRNQK6EVI4SM7Y6ODOMVGGAJOQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.324356281 (198h32m10.020205034s) [noise=true , address=77.90.31.11:4554 , piece_id=TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.331696878 (198h32m10.027545621s) [noise=true , address=45.91.116.4:4937 , piece_id=CWDA2IYNFUYHVDBJ74Y3CJKCZUBN4QYUEJYFKZNIBAUYYAXO3WFQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.334142676 (198h32m10.029991429s) [noise=true , address=45.91.116.10:4082 , piece_id=RW62GCPC56K2IMOFF276GQECXRKW6NY6W44TNAYFNDGQS3UK7EEA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.346539031 (198h32m10.042387794s) [noise=true , address=77.90.31.10:4519 , piece_id=MIHLKGHCRTUFQSGBIFKIWPNACAJ4CQOEYRDIKA4KJLFP356VMY6A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.419504032 (198h32m10.115352785s) [noise=true , address=77.90.31.19:4843 , piece_id=JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:39.113150499 (198h32m10.808999252s) [noise=true , address=45.91.116.6:4973 , piece_id=Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:39.113177150 (198h32m10.809025903s) [seg_pos={0 0}, upload=174963]: Segment reached optimal threshold of 13 pieces. Nov 7 18:12:39.113381018 (198h32m10.809229772s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113390166 (198h32m10.809238919s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113460199 (198h32m10.809308952s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113470077 (198h32m10.809318821s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113780338 (198h32m10.809629091s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113788243 (198h32m10.809636986s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113858657 (198h32m10.8097074s) [seg_pos={0 0}, upload=174963]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 7 18:12:39.114030194 (198h32m10.809878947s) [upload=174963]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 7 18:12:39.114038290 (198h32m10.809887043s) [upload=174963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 7 18:12:39.451448640 (198h32m11.147297403s) [upload=174963]: Done uploading segments... Nov 7 18:12:39.451473607 (198h32m11.14732236s) [upload=174963]: Upload finished. err: <nil> Duration: 6.70590884s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [XXX.X..X.X.XXXX.X..XX.XXXXX.X...............................] duration: 6.059421365s trace: https://jaeger.dev.storj.io/trace/422f33ac8ae6a74a FAIL (2025-11-07 18:35:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ) context deadline exceeded; piecestore: (Node ID: 12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, Piece ID: JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ) context deadline exceeded ======LOGS====== Nov 7 18:12:35.791697270 (198h32m7.487546023s) [upload=174963]: Starting upload Nov 7 18:12:35.791791689 (198h32m7.487640452s) [upload=174963]: Done starting upload Nov 7 18:12:35.791885458 (198h32m7.487734211s) [upload=174963]: Uploading segments... Nov 7 18:12:35.792146455 (198h32m7.487995208s) [upload=174963]: Got next segment. Inline: false Nov 7 18:12:35.792195278 (198h32m7.488044021s) [upload=174963]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 7 18:12:35.792201820 (198h32m7.488050563s) [upload=174963]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 7 18:12:36.063867588 (198h32m7.759716341s) [seg_pos={0 0}, upload=174963]: Begin upload segment... Nov 7 18:12:36.063937832 (198h32m7.759786585s) [seg_pos={0 0}, upload=174963]: Done begin upload segment. Nov 7 18:12:36.063988689 (198h32m7.759837463s) [noise=true , address=145.79.227.27:4690 , piece_id=7CVNJTAEBVYJXRWNSFYFP3E7SKMY5XI4CV2JEZJLSXH4TA2F2JZQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.064083510 (198h32m7.759932263s) [noise=true , address=145.79.227.10:4403 , piece_id=JCMXNJ5PW5ULGOJZ5SGB423VW7VJPC5REYNSNWJZJXHGUQXUS3KA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.064896969 (198h32m7.760745762s) [noise=true , address=145.79.227.28:4726 , piece_id=D6DH5FHFB5ZTPANRPUWZKRYPFYUV2LS6AY5FQL5RBJZIUNM4FKLQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.065443520 (198h32m7.761292273s) [noise=true , address=77.90.31.11:4554 , piece_id=TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.065970974 (198h32m7.761819777s) [noise=true , address=145.79.227.8:4330 , piece_id=32XBCOEGSUTVUCCWVM2D7CB6FKMNLD6N2N4655B3ZYHNAIL4R2QQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.066422053 (198h32m7.762270806s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.067054427 (198h32m7.76290318s) [noise=true , address=77.90.31.10:4519 , piece_id=MIHLKGHCRTUFQSGBIFKIWPNACAJ4CQOEYRDIKA4KJLFP356VMY6A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.067645042 (198h32m7.763493795s) [noise=true , address=45.91.116.6:4973 , piece_id=Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.068379079 (198h32m7.764227842s) [noise=true , address=77.90.31.19:4843 , piece_id=JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.070931099 (198h32m7.766779862s) [noise=true , address=145.79.227.11:4732 , piece_id=VKSPYLTD6C6XBUFXXVVPHIPGFRCRNQK6EVI4SM7Y6ODOMVGGAJOQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.073100420 (198h32m7.768949173s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.075821872 (198h32m7.771670625s) [noise=true , address=45.91.116.4:4937 , piece_id=CWDA2IYNFUYHVDBJ74Y3CJKCZUBN4QYUEJYFKZNIBAUYYAXO3WFQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.078400863 (198h32m7.774249626s) [noise=true , address=45.91.116.10:4082 , piece_id=RW62GCPC56K2IMOFF276GQECXRKW6NY6W44TNAYFNDGQS3UK7EEA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.081230762 (198h32m7.777079515s) [noise=true , address=145.79.227.15:4876 , piece_id=ZPGKGNXRFEWRRJKTS7NKBEVQA7OAWNWYDR3DUTPHT3XVYRJSKWPQ, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.083298258 (198h32m7.779147012s) [noise=true , address=145.79.227.20:4438 , piece_id=V6WDUVDB7WHTDCTSYCFWS22UKRD5NMXGJ2GHLSFLVVOXZMZWLL4A, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:36.084602091 (198h32m7.780450844s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Uploading piece... Nov 7 18:12:37.640211006 (198h32m9.336059759s) [upload=174963]: Next returned nil segment Nov 7 18:12:37.640227878 (198h32m9.336076621s) [upload=174963]: Waiting for error group managing segments... Nov 7 18:12:38.007509819 (198h32m9.703358572s) [noise=true , address=145.79.227.10:4403 , piece_id=JCMXNJ5PW5ULGOJZ5SGB423VW7VJPC5REYNSNWJZJXHGUQXUS3KA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.015182750 (198h32m9.711031503s) [noise=true , address=145.79.227.28:4726 , piece_id=D6DH5FHFB5ZTPANRPUWZKRYPFYUV2LS6AY5FQL5RBJZIUNM4FKLQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.020400886 (198h32m9.71624964s) [noise=true , address=145.79.227.27:4690 , piece_id=7CVNJTAEBVYJXRWNSFYFP3E7SKMY5XI4CV2JEZJLSXH4TA2F2JZQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.062361869 (198h32m9.758210622s) [noise=true , address=145.79.227.15:4876 , piece_id=ZPGKGNXRFEWRRJKTS7NKBEVQA7OAWNWYDR3DUTPHT3XVYRJSKWPQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.096654439 (198h32m9.792503192s) [noise=true , address=145.79.227.20:4438 , piece_id=V6WDUVDB7WHTDCTSYCFWS22UKRD5NMXGJ2GHLSFLVVOXZMZWLL4A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.244365682 (198h32m9.940214435s) [noise=true , address=145.79.227.8:4330 , piece_id=32XBCOEGSUTVUCCWVM2D7CB6FKMNLD6N2N4655B3ZYHNAIL4R2QQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.245114297 (198h32m9.94096305s) [noise=true , address=145.79.227.11:4732 , piece_id=VKSPYLTD6C6XBUFXXVVPHIPGFRCRNQK6EVI4SM7Y6ODOMVGGAJOQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.324356281 (198h32m10.020205034s) [noise=true , address=77.90.31.11:4554 , piece_id=TGH7QSLUXLBSUWF5DGAAI64POWSCFLBUEXJXJBN7B2IUTFBOZX5A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.331696878 (198h32m10.027545621s) [noise=true , address=45.91.116.4:4937 , piece_id=CWDA2IYNFUYHVDBJ74Y3CJKCZUBN4QYUEJYFKZNIBAUYYAXO3WFQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.334142676 (198h32m10.029991429s) [noise=true , address=45.91.116.10:4082 , piece_id=RW62GCPC56K2IMOFF276GQECXRKW6NY6W44TNAYFNDGQS3UK7EEA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.346539031 (198h32m10.042387794s) [noise=true , address=77.90.31.10:4519 , piece_id=MIHLKGHCRTUFQSGBIFKIWPNACAJ4CQOEYRDIKA4KJLFP356VMY6A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:38.419504032 (198h32m10.115352785s) [noise=true , address=77.90.31.19:4843 , piece_id=JJMCK3LC3OAOGOICR7Q6EBKMTOY2YMIKJD5INPIU2PT4LHXZSDKQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:39.113150499 (198h32m10.808999252s) [noise=true , address=45.91.116.6:4973 , piece_id=Z56HTF4NSSXH7YXAFWZMB2ONFKUDYLDHO4OUAQHKBSDSKOEL5DSQ, seg_pos={0 0}, upload=174963]: Done uploading piece. err: <nil> Nov 7 18:12:39.113177150 (198h32m10.809025903s) [seg_pos={0 0}, upload=174963]: Segment reached optimal threshold of 13 pieces. Nov 7 18:12:39.113381018 (198h32m10.809229772s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113390166 (198h32m10.809238919s) [noise=true , address=45.91.116.14:4189 , piece_id=MHSTR4LEAZXFPH7G53SPYYG2VTFYGPHG6EUI7BHB3IB7BNHNSU7A, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113460199 (198h32m10.809308952s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113470077 (198h32m10.809318821s) [noise=true , address=45.91.116.5:4299 , piece_id=ZWDZ5ZPMPFGIJSZQ3EBWNW3TE2V3YP4K2ITQG6LOHQWVJ5G6Q3NA, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113780338 (198h32m10.809629091s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Nov 7 18:12:39.113788243 (198h32m10.809636986s) [noise=true , address=45.91.116.13:4153 , piece_id=G52YIRA5DSVEAXE45CE2G6FIIDW25B24BDKFP6QBVCJZJPXM2VHA, seg_pos={0 0}, upload=174963]: Long tail cancellation due to optimal threshold Nov 7 18:12:39.113858657 (198h32m10.8097074s) [seg_pos={0 0}, upload=174963]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 7 18:12:39.114030194 (198h32m10.809878947s) [upload=174963]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 7 18:12:39.114038290 (198h32m10.809887043s) [upload=174963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 7 18:12:39.451448640 (198h32m11.147297403s) [upload=174963]: Done uploading segments... Nov 7 18:12:39.451473607 (198h32m11.14732236s) [upload=174963]: Upload finished. err: <nil> Duration: 6.059453807s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)