Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 20 out of 60 is failed (threshold=6) [.X....XXX...XXXX.........X.X.....X..XXXXXX.X.....X........X.] duration: 3.563079807s trace: https://jaeger.dev.storj.io/trace/4f0c589bdb8ad2b8 FAIL (2025-12-30 07:30:01 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.563113743s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 20 out of 60 is failed (threshold=6) [..X....XXX...XXXX.........X.X.....X..XXXXXX.X.....X........X] duration: 5.001435869s trace: https://jaeger.dev.storj.io/trace/4f0c589bdb8ad2b8 FAIL (2025-12-30 07:29:40 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001456719s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [X...X.....X....XXX..X.....X...X..X.XX..X...X....XXX....X....] duration: 5.670853757s trace: https://jaeger.dev.storj.io/trace/5b5a31fe4475073 FAIL (2025-12-30 07:29:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 07:26:46.289184198 (591h57m46.160267243s) [upload=518946]: Starting upload Dec 30 07:26:46.289281163 (591h57m46.160364218s) [upload=518946]: Done starting upload Dec 30 07:26:46.289508094 (591h57m46.160591129s) [upload=518946]: Uploading segments... Dec 30 07:26:46.289589440 (591h57m46.160672495s) [upload=518946]: Got next segment. Inline: false Dec 30 07:26:46.289633714 (591h57m46.160716759s) [upload=518946]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 07:26:46.289639164 (591h57m46.160722209s) [upload=518946]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 07:26:46.532324195 (591h57m46.40340724s) [seg_pos={0 0}, upload=518946]: Begin upload segment... Dec 30 07:26:46.532410239 (591h57m46.403493274s) [seg_pos={0 0}, upload=518946]: Done begin upload segment. Dec 30 07:26:46.532448602 (591h57m46.403531637s) [noise=true , address=145.79.227.2:4114 , piece_id=F6QSNFGWCWQDQUCOICC3GNCHUPERG6PWBO2OIAQDCGRHPS5YA2PA, seg_pos={0 0}, upload=518946]: Uploading piece... Dec 30 07:26:46.532504338 (591h57m46.403587383s) [noise=true , address=45.91.116.8:4046 , piece_id=5YF5RU2ICVQJLAG43MN4K3AHEHGOQBQ57ECEKAUUX75XHJNKVNRQ, seg_pos={0 0}, upload=518946]: Uploading piece... Dec 30 07:26:46.533077460 (591h57m46.404160505s) [noise=true , address=145.79.227.39:4015 , piece_id=NGGRJ4EFWFGJSHIQ3YHBS3SQZVO4N4UWXIAB6JA32K6PVAUC5AZQ, seg_pos={0 0}, upload=518946]: Uploading piece... Dec 30 07:26:46.533549498 (591h57m46.404632533s) [noise=true , address=145.79.227.19:4078 , piece_id=AMXZNOQCZXM7LGL46SR7CS7CZ55BGTRTOZ5DJAYDMTBGB5NNDC3Q, seg_pos={0 0}, upload=518946]: Uploading piece... Dec 30 07:26:46.534157366 (591h57m46.405240401s) [noise=true , address=145.79.227.41:4085 , piece_id=7RGSCLOZ46YHEPE5ZKU6QMQHNW7SLV3VN6NCURYAUWBV2RWRHEEQ, seg_pos={0 0}, upload=518946]: Uploading piece... Dec 30 07:26:46.534782767 (591h57m46.405865802s) [noise=true , address=145.79.227.15:4874 , piece_id=GY5FWVL6PE77LP7455DHVEMQ2VKV3W6XL5NHN4PPJ7F45HPUC4IA, seg_pos={0 0}, upload=518946]: Uploading piece... Dec 30 07:26:46.535251168 (591h57m46.406334213s) [noise=true , address=145.79.227.51:4411 , piece_id=UIKRX5OVZIFAKUBTQQGRYMPMIXSFY43VUBC2KRCCGWWMFUWD2LSQ, seg_pos={0 0}, upload=518946]: Uploading piece... Dec 30 07:26:46.536749040 (591h57m46.407832085s) [noise=true , address=145.79.227.36:4517 , piece_id=DKJ2BMYTOU46NNNK6FRXT7V2V3UXRQPZURV3RH6CMU3BFCBIJC3Q, seg_pos={0 0}, upload=518946]: Uploading piece... Dec 30 07:26:46.538410154 (591h57m46.409493209s) [noise=true , address=145.79.227.14:4840 , piece_id=NI6LJNVGBHKOR4SAM7S5V3IRKEU4DG6ABLBI3VAZKLF4DOWXMJLQ, seg_pos={0 0}, upload=518946]: Uploading piece... Dec 30 07:26:46.539727031 (591h57m46.410810066s) [noise=true , address=45.91.116.5:4300 , piece_id=YZMX3AQIOSGZNCOGFBPEQ4DOIWQYX23SHVGMTNGDTIEEYTGGKVQQ, seg_pos={0 0}, upload=518946]: Uploading piece... Dec 30 07:26:46.541085869 (591h57m46.412168904s) [noise=true , address=145.79.227.32:4835 , piece_id=4HD4E2ME5CEPRUFERUH6NEFFFFGUYX2GLU2GF4OLB5SIMZCCGRQA, seg_pos={0 0}, upload=518946]: Uploading piece... Dec 30 07:26:46.541465432 (591h57m46.412548467s) [noise=true , address=145.79.227.17:4972 , piece_id=2QAO4DEM2BL4RRMJRD6L2T3XXEVJWLBW6JD6XOXM2DW3VWGAACLA, seg_pos={0 0}, upload=518946]: Uploading piece... Dec 30 07:26:46.541630516 (591h57m46.412713551s) [noise=true , address=45.91.116.4:4939 , piece_id=EX42IOXR53GUHE7FD7SZBESKBKIZQFG6EXOWP5N2SZKMN2WQIBEA, seg_pos={0 0}, upload=518946]: Uploading piece... Dec 30 07:26:46.543860943 (591h57m46.414943978s) [noise=true , address=145.79.227.22:4509 , piece_id=GXGMXCJKULDJLXTM2AGJA5VRVCVNRHKNV5NB2TTESXQ5TVPAYZKQ, seg_pos={0 0}, upload=518946]: Uploading piece... Dec 30 07:26:46.545096676 (591h57m46.416179711s) [noise=true , address=145.79.227.46:4603 , piece_id=OSFA5IDUJ5DGRO62ORYYSJFGDOF2DWVLENETXJU2DP7JMPUXFR4Q, seg_pos={0 0}, upload=518946]: Uploading piece... Dec 30 07:26:46.546477646 (591h57m46.417560691s) [noise=true , address=77.90.31.18:4805 , piece_id=PC3THDCHXRWOJPPGYCM4XVTUWJQJHHKIHUWU34YAECOYWS5IBX4Q, seg_pos={0 0}, upload=518946]: Uploading piece... Dec 30 07:26:48.130107348 (591h57m48.001190383s) [upload=518946]: Next returned nil segment Dec 30 07:26:48.130126845 (591h57m48.00120988s) [upload=518946]: Waiting for error group managing segments... Dec 30 07:26:48.522160289 (591h57m48.393243334s) [noise=true , address=145.79.227.2:4114 , piece_id=F6QSNFGWCWQDQUCOICC3GNCHUPERG6PWBO2OIAQDCGRHPS5YA2PA, seg_pos={0 0}, upload=518946]: Done uploading piece. err: <nil> Dec 30 07:26:48.529146272 (591h57m48.400229327s) [noise=true , address=145.79.227.46:4603 , piece_id=OSFA5IDUJ5DGRO62ORYYSJFGDOF2DWVLENETXJU2DP7JMPUXFR4Q, seg_pos={0 0}, upload=518946]: Done uploading piece. err: <nil> Dec 30 07:26:48.532844023 (591h57m48.403927058s) [noise=true , address=145.79.227.15:4874 , piece_id=GY5FWVL6PE77LP7455DHVEMQ2VKV3W6XL5NHN4PPJ7F45HPUC4IA, seg_pos={0 0}, upload=518946]: Done uploading piece. err: <nil> Dec 30 07:26:48.534193653 (591h57m48.405276688s) [noise=true , address=145.79.227.22:4509 , piece_id=GXGMXCJKULDJLXTM2AGJA5VRVCVNRHKNV5NB2TTESXQ5TVPAYZKQ, seg_pos={0 0}, upload=518946]: Done uploading piece. err: <nil> Dec 30 07:26:48.547639556 (591h57m48.418722631s) [noise=true , address=145.79.227.32:4835 , piece_id=4HD4E2ME5CEPRUFERUH6NEFFFFGUYX2GLU2GF4OLB5SIMZCCGRQA, seg_pos={0 0}, upload=518946]: Done uploading piece. err: <nil> Dec 30 07:26:48.554344823 (591h57m48.425427858s) [noise=true , address=145.79.227.39:4015 , piece_id=NGGRJ4EFWFGJSHIQ3YHBS3SQZVO4N4UWXIAB6JA32K6PVAUC5AZQ, seg_pos={0 0}, upload=518946]: Done uploading piece. err: <nil> Dec 30 07:26:48.555384473 (591h57m48.426467508s) [noise=true , address=145.79.227.51:4411 , piece_id=UIKRX5OVZIFAKUBTQQGRYMPMIXSFY43VUBC2KRCCGWWMFUWD2LSQ, seg_pos={0 0}, upload=518946]: Done uploading piece. err: <nil> Dec 30 07:26:48.557932195 (591h57m48.42901523s) [noise=true , address=145.79.227.36:4517 , piece_id=DKJ2BMYTOU46NNNK6FRXT7V2V3UXRQPZURV3RH6CMU3BFCBIJC3Q, seg_pos={0 0}, upload=518946]: Done uploading piece. err: <nil> Dec 30 07:26:48.601713982 (591h57m48.472797017s) [noise=true , address=145.79.227.19:4078 , piece_id=AMXZNOQCZXM7LGL46SR7CS7CZ55BGTRTOZ5DJAYDMTBGB5NNDC3Q, seg_pos={0 0}, upload=518946]: Done uploading piece. err: <nil> Dec 30 07:26:48.607727713 (591h57m48.478810748s) [noise=true , address=145.79.227.14:4840 , piece_id=NI6LJNVGBHKOR4SAM7S5V3IRKEU4DG6ABLBI3VAZKLF4DOWXMJLQ, seg_pos={0 0}, upload=518946]: Done uploading piece. err: <nil> Dec 30 07:26:48.613029328 (591h57m48.484112363s) [noise=true , address=145.79.227.41:4085 , piece_id=7RGSCLOZ46YHEPE5ZKU6QMQHNW7SLV3VN6NCURYAUWBV2RWRHEEQ, seg_pos={0 0}, upload=518946]: Done uploading piece. err: <nil> Dec 30 07:26:48.787823986 (591h57m48.658907021s) [noise=true , address=77.90.31.18:4805 , piece_id=PC3THDCHXRWOJPPGYCM4XVTUWJQJHHKIHUWU34YAECOYWS5IBX4Q, seg_pos={0 0}, upload=518946]: Done uploading piece. err: <nil> Dec 30 07:26:48.905112491 (591h57m48.776195526s) [noise=true , address=45.91.116.8:4046 , piece_id=5YF5RU2ICVQJLAG43MN4K3AHEHGOQBQ57ECEKAUUX75XHJNKVNRQ, seg_pos={0 0}, upload=518946]: Done uploading piece. err: <nil> Dec 30 07:26:48.905132369 (591h57m48.776215404s) [seg_pos={0 0}, upload=518946]: Segment reached optimal threshold of 13 pieces. Dec 30 07:26:48.905318543 (591h57m48.776401588s) [noise=true , address=45.91.116.4:4939 , piece_id=EX42IOXR53GUHE7FD7SZBESKBKIZQFG6EXOWP5N2SZKMN2WQIBEA, seg_pos={0 0}, upload=518946]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 30 07:26:48.905324355 (591h57m48.77640739s) [noise=true , address=45.91.116.4:4939 , piece_id=EX42IOXR53GUHE7FD7SZBESKBKIZQFG6EXOWP5N2SZKMN2WQIBEA, seg_pos={0 0}, upload=518946]: Long tail cancellation due to optimal threshold Dec 30 07:26:48.905423974 (591h57m48.776507009s) [noise=true , address=45.91.116.5:4300 , piece_id=YZMX3AQIOSGZNCOGFBPEQ4DOIWQYX23SHVGMTNGDTIEEYTGGKVQQ, seg_pos={0 0}, upload=518946]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Dec 30 07:26:48.905429255 (591h57m48.77651229s) [noise=true , address=45.91.116.5:4300 , piece_id=YZMX3AQIOSGZNCOGFBPEQ4DOIWQYX23SHVGMTNGDTIEEYTGGKVQQ, seg_pos={0 0}, upload=518946]: Long tail cancellation due to optimal threshold Dec 30 07:26:48.905524476 (591h57m48.776607511s) [noise=true , address=145.79.227.17:4972 , piece_id=2QAO4DEM2BL4RRMJRD6L2T3XXEVJWLBW6JD6XOXM2DW3VWGAACLA, seg_pos={0 0}, upload=518946]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: expected piece hash; context canceled; EOF Dec 30 07:26:48.905529866 (591h57m48.776612911s) [noise=true , address=145.79.227.17:4972 , piece_id=2QAO4DEM2BL4RRMJRD6L2T3XXEVJWLBW6JD6XOXM2DW3VWGAACLA, seg_pos={0 0}, upload=518946]: Long tail cancellation due to optimal threshold Dec 30 07:26:48.905545416 (591h57m48.776628451s) [seg_pos={0 0}, upload=518946]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 07:26:48.905609698 (591h57m48.776692733s) [upload=518946]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 07:26:48.905613896 (591h57m48.776696931s) [upload=518946]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 07:26:49.197354886 (591h57m49.068437921s) [upload=518946]: Done uploading segments... Dec 30 07:26:49.197373121 (591h57m49.068456156s) [upload=518946]: Upload finished. err: <nil> Duration: 5.670892571s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 15.441125334s trace: https://jaeger.dev.storj.io/trace/1d0a2f93cb06aaa4 FAIL (2025-12-30 07:28:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== loading piece... Dec 30 07:06:00.467281687 (591h37m0.338364722s) [noise=true , address=77.90.31.14:4693 , piece_id=CQ6A6DJJDCKQAAMCXW7FELWCWKAOI7YCCWKVDSTZEBXNHVKQRKPQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.467716555 (591h37m0.33879959s) [noise=true , address=145.79.227.40:4082 , piece_id=7UFAIMVNINXXXRPNXLGJ34YEO3ILVN2O7UFG3HHTIT2DKWVZSUPA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468044320 (591h37m0.339127365s) [noise=true , address=145.79.227.29:4955 , piece_id=CBV7RUAZ6QAZE3VDTZKOGTIMOY66VUJTSYQSZRC7WLUD53UWVJTQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468308362 (591h37m0.339391407s) [noise=true , address=145.79.227.17:4000 , piece_id=KN3NDKB4NODHNXUWWQQ4YBWBPJCNE7YHYEMLQ4UITRCFZB4BRSMA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468565703 (591h37m0.339648748s) [noise=true , address=145.79.227.46:4604 , piece_id=NY5A6JZW4F6E4CSX3ERMKSAL2R3IYSKQ2EVHZSPF6ZOHWQFXBYWA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468843542 (591h37m0.339926587s) [noise=true , address=77.90.31.17:4781 , piece_id=TFUDJVQZFLS2XTZEGBPTV23WYLNHZMGK5ZXNDJSMW5HSNZXWXBBA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.469105310 (591h37m0.340188345s) [noise=true , address=145.79.227.35:4494 , piece_id=U4P6QB4ZRFGAJOHVSSTZHCG4PTBGII4LXAAJXXVN5RVR4WUWUBBQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.470034750 (591h37m0.341117795s) [noise=true , address=145.79.227.49:4365 , piece_id=Z357V6BZEGQOFLFH32RQLYJJZDLURVHZAHKW3H3ZIJXMQC3ZIFBQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.470487622 (591h37m0.341570657s) [noise=true , address=64.49.13.169:4713 , piece_id=YS6FRVS5V3BECH3TJQRJBGB5SSYYGUBK2HMO25PZEUOUSC7ENNZQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.470809334 (591h37m0.341892379s) [noise=true , address=77.90.31.10:4530 , piece_id=LNRM5YW2MZLXHWHGNUMP5I7Y5RRNRESBUA2M3SQKDFEQXZ25RJ3A, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.471118874 (591h37m0.342201919s) [noise=true , address=145.79.227.44:4221 , piece_id=GDKGEN3QCGFIICNST42FA2ZLLTQCHL2QAZ7AMUNPJIGYTSHDMOJQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.471431078 (591h37m0.342514113s) [noise=true , address=45.91.116.4:4963 , piece_id=FUGWNAY33ZGP6J57FIODDQQJ2UH46ABT52AU4Y7Y74Q2AOTCXWFA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.471741530 (591h37m0.342824575s) [noise=true , address=193.149.164.143:4704 , piece_id=PQQZ6N4A7TZGGX5DVRE6NT6TSDJVQARR22F6MZ5Z6LYXLUKF4V7A, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.472073311 (591h37m0.343156356s) [noise=true , address=145.79.227.33:4901 , piece_id=FFTUVNIKVKRAZ5R44V6WLWNKKC5MQYA66RZPSMW6N5EQKY7NSGZQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:01.899399345 (591h37m1.77048239s) [upload=518641]: Next returned nil segment Dec 30 07:06:01.899418712 (591h37m1.770501747s) [upload=518641]: Waiting for error group managing segments... Dec 30 07:06:02.323303785 (591h37m2.194386821s) [noise=true , address=145.79.227.46:4604 , piece_id=NY5A6JZW4F6E4CSX3ERMKSAL2R3IYSKQ2EVHZSPF6ZOHWQFXBYWA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.339689637 (591h37m2.210772682s) [noise=true , address=145.79.227.17:4000 , piece_id=KN3NDKB4NODHNXUWWQQ4YBWBPJCNE7YHYEMLQ4UITRCFZB4BRSMA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.346460570 (591h37m2.217543615s) [noise=true , address=145.79.227.29:4955 , piece_id=CBV7RUAZ6QAZE3VDTZKOGTIMOY66VUJTSYQSZRC7WLUD53UWVJTQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.347028050 (591h37m2.218111095s) [noise=true , address=145.79.227.34:4931 , piece_id=TJ4NTHHJQ4VLPBIC7RW42DBFIBI5G7LJWBN53TPSEVAWHANHMTBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.427966770 (591h37m2.299049815s) [noise=true , address=145.79.227.5:4226 , piece_id=QKDFEH6UKIH4QYN6DE3YNH3FZPAYGNYMQSBCYGM7EAW7F4MDA66A, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.446440007 (591h37m2.317523052s) [noise=true , address=145.79.227.35:4494 , piece_id=U4P6QB4ZRFGAJOHVSSTZHCG4PTBGII4LXAAJXXVN5RVR4WUWUBBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.583477598 (591h37m2.454560643s) [noise=true , address=77.90.31.19:4859 , piece_id=XHNZMPC2MF4RALF2FZJY73UYN4XWOYJUL66RFRUXNPT64JZQE5KQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.613112418 (591h37m2.484195463s) [noise=true , address=77.90.31.17:4781 , piece_id=TFUDJVQZFLS2XTZEGBPTV23WYLNHZMGK5ZXNDJSMW5HSNZXWXBBA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.631421591 (591h37m2.502504626s) [noise=true , address=77.90.31.10:4530 , piece_id=LNRM5YW2MZLXHWHGNUMP5I7Y5RRNRESBUA2M3SQKDFEQXZ25RJ3A, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.944663583 (591h37m2.815746628s) [noise=true , address=145.79.227.49:4365 , piece_id=Z357V6BZEGQOFLFH32RQLYJJZDLURVHZAHKW3H3ZIJXMQC3ZIFBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.195837408 (591h37m3.066920453s) [noise=true , address=193.149.164.143:4704 , piece_id=PQQZ6N4A7TZGGX5DVRE6NT6TSDJVQARR22F6MZ5Z6LYXLUKF4V7A, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.269365478 (591h37m3.140448523s) [noise=true , address=145.79.227.44:4221 , piece_id=GDKGEN3QCGFIICNST42FA2ZLLTQCHL2QAZ7AMUNPJIGYTSHDMOJQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.272362485 (591h37m3.14344553s) [noise=true , address=145.79.227.30:4778 , piece_id=EVWGEZL3VCMHEM6ALESBLMU65XBE74CKVMFXMSHK2GCXFPI35RBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.340891504 (591h37m3.211974549s) [noise=true , address=145.79.227.40:4082 , piece_id=7UFAIMVNINXXXRPNXLGJ34YEO3ILVN2O7UFG3HHTIT2DKWVZSUPA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.340922954 (591h37m3.212005999s) [seg_pos={0 0}, upload=518641]: Segment reached optimal threshold of 14 pieces. Dec 30 07:06:03.341286095 (591h37m3.21236913s) [noise=true , address=77.90.31.14:4693 , piece_id=CQ6A6DJJDCKQAAMCXW7FELWCWKAOI7YCCWKVDSTZEBXNHVKQRKPQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RASrnvpAj4ToD4xNvbma1iKZEe2uHK8WosjfZwDDom8Qd17Ey): protocol: write tcp 188.166.225.170:47364->77.90.31.14:4693: use of closed network connection; write tcp 188.166.225.170:47364->77.90.31.14:4693: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47364->77.90.31.14:4693: use of closed network connection Dec 30 07:06:03.341317094 (591h37m3.212400139s) [noise=true , address=77.90.31.14:4693 , piece_id=CQ6A6DJJDCKQAAMCXW7FELWCWKAOI7YCCWKVDSTZEBXNHVKQRKPQ, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341322114 (591h37m3.212405159s) [noise=true , address=145.79.227.33:4901 , piece_id=FFTUVNIKVKRAZ5R44V6WLWNKKC5MQYA66RZPSMW6N5EQKY7NSGZQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1): protocol: expected piece hash; context canceled; EOF Dec 30 07:06:03.341336261 (591h37m3.212419306s) [noise=true , address=145.79.227.33:4901 , piece_id=FFTUVNIKVKRAZ5R44V6WLWNKKC5MQYA66RZPSMW6N5EQKY7NSGZQ, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341546141 (591h37m3.212629196s) [noise=true , address=45.91.116.4:4963 , piece_id=FUGWNAY33ZGP6J57FIODDQQJ2UH46ABT52AU4Y7Y74Q2AOTCXWFA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Eki7jYegwvkdixR5pznQBbSeJPn9GjGofxnUApidQF4kRNBK5): protocol: expected piece hash; write tcp 188.166.225.170:51728->45.91.116.4:4963: use of closed network connection; write tcp 188.166.225.170:51728->45.91.116.4:4963: use of closed network connection; EOF Dec 30 07:06:03.341555549 (591h37m3.212638594s) [noise=true , address=45.91.116.4:4963 , piece_id=FUGWNAY33ZGP6J57FIODDQQJ2UH46ABT52AU4Y7Y74Q2AOTCXWFA, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341759577 (591h37m3.212842622s) [noise=true , address=64.49.13.169:4713 , piece_id=YS6FRVS5V3BECH3TJQRJBGB5SSYYGUBK2HMO25PZEUOUSC7ENNZQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17z3AiUfbxBX2eMDnUnqo1bbgojuQxJro7PsEaPjYzQjHQ3D6K): protocol: write tcp 188.166.225.170:42002->64.49.13.169:4713: use of closed network connection; write tcp 188.166.225.170:42002->64.49.13.169:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42002->64.49.13.169:4713: use of closed network connection Dec 30 07:06:03.341769897 (591h37m3.212852942s) [noise=true , address=64.49.13.169:4713 , piece_id=YS6FRVS5V3BECH3TJQRJBGB5SSYYGUBK2HMO25PZEUOUSC7ENNZQ, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341869927 (591h37m3.212952972s) [seg_pos={0 0}, upload=518641]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 07:06:03.342018730 (591h37m3.213101775s) [upload=518641]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 07:06:03.342025634 (591h37m3.213108679s) [upload=518641]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 07:06:03.635315934 (591h37m3.506398979s) [upload=518641]: Done uploading segments... Dec 30 07:06:03.635343146 (591h37m3.506426191s) [upload=518641]: Upload finished. err: <nil> Duration: 15.441151815s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 19 out of 60 is failed (threshold=6) [...X....XXX...XXXX.........X.X.....X..XXXXXX.X.....X........] duration: 3.958234668s trace: https://jaeger.dev.storj.io/trace/220b42f7e5b29694 FAIL (2025-12-30 07:28:44 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.958258954s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [XX...X.....X....XXX..X.....X...X..X.XX..X...X....XXX....X...] duration: 8.603406011s trace: https://jaeger.dev.storj.io/trace/5b5a31fe4475073 FAIL (2025-12-30 07:28:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 07:26:46.289184198 (591h57m46.160267243s) [upload=518946]: Starting upload Dec 30 07:26:46.289281163 (591h57m46.160364218s) [upload=518946]: Done starting upload Dec 30 07:26:46.289508094 (591h57m46.160591129s) [upload=518946]: Uploading segments... Dec 30 07:26:46.289589440 (591h57m46.160672495s) [upload=518946]: Got next segment. Inline: false Dec 30 07:26:46.289633714 (591h57m46.160716759s) [upload=518946]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 07:26:46.289639164 (591h57m46.160722209s) [upload=518946]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 07:26:46.532324195 (591h57m46.40340724s) [seg_pos={0 0}, upload=518946]: Begin upload segment... Dec 30 07:26:46.532410239 (591h57m46.403493274s) [seg_pos={0 0}, upload=518946]: Done begin upload segment. Dec 30 07:26:46.532448602 (591h57m46.403531637s) [noise=true , address=145.79.227.2:4114 , piece_id=F6QSNFGWCWQDQUCOICC3GNCHUPERG6PWBO2OIAQDCGRHPS5YA2PA, seg_pos={0 0}, upload=518946]: Uploading piece... Dec 30 07:26:46.532504338 (591h57m46.403587383s) [noise=true , address=45.91.116.8:4046 , piece_id=5YF5RU2ICVQJLAG43MN4K3AHEHGOQBQ57ECEKAUUX75XHJNKVNRQ, seg_pos={0 0}, upload=518946]: Uploading piece... Dec 30 07:26:46.533077460 (591h57m46.404160505s) [noise=true , address=145.79.227.39:4015 , piece_id=NGGRJ4EFWFGJSHIQ3YHBS3SQZVO4N4UWXIAB6JA32K6PVAUC5AZQ, seg_pos={0 0}, upload=518946]: Uploading piece... Dec 30 07:26:46.533549498 (591h57m46.404632533s) [noise=true , address=145.79.227.19:4078 , piece_id=AMXZNOQCZXM7LGL46SR7CS7CZ55BGTRTOZ5DJAYDMTBGB5NNDC3Q, seg_pos={0 0}, upload=518946]: Uploading piece... Dec 30 07:26:46.534157366 (591h57m46.405240401s) [noise=true , address=145.79.227.41:4085 , piece_id=7RGSCLOZ46YHEPE5ZKU6QMQHNW7SLV3VN6NCURYAUWBV2RWRHEEQ, seg_pos={0 0}, upload=518946]: Uploading piece... Dec 30 07:26:46.534782767 (591h57m46.405865802s) [noise=true , address=145.79.227.15:4874 , piece_id=GY5FWVL6PE77LP7455DHVEMQ2VKV3W6XL5NHN4PPJ7F45HPUC4IA, seg_pos={0 0}, upload=518946]: Uploading piece... Dec 30 07:26:46.535251168 (591h57m46.406334213s) [noise=true , address=145.79.227.51:4411 , piece_id=UIKRX5OVZIFAKUBTQQGRYMPMIXSFY43VUBC2KRCCGWWMFUWD2LSQ, seg_pos={0 0}, upload=518946]: Uploading piece... Dec 30 07:26:46.536749040 (591h57m46.407832085s) [noise=true , address=145.79.227.36:4517 , piece_id=DKJ2BMYTOU46NNNK6FRXT7V2V3UXRQPZURV3RH6CMU3BFCBIJC3Q, seg_pos={0 0}, upload=518946]: Uploading piece... Dec 30 07:26:46.538410154 (591h57m46.409493209s) [noise=true , address=145.79.227.14:4840 , piece_id=NI6LJNVGBHKOR4SAM7S5V3IRKEU4DG6ABLBI3VAZKLF4DOWXMJLQ, seg_pos={0 0}, upload=518946]: Uploading piece... Dec 30 07:26:46.539727031 (591h57m46.410810066s) [noise=true , address=45.91.116.5:4300 , piece_id=YZMX3AQIOSGZNCOGFBPEQ4DOIWQYX23SHVGMTNGDTIEEYTGGKVQQ, seg_pos={0 0}, upload=518946]: Uploading piece... Dec 30 07:26:46.541085869 (591h57m46.412168904s) [noise=true , address=145.79.227.32:4835 , piece_id=4HD4E2ME5CEPRUFERUH6NEFFFFGUYX2GLU2GF4OLB5SIMZCCGRQA, seg_pos={0 0}, upload=518946]: Uploading piece... Dec 30 07:26:46.541465432 (591h57m46.412548467s) [noise=true , address=145.79.227.17:4972 , piece_id=2QAO4DEM2BL4RRMJRD6L2T3XXEVJWLBW6JD6XOXM2DW3VWGAACLA, seg_pos={0 0}, upload=518946]: Uploading piece... Dec 30 07:26:46.541630516 (591h57m46.412713551s) [noise=true , address=45.91.116.4:4939 , piece_id=EX42IOXR53GUHE7FD7SZBESKBKIZQFG6EXOWP5N2SZKMN2WQIBEA, seg_pos={0 0}, upload=518946]: Uploading piece... Dec 30 07:26:46.543860943 (591h57m46.414943978s) [noise=true , address=145.79.227.22:4509 , piece_id=GXGMXCJKULDJLXTM2AGJA5VRVCVNRHKNV5NB2TTESXQ5TVPAYZKQ, seg_pos={0 0}, upload=518946]: Uploading piece... Dec 30 07:26:46.545096676 (591h57m46.416179711s) [noise=true , address=145.79.227.46:4603 , piece_id=OSFA5IDUJ5DGRO62ORYYSJFGDOF2DWVLENETXJU2DP7JMPUXFR4Q, seg_pos={0 0}, upload=518946]: Uploading piece... Dec 30 07:26:46.546477646 (591h57m46.417560691s) [noise=true , address=77.90.31.18:4805 , piece_id=PC3THDCHXRWOJPPGYCM4XVTUWJQJHHKIHUWU34YAECOYWS5IBX4Q, seg_pos={0 0}, upload=518946]: Uploading piece... Dec 30 07:26:48.130107348 (591h57m48.001190383s) [upload=518946]: Next returned nil segment Dec 30 07:26:48.130126845 (591h57m48.00120988s) [upload=518946]: Waiting for error group managing segments... Dec 30 07:26:48.522160289 (591h57m48.393243334s) [noise=true , address=145.79.227.2:4114 , piece_id=F6QSNFGWCWQDQUCOICC3GNCHUPERG6PWBO2OIAQDCGRHPS5YA2PA, seg_pos={0 0}, upload=518946]: Done uploading piece. err: <nil> Dec 30 07:26:48.529146272 (591h57m48.400229327s) [noise=true , address=145.79.227.46:4603 , piece_id=OSFA5IDUJ5DGRO62ORYYSJFGDOF2DWVLENETXJU2DP7JMPUXFR4Q, seg_pos={0 0}, upload=518946]: Done uploading piece. err: <nil> Dec 30 07:26:48.532844023 (591h57m48.403927058s) [noise=true , address=145.79.227.15:4874 , piece_id=GY5FWVL6PE77LP7455DHVEMQ2VKV3W6XL5NHN4PPJ7F45HPUC4IA, seg_pos={0 0}, upload=518946]: Done uploading piece. err: <nil> Dec 30 07:26:48.534193653 (591h57m48.405276688s) [noise=true , address=145.79.227.22:4509 , piece_id=GXGMXCJKULDJLXTM2AGJA5VRVCVNRHKNV5NB2TTESXQ5TVPAYZKQ, seg_pos={0 0}, upload=518946]: Done uploading piece. err: <nil> Dec 30 07:26:48.547639556 (591h57m48.418722631s) [noise=true , address=145.79.227.32:4835 , piece_id=4HD4E2ME5CEPRUFERUH6NEFFFFGUYX2GLU2GF4OLB5SIMZCCGRQA, seg_pos={0 0}, upload=518946]: Done uploading piece. err: <nil> Dec 30 07:26:48.554344823 (591h57m48.425427858s) [noise=true , address=145.79.227.39:4015 , piece_id=NGGRJ4EFWFGJSHIQ3YHBS3SQZVO4N4UWXIAB6JA32K6PVAUC5AZQ, seg_pos={0 0}, upload=518946]: Done uploading piece. err: <nil> Dec 30 07:26:48.555384473 (591h57m48.426467508s) [noise=true , address=145.79.227.51:4411 , piece_id=UIKRX5OVZIFAKUBTQQGRYMPMIXSFY43VUBC2KRCCGWWMFUWD2LSQ, seg_pos={0 0}, upload=518946]: Done uploading piece. err: <nil> Dec 30 07:26:48.557932195 (591h57m48.42901523s) [noise=true , address=145.79.227.36:4517 , piece_id=DKJ2BMYTOU46NNNK6FRXT7V2V3UXRQPZURV3RH6CMU3BFCBIJC3Q, seg_pos={0 0}, upload=518946]: Done uploading piece. err: <nil> Dec 30 07:26:48.601713982 (591h57m48.472797017s) [noise=true , address=145.79.227.19:4078 , piece_id=AMXZNOQCZXM7LGL46SR7CS7CZ55BGTRTOZ5DJAYDMTBGB5NNDC3Q, seg_pos={0 0}, upload=518946]: Done uploading piece. err: <nil> Dec 30 07:26:48.607727713 (591h57m48.478810748s) [noise=true , address=145.79.227.14:4840 , piece_id=NI6LJNVGBHKOR4SAM7S5V3IRKEU4DG6ABLBI3VAZKLF4DOWXMJLQ, seg_pos={0 0}, upload=518946]: Done uploading piece. err: <nil> Dec 30 07:26:48.613029328 (591h57m48.484112363s) [noise=true , address=145.79.227.41:4085 , piece_id=7RGSCLOZ46YHEPE5ZKU6QMQHNW7SLV3VN6NCURYAUWBV2RWRHEEQ, seg_pos={0 0}, upload=518946]: Done uploading piece. err: <nil> Dec 30 07:26:48.787823986 (591h57m48.658907021s) [noise=true , address=77.90.31.18:4805 , piece_id=PC3THDCHXRWOJPPGYCM4XVTUWJQJHHKIHUWU34YAECOYWS5IBX4Q, seg_pos={0 0}, upload=518946]: Done uploading piece. err: <nil> Dec 30 07:26:48.905112491 (591h57m48.776195526s) [noise=true , address=45.91.116.8:4046 , piece_id=5YF5RU2ICVQJLAG43MN4K3AHEHGOQBQ57ECEKAUUX75XHJNKVNRQ, seg_pos={0 0}, upload=518946]: Done uploading piece. err: <nil> Dec 30 07:26:48.905132369 (591h57m48.776215404s) [seg_pos={0 0}, upload=518946]: Segment reached optimal threshold of 13 pieces. Dec 30 07:26:48.905318543 (591h57m48.776401588s) [noise=true , address=45.91.116.4:4939 , piece_id=EX42IOXR53GUHE7FD7SZBESKBKIZQFG6EXOWP5N2SZKMN2WQIBEA, seg_pos={0 0}, upload=518946]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 30 07:26:48.905324355 (591h57m48.77640739s) [noise=true , address=45.91.116.4:4939 , piece_id=EX42IOXR53GUHE7FD7SZBESKBKIZQFG6EXOWP5N2SZKMN2WQIBEA, seg_pos={0 0}, upload=518946]: Long tail cancellation due to optimal threshold Dec 30 07:26:48.905423974 (591h57m48.776507009s) [noise=true , address=45.91.116.5:4300 , piece_id=YZMX3AQIOSGZNCOGFBPEQ4DOIWQYX23SHVGMTNGDTIEEYTGGKVQQ, seg_pos={0 0}, upload=518946]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Dec 30 07:26:48.905429255 (591h57m48.77651229s) [noise=true , address=45.91.116.5:4300 , piece_id=YZMX3AQIOSGZNCOGFBPEQ4DOIWQYX23SHVGMTNGDTIEEYTGGKVQQ, seg_pos={0 0}, upload=518946]: Long tail cancellation due to optimal threshold Dec 30 07:26:48.905524476 (591h57m48.776607511s) [noise=true , address=145.79.227.17:4972 , piece_id=2QAO4DEM2BL4RRMJRD6L2T3XXEVJWLBW6JD6XOXM2DW3VWGAACLA, seg_pos={0 0}, upload=518946]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: expected piece hash; context canceled; EOF Dec 30 07:26:48.905529866 (591h57m48.776612911s) [noise=true , address=145.79.227.17:4972 , piece_id=2QAO4DEM2BL4RRMJRD6L2T3XXEVJWLBW6JD6XOXM2DW3VWGAACLA, seg_pos={0 0}, upload=518946]: Long tail cancellation due to optimal threshold Dec 30 07:26:48.905545416 (591h57m48.776628451s) [seg_pos={0 0}, upload=518946]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 07:26:48.905609698 (591h57m48.776692733s) [upload=518946]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 07:26:48.905613896 (591h57m48.776696931s) [upload=518946]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 07:26:49.197354886 (591h57m49.068437921s) [upload=518946]: Done uploading segments... Dec 30 07:26:49.197373121 (591h57m49.068456156s) [upload=518946]: Upload finished. err: <nil> Duration: 8.60343233s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 41.548759399s trace: https://jaeger.dev.storj.io/trace/1d0a2f93cb06aaa4 FAIL (2025-12-30 07:28:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== loading piece... Dec 30 07:06:00.467281687 (591h37m0.338364722s) [noise=true , address=77.90.31.14:4693 , piece_id=CQ6A6DJJDCKQAAMCXW7FELWCWKAOI7YCCWKVDSTZEBXNHVKQRKPQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.467716555 (591h37m0.33879959s) [noise=true , address=145.79.227.40:4082 , piece_id=7UFAIMVNINXXXRPNXLGJ34YEO3ILVN2O7UFG3HHTIT2DKWVZSUPA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468044320 (591h37m0.339127365s) [noise=true , address=145.79.227.29:4955 , piece_id=CBV7RUAZ6QAZE3VDTZKOGTIMOY66VUJTSYQSZRC7WLUD53UWVJTQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468308362 (591h37m0.339391407s) [noise=true , address=145.79.227.17:4000 , piece_id=KN3NDKB4NODHNXUWWQQ4YBWBPJCNE7YHYEMLQ4UITRCFZB4BRSMA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468565703 (591h37m0.339648748s) [noise=true , address=145.79.227.46:4604 , piece_id=NY5A6JZW4F6E4CSX3ERMKSAL2R3IYSKQ2EVHZSPF6ZOHWQFXBYWA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468843542 (591h37m0.339926587s) [noise=true , address=77.90.31.17:4781 , piece_id=TFUDJVQZFLS2XTZEGBPTV23WYLNHZMGK5ZXNDJSMW5HSNZXWXBBA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.469105310 (591h37m0.340188345s) [noise=true , address=145.79.227.35:4494 , piece_id=U4P6QB4ZRFGAJOHVSSTZHCG4PTBGII4LXAAJXXVN5RVR4WUWUBBQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.470034750 (591h37m0.341117795s) [noise=true , address=145.79.227.49:4365 , piece_id=Z357V6BZEGQOFLFH32RQLYJJZDLURVHZAHKW3H3ZIJXMQC3ZIFBQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.470487622 (591h37m0.341570657s) [noise=true , address=64.49.13.169:4713 , piece_id=YS6FRVS5V3BECH3TJQRJBGB5SSYYGUBK2HMO25PZEUOUSC7ENNZQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.470809334 (591h37m0.341892379s) [noise=true , address=77.90.31.10:4530 , piece_id=LNRM5YW2MZLXHWHGNUMP5I7Y5RRNRESBUA2M3SQKDFEQXZ25RJ3A, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.471118874 (591h37m0.342201919s) [noise=true , address=145.79.227.44:4221 , piece_id=GDKGEN3QCGFIICNST42FA2ZLLTQCHL2QAZ7AMUNPJIGYTSHDMOJQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.471431078 (591h37m0.342514113s) [noise=true , address=45.91.116.4:4963 , piece_id=FUGWNAY33ZGP6J57FIODDQQJ2UH46ABT52AU4Y7Y74Q2AOTCXWFA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.471741530 (591h37m0.342824575s) [noise=true , address=193.149.164.143:4704 , piece_id=PQQZ6N4A7TZGGX5DVRE6NT6TSDJVQARR22F6MZ5Z6LYXLUKF4V7A, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.472073311 (591h37m0.343156356s) [noise=true , address=145.79.227.33:4901 , piece_id=FFTUVNIKVKRAZ5R44V6WLWNKKC5MQYA66RZPSMW6N5EQKY7NSGZQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:01.899399345 (591h37m1.77048239s) [upload=518641]: Next returned nil segment Dec 30 07:06:01.899418712 (591h37m1.770501747s) [upload=518641]: Waiting for error group managing segments... Dec 30 07:06:02.323303785 (591h37m2.194386821s) [noise=true , address=145.79.227.46:4604 , piece_id=NY5A6JZW4F6E4CSX3ERMKSAL2R3IYSKQ2EVHZSPF6ZOHWQFXBYWA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.339689637 (591h37m2.210772682s) [noise=true , address=145.79.227.17:4000 , piece_id=KN3NDKB4NODHNXUWWQQ4YBWBPJCNE7YHYEMLQ4UITRCFZB4BRSMA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.346460570 (591h37m2.217543615s) [noise=true , address=145.79.227.29:4955 , piece_id=CBV7RUAZ6QAZE3VDTZKOGTIMOY66VUJTSYQSZRC7WLUD53UWVJTQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.347028050 (591h37m2.218111095s) [noise=true , address=145.79.227.34:4931 , piece_id=TJ4NTHHJQ4VLPBIC7RW42DBFIBI5G7LJWBN53TPSEVAWHANHMTBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.427966770 (591h37m2.299049815s) [noise=true , address=145.79.227.5:4226 , piece_id=QKDFEH6UKIH4QYN6DE3YNH3FZPAYGNYMQSBCYGM7EAW7F4MDA66A, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.446440007 (591h37m2.317523052s) [noise=true , address=145.79.227.35:4494 , piece_id=U4P6QB4ZRFGAJOHVSSTZHCG4PTBGII4LXAAJXXVN5RVR4WUWUBBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.583477598 (591h37m2.454560643s) [noise=true , address=77.90.31.19:4859 , piece_id=XHNZMPC2MF4RALF2FZJY73UYN4XWOYJUL66RFRUXNPT64JZQE5KQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.613112418 (591h37m2.484195463s) [noise=true , address=77.90.31.17:4781 , piece_id=TFUDJVQZFLS2XTZEGBPTV23WYLNHZMGK5ZXNDJSMW5HSNZXWXBBA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.631421591 (591h37m2.502504626s) [noise=true , address=77.90.31.10:4530 , piece_id=LNRM5YW2MZLXHWHGNUMP5I7Y5RRNRESBUA2M3SQKDFEQXZ25RJ3A, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.944663583 (591h37m2.815746628s) [noise=true , address=145.79.227.49:4365 , piece_id=Z357V6BZEGQOFLFH32RQLYJJZDLURVHZAHKW3H3ZIJXMQC3ZIFBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.195837408 (591h37m3.066920453s) [noise=true , address=193.149.164.143:4704 , piece_id=PQQZ6N4A7TZGGX5DVRE6NT6TSDJVQARR22F6MZ5Z6LYXLUKF4V7A, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.269365478 (591h37m3.140448523s) [noise=true , address=145.79.227.44:4221 , piece_id=GDKGEN3QCGFIICNST42FA2ZLLTQCHL2QAZ7AMUNPJIGYTSHDMOJQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.272362485 (591h37m3.14344553s) [noise=true , address=145.79.227.30:4778 , piece_id=EVWGEZL3VCMHEM6ALESBLMU65XBE74CKVMFXMSHK2GCXFPI35RBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.340891504 (591h37m3.211974549s) [noise=true , address=145.79.227.40:4082 , piece_id=7UFAIMVNINXXXRPNXLGJ34YEO3ILVN2O7UFG3HHTIT2DKWVZSUPA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.340922954 (591h37m3.212005999s) [seg_pos={0 0}, upload=518641]: Segment reached optimal threshold of 14 pieces. Dec 30 07:06:03.341286095 (591h37m3.21236913s) [noise=true , address=77.90.31.14:4693 , piece_id=CQ6A6DJJDCKQAAMCXW7FELWCWKAOI7YCCWKVDSTZEBXNHVKQRKPQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RASrnvpAj4ToD4xNvbma1iKZEe2uHK8WosjfZwDDom8Qd17Ey): protocol: write tcp 188.166.225.170:47364->77.90.31.14:4693: use of closed network connection; write tcp 188.166.225.170:47364->77.90.31.14:4693: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47364->77.90.31.14:4693: use of closed network connection Dec 30 07:06:03.341317094 (591h37m3.212400139s) [noise=true , address=77.90.31.14:4693 , piece_id=CQ6A6DJJDCKQAAMCXW7FELWCWKAOI7YCCWKVDSTZEBXNHVKQRKPQ, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341322114 (591h37m3.212405159s) [noise=true , address=145.79.227.33:4901 , piece_id=FFTUVNIKVKRAZ5R44V6WLWNKKC5MQYA66RZPSMW6N5EQKY7NSGZQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1): protocol: expected piece hash; context canceled; EOF Dec 30 07:06:03.341336261 (591h37m3.212419306s) [noise=true , address=145.79.227.33:4901 , piece_id=FFTUVNIKVKRAZ5R44V6WLWNKKC5MQYA66RZPSMW6N5EQKY7NSGZQ, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341546141 (591h37m3.212629196s) [noise=true , address=45.91.116.4:4963 , piece_id=FUGWNAY33ZGP6J57FIODDQQJ2UH46ABT52AU4Y7Y74Q2AOTCXWFA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Eki7jYegwvkdixR5pznQBbSeJPn9GjGofxnUApidQF4kRNBK5): protocol: expected piece hash; write tcp 188.166.225.170:51728->45.91.116.4:4963: use of closed network connection; write tcp 188.166.225.170:51728->45.91.116.4:4963: use of closed network connection; EOF Dec 30 07:06:03.341555549 (591h37m3.212638594s) [noise=true , address=45.91.116.4:4963 , piece_id=FUGWNAY33ZGP6J57FIODDQQJ2UH46ABT52AU4Y7Y74Q2AOTCXWFA, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341759577 (591h37m3.212842622s) [noise=true , address=64.49.13.169:4713 , piece_id=YS6FRVS5V3BECH3TJQRJBGB5SSYYGUBK2HMO25PZEUOUSC7ENNZQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17z3AiUfbxBX2eMDnUnqo1bbgojuQxJro7PsEaPjYzQjHQ3D6K): protocol: write tcp 188.166.225.170:42002->64.49.13.169:4713: use of closed network connection; write tcp 188.166.225.170:42002->64.49.13.169:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42002->64.49.13.169:4713: use of closed network connection Dec 30 07:06:03.341769897 (591h37m3.212852942s) [noise=true , address=64.49.13.169:4713 , piece_id=YS6FRVS5V3BECH3TJQRJBGB5SSYYGUBK2HMO25PZEUOUSC7ENNZQ, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341869927 (591h37m3.212952972s) [seg_pos={0 0}, upload=518641]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 07:06:03.342018730 (591h37m3.213101775s) [upload=518641]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 07:06:03.342025634 (591h37m3.213108679s) [upload=518641]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 07:06:03.635315934 (591h37m3.506398979s) [upload=518641]: Done uploading segments... Dec 30 07:06:03.635343146 (591h37m3.506426191s) [upload=518641]: Upload finished. err: <nil> Duration: 41.548774037s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [.XX...X.....X....XXX..X.....X...X..X.XX..X...X....XXX....X..] duration: 5.893024434s trace: https://jaeger.dev.storj.io/trace/5b5a31fe4475073 FAIL (2025-12-30 07:27:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 07:26:46.289184198 (591h57m46.160267243s) [upload=518946]: Starting upload Dec 30 07:26:46.289281163 (591h57m46.160364218s) [upload=518946]: Done starting upload Dec 30 07:26:46.289508094 (591h57m46.160591129s) [upload=518946]: Uploading segments... Dec 30 07:26:46.289589440 (591h57m46.160672495s) [upload=518946]: Got next segment. Inline: false Dec 30 07:26:46.289633714 (591h57m46.160716759s) [upload=518946]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 07:26:46.289639164 (591h57m46.160722209s) [upload=518946]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 07:26:46.532324195 (591h57m46.40340724s) [seg_pos={0 0}, upload=518946]: Begin upload segment... Dec 30 07:26:46.532410239 (591h57m46.403493274s) [seg_pos={0 0}, upload=518946]: Done begin upload segment. Dec 30 07:26:46.532448602 (591h57m46.403531637s) [noise=true , address=145.79.227.2:4114 , piece_id=F6QSNFGWCWQDQUCOICC3GNCHUPERG6PWBO2OIAQDCGRHPS5YA2PA, seg_pos={0 0}, upload=518946]: Uploading piece... Dec 30 07:26:46.532504338 (591h57m46.403587383s) [noise=true , address=45.91.116.8:4046 , piece_id=5YF5RU2ICVQJLAG43MN4K3AHEHGOQBQ57ECEKAUUX75XHJNKVNRQ, seg_pos={0 0}, upload=518946]: Uploading piece... Dec 30 07:26:46.533077460 (591h57m46.404160505s) [noise=true , address=145.79.227.39:4015 , piece_id=NGGRJ4EFWFGJSHIQ3YHBS3SQZVO4N4UWXIAB6JA32K6PVAUC5AZQ, seg_pos={0 0}, upload=518946]: Uploading piece... Dec 30 07:26:46.533549498 (591h57m46.404632533s) [noise=true , address=145.79.227.19:4078 , piece_id=AMXZNOQCZXM7LGL46SR7CS7CZ55BGTRTOZ5DJAYDMTBGB5NNDC3Q, seg_pos={0 0}, upload=518946]: Uploading piece... Dec 30 07:26:46.534157366 (591h57m46.405240401s) [noise=true , address=145.79.227.41:4085 , piece_id=7RGSCLOZ46YHEPE5ZKU6QMQHNW7SLV3VN6NCURYAUWBV2RWRHEEQ, seg_pos={0 0}, upload=518946]: Uploading piece... Dec 30 07:26:46.534782767 (591h57m46.405865802s) [noise=true , address=145.79.227.15:4874 , piece_id=GY5FWVL6PE77LP7455DHVEMQ2VKV3W6XL5NHN4PPJ7F45HPUC4IA, seg_pos={0 0}, upload=518946]: Uploading piece... Dec 30 07:26:46.535251168 (591h57m46.406334213s) [noise=true , address=145.79.227.51:4411 , piece_id=UIKRX5OVZIFAKUBTQQGRYMPMIXSFY43VUBC2KRCCGWWMFUWD2LSQ, seg_pos={0 0}, upload=518946]: Uploading piece... Dec 30 07:26:46.536749040 (591h57m46.407832085s) [noise=true , address=145.79.227.36:4517 , piece_id=DKJ2BMYTOU46NNNK6FRXT7V2V3UXRQPZURV3RH6CMU3BFCBIJC3Q, seg_pos={0 0}, upload=518946]: Uploading piece... Dec 30 07:26:46.538410154 (591h57m46.409493209s) [noise=true , address=145.79.227.14:4840 , piece_id=NI6LJNVGBHKOR4SAM7S5V3IRKEU4DG6ABLBI3VAZKLF4DOWXMJLQ, seg_pos={0 0}, upload=518946]: Uploading piece... Dec 30 07:26:46.539727031 (591h57m46.410810066s) [noise=true , address=45.91.116.5:4300 , piece_id=YZMX3AQIOSGZNCOGFBPEQ4DOIWQYX23SHVGMTNGDTIEEYTGGKVQQ, seg_pos={0 0}, upload=518946]: Uploading piece... Dec 30 07:26:46.541085869 (591h57m46.412168904s) [noise=true , address=145.79.227.32:4835 , piece_id=4HD4E2ME5CEPRUFERUH6NEFFFFGUYX2GLU2GF4OLB5SIMZCCGRQA, seg_pos={0 0}, upload=518946]: Uploading piece... Dec 30 07:26:46.541465432 (591h57m46.412548467s) [noise=true , address=145.79.227.17:4972 , piece_id=2QAO4DEM2BL4RRMJRD6L2T3XXEVJWLBW6JD6XOXM2DW3VWGAACLA, seg_pos={0 0}, upload=518946]: Uploading piece... Dec 30 07:26:46.541630516 (591h57m46.412713551s) [noise=true , address=45.91.116.4:4939 , piece_id=EX42IOXR53GUHE7FD7SZBESKBKIZQFG6EXOWP5N2SZKMN2WQIBEA, seg_pos={0 0}, upload=518946]: Uploading piece... Dec 30 07:26:46.543860943 (591h57m46.414943978s) [noise=true , address=145.79.227.22:4509 , piece_id=GXGMXCJKULDJLXTM2AGJA5VRVCVNRHKNV5NB2TTESXQ5TVPAYZKQ, seg_pos={0 0}, upload=518946]: Uploading piece... Dec 30 07:26:46.545096676 (591h57m46.416179711s) [noise=true , address=145.79.227.46:4603 , piece_id=OSFA5IDUJ5DGRO62ORYYSJFGDOF2DWVLENETXJU2DP7JMPUXFR4Q, seg_pos={0 0}, upload=518946]: Uploading piece... Dec 30 07:26:46.546477646 (591h57m46.417560691s) [noise=true , address=77.90.31.18:4805 , piece_id=PC3THDCHXRWOJPPGYCM4XVTUWJQJHHKIHUWU34YAECOYWS5IBX4Q, seg_pos={0 0}, upload=518946]: Uploading piece... Dec 30 07:26:48.130107348 (591h57m48.001190383s) [upload=518946]: Next returned nil segment Dec 30 07:26:48.130126845 (591h57m48.00120988s) [upload=518946]: Waiting for error group managing segments... Dec 30 07:26:48.522160289 (591h57m48.393243334s) [noise=true , address=145.79.227.2:4114 , piece_id=F6QSNFGWCWQDQUCOICC3GNCHUPERG6PWBO2OIAQDCGRHPS5YA2PA, seg_pos={0 0}, upload=518946]: Done uploading piece. err: <nil> Dec 30 07:26:48.529146272 (591h57m48.400229327s) [noise=true , address=145.79.227.46:4603 , piece_id=OSFA5IDUJ5DGRO62ORYYSJFGDOF2DWVLENETXJU2DP7JMPUXFR4Q, seg_pos={0 0}, upload=518946]: Done uploading piece. err: <nil> Dec 30 07:26:48.532844023 (591h57m48.403927058s) [noise=true , address=145.79.227.15:4874 , piece_id=GY5FWVL6PE77LP7455DHVEMQ2VKV3W6XL5NHN4PPJ7F45HPUC4IA, seg_pos={0 0}, upload=518946]: Done uploading piece. err: <nil> Dec 30 07:26:48.534193653 (591h57m48.405276688s) [noise=true , address=145.79.227.22:4509 , piece_id=GXGMXCJKULDJLXTM2AGJA5VRVCVNRHKNV5NB2TTESXQ5TVPAYZKQ, seg_pos={0 0}, upload=518946]: Done uploading piece. err: <nil> Dec 30 07:26:48.547639556 (591h57m48.418722631s) [noise=true , address=145.79.227.32:4835 , piece_id=4HD4E2ME5CEPRUFERUH6NEFFFFGUYX2GLU2GF4OLB5SIMZCCGRQA, seg_pos={0 0}, upload=518946]: Done uploading piece. err: <nil> Dec 30 07:26:48.554344823 (591h57m48.425427858s) [noise=true , address=145.79.227.39:4015 , piece_id=NGGRJ4EFWFGJSHIQ3YHBS3SQZVO4N4UWXIAB6JA32K6PVAUC5AZQ, seg_pos={0 0}, upload=518946]: Done uploading piece. err: <nil> Dec 30 07:26:48.555384473 (591h57m48.426467508s) [noise=true , address=145.79.227.51:4411 , piece_id=UIKRX5OVZIFAKUBTQQGRYMPMIXSFY43VUBC2KRCCGWWMFUWD2LSQ, seg_pos={0 0}, upload=518946]: Done uploading piece. err: <nil> Dec 30 07:26:48.557932195 (591h57m48.42901523s) [noise=true , address=145.79.227.36:4517 , piece_id=DKJ2BMYTOU46NNNK6FRXT7V2V3UXRQPZURV3RH6CMU3BFCBIJC3Q, seg_pos={0 0}, upload=518946]: Done uploading piece. err: <nil> Dec 30 07:26:48.601713982 (591h57m48.472797017s) [noise=true , address=145.79.227.19:4078 , piece_id=AMXZNOQCZXM7LGL46SR7CS7CZ55BGTRTOZ5DJAYDMTBGB5NNDC3Q, seg_pos={0 0}, upload=518946]: Done uploading piece. err: <nil> Dec 30 07:26:48.607727713 (591h57m48.478810748s) [noise=true , address=145.79.227.14:4840 , piece_id=NI6LJNVGBHKOR4SAM7S5V3IRKEU4DG6ABLBI3VAZKLF4DOWXMJLQ, seg_pos={0 0}, upload=518946]: Done uploading piece. err: <nil> Dec 30 07:26:48.613029328 (591h57m48.484112363s) [noise=true , address=145.79.227.41:4085 , piece_id=7RGSCLOZ46YHEPE5ZKU6QMQHNW7SLV3VN6NCURYAUWBV2RWRHEEQ, seg_pos={0 0}, upload=518946]: Done uploading piece. err: <nil> Dec 30 07:26:48.787823986 (591h57m48.658907021s) [noise=true , address=77.90.31.18:4805 , piece_id=PC3THDCHXRWOJPPGYCM4XVTUWJQJHHKIHUWU34YAECOYWS5IBX4Q, seg_pos={0 0}, upload=518946]: Done uploading piece. err: <nil> Dec 30 07:26:48.905112491 (591h57m48.776195526s) [noise=true , address=45.91.116.8:4046 , piece_id=5YF5RU2ICVQJLAG43MN4K3AHEHGOQBQ57ECEKAUUX75XHJNKVNRQ, seg_pos={0 0}, upload=518946]: Done uploading piece. err: <nil> Dec 30 07:26:48.905132369 (591h57m48.776215404s) [seg_pos={0 0}, upload=518946]: Segment reached optimal threshold of 13 pieces. Dec 30 07:26:48.905318543 (591h57m48.776401588s) [noise=true , address=45.91.116.4:4939 , piece_id=EX42IOXR53GUHE7FD7SZBESKBKIZQFG6EXOWP5N2SZKMN2WQIBEA, seg_pos={0 0}, upload=518946]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 30 07:26:48.905324355 (591h57m48.77640739s) [noise=true , address=45.91.116.4:4939 , piece_id=EX42IOXR53GUHE7FD7SZBESKBKIZQFG6EXOWP5N2SZKMN2WQIBEA, seg_pos={0 0}, upload=518946]: Long tail cancellation due to optimal threshold Dec 30 07:26:48.905423974 (591h57m48.776507009s) [noise=true , address=45.91.116.5:4300 , piece_id=YZMX3AQIOSGZNCOGFBPEQ4DOIWQYX23SHVGMTNGDTIEEYTGGKVQQ, seg_pos={0 0}, upload=518946]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Dec 30 07:26:48.905429255 (591h57m48.77651229s) [noise=true , address=45.91.116.5:4300 , piece_id=YZMX3AQIOSGZNCOGFBPEQ4DOIWQYX23SHVGMTNGDTIEEYTGGKVQQ, seg_pos={0 0}, upload=518946]: Long tail cancellation due to optimal threshold Dec 30 07:26:48.905524476 (591h57m48.776607511s) [noise=true , address=145.79.227.17:4972 , piece_id=2QAO4DEM2BL4RRMJRD6L2T3XXEVJWLBW6JD6XOXM2DW3VWGAACLA, seg_pos={0 0}, upload=518946]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: expected piece hash; context canceled; EOF Dec 30 07:26:48.905529866 (591h57m48.776612911s) [noise=true , address=145.79.227.17:4972 , piece_id=2QAO4DEM2BL4RRMJRD6L2T3XXEVJWLBW6JD6XOXM2DW3VWGAACLA, seg_pos={0 0}, upload=518946]: Long tail cancellation due to optimal threshold Dec 30 07:26:48.905545416 (591h57m48.776628451s) [seg_pos={0 0}, upload=518946]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 07:26:48.905609698 (591h57m48.776692733s) [upload=518946]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 07:26:48.905613896 (591h57m48.776696931s) [upload=518946]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 07:26:49.197354886 (591h57m49.068437921s) [upload=518946]: Done uploading segments... Dec 30 07:26:49.197373121 (591h57m49.068456156s) [upload=518946]: Upload finished. err: <nil> Duration: 5.893056494s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 41.033798017s trace: https://jaeger.dev.storj.io/trace/1d0a2f93cb06aaa4 FAIL (2025-12-30 07:27:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== loading piece... Dec 30 07:06:00.467281687 (591h37m0.338364722s) [noise=true , address=77.90.31.14:4693 , piece_id=CQ6A6DJJDCKQAAMCXW7FELWCWKAOI7YCCWKVDSTZEBXNHVKQRKPQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.467716555 (591h37m0.33879959s) [noise=true , address=145.79.227.40:4082 , piece_id=7UFAIMVNINXXXRPNXLGJ34YEO3ILVN2O7UFG3HHTIT2DKWVZSUPA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468044320 (591h37m0.339127365s) [noise=true , address=145.79.227.29:4955 , piece_id=CBV7RUAZ6QAZE3VDTZKOGTIMOY66VUJTSYQSZRC7WLUD53UWVJTQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468308362 (591h37m0.339391407s) [noise=true , address=145.79.227.17:4000 , piece_id=KN3NDKB4NODHNXUWWQQ4YBWBPJCNE7YHYEMLQ4UITRCFZB4BRSMA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468565703 (591h37m0.339648748s) [noise=true , address=145.79.227.46:4604 , piece_id=NY5A6JZW4F6E4CSX3ERMKSAL2R3IYSKQ2EVHZSPF6ZOHWQFXBYWA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468843542 (591h37m0.339926587s) [noise=true , address=77.90.31.17:4781 , piece_id=TFUDJVQZFLS2XTZEGBPTV23WYLNHZMGK5ZXNDJSMW5HSNZXWXBBA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.469105310 (591h37m0.340188345s) [noise=true , address=145.79.227.35:4494 , piece_id=U4P6QB4ZRFGAJOHVSSTZHCG4PTBGII4LXAAJXXVN5RVR4WUWUBBQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.470034750 (591h37m0.341117795s) [noise=true , address=145.79.227.49:4365 , piece_id=Z357V6BZEGQOFLFH32RQLYJJZDLURVHZAHKW3H3ZIJXMQC3ZIFBQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.470487622 (591h37m0.341570657s) [noise=true , address=64.49.13.169:4713 , piece_id=YS6FRVS5V3BECH3TJQRJBGB5SSYYGUBK2HMO25PZEUOUSC7ENNZQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.470809334 (591h37m0.341892379s) [noise=true , address=77.90.31.10:4530 , piece_id=LNRM5YW2MZLXHWHGNUMP5I7Y5RRNRESBUA2M3SQKDFEQXZ25RJ3A, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.471118874 (591h37m0.342201919s) [noise=true , address=145.79.227.44:4221 , piece_id=GDKGEN3QCGFIICNST42FA2ZLLTQCHL2QAZ7AMUNPJIGYTSHDMOJQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.471431078 (591h37m0.342514113s) [noise=true , address=45.91.116.4:4963 , piece_id=FUGWNAY33ZGP6J57FIODDQQJ2UH46ABT52AU4Y7Y74Q2AOTCXWFA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.471741530 (591h37m0.342824575s) [noise=true , address=193.149.164.143:4704 , piece_id=PQQZ6N4A7TZGGX5DVRE6NT6TSDJVQARR22F6MZ5Z6LYXLUKF4V7A, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.472073311 (591h37m0.343156356s) [noise=true , address=145.79.227.33:4901 , piece_id=FFTUVNIKVKRAZ5R44V6WLWNKKC5MQYA66RZPSMW6N5EQKY7NSGZQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:01.899399345 (591h37m1.77048239s) [upload=518641]: Next returned nil segment Dec 30 07:06:01.899418712 (591h37m1.770501747s) [upload=518641]: Waiting for error group managing segments... Dec 30 07:06:02.323303785 (591h37m2.194386821s) [noise=true , address=145.79.227.46:4604 , piece_id=NY5A6JZW4F6E4CSX3ERMKSAL2R3IYSKQ2EVHZSPF6ZOHWQFXBYWA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.339689637 (591h37m2.210772682s) [noise=true , address=145.79.227.17:4000 , piece_id=KN3NDKB4NODHNXUWWQQ4YBWBPJCNE7YHYEMLQ4UITRCFZB4BRSMA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.346460570 (591h37m2.217543615s) [noise=true , address=145.79.227.29:4955 , piece_id=CBV7RUAZ6QAZE3VDTZKOGTIMOY66VUJTSYQSZRC7WLUD53UWVJTQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.347028050 (591h37m2.218111095s) [noise=true , address=145.79.227.34:4931 , piece_id=TJ4NTHHJQ4VLPBIC7RW42DBFIBI5G7LJWBN53TPSEVAWHANHMTBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.427966770 (591h37m2.299049815s) [noise=true , address=145.79.227.5:4226 , piece_id=QKDFEH6UKIH4QYN6DE3YNH3FZPAYGNYMQSBCYGM7EAW7F4MDA66A, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.446440007 (591h37m2.317523052s) [noise=true , address=145.79.227.35:4494 , piece_id=U4P6QB4ZRFGAJOHVSSTZHCG4PTBGII4LXAAJXXVN5RVR4WUWUBBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.583477598 (591h37m2.454560643s) [noise=true , address=77.90.31.19:4859 , piece_id=XHNZMPC2MF4RALF2FZJY73UYN4XWOYJUL66RFRUXNPT64JZQE5KQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.613112418 (591h37m2.484195463s) [noise=true , address=77.90.31.17:4781 , piece_id=TFUDJVQZFLS2XTZEGBPTV23WYLNHZMGK5ZXNDJSMW5HSNZXWXBBA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.631421591 (591h37m2.502504626s) [noise=true , address=77.90.31.10:4530 , piece_id=LNRM5YW2MZLXHWHGNUMP5I7Y5RRNRESBUA2M3SQKDFEQXZ25RJ3A, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.944663583 (591h37m2.815746628s) [noise=true , address=145.79.227.49:4365 , piece_id=Z357V6BZEGQOFLFH32RQLYJJZDLURVHZAHKW3H3ZIJXMQC3ZIFBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.195837408 (591h37m3.066920453s) [noise=true , address=193.149.164.143:4704 , piece_id=PQQZ6N4A7TZGGX5DVRE6NT6TSDJVQARR22F6MZ5Z6LYXLUKF4V7A, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.269365478 (591h37m3.140448523s) [noise=true , address=145.79.227.44:4221 , piece_id=GDKGEN3QCGFIICNST42FA2ZLLTQCHL2QAZ7AMUNPJIGYTSHDMOJQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.272362485 (591h37m3.14344553s) [noise=true , address=145.79.227.30:4778 , piece_id=EVWGEZL3VCMHEM6ALESBLMU65XBE74CKVMFXMSHK2GCXFPI35RBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.340891504 (591h37m3.211974549s) [noise=true , address=145.79.227.40:4082 , piece_id=7UFAIMVNINXXXRPNXLGJ34YEO3ILVN2O7UFG3HHTIT2DKWVZSUPA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.340922954 (591h37m3.212005999s) [seg_pos={0 0}, upload=518641]: Segment reached optimal threshold of 14 pieces. Dec 30 07:06:03.341286095 (591h37m3.21236913s) [noise=true , address=77.90.31.14:4693 , piece_id=CQ6A6DJJDCKQAAMCXW7FELWCWKAOI7YCCWKVDSTZEBXNHVKQRKPQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RASrnvpAj4ToD4xNvbma1iKZEe2uHK8WosjfZwDDom8Qd17Ey): protocol: write tcp 188.166.225.170:47364->77.90.31.14:4693: use of closed network connection; write tcp 188.166.225.170:47364->77.90.31.14:4693: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47364->77.90.31.14:4693: use of closed network connection Dec 30 07:06:03.341317094 (591h37m3.212400139s) [noise=true , address=77.90.31.14:4693 , piece_id=CQ6A6DJJDCKQAAMCXW7FELWCWKAOI7YCCWKVDSTZEBXNHVKQRKPQ, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341322114 (591h37m3.212405159s) [noise=true , address=145.79.227.33:4901 , piece_id=FFTUVNIKVKRAZ5R44V6WLWNKKC5MQYA66RZPSMW6N5EQKY7NSGZQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1): protocol: expected piece hash; context canceled; EOF Dec 30 07:06:03.341336261 (591h37m3.212419306s) [noise=true , address=145.79.227.33:4901 , piece_id=FFTUVNIKVKRAZ5R44V6WLWNKKC5MQYA66RZPSMW6N5EQKY7NSGZQ, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341546141 (591h37m3.212629196s) [noise=true , address=45.91.116.4:4963 , piece_id=FUGWNAY33ZGP6J57FIODDQQJ2UH46ABT52AU4Y7Y74Q2AOTCXWFA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Eki7jYegwvkdixR5pznQBbSeJPn9GjGofxnUApidQF4kRNBK5): protocol: expected piece hash; write tcp 188.166.225.170:51728->45.91.116.4:4963: use of closed network connection; write tcp 188.166.225.170:51728->45.91.116.4:4963: use of closed network connection; EOF Dec 30 07:06:03.341555549 (591h37m3.212638594s) [noise=true , address=45.91.116.4:4963 , piece_id=FUGWNAY33ZGP6J57FIODDQQJ2UH46ABT52AU4Y7Y74Q2AOTCXWFA, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341759577 (591h37m3.212842622s) [noise=true , address=64.49.13.169:4713 , piece_id=YS6FRVS5V3BECH3TJQRJBGB5SSYYGUBK2HMO25PZEUOUSC7ENNZQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17z3AiUfbxBX2eMDnUnqo1bbgojuQxJro7PsEaPjYzQjHQ3D6K): protocol: write tcp 188.166.225.170:42002->64.49.13.169:4713: use of closed network connection; write tcp 188.166.225.170:42002->64.49.13.169:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42002->64.49.13.169:4713: use of closed network connection Dec 30 07:06:03.341769897 (591h37m3.212852942s) [noise=true , address=64.49.13.169:4713 , piece_id=YS6FRVS5V3BECH3TJQRJBGB5SSYYGUBK2HMO25PZEUOUSC7ENNZQ, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341869927 (591h37m3.212952972s) [seg_pos={0 0}, upload=518641]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 07:06:03.342018730 (591h37m3.213101775s) [upload=518641]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 07:06:03.342025634 (591h37m3.213108679s) [upload=518641]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 07:06:03.635315934 (591h37m3.506398979s) [upload=518641]: Done uploading segments... Dec 30 07:06:03.635343146 (591h37m3.506426191s) [upload=518641]: Upload finished. err: <nil> Duration: 41.033856578s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 19 out of 60 is failed (threshold=6) [....X....XXX...XXXX.........X.X.....X..XXXXXX.X.....X.......] duration: 3.527484753s trace: https://jaeger.dev.storj.io/trace/220b42f7e5b29694 FAIL (2025-12-30 07:27:29 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.527512274s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [..XX...X.....X....XXX..X.....X...X..X.XX..X...X....XXX....X.] duration: 5.910664317s trace: https://jaeger.dev.storj.io/trace/5b5a31fe4475073 FAIL (2025-12-30 07:27:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 07:26:46.289184198 (591h57m46.160267243s) [upload=518946]: Starting upload Dec 30 07:26:46.289281163 (591h57m46.160364218s) [upload=518946]: Done starting upload Dec 30 07:26:46.289508094 (591h57m46.160591129s) [upload=518946]: Uploading segments... Dec 30 07:26:46.289589440 (591h57m46.160672495s) [upload=518946]: Got next segment. Inline: false Dec 30 07:26:46.289633714 (591h57m46.160716759s) [upload=518946]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 07:26:46.289639164 (591h57m46.160722209s) [upload=518946]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 07:26:46.532324195 (591h57m46.40340724s) [seg_pos={0 0}, upload=518946]: Begin upload segment... Dec 30 07:26:46.532410239 (591h57m46.403493274s) [seg_pos={0 0}, upload=518946]: Done begin upload segment. Dec 30 07:26:46.532448602 (591h57m46.403531637s) [noise=true , address=145.79.227.2:4114 , piece_id=F6QSNFGWCWQDQUCOICC3GNCHUPERG6PWBO2OIAQDCGRHPS5YA2PA, seg_pos={0 0}, upload=518946]: Uploading piece... Dec 30 07:26:46.532504338 (591h57m46.403587383s) [noise=true , address=45.91.116.8:4046 , piece_id=5YF5RU2ICVQJLAG43MN4K3AHEHGOQBQ57ECEKAUUX75XHJNKVNRQ, seg_pos={0 0}, upload=518946]: Uploading piece... Dec 30 07:26:46.533077460 (591h57m46.404160505s) [noise=true , address=145.79.227.39:4015 , piece_id=NGGRJ4EFWFGJSHIQ3YHBS3SQZVO4N4UWXIAB6JA32K6PVAUC5AZQ, seg_pos={0 0}, upload=518946]: Uploading piece... Dec 30 07:26:46.533549498 (591h57m46.404632533s) [noise=true , address=145.79.227.19:4078 , piece_id=AMXZNOQCZXM7LGL46SR7CS7CZ55BGTRTOZ5DJAYDMTBGB5NNDC3Q, seg_pos={0 0}, upload=518946]: Uploading piece... Dec 30 07:26:46.534157366 (591h57m46.405240401s) [noise=true , address=145.79.227.41:4085 , piece_id=7RGSCLOZ46YHEPE5ZKU6QMQHNW7SLV3VN6NCURYAUWBV2RWRHEEQ, seg_pos={0 0}, upload=518946]: Uploading piece... Dec 30 07:26:46.534782767 (591h57m46.405865802s) [noise=true , address=145.79.227.15:4874 , piece_id=GY5FWVL6PE77LP7455DHVEMQ2VKV3W6XL5NHN4PPJ7F45HPUC4IA, seg_pos={0 0}, upload=518946]: Uploading piece... Dec 30 07:26:46.535251168 (591h57m46.406334213s) [noise=true , address=145.79.227.51:4411 , piece_id=UIKRX5OVZIFAKUBTQQGRYMPMIXSFY43VUBC2KRCCGWWMFUWD2LSQ, seg_pos={0 0}, upload=518946]: Uploading piece... Dec 30 07:26:46.536749040 (591h57m46.407832085s) [noise=true , address=145.79.227.36:4517 , piece_id=DKJ2BMYTOU46NNNK6FRXT7V2V3UXRQPZURV3RH6CMU3BFCBIJC3Q, seg_pos={0 0}, upload=518946]: Uploading piece... Dec 30 07:26:46.538410154 (591h57m46.409493209s) [noise=true , address=145.79.227.14:4840 , piece_id=NI6LJNVGBHKOR4SAM7S5V3IRKEU4DG6ABLBI3VAZKLF4DOWXMJLQ, seg_pos={0 0}, upload=518946]: Uploading piece... Dec 30 07:26:46.539727031 (591h57m46.410810066s) [noise=true , address=45.91.116.5:4300 , piece_id=YZMX3AQIOSGZNCOGFBPEQ4DOIWQYX23SHVGMTNGDTIEEYTGGKVQQ, seg_pos={0 0}, upload=518946]: Uploading piece... Dec 30 07:26:46.541085869 (591h57m46.412168904s) [noise=true , address=145.79.227.32:4835 , piece_id=4HD4E2ME5CEPRUFERUH6NEFFFFGUYX2GLU2GF4OLB5SIMZCCGRQA, seg_pos={0 0}, upload=518946]: Uploading piece... Dec 30 07:26:46.541465432 (591h57m46.412548467s) [noise=true , address=145.79.227.17:4972 , piece_id=2QAO4DEM2BL4RRMJRD6L2T3XXEVJWLBW6JD6XOXM2DW3VWGAACLA, seg_pos={0 0}, upload=518946]: Uploading piece... Dec 30 07:26:46.541630516 (591h57m46.412713551s) [noise=true , address=45.91.116.4:4939 , piece_id=EX42IOXR53GUHE7FD7SZBESKBKIZQFG6EXOWP5N2SZKMN2WQIBEA, seg_pos={0 0}, upload=518946]: Uploading piece... Dec 30 07:26:46.543860943 (591h57m46.414943978s) [noise=true , address=145.79.227.22:4509 , piece_id=GXGMXCJKULDJLXTM2AGJA5VRVCVNRHKNV5NB2TTESXQ5TVPAYZKQ, seg_pos={0 0}, upload=518946]: Uploading piece... Dec 30 07:26:46.545096676 (591h57m46.416179711s) [noise=true , address=145.79.227.46:4603 , piece_id=OSFA5IDUJ5DGRO62ORYYSJFGDOF2DWVLENETXJU2DP7JMPUXFR4Q, seg_pos={0 0}, upload=518946]: Uploading piece... Dec 30 07:26:46.546477646 (591h57m46.417560691s) [noise=true , address=77.90.31.18:4805 , piece_id=PC3THDCHXRWOJPPGYCM4XVTUWJQJHHKIHUWU34YAECOYWS5IBX4Q, seg_pos={0 0}, upload=518946]: Uploading piece... Dec 30 07:26:48.130107348 (591h57m48.001190383s) [upload=518946]: Next returned nil segment Dec 30 07:26:48.130126845 (591h57m48.00120988s) [upload=518946]: Waiting for error group managing segments... Dec 30 07:26:48.522160289 (591h57m48.393243334s) [noise=true , address=145.79.227.2:4114 , piece_id=F6QSNFGWCWQDQUCOICC3GNCHUPERG6PWBO2OIAQDCGRHPS5YA2PA, seg_pos={0 0}, upload=518946]: Done uploading piece. err: <nil> Dec 30 07:26:48.529146272 (591h57m48.400229327s) [noise=true , address=145.79.227.46:4603 , piece_id=OSFA5IDUJ5DGRO62ORYYSJFGDOF2DWVLENETXJU2DP7JMPUXFR4Q, seg_pos={0 0}, upload=518946]: Done uploading piece. err: <nil> Dec 30 07:26:48.532844023 (591h57m48.403927058s) [noise=true , address=145.79.227.15:4874 , piece_id=GY5FWVL6PE77LP7455DHVEMQ2VKV3W6XL5NHN4PPJ7F45HPUC4IA, seg_pos={0 0}, upload=518946]: Done uploading piece. err: <nil> Dec 30 07:26:48.534193653 (591h57m48.405276688s) [noise=true , address=145.79.227.22:4509 , piece_id=GXGMXCJKULDJLXTM2AGJA5VRVCVNRHKNV5NB2TTESXQ5TVPAYZKQ, seg_pos={0 0}, upload=518946]: Done uploading piece. err: <nil> Dec 30 07:26:48.547639556 (591h57m48.418722631s) [noise=true , address=145.79.227.32:4835 , piece_id=4HD4E2ME5CEPRUFERUH6NEFFFFGUYX2GLU2GF4OLB5SIMZCCGRQA, seg_pos={0 0}, upload=518946]: Done uploading piece. err: <nil> Dec 30 07:26:48.554344823 (591h57m48.425427858s) [noise=true , address=145.79.227.39:4015 , piece_id=NGGRJ4EFWFGJSHIQ3YHBS3SQZVO4N4UWXIAB6JA32K6PVAUC5AZQ, seg_pos={0 0}, upload=518946]: Done uploading piece. err: <nil> Dec 30 07:26:48.555384473 (591h57m48.426467508s) [noise=true , address=145.79.227.51:4411 , piece_id=UIKRX5OVZIFAKUBTQQGRYMPMIXSFY43VUBC2KRCCGWWMFUWD2LSQ, seg_pos={0 0}, upload=518946]: Done uploading piece. err: <nil> Dec 30 07:26:48.557932195 (591h57m48.42901523s) [noise=true , address=145.79.227.36:4517 , piece_id=DKJ2BMYTOU46NNNK6FRXT7V2V3UXRQPZURV3RH6CMU3BFCBIJC3Q, seg_pos={0 0}, upload=518946]: Done uploading piece. err: <nil> Dec 30 07:26:48.601713982 (591h57m48.472797017s) [noise=true , address=145.79.227.19:4078 , piece_id=AMXZNOQCZXM7LGL46SR7CS7CZ55BGTRTOZ5DJAYDMTBGB5NNDC3Q, seg_pos={0 0}, upload=518946]: Done uploading piece. err: <nil> Dec 30 07:26:48.607727713 (591h57m48.478810748s) [noise=true , address=145.79.227.14:4840 , piece_id=NI6LJNVGBHKOR4SAM7S5V3IRKEU4DG6ABLBI3VAZKLF4DOWXMJLQ, seg_pos={0 0}, upload=518946]: Done uploading piece. err: <nil> Dec 30 07:26:48.613029328 (591h57m48.484112363s) [noise=true , address=145.79.227.41:4085 , piece_id=7RGSCLOZ46YHEPE5ZKU6QMQHNW7SLV3VN6NCURYAUWBV2RWRHEEQ, seg_pos={0 0}, upload=518946]: Done uploading piece. err: <nil> Dec 30 07:26:48.787823986 (591h57m48.658907021s) [noise=true , address=77.90.31.18:4805 , piece_id=PC3THDCHXRWOJPPGYCM4XVTUWJQJHHKIHUWU34YAECOYWS5IBX4Q, seg_pos={0 0}, upload=518946]: Done uploading piece. err: <nil> Dec 30 07:26:48.905112491 (591h57m48.776195526s) [noise=true , address=45.91.116.8:4046 , piece_id=5YF5RU2ICVQJLAG43MN4K3AHEHGOQBQ57ECEKAUUX75XHJNKVNRQ, seg_pos={0 0}, upload=518946]: Done uploading piece. err: <nil> Dec 30 07:26:48.905132369 (591h57m48.776215404s) [seg_pos={0 0}, upload=518946]: Segment reached optimal threshold of 13 pieces. Dec 30 07:26:48.905318543 (591h57m48.776401588s) [noise=true , address=45.91.116.4:4939 , piece_id=EX42IOXR53GUHE7FD7SZBESKBKIZQFG6EXOWP5N2SZKMN2WQIBEA, seg_pos={0 0}, upload=518946]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 30 07:26:48.905324355 (591h57m48.77640739s) [noise=true , address=45.91.116.4:4939 , piece_id=EX42IOXR53GUHE7FD7SZBESKBKIZQFG6EXOWP5N2SZKMN2WQIBEA, seg_pos={0 0}, upload=518946]: Long tail cancellation due to optimal threshold Dec 30 07:26:48.905423974 (591h57m48.776507009s) [noise=true , address=45.91.116.5:4300 , piece_id=YZMX3AQIOSGZNCOGFBPEQ4DOIWQYX23SHVGMTNGDTIEEYTGGKVQQ, seg_pos={0 0}, upload=518946]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Dec 30 07:26:48.905429255 (591h57m48.77651229s) [noise=true , address=45.91.116.5:4300 , piece_id=YZMX3AQIOSGZNCOGFBPEQ4DOIWQYX23SHVGMTNGDTIEEYTGGKVQQ, seg_pos={0 0}, upload=518946]: Long tail cancellation due to optimal threshold Dec 30 07:26:48.905524476 (591h57m48.776607511s) [noise=true , address=145.79.227.17:4972 , piece_id=2QAO4DEM2BL4RRMJRD6L2T3XXEVJWLBW6JD6XOXM2DW3VWGAACLA, seg_pos={0 0}, upload=518946]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: expected piece hash; context canceled; EOF Dec 30 07:26:48.905529866 (591h57m48.776612911s) [noise=true , address=145.79.227.17:4972 , piece_id=2QAO4DEM2BL4RRMJRD6L2T3XXEVJWLBW6JD6XOXM2DW3VWGAACLA, seg_pos={0 0}, upload=518946]: Long tail cancellation due to optimal threshold Dec 30 07:26:48.905545416 (591h57m48.776628451s) [seg_pos={0 0}, upload=518946]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 07:26:48.905609698 (591h57m48.776692733s) [upload=518946]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 07:26:48.905613896 (591h57m48.776696931s) [upload=518946]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 07:26:49.197354886 (591h57m49.068437921s) [upload=518946]: Done uploading segments... Dec 30 07:26:49.197373121 (591h57m49.068456156s) [upload=518946]: Upload finished. err: <nil> Duration: 5.910682021s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 6.313028832s trace: https://jaeger.dev.storj.io/trace/4c2f4efaff0e0ea0 FAIL (2025-12-30 07:27:20 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== load=518292]: Uploading piece... Dec 30 06:41:13.598100152 (591h12m13.469183187s) [noise=true , address=81.171.14.70:4041 , piece_id=NORNA6WFTGPNOHU7MPTNNB5ONTBS5DJGCN2NWIZAEPYSUSIOSGEQ, seg_pos={0 0}, upload=518292]: Uploading piece... Dec 30 06:41:13.598587971 (591h12m13.469671016s) [noise=true , address=89.149.201.147:4007 , piece_id=HZLSYGSBP4TJH5V5XF3Q3AETJU625ERNDBD3VXW2HUZTTPAHLBMA, seg_pos={0 0}, upload=518292]: Uploading piece... Dec 30 06:41:13.599093143 (591h12m13.470176178s) [noise=true , address=89.149.201.147:4008 , piece_id=XYFRNJVIFFZA5S36KFPGZPVCPLRL2OW5QBJD3UR37VPPA2RX3RJA, seg_pos={0 0}, upload=518292]: Uploading piece... Dec 30 06:41:13.599699999 (591h12m13.470783034s) [noise=true , address=81.171.14.70:4039 , piece_id=DEJDXFDHOSAMIP2FN5LJBJNBB4QLJSE2AUDW6IMA6IC4VRY7VAGQ, seg_pos={0 0}, upload=518292]: Uploading piece... Dec 30 06:41:13.600136169 (591h12m13.471219204s) [noise=true , address=185.167.204.205:4034 , piece_id=OWZHQB5ZDT2QZ2QAHZJVRWQ222MX6CWLKJUOIAAGSMAVLWGJAOBA, seg_pos={0 0}, upload=518292]: Uploading piece... Dec 30 06:41:13.600543184 (591h12m13.471626229s) [noise=true , address=185.167.204.206:4058 , piece_id=ROOCGK64M5RHIWNAVCSZNYTZ6M2ORWB7MJ27XJVDTFVQIKJZHY4A, seg_pos={0 0}, upload=518292]: Uploading piece... Dec 30 06:41:13.601048286 (591h12m13.472131332s) [noise=true , address=81.171.14.70:4046 , piece_id=B2MTUNKMUP5S7XW4SVQ5QI2ZO3VFB5QVAKBBY4BIZIUILM6O7VYQ, seg_pos={0 0}, upload=518292]: Uploading piece... Dec 30 06:41:13.601698784 (591h12m13.472781819s) [noise=true , address=185.167.204.206:4064 , piece_id=H4PP7HPHXRX2LOZ4776AMWAJDN7OC435E2K4BXI7XOZYKTB4YUNA, seg_pos={0 0}, upload=518292]: Uploading piece... Dec 30 06:41:13.602168259 (591h12m13.473251304s) [noise=true , address=185.167.204.206:4065 , piece_id=LD3J4656Q5QEUCMJ5OKEYATEVRIKSXO7XGARTN25BVM7FDIYOWKA, seg_pos={0 0}, upload=518292]: Uploading piece... Dec 30 06:41:13.602782708 (591h12m13.473865753s) [noise=true , address=81.171.14.70:4027 , piece_id=6WEORWX5QOMFSJUEADIIBHHIPKNFP4NJPZAJADBUTPUSG7MZJMOQ, seg_pos={0 0}, upload=518292]: Uploading piece... Dec 30 06:41:14.423395002 (591h12m14.294478047s) [upload=518292]: Next returned nil segment Dec 30 06:41:14.423414068 (591h12m14.294497103s) [upload=518292]: Waiting for error group managing segments... Dec 30 06:41:14.817650316 (591h12m14.688733351s) [noise=true , address=185.167.204.205:4029 , piece_id=KUOMMYJ2F4NIK6VN4C5M3Y3TMHYEZH7QFNJ6K3CLOAVI5R6VKTHQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:14.817870054 (591h12m14.688953099s) [noise=true , address=185.167.204.205:4019 , piece_id=H6AH654VGX6NHBUKODIWGL7I2RKE3DDMY4UM6VDONP2EP3NPDJBA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:14.818623698 (591h12m14.689706744s) [noise=true , address=185.167.204.206:4065 , piece_id=LD3J4656Q5QEUCMJ5OKEYATEVRIKSXO7XGARTN25BVM7FDIYOWKA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:14.818932436 (591h12m14.690015481s) [noise=true , address=185.167.204.206:4061 , piece_id=T2XS4IFPQSVS3T45SPQ5ZDDVT2VMTMXHKL2ZMX4OIOPRR67SCD2Q, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:14.819147046 (591h12m14.690230091s) [noise=true , address=185.167.204.206:4071 , piece_id=QI2TOY5KCDZVWCCLMAJXVUWD2HB7S7IOTEAV2EHAMUNHKOFKTNYA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:14.819678578 (591h12m14.690761623s) [noise=true , address=185.167.204.206:4059 , piece_id=LMOGTWX2YIRR3EKNROK43O2B27S5L5U6GYB7HBVRBPQPH4WDDWZQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:14.819905028 (591h12m14.690988063s) [noise=true , address=185.167.204.206:4058 , piece_id=ROOCGK64M5RHIWNAVCSZNYTZ6M2ORWB7MJ27XJVDTFVQIKJZHY4A, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:14.820479312 (591h12m14.691562357s) [noise=true , address=185.167.204.205:4021 , piece_id=KP7J5XSY4SKSKZG7XXFZMNKMRP4472PXI7CW2UQBZICN4MAFO5RQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:14.823149296 (591h12m14.694232331s) [noise=true , address=185.167.204.205:4034 , piece_id=OWZHQB5ZDT2QZ2QAHZJVRWQ222MX6CWLKJUOIAAGSMAVLWGJAOBA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:14.827906996 (591h12m14.698990041s) [noise=true , address=185.167.204.206:4055 , piece_id=VYEDR6XHZBZMTVJ5U5OF67WZOGVMCMQLXXBQUI7DHBCCVQ2RRERQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:14.828003769 (591h12m14.699086814s) [noise=true , address=185.167.204.205:4016 , piece_id=I6MUDLAXUUDJL2C4HFE22C2L56A7D6HSCNWXKWJ7OKO2N4RA4W7Q, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:14.828893203 (591h12m14.699976238s) [noise=true , address=185.167.204.206:4064 , piece_id=H4PP7HPHXRX2LOZ4776AMWAJDN7OC435E2K4BXI7XOZYKTB4YUNA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:14.829477074 (591h12m14.700560109s) [noise=true , address=185.167.204.205:4017 , piece_id=CZP65KETMGOHFDMV7KENRL4V36UHBPWRDCBBPUM5NE5L2PV4AQCQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:14.829774972 (591h12m14.700858007s) [noise=true , address=185.167.204.206:4043 , piece_id=G7VZYID4TXGNPDN63NQULXOVSX42WMFFNPBX6KQJ5L2WSW4D2YGQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:14.831734493 (591h12m14.702817528s) [noise=true , address=185.167.204.206:4049 , piece_id=BQ7JEGSGPIFJUGZFBYRPDMBNPZ24BG6TEG4VUM4JUBTE4T2RF4SA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:15.405688561 (591h12m15.276771606s) [noise=true , address=81.171.14.70:4027 , piece_id=6WEORWX5QOMFSJUEADIIBHHIPKNFP4NJPZAJADBUTPUSG7MZJMOQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:15.417110440 (591h12m15.288193485s) [noise=true , address=81.171.14.70:4031 , piece_id=U6Z4SNK32ZSWQ7NDR7ACXGRC4BADDOPR65ILBR64QLGVNDBADFFQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:15.425600446 (591h12m15.296683481s) [noise=true , address=81.171.14.70:4037 , piece_id=6A25LEIAQEJB3DNCTXQT6K7ZSKIGOW4QUGMNKLOG3CT2XXH25PSA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:15.499056228 (591h12m15.370139263s) [noise=true , address=81.171.14.70:4038 , piece_id=3WRHW3VGXWHOEOV5Q6CLYSF7MUVWS7QHTNHLSTCKAUERPHAH3I3Q, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:15.514998925 (591h12m15.38608197s) [noise=true , address=81.171.14.70:4041 , piece_id=NORNA6WFTGPNOHU7MPTNNB5ONTBS5DJGCN2NWIZAEPYSUSIOSGEQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:15.528579544 (591h12m15.399662589s) [noise=true , address=81.171.14.70:4030 , piece_id=KITOJIT34CHS2QZBXCW2SJPZB4RHEHCMJ6IYQ4Z5KLL24YAWBIUQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:15.528991398 (591h12m15.400074434s) [noise=true , address=81.171.14.70:4047 , piece_id=ZWV5DP4YC3MSYDWEQH4ZK3VR4WZ4DUTI4VL3WVLQKFIEKIW44LOQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:15.534844544 (591h12m15.405927579s) [noise=true , address=81.171.14.70:4035 , piece_id=XCLL5SOAPJSSGF5YYQH6ZKGOHMLS6RI6H26FOW5XIJRY3TWYNEGA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:15.915614910 (591h12m15.786697945s) [noise=true , address=81.171.14.70:4039 , piece_id=DEJDXFDHOSAMIP2FN5LJBJNBB4QLJSE2AUDW6IMA6IC4VRY7VAGQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:15.967820176 (591h12m15.838903211s) [noise=true , address=81.171.14.70:4046 , piece_id=B2MTUNKMUP5S7XW4SVQ5QI2ZO3VFB5QVAKBBY4BIZIUILM6O7VYQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:16.172116128 (591h12m16.043199163s) [noise=true , address=89.149.201.147:4007 , piece_id=HZLSYGSBP4TJH5V5XF3Q3AETJU625ERNDBD3VXW2HUZTTPAHLBMA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:16.172137559 (591h12m16.043220604s) [seg_pos={0 0}, upload=518292]: Segment reached optimal threshold of 26 pieces. Dec 30 06:41:16.172336949 (591h12m16.043419994s) [noise=true , address=89.149.201.147:4008 , piece_id=XYFRNJVIFFZA5S36KFPGZPVCPLRL2OW5QBJD3UR37VPPA2RX3RJA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QN8VP6o7qGw9Dd8EafU9xe7s7Cuf1FyqPMbnRaznrfPxfEUUn): protocol: expected piece hash; context canceled; EOF Dec 30 06:41:16.172344213 (591h12m16.043427258s) [noise=true , address=89.149.201.147:4008 , piece_id=XYFRNJVIFFZA5S36KFPGZPVCPLRL2OW5QBJD3UR37VPPA2RX3RJA, seg_pos={0 0}, upload=518292]: Long tail cancellation due to optimal threshold Dec 30 06:41:16.172554002 (591h12m16.043637037s) [noise=true , address=89.149.201.147:4020 , piece_id=N7G5DGEDYVKBTO5S6BYW3IYEOK7HAMK5JQIBMVPMPYRDHXODWEGA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YeKHJH7CoouLFJYENJhzVLNkA32hJaqjLz3dXhr78eaSGtcic): protocol: expected piece hash; context canceled; EOF Dec 30 06:41:16.172565233 (591h12m16.043648278s) [noise=true , address=89.149.201.147:4020 , piece_id=N7G5DGEDYVKBTO5S6BYW3IYEOK7HAMK5JQIBMVPMPYRDHXODWEGA, seg_pos={0 0}, upload=518292]: Long tail cancellation due to optimal threshold Dec 30 06:41:16.172587506 (591h12m16.043670541s) [seg_pos={0 0}, upload=518292]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 30 06:41:16.172717945 (591h12m16.04380098s) [upload=518292]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 06:41:16.172724567 (591h12m16.043807602s) [upload=518292]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 06:41:16.404092237 (591h12m16.275175292s) [upload=518292]: Done uploading segments... Dec 30 06:41:16.404119940 (591h12m16.275202975s) [upload=518292]: Upload finished. err: <nil> Duration: 6.313050193s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [...XX...X.....X....XXX..X.....X...X..X.XX..X...X....XXX....X] duration: 30.003747149s trace: https://jaeger.dev.storj.io/trace/5b5a31fe4475073 FAIL (2025-12-30 07:27:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 07:26:46.289184198 (591h57m46.160267243s) [upload=518946]: Starting upload Dec 30 07:26:46.289281163 (591h57m46.160364218s) [upload=518946]: Done starting upload Dec 30 07:26:46.289508094 (591h57m46.160591129s) [upload=518946]: Uploading segments... Dec 30 07:26:46.289589440 (591h57m46.160672495s) [upload=518946]: Got next segment. Inline: false Dec 30 07:26:46.289633714 (591h57m46.160716759s) [upload=518946]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 07:26:46.289639164 (591h57m46.160722209s) [upload=518946]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 07:26:46.532324195 (591h57m46.40340724s) [seg_pos={0 0}, upload=518946]: Begin upload segment... Dec 30 07:26:46.532410239 (591h57m46.403493274s) [seg_pos={0 0}, upload=518946]: Done begin upload segment. Dec 30 07:26:46.532448602 (591h57m46.403531637s) [noise=true , address=145.79.227.2:4114 , piece_id=F6QSNFGWCWQDQUCOICC3GNCHUPERG6PWBO2OIAQDCGRHPS5YA2PA, seg_pos={0 0}, upload=518946]: Uploading piece... Dec 30 07:26:46.532504338 (591h57m46.403587383s) [noise=true , address=45.91.116.8:4046 , piece_id=5YF5RU2ICVQJLAG43MN4K3AHEHGOQBQ57ECEKAUUX75XHJNKVNRQ, seg_pos={0 0}, upload=518946]: Uploading piece... Dec 30 07:26:46.533077460 (591h57m46.404160505s) [noise=true , address=145.79.227.39:4015 , piece_id=NGGRJ4EFWFGJSHIQ3YHBS3SQZVO4N4UWXIAB6JA32K6PVAUC5AZQ, seg_pos={0 0}, upload=518946]: Uploading piece... Dec 30 07:26:46.533549498 (591h57m46.404632533s) [noise=true , address=145.79.227.19:4078 , piece_id=AMXZNOQCZXM7LGL46SR7CS7CZ55BGTRTOZ5DJAYDMTBGB5NNDC3Q, seg_pos={0 0}, upload=518946]: Uploading piece... Dec 30 07:26:46.534157366 (591h57m46.405240401s) [noise=true , address=145.79.227.41:4085 , piece_id=7RGSCLOZ46YHEPE5ZKU6QMQHNW7SLV3VN6NCURYAUWBV2RWRHEEQ, seg_pos={0 0}, upload=518946]: Uploading piece... Dec 30 07:26:46.534782767 (591h57m46.405865802s) [noise=true , address=145.79.227.15:4874 , piece_id=GY5FWVL6PE77LP7455DHVEMQ2VKV3W6XL5NHN4PPJ7F45HPUC4IA, seg_pos={0 0}, upload=518946]: Uploading piece... Dec 30 07:26:46.535251168 (591h57m46.406334213s) [noise=true , address=145.79.227.51:4411 , piece_id=UIKRX5OVZIFAKUBTQQGRYMPMIXSFY43VUBC2KRCCGWWMFUWD2LSQ, seg_pos={0 0}, upload=518946]: Uploading piece... Dec 30 07:26:46.536749040 (591h57m46.407832085s) [noise=true , address=145.79.227.36:4517 , piece_id=DKJ2BMYTOU46NNNK6FRXT7V2V3UXRQPZURV3RH6CMU3BFCBIJC3Q, seg_pos={0 0}, upload=518946]: Uploading piece... Dec 30 07:26:46.538410154 (591h57m46.409493209s) [noise=true , address=145.79.227.14:4840 , piece_id=NI6LJNVGBHKOR4SAM7S5V3IRKEU4DG6ABLBI3VAZKLF4DOWXMJLQ, seg_pos={0 0}, upload=518946]: Uploading piece... Dec 30 07:26:46.539727031 (591h57m46.410810066s) [noise=true , address=45.91.116.5:4300 , piece_id=YZMX3AQIOSGZNCOGFBPEQ4DOIWQYX23SHVGMTNGDTIEEYTGGKVQQ, seg_pos={0 0}, upload=518946]: Uploading piece... Dec 30 07:26:46.541085869 (591h57m46.412168904s) [noise=true , address=145.79.227.32:4835 , piece_id=4HD4E2ME5CEPRUFERUH6NEFFFFGUYX2GLU2GF4OLB5SIMZCCGRQA, seg_pos={0 0}, upload=518946]: Uploading piece... Dec 30 07:26:46.541465432 (591h57m46.412548467s) [noise=true , address=145.79.227.17:4972 , piece_id=2QAO4DEM2BL4RRMJRD6L2T3XXEVJWLBW6JD6XOXM2DW3VWGAACLA, seg_pos={0 0}, upload=518946]: Uploading piece... Dec 30 07:26:46.541630516 (591h57m46.412713551s) [noise=true , address=45.91.116.4:4939 , piece_id=EX42IOXR53GUHE7FD7SZBESKBKIZQFG6EXOWP5N2SZKMN2WQIBEA, seg_pos={0 0}, upload=518946]: Uploading piece... Dec 30 07:26:46.543860943 (591h57m46.414943978s) [noise=true , address=145.79.227.22:4509 , piece_id=GXGMXCJKULDJLXTM2AGJA5VRVCVNRHKNV5NB2TTESXQ5TVPAYZKQ, seg_pos={0 0}, upload=518946]: Uploading piece... Dec 30 07:26:46.545096676 (591h57m46.416179711s) [noise=true , address=145.79.227.46:4603 , piece_id=OSFA5IDUJ5DGRO62ORYYSJFGDOF2DWVLENETXJU2DP7JMPUXFR4Q, seg_pos={0 0}, upload=518946]: Uploading piece... Dec 30 07:26:46.546477646 (591h57m46.417560691s) [noise=true , address=77.90.31.18:4805 , piece_id=PC3THDCHXRWOJPPGYCM4XVTUWJQJHHKIHUWU34YAECOYWS5IBX4Q, seg_pos={0 0}, upload=518946]: Uploading piece... Dec 30 07:26:48.130107348 (591h57m48.001190383s) [upload=518946]: Next returned nil segment Dec 30 07:26:48.130126845 (591h57m48.00120988s) [upload=518946]: Waiting for error group managing segments... Dec 30 07:26:48.522160289 (591h57m48.393243334s) [noise=true , address=145.79.227.2:4114 , piece_id=F6QSNFGWCWQDQUCOICC3GNCHUPERG6PWBO2OIAQDCGRHPS5YA2PA, seg_pos={0 0}, upload=518946]: Done uploading piece. err: <nil> Dec 30 07:26:48.529146272 (591h57m48.400229327s) [noise=true , address=145.79.227.46:4603 , piece_id=OSFA5IDUJ5DGRO62ORYYSJFGDOF2DWVLENETXJU2DP7JMPUXFR4Q, seg_pos={0 0}, upload=518946]: Done uploading piece. err: <nil> Dec 30 07:26:48.532844023 (591h57m48.403927058s) [noise=true , address=145.79.227.15:4874 , piece_id=GY5FWVL6PE77LP7455DHVEMQ2VKV3W6XL5NHN4PPJ7F45HPUC4IA, seg_pos={0 0}, upload=518946]: Done uploading piece. err: <nil> Dec 30 07:26:48.534193653 (591h57m48.405276688s) [noise=true , address=145.79.227.22:4509 , piece_id=GXGMXCJKULDJLXTM2AGJA5VRVCVNRHKNV5NB2TTESXQ5TVPAYZKQ, seg_pos={0 0}, upload=518946]: Done uploading piece. err: <nil> Dec 30 07:26:48.547639556 (591h57m48.418722631s) [noise=true , address=145.79.227.32:4835 , piece_id=4HD4E2ME5CEPRUFERUH6NEFFFFGUYX2GLU2GF4OLB5SIMZCCGRQA, seg_pos={0 0}, upload=518946]: Done uploading piece. err: <nil> Dec 30 07:26:48.554344823 (591h57m48.425427858s) [noise=true , address=145.79.227.39:4015 , piece_id=NGGRJ4EFWFGJSHIQ3YHBS3SQZVO4N4UWXIAB6JA32K6PVAUC5AZQ, seg_pos={0 0}, upload=518946]: Done uploading piece. err: <nil> Dec 30 07:26:48.555384473 (591h57m48.426467508s) [noise=true , address=145.79.227.51:4411 , piece_id=UIKRX5OVZIFAKUBTQQGRYMPMIXSFY43VUBC2KRCCGWWMFUWD2LSQ, seg_pos={0 0}, upload=518946]: Done uploading piece. err: <nil> Dec 30 07:26:48.557932195 (591h57m48.42901523s) [noise=true , address=145.79.227.36:4517 , piece_id=DKJ2BMYTOU46NNNK6FRXT7V2V3UXRQPZURV3RH6CMU3BFCBIJC3Q, seg_pos={0 0}, upload=518946]: Done uploading piece. err: <nil> Dec 30 07:26:48.601713982 (591h57m48.472797017s) [noise=true , address=145.79.227.19:4078 , piece_id=AMXZNOQCZXM7LGL46SR7CS7CZ55BGTRTOZ5DJAYDMTBGB5NNDC3Q, seg_pos={0 0}, upload=518946]: Done uploading piece. err: <nil> Dec 30 07:26:48.607727713 (591h57m48.478810748s) [noise=true , address=145.79.227.14:4840 , piece_id=NI6LJNVGBHKOR4SAM7S5V3IRKEU4DG6ABLBI3VAZKLF4DOWXMJLQ, seg_pos={0 0}, upload=518946]: Done uploading piece. err: <nil> Dec 30 07:26:48.613029328 (591h57m48.484112363s) [noise=true , address=145.79.227.41:4085 , piece_id=7RGSCLOZ46YHEPE5ZKU6QMQHNW7SLV3VN6NCURYAUWBV2RWRHEEQ, seg_pos={0 0}, upload=518946]: Done uploading piece. err: <nil> Dec 30 07:26:48.787823986 (591h57m48.658907021s) [noise=true , address=77.90.31.18:4805 , piece_id=PC3THDCHXRWOJPPGYCM4XVTUWJQJHHKIHUWU34YAECOYWS5IBX4Q, seg_pos={0 0}, upload=518946]: Done uploading piece. err: <nil> Dec 30 07:26:48.905112491 (591h57m48.776195526s) [noise=true , address=45.91.116.8:4046 , piece_id=5YF5RU2ICVQJLAG43MN4K3AHEHGOQBQ57ECEKAUUX75XHJNKVNRQ, seg_pos={0 0}, upload=518946]: Done uploading piece. err: <nil> Dec 30 07:26:48.905132369 (591h57m48.776215404s) [seg_pos={0 0}, upload=518946]: Segment reached optimal threshold of 13 pieces. Dec 30 07:26:48.905318543 (591h57m48.776401588s) [noise=true , address=45.91.116.4:4939 , piece_id=EX42IOXR53GUHE7FD7SZBESKBKIZQFG6EXOWP5N2SZKMN2WQIBEA, seg_pos={0 0}, upload=518946]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 30 07:26:48.905324355 (591h57m48.77640739s) [noise=true , address=45.91.116.4:4939 , piece_id=EX42IOXR53GUHE7FD7SZBESKBKIZQFG6EXOWP5N2SZKMN2WQIBEA, seg_pos={0 0}, upload=518946]: Long tail cancellation due to optimal threshold Dec 30 07:26:48.905423974 (591h57m48.776507009s) [noise=true , address=45.91.116.5:4300 , piece_id=YZMX3AQIOSGZNCOGFBPEQ4DOIWQYX23SHVGMTNGDTIEEYTGGKVQQ, seg_pos={0 0}, upload=518946]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Dec 30 07:26:48.905429255 (591h57m48.77651229s) [noise=true , address=45.91.116.5:4300 , piece_id=YZMX3AQIOSGZNCOGFBPEQ4DOIWQYX23SHVGMTNGDTIEEYTGGKVQQ, seg_pos={0 0}, upload=518946]: Long tail cancellation due to optimal threshold Dec 30 07:26:48.905524476 (591h57m48.776607511s) [noise=true , address=145.79.227.17:4972 , piece_id=2QAO4DEM2BL4RRMJRD6L2T3XXEVJWLBW6JD6XOXM2DW3VWGAACLA, seg_pos={0 0}, upload=518946]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: expected piece hash; context canceled; EOF Dec 30 07:26:48.905529866 (591h57m48.776612911s) [noise=true , address=145.79.227.17:4972 , piece_id=2QAO4DEM2BL4RRMJRD6L2T3XXEVJWLBW6JD6XOXM2DW3VWGAACLA, seg_pos={0 0}, upload=518946]: Long tail cancellation due to optimal threshold Dec 30 07:26:48.905545416 (591h57m48.776628451s) [seg_pos={0 0}, upload=518946]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 07:26:48.905609698 (591h57m48.776692733s) [upload=518946]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 07:26:48.905613896 (591h57m48.776696931s) [upload=518946]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 07:26:49.197354886 (591h57m49.068437921s) [upload=518946]: Done uploading segments... Dec 30 07:26:49.197373121 (591h57m49.068456156s) [upload=518946]: Upload finished. err: <nil> Duration: 30.003768941s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 19 out of 60 is failed (threshold=6) [.....X....XXX...XXXX.........X.X.....X..XXXXXX.X.....X......] duration: 3.615685508s trace: https://jaeger.dev.storj.io/trace/220b42f7e5b29694 FAIL (2025-12-30 07:26:49 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.615710126s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 6.69611607s trace: https://jaeger.dev.storj.io/trace/4c2f4efaff0e0ea0 FAIL (2025-12-30 07:26:47 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== load=518292]: Uploading piece... Dec 30 06:41:13.598100152 (591h12m13.469183187s) [noise=true , address=81.171.14.70:4041 , piece_id=NORNA6WFTGPNOHU7MPTNNB5ONTBS5DJGCN2NWIZAEPYSUSIOSGEQ, seg_pos={0 0}, upload=518292]: Uploading piece... Dec 30 06:41:13.598587971 (591h12m13.469671016s) [noise=true , address=89.149.201.147:4007 , piece_id=HZLSYGSBP4TJH5V5XF3Q3AETJU625ERNDBD3VXW2HUZTTPAHLBMA, seg_pos={0 0}, upload=518292]: Uploading piece... Dec 30 06:41:13.599093143 (591h12m13.470176178s) [noise=true , address=89.149.201.147:4008 , piece_id=XYFRNJVIFFZA5S36KFPGZPVCPLRL2OW5QBJD3UR37VPPA2RX3RJA, seg_pos={0 0}, upload=518292]: Uploading piece... Dec 30 06:41:13.599699999 (591h12m13.470783034s) [noise=true , address=81.171.14.70:4039 , piece_id=DEJDXFDHOSAMIP2FN5LJBJNBB4QLJSE2AUDW6IMA6IC4VRY7VAGQ, seg_pos={0 0}, upload=518292]: Uploading piece... Dec 30 06:41:13.600136169 (591h12m13.471219204s) [noise=true , address=185.167.204.205:4034 , piece_id=OWZHQB5ZDT2QZ2QAHZJVRWQ222MX6CWLKJUOIAAGSMAVLWGJAOBA, seg_pos={0 0}, upload=518292]: Uploading piece... Dec 30 06:41:13.600543184 (591h12m13.471626229s) [noise=true , address=185.167.204.206:4058 , piece_id=ROOCGK64M5RHIWNAVCSZNYTZ6M2ORWB7MJ27XJVDTFVQIKJZHY4A, seg_pos={0 0}, upload=518292]: Uploading piece... Dec 30 06:41:13.601048286 (591h12m13.472131332s) [noise=true , address=81.171.14.70:4046 , piece_id=B2MTUNKMUP5S7XW4SVQ5QI2ZO3VFB5QVAKBBY4BIZIUILM6O7VYQ, seg_pos={0 0}, upload=518292]: Uploading piece... Dec 30 06:41:13.601698784 (591h12m13.472781819s) [noise=true , address=185.167.204.206:4064 , piece_id=H4PP7HPHXRX2LOZ4776AMWAJDN7OC435E2K4BXI7XOZYKTB4YUNA, seg_pos={0 0}, upload=518292]: Uploading piece... Dec 30 06:41:13.602168259 (591h12m13.473251304s) [noise=true , address=185.167.204.206:4065 , piece_id=LD3J4656Q5QEUCMJ5OKEYATEVRIKSXO7XGARTN25BVM7FDIYOWKA, seg_pos={0 0}, upload=518292]: Uploading piece... Dec 30 06:41:13.602782708 (591h12m13.473865753s) [noise=true , address=81.171.14.70:4027 , piece_id=6WEORWX5QOMFSJUEADIIBHHIPKNFP4NJPZAJADBUTPUSG7MZJMOQ, seg_pos={0 0}, upload=518292]: Uploading piece... Dec 30 06:41:14.423395002 (591h12m14.294478047s) [upload=518292]: Next returned nil segment Dec 30 06:41:14.423414068 (591h12m14.294497103s) [upload=518292]: Waiting for error group managing segments... Dec 30 06:41:14.817650316 (591h12m14.688733351s) [noise=true , address=185.167.204.205:4029 , piece_id=KUOMMYJ2F4NIK6VN4C5M3Y3TMHYEZH7QFNJ6K3CLOAVI5R6VKTHQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:14.817870054 (591h12m14.688953099s) [noise=true , address=185.167.204.205:4019 , piece_id=H6AH654VGX6NHBUKODIWGL7I2RKE3DDMY4UM6VDONP2EP3NPDJBA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:14.818623698 (591h12m14.689706744s) [noise=true , address=185.167.204.206:4065 , piece_id=LD3J4656Q5QEUCMJ5OKEYATEVRIKSXO7XGARTN25BVM7FDIYOWKA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:14.818932436 (591h12m14.690015481s) [noise=true , address=185.167.204.206:4061 , piece_id=T2XS4IFPQSVS3T45SPQ5ZDDVT2VMTMXHKL2ZMX4OIOPRR67SCD2Q, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:14.819147046 (591h12m14.690230091s) [noise=true , address=185.167.204.206:4071 , piece_id=QI2TOY5KCDZVWCCLMAJXVUWD2HB7S7IOTEAV2EHAMUNHKOFKTNYA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:14.819678578 (591h12m14.690761623s) [noise=true , address=185.167.204.206:4059 , piece_id=LMOGTWX2YIRR3EKNROK43O2B27S5L5U6GYB7HBVRBPQPH4WDDWZQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:14.819905028 (591h12m14.690988063s) [noise=true , address=185.167.204.206:4058 , piece_id=ROOCGK64M5RHIWNAVCSZNYTZ6M2ORWB7MJ27XJVDTFVQIKJZHY4A, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:14.820479312 (591h12m14.691562357s) [noise=true , address=185.167.204.205:4021 , piece_id=KP7J5XSY4SKSKZG7XXFZMNKMRP4472PXI7CW2UQBZICN4MAFO5RQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:14.823149296 (591h12m14.694232331s) [noise=true , address=185.167.204.205:4034 , piece_id=OWZHQB5ZDT2QZ2QAHZJVRWQ222MX6CWLKJUOIAAGSMAVLWGJAOBA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:14.827906996 (591h12m14.698990041s) [noise=true , address=185.167.204.206:4055 , piece_id=VYEDR6XHZBZMTVJ5U5OF67WZOGVMCMQLXXBQUI7DHBCCVQ2RRERQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:14.828003769 (591h12m14.699086814s) [noise=true , address=185.167.204.205:4016 , piece_id=I6MUDLAXUUDJL2C4HFE22C2L56A7D6HSCNWXKWJ7OKO2N4RA4W7Q, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:14.828893203 (591h12m14.699976238s) [noise=true , address=185.167.204.206:4064 , piece_id=H4PP7HPHXRX2LOZ4776AMWAJDN7OC435E2K4BXI7XOZYKTB4YUNA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:14.829477074 (591h12m14.700560109s) [noise=true , address=185.167.204.205:4017 , piece_id=CZP65KETMGOHFDMV7KENRL4V36UHBPWRDCBBPUM5NE5L2PV4AQCQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:14.829774972 (591h12m14.700858007s) [noise=true , address=185.167.204.206:4043 , piece_id=G7VZYID4TXGNPDN63NQULXOVSX42WMFFNPBX6KQJ5L2WSW4D2YGQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:14.831734493 (591h12m14.702817528s) [noise=true , address=185.167.204.206:4049 , piece_id=BQ7JEGSGPIFJUGZFBYRPDMBNPZ24BG6TEG4VUM4JUBTE4T2RF4SA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:15.405688561 (591h12m15.276771606s) [noise=true , address=81.171.14.70:4027 , piece_id=6WEORWX5QOMFSJUEADIIBHHIPKNFP4NJPZAJADBUTPUSG7MZJMOQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:15.417110440 (591h12m15.288193485s) [noise=true , address=81.171.14.70:4031 , piece_id=U6Z4SNK32ZSWQ7NDR7ACXGRC4BADDOPR65ILBR64QLGVNDBADFFQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:15.425600446 (591h12m15.296683481s) [noise=true , address=81.171.14.70:4037 , piece_id=6A25LEIAQEJB3DNCTXQT6K7ZSKIGOW4QUGMNKLOG3CT2XXH25PSA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:15.499056228 (591h12m15.370139263s) [noise=true , address=81.171.14.70:4038 , piece_id=3WRHW3VGXWHOEOV5Q6CLYSF7MUVWS7QHTNHLSTCKAUERPHAH3I3Q, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:15.514998925 (591h12m15.38608197s) [noise=true , address=81.171.14.70:4041 , piece_id=NORNA6WFTGPNOHU7MPTNNB5ONTBS5DJGCN2NWIZAEPYSUSIOSGEQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:15.528579544 (591h12m15.399662589s) [noise=true , address=81.171.14.70:4030 , piece_id=KITOJIT34CHS2QZBXCW2SJPZB4RHEHCMJ6IYQ4Z5KLL24YAWBIUQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:15.528991398 (591h12m15.400074434s) [noise=true , address=81.171.14.70:4047 , piece_id=ZWV5DP4YC3MSYDWEQH4ZK3VR4WZ4DUTI4VL3WVLQKFIEKIW44LOQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:15.534844544 (591h12m15.405927579s) [noise=true , address=81.171.14.70:4035 , piece_id=XCLL5SOAPJSSGF5YYQH6ZKGOHMLS6RI6H26FOW5XIJRY3TWYNEGA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:15.915614910 (591h12m15.786697945s) [noise=true , address=81.171.14.70:4039 , piece_id=DEJDXFDHOSAMIP2FN5LJBJNBB4QLJSE2AUDW6IMA6IC4VRY7VAGQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:15.967820176 (591h12m15.838903211s) [noise=true , address=81.171.14.70:4046 , piece_id=B2MTUNKMUP5S7XW4SVQ5QI2ZO3VFB5QVAKBBY4BIZIUILM6O7VYQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:16.172116128 (591h12m16.043199163s) [noise=true , address=89.149.201.147:4007 , piece_id=HZLSYGSBP4TJH5V5XF3Q3AETJU625ERNDBD3VXW2HUZTTPAHLBMA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:16.172137559 (591h12m16.043220604s) [seg_pos={0 0}, upload=518292]: Segment reached optimal threshold of 26 pieces. Dec 30 06:41:16.172336949 (591h12m16.043419994s) [noise=true , address=89.149.201.147:4008 , piece_id=XYFRNJVIFFZA5S36KFPGZPVCPLRL2OW5QBJD3UR37VPPA2RX3RJA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QN8VP6o7qGw9Dd8EafU9xe7s7Cuf1FyqPMbnRaznrfPxfEUUn): protocol: expected piece hash; context canceled; EOF Dec 30 06:41:16.172344213 (591h12m16.043427258s) [noise=true , address=89.149.201.147:4008 , piece_id=XYFRNJVIFFZA5S36KFPGZPVCPLRL2OW5QBJD3UR37VPPA2RX3RJA, seg_pos={0 0}, upload=518292]: Long tail cancellation due to optimal threshold Dec 30 06:41:16.172554002 (591h12m16.043637037s) [noise=true , address=89.149.201.147:4020 , piece_id=N7G5DGEDYVKBTO5S6BYW3IYEOK7HAMK5JQIBMVPMPYRDHXODWEGA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YeKHJH7CoouLFJYENJhzVLNkA32hJaqjLz3dXhr78eaSGtcic): protocol: expected piece hash; context canceled; EOF Dec 30 06:41:16.172565233 (591h12m16.043648278s) [noise=true , address=89.149.201.147:4020 , piece_id=N7G5DGEDYVKBTO5S6BYW3IYEOK7HAMK5JQIBMVPMPYRDHXODWEGA, seg_pos={0 0}, upload=518292]: Long tail cancellation due to optimal threshold Dec 30 06:41:16.172587506 (591h12m16.043670541s) [seg_pos={0 0}, upload=518292]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 30 06:41:16.172717945 (591h12m16.04380098s) [upload=518292]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 06:41:16.172724567 (591h12m16.043807602s) [upload=518292]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 06:41:16.404092237 (591h12m16.275175292s) [upload=518292]: Done uploading segments... Dec 30 06:41:16.404119940 (591h12m16.275202975s) [upload=518292]: Upload finished. err: <nil> Duration: 6.696212534s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [....XX...X.....X....XXX..X.....X...X..X.XX..X...X....XXX....] duration: 5.959653677s trace: https://jaeger.dev.storj.io/trace/5930c8857cdaeba1 FAIL (2025-12-30 07:26:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: C5PXYN2XME2CS2N7LUEKJ76R7ZOJMNYMYXUMS42JCEKYZTDALSSQ) context deadline exceeded; piecestore: (Node ID: 12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9, Piece ID: ZQDEAEO35DTGL37JA2MYQLL3GWGJFPRD7PU4HTOSUXJOX23G4WAA) context deadline exceeded ======LOGS====== Dec 30 07:23:04.168104272 (591h54m4.039187317s) [upload=518894]: Starting upload Dec 30 07:23:04.168243527 (591h54m4.039326572s) [upload=518894]: Done starting upload Dec 30 07:23:04.168324552 (591h54m4.039407597s) [upload=518894]: Uploading segments... Dec 30 07:23:04.168580919 (591h54m4.039663965s) [upload=518894]: Got next segment. Inline: false Dec 30 07:23:04.168639501 (591h54m4.039722546s) [upload=518894]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 07:23:04.168653718 (591h54m4.039736763s) [upload=518894]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 07:23:04.411075136 (591h54m4.282158191s) [seg_pos={0 0}, upload=518894]: Begin upload segment... Dec 30 07:23:04.411153746 (591h54m4.282236791s) [seg_pos={0 0}, upload=518894]: Done begin upload segment. Dec 30 07:23:04.411194513 (591h54m4.282277558s) [noise=true , address=145.79.227.41:4087 , piece_id=PMBO6I23O5BPHZP6R3HRF6RLWVRGDJGJYLKALUQSRO2Y5Q5JBBRA, seg_pos={0 0}, upload=518894]: Uploading piece... Dec 30 07:23:04.411378775 (591h54m4.28246182s) [noise=true , address=145.79.227.48:4303 , piece_id=NJELZTCH2CL6PKTGXDIKNCQW2BG4TQTS7AVBRNZ3M4N536NE4IFA, seg_pos={0 0}, upload=518894]: Uploading piece... Dec 30 07:23:04.412986175 (591h54m4.28406922s) [noise=true , address=77.90.31.14:4664 , piece_id=ZQDEAEO35DTGL37JA2MYQLL3GWGJFPRD7PU4HTOSUXJOX23G4WAA, seg_pos={0 0}, upload=518894]: Uploading piece... Dec 30 07:23:04.413137623 (591h54m4.284220668s) [noise=true , address=145.79.227.14:4839 , piece_id=IE5O6S4ACP255IIMCSZJZPXCDEOUAM35QOJYTYPJFZ4LQER5M5PA, seg_pos={0 0}, upload=518894]: Uploading piece... Dec 30 07:23:04.413689645 (591h54m4.28477268s) [noise=true , address=145.79.227.19:4079 , piece_id=XDPDXVIUOM2E7RSFXUCISOSGG37URYAVNPU2SOHTGZ4FYEQWWRRQ, seg_pos={0 0}, upload=518894]: Uploading piece... Dec 30 07:23:04.414362846 (591h54m4.285445891s) [noise=true , address=145.79.227.39:4013 , piece_id=4YRNS665OHS57GBBC2IWVEMHF6IDGENB6YLMFIMY6Y4QUX3VFEZQ, seg_pos={0 0}, upload=518894]: Uploading piece... Dec 30 07:23:04.414733071 (591h54m4.285816127s) [noise=true , address=45.91.116.8:4047 , piece_id=C5PXYN2XME2CS2N7LUEKJ76R7ZOJMNYMYXUMS42JCEKYZTDALSSQ, seg_pos={0 0}, upload=518894]: Uploading piece... Dec 30 07:23:04.415482148 (591h54m4.286565193s) [noise=true , address=145.79.227.22:4510 , piece_id=STV6UFDLOEJBKC4Q4UMPXDJGZSCAHOREHSCZ7E4WBK2NY64PTYEQ, seg_pos={0 0}, upload=518894]: Uploading piece... Dec 30 07:23:04.415983963 (591h54m4.287067009s) [noise=true , address=45.91.116.13:4155 , piece_id=TD5IE6RMZKH6AZMKFLMY4OBGML22MTC33M5IEN66FWCPXMQFSPZQ, seg_pos={0 0}, upload=518894]: Uploading piece... Dec 30 07:23:04.415998290 (591h54m4.287081335s) [noise=true , address=145.79.227.46:4601 , piece_id=ZRW3L66YC4GLEEG6KNFSGN6ZU3AP3TICSLGNBCNOAEE7R3CGLSFA, seg_pos={0 0}, upload=518894]: Uploading piece... Dec 30 07:23:04.416524051 (591h54m4.287607086s) [noise=true , address=45.91.116.7:4010 , piece_id=RW5RJ4QTUD4VCCG42APNNMBOFXRDM3Y3FDVRUPKGIKVDLXFEICZQ, seg_pos={0 0}, upload=518894]: Uploading piece... Dec 30 07:23:04.417877038 (591h54m4.288960083s) [noise=true , address=145.79.227.42:4122 , piece_id=UU7XD5MPC2SOM52PDSVXVKPSEZNECEODZV3YAD7VZIRPED2V6MFQ, seg_pos={0 0}, upload=518894]: Uploading piece... Dec 30 07:23:04.417988591 (591h54m4.289071636s) [noise=true , address=145.79.227.12:4765 , piece_id=YMMZMFWOPNIMBVBZ3WD2PZ7YZA3PG5KNHWMF4JQNP4KFCYNI3QWQ, seg_pos={0 0}, upload=518894]: Uploading piece... Dec 30 07:23:04.418918321 (591h54m4.290001366s) [noise=true , address=145.79.227.36:4519 , piece_id=LGZU4JRPDPNLZAG5DFVJOU5Y4Z7EVZFNNSVRHTW232VIDCAQP2HQ, seg_pos={0 0}, upload=518894]: Uploading piece... Dec 30 07:23:04.419094546 (591h54m4.290177591s) [noise=true , address=45.91.116.12:4226 , piece_id=5AI7NMRMTF4DFJALI3FZLAW7S5KW4LBKGLSHZI5OPO35HTJF3YGA, seg_pos={0 0}, upload=518894]: Uploading piece... Dec 30 07:23:04.420782710 (591h54m4.291865755s) [noise=true , address=145.79.227.50:4374 , piece_id=XRDF3IFLXWOJM6MAHF2IB6PMG3CPK3ZI7CIQDKT3DAGDDP4B7URQ, seg_pos={0 0}, upload=518894]: Uploading piece... Dec 30 07:23:06.023176699 (591h54m5.894259745s) [upload=518894]: Next returned nil segment Dec 30 07:23:06.023195695 (591h54m5.89427874s) [upload=518894]: Waiting for error group managing segments... Dec 30 07:23:06.406060489 (591h54m6.277143524s) [noise=true , address=145.79.227.41:4087 , piece_id=PMBO6I23O5BPHZP6R3HRF6RLWVRGDJGJYLKALUQSRO2Y5Q5JBBRA, seg_pos={0 0}, upload=518894]: Done uploading piece. err: <nil> Dec 30 07:23:06.406878477 (591h54m6.277961522s) [noise=true , address=145.79.227.14:4839 , piece_id=IE5O6S4ACP255IIMCSZJZPXCDEOUAM35QOJYTYPJFZ4LQER5M5PA, seg_pos={0 0}, upload=518894]: Done uploading piece. err: <nil> Dec 30 07:23:06.411300568 (591h54m6.282383603s) [noise=true , address=145.79.227.48:4303 , piece_id=NJELZTCH2CL6PKTGXDIKNCQW2BG4TQTS7AVBRNZ3M4N536NE4IFA, seg_pos={0 0}, upload=518894]: Done uploading piece. err: <nil> Dec 30 07:23:06.412528676 (591h54m6.283611721s) [noise=true , address=145.79.227.42:4122 , piece_id=UU7XD5MPC2SOM52PDSVXVKPSEZNECEODZV3YAD7VZIRPED2V6MFQ, seg_pos={0 0}, upload=518894]: Done uploading piece. err: <nil> Dec 30 07:23:06.415731735 (591h54m6.28681478s) [noise=true , address=145.79.227.50:4374 , piece_id=XRDF3IFLXWOJM6MAHF2IB6PMG3CPK3ZI7CIQDKT3DAGDDP4B7URQ, seg_pos={0 0}, upload=518894]: Done uploading piece. err: <nil> Dec 30 07:23:06.424674564 (591h54m6.295757599s) [noise=true , address=145.79.227.12:4765 , piece_id=YMMZMFWOPNIMBVBZ3WD2PZ7YZA3PG5KNHWMF4JQNP4KFCYNI3QWQ, seg_pos={0 0}, upload=518894]: Done uploading piece. err: <nil> Dec 30 07:23:06.427645781 (591h54m6.298728816s) [noise=true , address=145.79.227.19:4079 , piece_id=XDPDXVIUOM2E7RSFXUCISOSGG37URYAVNPU2SOHTGZ4FYEQWWRRQ, seg_pos={0 0}, upload=518894]: Done uploading piece. err: <nil> Dec 30 07:23:06.431967981 (591h54m6.303051016s) [noise=true , address=145.79.227.46:4601 , piece_id=ZRW3L66YC4GLEEG6KNFSGN6ZU3AP3TICSLGNBCNOAEE7R3CGLSFA, seg_pos={0 0}, upload=518894]: Done uploading piece. err: <nil> Dec 30 07:23:06.432564157 (591h54m6.303647192s) [noise=true , address=145.79.227.39:4013 , piece_id=4YRNS665OHS57GBBC2IWVEMHF6IDGENB6YLMFIMY6Y4QUX3VFEZQ, seg_pos={0 0}, upload=518894]: Done uploading piece. err: <nil> Dec 30 07:23:06.436102254 (591h54m6.307185289s) [noise=true , address=145.79.227.36:4519 , piece_id=LGZU4JRPDPNLZAG5DFVJOU5Y4Z7EVZFNNSVRHTW232VIDCAQP2HQ, seg_pos={0 0}, upload=518894]: Done uploading piece. err: <nil> Dec 30 07:23:06.490019731 (591h54m6.361102766s) [noise=true , address=145.79.227.22:4510 , piece_id=STV6UFDLOEJBKC4Q4UMPXDJGZSCAHOREHSCZ7E4WBK2NY64PTYEQ, seg_pos={0 0}, upload=518894]: Done uploading piece. err: <nil> Dec 30 07:23:06.604939257 (591h54m6.476022292s) [noise=true , address=77.90.31.14:4664 , piece_id=ZQDEAEO35DTGL37JA2MYQLL3GWGJFPRD7PU4HTOSUXJOX23G4WAA, seg_pos={0 0}, upload=518894]: Done uploading piece. err: <nil> Dec 30 07:23:06.832441416 (591h54m6.703524461s) [noise=true , address=45.91.116.8:4047 , piece_id=C5PXYN2XME2CS2N7LUEKJ76R7ZOJMNYMYXUMS42JCEKYZTDALSSQ, seg_pos={0 0}, upload=518894]: Done uploading piece. err: <nil> Dec 30 07:23:06.832462966 (591h54m6.703546001s) [seg_pos={0 0}, upload=518894]: Segment reached optimal threshold of 13 pieces. Dec 30 07:23:06.832758960 (591h54m6.703842005s) [noise=true , address=45.91.116.12:4226 , piece_id=5AI7NMRMTF4DFJALI3FZLAW7S5KW4LBKGLSHZI5OPO35HTJF3YGA, seg_pos={0 0}, upload=518894]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 30 07:23:06.832769270 (591h54m6.703852315s) [noise=true , address=45.91.116.12:4226 , piece_id=5AI7NMRMTF4DFJALI3FZLAW7S5KW4LBKGLSHZI5OPO35HTJF3YGA, seg_pos={0 0}, upload=518894]: Long tail cancellation due to optimal threshold Dec 30 07:23:06.832844412 (591h54m6.703927458s) [noise=true , address=45.91.116.13:4155 , piece_id=TD5IE6RMZKH6AZMKFLMY4OBGML22MTC33M5IEN66FWCPXMQFSPZQ, seg_pos={0 0}, upload=518894]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 30 07:23:06.832861305 (591h54m6.70394434s) [noise=true , address=45.91.116.13:4155 , piece_id=TD5IE6RMZKH6AZMKFLMY4OBGML22MTC33M5IEN66FWCPXMQFSPZQ, seg_pos={0 0}, upload=518894]: Long tail cancellation due to optimal threshold Dec 30 07:23:06.833039615 (591h54m6.70412266s) [noise=true , address=45.91.116.7:4010 , piece_id=RW5RJ4QTUD4VCCG42APNNMBOFXRDM3Y3FDVRUPKGIKVDLXFEICZQ, seg_pos={0 0}, upload=518894]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: read tcp 188.166.225.170:37190->45.91.116.7:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 30 07:23:06.833047369 (591h54m6.704130404s) [noise=true , address=45.91.116.7:4010 , piece_id=RW5RJ4QTUD4VCCG42APNNMBOFXRDM3Y3FDVRUPKGIKVDLXFEICZQ, seg_pos={0 0}, upload=518894]: Long tail cancellation due to optimal threshold Dec 30 07:23:06.833068700 (591h54m6.704151745s) [seg_pos={0 0}, upload=518894]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 07:23:06.833152559 (591h54m6.704235604s) [upload=518894]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 07:23:06.833157018 (591h54m6.704240053s) [upload=518894]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 07:23:07.126513614 (591h54m6.997596659s) [upload=518894]: Done uploading segments... Dec 30 07:23:07.126531589 (591h54m6.997614624s) [upload=518894]: Upload finished. err: <nil> Duration: 5.959679467s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 38.001758991s trace: https://jaeger.dev.storj.io/trace/1d0a2f93cb06aaa4 FAIL (2025-12-30 07:26:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== loading piece... Dec 30 07:06:00.467281687 (591h37m0.338364722s) [noise=true , address=77.90.31.14:4693 , piece_id=CQ6A6DJJDCKQAAMCXW7FELWCWKAOI7YCCWKVDSTZEBXNHVKQRKPQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.467716555 (591h37m0.33879959s) [noise=true , address=145.79.227.40:4082 , piece_id=7UFAIMVNINXXXRPNXLGJ34YEO3ILVN2O7UFG3HHTIT2DKWVZSUPA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468044320 (591h37m0.339127365s) [noise=true , address=145.79.227.29:4955 , piece_id=CBV7RUAZ6QAZE3VDTZKOGTIMOY66VUJTSYQSZRC7WLUD53UWVJTQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468308362 (591h37m0.339391407s) [noise=true , address=145.79.227.17:4000 , piece_id=KN3NDKB4NODHNXUWWQQ4YBWBPJCNE7YHYEMLQ4UITRCFZB4BRSMA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468565703 (591h37m0.339648748s) [noise=true , address=145.79.227.46:4604 , piece_id=NY5A6JZW4F6E4CSX3ERMKSAL2R3IYSKQ2EVHZSPF6ZOHWQFXBYWA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468843542 (591h37m0.339926587s) [noise=true , address=77.90.31.17:4781 , piece_id=TFUDJVQZFLS2XTZEGBPTV23WYLNHZMGK5ZXNDJSMW5HSNZXWXBBA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.469105310 (591h37m0.340188345s) [noise=true , address=145.79.227.35:4494 , piece_id=U4P6QB4ZRFGAJOHVSSTZHCG4PTBGII4LXAAJXXVN5RVR4WUWUBBQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.470034750 (591h37m0.341117795s) [noise=true , address=145.79.227.49:4365 , piece_id=Z357V6BZEGQOFLFH32RQLYJJZDLURVHZAHKW3H3ZIJXMQC3ZIFBQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.470487622 (591h37m0.341570657s) [noise=true , address=64.49.13.169:4713 , piece_id=YS6FRVS5V3BECH3TJQRJBGB5SSYYGUBK2HMO25PZEUOUSC7ENNZQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.470809334 (591h37m0.341892379s) [noise=true , address=77.90.31.10:4530 , piece_id=LNRM5YW2MZLXHWHGNUMP5I7Y5RRNRESBUA2M3SQKDFEQXZ25RJ3A, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.471118874 (591h37m0.342201919s) [noise=true , address=145.79.227.44:4221 , piece_id=GDKGEN3QCGFIICNST42FA2ZLLTQCHL2QAZ7AMUNPJIGYTSHDMOJQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.471431078 (591h37m0.342514113s) [noise=true , address=45.91.116.4:4963 , piece_id=FUGWNAY33ZGP6J57FIODDQQJ2UH46ABT52AU4Y7Y74Q2AOTCXWFA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.471741530 (591h37m0.342824575s) [noise=true , address=193.149.164.143:4704 , piece_id=PQQZ6N4A7TZGGX5DVRE6NT6TSDJVQARR22F6MZ5Z6LYXLUKF4V7A, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.472073311 (591h37m0.343156356s) [noise=true , address=145.79.227.33:4901 , piece_id=FFTUVNIKVKRAZ5R44V6WLWNKKC5MQYA66RZPSMW6N5EQKY7NSGZQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:01.899399345 (591h37m1.77048239s) [upload=518641]: Next returned nil segment Dec 30 07:06:01.899418712 (591h37m1.770501747s) [upload=518641]: Waiting for error group managing segments... Dec 30 07:06:02.323303785 (591h37m2.194386821s) [noise=true , address=145.79.227.46:4604 , piece_id=NY5A6JZW4F6E4CSX3ERMKSAL2R3IYSKQ2EVHZSPF6ZOHWQFXBYWA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.339689637 (591h37m2.210772682s) [noise=true , address=145.79.227.17:4000 , piece_id=KN3NDKB4NODHNXUWWQQ4YBWBPJCNE7YHYEMLQ4UITRCFZB4BRSMA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.346460570 (591h37m2.217543615s) [noise=true , address=145.79.227.29:4955 , piece_id=CBV7RUAZ6QAZE3VDTZKOGTIMOY66VUJTSYQSZRC7WLUD53UWVJTQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.347028050 (591h37m2.218111095s) [noise=true , address=145.79.227.34:4931 , piece_id=TJ4NTHHJQ4VLPBIC7RW42DBFIBI5G7LJWBN53TPSEVAWHANHMTBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.427966770 (591h37m2.299049815s) [noise=true , address=145.79.227.5:4226 , piece_id=QKDFEH6UKIH4QYN6DE3YNH3FZPAYGNYMQSBCYGM7EAW7F4MDA66A, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.446440007 (591h37m2.317523052s) [noise=true , address=145.79.227.35:4494 , piece_id=U4P6QB4ZRFGAJOHVSSTZHCG4PTBGII4LXAAJXXVN5RVR4WUWUBBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.583477598 (591h37m2.454560643s) [noise=true , address=77.90.31.19:4859 , piece_id=XHNZMPC2MF4RALF2FZJY73UYN4XWOYJUL66RFRUXNPT64JZQE5KQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.613112418 (591h37m2.484195463s) [noise=true , address=77.90.31.17:4781 , piece_id=TFUDJVQZFLS2XTZEGBPTV23WYLNHZMGK5ZXNDJSMW5HSNZXWXBBA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.631421591 (591h37m2.502504626s) [noise=true , address=77.90.31.10:4530 , piece_id=LNRM5YW2MZLXHWHGNUMP5I7Y5RRNRESBUA2M3SQKDFEQXZ25RJ3A, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.944663583 (591h37m2.815746628s) [noise=true , address=145.79.227.49:4365 , piece_id=Z357V6BZEGQOFLFH32RQLYJJZDLURVHZAHKW3H3ZIJXMQC3ZIFBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.195837408 (591h37m3.066920453s) [noise=true , address=193.149.164.143:4704 , piece_id=PQQZ6N4A7TZGGX5DVRE6NT6TSDJVQARR22F6MZ5Z6LYXLUKF4V7A, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.269365478 (591h37m3.140448523s) [noise=true , address=145.79.227.44:4221 , piece_id=GDKGEN3QCGFIICNST42FA2ZLLTQCHL2QAZ7AMUNPJIGYTSHDMOJQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.272362485 (591h37m3.14344553s) [noise=true , address=145.79.227.30:4778 , piece_id=EVWGEZL3VCMHEM6ALESBLMU65XBE74CKVMFXMSHK2GCXFPI35RBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.340891504 (591h37m3.211974549s) [noise=true , address=145.79.227.40:4082 , piece_id=7UFAIMVNINXXXRPNXLGJ34YEO3ILVN2O7UFG3HHTIT2DKWVZSUPA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.340922954 (591h37m3.212005999s) [seg_pos={0 0}, upload=518641]: Segment reached optimal threshold of 14 pieces. Dec 30 07:06:03.341286095 (591h37m3.21236913s) [noise=true , address=77.90.31.14:4693 , piece_id=CQ6A6DJJDCKQAAMCXW7FELWCWKAOI7YCCWKVDSTZEBXNHVKQRKPQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RASrnvpAj4ToD4xNvbma1iKZEe2uHK8WosjfZwDDom8Qd17Ey): protocol: write tcp 188.166.225.170:47364->77.90.31.14:4693: use of closed network connection; write tcp 188.166.225.170:47364->77.90.31.14:4693: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47364->77.90.31.14:4693: use of closed network connection Dec 30 07:06:03.341317094 (591h37m3.212400139s) [noise=true , address=77.90.31.14:4693 , piece_id=CQ6A6DJJDCKQAAMCXW7FELWCWKAOI7YCCWKVDSTZEBXNHVKQRKPQ, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341322114 (591h37m3.212405159s) [noise=true , address=145.79.227.33:4901 , piece_id=FFTUVNIKVKRAZ5R44V6WLWNKKC5MQYA66RZPSMW6N5EQKY7NSGZQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1): protocol: expected piece hash; context canceled; EOF Dec 30 07:06:03.341336261 (591h37m3.212419306s) [noise=true , address=145.79.227.33:4901 , piece_id=FFTUVNIKVKRAZ5R44V6WLWNKKC5MQYA66RZPSMW6N5EQKY7NSGZQ, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341546141 (591h37m3.212629196s) [noise=true , address=45.91.116.4:4963 , piece_id=FUGWNAY33ZGP6J57FIODDQQJ2UH46ABT52AU4Y7Y74Q2AOTCXWFA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Eki7jYegwvkdixR5pznQBbSeJPn9GjGofxnUApidQF4kRNBK5): protocol: expected piece hash; write tcp 188.166.225.170:51728->45.91.116.4:4963: use of closed network connection; write tcp 188.166.225.170:51728->45.91.116.4:4963: use of closed network connection; EOF Dec 30 07:06:03.341555549 (591h37m3.212638594s) [noise=true , address=45.91.116.4:4963 , piece_id=FUGWNAY33ZGP6J57FIODDQQJ2UH46ABT52AU4Y7Y74Q2AOTCXWFA, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341759577 (591h37m3.212842622s) [noise=true , address=64.49.13.169:4713 , piece_id=YS6FRVS5V3BECH3TJQRJBGB5SSYYGUBK2HMO25PZEUOUSC7ENNZQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17z3AiUfbxBX2eMDnUnqo1bbgojuQxJro7PsEaPjYzQjHQ3D6K): protocol: write tcp 188.166.225.170:42002->64.49.13.169:4713: use of closed network connection; write tcp 188.166.225.170:42002->64.49.13.169:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42002->64.49.13.169:4713: use of closed network connection Dec 30 07:06:03.341769897 (591h37m3.212852942s) [noise=true , address=64.49.13.169:4713 , piece_id=YS6FRVS5V3BECH3TJQRJBGB5SSYYGUBK2HMO25PZEUOUSC7ENNZQ, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341869927 (591h37m3.212952972s) [seg_pos={0 0}, upload=518641]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 07:06:03.342018730 (591h37m3.213101775s) [upload=518641]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 07:06:03.342025634 (591h37m3.213108679s) [upload=518641]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 07:06:03.635315934 (591h37m3.506398979s) [upload=518641]: Done uploading segments... Dec 30 07:06:03.635343146 (591h37m3.506426191s) [upload=518641]: Upload finished. err: <nil> Duration: 38.001788657s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 20 out of 60 is failed (threshold=6) [X.....X....XXX...XXXX.........X.X.....X..XXXXXX.X.....X.....] duration: 3.459579498s trace: https://jaeger.dev.storj.io/trace/220b42f7e5b29694 FAIL (2025-12-30 07:26:23 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.459608693s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 7.491018149s trace: https://jaeger.dev.storj.io/trace/4c2f4efaff0e0ea0 FAIL (2025-12-30 07:25:59 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== load=518292]: Uploading piece... Dec 30 06:41:13.598100152 (591h12m13.469183187s) [noise=true , address=81.171.14.70:4041 , piece_id=NORNA6WFTGPNOHU7MPTNNB5ONTBS5DJGCN2NWIZAEPYSUSIOSGEQ, seg_pos={0 0}, upload=518292]: Uploading piece... Dec 30 06:41:13.598587971 (591h12m13.469671016s) [noise=true , address=89.149.201.147:4007 , piece_id=HZLSYGSBP4TJH5V5XF3Q3AETJU625ERNDBD3VXW2HUZTTPAHLBMA, seg_pos={0 0}, upload=518292]: Uploading piece... Dec 30 06:41:13.599093143 (591h12m13.470176178s) [noise=true , address=89.149.201.147:4008 , piece_id=XYFRNJVIFFZA5S36KFPGZPVCPLRL2OW5QBJD3UR37VPPA2RX3RJA, seg_pos={0 0}, upload=518292]: Uploading piece... Dec 30 06:41:13.599699999 (591h12m13.470783034s) [noise=true , address=81.171.14.70:4039 , piece_id=DEJDXFDHOSAMIP2FN5LJBJNBB4QLJSE2AUDW6IMA6IC4VRY7VAGQ, seg_pos={0 0}, upload=518292]: Uploading piece... Dec 30 06:41:13.600136169 (591h12m13.471219204s) [noise=true , address=185.167.204.205:4034 , piece_id=OWZHQB5ZDT2QZ2QAHZJVRWQ222MX6CWLKJUOIAAGSMAVLWGJAOBA, seg_pos={0 0}, upload=518292]: Uploading piece... Dec 30 06:41:13.600543184 (591h12m13.471626229s) [noise=true , address=185.167.204.206:4058 , piece_id=ROOCGK64M5RHIWNAVCSZNYTZ6M2ORWB7MJ27XJVDTFVQIKJZHY4A, seg_pos={0 0}, upload=518292]: Uploading piece... Dec 30 06:41:13.601048286 (591h12m13.472131332s) [noise=true , address=81.171.14.70:4046 , piece_id=B2MTUNKMUP5S7XW4SVQ5QI2ZO3VFB5QVAKBBY4BIZIUILM6O7VYQ, seg_pos={0 0}, upload=518292]: Uploading piece... Dec 30 06:41:13.601698784 (591h12m13.472781819s) [noise=true , address=185.167.204.206:4064 , piece_id=H4PP7HPHXRX2LOZ4776AMWAJDN7OC435E2K4BXI7XOZYKTB4YUNA, seg_pos={0 0}, upload=518292]: Uploading piece... Dec 30 06:41:13.602168259 (591h12m13.473251304s) [noise=true , address=185.167.204.206:4065 , piece_id=LD3J4656Q5QEUCMJ5OKEYATEVRIKSXO7XGARTN25BVM7FDIYOWKA, seg_pos={0 0}, upload=518292]: Uploading piece... Dec 30 06:41:13.602782708 (591h12m13.473865753s) [noise=true , address=81.171.14.70:4027 , piece_id=6WEORWX5QOMFSJUEADIIBHHIPKNFP4NJPZAJADBUTPUSG7MZJMOQ, seg_pos={0 0}, upload=518292]: Uploading piece... Dec 30 06:41:14.423395002 (591h12m14.294478047s) [upload=518292]: Next returned nil segment Dec 30 06:41:14.423414068 (591h12m14.294497103s) [upload=518292]: Waiting for error group managing segments... Dec 30 06:41:14.817650316 (591h12m14.688733351s) [noise=true , address=185.167.204.205:4029 , piece_id=KUOMMYJ2F4NIK6VN4C5M3Y3TMHYEZH7QFNJ6K3CLOAVI5R6VKTHQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:14.817870054 (591h12m14.688953099s) [noise=true , address=185.167.204.205:4019 , piece_id=H6AH654VGX6NHBUKODIWGL7I2RKE3DDMY4UM6VDONP2EP3NPDJBA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:14.818623698 (591h12m14.689706744s) [noise=true , address=185.167.204.206:4065 , piece_id=LD3J4656Q5QEUCMJ5OKEYATEVRIKSXO7XGARTN25BVM7FDIYOWKA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:14.818932436 (591h12m14.690015481s) [noise=true , address=185.167.204.206:4061 , piece_id=T2XS4IFPQSVS3T45SPQ5ZDDVT2VMTMXHKL2ZMX4OIOPRR67SCD2Q, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:14.819147046 (591h12m14.690230091s) [noise=true , address=185.167.204.206:4071 , piece_id=QI2TOY5KCDZVWCCLMAJXVUWD2HB7S7IOTEAV2EHAMUNHKOFKTNYA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:14.819678578 (591h12m14.690761623s) [noise=true , address=185.167.204.206:4059 , piece_id=LMOGTWX2YIRR3EKNROK43O2B27S5L5U6GYB7HBVRBPQPH4WDDWZQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:14.819905028 (591h12m14.690988063s) [noise=true , address=185.167.204.206:4058 , piece_id=ROOCGK64M5RHIWNAVCSZNYTZ6M2ORWB7MJ27XJVDTFVQIKJZHY4A, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:14.820479312 (591h12m14.691562357s) [noise=true , address=185.167.204.205:4021 , piece_id=KP7J5XSY4SKSKZG7XXFZMNKMRP4472PXI7CW2UQBZICN4MAFO5RQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:14.823149296 (591h12m14.694232331s) [noise=true , address=185.167.204.205:4034 , piece_id=OWZHQB5ZDT2QZ2QAHZJVRWQ222MX6CWLKJUOIAAGSMAVLWGJAOBA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:14.827906996 (591h12m14.698990041s) [noise=true , address=185.167.204.206:4055 , piece_id=VYEDR6XHZBZMTVJ5U5OF67WZOGVMCMQLXXBQUI7DHBCCVQ2RRERQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:14.828003769 (591h12m14.699086814s) [noise=true , address=185.167.204.205:4016 , piece_id=I6MUDLAXUUDJL2C4HFE22C2L56A7D6HSCNWXKWJ7OKO2N4RA4W7Q, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:14.828893203 (591h12m14.699976238s) [noise=true , address=185.167.204.206:4064 , piece_id=H4PP7HPHXRX2LOZ4776AMWAJDN7OC435E2K4BXI7XOZYKTB4YUNA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:14.829477074 (591h12m14.700560109s) [noise=true , address=185.167.204.205:4017 , piece_id=CZP65KETMGOHFDMV7KENRL4V36UHBPWRDCBBPUM5NE5L2PV4AQCQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:14.829774972 (591h12m14.700858007s) [noise=true , address=185.167.204.206:4043 , piece_id=G7VZYID4TXGNPDN63NQULXOVSX42WMFFNPBX6KQJ5L2WSW4D2YGQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:14.831734493 (591h12m14.702817528s) [noise=true , address=185.167.204.206:4049 , piece_id=BQ7JEGSGPIFJUGZFBYRPDMBNPZ24BG6TEG4VUM4JUBTE4T2RF4SA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:15.405688561 (591h12m15.276771606s) [noise=true , address=81.171.14.70:4027 , piece_id=6WEORWX5QOMFSJUEADIIBHHIPKNFP4NJPZAJADBUTPUSG7MZJMOQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:15.417110440 (591h12m15.288193485s) [noise=true , address=81.171.14.70:4031 , piece_id=U6Z4SNK32ZSWQ7NDR7ACXGRC4BADDOPR65ILBR64QLGVNDBADFFQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:15.425600446 (591h12m15.296683481s) [noise=true , address=81.171.14.70:4037 , piece_id=6A25LEIAQEJB3DNCTXQT6K7ZSKIGOW4QUGMNKLOG3CT2XXH25PSA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:15.499056228 (591h12m15.370139263s) [noise=true , address=81.171.14.70:4038 , piece_id=3WRHW3VGXWHOEOV5Q6CLYSF7MUVWS7QHTNHLSTCKAUERPHAH3I3Q, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:15.514998925 (591h12m15.38608197s) [noise=true , address=81.171.14.70:4041 , piece_id=NORNA6WFTGPNOHU7MPTNNB5ONTBS5DJGCN2NWIZAEPYSUSIOSGEQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:15.528579544 (591h12m15.399662589s) [noise=true , address=81.171.14.70:4030 , piece_id=KITOJIT34CHS2QZBXCW2SJPZB4RHEHCMJ6IYQ4Z5KLL24YAWBIUQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:15.528991398 (591h12m15.400074434s) [noise=true , address=81.171.14.70:4047 , piece_id=ZWV5DP4YC3MSYDWEQH4ZK3VR4WZ4DUTI4VL3WVLQKFIEKIW44LOQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:15.534844544 (591h12m15.405927579s) [noise=true , address=81.171.14.70:4035 , piece_id=XCLL5SOAPJSSGF5YYQH6ZKGOHMLS6RI6H26FOW5XIJRY3TWYNEGA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:15.915614910 (591h12m15.786697945s) [noise=true , address=81.171.14.70:4039 , piece_id=DEJDXFDHOSAMIP2FN5LJBJNBB4QLJSE2AUDW6IMA6IC4VRY7VAGQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:15.967820176 (591h12m15.838903211s) [noise=true , address=81.171.14.70:4046 , piece_id=B2MTUNKMUP5S7XW4SVQ5QI2ZO3VFB5QVAKBBY4BIZIUILM6O7VYQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:16.172116128 (591h12m16.043199163s) [noise=true , address=89.149.201.147:4007 , piece_id=HZLSYGSBP4TJH5V5XF3Q3AETJU625ERNDBD3VXW2HUZTTPAHLBMA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:16.172137559 (591h12m16.043220604s) [seg_pos={0 0}, upload=518292]: Segment reached optimal threshold of 26 pieces. Dec 30 06:41:16.172336949 (591h12m16.043419994s) [noise=true , address=89.149.201.147:4008 , piece_id=XYFRNJVIFFZA5S36KFPGZPVCPLRL2OW5QBJD3UR37VPPA2RX3RJA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QN8VP6o7qGw9Dd8EafU9xe7s7Cuf1FyqPMbnRaznrfPxfEUUn): protocol: expected piece hash; context canceled; EOF Dec 30 06:41:16.172344213 (591h12m16.043427258s) [noise=true , address=89.149.201.147:4008 , piece_id=XYFRNJVIFFZA5S36KFPGZPVCPLRL2OW5QBJD3UR37VPPA2RX3RJA, seg_pos={0 0}, upload=518292]: Long tail cancellation due to optimal threshold Dec 30 06:41:16.172554002 (591h12m16.043637037s) [noise=true , address=89.149.201.147:4020 , piece_id=N7G5DGEDYVKBTO5S6BYW3IYEOK7HAMK5JQIBMVPMPYRDHXODWEGA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YeKHJH7CoouLFJYENJhzVLNkA32hJaqjLz3dXhr78eaSGtcic): protocol: expected piece hash; context canceled; EOF Dec 30 06:41:16.172565233 (591h12m16.043648278s) [noise=true , address=89.149.201.147:4020 , piece_id=N7G5DGEDYVKBTO5S6BYW3IYEOK7HAMK5JQIBMVPMPYRDHXODWEGA, seg_pos={0 0}, upload=518292]: Long tail cancellation due to optimal threshold Dec 30 06:41:16.172587506 (591h12m16.043670541s) [seg_pos={0 0}, upload=518292]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 30 06:41:16.172717945 (591h12m16.04380098s) [upload=518292]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 06:41:16.172724567 (591h12m16.043807602s) [upload=518292]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 06:41:16.404092237 (591h12m16.275175292s) [upload=518292]: Done uploading segments... Dec 30 06:41:16.404119940 (591h12m16.275202975s) [upload=518292]: Upload finished. err: <nil> Duration: 7.491050821s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [X....XX...X.....X....XXX..X.....X...X..X.XX..X...X....XXX...] duration: 5.915555119s trace: https://jaeger.dev.storj.io/trace/5930c8857cdaeba1 FAIL (2025-12-30 07:25:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: C5PXYN2XME2CS2N7LUEKJ76R7ZOJMNYMYXUMS42JCEKYZTDALSSQ) context deadline exceeded; piecestore: (Node ID: 12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9, Piece ID: ZQDEAEO35DTGL37JA2MYQLL3GWGJFPRD7PU4HTOSUXJOX23G4WAA) context deadline exceeded ======LOGS====== Dec 30 07:23:04.168104272 (591h54m4.039187317s) [upload=518894]: Starting upload Dec 30 07:23:04.168243527 (591h54m4.039326572s) [upload=518894]: Done starting upload Dec 30 07:23:04.168324552 (591h54m4.039407597s) [upload=518894]: Uploading segments... Dec 30 07:23:04.168580919 (591h54m4.039663965s) [upload=518894]: Got next segment. Inline: false Dec 30 07:23:04.168639501 (591h54m4.039722546s) [upload=518894]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 07:23:04.168653718 (591h54m4.039736763s) [upload=518894]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 07:23:04.411075136 (591h54m4.282158191s) [seg_pos={0 0}, upload=518894]: Begin upload segment... Dec 30 07:23:04.411153746 (591h54m4.282236791s) [seg_pos={0 0}, upload=518894]: Done begin upload segment. Dec 30 07:23:04.411194513 (591h54m4.282277558s) [noise=true , address=145.79.227.41:4087 , piece_id=PMBO6I23O5BPHZP6R3HRF6RLWVRGDJGJYLKALUQSRO2Y5Q5JBBRA, seg_pos={0 0}, upload=518894]: Uploading piece... Dec 30 07:23:04.411378775 (591h54m4.28246182s) [noise=true , address=145.79.227.48:4303 , piece_id=NJELZTCH2CL6PKTGXDIKNCQW2BG4TQTS7AVBRNZ3M4N536NE4IFA, seg_pos={0 0}, upload=518894]: Uploading piece... Dec 30 07:23:04.412986175 (591h54m4.28406922s) [noise=true , address=77.90.31.14:4664 , piece_id=ZQDEAEO35DTGL37JA2MYQLL3GWGJFPRD7PU4HTOSUXJOX23G4WAA, seg_pos={0 0}, upload=518894]: Uploading piece... Dec 30 07:23:04.413137623 (591h54m4.284220668s) [noise=true , address=145.79.227.14:4839 , piece_id=IE5O6S4ACP255IIMCSZJZPXCDEOUAM35QOJYTYPJFZ4LQER5M5PA, seg_pos={0 0}, upload=518894]: Uploading piece... Dec 30 07:23:04.413689645 (591h54m4.28477268s) [noise=true , address=145.79.227.19:4079 , piece_id=XDPDXVIUOM2E7RSFXUCISOSGG37URYAVNPU2SOHTGZ4FYEQWWRRQ, seg_pos={0 0}, upload=518894]: Uploading piece... Dec 30 07:23:04.414362846 (591h54m4.285445891s) [noise=true , address=145.79.227.39:4013 , piece_id=4YRNS665OHS57GBBC2IWVEMHF6IDGENB6YLMFIMY6Y4QUX3VFEZQ, seg_pos={0 0}, upload=518894]: Uploading piece... Dec 30 07:23:04.414733071 (591h54m4.285816127s) [noise=true , address=45.91.116.8:4047 , piece_id=C5PXYN2XME2CS2N7LUEKJ76R7ZOJMNYMYXUMS42JCEKYZTDALSSQ, seg_pos={0 0}, upload=518894]: Uploading piece... Dec 30 07:23:04.415482148 (591h54m4.286565193s) [noise=true , address=145.79.227.22:4510 , piece_id=STV6UFDLOEJBKC4Q4UMPXDJGZSCAHOREHSCZ7E4WBK2NY64PTYEQ, seg_pos={0 0}, upload=518894]: Uploading piece... Dec 30 07:23:04.415983963 (591h54m4.287067009s) [noise=true , address=45.91.116.13:4155 , piece_id=TD5IE6RMZKH6AZMKFLMY4OBGML22MTC33M5IEN66FWCPXMQFSPZQ, seg_pos={0 0}, upload=518894]: Uploading piece... Dec 30 07:23:04.415998290 (591h54m4.287081335s) [noise=true , address=145.79.227.46:4601 , piece_id=ZRW3L66YC4GLEEG6KNFSGN6ZU3AP3TICSLGNBCNOAEE7R3CGLSFA, seg_pos={0 0}, upload=518894]: Uploading piece... Dec 30 07:23:04.416524051 (591h54m4.287607086s) [noise=true , address=45.91.116.7:4010 , piece_id=RW5RJ4QTUD4VCCG42APNNMBOFXRDM3Y3FDVRUPKGIKVDLXFEICZQ, seg_pos={0 0}, upload=518894]: Uploading piece... Dec 30 07:23:04.417877038 (591h54m4.288960083s) [noise=true , address=145.79.227.42:4122 , piece_id=UU7XD5MPC2SOM52PDSVXVKPSEZNECEODZV3YAD7VZIRPED2V6MFQ, seg_pos={0 0}, upload=518894]: Uploading piece... Dec 30 07:23:04.417988591 (591h54m4.289071636s) [noise=true , address=145.79.227.12:4765 , piece_id=YMMZMFWOPNIMBVBZ3WD2PZ7YZA3PG5KNHWMF4JQNP4KFCYNI3QWQ, seg_pos={0 0}, upload=518894]: Uploading piece... Dec 30 07:23:04.418918321 (591h54m4.290001366s) [noise=true , address=145.79.227.36:4519 , piece_id=LGZU4JRPDPNLZAG5DFVJOU5Y4Z7EVZFNNSVRHTW232VIDCAQP2HQ, seg_pos={0 0}, upload=518894]: Uploading piece... Dec 30 07:23:04.419094546 (591h54m4.290177591s) [noise=true , address=45.91.116.12:4226 , piece_id=5AI7NMRMTF4DFJALI3FZLAW7S5KW4LBKGLSHZI5OPO35HTJF3YGA, seg_pos={0 0}, upload=518894]: Uploading piece... Dec 30 07:23:04.420782710 (591h54m4.291865755s) [noise=true , address=145.79.227.50:4374 , piece_id=XRDF3IFLXWOJM6MAHF2IB6PMG3CPK3ZI7CIQDKT3DAGDDP4B7URQ, seg_pos={0 0}, upload=518894]: Uploading piece... Dec 30 07:23:06.023176699 (591h54m5.894259745s) [upload=518894]: Next returned nil segment Dec 30 07:23:06.023195695 (591h54m5.89427874s) [upload=518894]: Waiting for error group managing segments... Dec 30 07:23:06.406060489 (591h54m6.277143524s) [noise=true , address=145.79.227.41:4087 , piece_id=PMBO6I23O5BPHZP6R3HRF6RLWVRGDJGJYLKALUQSRO2Y5Q5JBBRA, seg_pos={0 0}, upload=518894]: Done uploading piece. err: <nil> Dec 30 07:23:06.406878477 (591h54m6.277961522s) [noise=true , address=145.79.227.14:4839 , piece_id=IE5O6S4ACP255IIMCSZJZPXCDEOUAM35QOJYTYPJFZ4LQER5M5PA, seg_pos={0 0}, upload=518894]: Done uploading piece. err: <nil> Dec 30 07:23:06.411300568 (591h54m6.282383603s) [noise=true , address=145.79.227.48:4303 , piece_id=NJELZTCH2CL6PKTGXDIKNCQW2BG4TQTS7AVBRNZ3M4N536NE4IFA, seg_pos={0 0}, upload=518894]: Done uploading piece. err: <nil> Dec 30 07:23:06.412528676 (591h54m6.283611721s) [noise=true , address=145.79.227.42:4122 , piece_id=UU7XD5MPC2SOM52PDSVXVKPSEZNECEODZV3YAD7VZIRPED2V6MFQ, seg_pos={0 0}, upload=518894]: Done uploading piece. err: <nil> Dec 30 07:23:06.415731735 (591h54m6.28681478s) [noise=true , address=145.79.227.50:4374 , piece_id=XRDF3IFLXWOJM6MAHF2IB6PMG3CPK3ZI7CIQDKT3DAGDDP4B7URQ, seg_pos={0 0}, upload=518894]: Done uploading piece. err: <nil> Dec 30 07:23:06.424674564 (591h54m6.295757599s) [noise=true , address=145.79.227.12:4765 , piece_id=YMMZMFWOPNIMBVBZ3WD2PZ7YZA3PG5KNHWMF4JQNP4KFCYNI3QWQ, seg_pos={0 0}, upload=518894]: Done uploading piece. err: <nil> Dec 30 07:23:06.427645781 (591h54m6.298728816s) [noise=true , address=145.79.227.19:4079 , piece_id=XDPDXVIUOM2E7RSFXUCISOSGG37URYAVNPU2SOHTGZ4FYEQWWRRQ, seg_pos={0 0}, upload=518894]: Done uploading piece. err: <nil> Dec 30 07:23:06.431967981 (591h54m6.303051016s) [noise=true , address=145.79.227.46:4601 , piece_id=ZRW3L66YC4GLEEG6KNFSGN6ZU3AP3TICSLGNBCNOAEE7R3CGLSFA, seg_pos={0 0}, upload=518894]: Done uploading piece. err: <nil> Dec 30 07:23:06.432564157 (591h54m6.303647192s) [noise=true , address=145.79.227.39:4013 , piece_id=4YRNS665OHS57GBBC2IWVEMHF6IDGENB6YLMFIMY6Y4QUX3VFEZQ, seg_pos={0 0}, upload=518894]: Done uploading piece. err: <nil> Dec 30 07:23:06.436102254 (591h54m6.307185289s) [noise=true , address=145.79.227.36:4519 , piece_id=LGZU4JRPDPNLZAG5DFVJOU5Y4Z7EVZFNNSVRHTW232VIDCAQP2HQ, seg_pos={0 0}, upload=518894]: Done uploading piece. err: <nil> Dec 30 07:23:06.490019731 (591h54m6.361102766s) [noise=true , address=145.79.227.22:4510 , piece_id=STV6UFDLOEJBKC4Q4UMPXDJGZSCAHOREHSCZ7E4WBK2NY64PTYEQ, seg_pos={0 0}, upload=518894]: Done uploading piece. err: <nil> Dec 30 07:23:06.604939257 (591h54m6.476022292s) [noise=true , address=77.90.31.14:4664 , piece_id=ZQDEAEO35DTGL37JA2MYQLL3GWGJFPRD7PU4HTOSUXJOX23G4WAA, seg_pos={0 0}, upload=518894]: Done uploading piece. err: <nil> Dec 30 07:23:06.832441416 (591h54m6.703524461s) [noise=true , address=45.91.116.8:4047 , piece_id=C5PXYN2XME2CS2N7LUEKJ76R7ZOJMNYMYXUMS42JCEKYZTDALSSQ, seg_pos={0 0}, upload=518894]: Done uploading piece. err: <nil> Dec 30 07:23:06.832462966 (591h54m6.703546001s) [seg_pos={0 0}, upload=518894]: Segment reached optimal threshold of 13 pieces. Dec 30 07:23:06.832758960 (591h54m6.703842005s) [noise=true , address=45.91.116.12:4226 , piece_id=5AI7NMRMTF4DFJALI3FZLAW7S5KW4LBKGLSHZI5OPO35HTJF3YGA, seg_pos={0 0}, upload=518894]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 30 07:23:06.832769270 (591h54m6.703852315s) [noise=true , address=45.91.116.12:4226 , piece_id=5AI7NMRMTF4DFJALI3FZLAW7S5KW4LBKGLSHZI5OPO35HTJF3YGA, seg_pos={0 0}, upload=518894]: Long tail cancellation due to optimal threshold Dec 30 07:23:06.832844412 (591h54m6.703927458s) [noise=true , address=45.91.116.13:4155 , piece_id=TD5IE6RMZKH6AZMKFLMY4OBGML22MTC33M5IEN66FWCPXMQFSPZQ, seg_pos={0 0}, upload=518894]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 30 07:23:06.832861305 (591h54m6.70394434s) [noise=true , address=45.91.116.13:4155 , piece_id=TD5IE6RMZKH6AZMKFLMY4OBGML22MTC33M5IEN66FWCPXMQFSPZQ, seg_pos={0 0}, upload=518894]: Long tail cancellation due to optimal threshold Dec 30 07:23:06.833039615 (591h54m6.70412266s) [noise=true , address=45.91.116.7:4010 , piece_id=RW5RJ4QTUD4VCCG42APNNMBOFXRDM3Y3FDVRUPKGIKVDLXFEICZQ, seg_pos={0 0}, upload=518894]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: read tcp 188.166.225.170:37190->45.91.116.7:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 30 07:23:06.833047369 (591h54m6.704130404s) [noise=true , address=45.91.116.7:4010 , piece_id=RW5RJ4QTUD4VCCG42APNNMBOFXRDM3Y3FDVRUPKGIKVDLXFEICZQ, seg_pos={0 0}, upload=518894]: Long tail cancellation due to optimal threshold Dec 30 07:23:06.833068700 (591h54m6.704151745s) [seg_pos={0 0}, upload=518894]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 07:23:06.833152559 (591h54m6.704235604s) [upload=518894]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 07:23:06.833157018 (591h54m6.704240053s) [upload=518894]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 07:23:07.126513614 (591h54m6.997596659s) [upload=518894]: Done uploading segments... Dec 30 07:23:07.126531589 (591h54m6.997614624s) [upload=518894]: Upload finished. err: <nil> Duration: 5.915590164s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 20 out of 60 is failed (threshold=6) [.X.....X....XXX...XXXX.........X.X.....X..XXXXXX.X.....X....] duration: 3.351176382s trace: https://jaeger.dev.storj.io/trace/220b42f7e5b29694 FAIL (2025-12-30 07:25:48 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.351214945s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 47.152963264s trace: https://jaeger.dev.storj.io/trace/1d0a2f93cb06aaa4 FAIL (2025-12-30 07:25:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== loading piece... Dec 30 07:06:00.467281687 (591h37m0.338364722s) [noise=true , address=77.90.31.14:4693 , piece_id=CQ6A6DJJDCKQAAMCXW7FELWCWKAOI7YCCWKVDSTZEBXNHVKQRKPQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.467716555 (591h37m0.33879959s) [noise=true , address=145.79.227.40:4082 , piece_id=7UFAIMVNINXXXRPNXLGJ34YEO3ILVN2O7UFG3HHTIT2DKWVZSUPA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468044320 (591h37m0.339127365s) [noise=true , address=145.79.227.29:4955 , piece_id=CBV7RUAZ6QAZE3VDTZKOGTIMOY66VUJTSYQSZRC7WLUD53UWVJTQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468308362 (591h37m0.339391407s) [noise=true , address=145.79.227.17:4000 , piece_id=KN3NDKB4NODHNXUWWQQ4YBWBPJCNE7YHYEMLQ4UITRCFZB4BRSMA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468565703 (591h37m0.339648748s) [noise=true , address=145.79.227.46:4604 , piece_id=NY5A6JZW4F6E4CSX3ERMKSAL2R3IYSKQ2EVHZSPF6ZOHWQFXBYWA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468843542 (591h37m0.339926587s) [noise=true , address=77.90.31.17:4781 , piece_id=TFUDJVQZFLS2XTZEGBPTV23WYLNHZMGK5ZXNDJSMW5HSNZXWXBBA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.469105310 (591h37m0.340188345s) [noise=true , address=145.79.227.35:4494 , piece_id=U4P6QB4ZRFGAJOHVSSTZHCG4PTBGII4LXAAJXXVN5RVR4WUWUBBQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.470034750 (591h37m0.341117795s) [noise=true , address=145.79.227.49:4365 , piece_id=Z357V6BZEGQOFLFH32RQLYJJZDLURVHZAHKW3H3ZIJXMQC3ZIFBQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.470487622 (591h37m0.341570657s) [noise=true , address=64.49.13.169:4713 , piece_id=YS6FRVS5V3BECH3TJQRJBGB5SSYYGUBK2HMO25PZEUOUSC7ENNZQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.470809334 (591h37m0.341892379s) [noise=true , address=77.90.31.10:4530 , piece_id=LNRM5YW2MZLXHWHGNUMP5I7Y5RRNRESBUA2M3SQKDFEQXZ25RJ3A, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.471118874 (591h37m0.342201919s) [noise=true , address=145.79.227.44:4221 , piece_id=GDKGEN3QCGFIICNST42FA2ZLLTQCHL2QAZ7AMUNPJIGYTSHDMOJQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.471431078 (591h37m0.342514113s) [noise=true , address=45.91.116.4:4963 , piece_id=FUGWNAY33ZGP6J57FIODDQQJ2UH46ABT52AU4Y7Y74Q2AOTCXWFA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.471741530 (591h37m0.342824575s) [noise=true , address=193.149.164.143:4704 , piece_id=PQQZ6N4A7TZGGX5DVRE6NT6TSDJVQARR22F6MZ5Z6LYXLUKF4V7A, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.472073311 (591h37m0.343156356s) [noise=true , address=145.79.227.33:4901 , piece_id=FFTUVNIKVKRAZ5R44V6WLWNKKC5MQYA66RZPSMW6N5EQKY7NSGZQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:01.899399345 (591h37m1.77048239s) [upload=518641]: Next returned nil segment Dec 30 07:06:01.899418712 (591h37m1.770501747s) [upload=518641]: Waiting for error group managing segments... Dec 30 07:06:02.323303785 (591h37m2.194386821s) [noise=true , address=145.79.227.46:4604 , piece_id=NY5A6JZW4F6E4CSX3ERMKSAL2R3IYSKQ2EVHZSPF6ZOHWQFXBYWA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.339689637 (591h37m2.210772682s) [noise=true , address=145.79.227.17:4000 , piece_id=KN3NDKB4NODHNXUWWQQ4YBWBPJCNE7YHYEMLQ4UITRCFZB4BRSMA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.346460570 (591h37m2.217543615s) [noise=true , address=145.79.227.29:4955 , piece_id=CBV7RUAZ6QAZE3VDTZKOGTIMOY66VUJTSYQSZRC7WLUD53UWVJTQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.347028050 (591h37m2.218111095s) [noise=true , address=145.79.227.34:4931 , piece_id=TJ4NTHHJQ4VLPBIC7RW42DBFIBI5G7LJWBN53TPSEVAWHANHMTBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.427966770 (591h37m2.299049815s) [noise=true , address=145.79.227.5:4226 , piece_id=QKDFEH6UKIH4QYN6DE3YNH3FZPAYGNYMQSBCYGM7EAW7F4MDA66A, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.446440007 (591h37m2.317523052s) [noise=true , address=145.79.227.35:4494 , piece_id=U4P6QB4ZRFGAJOHVSSTZHCG4PTBGII4LXAAJXXVN5RVR4WUWUBBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.583477598 (591h37m2.454560643s) [noise=true , address=77.90.31.19:4859 , piece_id=XHNZMPC2MF4RALF2FZJY73UYN4XWOYJUL66RFRUXNPT64JZQE5KQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.613112418 (591h37m2.484195463s) [noise=true , address=77.90.31.17:4781 , piece_id=TFUDJVQZFLS2XTZEGBPTV23WYLNHZMGK5ZXNDJSMW5HSNZXWXBBA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.631421591 (591h37m2.502504626s) [noise=true , address=77.90.31.10:4530 , piece_id=LNRM5YW2MZLXHWHGNUMP5I7Y5RRNRESBUA2M3SQKDFEQXZ25RJ3A, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.944663583 (591h37m2.815746628s) [noise=true , address=145.79.227.49:4365 , piece_id=Z357V6BZEGQOFLFH32RQLYJJZDLURVHZAHKW3H3ZIJXMQC3ZIFBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.195837408 (591h37m3.066920453s) [noise=true , address=193.149.164.143:4704 , piece_id=PQQZ6N4A7TZGGX5DVRE6NT6TSDJVQARR22F6MZ5Z6LYXLUKF4V7A, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.269365478 (591h37m3.140448523s) [noise=true , address=145.79.227.44:4221 , piece_id=GDKGEN3QCGFIICNST42FA2ZLLTQCHL2QAZ7AMUNPJIGYTSHDMOJQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.272362485 (591h37m3.14344553s) [noise=true , address=145.79.227.30:4778 , piece_id=EVWGEZL3VCMHEM6ALESBLMU65XBE74CKVMFXMSHK2GCXFPI35RBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.340891504 (591h37m3.211974549s) [noise=true , address=145.79.227.40:4082 , piece_id=7UFAIMVNINXXXRPNXLGJ34YEO3ILVN2O7UFG3HHTIT2DKWVZSUPA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.340922954 (591h37m3.212005999s) [seg_pos={0 0}, upload=518641]: Segment reached optimal threshold of 14 pieces. Dec 30 07:06:03.341286095 (591h37m3.21236913s) [noise=true , address=77.90.31.14:4693 , piece_id=CQ6A6DJJDCKQAAMCXW7FELWCWKAOI7YCCWKVDSTZEBXNHVKQRKPQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RASrnvpAj4ToD4xNvbma1iKZEe2uHK8WosjfZwDDom8Qd17Ey): protocol: write tcp 188.166.225.170:47364->77.90.31.14:4693: use of closed network connection; write tcp 188.166.225.170:47364->77.90.31.14:4693: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47364->77.90.31.14:4693: use of closed network connection Dec 30 07:06:03.341317094 (591h37m3.212400139s) [noise=true , address=77.90.31.14:4693 , piece_id=CQ6A6DJJDCKQAAMCXW7FELWCWKAOI7YCCWKVDSTZEBXNHVKQRKPQ, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341322114 (591h37m3.212405159s) [noise=true , address=145.79.227.33:4901 , piece_id=FFTUVNIKVKRAZ5R44V6WLWNKKC5MQYA66RZPSMW6N5EQKY7NSGZQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1): protocol: expected piece hash; context canceled; EOF Dec 30 07:06:03.341336261 (591h37m3.212419306s) [noise=true , address=145.79.227.33:4901 , piece_id=FFTUVNIKVKRAZ5R44V6WLWNKKC5MQYA66RZPSMW6N5EQKY7NSGZQ, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341546141 (591h37m3.212629196s) [noise=true , address=45.91.116.4:4963 , piece_id=FUGWNAY33ZGP6J57FIODDQQJ2UH46ABT52AU4Y7Y74Q2AOTCXWFA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Eki7jYegwvkdixR5pznQBbSeJPn9GjGofxnUApidQF4kRNBK5): protocol: expected piece hash; write tcp 188.166.225.170:51728->45.91.116.4:4963: use of closed network connection; write tcp 188.166.225.170:51728->45.91.116.4:4963: use of closed network connection; EOF Dec 30 07:06:03.341555549 (591h37m3.212638594s) [noise=true , address=45.91.116.4:4963 , piece_id=FUGWNAY33ZGP6J57FIODDQQJ2UH46ABT52AU4Y7Y74Q2AOTCXWFA, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341759577 (591h37m3.212842622s) [noise=true , address=64.49.13.169:4713 , piece_id=YS6FRVS5V3BECH3TJQRJBGB5SSYYGUBK2HMO25PZEUOUSC7ENNZQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17z3AiUfbxBX2eMDnUnqo1bbgojuQxJro7PsEaPjYzQjHQ3D6K): protocol: write tcp 188.166.225.170:42002->64.49.13.169:4713: use of closed network connection; write tcp 188.166.225.170:42002->64.49.13.169:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42002->64.49.13.169:4713: use of closed network connection Dec 30 07:06:03.341769897 (591h37m3.212852942s) [noise=true , address=64.49.13.169:4713 , piece_id=YS6FRVS5V3BECH3TJQRJBGB5SSYYGUBK2HMO25PZEUOUSC7ENNZQ, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341869927 (591h37m3.212952972s) [seg_pos={0 0}, upload=518641]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 07:06:03.342018730 (591h37m3.213101775s) [upload=518641]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 07:06:03.342025634 (591h37m3.213108679s) [upload=518641]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 07:06:03.635315934 (591h37m3.506398979s) [upload=518641]: Done uploading segments... Dec 30 07:06:03.635343146 (591h37m3.506426191s) [upload=518641]: Upload finished. err: <nil> Duration: 47.153005585s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 10.376930706s trace: https://jaeger.dev.storj.io/trace/4c2f4efaff0e0ea0 FAIL (2025-12-30 07:24:54 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== load=518292]: Uploading piece... Dec 30 06:41:13.598100152 (591h12m13.469183187s) [noise=true , address=81.171.14.70:4041 , piece_id=NORNA6WFTGPNOHU7MPTNNB5ONTBS5DJGCN2NWIZAEPYSUSIOSGEQ, seg_pos={0 0}, upload=518292]: Uploading piece... Dec 30 06:41:13.598587971 (591h12m13.469671016s) [noise=true , address=89.149.201.147:4007 , piece_id=HZLSYGSBP4TJH5V5XF3Q3AETJU625ERNDBD3VXW2HUZTTPAHLBMA, seg_pos={0 0}, upload=518292]: Uploading piece... Dec 30 06:41:13.599093143 (591h12m13.470176178s) [noise=true , address=89.149.201.147:4008 , piece_id=XYFRNJVIFFZA5S36KFPGZPVCPLRL2OW5QBJD3UR37VPPA2RX3RJA, seg_pos={0 0}, upload=518292]: Uploading piece... Dec 30 06:41:13.599699999 (591h12m13.470783034s) [noise=true , address=81.171.14.70:4039 , piece_id=DEJDXFDHOSAMIP2FN5LJBJNBB4QLJSE2AUDW6IMA6IC4VRY7VAGQ, seg_pos={0 0}, upload=518292]: Uploading piece... Dec 30 06:41:13.600136169 (591h12m13.471219204s) [noise=true , address=185.167.204.205:4034 , piece_id=OWZHQB5ZDT2QZ2QAHZJVRWQ222MX6CWLKJUOIAAGSMAVLWGJAOBA, seg_pos={0 0}, upload=518292]: Uploading piece... Dec 30 06:41:13.600543184 (591h12m13.471626229s) [noise=true , address=185.167.204.206:4058 , piece_id=ROOCGK64M5RHIWNAVCSZNYTZ6M2ORWB7MJ27XJVDTFVQIKJZHY4A, seg_pos={0 0}, upload=518292]: Uploading piece... Dec 30 06:41:13.601048286 (591h12m13.472131332s) [noise=true , address=81.171.14.70:4046 , piece_id=B2MTUNKMUP5S7XW4SVQ5QI2ZO3VFB5QVAKBBY4BIZIUILM6O7VYQ, seg_pos={0 0}, upload=518292]: Uploading piece... Dec 30 06:41:13.601698784 (591h12m13.472781819s) [noise=true , address=185.167.204.206:4064 , piece_id=H4PP7HPHXRX2LOZ4776AMWAJDN7OC435E2K4BXI7XOZYKTB4YUNA, seg_pos={0 0}, upload=518292]: Uploading piece... Dec 30 06:41:13.602168259 (591h12m13.473251304s) [noise=true , address=185.167.204.206:4065 , piece_id=LD3J4656Q5QEUCMJ5OKEYATEVRIKSXO7XGARTN25BVM7FDIYOWKA, seg_pos={0 0}, upload=518292]: Uploading piece... Dec 30 06:41:13.602782708 (591h12m13.473865753s) [noise=true , address=81.171.14.70:4027 , piece_id=6WEORWX5QOMFSJUEADIIBHHIPKNFP4NJPZAJADBUTPUSG7MZJMOQ, seg_pos={0 0}, upload=518292]: Uploading piece... Dec 30 06:41:14.423395002 (591h12m14.294478047s) [upload=518292]: Next returned nil segment Dec 30 06:41:14.423414068 (591h12m14.294497103s) [upload=518292]: Waiting for error group managing segments... Dec 30 06:41:14.817650316 (591h12m14.688733351s) [noise=true , address=185.167.204.205:4029 , piece_id=KUOMMYJ2F4NIK6VN4C5M3Y3TMHYEZH7QFNJ6K3CLOAVI5R6VKTHQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:14.817870054 (591h12m14.688953099s) [noise=true , address=185.167.204.205:4019 , piece_id=H6AH654VGX6NHBUKODIWGL7I2RKE3DDMY4UM6VDONP2EP3NPDJBA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:14.818623698 (591h12m14.689706744s) [noise=true , address=185.167.204.206:4065 , piece_id=LD3J4656Q5QEUCMJ5OKEYATEVRIKSXO7XGARTN25BVM7FDIYOWKA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:14.818932436 (591h12m14.690015481s) [noise=true , address=185.167.204.206:4061 , piece_id=T2XS4IFPQSVS3T45SPQ5ZDDVT2VMTMXHKL2ZMX4OIOPRR67SCD2Q, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:14.819147046 (591h12m14.690230091s) [noise=true , address=185.167.204.206:4071 , piece_id=QI2TOY5KCDZVWCCLMAJXVUWD2HB7S7IOTEAV2EHAMUNHKOFKTNYA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:14.819678578 (591h12m14.690761623s) [noise=true , address=185.167.204.206:4059 , piece_id=LMOGTWX2YIRR3EKNROK43O2B27S5L5U6GYB7HBVRBPQPH4WDDWZQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:14.819905028 (591h12m14.690988063s) [noise=true , address=185.167.204.206:4058 , piece_id=ROOCGK64M5RHIWNAVCSZNYTZ6M2ORWB7MJ27XJVDTFVQIKJZHY4A, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:14.820479312 (591h12m14.691562357s) [noise=true , address=185.167.204.205:4021 , piece_id=KP7J5XSY4SKSKZG7XXFZMNKMRP4472PXI7CW2UQBZICN4MAFO5RQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:14.823149296 (591h12m14.694232331s) [noise=true , address=185.167.204.205:4034 , piece_id=OWZHQB5ZDT2QZ2QAHZJVRWQ222MX6CWLKJUOIAAGSMAVLWGJAOBA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:14.827906996 (591h12m14.698990041s) [noise=true , address=185.167.204.206:4055 , piece_id=VYEDR6XHZBZMTVJ5U5OF67WZOGVMCMQLXXBQUI7DHBCCVQ2RRERQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:14.828003769 (591h12m14.699086814s) [noise=true , address=185.167.204.205:4016 , piece_id=I6MUDLAXUUDJL2C4HFE22C2L56A7D6HSCNWXKWJ7OKO2N4RA4W7Q, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:14.828893203 (591h12m14.699976238s) [noise=true , address=185.167.204.206:4064 , piece_id=H4PP7HPHXRX2LOZ4776AMWAJDN7OC435E2K4BXI7XOZYKTB4YUNA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:14.829477074 (591h12m14.700560109s) [noise=true , address=185.167.204.205:4017 , piece_id=CZP65KETMGOHFDMV7KENRL4V36UHBPWRDCBBPUM5NE5L2PV4AQCQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:14.829774972 (591h12m14.700858007s) [noise=true , address=185.167.204.206:4043 , piece_id=G7VZYID4TXGNPDN63NQULXOVSX42WMFFNPBX6KQJ5L2WSW4D2YGQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:14.831734493 (591h12m14.702817528s) [noise=true , address=185.167.204.206:4049 , piece_id=BQ7JEGSGPIFJUGZFBYRPDMBNPZ24BG6TEG4VUM4JUBTE4T2RF4SA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:15.405688561 (591h12m15.276771606s) [noise=true , address=81.171.14.70:4027 , piece_id=6WEORWX5QOMFSJUEADIIBHHIPKNFP4NJPZAJADBUTPUSG7MZJMOQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:15.417110440 (591h12m15.288193485s) [noise=true , address=81.171.14.70:4031 , piece_id=U6Z4SNK32ZSWQ7NDR7ACXGRC4BADDOPR65ILBR64QLGVNDBADFFQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:15.425600446 (591h12m15.296683481s) [noise=true , address=81.171.14.70:4037 , piece_id=6A25LEIAQEJB3DNCTXQT6K7ZSKIGOW4QUGMNKLOG3CT2XXH25PSA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:15.499056228 (591h12m15.370139263s) [noise=true , address=81.171.14.70:4038 , piece_id=3WRHW3VGXWHOEOV5Q6CLYSF7MUVWS7QHTNHLSTCKAUERPHAH3I3Q, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:15.514998925 (591h12m15.38608197s) [noise=true , address=81.171.14.70:4041 , piece_id=NORNA6WFTGPNOHU7MPTNNB5ONTBS5DJGCN2NWIZAEPYSUSIOSGEQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:15.528579544 (591h12m15.399662589s) [noise=true , address=81.171.14.70:4030 , piece_id=KITOJIT34CHS2QZBXCW2SJPZB4RHEHCMJ6IYQ4Z5KLL24YAWBIUQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:15.528991398 (591h12m15.400074434s) [noise=true , address=81.171.14.70:4047 , piece_id=ZWV5DP4YC3MSYDWEQH4ZK3VR4WZ4DUTI4VL3WVLQKFIEKIW44LOQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:15.534844544 (591h12m15.405927579s) [noise=true , address=81.171.14.70:4035 , piece_id=XCLL5SOAPJSSGF5YYQH6ZKGOHMLS6RI6H26FOW5XIJRY3TWYNEGA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:15.915614910 (591h12m15.786697945s) [noise=true , address=81.171.14.70:4039 , piece_id=DEJDXFDHOSAMIP2FN5LJBJNBB4QLJSE2AUDW6IMA6IC4VRY7VAGQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:15.967820176 (591h12m15.838903211s) [noise=true , address=81.171.14.70:4046 , piece_id=B2MTUNKMUP5S7XW4SVQ5QI2ZO3VFB5QVAKBBY4BIZIUILM6O7VYQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:16.172116128 (591h12m16.043199163s) [noise=true , address=89.149.201.147:4007 , piece_id=HZLSYGSBP4TJH5V5XF3Q3AETJU625ERNDBD3VXW2HUZTTPAHLBMA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:16.172137559 (591h12m16.043220604s) [seg_pos={0 0}, upload=518292]: Segment reached optimal threshold of 26 pieces. Dec 30 06:41:16.172336949 (591h12m16.043419994s) [noise=true , address=89.149.201.147:4008 , piece_id=XYFRNJVIFFZA5S36KFPGZPVCPLRL2OW5QBJD3UR37VPPA2RX3RJA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QN8VP6o7qGw9Dd8EafU9xe7s7Cuf1FyqPMbnRaznrfPxfEUUn): protocol: expected piece hash; context canceled; EOF Dec 30 06:41:16.172344213 (591h12m16.043427258s) [noise=true , address=89.149.201.147:4008 , piece_id=XYFRNJVIFFZA5S36KFPGZPVCPLRL2OW5QBJD3UR37VPPA2RX3RJA, seg_pos={0 0}, upload=518292]: Long tail cancellation due to optimal threshold Dec 30 06:41:16.172554002 (591h12m16.043637037s) [noise=true , address=89.149.201.147:4020 , piece_id=N7G5DGEDYVKBTO5S6BYW3IYEOK7HAMK5JQIBMVPMPYRDHXODWEGA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YeKHJH7CoouLFJYENJhzVLNkA32hJaqjLz3dXhr78eaSGtcic): protocol: expected piece hash; context canceled; EOF Dec 30 06:41:16.172565233 (591h12m16.043648278s) [noise=true , address=89.149.201.147:4020 , piece_id=N7G5DGEDYVKBTO5S6BYW3IYEOK7HAMK5JQIBMVPMPYRDHXODWEGA, seg_pos={0 0}, upload=518292]: Long tail cancellation due to optimal threshold Dec 30 06:41:16.172587506 (591h12m16.043670541s) [seg_pos={0 0}, upload=518292]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 30 06:41:16.172717945 (591h12m16.04380098s) [upload=518292]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 06:41:16.172724567 (591h12m16.043807602s) [upload=518292]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 06:41:16.404092237 (591h12m16.275175292s) [upload=518292]: Done uploading segments... Dec 30 06:41:16.404119940 (591h12m16.275202975s) [upload=518292]: Upload finished. err: <nil> Duration: 10.376956937s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 15.352298956s trace: https://jaeger.dev.storj.io/trace/1d0a2f93cb06aaa4 FAIL (2025-12-30 07:24:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== loading piece... Dec 30 07:06:00.467281687 (591h37m0.338364722s) [noise=true , address=77.90.31.14:4693 , piece_id=CQ6A6DJJDCKQAAMCXW7FELWCWKAOI7YCCWKVDSTZEBXNHVKQRKPQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.467716555 (591h37m0.33879959s) [noise=true , address=145.79.227.40:4082 , piece_id=7UFAIMVNINXXXRPNXLGJ34YEO3ILVN2O7UFG3HHTIT2DKWVZSUPA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468044320 (591h37m0.339127365s) [noise=true , address=145.79.227.29:4955 , piece_id=CBV7RUAZ6QAZE3VDTZKOGTIMOY66VUJTSYQSZRC7WLUD53UWVJTQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468308362 (591h37m0.339391407s) [noise=true , address=145.79.227.17:4000 , piece_id=KN3NDKB4NODHNXUWWQQ4YBWBPJCNE7YHYEMLQ4UITRCFZB4BRSMA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468565703 (591h37m0.339648748s) [noise=true , address=145.79.227.46:4604 , piece_id=NY5A6JZW4F6E4CSX3ERMKSAL2R3IYSKQ2EVHZSPF6ZOHWQFXBYWA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468843542 (591h37m0.339926587s) [noise=true , address=77.90.31.17:4781 , piece_id=TFUDJVQZFLS2XTZEGBPTV23WYLNHZMGK5ZXNDJSMW5HSNZXWXBBA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.469105310 (591h37m0.340188345s) [noise=true , address=145.79.227.35:4494 , piece_id=U4P6QB4ZRFGAJOHVSSTZHCG4PTBGII4LXAAJXXVN5RVR4WUWUBBQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.470034750 (591h37m0.341117795s) [noise=true , address=145.79.227.49:4365 , piece_id=Z357V6BZEGQOFLFH32RQLYJJZDLURVHZAHKW3H3ZIJXMQC3ZIFBQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.470487622 (591h37m0.341570657s) [noise=true , address=64.49.13.169:4713 , piece_id=YS6FRVS5V3BECH3TJQRJBGB5SSYYGUBK2HMO25PZEUOUSC7ENNZQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.470809334 (591h37m0.341892379s) [noise=true , address=77.90.31.10:4530 , piece_id=LNRM5YW2MZLXHWHGNUMP5I7Y5RRNRESBUA2M3SQKDFEQXZ25RJ3A, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.471118874 (591h37m0.342201919s) [noise=true , address=145.79.227.44:4221 , piece_id=GDKGEN3QCGFIICNST42FA2ZLLTQCHL2QAZ7AMUNPJIGYTSHDMOJQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.471431078 (591h37m0.342514113s) [noise=true , address=45.91.116.4:4963 , piece_id=FUGWNAY33ZGP6J57FIODDQQJ2UH46ABT52AU4Y7Y74Q2AOTCXWFA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.471741530 (591h37m0.342824575s) [noise=true , address=193.149.164.143:4704 , piece_id=PQQZ6N4A7TZGGX5DVRE6NT6TSDJVQARR22F6MZ5Z6LYXLUKF4V7A, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.472073311 (591h37m0.343156356s) [noise=true , address=145.79.227.33:4901 , piece_id=FFTUVNIKVKRAZ5R44V6WLWNKKC5MQYA66RZPSMW6N5EQKY7NSGZQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:01.899399345 (591h37m1.77048239s) [upload=518641]: Next returned nil segment Dec 30 07:06:01.899418712 (591h37m1.770501747s) [upload=518641]: Waiting for error group managing segments... Dec 30 07:06:02.323303785 (591h37m2.194386821s) [noise=true , address=145.79.227.46:4604 , piece_id=NY5A6JZW4F6E4CSX3ERMKSAL2R3IYSKQ2EVHZSPF6ZOHWQFXBYWA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.339689637 (591h37m2.210772682s) [noise=true , address=145.79.227.17:4000 , piece_id=KN3NDKB4NODHNXUWWQQ4YBWBPJCNE7YHYEMLQ4UITRCFZB4BRSMA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.346460570 (591h37m2.217543615s) [noise=true , address=145.79.227.29:4955 , piece_id=CBV7RUAZ6QAZE3VDTZKOGTIMOY66VUJTSYQSZRC7WLUD53UWVJTQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.347028050 (591h37m2.218111095s) [noise=true , address=145.79.227.34:4931 , piece_id=TJ4NTHHJQ4VLPBIC7RW42DBFIBI5G7LJWBN53TPSEVAWHANHMTBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.427966770 (591h37m2.299049815s) [noise=true , address=145.79.227.5:4226 , piece_id=QKDFEH6UKIH4QYN6DE3YNH3FZPAYGNYMQSBCYGM7EAW7F4MDA66A, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.446440007 (591h37m2.317523052s) [noise=true , address=145.79.227.35:4494 , piece_id=U4P6QB4ZRFGAJOHVSSTZHCG4PTBGII4LXAAJXXVN5RVR4WUWUBBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.583477598 (591h37m2.454560643s) [noise=true , address=77.90.31.19:4859 , piece_id=XHNZMPC2MF4RALF2FZJY73UYN4XWOYJUL66RFRUXNPT64JZQE5KQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.613112418 (591h37m2.484195463s) [noise=true , address=77.90.31.17:4781 , piece_id=TFUDJVQZFLS2XTZEGBPTV23WYLNHZMGK5ZXNDJSMW5HSNZXWXBBA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.631421591 (591h37m2.502504626s) [noise=true , address=77.90.31.10:4530 , piece_id=LNRM5YW2MZLXHWHGNUMP5I7Y5RRNRESBUA2M3SQKDFEQXZ25RJ3A, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.944663583 (591h37m2.815746628s) [noise=true , address=145.79.227.49:4365 , piece_id=Z357V6BZEGQOFLFH32RQLYJJZDLURVHZAHKW3H3ZIJXMQC3ZIFBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.195837408 (591h37m3.066920453s) [noise=true , address=193.149.164.143:4704 , piece_id=PQQZ6N4A7TZGGX5DVRE6NT6TSDJVQARR22F6MZ5Z6LYXLUKF4V7A, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.269365478 (591h37m3.140448523s) [noise=true , address=145.79.227.44:4221 , piece_id=GDKGEN3QCGFIICNST42FA2ZLLTQCHL2QAZ7AMUNPJIGYTSHDMOJQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.272362485 (591h37m3.14344553s) [noise=true , address=145.79.227.30:4778 , piece_id=EVWGEZL3VCMHEM6ALESBLMU65XBE74CKVMFXMSHK2GCXFPI35RBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.340891504 (591h37m3.211974549s) [noise=true , address=145.79.227.40:4082 , piece_id=7UFAIMVNINXXXRPNXLGJ34YEO3ILVN2O7UFG3HHTIT2DKWVZSUPA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.340922954 (591h37m3.212005999s) [seg_pos={0 0}, upload=518641]: Segment reached optimal threshold of 14 pieces. Dec 30 07:06:03.341286095 (591h37m3.21236913s) [noise=true , address=77.90.31.14:4693 , piece_id=CQ6A6DJJDCKQAAMCXW7FELWCWKAOI7YCCWKVDSTZEBXNHVKQRKPQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RASrnvpAj4ToD4xNvbma1iKZEe2uHK8WosjfZwDDom8Qd17Ey): protocol: write tcp 188.166.225.170:47364->77.90.31.14:4693: use of closed network connection; write tcp 188.166.225.170:47364->77.90.31.14:4693: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47364->77.90.31.14:4693: use of closed network connection Dec 30 07:06:03.341317094 (591h37m3.212400139s) [noise=true , address=77.90.31.14:4693 , piece_id=CQ6A6DJJDCKQAAMCXW7FELWCWKAOI7YCCWKVDSTZEBXNHVKQRKPQ, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341322114 (591h37m3.212405159s) [noise=true , address=145.79.227.33:4901 , piece_id=FFTUVNIKVKRAZ5R44V6WLWNKKC5MQYA66RZPSMW6N5EQKY7NSGZQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1): protocol: expected piece hash; context canceled; EOF Dec 30 07:06:03.341336261 (591h37m3.212419306s) [noise=true , address=145.79.227.33:4901 , piece_id=FFTUVNIKVKRAZ5R44V6WLWNKKC5MQYA66RZPSMW6N5EQKY7NSGZQ, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341546141 (591h37m3.212629196s) [noise=true , address=45.91.116.4:4963 , piece_id=FUGWNAY33ZGP6J57FIODDQQJ2UH46ABT52AU4Y7Y74Q2AOTCXWFA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Eki7jYegwvkdixR5pznQBbSeJPn9GjGofxnUApidQF4kRNBK5): protocol: expected piece hash; write tcp 188.166.225.170:51728->45.91.116.4:4963: use of closed network connection; write tcp 188.166.225.170:51728->45.91.116.4:4963: use of closed network connection; EOF Dec 30 07:06:03.341555549 (591h37m3.212638594s) [noise=true , address=45.91.116.4:4963 , piece_id=FUGWNAY33ZGP6J57FIODDQQJ2UH46ABT52AU4Y7Y74Q2AOTCXWFA, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341759577 (591h37m3.212842622s) [noise=true , address=64.49.13.169:4713 , piece_id=YS6FRVS5V3BECH3TJQRJBGB5SSYYGUBK2HMO25PZEUOUSC7ENNZQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17z3AiUfbxBX2eMDnUnqo1bbgojuQxJro7PsEaPjYzQjHQ3D6K): protocol: write tcp 188.166.225.170:42002->64.49.13.169:4713: use of closed network connection; write tcp 188.166.225.170:42002->64.49.13.169:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42002->64.49.13.169:4713: use of closed network connection Dec 30 07:06:03.341769897 (591h37m3.212852942s) [noise=true , address=64.49.13.169:4713 , piece_id=YS6FRVS5V3BECH3TJQRJBGB5SSYYGUBK2HMO25PZEUOUSC7ENNZQ, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341869927 (591h37m3.212952972s) [seg_pos={0 0}, upload=518641]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 07:06:03.342018730 (591h37m3.213101775s) [upload=518641]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 07:06:03.342025634 (591h37m3.213108679s) [upload=518641]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 07:06:03.635315934 (591h37m3.506398979s) [upload=518641]: Done uploading segments... Dec 30 07:06:03.635343146 (591h37m3.506426191s) [upload=518641]: Upload finished. err: <nil> Duration: 15.352331478s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [.X....XX...X.....X....XXX..X.....X...X..X.XX..X...X....XXX..] duration: 5.968268301s trace: https://jaeger.dev.storj.io/trace/5930c8857cdaeba1 FAIL (2025-12-30 07:24:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: C5PXYN2XME2CS2N7LUEKJ76R7ZOJMNYMYXUMS42JCEKYZTDALSSQ) context deadline exceeded; piecestore: (Node ID: 12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9, Piece ID: ZQDEAEO35DTGL37JA2MYQLL3GWGJFPRD7PU4HTOSUXJOX23G4WAA) context deadline exceeded ======LOGS====== Dec 30 07:23:04.168104272 (591h54m4.039187317s) [upload=518894]: Starting upload Dec 30 07:23:04.168243527 (591h54m4.039326572s) [upload=518894]: Done starting upload Dec 30 07:23:04.168324552 (591h54m4.039407597s) [upload=518894]: Uploading segments... Dec 30 07:23:04.168580919 (591h54m4.039663965s) [upload=518894]: Got next segment. Inline: false Dec 30 07:23:04.168639501 (591h54m4.039722546s) [upload=518894]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 07:23:04.168653718 (591h54m4.039736763s) [upload=518894]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 07:23:04.411075136 (591h54m4.282158191s) [seg_pos={0 0}, upload=518894]: Begin upload segment... Dec 30 07:23:04.411153746 (591h54m4.282236791s) [seg_pos={0 0}, upload=518894]: Done begin upload segment. Dec 30 07:23:04.411194513 (591h54m4.282277558s) [noise=true , address=145.79.227.41:4087 , piece_id=PMBO6I23O5BPHZP6R3HRF6RLWVRGDJGJYLKALUQSRO2Y5Q5JBBRA, seg_pos={0 0}, upload=518894]: Uploading piece... Dec 30 07:23:04.411378775 (591h54m4.28246182s) [noise=true , address=145.79.227.48:4303 , piece_id=NJELZTCH2CL6PKTGXDIKNCQW2BG4TQTS7AVBRNZ3M4N536NE4IFA, seg_pos={0 0}, upload=518894]: Uploading piece... Dec 30 07:23:04.412986175 (591h54m4.28406922s) [noise=true , address=77.90.31.14:4664 , piece_id=ZQDEAEO35DTGL37JA2MYQLL3GWGJFPRD7PU4HTOSUXJOX23G4WAA, seg_pos={0 0}, upload=518894]: Uploading piece... Dec 30 07:23:04.413137623 (591h54m4.284220668s) [noise=true , address=145.79.227.14:4839 , piece_id=IE5O6S4ACP255IIMCSZJZPXCDEOUAM35QOJYTYPJFZ4LQER5M5PA, seg_pos={0 0}, upload=518894]: Uploading piece... Dec 30 07:23:04.413689645 (591h54m4.28477268s) [noise=true , address=145.79.227.19:4079 , piece_id=XDPDXVIUOM2E7RSFXUCISOSGG37URYAVNPU2SOHTGZ4FYEQWWRRQ, seg_pos={0 0}, upload=518894]: Uploading piece... Dec 30 07:23:04.414362846 (591h54m4.285445891s) [noise=true , address=145.79.227.39:4013 , piece_id=4YRNS665OHS57GBBC2IWVEMHF6IDGENB6YLMFIMY6Y4QUX3VFEZQ, seg_pos={0 0}, upload=518894]: Uploading piece... Dec 30 07:23:04.414733071 (591h54m4.285816127s) [noise=true , address=45.91.116.8:4047 , piece_id=C5PXYN2XME2CS2N7LUEKJ76R7ZOJMNYMYXUMS42JCEKYZTDALSSQ, seg_pos={0 0}, upload=518894]: Uploading piece... Dec 30 07:23:04.415482148 (591h54m4.286565193s) [noise=true , address=145.79.227.22:4510 , piece_id=STV6UFDLOEJBKC4Q4UMPXDJGZSCAHOREHSCZ7E4WBK2NY64PTYEQ, seg_pos={0 0}, upload=518894]: Uploading piece... Dec 30 07:23:04.415983963 (591h54m4.287067009s) [noise=true , address=45.91.116.13:4155 , piece_id=TD5IE6RMZKH6AZMKFLMY4OBGML22MTC33M5IEN66FWCPXMQFSPZQ, seg_pos={0 0}, upload=518894]: Uploading piece... Dec 30 07:23:04.415998290 (591h54m4.287081335s) [noise=true , address=145.79.227.46:4601 , piece_id=ZRW3L66YC4GLEEG6KNFSGN6ZU3AP3TICSLGNBCNOAEE7R3CGLSFA, seg_pos={0 0}, upload=518894]: Uploading piece... Dec 30 07:23:04.416524051 (591h54m4.287607086s) [noise=true , address=45.91.116.7:4010 , piece_id=RW5RJ4QTUD4VCCG42APNNMBOFXRDM3Y3FDVRUPKGIKVDLXFEICZQ, seg_pos={0 0}, upload=518894]: Uploading piece... Dec 30 07:23:04.417877038 (591h54m4.288960083s) [noise=true , address=145.79.227.42:4122 , piece_id=UU7XD5MPC2SOM52PDSVXVKPSEZNECEODZV3YAD7VZIRPED2V6MFQ, seg_pos={0 0}, upload=518894]: Uploading piece... Dec 30 07:23:04.417988591 (591h54m4.289071636s) [noise=true , address=145.79.227.12:4765 , piece_id=YMMZMFWOPNIMBVBZ3WD2PZ7YZA3PG5KNHWMF4JQNP4KFCYNI3QWQ, seg_pos={0 0}, upload=518894]: Uploading piece... Dec 30 07:23:04.418918321 (591h54m4.290001366s) [noise=true , address=145.79.227.36:4519 , piece_id=LGZU4JRPDPNLZAG5DFVJOU5Y4Z7EVZFNNSVRHTW232VIDCAQP2HQ, seg_pos={0 0}, upload=518894]: Uploading piece... Dec 30 07:23:04.419094546 (591h54m4.290177591s) [noise=true , address=45.91.116.12:4226 , piece_id=5AI7NMRMTF4DFJALI3FZLAW7S5KW4LBKGLSHZI5OPO35HTJF3YGA, seg_pos={0 0}, upload=518894]: Uploading piece... Dec 30 07:23:04.420782710 (591h54m4.291865755s) [noise=true , address=145.79.227.50:4374 , piece_id=XRDF3IFLXWOJM6MAHF2IB6PMG3CPK3ZI7CIQDKT3DAGDDP4B7URQ, seg_pos={0 0}, upload=518894]: Uploading piece... Dec 30 07:23:06.023176699 (591h54m5.894259745s) [upload=518894]: Next returned nil segment Dec 30 07:23:06.023195695 (591h54m5.89427874s) [upload=518894]: Waiting for error group managing segments... Dec 30 07:23:06.406060489 (591h54m6.277143524s) [noise=true , address=145.79.227.41:4087 , piece_id=PMBO6I23O5BPHZP6R3HRF6RLWVRGDJGJYLKALUQSRO2Y5Q5JBBRA, seg_pos={0 0}, upload=518894]: Done uploading piece. err: <nil> Dec 30 07:23:06.406878477 (591h54m6.277961522s) [noise=true , address=145.79.227.14:4839 , piece_id=IE5O6S4ACP255IIMCSZJZPXCDEOUAM35QOJYTYPJFZ4LQER5M5PA, seg_pos={0 0}, upload=518894]: Done uploading piece. err: <nil> Dec 30 07:23:06.411300568 (591h54m6.282383603s) [noise=true , address=145.79.227.48:4303 , piece_id=NJELZTCH2CL6PKTGXDIKNCQW2BG4TQTS7AVBRNZ3M4N536NE4IFA, seg_pos={0 0}, upload=518894]: Done uploading piece. err: <nil> Dec 30 07:23:06.412528676 (591h54m6.283611721s) [noise=true , address=145.79.227.42:4122 , piece_id=UU7XD5MPC2SOM52PDSVXVKPSEZNECEODZV3YAD7VZIRPED2V6MFQ, seg_pos={0 0}, upload=518894]: Done uploading piece. err: <nil> Dec 30 07:23:06.415731735 (591h54m6.28681478s) [noise=true , address=145.79.227.50:4374 , piece_id=XRDF3IFLXWOJM6MAHF2IB6PMG3CPK3ZI7CIQDKT3DAGDDP4B7URQ, seg_pos={0 0}, upload=518894]: Done uploading piece. err: <nil> Dec 30 07:23:06.424674564 (591h54m6.295757599s) [noise=true , address=145.79.227.12:4765 , piece_id=YMMZMFWOPNIMBVBZ3WD2PZ7YZA3PG5KNHWMF4JQNP4KFCYNI3QWQ, seg_pos={0 0}, upload=518894]: Done uploading piece. err: <nil> Dec 30 07:23:06.427645781 (591h54m6.298728816s) [noise=true , address=145.79.227.19:4079 , piece_id=XDPDXVIUOM2E7RSFXUCISOSGG37URYAVNPU2SOHTGZ4FYEQWWRRQ, seg_pos={0 0}, upload=518894]: Done uploading piece. err: <nil> Dec 30 07:23:06.431967981 (591h54m6.303051016s) [noise=true , address=145.79.227.46:4601 , piece_id=ZRW3L66YC4GLEEG6KNFSGN6ZU3AP3TICSLGNBCNOAEE7R3CGLSFA, seg_pos={0 0}, upload=518894]: Done uploading piece. err: <nil> Dec 30 07:23:06.432564157 (591h54m6.303647192s) [noise=true , address=145.79.227.39:4013 , piece_id=4YRNS665OHS57GBBC2IWVEMHF6IDGENB6YLMFIMY6Y4QUX3VFEZQ, seg_pos={0 0}, upload=518894]: Done uploading piece. err: <nil> Dec 30 07:23:06.436102254 (591h54m6.307185289s) [noise=true , address=145.79.227.36:4519 , piece_id=LGZU4JRPDPNLZAG5DFVJOU5Y4Z7EVZFNNSVRHTW232VIDCAQP2HQ, seg_pos={0 0}, upload=518894]: Done uploading piece. err: <nil> Dec 30 07:23:06.490019731 (591h54m6.361102766s) [noise=true , address=145.79.227.22:4510 , piece_id=STV6UFDLOEJBKC4Q4UMPXDJGZSCAHOREHSCZ7E4WBK2NY64PTYEQ, seg_pos={0 0}, upload=518894]: Done uploading piece. err: <nil> Dec 30 07:23:06.604939257 (591h54m6.476022292s) [noise=true , address=77.90.31.14:4664 , piece_id=ZQDEAEO35DTGL37JA2MYQLL3GWGJFPRD7PU4HTOSUXJOX23G4WAA, seg_pos={0 0}, upload=518894]: Done uploading piece. err: <nil> Dec 30 07:23:06.832441416 (591h54m6.703524461s) [noise=true , address=45.91.116.8:4047 , piece_id=C5PXYN2XME2CS2N7LUEKJ76R7ZOJMNYMYXUMS42JCEKYZTDALSSQ, seg_pos={0 0}, upload=518894]: Done uploading piece. err: <nil> Dec 30 07:23:06.832462966 (591h54m6.703546001s) [seg_pos={0 0}, upload=518894]: Segment reached optimal threshold of 13 pieces. Dec 30 07:23:06.832758960 (591h54m6.703842005s) [noise=true , address=45.91.116.12:4226 , piece_id=5AI7NMRMTF4DFJALI3FZLAW7S5KW4LBKGLSHZI5OPO35HTJF3YGA, seg_pos={0 0}, upload=518894]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 30 07:23:06.832769270 (591h54m6.703852315s) [noise=true , address=45.91.116.12:4226 , piece_id=5AI7NMRMTF4DFJALI3FZLAW7S5KW4LBKGLSHZI5OPO35HTJF3YGA, seg_pos={0 0}, upload=518894]: Long tail cancellation due to optimal threshold Dec 30 07:23:06.832844412 (591h54m6.703927458s) [noise=true , address=45.91.116.13:4155 , piece_id=TD5IE6RMZKH6AZMKFLMY4OBGML22MTC33M5IEN66FWCPXMQFSPZQ, seg_pos={0 0}, upload=518894]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 30 07:23:06.832861305 (591h54m6.70394434s) [noise=true , address=45.91.116.13:4155 , piece_id=TD5IE6RMZKH6AZMKFLMY4OBGML22MTC33M5IEN66FWCPXMQFSPZQ, seg_pos={0 0}, upload=518894]: Long tail cancellation due to optimal threshold Dec 30 07:23:06.833039615 (591h54m6.70412266s) [noise=true , address=45.91.116.7:4010 , piece_id=RW5RJ4QTUD4VCCG42APNNMBOFXRDM3Y3FDVRUPKGIKVDLXFEICZQ, seg_pos={0 0}, upload=518894]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: read tcp 188.166.225.170:37190->45.91.116.7:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 30 07:23:06.833047369 (591h54m6.704130404s) [noise=true , address=45.91.116.7:4010 , piece_id=RW5RJ4QTUD4VCCG42APNNMBOFXRDM3Y3FDVRUPKGIKVDLXFEICZQ, seg_pos={0 0}, upload=518894]: Long tail cancellation due to optimal threshold Dec 30 07:23:06.833068700 (591h54m6.704151745s) [seg_pos={0 0}, upload=518894]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 07:23:06.833152559 (591h54m6.704235604s) [upload=518894]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 07:23:06.833157018 (591h54m6.704240053s) [upload=518894]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 07:23:07.126513614 (591h54m6.997596659s) [upload=518894]: Done uploading segments... Dec 30 07:23:07.126531589 (591h54m6.997614624s) [upload=518894]: Upload finished. err: <nil> Duration: 5.968293852s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 20 out of 60 is failed (threshold=6) [..X.....X....XXX...XXXX.........X.X.....X..XXXXXX.X.....X...] duration: 4.180193542s trace: https://jaeger.dev.storj.io/trace/220b42f7e5b29694 FAIL (2025-12-30 07:24:36 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.180218622s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 42.030103742s trace: https://jaeger.dev.storj.io/trace/1d0a2f93cb06aaa4 FAIL (2025-12-30 07:24:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== loading piece... Dec 30 07:06:00.467281687 (591h37m0.338364722s) [noise=true , address=77.90.31.14:4693 , piece_id=CQ6A6DJJDCKQAAMCXW7FELWCWKAOI7YCCWKVDSTZEBXNHVKQRKPQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.467716555 (591h37m0.33879959s) [noise=true , address=145.79.227.40:4082 , piece_id=7UFAIMVNINXXXRPNXLGJ34YEO3ILVN2O7UFG3HHTIT2DKWVZSUPA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468044320 (591h37m0.339127365s) [noise=true , address=145.79.227.29:4955 , piece_id=CBV7RUAZ6QAZE3VDTZKOGTIMOY66VUJTSYQSZRC7WLUD53UWVJTQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468308362 (591h37m0.339391407s) [noise=true , address=145.79.227.17:4000 , piece_id=KN3NDKB4NODHNXUWWQQ4YBWBPJCNE7YHYEMLQ4UITRCFZB4BRSMA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468565703 (591h37m0.339648748s) [noise=true , address=145.79.227.46:4604 , piece_id=NY5A6JZW4F6E4CSX3ERMKSAL2R3IYSKQ2EVHZSPF6ZOHWQFXBYWA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468843542 (591h37m0.339926587s) [noise=true , address=77.90.31.17:4781 , piece_id=TFUDJVQZFLS2XTZEGBPTV23WYLNHZMGK5ZXNDJSMW5HSNZXWXBBA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.469105310 (591h37m0.340188345s) [noise=true , address=145.79.227.35:4494 , piece_id=U4P6QB4ZRFGAJOHVSSTZHCG4PTBGII4LXAAJXXVN5RVR4WUWUBBQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.470034750 (591h37m0.341117795s) [noise=true , address=145.79.227.49:4365 , piece_id=Z357V6BZEGQOFLFH32RQLYJJZDLURVHZAHKW3H3ZIJXMQC3ZIFBQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.470487622 (591h37m0.341570657s) [noise=true , address=64.49.13.169:4713 , piece_id=YS6FRVS5V3BECH3TJQRJBGB5SSYYGUBK2HMO25PZEUOUSC7ENNZQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.470809334 (591h37m0.341892379s) [noise=true , address=77.90.31.10:4530 , piece_id=LNRM5YW2MZLXHWHGNUMP5I7Y5RRNRESBUA2M3SQKDFEQXZ25RJ3A, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.471118874 (591h37m0.342201919s) [noise=true , address=145.79.227.44:4221 , piece_id=GDKGEN3QCGFIICNST42FA2ZLLTQCHL2QAZ7AMUNPJIGYTSHDMOJQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.471431078 (591h37m0.342514113s) [noise=true , address=45.91.116.4:4963 , piece_id=FUGWNAY33ZGP6J57FIODDQQJ2UH46ABT52AU4Y7Y74Q2AOTCXWFA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.471741530 (591h37m0.342824575s) [noise=true , address=193.149.164.143:4704 , piece_id=PQQZ6N4A7TZGGX5DVRE6NT6TSDJVQARR22F6MZ5Z6LYXLUKF4V7A, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.472073311 (591h37m0.343156356s) [noise=true , address=145.79.227.33:4901 , piece_id=FFTUVNIKVKRAZ5R44V6WLWNKKC5MQYA66RZPSMW6N5EQKY7NSGZQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:01.899399345 (591h37m1.77048239s) [upload=518641]: Next returned nil segment Dec 30 07:06:01.899418712 (591h37m1.770501747s) [upload=518641]: Waiting for error group managing segments... Dec 30 07:06:02.323303785 (591h37m2.194386821s) [noise=true , address=145.79.227.46:4604 , piece_id=NY5A6JZW4F6E4CSX3ERMKSAL2R3IYSKQ2EVHZSPF6ZOHWQFXBYWA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.339689637 (591h37m2.210772682s) [noise=true , address=145.79.227.17:4000 , piece_id=KN3NDKB4NODHNXUWWQQ4YBWBPJCNE7YHYEMLQ4UITRCFZB4BRSMA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.346460570 (591h37m2.217543615s) [noise=true , address=145.79.227.29:4955 , piece_id=CBV7RUAZ6QAZE3VDTZKOGTIMOY66VUJTSYQSZRC7WLUD53UWVJTQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.347028050 (591h37m2.218111095s) [noise=true , address=145.79.227.34:4931 , piece_id=TJ4NTHHJQ4VLPBIC7RW42DBFIBI5G7LJWBN53TPSEVAWHANHMTBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.427966770 (591h37m2.299049815s) [noise=true , address=145.79.227.5:4226 , piece_id=QKDFEH6UKIH4QYN6DE3YNH3FZPAYGNYMQSBCYGM7EAW7F4MDA66A, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.446440007 (591h37m2.317523052s) [noise=true , address=145.79.227.35:4494 , piece_id=U4P6QB4ZRFGAJOHVSSTZHCG4PTBGII4LXAAJXXVN5RVR4WUWUBBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.583477598 (591h37m2.454560643s) [noise=true , address=77.90.31.19:4859 , piece_id=XHNZMPC2MF4RALF2FZJY73UYN4XWOYJUL66RFRUXNPT64JZQE5KQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.613112418 (591h37m2.484195463s) [noise=true , address=77.90.31.17:4781 , piece_id=TFUDJVQZFLS2XTZEGBPTV23WYLNHZMGK5ZXNDJSMW5HSNZXWXBBA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.631421591 (591h37m2.502504626s) [noise=true , address=77.90.31.10:4530 , piece_id=LNRM5YW2MZLXHWHGNUMP5I7Y5RRNRESBUA2M3SQKDFEQXZ25RJ3A, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.944663583 (591h37m2.815746628s) [noise=true , address=145.79.227.49:4365 , piece_id=Z357V6BZEGQOFLFH32RQLYJJZDLURVHZAHKW3H3ZIJXMQC3ZIFBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.195837408 (591h37m3.066920453s) [noise=true , address=193.149.164.143:4704 , piece_id=PQQZ6N4A7TZGGX5DVRE6NT6TSDJVQARR22F6MZ5Z6LYXLUKF4V7A, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.269365478 (591h37m3.140448523s) [noise=true , address=145.79.227.44:4221 , piece_id=GDKGEN3QCGFIICNST42FA2ZLLTQCHL2QAZ7AMUNPJIGYTSHDMOJQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.272362485 (591h37m3.14344553s) [noise=true , address=145.79.227.30:4778 , piece_id=EVWGEZL3VCMHEM6ALESBLMU65XBE74CKVMFXMSHK2GCXFPI35RBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.340891504 (591h37m3.211974549s) [noise=true , address=145.79.227.40:4082 , piece_id=7UFAIMVNINXXXRPNXLGJ34YEO3ILVN2O7UFG3HHTIT2DKWVZSUPA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.340922954 (591h37m3.212005999s) [seg_pos={0 0}, upload=518641]: Segment reached optimal threshold of 14 pieces. Dec 30 07:06:03.341286095 (591h37m3.21236913s) [noise=true , address=77.90.31.14:4693 , piece_id=CQ6A6DJJDCKQAAMCXW7FELWCWKAOI7YCCWKVDSTZEBXNHVKQRKPQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RASrnvpAj4ToD4xNvbma1iKZEe2uHK8WosjfZwDDom8Qd17Ey): protocol: write tcp 188.166.225.170:47364->77.90.31.14:4693: use of closed network connection; write tcp 188.166.225.170:47364->77.90.31.14:4693: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47364->77.90.31.14:4693: use of closed network connection Dec 30 07:06:03.341317094 (591h37m3.212400139s) [noise=true , address=77.90.31.14:4693 , piece_id=CQ6A6DJJDCKQAAMCXW7FELWCWKAOI7YCCWKVDSTZEBXNHVKQRKPQ, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341322114 (591h37m3.212405159s) [noise=true , address=145.79.227.33:4901 , piece_id=FFTUVNIKVKRAZ5R44V6WLWNKKC5MQYA66RZPSMW6N5EQKY7NSGZQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1): protocol: expected piece hash; context canceled; EOF Dec 30 07:06:03.341336261 (591h37m3.212419306s) [noise=true , address=145.79.227.33:4901 , piece_id=FFTUVNIKVKRAZ5R44V6WLWNKKC5MQYA66RZPSMW6N5EQKY7NSGZQ, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341546141 (591h37m3.212629196s) [noise=true , address=45.91.116.4:4963 , piece_id=FUGWNAY33ZGP6J57FIODDQQJ2UH46ABT52AU4Y7Y74Q2AOTCXWFA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Eki7jYegwvkdixR5pznQBbSeJPn9GjGofxnUApidQF4kRNBK5): protocol: expected piece hash; write tcp 188.166.225.170:51728->45.91.116.4:4963: use of closed network connection; write tcp 188.166.225.170:51728->45.91.116.4:4963: use of closed network connection; EOF Dec 30 07:06:03.341555549 (591h37m3.212638594s) [noise=true , address=45.91.116.4:4963 , piece_id=FUGWNAY33ZGP6J57FIODDQQJ2UH46ABT52AU4Y7Y74Q2AOTCXWFA, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341759577 (591h37m3.212842622s) [noise=true , address=64.49.13.169:4713 , piece_id=YS6FRVS5V3BECH3TJQRJBGB5SSYYGUBK2HMO25PZEUOUSC7ENNZQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17z3AiUfbxBX2eMDnUnqo1bbgojuQxJro7PsEaPjYzQjHQ3D6K): protocol: write tcp 188.166.225.170:42002->64.49.13.169:4713: use of closed network connection; write tcp 188.166.225.170:42002->64.49.13.169:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42002->64.49.13.169:4713: use of closed network connection Dec 30 07:06:03.341769897 (591h37m3.212852942s) [noise=true , address=64.49.13.169:4713 , piece_id=YS6FRVS5V3BECH3TJQRJBGB5SSYYGUBK2HMO25PZEUOUSC7ENNZQ, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341869927 (591h37m3.212952972s) [seg_pos={0 0}, upload=518641]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 07:06:03.342018730 (591h37m3.213101775s) [upload=518641]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 07:06:03.342025634 (591h37m3.213108679s) [upload=518641]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 07:06:03.635315934 (591h37m3.506398979s) [upload=518641]: Done uploading segments... Dec 30 07:06:03.635343146 (591h37m3.506426191s) [upload=518641]: Upload finished. err: <nil> Duration: 42.030118058s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 6.327745483s trace: https://jaeger.dev.storj.io/trace/4c2f4efaff0e0ea0 FAIL (2025-12-30 07:24:19 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== load=518292]: Uploading piece... Dec 30 06:41:13.598100152 (591h12m13.469183187s) [noise=true , address=81.171.14.70:4041 , piece_id=NORNA6WFTGPNOHU7MPTNNB5ONTBS5DJGCN2NWIZAEPYSUSIOSGEQ, seg_pos={0 0}, upload=518292]: Uploading piece... Dec 30 06:41:13.598587971 (591h12m13.469671016s) [noise=true , address=89.149.201.147:4007 , piece_id=HZLSYGSBP4TJH5V5XF3Q3AETJU625ERNDBD3VXW2HUZTTPAHLBMA, seg_pos={0 0}, upload=518292]: Uploading piece... Dec 30 06:41:13.599093143 (591h12m13.470176178s) [noise=true , address=89.149.201.147:4008 , piece_id=XYFRNJVIFFZA5S36KFPGZPVCPLRL2OW5QBJD3UR37VPPA2RX3RJA, seg_pos={0 0}, upload=518292]: Uploading piece... Dec 30 06:41:13.599699999 (591h12m13.470783034s) [noise=true , address=81.171.14.70:4039 , piece_id=DEJDXFDHOSAMIP2FN5LJBJNBB4QLJSE2AUDW6IMA6IC4VRY7VAGQ, seg_pos={0 0}, upload=518292]: Uploading piece... Dec 30 06:41:13.600136169 (591h12m13.471219204s) [noise=true , address=185.167.204.205:4034 , piece_id=OWZHQB5ZDT2QZ2QAHZJVRWQ222MX6CWLKJUOIAAGSMAVLWGJAOBA, seg_pos={0 0}, upload=518292]: Uploading piece... Dec 30 06:41:13.600543184 (591h12m13.471626229s) [noise=true , address=185.167.204.206:4058 , piece_id=ROOCGK64M5RHIWNAVCSZNYTZ6M2ORWB7MJ27XJVDTFVQIKJZHY4A, seg_pos={0 0}, upload=518292]: Uploading piece... Dec 30 06:41:13.601048286 (591h12m13.472131332s) [noise=true , address=81.171.14.70:4046 , piece_id=B2MTUNKMUP5S7XW4SVQ5QI2ZO3VFB5QVAKBBY4BIZIUILM6O7VYQ, seg_pos={0 0}, upload=518292]: Uploading piece... Dec 30 06:41:13.601698784 (591h12m13.472781819s) [noise=true , address=185.167.204.206:4064 , piece_id=H4PP7HPHXRX2LOZ4776AMWAJDN7OC435E2K4BXI7XOZYKTB4YUNA, seg_pos={0 0}, upload=518292]: Uploading piece... Dec 30 06:41:13.602168259 (591h12m13.473251304s) [noise=true , address=185.167.204.206:4065 , piece_id=LD3J4656Q5QEUCMJ5OKEYATEVRIKSXO7XGARTN25BVM7FDIYOWKA, seg_pos={0 0}, upload=518292]: Uploading piece... Dec 30 06:41:13.602782708 (591h12m13.473865753s) [noise=true , address=81.171.14.70:4027 , piece_id=6WEORWX5QOMFSJUEADIIBHHIPKNFP4NJPZAJADBUTPUSG7MZJMOQ, seg_pos={0 0}, upload=518292]: Uploading piece... Dec 30 06:41:14.423395002 (591h12m14.294478047s) [upload=518292]: Next returned nil segment Dec 30 06:41:14.423414068 (591h12m14.294497103s) [upload=518292]: Waiting for error group managing segments... Dec 30 06:41:14.817650316 (591h12m14.688733351s) [noise=true , address=185.167.204.205:4029 , piece_id=KUOMMYJ2F4NIK6VN4C5M3Y3TMHYEZH7QFNJ6K3CLOAVI5R6VKTHQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:14.817870054 (591h12m14.688953099s) [noise=true , address=185.167.204.205:4019 , piece_id=H6AH654VGX6NHBUKODIWGL7I2RKE3DDMY4UM6VDONP2EP3NPDJBA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:14.818623698 (591h12m14.689706744s) [noise=true , address=185.167.204.206:4065 , piece_id=LD3J4656Q5QEUCMJ5OKEYATEVRIKSXO7XGARTN25BVM7FDIYOWKA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:14.818932436 (591h12m14.690015481s) [noise=true , address=185.167.204.206:4061 , piece_id=T2XS4IFPQSVS3T45SPQ5ZDDVT2VMTMXHKL2ZMX4OIOPRR67SCD2Q, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:14.819147046 (591h12m14.690230091s) [noise=true , address=185.167.204.206:4071 , piece_id=QI2TOY5KCDZVWCCLMAJXVUWD2HB7S7IOTEAV2EHAMUNHKOFKTNYA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:14.819678578 (591h12m14.690761623s) [noise=true , address=185.167.204.206:4059 , piece_id=LMOGTWX2YIRR3EKNROK43O2B27S5L5U6GYB7HBVRBPQPH4WDDWZQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:14.819905028 (591h12m14.690988063s) [noise=true , address=185.167.204.206:4058 , piece_id=ROOCGK64M5RHIWNAVCSZNYTZ6M2ORWB7MJ27XJVDTFVQIKJZHY4A, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:14.820479312 (591h12m14.691562357s) [noise=true , address=185.167.204.205:4021 , piece_id=KP7J5XSY4SKSKZG7XXFZMNKMRP4472PXI7CW2UQBZICN4MAFO5RQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:14.823149296 (591h12m14.694232331s) [noise=true , address=185.167.204.205:4034 , piece_id=OWZHQB5ZDT2QZ2QAHZJVRWQ222MX6CWLKJUOIAAGSMAVLWGJAOBA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:14.827906996 (591h12m14.698990041s) [noise=true , address=185.167.204.206:4055 , piece_id=VYEDR6XHZBZMTVJ5U5OF67WZOGVMCMQLXXBQUI7DHBCCVQ2RRERQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:14.828003769 (591h12m14.699086814s) [noise=true , address=185.167.204.205:4016 , piece_id=I6MUDLAXUUDJL2C4HFE22C2L56A7D6HSCNWXKWJ7OKO2N4RA4W7Q, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:14.828893203 (591h12m14.699976238s) [noise=true , address=185.167.204.206:4064 , piece_id=H4PP7HPHXRX2LOZ4776AMWAJDN7OC435E2K4BXI7XOZYKTB4YUNA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:14.829477074 (591h12m14.700560109s) [noise=true , address=185.167.204.205:4017 , piece_id=CZP65KETMGOHFDMV7KENRL4V36UHBPWRDCBBPUM5NE5L2PV4AQCQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:14.829774972 (591h12m14.700858007s) [noise=true , address=185.167.204.206:4043 , piece_id=G7VZYID4TXGNPDN63NQULXOVSX42WMFFNPBX6KQJ5L2WSW4D2YGQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:14.831734493 (591h12m14.702817528s) [noise=true , address=185.167.204.206:4049 , piece_id=BQ7JEGSGPIFJUGZFBYRPDMBNPZ24BG6TEG4VUM4JUBTE4T2RF4SA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:15.405688561 (591h12m15.276771606s) [noise=true , address=81.171.14.70:4027 , piece_id=6WEORWX5QOMFSJUEADIIBHHIPKNFP4NJPZAJADBUTPUSG7MZJMOQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:15.417110440 (591h12m15.288193485s) [noise=true , address=81.171.14.70:4031 , piece_id=U6Z4SNK32ZSWQ7NDR7ACXGRC4BADDOPR65ILBR64QLGVNDBADFFQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:15.425600446 (591h12m15.296683481s) [noise=true , address=81.171.14.70:4037 , piece_id=6A25LEIAQEJB3DNCTXQT6K7ZSKIGOW4QUGMNKLOG3CT2XXH25PSA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:15.499056228 (591h12m15.370139263s) [noise=true , address=81.171.14.70:4038 , piece_id=3WRHW3VGXWHOEOV5Q6CLYSF7MUVWS7QHTNHLSTCKAUERPHAH3I3Q, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:15.514998925 (591h12m15.38608197s) [noise=true , address=81.171.14.70:4041 , piece_id=NORNA6WFTGPNOHU7MPTNNB5ONTBS5DJGCN2NWIZAEPYSUSIOSGEQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:15.528579544 (591h12m15.399662589s) [noise=true , address=81.171.14.70:4030 , piece_id=KITOJIT34CHS2QZBXCW2SJPZB4RHEHCMJ6IYQ4Z5KLL24YAWBIUQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:15.528991398 (591h12m15.400074434s) [noise=true , address=81.171.14.70:4047 , piece_id=ZWV5DP4YC3MSYDWEQH4ZK3VR4WZ4DUTI4VL3WVLQKFIEKIW44LOQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:15.534844544 (591h12m15.405927579s) [noise=true , address=81.171.14.70:4035 , piece_id=XCLL5SOAPJSSGF5YYQH6ZKGOHMLS6RI6H26FOW5XIJRY3TWYNEGA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:15.915614910 (591h12m15.786697945s) [noise=true , address=81.171.14.70:4039 , piece_id=DEJDXFDHOSAMIP2FN5LJBJNBB4QLJSE2AUDW6IMA6IC4VRY7VAGQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:15.967820176 (591h12m15.838903211s) [noise=true , address=81.171.14.70:4046 , piece_id=B2MTUNKMUP5S7XW4SVQ5QI2ZO3VFB5QVAKBBY4BIZIUILM6O7VYQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:16.172116128 (591h12m16.043199163s) [noise=true , address=89.149.201.147:4007 , piece_id=HZLSYGSBP4TJH5V5XF3Q3AETJU625ERNDBD3VXW2HUZTTPAHLBMA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:16.172137559 (591h12m16.043220604s) [seg_pos={0 0}, upload=518292]: Segment reached optimal threshold of 26 pieces. Dec 30 06:41:16.172336949 (591h12m16.043419994s) [noise=true , address=89.149.201.147:4008 , piece_id=XYFRNJVIFFZA5S36KFPGZPVCPLRL2OW5QBJD3UR37VPPA2RX3RJA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QN8VP6o7qGw9Dd8EafU9xe7s7Cuf1FyqPMbnRaznrfPxfEUUn): protocol: expected piece hash; context canceled; EOF Dec 30 06:41:16.172344213 (591h12m16.043427258s) [noise=true , address=89.149.201.147:4008 , piece_id=XYFRNJVIFFZA5S36KFPGZPVCPLRL2OW5QBJD3UR37VPPA2RX3RJA, seg_pos={0 0}, upload=518292]: Long tail cancellation due to optimal threshold Dec 30 06:41:16.172554002 (591h12m16.043637037s) [noise=true , address=89.149.201.147:4020 , piece_id=N7G5DGEDYVKBTO5S6BYW3IYEOK7HAMK5JQIBMVPMPYRDHXODWEGA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YeKHJH7CoouLFJYENJhzVLNkA32hJaqjLz3dXhr78eaSGtcic): protocol: expected piece hash; context canceled; EOF Dec 30 06:41:16.172565233 (591h12m16.043648278s) [noise=true , address=89.149.201.147:4020 , piece_id=N7G5DGEDYVKBTO5S6BYW3IYEOK7HAMK5JQIBMVPMPYRDHXODWEGA, seg_pos={0 0}, upload=518292]: Long tail cancellation due to optimal threshold Dec 30 06:41:16.172587506 (591h12m16.043670541s) [seg_pos={0 0}, upload=518292]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 30 06:41:16.172717945 (591h12m16.04380098s) [upload=518292]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 06:41:16.172724567 (591h12m16.043807602s) [upload=518292]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 06:41:16.404092237 (591h12m16.275175292s) [upload=518292]: Done uploading segments... Dec 30 06:41:16.404119940 (591h12m16.275202975s) [upload=518292]: Upload finished. err: <nil> Duration: 6.327771492s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 20 out of 60 is failed (threshold=6) [...X.....X....XXX...XXXX.........X.X.....X..XXXXXX.X.....X..] duration: 3.432132803s trace: https://jaeger.dev.storj.io/trace/220b42f7e5b29694 FAIL (2025-12-30 07:23:57 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.432158864s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [..X....XX...X.....X....XXX..X.....X...X..X.XX..X...X....XXX.] duration: 5.66093556s trace: https://jaeger.dev.storj.io/trace/5930c8857cdaeba1 FAIL (2025-12-30 07:23:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: C5PXYN2XME2CS2N7LUEKJ76R7ZOJMNYMYXUMS42JCEKYZTDALSSQ) context deadline exceeded; piecestore: (Node ID: 12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9, Piece ID: ZQDEAEO35DTGL37JA2MYQLL3GWGJFPRD7PU4HTOSUXJOX23G4WAA) context deadline exceeded ======LOGS====== Dec 30 07:23:04.168104272 (591h54m4.039187317s) [upload=518894]: Starting upload Dec 30 07:23:04.168243527 (591h54m4.039326572s) [upload=518894]: Done starting upload Dec 30 07:23:04.168324552 (591h54m4.039407597s) [upload=518894]: Uploading segments... Dec 30 07:23:04.168580919 (591h54m4.039663965s) [upload=518894]: Got next segment. Inline: false Dec 30 07:23:04.168639501 (591h54m4.039722546s) [upload=518894]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 07:23:04.168653718 (591h54m4.039736763s) [upload=518894]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 07:23:04.411075136 (591h54m4.282158191s) [seg_pos={0 0}, upload=518894]: Begin upload segment... Dec 30 07:23:04.411153746 (591h54m4.282236791s) [seg_pos={0 0}, upload=518894]: Done begin upload segment. Dec 30 07:23:04.411194513 (591h54m4.282277558s) [noise=true , address=145.79.227.41:4087 , piece_id=PMBO6I23O5BPHZP6R3HRF6RLWVRGDJGJYLKALUQSRO2Y5Q5JBBRA, seg_pos={0 0}, upload=518894]: Uploading piece... Dec 30 07:23:04.411378775 (591h54m4.28246182s) [noise=true , address=145.79.227.48:4303 , piece_id=NJELZTCH2CL6PKTGXDIKNCQW2BG4TQTS7AVBRNZ3M4N536NE4IFA, seg_pos={0 0}, upload=518894]: Uploading piece... Dec 30 07:23:04.412986175 (591h54m4.28406922s) [noise=true , address=77.90.31.14:4664 , piece_id=ZQDEAEO35DTGL37JA2MYQLL3GWGJFPRD7PU4HTOSUXJOX23G4WAA, seg_pos={0 0}, upload=518894]: Uploading piece... Dec 30 07:23:04.413137623 (591h54m4.284220668s) [noise=true , address=145.79.227.14:4839 , piece_id=IE5O6S4ACP255IIMCSZJZPXCDEOUAM35QOJYTYPJFZ4LQER5M5PA, seg_pos={0 0}, upload=518894]: Uploading piece... Dec 30 07:23:04.413689645 (591h54m4.28477268s) [noise=true , address=145.79.227.19:4079 , piece_id=XDPDXVIUOM2E7RSFXUCISOSGG37URYAVNPU2SOHTGZ4FYEQWWRRQ, seg_pos={0 0}, upload=518894]: Uploading piece... Dec 30 07:23:04.414362846 (591h54m4.285445891s) [noise=true , address=145.79.227.39:4013 , piece_id=4YRNS665OHS57GBBC2IWVEMHF6IDGENB6YLMFIMY6Y4QUX3VFEZQ, seg_pos={0 0}, upload=518894]: Uploading piece... Dec 30 07:23:04.414733071 (591h54m4.285816127s) [noise=true , address=45.91.116.8:4047 , piece_id=C5PXYN2XME2CS2N7LUEKJ76R7ZOJMNYMYXUMS42JCEKYZTDALSSQ, seg_pos={0 0}, upload=518894]: Uploading piece... Dec 30 07:23:04.415482148 (591h54m4.286565193s) [noise=true , address=145.79.227.22:4510 , piece_id=STV6UFDLOEJBKC4Q4UMPXDJGZSCAHOREHSCZ7E4WBK2NY64PTYEQ, seg_pos={0 0}, upload=518894]: Uploading piece... Dec 30 07:23:04.415983963 (591h54m4.287067009s) [noise=true , address=45.91.116.13:4155 , piece_id=TD5IE6RMZKH6AZMKFLMY4OBGML22MTC33M5IEN66FWCPXMQFSPZQ, seg_pos={0 0}, upload=518894]: Uploading piece... Dec 30 07:23:04.415998290 (591h54m4.287081335s) [noise=true , address=145.79.227.46:4601 , piece_id=ZRW3L66YC4GLEEG6KNFSGN6ZU3AP3TICSLGNBCNOAEE7R3CGLSFA, seg_pos={0 0}, upload=518894]: Uploading piece... Dec 30 07:23:04.416524051 (591h54m4.287607086s) [noise=true , address=45.91.116.7:4010 , piece_id=RW5RJ4QTUD4VCCG42APNNMBOFXRDM3Y3FDVRUPKGIKVDLXFEICZQ, seg_pos={0 0}, upload=518894]: Uploading piece... Dec 30 07:23:04.417877038 (591h54m4.288960083s) [noise=true , address=145.79.227.42:4122 , piece_id=UU7XD5MPC2SOM52PDSVXVKPSEZNECEODZV3YAD7VZIRPED2V6MFQ, seg_pos={0 0}, upload=518894]: Uploading piece... Dec 30 07:23:04.417988591 (591h54m4.289071636s) [noise=true , address=145.79.227.12:4765 , piece_id=YMMZMFWOPNIMBVBZ3WD2PZ7YZA3PG5KNHWMF4JQNP4KFCYNI3QWQ, seg_pos={0 0}, upload=518894]: Uploading piece... Dec 30 07:23:04.418918321 (591h54m4.290001366s) [noise=true , address=145.79.227.36:4519 , piece_id=LGZU4JRPDPNLZAG5DFVJOU5Y4Z7EVZFNNSVRHTW232VIDCAQP2HQ, seg_pos={0 0}, upload=518894]: Uploading piece... Dec 30 07:23:04.419094546 (591h54m4.290177591s) [noise=true , address=45.91.116.12:4226 , piece_id=5AI7NMRMTF4DFJALI3FZLAW7S5KW4LBKGLSHZI5OPO35HTJF3YGA, seg_pos={0 0}, upload=518894]: Uploading piece... Dec 30 07:23:04.420782710 (591h54m4.291865755s) [noise=true , address=145.79.227.50:4374 , piece_id=XRDF3IFLXWOJM6MAHF2IB6PMG3CPK3ZI7CIQDKT3DAGDDP4B7URQ, seg_pos={0 0}, upload=518894]: Uploading piece... Dec 30 07:23:06.023176699 (591h54m5.894259745s) [upload=518894]: Next returned nil segment Dec 30 07:23:06.023195695 (591h54m5.89427874s) [upload=518894]: Waiting for error group managing segments... Dec 30 07:23:06.406060489 (591h54m6.277143524s) [noise=true , address=145.79.227.41:4087 , piece_id=PMBO6I23O5BPHZP6R3HRF6RLWVRGDJGJYLKALUQSRO2Y5Q5JBBRA, seg_pos={0 0}, upload=518894]: Done uploading piece. err: <nil> Dec 30 07:23:06.406878477 (591h54m6.277961522s) [noise=true , address=145.79.227.14:4839 , piece_id=IE5O6S4ACP255IIMCSZJZPXCDEOUAM35QOJYTYPJFZ4LQER5M5PA, seg_pos={0 0}, upload=518894]: Done uploading piece. err: <nil> Dec 30 07:23:06.411300568 (591h54m6.282383603s) [noise=true , address=145.79.227.48:4303 , piece_id=NJELZTCH2CL6PKTGXDIKNCQW2BG4TQTS7AVBRNZ3M4N536NE4IFA, seg_pos={0 0}, upload=518894]: Done uploading piece. err: <nil> Dec 30 07:23:06.412528676 (591h54m6.283611721s) [noise=true , address=145.79.227.42:4122 , piece_id=UU7XD5MPC2SOM52PDSVXVKPSEZNECEODZV3YAD7VZIRPED2V6MFQ, seg_pos={0 0}, upload=518894]: Done uploading piece. err: <nil> Dec 30 07:23:06.415731735 (591h54m6.28681478s) [noise=true , address=145.79.227.50:4374 , piece_id=XRDF3IFLXWOJM6MAHF2IB6PMG3CPK3ZI7CIQDKT3DAGDDP4B7URQ, seg_pos={0 0}, upload=518894]: Done uploading piece. err: <nil> Dec 30 07:23:06.424674564 (591h54m6.295757599s) [noise=true , address=145.79.227.12:4765 , piece_id=YMMZMFWOPNIMBVBZ3WD2PZ7YZA3PG5KNHWMF4JQNP4KFCYNI3QWQ, seg_pos={0 0}, upload=518894]: Done uploading piece. err: <nil> Dec 30 07:23:06.427645781 (591h54m6.298728816s) [noise=true , address=145.79.227.19:4079 , piece_id=XDPDXVIUOM2E7RSFXUCISOSGG37URYAVNPU2SOHTGZ4FYEQWWRRQ, seg_pos={0 0}, upload=518894]: Done uploading piece. err: <nil> Dec 30 07:23:06.431967981 (591h54m6.303051016s) [noise=true , address=145.79.227.46:4601 , piece_id=ZRW3L66YC4GLEEG6KNFSGN6ZU3AP3TICSLGNBCNOAEE7R3CGLSFA, seg_pos={0 0}, upload=518894]: Done uploading piece. err: <nil> Dec 30 07:23:06.432564157 (591h54m6.303647192s) [noise=true , address=145.79.227.39:4013 , piece_id=4YRNS665OHS57GBBC2IWVEMHF6IDGENB6YLMFIMY6Y4QUX3VFEZQ, seg_pos={0 0}, upload=518894]: Done uploading piece. err: <nil> Dec 30 07:23:06.436102254 (591h54m6.307185289s) [noise=true , address=145.79.227.36:4519 , piece_id=LGZU4JRPDPNLZAG5DFVJOU5Y4Z7EVZFNNSVRHTW232VIDCAQP2HQ, seg_pos={0 0}, upload=518894]: Done uploading piece. err: <nil> Dec 30 07:23:06.490019731 (591h54m6.361102766s) [noise=true , address=145.79.227.22:4510 , piece_id=STV6UFDLOEJBKC4Q4UMPXDJGZSCAHOREHSCZ7E4WBK2NY64PTYEQ, seg_pos={0 0}, upload=518894]: Done uploading piece. err: <nil> Dec 30 07:23:06.604939257 (591h54m6.476022292s) [noise=true , address=77.90.31.14:4664 , piece_id=ZQDEAEO35DTGL37JA2MYQLL3GWGJFPRD7PU4HTOSUXJOX23G4WAA, seg_pos={0 0}, upload=518894]: Done uploading piece. err: <nil> Dec 30 07:23:06.832441416 (591h54m6.703524461s) [noise=true , address=45.91.116.8:4047 , piece_id=C5PXYN2XME2CS2N7LUEKJ76R7ZOJMNYMYXUMS42JCEKYZTDALSSQ, seg_pos={0 0}, upload=518894]: Done uploading piece. err: <nil> Dec 30 07:23:06.832462966 (591h54m6.703546001s) [seg_pos={0 0}, upload=518894]: Segment reached optimal threshold of 13 pieces. Dec 30 07:23:06.832758960 (591h54m6.703842005s) [noise=true , address=45.91.116.12:4226 , piece_id=5AI7NMRMTF4DFJALI3FZLAW7S5KW4LBKGLSHZI5OPO35HTJF3YGA, seg_pos={0 0}, upload=518894]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 30 07:23:06.832769270 (591h54m6.703852315s) [noise=true , address=45.91.116.12:4226 , piece_id=5AI7NMRMTF4DFJALI3FZLAW7S5KW4LBKGLSHZI5OPO35HTJF3YGA, seg_pos={0 0}, upload=518894]: Long tail cancellation due to optimal threshold Dec 30 07:23:06.832844412 (591h54m6.703927458s) [noise=true , address=45.91.116.13:4155 , piece_id=TD5IE6RMZKH6AZMKFLMY4OBGML22MTC33M5IEN66FWCPXMQFSPZQ, seg_pos={0 0}, upload=518894]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 30 07:23:06.832861305 (591h54m6.70394434s) [noise=true , address=45.91.116.13:4155 , piece_id=TD5IE6RMZKH6AZMKFLMY4OBGML22MTC33M5IEN66FWCPXMQFSPZQ, seg_pos={0 0}, upload=518894]: Long tail cancellation due to optimal threshold Dec 30 07:23:06.833039615 (591h54m6.70412266s) [noise=true , address=45.91.116.7:4010 , piece_id=RW5RJ4QTUD4VCCG42APNNMBOFXRDM3Y3FDVRUPKGIKVDLXFEICZQ, seg_pos={0 0}, upload=518894]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: read tcp 188.166.225.170:37190->45.91.116.7:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 30 07:23:06.833047369 (591h54m6.704130404s) [noise=true , address=45.91.116.7:4010 , piece_id=RW5RJ4QTUD4VCCG42APNNMBOFXRDM3Y3FDVRUPKGIKVDLXFEICZQ, seg_pos={0 0}, upload=518894]: Long tail cancellation due to optimal threshold Dec 30 07:23:06.833068700 (591h54m6.704151745s) [seg_pos={0 0}, upload=518894]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 07:23:06.833152559 (591h54m6.704235604s) [upload=518894]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 07:23:06.833157018 (591h54m6.704240053s) [upload=518894]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 07:23:07.126513614 (591h54m6.997596659s) [upload=518894]: Done uploading segments... Dec 30 07:23:07.126531589 (591h54m6.997614624s) [upload=518894]: Upload finished. err: <nil> Duration: 5.660953514s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 37.98419152s trace: https://jaeger.dev.storj.io/trace/1d0a2f93cb06aaa4 FAIL (2025-12-30 07:23:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== loading piece... Dec 30 07:06:00.467281687 (591h37m0.338364722s) [noise=true , address=77.90.31.14:4693 , piece_id=CQ6A6DJJDCKQAAMCXW7FELWCWKAOI7YCCWKVDSTZEBXNHVKQRKPQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.467716555 (591h37m0.33879959s) [noise=true , address=145.79.227.40:4082 , piece_id=7UFAIMVNINXXXRPNXLGJ34YEO3ILVN2O7UFG3HHTIT2DKWVZSUPA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468044320 (591h37m0.339127365s) [noise=true , address=145.79.227.29:4955 , piece_id=CBV7RUAZ6QAZE3VDTZKOGTIMOY66VUJTSYQSZRC7WLUD53UWVJTQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468308362 (591h37m0.339391407s) [noise=true , address=145.79.227.17:4000 , piece_id=KN3NDKB4NODHNXUWWQQ4YBWBPJCNE7YHYEMLQ4UITRCFZB4BRSMA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468565703 (591h37m0.339648748s) [noise=true , address=145.79.227.46:4604 , piece_id=NY5A6JZW4F6E4CSX3ERMKSAL2R3IYSKQ2EVHZSPF6ZOHWQFXBYWA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468843542 (591h37m0.339926587s) [noise=true , address=77.90.31.17:4781 , piece_id=TFUDJVQZFLS2XTZEGBPTV23WYLNHZMGK5ZXNDJSMW5HSNZXWXBBA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.469105310 (591h37m0.340188345s) [noise=true , address=145.79.227.35:4494 , piece_id=U4P6QB4ZRFGAJOHVSSTZHCG4PTBGII4LXAAJXXVN5RVR4WUWUBBQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.470034750 (591h37m0.341117795s) [noise=true , address=145.79.227.49:4365 , piece_id=Z357V6BZEGQOFLFH32RQLYJJZDLURVHZAHKW3H3ZIJXMQC3ZIFBQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.470487622 (591h37m0.341570657s) [noise=true , address=64.49.13.169:4713 , piece_id=YS6FRVS5V3BECH3TJQRJBGB5SSYYGUBK2HMO25PZEUOUSC7ENNZQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.470809334 (591h37m0.341892379s) [noise=true , address=77.90.31.10:4530 , piece_id=LNRM5YW2MZLXHWHGNUMP5I7Y5RRNRESBUA2M3SQKDFEQXZ25RJ3A, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.471118874 (591h37m0.342201919s) [noise=true , address=145.79.227.44:4221 , piece_id=GDKGEN3QCGFIICNST42FA2ZLLTQCHL2QAZ7AMUNPJIGYTSHDMOJQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.471431078 (591h37m0.342514113s) [noise=true , address=45.91.116.4:4963 , piece_id=FUGWNAY33ZGP6J57FIODDQQJ2UH46ABT52AU4Y7Y74Q2AOTCXWFA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.471741530 (591h37m0.342824575s) [noise=true , address=193.149.164.143:4704 , piece_id=PQQZ6N4A7TZGGX5DVRE6NT6TSDJVQARR22F6MZ5Z6LYXLUKF4V7A, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.472073311 (591h37m0.343156356s) [noise=true , address=145.79.227.33:4901 , piece_id=FFTUVNIKVKRAZ5R44V6WLWNKKC5MQYA66RZPSMW6N5EQKY7NSGZQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:01.899399345 (591h37m1.77048239s) [upload=518641]: Next returned nil segment Dec 30 07:06:01.899418712 (591h37m1.770501747s) [upload=518641]: Waiting for error group managing segments... Dec 30 07:06:02.323303785 (591h37m2.194386821s) [noise=true , address=145.79.227.46:4604 , piece_id=NY5A6JZW4F6E4CSX3ERMKSAL2R3IYSKQ2EVHZSPF6ZOHWQFXBYWA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.339689637 (591h37m2.210772682s) [noise=true , address=145.79.227.17:4000 , piece_id=KN3NDKB4NODHNXUWWQQ4YBWBPJCNE7YHYEMLQ4UITRCFZB4BRSMA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.346460570 (591h37m2.217543615s) [noise=true , address=145.79.227.29:4955 , piece_id=CBV7RUAZ6QAZE3VDTZKOGTIMOY66VUJTSYQSZRC7WLUD53UWVJTQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.347028050 (591h37m2.218111095s) [noise=true , address=145.79.227.34:4931 , piece_id=TJ4NTHHJQ4VLPBIC7RW42DBFIBI5G7LJWBN53TPSEVAWHANHMTBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.427966770 (591h37m2.299049815s) [noise=true , address=145.79.227.5:4226 , piece_id=QKDFEH6UKIH4QYN6DE3YNH3FZPAYGNYMQSBCYGM7EAW7F4MDA66A, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.446440007 (591h37m2.317523052s) [noise=true , address=145.79.227.35:4494 , piece_id=U4P6QB4ZRFGAJOHVSSTZHCG4PTBGII4LXAAJXXVN5RVR4WUWUBBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.583477598 (591h37m2.454560643s) [noise=true , address=77.90.31.19:4859 , piece_id=XHNZMPC2MF4RALF2FZJY73UYN4XWOYJUL66RFRUXNPT64JZQE5KQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.613112418 (591h37m2.484195463s) [noise=true , address=77.90.31.17:4781 , piece_id=TFUDJVQZFLS2XTZEGBPTV23WYLNHZMGK5ZXNDJSMW5HSNZXWXBBA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.631421591 (591h37m2.502504626s) [noise=true , address=77.90.31.10:4530 , piece_id=LNRM5YW2MZLXHWHGNUMP5I7Y5RRNRESBUA2M3SQKDFEQXZ25RJ3A, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.944663583 (591h37m2.815746628s) [noise=true , address=145.79.227.49:4365 , piece_id=Z357V6BZEGQOFLFH32RQLYJJZDLURVHZAHKW3H3ZIJXMQC3ZIFBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.195837408 (591h37m3.066920453s) [noise=true , address=193.149.164.143:4704 , piece_id=PQQZ6N4A7TZGGX5DVRE6NT6TSDJVQARR22F6MZ5Z6LYXLUKF4V7A, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.269365478 (591h37m3.140448523s) [noise=true , address=145.79.227.44:4221 , piece_id=GDKGEN3QCGFIICNST42FA2ZLLTQCHL2QAZ7AMUNPJIGYTSHDMOJQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.272362485 (591h37m3.14344553s) [noise=true , address=145.79.227.30:4778 , piece_id=EVWGEZL3VCMHEM6ALESBLMU65XBE74CKVMFXMSHK2GCXFPI35RBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.340891504 (591h37m3.211974549s) [noise=true , address=145.79.227.40:4082 , piece_id=7UFAIMVNINXXXRPNXLGJ34YEO3ILVN2O7UFG3HHTIT2DKWVZSUPA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.340922954 (591h37m3.212005999s) [seg_pos={0 0}, upload=518641]: Segment reached optimal threshold of 14 pieces. Dec 30 07:06:03.341286095 (591h37m3.21236913s) [noise=true , address=77.90.31.14:4693 , piece_id=CQ6A6DJJDCKQAAMCXW7FELWCWKAOI7YCCWKVDSTZEBXNHVKQRKPQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RASrnvpAj4ToD4xNvbma1iKZEe2uHK8WosjfZwDDom8Qd17Ey): protocol: write tcp 188.166.225.170:47364->77.90.31.14:4693: use of closed network connection; write tcp 188.166.225.170:47364->77.90.31.14:4693: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47364->77.90.31.14:4693: use of closed network connection Dec 30 07:06:03.341317094 (591h37m3.212400139s) [noise=true , address=77.90.31.14:4693 , piece_id=CQ6A6DJJDCKQAAMCXW7FELWCWKAOI7YCCWKVDSTZEBXNHVKQRKPQ, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341322114 (591h37m3.212405159s) [noise=true , address=145.79.227.33:4901 , piece_id=FFTUVNIKVKRAZ5R44V6WLWNKKC5MQYA66RZPSMW6N5EQKY7NSGZQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1): protocol: expected piece hash; context canceled; EOF Dec 30 07:06:03.341336261 (591h37m3.212419306s) [noise=true , address=145.79.227.33:4901 , piece_id=FFTUVNIKVKRAZ5R44V6WLWNKKC5MQYA66RZPSMW6N5EQKY7NSGZQ, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341546141 (591h37m3.212629196s) [noise=true , address=45.91.116.4:4963 , piece_id=FUGWNAY33ZGP6J57FIODDQQJ2UH46ABT52AU4Y7Y74Q2AOTCXWFA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Eki7jYegwvkdixR5pznQBbSeJPn9GjGofxnUApidQF4kRNBK5): protocol: expected piece hash; write tcp 188.166.225.170:51728->45.91.116.4:4963: use of closed network connection; write tcp 188.166.225.170:51728->45.91.116.4:4963: use of closed network connection; EOF Dec 30 07:06:03.341555549 (591h37m3.212638594s) [noise=true , address=45.91.116.4:4963 , piece_id=FUGWNAY33ZGP6J57FIODDQQJ2UH46ABT52AU4Y7Y74Q2AOTCXWFA, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341759577 (591h37m3.212842622s) [noise=true , address=64.49.13.169:4713 , piece_id=YS6FRVS5V3BECH3TJQRJBGB5SSYYGUBK2HMO25PZEUOUSC7ENNZQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17z3AiUfbxBX2eMDnUnqo1bbgojuQxJro7PsEaPjYzQjHQ3D6K): protocol: write tcp 188.166.225.170:42002->64.49.13.169:4713: use of closed network connection; write tcp 188.166.225.170:42002->64.49.13.169:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42002->64.49.13.169:4713: use of closed network connection Dec 30 07:06:03.341769897 (591h37m3.212852942s) [noise=true , address=64.49.13.169:4713 , piece_id=YS6FRVS5V3BECH3TJQRJBGB5SSYYGUBK2HMO25PZEUOUSC7ENNZQ, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341869927 (591h37m3.212952972s) [seg_pos={0 0}, upload=518641]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 07:06:03.342018730 (591h37m3.213101775s) [upload=518641]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 07:06:03.342025634 (591h37m3.213108679s) [upload=518641]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 07:06:03.635315934 (591h37m3.506398979s) [upload=518641]: Done uploading segments... Dec 30 07:06:03.635343146 (591h37m3.506426191s) [upload=518641]: Upload finished. err: <nil> Duration: 37.984219315s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [...X....XX...X.....X....XXX..X.....X...X..X.XX..X...X....XXX] duration: 30.002798678s trace: https://jaeger.dev.storj.io/trace/5930c8857cdaeba1 FAIL (2025-12-30 07:23:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: C5PXYN2XME2CS2N7LUEKJ76R7ZOJMNYMYXUMS42JCEKYZTDALSSQ) context deadline exceeded; piecestore: (Node ID: 12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9, Piece ID: ZQDEAEO35DTGL37JA2MYQLL3GWGJFPRD7PU4HTOSUXJOX23G4WAA) context deadline exceeded ======LOGS====== Dec 30 07:23:04.168104272 (591h54m4.039187317s) [upload=518894]: Starting upload Dec 30 07:23:04.168243527 (591h54m4.039326572s) [upload=518894]: Done starting upload Dec 30 07:23:04.168324552 (591h54m4.039407597s) [upload=518894]: Uploading segments... Dec 30 07:23:04.168580919 (591h54m4.039663965s) [upload=518894]: Got next segment. Inline: false Dec 30 07:23:04.168639501 (591h54m4.039722546s) [upload=518894]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 07:23:04.168653718 (591h54m4.039736763s) [upload=518894]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 07:23:04.411075136 (591h54m4.282158191s) [seg_pos={0 0}, upload=518894]: Begin upload segment... Dec 30 07:23:04.411153746 (591h54m4.282236791s) [seg_pos={0 0}, upload=518894]: Done begin upload segment. Dec 30 07:23:04.411194513 (591h54m4.282277558s) [noise=true , address=145.79.227.41:4087 , piece_id=PMBO6I23O5BPHZP6R3HRF6RLWVRGDJGJYLKALUQSRO2Y5Q5JBBRA, seg_pos={0 0}, upload=518894]: Uploading piece... Dec 30 07:23:04.411378775 (591h54m4.28246182s) [noise=true , address=145.79.227.48:4303 , piece_id=NJELZTCH2CL6PKTGXDIKNCQW2BG4TQTS7AVBRNZ3M4N536NE4IFA, seg_pos={0 0}, upload=518894]: Uploading piece... Dec 30 07:23:04.412986175 (591h54m4.28406922s) [noise=true , address=77.90.31.14:4664 , piece_id=ZQDEAEO35DTGL37JA2MYQLL3GWGJFPRD7PU4HTOSUXJOX23G4WAA, seg_pos={0 0}, upload=518894]: Uploading piece... Dec 30 07:23:04.413137623 (591h54m4.284220668s) [noise=true , address=145.79.227.14:4839 , piece_id=IE5O6S4ACP255IIMCSZJZPXCDEOUAM35QOJYTYPJFZ4LQER5M5PA, seg_pos={0 0}, upload=518894]: Uploading piece... Dec 30 07:23:04.413689645 (591h54m4.28477268s) [noise=true , address=145.79.227.19:4079 , piece_id=XDPDXVIUOM2E7RSFXUCISOSGG37URYAVNPU2SOHTGZ4FYEQWWRRQ, seg_pos={0 0}, upload=518894]: Uploading piece... Dec 30 07:23:04.414362846 (591h54m4.285445891s) [noise=true , address=145.79.227.39:4013 , piece_id=4YRNS665OHS57GBBC2IWVEMHF6IDGENB6YLMFIMY6Y4QUX3VFEZQ, seg_pos={0 0}, upload=518894]: Uploading piece... Dec 30 07:23:04.414733071 (591h54m4.285816127s) [noise=true , address=45.91.116.8:4047 , piece_id=C5PXYN2XME2CS2N7LUEKJ76R7ZOJMNYMYXUMS42JCEKYZTDALSSQ, seg_pos={0 0}, upload=518894]: Uploading piece... Dec 30 07:23:04.415482148 (591h54m4.286565193s) [noise=true , address=145.79.227.22:4510 , piece_id=STV6UFDLOEJBKC4Q4UMPXDJGZSCAHOREHSCZ7E4WBK2NY64PTYEQ, seg_pos={0 0}, upload=518894]: Uploading piece... Dec 30 07:23:04.415983963 (591h54m4.287067009s) [noise=true , address=45.91.116.13:4155 , piece_id=TD5IE6RMZKH6AZMKFLMY4OBGML22MTC33M5IEN66FWCPXMQFSPZQ, seg_pos={0 0}, upload=518894]: Uploading piece... Dec 30 07:23:04.415998290 (591h54m4.287081335s) [noise=true , address=145.79.227.46:4601 , piece_id=ZRW3L66YC4GLEEG6KNFSGN6ZU3AP3TICSLGNBCNOAEE7R3CGLSFA, seg_pos={0 0}, upload=518894]: Uploading piece... Dec 30 07:23:04.416524051 (591h54m4.287607086s) [noise=true , address=45.91.116.7:4010 , piece_id=RW5RJ4QTUD4VCCG42APNNMBOFXRDM3Y3FDVRUPKGIKVDLXFEICZQ, seg_pos={0 0}, upload=518894]: Uploading piece... Dec 30 07:23:04.417877038 (591h54m4.288960083s) [noise=true , address=145.79.227.42:4122 , piece_id=UU7XD5MPC2SOM52PDSVXVKPSEZNECEODZV3YAD7VZIRPED2V6MFQ, seg_pos={0 0}, upload=518894]: Uploading piece... Dec 30 07:23:04.417988591 (591h54m4.289071636s) [noise=true , address=145.79.227.12:4765 , piece_id=YMMZMFWOPNIMBVBZ3WD2PZ7YZA3PG5KNHWMF4JQNP4KFCYNI3QWQ, seg_pos={0 0}, upload=518894]: Uploading piece... Dec 30 07:23:04.418918321 (591h54m4.290001366s) [noise=true , address=145.79.227.36:4519 , piece_id=LGZU4JRPDPNLZAG5DFVJOU5Y4Z7EVZFNNSVRHTW232VIDCAQP2HQ, seg_pos={0 0}, upload=518894]: Uploading piece... Dec 30 07:23:04.419094546 (591h54m4.290177591s) [noise=true , address=45.91.116.12:4226 , piece_id=5AI7NMRMTF4DFJALI3FZLAW7S5KW4LBKGLSHZI5OPO35HTJF3YGA, seg_pos={0 0}, upload=518894]: Uploading piece... Dec 30 07:23:04.420782710 (591h54m4.291865755s) [noise=true , address=145.79.227.50:4374 , piece_id=XRDF3IFLXWOJM6MAHF2IB6PMG3CPK3ZI7CIQDKT3DAGDDP4B7URQ, seg_pos={0 0}, upload=518894]: Uploading piece... Dec 30 07:23:06.023176699 (591h54m5.894259745s) [upload=518894]: Next returned nil segment Dec 30 07:23:06.023195695 (591h54m5.89427874s) [upload=518894]: Waiting for error group managing segments... Dec 30 07:23:06.406060489 (591h54m6.277143524s) [noise=true , address=145.79.227.41:4087 , piece_id=PMBO6I23O5BPHZP6R3HRF6RLWVRGDJGJYLKALUQSRO2Y5Q5JBBRA, seg_pos={0 0}, upload=518894]: Done uploading piece. err: <nil> Dec 30 07:23:06.406878477 (591h54m6.277961522s) [noise=true , address=145.79.227.14:4839 , piece_id=IE5O6S4ACP255IIMCSZJZPXCDEOUAM35QOJYTYPJFZ4LQER5M5PA, seg_pos={0 0}, upload=518894]: Done uploading piece. err: <nil> Dec 30 07:23:06.411300568 (591h54m6.282383603s) [noise=true , address=145.79.227.48:4303 , piece_id=NJELZTCH2CL6PKTGXDIKNCQW2BG4TQTS7AVBRNZ3M4N536NE4IFA, seg_pos={0 0}, upload=518894]: Done uploading piece. err: <nil> Dec 30 07:23:06.412528676 (591h54m6.283611721s) [noise=true , address=145.79.227.42:4122 , piece_id=UU7XD5MPC2SOM52PDSVXVKPSEZNECEODZV3YAD7VZIRPED2V6MFQ, seg_pos={0 0}, upload=518894]: Done uploading piece. err: <nil> Dec 30 07:23:06.415731735 (591h54m6.28681478s) [noise=true , address=145.79.227.50:4374 , piece_id=XRDF3IFLXWOJM6MAHF2IB6PMG3CPK3ZI7CIQDKT3DAGDDP4B7URQ, seg_pos={0 0}, upload=518894]: Done uploading piece. err: <nil> Dec 30 07:23:06.424674564 (591h54m6.295757599s) [noise=true , address=145.79.227.12:4765 , piece_id=YMMZMFWOPNIMBVBZ3WD2PZ7YZA3PG5KNHWMF4JQNP4KFCYNI3QWQ, seg_pos={0 0}, upload=518894]: Done uploading piece. err: <nil> Dec 30 07:23:06.427645781 (591h54m6.298728816s) [noise=true , address=145.79.227.19:4079 , piece_id=XDPDXVIUOM2E7RSFXUCISOSGG37URYAVNPU2SOHTGZ4FYEQWWRRQ, seg_pos={0 0}, upload=518894]: Done uploading piece. err: <nil> Dec 30 07:23:06.431967981 (591h54m6.303051016s) [noise=true , address=145.79.227.46:4601 , piece_id=ZRW3L66YC4GLEEG6KNFSGN6ZU3AP3TICSLGNBCNOAEE7R3CGLSFA, seg_pos={0 0}, upload=518894]: Done uploading piece. err: <nil> Dec 30 07:23:06.432564157 (591h54m6.303647192s) [noise=true , address=145.79.227.39:4013 , piece_id=4YRNS665OHS57GBBC2IWVEMHF6IDGENB6YLMFIMY6Y4QUX3VFEZQ, seg_pos={0 0}, upload=518894]: Done uploading piece. err: <nil> Dec 30 07:23:06.436102254 (591h54m6.307185289s) [noise=true , address=145.79.227.36:4519 , piece_id=LGZU4JRPDPNLZAG5DFVJOU5Y4Z7EVZFNNSVRHTW232VIDCAQP2HQ, seg_pos={0 0}, upload=518894]: Done uploading piece. err: <nil> Dec 30 07:23:06.490019731 (591h54m6.361102766s) [noise=true , address=145.79.227.22:4510 , piece_id=STV6UFDLOEJBKC4Q4UMPXDJGZSCAHOREHSCZ7E4WBK2NY64PTYEQ, seg_pos={0 0}, upload=518894]: Done uploading piece. err: <nil> Dec 30 07:23:06.604939257 (591h54m6.476022292s) [noise=true , address=77.90.31.14:4664 , piece_id=ZQDEAEO35DTGL37JA2MYQLL3GWGJFPRD7PU4HTOSUXJOX23G4WAA, seg_pos={0 0}, upload=518894]: Done uploading piece. err: <nil> Dec 30 07:23:06.832441416 (591h54m6.703524461s) [noise=true , address=45.91.116.8:4047 , piece_id=C5PXYN2XME2CS2N7LUEKJ76R7ZOJMNYMYXUMS42JCEKYZTDALSSQ, seg_pos={0 0}, upload=518894]: Done uploading piece. err: <nil> Dec 30 07:23:06.832462966 (591h54m6.703546001s) [seg_pos={0 0}, upload=518894]: Segment reached optimal threshold of 13 pieces. Dec 30 07:23:06.832758960 (591h54m6.703842005s) [noise=true , address=45.91.116.12:4226 , piece_id=5AI7NMRMTF4DFJALI3FZLAW7S5KW4LBKGLSHZI5OPO35HTJF3YGA, seg_pos={0 0}, upload=518894]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 30 07:23:06.832769270 (591h54m6.703852315s) [noise=true , address=45.91.116.12:4226 , piece_id=5AI7NMRMTF4DFJALI3FZLAW7S5KW4LBKGLSHZI5OPO35HTJF3YGA, seg_pos={0 0}, upload=518894]: Long tail cancellation due to optimal threshold Dec 30 07:23:06.832844412 (591h54m6.703927458s) [noise=true , address=45.91.116.13:4155 , piece_id=TD5IE6RMZKH6AZMKFLMY4OBGML22MTC33M5IEN66FWCPXMQFSPZQ, seg_pos={0 0}, upload=518894]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 30 07:23:06.832861305 (591h54m6.70394434s) [noise=true , address=45.91.116.13:4155 , piece_id=TD5IE6RMZKH6AZMKFLMY4OBGML22MTC33M5IEN66FWCPXMQFSPZQ, seg_pos={0 0}, upload=518894]: Long tail cancellation due to optimal threshold Dec 30 07:23:06.833039615 (591h54m6.70412266s) [noise=true , address=45.91.116.7:4010 , piece_id=RW5RJ4QTUD4VCCG42APNNMBOFXRDM3Y3FDVRUPKGIKVDLXFEICZQ, seg_pos={0 0}, upload=518894]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: read tcp 188.166.225.170:37190->45.91.116.7:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 30 07:23:06.833047369 (591h54m6.704130404s) [noise=true , address=45.91.116.7:4010 , piece_id=RW5RJ4QTUD4VCCG42APNNMBOFXRDM3Y3FDVRUPKGIKVDLXFEICZQ, seg_pos={0 0}, upload=518894]: Long tail cancellation due to optimal threshold Dec 30 07:23:06.833068700 (591h54m6.704151745s) [seg_pos={0 0}, upload=518894]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 07:23:06.833152559 (591h54m6.704235604s) [upload=518894]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 07:23:06.833157018 (591h54m6.704240053s) [upload=518894]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 07:23:07.126513614 (591h54m6.997596659s) [upload=518894]: Done uploading segments... Dec 30 07:23:07.126531589 (591h54m6.997614624s) [upload=518894]: Upload finished. err: <nil> Duration: 30.002811122s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 7.211326093s trace: https://jaeger.dev.storj.io/trace/4c2f4efaff0e0ea0 FAIL (2025-12-30 07:23:30 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== load=518292]: Uploading piece... Dec 30 06:41:13.598100152 (591h12m13.469183187s) [noise=true , address=81.171.14.70:4041 , piece_id=NORNA6WFTGPNOHU7MPTNNB5ONTBS5DJGCN2NWIZAEPYSUSIOSGEQ, seg_pos={0 0}, upload=518292]: Uploading piece... Dec 30 06:41:13.598587971 (591h12m13.469671016s) [noise=true , address=89.149.201.147:4007 , piece_id=HZLSYGSBP4TJH5V5XF3Q3AETJU625ERNDBD3VXW2HUZTTPAHLBMA, seg_pos={0 0}, upload=518292]: Uploading piece... Dec 30 06:41:13.599093143 (591h12m13.470176178s) [noise=true , address=89.149.201.147:4008 , piece_id=XYFRNJVIFFZA5S36KFPGZPVCPLRL2OW5QBJD3UR37VPPA2RX3RJA, seg_pos={0 0}, upload=518292]: Uploading piece... Dec 30 06:41:13.599699999 (591h12m13.470783034s) [noise=true , address=81.171.14.70:4039 , piece_id=DEJDXFDHOSAMIP2FN5LJBJNBB4QLJSE2AUDW6IMA6IC4VRY7VAGQ, seg_pos={0 0}, upload=518292]: Uploading piece... Dec 30 06:41:13.600136169 (591h12m13.471219204s) [noise=true , address=185.167.204.205:4034 , piece_id=OWZHQB5ZDT2QZ2QAHZJVRWQ222MX6CWLKJUOIAAGSMAVLWGJAOBA, seg_pos={0 0}, upload=518292]: Uploading piece... Dec 30 06:41:13.600543184 (591h12m13.471626229s) [noise=true , address=185.167.204.206:4058 , piece_id=ROOCGK64M5RHIWNAVCSZNYTZ6M2ORWB7MJ27XJVDTFVQIKJZHY4A, seg_pos={0 0}, upload=518292]: Uploading piece... Dec 30 06:41:13.601048286 (591h12m13.472131332s) [noise=true , address=81.171.14.70:4046 , piece_id=B2MTUNKMUP5S7XW4SVQ5QI2ZO3VFB5QVAKBBY4BIZIUILM6O7VYQ, seg_pos={0 0}, upload=518292]: Uploading piece... Dec 30 06:41:13.601698784 (591h12m13.472781819s) [noise=true , address=185.167.204.206:4064 , piece_id=H4PP7HPHXRX2LOZ4776AMWAJDN7OC435E2K4BXI7XOZYKTB4YUNA, seg_pos={0 0}, upload=518292]: Uploading piece... Dec 30 06:41:13.602168259 (591h12m13.473251304s) [noise=true , address=185.167.204.206:4065 , piece_id=LD3J4656Q5QEUCMJ5OKEYATEVRIKSXO7XGARTN25BVM7FDIYOWKA, seg_pos={0 0}, upload=518292]: Uploading piece... Dec 30 06:41:13.602782708 (591h12m13.473865753s) [noise=true , address=81.171.14.70:4027 , piece_id=6WEORWX5QOMFSJUEADIIBHHIPKNFP4NJPZAJADBUTPUSG7MZJMOQ, seg_pos={0 0}, upload=518292]: Uploading piece... Dec 30 06:41:14.423395002 (591h12m14.294478047s) [upload=518292]: Next returned nil segment Dec 30 06:41:14.423414068 (591h12m14.294497103s) [upload=518292]: Waiting for error group managing segments... Dec 30 06:41:14.817650316 (591h12m14.688733351s) [noise=true , address=185.167.204.205:4029 , piece_id=KUOMMYJ2F4NIK6VN4C5M3Y3TMHYEZH7QFNJ6K3CLOAVI5R6VKTHQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:14.817870054 (591h12m14.688953099s) [noise=true , address=185.167.204.205:4019 , piece_id=H6AH654VGX6NHBUKODIWGL7I2RKE3DDMY4UM6VDONP2EP3NPDJBA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:14.818623698 (591h12m14.689706744s) [noise=true , address=185.167.204.206:4065 , piece_id=LD3J4656Q5QEUCMJ5OKEYATEVRIKSXO7XGARTN25BVM7FDIYOWKA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:14.818932436 (591h12m14.690015481s) [noise=true , address=185.167.204.206:4061 , piece_id=T2XS4IFPQSVS3T45SPQ5ZDDVT2VMTMXHKL2ZMX4OIOPRR67SCD2Q, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:14.819147046 (591h12m14.690230091s) [noise=true , address=185.167.204.206:4071 , piece_id=QI2TOY5KCDZVWCCLMAJXVUWD2HB7S7IOTEAV2EHAMUNHKOFKTNYA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:14.819678578 (591h12m14.690761623s) [noise=true , address=185.167.204.206:4059 , piece_id=LMOGTWX2YIRR3EKNROK43O2B27S5L5U6GYB7HBVRBPQPH4WDDWZQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:14.819905028 (591h12m14.690988063s) [noise=true , address=185.167.204.206:4058 , piece_id=ROOCGK64M5RHIWNAVCSZNYTZ6M2ORWB7MJ27XJVDTFVQIKJZHY4A, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:14.820479312 (591h12m14.691562357s) [noise=true , address=185.167.204.205:4021 , piece_id=KP7J5XSY4SKSKZG7XXFZMNKMRP4472PXI7CW2UQBZICN4MAFO5RQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:14.823149296 (591h12m14.694232331s) [noise=true , address=185.167.204.205:4034 , piece_id=OWZHQB5ZDT2QZ2QAHZJVRWQ222MX6CWLKJUOIAAGSMAVLWGJAOBA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:14.827906996 (591h12m14.698990041s) [noise=true , address=185.167.204.206:4055 , piece_id=VYEDR6XHZBZMTVJ5U5OF67WZOGVMCMQLXXBQUI7DHBCCVQ2RRERQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:14.828003769 (591h12m14.699086814s) [noise=true , address=185.167.204.205:4016 , piece_id=I6MUDLAXUUDJL2C4HFE22C2L56A7D6HSCNWXKWJ7OKO2N4RA4W7Q, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:14.828893203 (591h12m14.699976238s) [noise=true , address=185.167.204.206:4064 , piece_id=H4PP7HPHXRX2LOZ4776AMWAJDN7OC435E2K4BXI7XOZYKTB4YUNA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:14.829477074 (591h12m14.700560109s) [noise=true , address=185.167.204.205:4017 , piece_id=CZP65KETMGOHFDMV7KENRL4V36UHBPWRDCBBPUM5NE5L2PV4AQCQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:14.829774972 (591h12m14.700858007s) [noise=true , address=185.167.204.206:4043 , piece_id=G7VZYID4TXGNPDN63NQULXOVSX42WMFFNPBX6KQJ5L2WSW4D2YGQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:14.831734493 (591h12m14.702817528s) [noise=true , address=185.167.204.206:4049 , piece_id=BQ7JEGSGPIFJUGZFBYRPDMBNPZ24BG6TEG4VUM4JUBTE4T2RF4SA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:15.405688561 (591h12m15.276771606s) [noise=true , address=81.171.14.70:4027 , piece_id=6WEORWX5QOMFSJUEADIIBHHIPKNFP4NJPZAJADBUTPUSG7MZJMOQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:15.417110440 (591h12m15.288193485s) [noise=true , address=81.171.14.70:4031 , piece_id=U6Z4SNK32ZSWQ7NDR7ACXGRC4BADDOPR65ILBR64QLGVNDBADFFQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:15.425600446 (591h12m15.296683481s) [noise=true , address=81.171.14.70:4037 , piece_id=6A25LEIAQEJB3DNCTXQT6K7ZSKIGOW4QUGMNKLOG3CT2XXH25PSA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:15.499056228 (591h12m15.370139263s) [noise=true , address=81.171.14.70:4038 , piece_id=3WRHW3VGXWHOEOV5Q6CLYSF7MUVWS7QHTNHLSTCKAUERPHAH3I3Q, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:15.514998925 (591h12m15.38608197s) [noise=true , address=81.171.14.70:4041 , piece_id=NORNA6WFTGPNOHU7MPTNNB5ONTBS5DJGCN2NWIZAEPYSUSIOSGEQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:15.528579544 (591h12m15.399662589s) [noise=true , address=81.171.14.70:4030 , piece_id=KITOJIT34CHS2QZBXCW2SJPZB4RHEHCMJ6IYQ4Z5KLL24YAWBIUQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:15.528991398 (591h12m15.400074434s) [noise=true , address=81.171.14.70:4047 , piece_id=ZWV5DP4YC3MSYDWEQH4ZK3VR4WZ4DUTI4VL3WVLQKFIEKIW44LOQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:15.534844544 (591h12m15.405927579s) [noise=true , address=81.171.14.70:4035 , piece_id=XCLL5SOAPJSSGF5YYQH6ZKGOHMLS6RI6H26FOW5XIJRY3TWYNEGA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:15.915614910 (591h12m15.786697945s) [noise=true , address=81.171.14.70:4039 , piece_id=DEJDXFDHOSAMIP2FN5LJBJNBB4QLJSE2AUDW6IMA6IC4VRY7VAGQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:15.967820176 (591h12m15.838903211s) [noise=true , address=81.171.14.70:4046 , piece_id=B2MTUNKMUP5S7XW4SVQ5QI2ZO3VFB5QVAKBBY4BIZIUILM6O7VYQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:16.172116128 (591h12m16.043199163s) [noise=true , address=89.149.201.147:4007 , piece_id=HZLSYGSBP4TJH5V5XF3Q3AETJU625ERNDBD3VXW2HUZTTPAHLBMA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:16.172137559 (591h12m16.043220604s) [seg_pos={0 0}, upload=518292]: Segment reached optimal threshold of 26 pieces. Dec 30 06:41:16.172336949 (591h12m16.043419994s) [noise=true , address=89.149.201.147:4008 , piece_id=XYFRNJVIFFZA5S36KFPGZPVCPLRL2OW5QBJD3UR37VPPA2RX3RJA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QN8VP6o7qGw9Dd8EafU9xe7s7Cuf1FyqPMbnRaznrfPxfEUUn): protocol: expected piece hash; context canceled; EOF Dec 30 06:41:16.172344213 (591h12m16.043427258s) [noise=true , address=89.149.201.147:4008 , piece_id=XYFRNJVIFFZA5S36KFPGZPVCPLRL2OW5QBJD3UR37VPPA2RX3RJA, seg_pos={0 0}, upload=518292]: Long tail cancellation due to optimal threshold Dec 30 06:41:16.172554002 (591h12m16.043637037s) [noise=true , address=89.149.201.147:4020 , piece_id=N7G5DGEDYVKBTO5S6BYW3IYEOK7HAMK5JQIBMVPMPYRDHXODWEGA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YeKHJH7CoouLFJYENJhzVLNkA32hJaqjLz3dXhr78eaSGtcic): protocol: expected piece hash; context canceled; EOF Dec 30 06:41:16.172565233 (591h12m16.043648278s) [noise=true , address=89.149.201.147:4020 , piece_id=N7G5DGEDYVKBTO5S6BYW3IYEOK7HAMK5JQIBMVPMPYRDHXODWEGA, seg_pos={0 0}, upload=518292]: Long tail cancellation due to optimal threshold Dec 30 06:41:16.172587506 (591h12m16.043670541s) [seg_pos={0 0}, upload=518292]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 30 06:41:16.172717945 (591h12m16.04380098s) [upload=518292]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 06:41:16.172724567 (591h12m16.043807602s) [upload=518292]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 06:41:16.404092237 (591h12m16.275175292s) [upload=518292]: Done uploading segments... Dec 30 06:41:16.404119940 (591h12m16.275202975s) [upload=518292]: Upload finished. err: <nil> Duration: 7.211353063s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 20 out of 60 is failed (threshold=6) [....X.....X....XXX...XXXX.........X.X.....X..XXXXXX.X.....X.] duration: 3.681528133s trace: https://jaeger.dev.storj.io/trace/220b42f7e5b29694 FAIL (2025-12-30 07:23:21 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.681547961s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [....X....XX...X.....X....XXX..X.....X...X..X.XX..X...X....XX] duration: 30.003301988s trace: https://jaeger.dev.storj.io/trace/1dc546dd79d5c2c FAIL (2025-12-30 07:23:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 0 07:22:33.911538580 (591h53m33.782621626s) [upload=518884]: Starting upload Dec 30 07:22:33.911595027 (591h53m33.782678062s) [upload=518884]: Done starting upload Dec 30 07:22:33.911643480 (591h53m33.782726515s) [upload=518884]: Uploading segments... Dec 30 07:22:33.911770732 (591h53m33.782853767s) [upload=518884]: Got next segment. Inline: false Dec 30 07:22:33.911802603 (591h53m33.782885638s) [upload=518884]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 07:22:33.911806891 (591h53m33.782889926s) [upload=518884]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 07:22:34.159166894 (591h53m34.030249929s) [seg_pos={0 0}, upload=518884]: Begin upload segment... Dec 30 07:22:34.159213574 (591h53m34.030296609s) [seg_pos={0 0}, upload=518884]: Done begin upload segment. Dec 30 07:22:34.159305147 (591h53m34.030388182s) [noise=true , address=145.79.227.38:4978 , piece_id=5RS5YJSBP7VIWRUR4JH346ZMGH2XO7JQK452A5ZECBVBGOFAPLQQ, seg_pos={0 0}, upload=518884]: Uploading piece... Dec 30 07:22:34.159321489 (591h53m34.030404534s) [noise=true , address=145.79.227.21:4475 , piece_id=KYUZA6Z6IUV2CJS2I7GY45MMMEHGSJ5N647DT7LGD4GKM3PTSDBQ, seg_pos={0 0}, upload=518884]: Uploading piece... Dec 30 07:22:34.159986776 (591h53m34.031069821s) [noise=true , address=45.91.116.9:4263 , piece_id=63FNSQKFPRUQLORNBFPNQBRWFQSUIDUQ3A5C26PSCV4JDWAW7MSA, seg_pos={0 0}, upload=518884]: Uploading piece... Dec 30 07:22:34.160302787 (591h53m34.031385832s) [noise=true , address=45.91.116.10:4081 , piece_id=FODM3DRWVIDH2CKAFMM43RG7IFLBWMWGNBEYUAOSYPDJ5KALAY5Q, seg_pos={0 0}, upload=518884]: Uploading piece... Dec 30 07:22:34.161115214 (591h53m34.032198249s) [noise=true , address=45.91.116.6:4973 , piece_id=DHFEMP7ZMUJ3JHN7TY2B6OQW4LBV7MEBDAZRJPBKJLOTSAC2GMGA, seg_pos={0 0}, upload=518884]: Uploading piece... Dec 30 07:22:34.162500542 (591h53m34.033583587s) [noise=true , address=77.90.31.13:4629 , piece_id=CST4E7R445UKYPUGX7FWJ6SKW2SX6QRSHR3YNYTAK2R6SKFZLUVQ, seg_pos={0 0}, upload=518884]: Uploading piece... Dec 30 07:22:34.162895704 (591h53m34.033978749s) [noise=true , address=77.90.31.17:4773 , piece_id=LRDHX5PI7PNG4NPLYAD7KA2KUCX4QOKX6T6W4NUGYZBQ7OGOKZPQ, seg_pos={0 0}, upload=518884]: Uploading piece... Dec 30 07:22:34.163197700 (591h53m34.034280745s) [noise=true , address=45.91.116.8:4047 , piece_id=S2UAMNBGWO5Y6EILT3O6JLJKFQSPIFSIL6B7KLHNW5XU7BUEEF2Q, seg_pos={0 0}, upload=518884]: Uploading piece... Dec 30 07:22:34.163500026 (591h53m34.034583061s) [noise=true , address=145.79.227.41:4086 , piece_id=FU77H6PGAIE2QETSTXUI5DZZ6BSDWIXIFBFM7GMUL4GQVUYPDW4Q, seg_pos={0 0}, upload=518884]: Uploading piece... Dec 30 07:22:34.163801840 (591h53m34.034884875s) [noise=true , address=45.91.116.7:4009 , piece_id=FJQIXPF2Z3WBBIRB2B6BTBBRR735MJE5UTYBO6FGFWCD4BWO6KXQ, seg_pos={0 0}, upload=518884]: Uploading piece... Dec 30 07:22:34.164085199 (591h53m34.035168244s) [noise=true , address=145.79.227.31:4799 , piece_id=G6NNCGJHGWT24YSOR57HSNHDEHNERAR4B6Q2IUZDRHZGJZNN34HA, seg_pos={0 0}, upload=518884]: Uploading piece... Dec 30 07:22:34.165479164 (591h53m34.036562209s) [noise=true , address=145.79.227.5:4222 , piece_id=75QT3DGLPARVRHDNFWYD7ZSERRN5Q35WEQQAPBNUP72SFWE2BLIQ, seg_pos={0 0}, upload=518884]: Uploading piece... Dec 30 07:22:34.167587007 (591h53m34.038670043s) [noise=true , address=145.79.227.26:4654 , piece_id=G63J7LQE3VW2PYKK6H3MEOSYUC3ZKOPMFNHYEWET3QWOT5PDNBWQ, seg_pos={0 0}, upload=518884]: Uploading piece... Dec 30 07:22:34.175834923 (591h53m34.046917958s) [noise=true , address=145.79.227.14:4840 , piece_id=TX2FIESK5EOJ72K56HBDX257GVLSXAEFAIWVD5LQI3HRVTCTJMTQ, seg_pos={0 0}, upload=518884]: Uploading piece... Dec 30 07:22:34.176125175 (591h53m34.04720821s) [noise=true , address=145.79.227.25:4619 , piece_id=GDHS7GC6VPOGGIARIPTLUV4GL6KZ6ELQINRLR43B3MOK7DR4YSQQ, seg_pos={0 0}, upload=518884]: Uploading piece... Dec 30 07:22:34.177393070 (591h53m34.048476105s) [noise=true , address=145.79.227.47:4267 , piece_id=GQYYEPL4INWN6KORITMXXWJNLFNE6YDE5FDL3Y7WTFGL2XQPGUZQ, seg_pos={0 0}, upload=518884]: Uploading piece... Dec 30 07:22:35.867827021 (591h53m35.738910056s) [upload=518884]: Next returned nil segment Dec 30 07:22:35.867849223 (591h53m35.738932258s) [upload=518884]: Waiting for error group managing segments... Dec 30 07:22:36.153162975 (591h53m36.02424601s) [noise=true , address=145.79.227.38:4978 , piece_id=5RS5YJSBP7VIWRUR4JH346ZMGH2XO7JQK452A5ZECBVBGOFAPLQQ, seg_pos={0 0}, upload=518884]: Done uploading piece. err: <nil> Dec 30 07:22:36.153862787 (591h53m36.024945822s) [noise=true , address=145.79.227.41:4086 , piece_id=FU77H6PGAIE2QETSTXUI5DZZ6BSDWIXIFBFM7GMUL4GQVUYPDW4Q, seg_pos={0 0}, upload=518884]: Done uploading piece. err: <nil> Dec 30 07:22:36.155676451 (591h53m36.026759486s) [noise=true , address=145.79.227.14:4840 , piece_id=TX2FIESK5EOJ72K56HBDX257GVLSXAEFAIWVD5LQI3HRVTCTJMTQ, seg_pos={0 0}, upload=518884]: Done uploading piece. err: <nil> Dec 30 07:22:36.167446423 (591h53m36.038529468s) [noise=true , address=145.79.227.21:4475 , piece_id=KYUZA6Z6IUV2CJS2I7GY45MMMEHGSJ5N647DT7LGD4GKM3PTSDBQ, seg_pos={0 0}, upload=518884]: Done uploading piece. err: <nil> Dec 30 07:22:36.171031710 (591h53m36.042114755s) [noise=true , address=145.79.227.31:4799 , piece_id=G6NNCGJHGWT24YSOR57HSNHDEHNERAR4B6Q2IUZDRHZGJZNN34HA, seg_pos={0 0}, upload=518884]: Done uploading piece. err: <nil> Dec 30 07:22:36.172519343 (591h53m36.043602378s) [noise=true , address=145.79.227.25:4619 , piece_id=GDHS7GC6VPOGGIARIPTLUV4GL6KZ6ELQINRLR43B3MOK7DR4YSQQ, seg_pos={0 0}, upload=518884]: Done uploading piece. err: <nil> Dec 30 07:22:36.175336387 (591h53m36.046419432s) [noise=true , address=145.79.227.26:4654 , piece_id=G63J7LQE3VW2PYKK6H3MEOSYUC3ZKOPMFNHYEWET3QWOT5PDNBWQ, seg_pos={0 0}, upload=518884]: Done uploading piece. err: <nil> Dec 30 07:22:36.190028733 (591h53m36.061111779s) [noise=true , address=145.79.227.47:4267 , piece_id=GQYYEPL4INWN6KORITMXXWJNLFNE6YDE5FDL3Y7WTFGL2XQPGUZQ, seg_pos={0 0}, upload=518884]: Done uploading piece. err: <nil> Dec 30 07:22:36.237763037 (591h53m36.108846072s) [noise=true , address=145.79.227.5:4222 , piece_id=75QT3DGLPARVRHDNFWYD7ZSERRN5Q35WEQQAPBNUP72SFWE2BLIQ, seg_pos={0 0}, upload=518884]: Done uploading piece. err: <nil> Dec 30 07:22:36.338632638 (591h53m36.209715673s) [noise=true , address=77.90.31.17:4773 , piece_id=LRDHX5PI7PNG4NPLYAD7KA2KUCX4QOKX6T6W4NUGYZBQ7OGOKZPQ, seg_pos={0 0}, upload=518884]: Done uploading piece. err: <nil> Dec 30 07:22:36.432368706 (591h53m36.303451751s) [noise=true , address=77.90.31.13:4629 , piece_id=CST4E7R445UKYPUGX7FWJ6SKW2SX6QRSHR3YNYTAK2R6SKFZLUVQ, seg_pos={0 0}, upload=518884]: Done uploading piece. err: <nil> Dec 30 07:22:36.465294162 (591h53m36.336377207s) [noise=true , address=45.91.116.6:4973 , piece_id=DHFEMP7ZMUJ3JHN7TY2B6OQW4LBV7MEBDAZRJPBKJLOTSAC2GMGA, seg_pos={0 0}, upload=518884]: Done uploading piece. err: <nil> Dec 30 07:22:36.529632512 (591h53m36.400715557s) [noise=true , address=45.91.116.10:4081 , piece_id=FODM3DRWVIDH2CKAFMM43RG7IFLBWMWGNBEYUAOSYPDJ5KALAY5Q, seg_pos={0 0}, upload=518884]: Done uploading piece. err: <nil> Dec 30 07:22:36.529657099 (591h53m36.400740144s) [seg_pos={0 0}, upload=518884]: Segment reached optimal threshold of 13 pieces. Dec 30 07:22:36.529967460 (591h53m36.401050505s) [noise=true , address=45.91.116.7:4009 , piece_id=FJQIXPF2Z3WBBIRB2B6BTBBRR735MJE5UTYBO6FGFWCD4BWO6KXQ, seg_pos={0 0}, upload=518884]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: write tcp 188.166.225.170:53472->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:53472->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53472->45.91.116.7:4009: use of closed network connection Dec 30 07:22:36.529977138 (591h53m36.401060173s) [noise=true , address=45.91.116.7:4009 , piece_id=FJQIXPF2Z3WBBIRB2B6BTBBRR735MJE5UTYBO6FGFWCD4BWO6KXQ, seg_pos={0 0}, upload=518884]: Long tail cancellation due to optimal threshold Dec 30 07:22:36.530108419 (591h53m36.401191464s) [noise=true , address=45.91.116.8:4047 , piece_id=S2UAMNBGWO5Y6EILT3O6JLJKFQSPIFSIL6B7KLHNW5XU7BUEEF2Q, seg_pos={0 0}, upload=518884]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 30 07:22:36.530150048 (591h53m36.401233093s) [noise=true , address=45.91.116.8:4047 , piece_id=S2UAMNBGWO5Y6EILT3O6JLJKFQSPIFSIL6B7KLHNW5XU7BUEEF2Q, seg_pos={0 0}, upload=518884]: Long tail cancellation due to optimal threshold Dec 30 07:22:36.530199071 (591h53m36.401282126s) [noise=true , address=45.91.116.9:4263 , piece_id=63FNSQKFPRUQLORNBFPNQBRWFQSUIDUQ3A5C26PSCV4JDWAW7MSA, seg_pos={0 0}, upload=518884]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 30 07:22:36.530208528 (591h53m36.401291573s) [noise=true , address=45.91.116.9:4263 , piece_id=63FNSQKFPRUQLORNBFPNQBRWFQSUIDUQ3A5C26PSCV4JDWAW7MSA, seg_pos={0 0}, upload=518884]: Long tail cancellation due to optimal threshold Dec 30 07:22:36.530288251 (591h53m36.401371286s) [seg_pos={0 0}, upload=518884]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 07:22:36.530438176 (591h53m36.401521211s) [upload=518884]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 07:22:36.530445109 (591h53m36.401528154s) [upload=518884]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 07:22:36.824681480 (591h53m36.695764525s) [upload=518884]: Done uploading segments... Dec 30 07:22:36.824709334 (591h53m36.695792379s) [upload=518884]: Upload finished. err: <nil> Duration: 30.003316224s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 36.670059823s trace: https://jaeger.dev.storj.io/trace/1d0a2f93cb06aaa4 FAIL (2025-12-30 07:22:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== loading piece... Dec 30 07:06:00.467281687 (591h37m0.338364722s) [noise=true , address=77.90.31.14:4693 , piece_id=CQ6A6DJJDCKQAAMCXW7FELWCWKAOI7YCCWKVDSTZEBXNHVKQRKPQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.467716555 (591h37m0.33879959s) [noise=true , address=145.79.227.40:4082 , piece_id=7UFAIMVNINXXXRPNXLGJ34YEO3ILVN2O7UFG3HHTIT2DKWVZSUPA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468044320 (591h37m0.339127365s) [noise=true , address=145.79.227.29:4955 , piece_id=CBV7RUAZ6QAZE3VDTZKOGTIMOY66VUJTSYQSZRC7WLUD53UWVJTQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468308362 (591h37m0.339391407s) [noise=true , address=145.79.227.17:4000 , piece_id=KN3NDKB4NODHNXUWWQQ4YBWBPJCNE7YHYEMLQ4UITRCFZB4BRSMA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468565703 (591h37m0.339648748s) [noise=true , address=145.79.227.46:4604 , piece_id=NY5A6JZW4F6E4CSX3ERMKSAL2R3IYSKQ2EVHZSPF6ZOHWQFXBYWA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468843542 (591h37m0.339926587s) [noise=true , address=77.90.31.17:4781 , piece_id=TFUDJVQZFLS2XTZEGBPTV23WYLNHZMGK5ZXNDJSMW5HSNZXWXBBA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.469105310 (591h37m0.340188345s) [noise=true , address=145.79.227.35:4494 , piece_id=U4P6QB4ZRFGAJOHVSSTZHCG4PTBGII4LXAAJXXVN5RVR4WUWUBBQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.470034750 (591h37m0.341117795s) [noise=true , address=145.79.227.49:4365 , piece_id=Z357V6BZEGQOFLFH32RQLYJJZDLURVHZAHKW3H3ZIJXMQC3ZIFBQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.470487622 (591h37m0.341570657s) [noise=true , address=64.49.13.169:4713 , piece_id=YS6FRVS5V3BECH3TJQRJBGB5SSYYGUBK2HMO25PZEUOUSC7ENNZQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.470809334 (591h37m0.341892379s) [noise=true , address=77.90.31.10:4530 , piece_id=LNRM5YW2MZLXHWHGNUMP5I7Y5RRNRESBUA2M3SQKDFEQXZ25RJ3A, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.471118874 (591h37m0.342201919s) [noise=true , address=145.79.227.44:4221 , piece_id=GDKGEN3QCGFIICNST42FA2ZLLTQCHL2QAZ7AMUNPJIGYTSHDMOJQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.471431078 (591h37m0.342514113s) [noise=true , address=45.91.116.4:4963 , piece_id=FUGWNAY33ZGP6J57FIODDQQJ2UH46ABT52AU4Y7Y74Q2AOTCXWFA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.471741530 (591h37m0.342824575s) [noise=true , address=193.149.164.143:4704 , piece_id=PQQZ6N4A7TZGGX5DVRE6NT6TSDJVQARR22F6MZ5Z6LYXLUKF4V7A, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.472073311 (591h37m0.343156356s) [noise=true , address=145.79.227.33:4901 , piece_id=FFTUVNIKVKRAZ5R44V6WLWNKKC5MQYA66RZPSMW6N5EQKY7NSGZQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:01.899399345 (591h37m1.77048239s) [upload=518641]: Next returned nil segment Dec 30 07:06:01.899418712 (591h37m1.770501747s) [upload=518641]: Waiting for error group managing segments... Dec 30 07:06:02.323303785 (591h37m2.194386821s) [noise=true , address=145.79.227.46:4604 , piece_id=NY5A6JZW4F6E4CSX3ERMKSAL2R3IYSKQ2EVHZSPF6ZOHWQFXBYWA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.339689637 (591h37m2.210772682s) [noise=true , address=145.79.227.17:4000 , piece_id=KN3NDKB4NODHNXUWWQQ4YBWBPJCNE7YHYEMLQ4UITRCFZB4BRSMA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.346460570 (591h37m2.217543615s) [noise=true , address=145.79.227.29:4955 , piece_id=CBV7RUAZ6QAZE3VDTZKOGTIMOY66VUJTSYQSZRC7WLUD53UWVJTQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.347028050 (591h37m2.218111095s) [noise=true , address=145.79.227.34:4931 , piece_id=TJ4NTHHJQ4VLPBIC7RW42DBFIBI5G7LJWBN53TPSEVAWHANHMTBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.427966770 (591h37m2.299049815s) [noise=true , address=145.79.227.5:4226 , piece_id=QKDFEH6UKIH4QYN6DE3YNH3FZPAYGNYMQSBCYGM7EAW7F4MDA66A, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.446440007 (591h37m2.317523052s) [noise=true , address=145.79.227.35:4494 , piece_id=U4P6QB4ZRFGAJOHVSSTZHCG4PTBGII4LXAAJXXVN5RVR4WUWUBBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.583477598 (591h37m2.454560643s) [noise=true , address=77.90.31.19:4859 , piece_id=XHNZMPC2MF4RALF2FZJY73UYN4XWOYJUL66RFRUXNPT64JZQE5KQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.613112418 (591h37m2.484195463s) [noise=true , address=77.90.31.17:4781 , piece_id=TFUDJVQZFLS2XTZEGBPTV23WYLNHZMGK5ZXNDJSMW5HSNZXWXBBA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.631421591 (591h37m2.502504626s) [noise=true , address=77.90.31.10:4530 , piece_id=LNRM5YW2MZLXHWHGNUMP5I7Y5RRNRESBUA2M3SQKDFEQXZ25RJ3A, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.944663583 (591h37m2.815746628s) [noise=true , address=145.79.227.49:4365 , piece_id=Z357V6BZEGQOFLFH32RQLYJJZDLURVHZAHKW3H3ZIJXMQC3ZIFBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.195837408 (591h37m3.066920453s) [noise=true , address=193.149.164.143:4704 , piece_id=PQQZ6N4A7TZGGX5DVRE6NT6TSDJVQARR22F6MZ5Z6LYXLUKF4V7A, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.269365478 (591h37m3.140448523s) [noise=true , address=145.79.227.44:4221 , piece_id=GDKGEN3QCGFIICNST42FA2ZLLTQCHL2QAZ7AMUNPJIGYTSHDMOJQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.272362485 (591h37m3.14344553s) [noise=true , address=145.79.227.30:4778 , piece_id=EVWGEZL3VCMHEM6ALESBLMU65XBE74CKVMFXMSHK2GCXFPI35RBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.340891504 (591h37m3.211974549s) [noise=true , address=145.79.227.40:4082 , piece_id=7UFAIMVNINXXXRPNXLGJ34YEO3ILVN2O7UFG3HHTIT2DKWVZSUPA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.340922954 (591h37m3.212005999s) [seg_pos={0 0}, upload=518641]: Segment reached optimal threshold of 14 pieces. Dec 30 07:06:03.341286095 (591h37m3.21236913s) [noise=true , address=77.90.31.14:4693 , piece_id=CQ6A6DJJDCKQAAMCXW7FELWCWKAOI7YCCWKVDSTZEBXNHVKQRKPQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RASrnvpAj4ToD4xNvbma1iKZEe2uHK8WosjfZwDDom8Qd17Ey): protocol: write tcp 188.166.225.170:47364->77.90.31.14:4693: use of closed network connection; write tcp 188.166.225.170:47364->77.90.31.14:4693: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47364->77.90.31.14:4693: use of closed network connection Dec 30 07:06:03.341317094 (591h37m3.212400139s) [noise=true , address=77.90.31.14:4693 , piece_id=CQ6A6DJJDCKQAAMCXW7FELWCWKAOI7YCCWKVDSTZEBXNHVKQRKPQ, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341322114 (591h37m3.212405159s) [noise=true , address=145.79.227.33:4901 , piece_id=FFTUVNIKVKRAZ5R44V6WLWNKKC5MQYA66RZPSMW6N5EQKY7NSGZQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1): protocol: expected piece hash; context canceled; EOF Dec 30 07:06:03.341336261 (591h37m3.212419306s) [noise=true , address=145.79.227.33:4901 , piece_id=FFTUVNIKVKRAZ5R44V6WLWNKKC5MQYA66RZPSMW6N5EQKY7NSGZQ, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341546141 (591h37m3.212629196s) [noise=true , address=45.91.116.4:4963 , piece_id=FUGWNAY33ZGP6J57FIODDQQJ2UH46ABT52AU4Y7Y74Q2AOTCXWFA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Eki7jYegwvkdixR5pznQBbSeJPn9GjGofxnUApidQF4kRNBK5): protocol: expected piece hash; write tcp 188.166.225.170:51728->45.91.116.4:4963: use of closed network connection; write tcp 188.166.225.170:51728->45.91.116.4:4963: use of closed network connection; EOF Dec 30 07:06:03.341555549 (591h37m3.212638594s) [noise=true , address=45.91.116.4:4963 , piece_id=FUGWNAY33ZGP6J57FIODDQQJ2UH46ABT52AU4Y7Y74Q2AOTCXWFA, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341759577 (591h37m3.212842622s) [noise=true , address=64.49.13.169:4713 , piece_id=YS6FRVS5V3BECH3TJQRJBGB5SSYYGUBK2HMO25PZEUOUSC7ENNZQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17z3AiUfbxBX2eMDnUnqo1bbgojuQxJro7PsEaPjYzQjHQ3D6K): protocol: write tcp 188.166.225.170:42002->64.49.13.169:4713: use of closed network connection; write tcp 188.166.225.170:42002->64.49.13.169:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42002->64.49.13.169:4713: use of closed network connection Dec 30 07:06:03.341769897 (591h37m3.212852942s) [noise=true , address=64.49.13.169:4713 , piece_id=YS6FRVS5V3BECH3TJQRJBGB5SSYYGUBK2HMO25PZEUOUSC7ENNZQ, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341869927 (591h37m3.212952972s) [seg_pos={0 0}, upload=518641]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 07:06:03.342018730 (591h37m3.213101775s) [upload=518641]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 07:06:03.342025634 (591h37m3.213108679s) [upload=518641]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 07:06:03.635315934 (591h37m3.506398979s) [upload=518641]: Done uploading segments... Dec 30 07:06:03.635343146 (591h37m3.506426191s) [upload=518641]: Upload finished. err: <nil> Duration: 36.670090462s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 6.68605694s trace: https://jaeger.dev.storj.io/trace/4c2f4efaff0e0ea0 FAIL (2025-12-30 07:22:26 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== load=518292]: Uploading piece... Dec 30 06:41:13.598100152 (591h12m13.469183187s) [noise=true , address=81.171.14.70:4041 , piece_id=NORNA6WFTGPNOHU7MPTNNB5ONTBS5DJGCN2NWIZAEPYSUSIOSGEQ, seg_pos={0 0}, upload=518292]: Uploading piece... Dec 30 06:41:13.598587971 (591h12m13.469671016s) [noise=true , address=89.149.201.147:4007 , piece_id=HZLSYGSBP4TJH5V5XF3Q3AETJU625ERNDBD3VXW2HUZTTPAHLBMA, seg_pos={0 0}, upload=518292]: Uploading piece... Dec 30 06:41:13.599093143 (591h12m13.470176178s) [noise=true , address=89.149.201.147:4008 , piece_id=XYFRNJVIFFZA5S36KFPGZPVCPLRL2OW5QBJD3UR37VPPA2RX3RJA, seg_pos={0 0}, upload=518292]: Uploading piece... Dec 30 06:41:13.599699999 (591h12m13.470783034s) [noise=true , address=81.171.14.70:4039 , piece_id=DEJDXFDHOSAMIP2FN5LJBJNBB4QLJSE2AUDW6IMA6IC4VRY7VAGQ, seg_pos={0 0}, upload=518292]: Uploading piece... Dec 30 06:41:13.600136169 (591h12m13.471219204s) [noise=true , address=185.167.204.205:4034 , piece_id=OWZHQB5ZDT2QZ2QAHZJVRWQ222MX6CWLKJUOIAAGSMAVLWGJAOBA, seg_pos={0 0}, upload=518292]: Uploading piece... Dec 30 06:41:13.600543184 (591h12m13.471626229s) [noise=true , address=185.167.204.206:4058 , piece_id=ROOCGK64M5RHIWNAVCSZNYTZ6M2ORWB7MJ27XJVDTFVQIKJZHY4A, seg_pos={0 0}, upload=518292]: Uploading piece... Dec 30 06:41:13.601048286 (591h12m13.472131332s) [noise=true , address=81.171.14.70:4046 , piece_id=B2MTUNKMUP5S7XW4SVQ5QI2ZO3VFB5QVAKBBY4BIZIUILM6O7VYQ, seg_pos={0 0}, upload=518292]: Uploading piece... Dec 30 06:41:13.601698784 (591h12m13.472781819s) [noise=true , address=185.167.204.206:4064 , piece_id=H4PP7HPHXRX2LOZ4776AMWAJDN7OC435E2K4BXI7XOZYKTB4YUNA, seg_pos={0 0}, upload=518292]: Uploading piece... Dec 30 06:41:13.602168259 (591h12m13.473251304s) [noise=true , address=185.167.204.206:4065 , piece_id=LD3J4656Q5QEUCMJ5OKEYATEVRIKSXO7XGARTN25BVM7FDIYOWKA, seg_pos={0 0}, upload=518292]: Uploading piece... Dec 30 06:41:13.602782708 (591h12m13.473865753s) [noise=true , address=81.171.14.70:4027 , piece_id=6WEORWX5QOMFSJUEADIIBHHIPKNFP4NJPZAJADBUTPUSG7MZJMOQ, seg_pos={0 0}, upload=518292]: Uploading piece... Dec 30 06:41:14.423395002 (591h12m14.294478047s) [upload=518292]: Next returned nil segment Dec 30 06:41:14.423414068 (591h12m14.294497103s) [upload=518292]: Waiting for error group managing segments... Dec 30 06:41:14.817650316 (591h12m14.688733351s) [noise=true , address=185.167.204.205:4029 , piece_id=KUOMMYJ2F4NIK6VN4C5M3Y3TMHYEZH7QFNJ6K3CLOAVI5R6VKTHQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:14.817870054 (591h12m14.688953099s) [noise=true , address=185.167.204.205:4019 , piece_id=H6AH654VGX6NHBUKODIWGL7I2RKE3DDMY4UM6VDONP2EP3NPDJBA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:14.818623698 (591h12m14.689706744s) [noise=true , address=185.167.204.206:4065 , piece_id=LD3J4656Q5QEUCMJ5OKEYATEVRIKSXO7XGARTN25BVM7FDIYOWKA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:14.818932436 (591h12m14.690015481s) [noise=true , address=185.167.204.206:4061 , piece_id=T2XS4IFPQSVS3T45SPQ5ZDDVT2VMTMXHKL2ZMX4OIOPRR67SCD2Q, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:14.819147046 (591h12m14.690230091s) [noise=true , address=185.167.204.206:4071 , piece_id=QI2TOY5KCDZVWCCLMAJXVUWD2HB7S7IOTEAV2EHAMUNHKOFKTNYA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:14.819678578 (591h12m14.690761623s) [noise=true , address=185.167.204.206:4059 , piece_id=LMOGTWX2YIRR3EKNROK43O2B27S5L5U6GYB7HBVRBPQPH4WDDWZQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:14.819905028 (591h12m14.690988063s) [noise=true , address=185.167.204.206:4058 , piece_id=ROOCGK64M5RHIWNAVCSZNYTZ6M2ORWB7MJ27XJVDTFVQIKJZHY4A, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:14.820479312 (591h12m14.691562357s) [noise=true , address=185.167.204.205:4021 , piece_id=KP7J5XSY4SKSKZG7XXFZMNKMRP4472PXI7CW2UQBZICN4MAFO5RQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:14.823149296 (591h12m14.694232331s) [noise=true , address=185.167.204.205:4034 , piece_id=OWZHQB5ZDT2QZ2QAHZJVRWQ222MX6CWLKJUOIAAGSMAVLWGJAOBA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:14.827906996 (591h12m14.698990041s) [noise=true , address=185.167.204.206:4055 , piece_id=VYEDR6XHZBZMTVJ5U5OF67WZOGVMCMQLXXBQUI7DHBCCVQ2RRERQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:14.828003769 (591h12m14.699086814s) [noise=true , address=185.167.204.205:4016 , piece_id=I6MUDLAXUUDJL2C4HFE22C2L56A7D6HSCNWXKWJ7OKO2N4RA4W7Q, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:14.828893203 (591h12m14.699976238s) [noise=true , address=185.167.204.206:4064 , piece_id=H4PP7HPHXRX2LOZ4776AMWAJDN7OC435E2K4BXI7XOZYKTB4YUNA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:14.829477074 (591h12m14.700560109s) [noise=true , address=185.167.204.205:4017 , piece_id=CZP65KETMGOHFDMV7KENRL4V36UHBPWRDCBBPUM5NE5L2PV4AQCQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:14.829774972 (591h12m14.700858007s) [noise=true , address=185.167.204.206:4043 , piece_id=G7VZYID4TXGNPDN63NQULXOVSX42WMFFNPBX6KQJ5L2WSW4D2YGQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:14.831734493 (591h12m14.702817528s) [noise=true , address=185.167.204.206:4049 , piece_id=BQ7JEGSGPIFJUGZFBYRPDMBNPZ24BG6TEG4VUM4JUBTE4T2RF4SA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:15.405688561 (591h12m15.276771606s) [noise=true , address=81.171.14.70:4027 , piece_id=6WEORWX5QOMFSJUEADIIBHHIPKNFP4NJPZAJADBUTPUSG7MZJMOQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:15.417110440 (591h12m15.288193485s) [noise=true , address=81.171.14.70:4031 , piece_id=U6Z4SNK32ZSWQ7NDR7ACXGRC4BADDOPR65ILBR64QLGVNDBADFFQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:15.425600446 (591h12m15.296683481s) [noise=true , address=81.171.14.70:4037 , piece_id=6A25LEIAQEJB3DNCTXQT6K7ZSKIGOW4QUGMNKLOG3CT2XXH25PSA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:15.499056228 (591h12m15.370139263s) [noise=true , address=81.171.14.70:4038 , piece_id=3WRHW3VGXWHOEOV5Q6CLYSF7MUVWS7QHTNHLSTCKAUERPHAH3I3Q, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:15.514998925 (591h12m15.38608197s) [noise=true , address=81.171.14.70:4041 , piece_id=NORNA6WFTGPNOHU7MPTNNB5ONTBS5DJGCN2NWIZAEPYSUSIOSGEQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:15.528579544 (591h12m15.399662589s) [noise=true , address=81.171.14.70:4030 , piece_id=KITOJIT34CHS2QZBXCW2SJPZB4RHEHCMJ6IYQ4Z5KLL24YAWBIUQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:15.528991398 (591h12m15.400074434s) [noise=true , address=81.171.14.70:4047 , piece_id=ZWV5DP4YC3MSYDWEQH4ZK3VR4WZ4DUTI4VL3WVLQKFIEKIW44LOQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:15.534844544 (591h12m15.405927579s) [noise=true , address=81.171.14.70:4035 , piece_id=XCLL5SOAPJSSGF5YYQH6ZKGOHMLS6RI6H26FOW5XIJRY3TWYNEGA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:15.915614910 (591h12m15.786697945s) [noise=true , address=81.171.14.70:4039 , piece_id=DEJDXFDHOSAMIP2FN5LJBJNBB4QLJSE2AUDW6IMA6IC4VRY7VAGQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:15.967820176 (591h12m15.838903211s) [noise=true , address=81.171.14.70:4046 , piece_id=B2MTUNKMUP5S7XW4SVQ5QI2ZO3VFB5QVAKBBY4BIZIUILM6O7VYQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:16.172116128 (591h12m16.043199163s) [noise=true , address=89.149.201.147:4007 , piece_id=HZLSYGSBP4TJH5V5XF3Q3AETJU625ERNDBD3VXW2HUZTTPAHLBMA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:16.172137559 (591h12m16.043220604s) [seg_pos={0 0}, upload=518292]: Segment reached optimal threshold of 26 pieces. Dec 30 06:41:16.172336949 (591h12m16.043419994s) [noise=true , address=89.149.201.147:4008 , piece_id=XYFRNJVIFFZA5S36KFPGZPVCPLRL2OW5QBJD3UR37VPPA2RX3RJA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QN8VP6o7qGw9Dd8EafU9xe7s7Cuf1FyqPMbnRaznrfPxfEUUn): protocol: expected piece hash; context canceled; EOF Dec 30 06:41:16.172344213 (591h12m16.043427258s) [noise=true , address=89.149.201.147:4008 , piece_id=XYFRNJVIFFZA5S36KFPGZPVCPLRL2OW5QBJD3UR37VPPA2RX3RJA, seg_pos={0 0}, upload=518292]: Long tail cancellation due to optimal threshold Dec 30 06:41:16.172554002 (591h12m16.043637037s) [noise=true , address=89.149.201.147:4020 , piece_id=N7G5DGEDYVKBTO5S6BYW3IYEOK7HAMK5JQIBMVPMPYRDHXODWEGA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YeKHJH7CoouLFJYENJhzVLNkA32hJaqjLz3dXhr78eaSGtcic): protocol: expected piece hash; context canceled; EOF Dec 30 06:41:16.172565233 (591h12m16.043648278s) [noise=true , address=89.149.201.147:4020 , piece_id=N7G5DGEDYVKBTO5S6BYW3IYEOK7HAMK5JQIBMVPMPYRDHXODWEGA, seg_pos={0 0}, upload=518292]: Long tail cancellation due to optimal threshold Dec 30 06:41:16.172587506 (591h12m16.043670541s) [seg_pos={0 0}, upload=518292]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 30 06:41:16.172717945 (591h12m16.04380098s) [upload=518292]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 06:41:16.172724567 (591h12m16.043807602s) [upload=518292]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 06:41:16.404092237 (591h12m16.275175292s) [upload=518292]: Done uploading segments... Dec 30 06:41:16.404119940 (591h12m16.275202975s) [upload=518292]: Upload finished. err: <nil> Duration: 6.686077189s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 15.577696246s trace: https://jaeger.dev.storj.io/trace/1d0a2f93cb06aaa4 FAIL (2025-12-30 07:22:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== loading piece... Dec 30 07:06:00.467281687 (591h37m0.338364722s) [noise=true , address=77.90.31.14:4693 , piece_id=CQ6A6DJJDCKQAAMCXW7FELWCWKAOI7YCCWKVDSTZEBXNHVKQRKPQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.467716555 (591h37m0.33879959s) [noise=true , address=145.79.227.40:4082 , piece_id=7UFAIMVNINXXXRPNXLGJ34YEO3ILVN2O7UFG3HHTIT2DKWVZSUPA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468044320 (591h37m0.339127365s) [noise=true , address=145.79.227.29:4955 , piece_id=CBV7RUAZ6QAZE3VDTZKOGTIMOY66VUJTSYQSZRC7WLUD53UWVJTQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468308362 (591h37m0.339391407s) [noise=true , address=145.79.227.17:4000 , piece_id=KN3NDKB4NODHNXUWWQQ4YBWBPJCNE7YHYEMLQ4UITRCFZB4BRSMA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468565703 (591h37m0.339648748s) [noise=true , address=145.79.227.46:4604 , piece_id=NY5A6JZW4F6E4CSX3ERMKSAL2R3IYSKQ2EVHZSPF6ZOHWQFXBYWA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468843542 (591h37m0.339926587s) [noise=true , address=77.90.31.17:4781 , piece_id=TFUDJVQZFLS2XTZEGBPTV23WYLNHZMGK5ZXNDJSMW5HSNZXWXBBA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.469105310 (591h37m0.340188345s) [noise=true , address=145.79.227.35:4494 , piece_id=U4P6QB4ZRFGAJOHVSSTZHCG4PTBGII4LXAAJXXVN5RVR4WUWUBBQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.470034750 (591h37m0.341117795s) [noise=true , address=145.79.227.49:4365 , piece_id=Z357V6BZEGQOFLFH32RQLYJJZDLURVHZAHKW3H3ZIJXMQC3ZIFBQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.470487622 (591h37m0.341570657s) [noise=true , address=64.49.13.169:4713 , piece_id=YS6FRVS5V3BECH3TJQRJBGB5SSYYGUBK2HMO25PZEUOUSC7ENNZQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.470809334 (591h37m0.341892379s) [noise=true , address=77.90.31.10:4530 , piece_id=LNRM5YW2MZLXHWHGNUMP5I7Y5RRNRESBUA2M3SQKDFEQXZ25RJ3A, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.471118874 (591h37m0.342201919s) [noise=true , address=145.79.227.44:4221 , piece_id=GDKGEN3QCGFIICNST42FA2ZLLTQCHL2QAZ7AMUNPJIGYTSHDMOJQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.471431078 (591h37m0.342514113s) [noise=true , address=45.91.116.4:4963 , piece_id=FUGWNAY33ZGP6J57FIODDQQJ2UH46ABT52AU4Y7Y74Q2AOTCXWFA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.471741530 (591h37m0.342824575s) [noise=true , address=193.149.164.143:4704 , piece_id=PQQZ6N4A7TZGGX5DVRE6NT6TSDJVQARR22F6MZ5Z6LYXLUKF4V7A, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.472073311 (591h37m0.343156356s) [noise=true , address=145.79.227.33:4901 , piece_id=FFTUVNIKVKRAZ5R44V6WLWNKKC5MQYA66RZPSMW6N5EQKY7NSGZQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:01.899399345 (591h37m1.77048239s) [upload=518641]: Next returned nil segment Dec 30 07:06:01.899418712 (591h37m1.770501747s) [upload=518641]: Waiting for error group managing segments... Dec 30 07:06:02.323303785 (591h37m2.194386821s) [noise=true , address=145.79.227.46:4604 , piece_id=NY5A6JZW4F6E4CSX3ERMKSAL2R3IYSKQ2EVHZSPF6ZOHWQFXBYWA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.339689637 (591h37m2.210772682s) [noise=true , address=145.79.227.17:4000 , piece_id=KN3NDKB4NODHNXUWWQQ4YBWBPJCNE7YHYEMLQ4UITRCFZB4BRSMA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.346460570 (591h37m2.217543615s) [noise=true , address=145.79.227.29:4955 , piece_id=CBV7RUAZ6QAZE3VDTZKOGTIMOY66VUJTSYQSZRC7WLUD53UWVJTQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.347028050 (591h37m2.218111095s) [noise=true , address=145.79.227.34:4931 , piece_id=TJ4NTHHJQ4VLPBIC7RW42DBFIBI5G7LJWBN53TPSEVAWHANHMTBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.427966770 (591h37m2.299049815s) [noise=true , address=145.79.227.5:4226 , piece_id=QKDFEH6UKIH4QYN6DE3YNH3FZPAYGNYMQSBCYGM7EAW7F4MDA66A, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.446440007 (591h37m2.317523052s) [noise=true , address=145.79.227.35:4494 , piece_id=U4P6QB4ZRFGAJOHVSSTZHCG4PTBGII4LXAAJXXVN5RVR4WUWUBBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.583477598 (591h37m2.454560643s) [noise=true , address=77.90.31.19:4859 , piece_id=XHNZMPC2MF4RALF2FZJY73UYN4XWOYJUL66RFRUXNPT64JZQE5KQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.613112418 (591h37m2.484195463s) [noise=true , address=77.90.31.17:4781 , piece_id=TFUDJVQZFLS2XTZEGBPTV23WYLNHZMGK5ZXNDJSMW5HSNZXWXBBA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.631421591 (591h37m2.502504626s) [noise=true , address=77.90.31.10:4530 , piece_id=LNRM5YW2MZLXHWHGNUMP5I7Y5RRNRESBUA2M3SQKDFEQXZ25RJ3A, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.944663583 (591h37m2.815746628s) [noise=true , address=145.79.227.49:4365 , piece_id=Z357V6BZEGQOFLFH32RQLYJJZDLURVHZAHKW3H3ZIJXMQC3ZIFBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.195837408 (591h37m3.066920453s) [noise=true , address=193.149.164.143:4704 , piece_id=PQQZ6N4A7TZGGX5DVRE6NT6TSDJVQARR22F6MZ5Z6LYXLUKF4V7A, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.269365478 (591h37m3.140448523s) [noise=true , address=145.79.227.44:4221 , piece_id=GDKGEN3QCGFIICNST42FA2ZLLTQCHL2QAZ7AMUNPJIGYTSHDMOJQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.272362485 (591h37m3.14344553s) [noise=true , address=145.79.227.30:4778 , piece_id=EVWGEZL3VCMHEM6ALESBLMU65XBE74CKVMFXMSHK2GCXFPI35RBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.340891504 (591h37m3.211974549s) [noise=true , address=145.79.227.40:4082 , piece_id=7UFAIMVNINXXXRPNXLGJ34YEO3ILVN2O7UFG3HHTIT2DKWVZSUPA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.340922954 (591h37m3.212005999s) [seg_pos={0 0}, upload=518641]: Segment reached optimal threshold of 14 pieces. Dec 30 07:06:03.341286095 (591h37m3.21236913s) [noise=true , address=77.90.31.14:4693 , piece_id=CQ6A6DJJDCKQAAMCXW7FELWCWKAOI7YCCWKVDSTZEBXNHVKQRKPQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RASrnvpAj4ToD4xNvbma1iKZEe2uHK8WosjfZwDDom8Qd17Ey): protocol: write tcp 188.166.225.170:47364->77.90.31.14:4693: use of closed network connection; write tcp 188.166.225.170:47364->77.90.31.14:4693: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47364->77.90.31.14:4693: use of closed network connection Dec 30 07:06:03.341317094 (591h37m3.212400139s) [noise=true , address=77.90.31.14:4693 , piece_id=CQ6A6DJJDCKQAAMCXW7FELWCWKAOI7YCCWKVDSTZEBXNHVKQRKPQ, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341322114 (591h37m3.212405159s) [noise=true , address=145.79.227.33:4901 , piece_id=FFTUVNIKVKRAZ5R44V6WLWNKKC5MQYA66RZPSMW6N5EQKY7NSGZQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1): protocol: expected piece hash; context canceled; EOF Dec 30 07:06:03.341336261 (591h37m3.212419306s) [noise=true , address=145.79.227.33:4901 , piece_id=FFTUVNIKVKRAZ5R44V6WLWNKKC5MQYA66RZPSMW6N5EQKY7NSGZQ, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341546141 (591h37m3.212629196s) [noise=true , address=45.91.116.4:4963 , piece_id=FUGWNAY33ZGP6J57FIODDQQJ2UH46ABT52AU4Y7Y74Q2AOTCXWFA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Eki7jYegwvkdixR5pznQBbSeJPn9GjGofxnUApidQF4kRNBK5): protocol: expected piece hash; write tcp 188.166.225.170:51728->45.91.116.4:4963: use of closed network connection; write tcp 188.166.225.170:51728->45.91.116.4:4963: use of closed network connection; EOF Dec 30 07:06:03.341555549 (591h37m3.212638594s) [noise=true , address=45.91.116.4:4963 , piece_id=FUGWNAY33ZGP6J57FIODDQQJ2UH46ABT52AU4Y7Y74Q2AOTCXWFA, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341759577 (591h37m3.212842622s) [noise=true , address=64.49.13.169:4713 , piece_id=YS6FRVS5V3BECH3TJQRJBGB5SSYYGUBK2HMO25PZEUOUSC7ENNZQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17z3AiUfbxBX2eMDnUnqo1bbgojuQxJro7PsEaPjYzQjHQ3D6K): protocol: write tcp 188.166.225.170:42002->64.49.13.169:4713: use of closed network connection; write tcp 188.166.225.170:42002->64.49.13.169:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42002->64.49.13.169:4713: use of closed network connection Dec 30 07:06:03.341769897 (591h37m3.212852942s) [noise=true , address=64.49.13.169:4713 , piece_id=YS6FRVS5V3BECH3TJQRJBGB5SSYYGUBK2HMO25PZEUOUSC7ENNZQ, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341869927 (591h37m3.212952972s) [seg_pos={0 0}, upload=518641]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 07:06:03.342018730 (591h37m3.213101775s) [upload=518641]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 07:06:03.342025634 (591h37m3.213108679s) [upload=518641]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 07:06:03.635315934 (591h37m3.506398979s) [upload=518641]: Done uploading segments... Dec 30 07:06:03.635343146 (591h37m3.506426191s) [upload=518641]: Upload finished. err: <nil> Duration: 15.577727725s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 20 out of 60 is failed (threshold=6) [.....X.....X....XXX...XXXX.........X.X.....X..XXXXXX.X.....X] duration: 5.001589328s trace: https://jaeger.dev.storj.io/trace/220b42f7e5b29694 FAIL (2025-12-30 07:22:12 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001617512s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [.....X....XX...X.....X....XXX..X.....X...X..X.XX..X...X....X] duration: 30.00493874s trace: https://jaeger.dev.storj.io/trace/72955df8f415476c FAIL (2025-12-30 07:22:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, Piece ID: NUCCFGKVNDB2HWYMZXE6PIAM472VHZLKBIQBUJBBTDALEFVCPCGQ) context deadline exceeded; piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: G2R45NA2RTLJB5WSFFUWWJ6MHTEXT2V7PMYCHJAIEUJJIOQ3NTWA) context deadline exceeded ======LOGS====== Dec 30 07:21:33.874476950 (591h52m33.745559996s) [upload=518872]: Starting upload Dec 30 07:21:33.875096450 (591h52m33.746179495s) [upload=518872]: Done starting upload Dec 30 07:21:33.875616360 (591h52m33.746699405s) [upload=518872]: Uploading segments... Dec 30 07:21:33.879700117 (591h52m33.750783162s) [upload=518872]: Got next segment. Inline: false Dec 30 07:21:33.879822069 (591h52m33.750905114s) [upload=518872]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 07:21:33.879829663 (591h52m33.750912708s) [upload=518872]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 07:21:34.127019803 (591h52m33.998102838s) [seg_pos={0 0}, upload=518872]: Begin upload segment... Dec 30 07:21:34.127085268 (591h52m33.998168313s) [seg_pos={0 0}, upload=518872]: Done begin upload segment. Dec 30 07:21:34.127141053 (591h52m33.998224098s) [noise=true , address=77.90.31.14:4662 , piece_id=NUCCFGKVNDB2HWYMZXE6PIAM472VHZLKBIQBUJBBTDALEFVCPCGQ, seg_pos={0 0}, upload=518872]: Uploading piece... Dec 30 07:21:34.127311137 (591h52m33.998394182s) [noise=true , address=77.90.31.15:4700 , piece_id=MPDDKCPATP2NCAMJKDR6FVTUZAPAWV6CIAULMP3KV7VMSBAJEU5Q, seg_pos={0 0}, upload=518872]: Uploading piece... Dec 30 07:21:34.128037671 (591h52m33.999120706s) [noise=true , address=145.79.227.22:4510 , piece_id=TIJOTLIXLKJO77QKMIIPREJKY7GGRVD3PJCNW7Z6443ERVJYB7CA, seg_pos={0 0}, upload=518872]: Uploading piece... Dec 30 07:21:34.128365064 (591h52m33.999448109s) [noise=true , address=145.79.227.10:4401 , piece_id=3YF36JMMSGXC7ZMX3OZ6XLL43KFCG4ATKWMSN76SQXGNZZGEHUUA, seg_pos={0 0}, upload=518872]: Uploading piece... Dec 30 07:21:34.128446339 (591h52m33.999529384s) [noise=true , address=45.91.116.9:4261 , piece_id=G2R45NA2RTLJB5WSFFUWWJ6MHTEXT2V7PMYCHJAIEUJJIOQ3NTWA, seg_pos={0 0}, upload=518872]: Uploading piece... Dec 30 07:21:34.128744276 (591h52m33.999827311s) [noise=true , address=145.79.227.6:4258 , piece_id=4FUSCFRNA57FINLYX53ISNOGUA5UB3ZMIQET72JC5OPG42KM4UFQ, seg_pos={0 0}, upload=518872]: Uploading piece... Dec 30 07:21:34.128852532 (591h52m33.999935567s) [noise=true , address=145.79.227.45:4231 , piece_id=U3PPN6RJFLYHERKLUF4NFJ36YLPYUKLPPI6WYV7OXB2SJGZRYGRQ, seg_pos={0 0}, upload=518872]: Uploading piece... Dec 30 07:21:34.129087069 (591h52m34.000170104s) [noise=true , address=145.79.227.8:4329 , piece_id=QC326E6PNBFFVYFIUPPZBQGMKEA7NWKX3Q7FDCKJV4PFYM2YYWQQ, seg_pos={0 0}, upload=518872]: Uploading piece... Dec 30 07:21:34.129247485 (591h52m34.00033052s) [noise=true , address=145.79.227.43:4159 , piece_id=I5HTLH7RN6NJY77GMAVG5A6XX7J4VQKRFNQV3RSZIGDHVCQHD67Q, seg_pos={0 0}, upload=518872]: Uploading piece... Dec 30 07:21:34.129412799 (591h52m34.000495834s) [noise=true , address=145.79.227.23:4547 , piece_id=QCPV35EIWFON3SSOOD3K4JRH7GSURKNYTVRDQU2RI7OPDH3BL4XQ, seg_pos={0 0}, upload=518872]: Uploading piece... Dec 30 07:21:34.129651313 (591h52m34.000734358s) [noise=true , address=145.79.227.17:4970 , piece_id=NNOSIPAXFPUTZPSPJ45WCBQ4EP2XLAKSRKXEDW5PFIDDTVGM3ZRA, seg_pos={0 0}, upload=518872]: Uploading piece... Dec 30 07:21:34.129740764 (591h52m34.000823799s) [noise=true , address=77.90.31.13:4627 , piece_id=6H3TX2623HQWCCONEAPCKYSIJHYHOYUZXHE2S3DOGBKRNLQZFTXA, seg_pos={0 0}, upload=518872]: Uploading piece... Dec 30 07:21:34.130124214 (591h52m34.001207259s) [noise=true , address=145.79.227.47:4267 , piece_id=JOJA4EEQVTY2MTGEUW6KD6CNTS3QMDKGNOFRQ44PPNGU5AS7TGMA, seg_pos={0 0}, upload=518872]: Uploading piece... Dec 30 07:21:34.130626500 (591h52m34.001709535s) [noise=true , address=77.90.31.16:4735 , piece_id=G3JKC4RYDHIZG7RM7LTPG6QAPPAVF4JA37IVNIKY2MSNZF3CVJVA, seg_pos={0 0}, upload=518872]: Uploading piece... Dec 30 07:21:34.131139798 (591h52m34.002222833s) [noise=true , address=145.79.227.18:4007 , piece_id=OOBR664H7YWQVJ246ALBC2YK2H2PCH6RUJKJ76O4A2OC73BEMJNQ, seg_pos={0 0}, upload=518872]: Uploading piece... Dec 30 07:21:34.131659818 (591h52m34.002742863s) [noise=true , address=145.79.227.39:4015 , piece_id=NS2WKQ7RHNL65FCHVXYIXAM7U4WYL5K7KHPAKCMDF3SUPWVBG66Q, seg_pos={0 0}, upload=518872]: Uploading piece... Dec 30 07:21:35.736016414 (591h52m35.607099459s) [upload=518872]: Next returned nil segment Dec 30 07:21:35.736031594 (591h52m35.607114639s) [upload=518872]: Waiting for error group managing segments... Dec 30 07:21:36.113551931 (591h52m35.984634967s) [noise=true , address=145.79.227.45:4231 , piece_id=U3PPN6RJFLYHERKLUF4NFJ36YLPYUKLPPI6WYV7OXB2SJGZRYGRQ, seg_pos={0 0}, upload=518872]: Done uploading piece. err: <nil> Dec 30 07:21:36.117929136 (591h52m35.989012171s) [noise=true , address=145.79.227.22:4510 , piece_id=TIJOTLIXLKJO77QKMIIPREJKY7GGRVD3PJCNW7Z6443ERVJYB7CA, seg_pos={0 0}, upload=518872]: Done uploading piece. err: <nil> Dec 30 07:21:36.122281483 (591h52m35.993364528s) [noise=true , address=145.79.227.10:4401 , piece_id=3YF36JMMSGXC7ZMX3OZ6XLL43KFCG4ATKWMSN76SQXGNZZGEHUUA, seg_pos={0 0}, upload=518872]: Done uploading piece. err: <nil> Dec 30 07:21:36.132492026 (591h52m36.003575061s) [noise=true , address=145.79.227.39:4015 , piece_id=NS2WKQ7RHNL65FCHVXYIXAM7U4WYL5K7KHPAKCMDF3SUPWVBG66Q, seg_pos={0 0}, upload=518872]: Done uploading piece. err: <nil> Dec 30 07:21:36.136547528 (591h52m36.007630574s) [noise=true , address=145.79.227.18:4007 , piece_id=OOBR664H7YWQVJ246ALBC2YK2H2PCH6RUJKJ76O4A2OC73BEMJNQ, seg_pos={0 0}, upload=518872]: Done uploading piece. err: <nil> Dec 30 07:21:36.140708571 (591h52m36.011791606s) [noise=true , address=145.79.227.17:4970 , piece_id=NNOSIPAXFPUTZPSPJ45WCBQ4EP2XLAKSRKXEDW5PFIDDTVGM3ZRA, seg_pos={0 0}, upload=518872]: Done uploading piece. err: <nil> Dec 30 07:21:36.148490330 (591h52m36.019573365s) [noise=true , address=145.79.227.6:4258 , piece_id=4FUSCFRNA57FINLYX53ISNOGUA5UB3ZMIQET72JC5OPG42KM4UFQ, seg_pos={0 0}, upload=518872]: Done uploading piece. err: <nil> Dec 30 07:21:36.149041148 (591h52m36.020124183s) [noise=true , address=145.79.227.47:4267 , piece_id=JOJA4EEQVTY2MTGEUW6KD6CNTS3QMDKGNOFRQ44PPNGU5AS7TGMA, seg_pos={0 0}, upload=518872]: Done uploading piece. err: <nil> Dec 30 07:21:36.203162774 (591h52m36.074245809s) [noise=true , address=145.79.227.8:4329 , piece_id=QC326E6PNBFFVYFIUPPZBQGMKEA7NWKX3Q7FDCKJV4PFYM2YYWQQ, seg_pos={0 0}, upload=518872]: Done uploading piece. err: <nil> Dec 30 07:21:36.203926228 (591h52m36.075009273s) [noise=true , address=145.79.227.23:4547 , piece_id=QCPV35EIWFON3SSOOD3K4JRH7GSURKNYTVRDQU2RI7OPDH3BL4XQ, seg_pos={0 0}, upload=518872]: Done uploading piece. err: <nil> Dec 30 07:21:36.389368959 (591h52m36.260452004s) [noise=true , address=77.90.31.15:4700 , piece_id=MPDDKCPATP2NCAMJKDR6FVTUZAPAWV6CIAULMP3KV7VMSBAJEU5Q, seg_pos={0 0}, upload=518872]: Done uploading piece. err: <nil> Dec 30 07:21:36.427782425 (591h52m36.29886546s) [noise=true , address=77.90.31.14:4662 , piece_id=NUCCFGKVNDB2HWYMZXE6PIAM472VHZLKBIQBUJBBTDALEFVCPCGQ, seg_pos={0 0}, upload=518872]: Done uploading piece. err: <nil> Dec 30 07:21:36.439840445 (591h52m36.31092349s) [noise=true , address=45.91.116.9:4261 , piece_id=G2R45NA2RTLJB5WSFFUWWJ6MHTEXT2V7PMYCHJAIEUJJIOQ3NTWA, seg_pos={0 0}, upload=518872]: Done uploading piece. err: <nil> Dec 30 07:21:36.439862728 (591h52m36.310945773s) [seg_pos={0 0}, upload=518872]: Segment reached optimal threshold of 13 pieces. Dec 30 07:21:36.440033533 (591h52m36.311116568s) [noise=true , address=77.90.31.13:4627 , piece_id=6H3TX2623HQWCCONEAPCKYSIJHYHOYUZXHE2S3DOGBKRNLQZFTXA, seg_pos={0 0}, upload=518872]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: expected piece hash; context canceled; EOF Dec 30 07:21:36.440040506 (591h52m36.311123541s) [noise=true , address=77.90.31.13:4627 , piece_id=6H3TX2623HQWCCONEAPCKYSIJHYHOYUZXHE2S3DOGBKRNLQZFTXA, seg_pos={0 0}, upload=518872]: Long tail cancellation due to optimal threshold Dec 30 07:21:36.440146017 (591h52m36.311229052s) [noise=true , address=145.79.227.43:4159 , piece_id=I5HTLH7RN6NJY77GMAVG5A6XX7J4VQKRFNQV3RSZIGDHVCQHD67Q, seg_pos={0 0}, upload=518872]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA): protocol: expected piece hash; context canceled; EOF Dec 30 07:21:36.440153070 (591h52m36.311236115s) [noise=true , address=145.79.227.43:4159 , piece_id=I5HTLH7RN6NJY77GMAVG5A6XX7J4VQKRFNQV3RSZIGDHVCQHD67Q, seg_pos={0 0}, upload=518872]: Long tail cancellation due to optimal threshold Dec 30 07:21:36.440239165 (591h52m36.31132221s) [noise=true , address=77.90.31.16:4735 , piece_id=G3JKC4RYDHIZG7RM7LTPG6QAPPAVF4JA37IVNIKY2MSNZF3CVJVA, seg_pos={0 0}, upload=518872]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: read tcp 188.166.225.170:38128->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 30 07:21:36.440250466 (591h52m36.311333511s) [noise=true , address=77.90.31.16:4735 , piece_id=G3JKC4RYDHIZG7RM7LTPG6QAPPAVF4JA37IVNIKY2MSNZF3CVJVA, seg_pos={0 0}, upload=518872]: Long tail cancellation due to optimal threshold Dec 30 07:21:36.440295793 (591h52m36.311378828s) [seg_pos={0 0}, upload=518872]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 07:21:36.440362569 (591h52m36.311445604s) [upload=518872]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 07:21:36.440367820 (591h52m36.311450855s) [upload=518872]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 07:21:36.725441043 (591h52m36.596524078s) [upload=518872]: Done uploading segments... Dec 30 07:21:36.725461252 (591h52m36.596544297s) [upload=518872]: Upload finished. err: <nil> Duration: 30.004966532s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 20 out of 60 is failed (threshold=6) [X.....X.....X....XXX...XXXX.........X.X.....X..XXXXXX.X.....] duration: 3.595461381s trace: https://jaeger.dev.storj.io/trace/14547861cf082dcb FAIL (2025-12-30 07:21:36 UTC) orbiter warning: context deadline exceeded Duration: 3.595490497s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 29.792254798s trace: https://jaeger.dev.storj.io/trace/1d0a2f93cb06aaa4 FAIL (2025-12-30 07:21:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== loading piece... Dec 30 07:06:00.467281687 (591h37m0.338364722s) [noise=true , address=77.90.31.14:4693 , piece_id=CQ6A6DJJDCKQAAMCXW7FELWCWKAOI7YCCWKVDSTZEBXNHVKQRKPQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.467716555 (591h37m0.33879959s) [noise=true , address=145.79.227.40:4082 , piece_id=7UFAIMVNINXXXRPNXLGJ34YEO3ILVN2O7UFG3HHTIT2DKWVZSUPA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468044320 (591h37m0.339127365s) [noise=true , address=145.79.227.29:4955 , piece_id=CBV7RUAZ6QAZE3VDTZKOGTIMOY66VUJTSYQSZRC7WLUD53UWVJTQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468308362 (591h37m0.339391407s) [noise=true , address=145.79.227.17:4000 , piece_id=KN3NDKB4NODHNXUWWQQ4YBWBPJCNE7YHYEMLQ4UITRCFZB4BRSMA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468565703 (591h37m0.339648748s) [noise=true , address=145.79.227.46:4604 , piece_id=NY5A6JZW4F6E4CSX3ERMKSAL2R3IYSKQ2EVHZSPF6ZOHWQFXBYWA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468843542 (591h37m0.339926587s) [noise=true , address=77.90.31.17:4781 , piece_id=TFUDJVQZFLS2XTZEGBPTV23WYLNHZMGK5ZXNDJSMW5HSNZXWXBBA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.469105310 (591h37m0.340188345s) [noise=true , address=145.79.227.35:4494 , piece_id=U4P6QB4ZRFGAJOHVSSTZHCG4PTBGII4LXAAJXXVN5RVR4WUWUBBQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.470034750 (591h37m0.341117795s) [noise=true , address=145.79.227.49:4365 , piece_id=Z357V6BZEGQOFLFH32RQLYJJZDLURVHZAHKW3H3ZIJXMQC3ZIFBQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.470487622 (591h37m0.341570657s) [noise=true , address=64.49.13.169:4713 , piece_id=YS6FRVS5V3BECH3TJQRJBGB5SSYYGUBK2HMO25PZEUOUSC7ENNZQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.470809334 (591h37m0.341892379s) [noise=true , address=77.90.31.10:4530 , piece_id=LNRM5YW2MZLXHWHGNUMP5I7Y5RRNRESBUA2M3SQKDFEQXZ25RJ3A, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.471118874 (591h37m0.342201919s) [noise=true , address=145.79.227.44:4221 , piece_id=GDKGEN3QCGFIICNST42FA2ZLLTQCHL2QAZ7AMUNPJIGYTSHDMOJQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.471431078 (591h37m0.342514113s) [noise=true , address=45.91.116.4:4963 , piece_id=FUGWNAY33ZGP6J57FIODDQQJ2UH46ABT52AU4Y7Y74Q2AOTCXWFA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.471741530 (591h37m0.342824575s) [noise=true , address=193.149.164.143:4704 , piece_id=PQQZ6N4A7TZGGX5DVRE6NT6TSDJVQARR22F6MZ5Z6LYXLUKF4V7A, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.472073311 (591h37m0.343156356s) [noise=true , address=145.79.227.33:4901 , piece_id=FFTUVNIKVKRAZ5R44V6WLWNKKC5MQYA66RZPSMW6N5EQKY7NSGZQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:01.899399345 (591h37m1.77048239s) [upload=518641]: Next returned nil segment Dec 30 07:06:01.899418712 (591h37m1.770501747s) [upload=518641]: Waiting for error group managing segments... Dec 30 07:06:02.323303785 (591h37m2.194386821s) [noise=true , address=145.79.227.46:4604 , piece_id=NY5A6JZW4F6E4CSX3ERMKSAL2R3IYSKQ2EVHZSPF6ZOHWQFXBYWA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.339689637 (591h37m2.210772682s) [noise=true , address=145.79.227.17:4000 , piece_id=KN3NDKB4NODHNXUWWQQ4YBWBPJCNE7YHYEMLQ4UITRCFZB4BRSMA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.346460570 (591h37m2.217543615s) [noise=true , address=145.79.227.29:4955 , piece_id=CBV7RUAZ6QAZE3VDTZKOGTIMOY66VUJTSYQSZRC7WLUD53UWVJTQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.347028050 (591h37m2.218111095s) [noise=true , address=145.79.227.34:4931 , piece_id=TJ4NTHHJQ4VLPBIC7RW42DBFIBI5G7LJWBN53TPSEVAWHANHMTBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.427966770 (591h37m2.299049815s) [noise=true , address=145.79.227.5:4226 , piece_id=QKDFEH6UKIH4QYN6DE3YNH3FZPAYGNYMQSBCYGM7EAW7F4MDA66A, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.446440007 (591h37m2.317523052s) [noise=true , address=145.79.227.35:4494 , piece_id=U4P6QB4ZRFGAJOHVSSTZHCG4PTBGII4LXAAJXXVN5RVR4WUWUBBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.583477598 (591h37m2.454560643s) [noise=true , address=77.90.31.19:4859 , piece_id=XHNZMPC2MF4RALF2FZJY73UYN4XWOYJUL66RFRUXNPT64JZQE5KQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.613112418 (591h37m2.484195463s) [noise=true , address=77.90.31.17:4781 , piece_id=TFUDJVQZFLS2XTZEGBPTV23WYLNHZMGK5ZXNDJSMW5HSNZXWXBBA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.631421591 (591h37m2.502504626s) [noise=true , address=77.90.31.10:4530 , piece_id=LNRM5YW2MZLXHWHGNUMP5I7Y5RRNRESBUA2M3SQKDFEQXZ25RJ3A, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.944663583 (591h37m2.815746628s) [noise=true , address=145.79.227.49:4365 , piece_id=Z357V6BZEGQOFLFH32RQLYJJZDLURVHZAHKW3H3ZIJXMQC3ZIFBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.195837408 (591h37m3.066920453s) [noise=true , address=193.149.164.143:4704 , piece_id=PQQZ6N4A7TZGGX5DVRE6NT6TSDJVQARR22F6MZ5Z6LYXLUKF4V7A, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.269365478 (591h37m3.140448523s) [noise=true , address=145.79.227.44:4221 , piece_id=GDKGEN3QCGFIICNST42FA2ZLLTQCHL2QAZ7AMUNPJIGYTSHDMOJQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.272362485 (591h37m3.14344553s) [noise=true , address=145.79.227.30:4778 , piece_id=EVWGEZL3VCMHEM6ALESBLMU65XBE74CKVMFXMSHK2GCXFPI35RBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.340891504 (591h37m3.211974549s) [noise=true , address=145.79.227.40:4082 , piece_id=7UFAIMVNINXXXRPNXLGJ34YEO3ILVN2O7UFG3HHTIT2DKWVZSUPA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.340922954 (591h37m3.212005999s) [seg_pos={0 0}, upload=518641]: Segment reached optimal threshold of 14 pieces. Dec 30 07:06:03.341286095 (591h37m3.21236913s) [noise=true , address=77.90.31.14:4693 , piece_id=CQ6A6DJJDCKQAAMCXW7FELWCWKAOI7YCCWKVDSTZEBXNHVKQRKPQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RASrnvpAj4ToD4xNvbma1iKZEe2uHK8WosjfZwDDom8Qd17Ey): protocol: write tcp 188.166.225.170:47364->77.90.31.14:4693: use of closed network connection; write tcp 188.166.225.170:47364->77.90.31.14:4693: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47364->77.90.31.14:4693: use of closed network connection Dec 30 07:06:03.341317094 (591h37m3.212400139s) [noise=true , address=77.90.31.14:4693 , piece_id=CQ6A6DJJDCKQAAMCXW7FELWCWKAOI7YCCWKVDSTZEBXNHVKQRKPQ, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341322114 (591h37m3.212405159s) [noise=true , address=145.79.227.33:4901 , piece_id=FFTUVNIKVKRAZ5R44V6WLWNKKC5MQYA66RZPSMW6N5EQKY7NSGZQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1): protocol: expected piece hash; context canceled; EOF Dec 30 07:06:03.341336261 (591h37m3.212419306s) [noise=true , address=145.79.227.33:4901 , piece_id=FFTUVNIKVKRAZ5R44V6WLWNKKC5MQYA66RZPSMW6N5EQKY7NSGZQ, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341546141 (591h37m3.212629196s) [noise=true , address=45.91.116.4:4963 , piece_id=FUGWNAY33ZGP6J57FIODDQQJ2UH46ABT52AU4Y7Y74Q2AOTCXWFA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Eki7jYegwvkdixR5pznQBbSeJPn9GjGofxnUApidQF4kRNBK5): protocol: expected piece hash; write tcp 188.166.225.170:51728->45.91.116.4:4963: use of closed network connection; write tcp 188.166.225.170:51728->45.91.116.4:4963: use of closed network connection; EOF Dec 30 07:06:03.341555549 (591h37m3.212638594s) [noise=true , address=45.91.116.4:4963 , piece_id=FUGWNAY33ZGP6J57FIODDQQJ2UH46ABT52AU4Y7Y74Q2AOTCXWFA, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341759577 (591h37m3.212842622s) [noise=true , address=64.49.13.169:4713 , piece_id=YS6FRVS5V3BECH3TJQRJBGB5SSYYGUBK2HMO25PZEUOUSC7ENNZQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17z3AiUfbxBX2eMDnUnqo1bbgojuQxJro7PsEaPjYzQjHQ3D6K): protocol: write tcp 188.166.225.170:42002->64.49.13.169:4713: use of closed network connection; write tcp 188.166.225.170:42002->64.49.13.169:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42002->64.49.13.169:4713: use of closed network connection Dec 30 07:06:03.341769897 (591h37m3.212852942s) [noise=true , address=64.49.13.169:4713 , piece_id=YS6FRVS5V3BECH3TJQRJBGB5SSYYGUBK2HMO25PZEUOUSC7ENNZQ, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341869927 (591h37m3.212952972s) [seg_pos={0 0}, upload=518641]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 07:06:03.342018730 (591h37m3.213101775s) [upload=518641]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 07:06:03.342025634 (591h37m3.213108679s) [upload=518641]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 07:06:03.635315934 (591h37m3.506398979s) [upload=518641]: Done uploading segments... Dec 30 07:06:03.635343146 (591h37m3.506426191s) [upload=518641]: Upload finished. err: <nil> Duration: 29.792291468s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 6.789537811s trace: https://jaeger.dev.storj.io/trace/4c2f4efaff0e0ea0 FAIL (2025-12-30 07:21:26 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== load=518292]: Uploading piece... Dec 30 06:41:13.598100152 (591h12m13.469183187s) [noise=true , address=81.171.14.70:4041 , piece_id=NORNA6WFTGPNOHU7MPTNNB5ONTBS5DJGCN2NWIZAEPYSUSIOSGEQ, seg_pos={0 0}, upload=518292]: Uploading piece... Dec 30 06:41:13.598587971 (591h12m13.469671016s) [noise=true , address=89.149.201.147:4007 , piece_id=HZLSYGSBP4TJH5V5XF3Q3AETJU625ERNDBD3VXW2HUZTTPAHLBMA, seg_pos={0 0}, upload=518292]: Uploading piece... Dec 30 06:41:13.599093143 (591h12m13.470176178s) [noise=true , address=89.149.201.147:4008 , piece_id=XYFRNJVIFFZA5S36KFPGZPVCPLRL2OW5QBJD3UR37VPPA2RX3RJA, seg_pos={0 0}, upload=518292]: Uploading piece... Dec 30 06:41:13.599699999 (591h12m13.470783034s) [noise=true , address=81.171.14.70:4039 , piece_id=DEJDXFDHOSAMIP2FN5LJBJNBB4QLJSE2AUDW6IMA6IC4VRY7VAGQ, seg_pos={0 0}, upload=518292]: Uploading piece... Dec 30 06:41:13.600136169 (591h12m13.471219204s) [noise=true , address=185.167.204.205:4034 , piece_id=OWZHQB5ZDT2QZ2QAHZJVRWQ222MX6CWLKJUOIAAGSMAVLWGJAOBA, seg_pos={0 0}, upload=518292]: Uploading piece... Dec 30 06:41:13.600543184 (591h12m13.471626229s) [noise=true , address=185.167.204.206:4058 , piece_id=ROOCGK64M5RHIWNAVCSZNYTZ6M2ORWB7MJ27XJVDTFVQIKJZHY4A, seg_pos={0 0}, upload=518292]: Uploading piece... Dec 30 06:41:13.601048286 (591h12m13.472131332s) [noise=true , address=81.171.14.70:4046 , piece_id=B2MTUNKMUP5S7XW4SVQ5QI2ZO3VFB5QVAKBBY4BIZIUILM6O7VYQ, seg_pos={0 0}, upload=518292]: Uploading piece... Dec 30 06:41:13.601698784 (591h12m13.472781819s) [noise=true , address=185.167.204.206:4064 , piece_id=H4PP7HPHXRX2LOZ4776AMWAJDN7OC435E2K4BXI7XOZYKTB4YUNA, seg_pos={0 0}, upload=518292]: Uploading piece... Dec 30 06:41:13.602168259 (591h12m13.473251304s) [noise=true , address=185.167.204.206:4065 , piece_id=LD3J4656Q5QEUCMJ5OKEYATEVRIKSXO7XGARTN25BVM7FDIYOWKA, seg_pos={0 0}, upload=518292]: Uploading piece... Dec 30 06:41:13.602782708 (591h12m13.473865753s) [noise=true , address=81.171.14.70:4027 , piece_id=6WEORWX5QOMFSJUEADIIBHHIPKNFP4NJPZAJADBUTPUSG7MZJMOQ, seg_pos={0 0}, upload=518292]: Uploading piece... Dec 30 06:41:14.423395002 (591h12m14.294478047s) [upload=518292]: Next returned nil segment Dec 30 06:41:14.423414068 (591h12m14.294497103s) [upload=518292]: Waiting for error group managing segments... Dec 30 06:41:14.817650316 (591h12m14.688733351s) [noise=true , address=185.167.204.205:4029 , piece_id=KUOMMYJ2F4NIK6VN4C5M3Y3TMHYEZH7QFNJ6K3CLOAVI5R6VKTHQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:14.817870054 (591h12m14.688953099s) [noise=true , address=185.167.204.205:4019 , piece_id=H6AH654VGX6NHBUKODIWGL7I2RKE3DDMY4UM6VDONP2EP3NPDJBA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:14.818623698 (591h12m14.689706744s) [noise=true , address=185.167.204.206:4065 , piece_id=LD3J4656Q5QEUCMJ5OKEYATEVRIKSXO7XGARTN25BVM7FDIYOWKA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:14.818932436 (591h12m14.690015481s) [noise=true , address=185.167.204.206:4061 , piece_id=T2XS4IFPQSVS3T45SPQ5ZDDVT2VMTMXHKL2ZMX4OIOPRR67SCD2Q, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:14.819147046 (591h12m14.690230091s) [noise=true , address=185.167.204.206:4071 , piece_id=QI2TOY5KCDZVWCCLMAJXVUWD2HB7S7IOTEAV2EHAMUNHKOFKTNYA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:14.819678578 (591h12m14.690761623s) [noise=true , address=185.167.204.206:4059 , piece_id=LMOGTWX2YIRR3EKNROK43O2B27S5L5U6GYB7HBVRBPQPH4WDDWZQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:14.819905028 (591h12m14.690988063s) [noise=true , address=185.167.204.206:4058 , piece_id=ROOCGK64M5RHIWNAVCSZNYTZ6M2ORWB7MJ27XJVDTFVQIKJZHY4A, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:14.820479312 (591h12m14.691562357s) [noise=true , address=185.167.204.205:4021 , piece_id=KP7J5XSY4SKSKZG7XXFZMNKMRP4472PXI7CW2UQBZICN4MAFO5RQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:14.823149296 (591h12m14.694232331s) [noise=true , address=185.167.204.205:4034 , piece_id=OWZHQB5ZDT2QZ2QAHZJVRWQ222MX6CWLKJUOIAAGSMAVLWGJAOBA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:14.827906996 (591h12m14.698990041s) [noise=true , address=185.167.204.206:4055 , piece_id=VYEDR6XHZBZMTVJ5U5OF67WZOGVMCMQLXXBQUI7DHBCCVQ2RRERQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:14.828003769 (591h12m14.699086814s) [noise=true , address=185.167.204.205:4016 , piece_id=I6MUDLAXUUDJL2C4HFE22C2L56A7D6HSCNWXKWJ7OKO2N4RA4W7Q, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:14.828893203 (591h12m14.699976238s) [noise=true , address=185.167.204.206:4064 , piece_id=H4PP7HPHXRX2LOZ4776AMWAJDN7OC435E2K4BXI7XOZYKTB4YUNA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:14.829477074 (591h12m14.700560109s) [noise=true , address=185.167.204.205:4017 , piece_id=CZP65KETMGOHFDMV7KENRL4V36UHBPWRDCBBPUM5NE5L2PV4AQCQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:14.829774972 (591h12m14.700858007s) [noise=true , address=185.167.204.206:4043 , piece_id=G7VZYID4TXGNPDN63NQULXOVSX42WMFFNPBX6KQJ5L2WSW4D2YGQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:14.831734493 (591h12m14.702817528s) [noise=true , address=185.167.204.206:4049 , piece_id=BQ7JEGSGPIFJUGZFBYRPDMBNPZ24BG6TEG4VUM4JUBTE4T2RF4SA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:15.405688561 (591h12m15.276771606s) [noise=true , address=81.171.14.70:4027 , piece_id=6WEORWX5QOMFSJUEADIIBHHIPKNFP4NJPZAJADBUTPUSG7MZJMOQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:15.417110440 (591h12m15.288193485s) [noise=true , address=81.171.14.70:4031 , piece_id=U6Z4SNK32ZSWQ7NDR7ACXGRC4BADDOPR65ILBR64QLGVNDBADFFQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:15.425600446 (591h12m15.296683481s) [noise=true , address=81.171.14.70:4037 , piece_id=6A25LEIAQEJB3DNCTXQT6K7ZSKIGOW4QUGMNKLOG3CT2XXH25PSA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:15.499056228 (591h12m15.370139263s) [noise=true , address=81.171.14.70:4038 , piece_id=3WRHW3VGXWHOEOV5Q6CLYSF7MUVWS7QHTNHLSTCKAUERPHAH3I3Q, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:15.514998925 (591h12m15.38608197s) [noise=true , address=81.171.14.70:4041 , piece_id=NORNA6WFTGPNOHU7MPTNNB5ONTBS5DJGCN2NWIZAEPYSUSIOSGEQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:15.528579544 (591h12m15.399662589s) [noise=true , address=81.171.14.70:4030 , piece_id=KITOJIT34CHS2QZBXCW2SJPZB4RHEHCMJ6IYQ4Z5KLL24YAWBIUQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:15.528991398 (591h12m15.400074434s) [noise=true , address=81.171.14.70:4047 , piece_id=ZWV5DP4YC3MSYDWEQH4ZK3VR4WZ4DUTI4VL3WVLQKFIEKIW44LOQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:15.534844544 (591h12m15.405927579s) [noise=true , address=81.171.14.70:4035 , piece_id=XCLL5SOAPJSSGF5YYQH6ZKGOHMLS6RI6H26FOW5XIJRY3TWYNEGA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:15.915614910 (591h12m15.786697945s) [noise=true , address=81.171.14.70:4039 , piece_id=DEJDXFDHOSAMIP2FN5LJBJNBB4QLJSE2AUDW6IMA6IC4VRY7VAGQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:15.967820176 (591h12m15.838903211s) [noise=true , address=81.171.14.70:4046 , piece_id=B2MTUNKMUP5S7XW4SVQ5QI2ZO3VFB5QVAKBBY4BIZIUILM6O7VYQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:16.172116128 (591h12m16.043199163s) [noise=true , address=89.149.201.147:4007 , piece_id=HZLSYGSBP4TJH5V5XF3Q3AETJU625ERNDBD3VXW2HUZTTPAHLBMA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:16.172137559 (591h12m16.043220604s) [seg_pos={0 0}, upload=518292]: Segment reached optimal threshold of 26 pieces. Dec 30 06:41:16.172336949 (591h12m16.043419994s) [noise=true , address=89.149.201.147:4008 , piece_id=XYFRNJVIFFZA5S36KFPGZPVCPLRL2OW5QBJD3UR37VPPA2RX3RJA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QN8VP6o7qGw9Dd8EafU9xe7s7Cuf1FyqPMbnRaznrfPxfEUUn): protocol: expected piece hash; context canceled; EOF Dec 30 06:41:16.172344213 (591h12m16.043427258s) [noise=true , address=89.149.201.147:4008 , piece_id=XYFRNJVIFFZA5S36KFPGZPVCPLRL2OW5QBJD3UR37VPPA2RX3RJA, seg_pos={0 0}, upload=518292]: Long tail cancellation due to optimal threshold Dec 30 06:41:16.172554002 (591h12m16.043637037s) [noise=true , address=89.149.201.147:4020 , piece_id=N7G5DGEDYVKBTO5S6BYW3IYEOK7HAMK5JQIBMVPMPYRDHXODWEGA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YeKHJH7CoouLFJYENJhzVLNkA32hJaqjLz3dXhr78eaSGtcic): protocol: expected piece hash; context canceled; EOF Dec 30 06:41:16.172565233 (591h12m16.043648278s) [noise=true , address=89.149.201.147:4020 , piece_id=N7G5DGEDYVKBTO5S6BYW3IYEOK7HAMK5JQIBMVPMPYRDHXODWEGA, seg_pos={0 0}, upload=518292]: Long tail cancellation due to optimal threshold Dec 30 06:41:16.172587506 (591h12m16.043670541s) [seg_pos={0 0}, upload=518292]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 30 06:41:16.172717945 (591h12m16.04380098s) [upload=518292]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 06:41:16.172724567 (591h12m16.043807602s) [upload=518292]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 06:41:16.404092237 (591h12m16.275175292s) [upload=518292]: Done uploading segments... Dec 30 06:41:16.404119940 (591h12m16.275202975s) [upload=518292]: Upload finished. err: <nil> Duration: 6.789556096s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 36.990277445s trace: https://jaeger.dev.storj.io/trace/1d0a2f93cb06aaa4 FAIL (2025-12-30 07:20:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== loading piece... Dec 30 07:06:00.467281687 (591h37m0.338364722s) [noise=true , address=77.90.31.14:4693 , piece_id=CQ6A6DJJDCKQAAMCXW7FELWCWKAOI7YCCWKVDSTZEBXNHVKQRKPQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.467716555 (591h37m0.33879959s) [noise=true , address=145.79.227.40:4082 , piece_id=7UFAIMVNINXXXRPNXLGJ34YEO3ILVN2O7UFG3HHTIT2DKWVZSUPA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468044320 (591h37m0.339127365s) [noise=true , address=145.79.227.29:4955 , piece_id=CBV7RUAZ6QAZE3VDTZKOGTIMOY66VUJTSYQSZRC7WLUD53UWVJTQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468308362 (591h37m0.339391407s) [noise=true , address=145.79.227.17:4000 , piece_id=KN3NDKB4NODHNXUWWQQ4YBWBPJCNE7YHYEMLQ4UITRCFZB4BRSMA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468565703 (591h37m0.339648748s) [noise=true , address=145.79.227.46:4604 , piece_id=NY5A6JZW4F6E4CSX3ERMKSAL2R3IYSKQ2EVHZSPF6ZOHWQFXBYWA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468843542 (591h37m0.339926587s) [noise=true , address=77.90.31.17:4781 , piece_id=TFUDJVQZFLS2XTZEGBPTV23WYLNHZMGK5ZXNDJSMW5HSNZXWXBBA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.469105310 (591h37m0.340188345s) [noise=true , address=145.79.227.35:4494 , piece_id=U4P6QB4ZRFGAJOHVSSTZHCG4PTBGII4LXAAJXXVN5RVR4WUWUBBQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.470034750 (591h37m0.341117795s) [noise=true , address=145.79.227.49:4365 , piece_id=Z357V6BZEGQOFLFH32RQLYJJZDLURVHZAHKW3H3ZIJXMQC3ZIFBQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.470487622 (591h37m0.341570657s) [noise=true , address=64.49.13.169:4713 , piece_id=YS6FRVS5V3BECH3TJQRJBGB5SSYYGUBK2HMO25PZEUOUSC7ENNZQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.470809334 (591h37m0.341892379s) [noise=true , address=77.90.31.10:4530 , piece_id=LNRM5YW2MZLXHWHGNUMP5I7Y5RRNRESBUA2M3SQKDFEQXZ25RJ3A, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.471118874 (591h37m0.342201919s) [noise=true , address=145.79.227.44:4221 , piece_id=GDKGEN3QCGFIICNST42FA2ZLLTQCHL2QAZ7AMUNPJIGYTSHDMOJQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.471431078 (591h37m0.342514113s) [noise=true , address=45.91.116.4:4963 , piece_id=FUGWNAY33ZGP6J57FIODDQQJ2UH46ABT52AU4Y7Y74Q2AOTCXWFA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.471741530 (591h37m0.342824575s) [noise=true , address=193.149.164.143:4704 , piece_id=PQQZ6N4A7TZGGX5DVRE6NT6TSDJVQARR22F6MZ5Z6LYXLUKF4V7A, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.472073311 (591h37m0.343156356s) [noise=true , address=145.79.227.33:4901 , piece_id=FFTUVNIKVKRAZ5R44V6WLWNKKC5MQYA66RZPSMW6N5EQKY7NSGZQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:01.899399345 (591h37m1.77048239s) [upload=518641]: Next returned nil segment Dec 30 07:06:01.899418712 (591h37m1.770501747s) [upload=518641]: Waiting for error group managing segments... Dec 30 07:06:02.323303785 (591h37m2.194386821s) [noise=true , address=145.79.227.46:4604 , piece_id=NY5A6JZW4F6E4CSX3ERMKSAL2R3IYSKQ2EVHZSPF6ZOHWQFXBYWA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.339689637 (591h37m2.210772682s) [noise=true , address=145.79.227.17:4000 , piece_id=KN3NDKB4NODHNXUWWQQ4YBWBPJCNE7YHYEMLQ4UITRCFZB4BRSMA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.346460570 (591h37m2.217543615s) [noise=true , address=145.79.227.29:4955 , piece_id=CBV7RUAZ6QAZE3VDTZKOGTIMOY66VUJTSYQSZRC7WLUD53UWVJTQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.347028050 (591h37m2.218111095s) [noise=true , address=145.79.227.34:4931 , piece_id=TJ4NTHHJQ4VLPBIC7RW42DBFIBI5G7LJWBN53TPSEVAWHANHMTBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.427966770 (591h37m2.299049815s) [noise=true , address=145.79.227.5:4226 , piece_id=QKDFEH6UKIH4QYN6DE3YNH3FZPAYGNYMQSBCYGM7EAW7F4MDA66A, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.446440007 (591h37m2.317523052s) [noise=true , address=145.79.227.35:4494 , piece_id=U4P6QB4ZRFGAJOHVSSTZHCG4PTBGII4LXAAJXXVN5RVR4WUWUBBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.583477598 (591h37m2.454560643s) [noise=true , address=77.90.31.19:4859 , piece_id=XHNZMPC2MF4RALF2FZJY73UYN4XWOYJUL66RFRUXNPT64JZQE5KQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.613112418 (591h37m2.484195463s) [noise=true , address=77.90.31.17:4781 , piece_id=TFUDJVQZFLS2XTZEGBPTV23WYLNHZMGK5ZXNDJSMW5HSNZXWXBBA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.631421591 (591h37m2.502504626s) [noise=true , address=77.90.31.10:4530 , piece_id=LNRM5YW2MZLXHWHGNUMP5I7Y5RRNRESBUA2M3SQKDFEQXZ25RJ3A, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.944663583 (591h37m2.815746628s) [noise=true , address=145.79.227.49:4365 , piece_id=Z357V6BZEGQOFLFH32RQLYJJZDLURVHZAHKW3H3ZIJXMQC3ZIFBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.195837408 (591h37m3.066920453s) [noise=true , address=193.149.164.143:4704 , piece_id=PQQZ6N4A7TZGGX5DVRE6NT6TSDJVQARR22F6MZ5Z6LYXLUKF4V7A, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.269365478 (591h37m3.140448523s) [noise=true , address=145.79.227.44:4221 , piece_id=GDKGEN3QCGFIICNST42FA2ZLLTQCHL2QAZ7AMUNPJIGYTSHDMOJQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.272362485 (591h37m3.14344553s) [noise=true , address=145.79.227.30:4778 , piece_id=EVWGEZL3VCMHEM6ALESBLMU65XBE74CKVMFXMSHK2GCXFPI35RBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.340891504 (591h37m3.211974549s) [noise=true , address=145.79.227.40:4082 , piece_id=7UFAIMVNINXXXRPNXLGJ34YEO3ILVN2O7UFG3HHTIT2DKWVZSUPA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.340922954 (591h37m3.212005999s) [seg_pos={0 0}, upload=518641]: Segment reached optimal threshold of 14 pieces. Dec 30 07:06:03.341286095 (591h37m3.21236913s) [noise=true , address=77.90.31.14:4693 , piece_id=CQ6A6DJJDCKQAAMCXW7FELWCWKAOI7YCCWKVDSTZEBXNHVKQRKPQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RASrnvpAj4ToD4xNvbma1iKZEe2uHK8WosjfZwDDom8Qd17Ey): protocol: write tcp 188.166.225.170:47364->77.90.31.14:4693: use of closed network connection; write tcp 188.166.225.170:47364->77.90.31.14:4693: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47364->77.90.31.14:4693: use of closed network connection Dec 30 07:06:03.341317094 (591h37m3.212400139s) [noise=true , address=77.90.31.14:4693 , piece_id=CQ6A6DJJDCKQAAMCXW7FELWCWKAOI7YCCWKVDSTZEBXNHVKQRKPQ, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341322114 (591h37m3.212405159s) [noise=true , address=145.79.227.33:4901 , piece_id=FFTUVNIKVKRAZ5R44V6WLWNKKC5MQYA66RZPSMW6N5EQKY7NSGZQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1): protocol: expected piece hash; context canceled; EOF Dec 30 07:06:03.341336261 (591h37m3.212419306s) [noise=true , address=145.79.227.33:4901 , piece_id=FFTUVNIKVKRAZ5R44V6WLWNKKC5MQYA66RZPSMW6N5EQKY7NSGZQ, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341546141 (591h37m3.212629196s) [noise=true , address=45.91.116.4:4963 , piece_id=FUGWNAY33ZGP6J57FIODDQQJ2UH46ABT52AU4Y7Y74Q2AOTCXWFA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Eki7jYegwvkdixR5pznQBbSeJPn9GjGofxnUApidQF4kRNBK5): protocol: expected piece hash; write tcp 188.166.225.170:51728->45.91.116.4:4963: use of closed network connection; write tcp 188.166.225.170:51728->45.91.116.4:4963: use of closed network connection; EOF Dec 30 07:06:03.341555549 (591h37m3.212638594s) [noise=true , address=45.91.116.4:4963 , piece_id=FUGWNAY33ZGP6J57FIODDQQJ2UH46ABT52AU4Y7Y74Q2AOTCXWFA, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341759577 (591h37m3.212842622s) [noise=true , address=64.49.13.169:4713 , piece_id=YS6FRVS5V3BECH3TJQRJBGB5SSYYGUBK2HMO25PZEUOUSC7ENNZQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17z3AiUfbxBX2eMDnUnqo1bbgojuQxJro7PsEaPjYzQjHQ3D6K): protocol: write tcp 188.166.225.170:42002->64.49.13.169:4713: use of closed network connection; write tcp 188.166.225.170:42002->64.49.13.169:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42002->64.49.13.169:4713: use of closed network connection Dec 30 07:06:03.341769897 (591h37m3.212852942s) [noise=true , address=64.49.13.169:4713 , piece_id=YS6FRVS5V3BECH3TJQRJBGB5SSYYGUBK2HMO25PZEUOUSC7ENNZQ, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341869927 (591h37m3.212952972s) [seg_pos={0 0}, upload=518641]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 07:06:03.342018730 (591h37m3.213101775s) [upload=518641]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 07:06:03.342025634 (591h37m3.213108679s) [upload=518641]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 07:06:03.635315934 (591h37m3.506398979s) [upload=518641]: Done uploading segments... Dec 30 07:06:03.635343146 (591h37m3.506426191s) [upload=518641]: Upload finished. err: <nil> Duration: 36.990293154s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [X.....X....XX...X.....X....XXX..X.....X...X..X.XX..X...X....] duration: 5.929179684s trace: https://jaeger.dev.storj.io/trace/5beb0f64287afad6 FAIL (2025-12-30 07:20:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: 3TS6SBJ3HMLLLPRF2SQB7BJMWS3IN7G7ZYDXEWRTH7UPXVJNEV2A) context deadline exceeded; piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: Y6DQST7V76RF2QRNYCAUG7JW44WSDE5UB2ZDDDQGWDZYCVTXFNSQ) context deadline exceeded ======LOGS====== Dec 30 07:17:28.850872807 (591h48m28.721955842s) [upload=518812]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 07:17:28.850877576 (591h48m28.721960621s) [upload=518812]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 07:17:29.093052385 (591h48m28.96413543s) [seg_pos={0 0}, upload=518812]: Begin upload segment... Dec 30 07:17:29.093119923 (591h48m28.964202958s) [seg_pos={0 0}, upload=518812]: Done begin upload segment. Dec 30 07:17:29.093167074 (591h48m28.964250109s) [noise=true , address=45.91.116.7:4010 , piece_id=GC5BNYAHKDHMWND3M63XOSS4VTI2EDEXE2T2P3NOYLY4FPG7U7NA, seg_pos={0 0}, upload=518812]: Uploading piece... Dec 30 07:17:29.094549415 (591h48m28.96563246s) [noise=true , address=45.91.116.13:4155 , piece_id=DCNMHT6GSOEA7SBGMEUGJO6TGVUQDQYP3CYOPPHN5ACONACAA5NQ, seg_pos={0 0}, upload=518812]: Uploading piece... Dec 30 07:17:29.094595925 (591h48m28.96567897s) [noise=true , address=77.90.31.21:4877 , piece_id=SMIERBLZBJ7K66DKNPSWHMIMKDDZ66OIZTYJKP7RPB5CEZSQ24EQ, seg_pos={0 0}, upload=518812]: Uploading piece... Dec 30 07:17:29.094725961 (591h48m28.965809007s) [noise=true , address=45.91.116.11:4118 , piece_id=Y6DQST7V76RF2QRNYCAUG7JW44WSDE5UB2ZDDDQGWDZYCVTXFNSQ, seg_pos={0 0}, upload=518812]: Uploading piece... Dec 30 07:17:29.095084375 (591h48m28.96616741s) [noise=true , address=145.79.227.14:4840 , piece_id=LNUZRYHY5JNMVXSYR26NONQCVVCQIME4BQVPC6373MASTPR2ZYNA, seg_pos={0 0}, upload=518812]: Uploading piece... Dec 30 07:17:29.095484366 (591h48m28.966567401s) [noise=true , address=145.79.227.44:4193 , piece_id=FUTYTVGBO7XDAADY4NEBNXQAMFDZKC22AOPVOFPLT5I5WRITLOOQ, seg_pos={0 0}, upload=518812]: Uploading piece... Dec 30 07:17:29.095768637 (591h48m28.966851682s) [noise=true , address=45.91.116.12:4225 , piece_id=DDTKXUON47NUOFBAZJCAF3CTDXNEVY7UV24KPNZZRLS4AWQKFJLQ, seg_pos={0 0}, upload=518812]: Uploading piece... Dec 30 07:17:29.095791010 (591h48m28.966874085s) [noise=true , address=145.79.227.16:4910 , piece_id=GUTA6YH2OQDSP44SMMJWQCXLX2KSEREY2UJOKEGWQOSYMMQCH45A, seg_pos={0 0}, upload=518812]: Uploading piece... Dec 30 07:17:29.096359652 (591h48m28.967442697s) [noise=true , address=145.79.227.20:4439 , piece_id=OIMXD3C4KXUFZ3BSQALCY64CNPSVN3CYPAR3LESG4KMZSNDR27MQ, seg_pos={0 0}, upload=518812]: Uploading piece... Dec 30 07:17:29.096949476 (591h48m28.968032521s) [noise=true , address=77.90.31.11:4554 , piece_id=3TS6SBJ3HMLLLPRF2SQB7BJMWS3IN7G7ZYDXEWRTH7UPXVJNEV2A, seg_pos={0 0}, upload=518812]: Uploading piece... Dec 30 07:17:29.097643046 (591h48m28.968726091s) [noise=true , address=45.91.116.4:4940 , piece_id=WDTVLWU7JC5HRKZF44WXGHWSD2YXZVH24WNRYR4KGUIOIEFYZJWA, seg_pos={0 0}, upload=518812]: Uploading piece... Dec 30 07:17:29.098025625 (591h48m28.96910866s) [noise=true , address=145.79.227.21:4473 , piece_id=3D6AQBT6T63OHVSAZC26FAYMBB4ZQSGSR4QJOV6ALCM5PAURWMJQ, seg_pos={0 0}, upload=518812]: Uploading piece... Dec 30 07:17:29.098443681 (591h48m28.969526726s) [noise=true , address=45.91.116.8:4045 , piece_id=BLK335NDLSNVM5JJXD2YPY4YGPTBU5GLJRLVMLOD2CL3NAWD5QAQ, seg_pos={0 0}, upload=518812]: Uploading piece... Dec 30 07:17:29.098740195 (591h48m28.96982324s) [noise=true , address=77.90.31.16:4736 , piece_id=F7BGEHXDVCASQ4LCWCRZBJVSSKXSSDVTFRHXGQJG5DSGOPDAE5QA, seg_pos={0 0}, upload=518812]: Uploading piece... Dec 30 07:17:29.099044865 (591h48m28.97012791s) [noise=true , address=77.90.31.10:4520 , piece_id=J5MNM7IF5PHWHMSNJ7LG4OON7BWZLL47XXFXQXPK54JP5FBRYPQA, seg_pos={0 0}, upload=518812]: Uploading piece... Dec 30 07:17:29.099476949 (591h48m28.970559984s) [noise=true , address=77.90.31.18:4805 , piece_id=DHSEC5JSVGU6752SZXEKKUP5PYP5GCHWQRACEBUNXYN6YA4QXOZA, seg_pos={0 0}, upload=518812]: Uploading piece... Dec 30 07:17:30.802773702 (591h48m30.673856737s) [upload=518812]: Next returned nil segment Dec 30 07:17:30.802787058 (591h48m30.673870093s) [upload=518812]: Waiting for error group managing segments... Dec 30 07:17:31.088090380 (591h48m30.959173415s) [noise=true , address=145.79.227.16:4910 , piece_id=GUTA6YH2OQDSP44SMMJWQCXLX2KSEREY2UJOKEGWQOSYMMQCH45A, seg_pos={0 0}, upload=518812]: Done uploading piece. err: <nil> Dec 30 07:17:31.088850928 (591h48m30.959933963s) [noise=true , address=145.79.227.21:4473 , piece_id=3D6AQBT6T63OHVSAZC26FAYMBB4ZQSGSR4QJOV6ALCM5PAURWMJQ, seg_pos={0 0}, upload=518812]: Done uploading piece. err: <nil> Dec 30 07:17:31.251281280 (591h48m31.122364325s) [noise=true , address=77.90.31.18:4805 , piece_id=DHSEC5JSVGU6752SZXEKKUP5PYP5GCHWQRACEBUNXYN6YA4QXOZA, seg_pos={0 0}, upload=518812]: Done uploading piece. err: <nil> Dec 30 07:17:31.280445815 (591h48m31.15152885s) [noise=true , address=77.90.31.16:4736 , piece_id=F7BGEHXDVCASQ4LCWCRZBJVSSKXSSDVTFRHXGQJG5DSGOPDAE5QA, seg_pos={0 0}, upload=518812]: Done uploading piece. err: <nil> Dec 30 07:17:31.281407896 (591h48m31.152490941s) [noise=true , address=77.90.31.11:4554 , piece_id=3TS6SBJ3HMLLLPRF2SQB7BJMWS3IN7G7ZYDXEWRTH7UPXVJNEV2A, seg_pos={0 0}, upload=518812]: Done uploading piece. err: <nil> Dec 30 07:17:31.374287703 (591h48m31.245370738s) [noise=true , address=45.91.116.12:4225 , piece_id=DDTKXUON47NUOFBAZJCAF3CTDXNEVY7UV24KPNZZRLS4AWQKFJLQ, seg_pos={0 0}, upload=518812]: Done uploading piece. err: <nil> Dec 30 07:17:31.437914779 (591h48m31.308997814s) [noise=true , address=77.90.31.21:4877 , piece_id=SMIERBLZBJ7K66DKNPSWHMIMKDDZ66OIZTYJKP7RPB5CEZSQ24EQ, seg_pos={0 0}, upload=518812]: Done uploading piece. err: <nil> Dec 30 07:17:31.481026952 (591h48m31.352109987s) [noise=true , address=45.91.116.7:4010 , piece_id=GC5BNYAHKDHMWND3M63XOSS4VTI2EDEXE2T2P3NOYLY4FPG7U7NA, seg_pos={0 0}, upload=518812]: Done uploading piece. err: <nil> Dec 30 07:17:31.496767514 (591h48m31.367850549s) [noise=true , address=45.91.116.8:4045 , piece_id=BLK335NDLSNVM5JJXD2YPY4YGPTBU5GLJRLVMLOD2CL3NAWD5QAQ, seg_pos={0 0}, upload=518812]: Done uploading piece. err: <nil> Dec 30 07:17:31.525092179 (591h48m31.396175224s) [noise=true , address=45.91.116.13:4155 , piece_id=DCNMHT6GSOEA7SBGMEUGJO6TGVUQDQYP3CYOPPHN5ACONACAA5NQ, seg_pos={0 0}, upload=518812]: Done uploading piece. err: <nil> Dec 30 07:17:31.991029587 (591h48m31.862112632s) [noise=true , address=77.90.31.10:4520 , piece_id=J5MNM7IF5PHWHMSNJ7LG4OON7BWZLL47XXFXQXPK54JP5FBRYPQA, seg_pos={0 0}, upload=518812]: Done uploading piece. err: <nil> Dec 30 07:17:32.130417734 (591h48m32.001500779s) [noise=true , address=45.91.116.4:4940 , piece_id=WDTVLWU7JC5HRKZF44WXGHWSD2YXZVH24WNRYR4KGUIOIEFYZJWA, seg_pos={0 0}, upload=518812]: Done uploading piece. err: <nil> Dec 30 07:17:33.003450795 (591h48m32.87453384s) [noise=true , address=45.91.116.11:4118 , piece_id=Y6DQST7V76RF2QRNYCAUG7JW44WSDE5UB2ZDDDQGWDZYCVTXFNSQ, seg_pos={0 0}, upload=518812]: Done uploading piece. err: <nil> Dec 30 07:17:33.003471545 (591h48m32.87455459s) [seg_pos={0 0}, upload=518812]: Segment reached optimal threshold of 13 pieces. Dec 30 07:17:33.003690913 (591h48m32.874773948s) [noise=true , address=145.79.227.44:4193 , piece_id=FUTYTVGBO7XDAADY4NEBNXQAMFDZKC22AOPVOFPLT5I5WRITLOOQ, seg_pos={0 0}, upload=518812]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: write tcp 188.166.225.170:54804->145.79.227.44:4193: use of closed network connection; write tcp 188.166.225.170:54804->145.79.227.44:4193: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54804->145.79.227.44:4193: use of closed network connection Dec 30 07:17:33.003698647 (591h48m32.874781682s) [noise=true , address=145.79.227.44:4193 , piece_id=FUTYTVGBO7XDAADY4NEBNXQAMFDZKC22AOPVOFPLT5I5WRITLOOQ, seg_pos={0 0}, upload=518812]: Long tail cancellation due to optimal threshold Dec 30 07:17:33.004107296 (591h48m32.875190341s) [noise=true , address=145.79.227.20:4439 , piece_id=OIMXD3C4KXUFZ3BSQALCY64CNPSVN3CYPAR3LESG4KMZSNDR27MQ, seg_pos={0 0}, upload=518812]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn): protocol: read tcp 188.166.225.170:53380->145.79.227.20:4439: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 30 07:17:33.004116623 (591h48m32.875199668s) [noise=true , address=145.79.227.20:4439 , piece_id=OIMXD3C4KXUFZ3BSQALCY64CNPSVN3CYPAR3LESG4KMZSNDR27MQ, seg_pos={0 0}, upload=518812]: Long tail cancellation due to optimal threshold Dec 30 07:17:33.004153644 (591h48m32.875236689s) [noise=true , address=145.79.227.14:4840 , piece_id=LNUZRYHY5JNMVXSYR26NONQCVVCQIME4BQVPC6373MASTPR2ZYNA, seg_pos={0 0}, upload=518812]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: write tcp 188.166.225.170:33654->145.79.227.14:4840: use of closed network connection; write tcp 188.166.225.170:33654->145.79.227.14:4840: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33654->145.79.227.14:4840: use of closed network connection Dec 30 07:17:33.004161128 (591h48m32.875244173s) [noise=true , address=145.79.227.14:4840 , piece_id=LNUZRYHY5JNMVXSYR26NONQCVVCQIME4BQVPC6373MASTPR2ZYNA, seg_pos={0 0}, upload=518812]: Long tail cancellation due to optimal threshold Dec 30 07:17:33.004179413 (591h48m32.875262458s) [seg_pos={0 0}, upload=518812]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 07:17:33.004280906 (591h48m32.875363941s) [upload=518812]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 07:17:33.004287398 (591h48m32.875370443s) [upload=518812]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 07:17:33.288518488 (591h48m33.159601523s) [upload=518812]: Done uploading segments... Dec 30 07:17:33.288538457 (591h48m33.159621502s) [upload=518812]: Upload finished. err: <nil> Duration: 5.929209681s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 8.417587754s trace: https://jaeger.dev.storj.io/trace/4c2f4efaff0e0ea0 FAIL (2025-12-30 07:20:29 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== load=518292]: Uploading piece... Dec 30 06:41:13.598100152 (591h12m13.469183187s) [noise=true , address=81.171.14.70:4041 , piece_id=NORNA6WFTGPNOHU7MPTNNB5ONTBS5DJGCN2NWIZAEPYSUSIOSGEQ, seg_pos={0 0}, upload=518292]: Uploading piece... Dec 30 06:41:13.598587971 (591h12m13.469671016s) [noise=true , address=89.149.201.147:4007 , piece_id=HZLSYGSBP4TJH5V5XF3Q3AETJU625ERNDBD3VXW2HUZTTPAHLBMA, seg_pos={0 0}, upload=518292]: Uploading piece... Dec 30 06:41:13.599093143 (591h12m13.470176178s) [noise=true , address=89.149.201.147:4008 , piece_id=XYFRNJVIFFZA5S36KFPGZPVCPLRL2OW5QBJD3UR37VPPA2RX3RJA, seg_pos={0 0}, upload=518292]: Uploading piece... Dec 30 06:41:13.599699999 (591h12m13.470783034s) [noise=true , address=81.171.14.70:4039 , piece_id=DEJDXFDHOSAMIP2FN5LJBJNBB4QLJSE2AUDW6IMA6IC4VRY7VAGQ, seg_pos={0 0}, upload=518292]: Uploading piece... Dec 30 06:41:13.600136169 (591h12m13.471219204s) [noise=true , address=185.167.204.205:4034 , piece_id=OWZHQB5ZDT2QZ2QAHZJVRWQ222MX6CWLKJUOIAAGSMAVLWGJAOBA, seg_pos={0 0}, upload=518292]: Uploading piece... Dec 30 06:41:13.600543184 (591h12m13.471626229s) [noise=true , address=185.167.204.206:4058 , piece_id=ROOCGK64M5RHIWNAVCSZNYTZ6M2ORWB7MJ27XJVDTFVQIKJZHY4A, seg_pos={0 0}, upload=518292]: Uploading piece... Dec 30 06:41:13.601048286 (591h12m13.472131332s) [noise=true , address=81.171.14.70:4046 , piece_id=B2MTUNKMUP5S7XW4SVQ5QI2ZO3VFB5QVAKBBY4BIZIUILM6O7VYQ, seg_pos={0 0}, upload=518292]: Uploading piece... Dec 30 06:41:13.601698784 (591h12m13.472781819s) [noise=true , address=185.167.204.206:4064 , piece_id=H4PP7HPHXRX2LOZ4776AMWAJDN7OC435E2K4BXI7XOZYKTB4YUNA, seg_pos={0 0}, upload=518292]: Uploading piece... Dec 30 06:41:13.602168259 (591h12m13.473251304s) [noise=true , address=185.167.204.206:4065 , piece_id=LD3J4656Q5QEUCMJ5OKEYATEVRIKSXO7XGARTN25BVM7FDIYOWKA, seg_pos={0 0}, upload=518292]: Uploading piece... Dec 30 06:41:13.602782708 (591h12m13.473865753s) [noise=true , address=81.171.14.70:4027 , piece_id=6WEORWX5QOMFSJUEADIIBHHIPKNFP4NJPZAJADBUTPUSG7MZJMOQ, seg_pos={0 0}, upload=518292]: Uploading piece... Dec 30 06:41:14.423395002 (591h12m14.294478047s) [upload=518292]: Next returned nil segment Dec 30 06:41:14.423414068 (591h12m14.294497103s) [upload=518292]: Waiting for error group managing segments... Dec 30 06:41:14.817650316 (591h12m14.688733351s) [noise=true , address=185.167.204.205:4029 , piece_id=KUOMMYJ2F4NIK6VN4C5M3Y3TMHYEZH7QFNJ6K3CLOAVI5R6VKTHQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:14.817870054 (591h12m14.688953099s) [noise=true , address=185.167.204.205:4019 , piece_id=H6AH654VGX6NHBUKODIWGL7I2RKE3DDMY4UM6VDONP2EP3NPDJBA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:14.818623698 (591h12m14.689706744s) [noise=true , address=185.167.204.206:4065 , piece_id=LD3J4656Q5QEUCMJ5OKEYATEVRIKSXO7XGARTN25BVM7FDIYOWKA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:14.818932436 (591h12m14.690015481s) [noise=true , address=185.167.204.206:4061 , piece_id=T2XS4IFPQSVS3T45SPQ5ZDDVT2VMTMXHKL2ZMX4OIOPRR67SCD2Q, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:14.819147046 (591h12m14.690230091s) [noise=true , address=185.167.204.206:4071 , piece_id=QI2TOY5KCDZVWCCLMAJXVUWD2HB7S7IOTEAV2EHAMUNHKOFKTNYA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:14.819678578 (591h12m14.690761623s) [noise=true , address=185.167.204.206:4059 , piece_id=LMOGTWX2YIRR3EKNROK43O2B27S5L5U6GYB7HBVRBPQPH4WDDWZQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:14.819905028 (591h12m14.690988063s) [noise=true , address=185.167.204.206:4058 , piece_id=ROOCGK64M5RHIWNAVCSZNYTZ6M2ORWB7MJ27XJVDTFVQIKJZHY4A, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:14.820479312 (591h12m14.691562357s) [noise=true , address=185.167.204.205:4021 , piece_id=KP7J5XSY4SKSKZG7XXFZMNKMRP4472PXI7CW2UQBZICN4MAFO5RQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:14.823149296 (591h12m14.694232331s) [noise=true , address=185.167.204.205:4034 , piece_id=OWZHQB5ZDT2QZ2QAHZJVRWQ222MX6CWLKJUOIAAGSMAVLWGJAOBA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:14.827906996 (591h12m14.698990041s) [noise=true , address=185.167.204.206:4055 , piece_id=VYEDR6XHZBZMTVJ5U5OF67WZOGVMCMQLXXBQUI7DHBCCVQ2RRERQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:14.828003769 (591h12m14.699086814s) [noise=true , address=185.167.204.205:4016 , piece_id=I6MUDLAXUUDJL2C4HFE22C2L56A7D6HSCNWXKWJ7OKO2N4RA4W7Q, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:14.828893203 (591h12m14.699976238s) [noise=true , address=185.167.204.206:4064 , piece_id=H4PP7HPHXRX2LOZ4776AMWAJDN7OC435E2K4BXI7XOZYKTB4YUNA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:14.829477074 (591h12m14.700560109s) [noise=true , address=185.167.204.205:4017 , piece_id=CZP65KETMGOHFDMV7KENRL4V36UHBPWRDCBBPUM5NE5L2PV4AQCQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:14.829774972 (591h12m14.700858007s) [noise=true , address=185.167.204.206:4043 , piece_id=G7VZYID4TXGNPDN63NQULXOVSX42WMFFNPBX6KQJ5L2WSW4D2YGQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:14.831734493 (591h12m14.702817528s) [noise=true , address=185.167.204.206:4049 , piece_id=BQ7JEGSGPIFJUGZFBYRPDMBNPZ24BG6TEG4VUM4JUBTE4T2RF4SA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:15.405688561 (591h12m15.276771606s) [noise=true , address=81.171.14.70:4027 , piece_id=6WEORWX5QOMFSJUEADIIBHHIPKNFP4NJPZAJADBUTPUSG7MZJMOQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:15.417110440 (591h12m15.288193485s) [noise=true , address=81.171.14.70:4031 , piece_id=U6Z4SNK32ZSWQ7NDR7ACXGRC4BADDOPR65ILBR64QLGVNDBADFFQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:15.425600446 (591h12m15.296683481s) [noise=true , address=81.171.14.70:4037 , piece_id=6A25LEIAQEJB3DNCTXQT6K7ZSKIGOW4QUGMNKLOG3CT2XXH25PSA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:15.499056228 (591h12m15.370139263s) [noise=true , address=81.171.14.70:4038 , piece_id=3WRHW3VGXWHOEOV5Q6CLYSF7MUVWS7QHTNHLSTCKAUERPHAH3I3Q, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:15.514998925 (591h12m15.38608197s) [noise=true , address=81.171.14.70:4041 , piece_id=NORNA6WFTGPNOHU7MPTNNB5ONTBS5DJGCN2NWIZAEPYSUSIOSGEQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:15.528579544 (591h12m15.399662589s) [noise=true , address=81.171.14.70:4030 , piece_id=KITOJIT34CHS2QZBXCW2SJPZB4RHEHCMJ6IYQ4Z5KLL24YAWBIUQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:15.528991398 (591h12m15.400074434s) [noise=true , address=81.171.14.70:4047 , piece_id=ZWV5DP4YC3MSYDWEQH4ZK3VR4WZ4DUTI4VL3WVLQKFIEKIW44LOQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:15.534844544 (591h12m15.405927579s) [noise=true , address=81.171.14.70:4035 , piece_id=XCLL5SOAPJSSGF5YYQH6ZKGOHMLS6RI6H26FOW5XIJRY3TWYNEGA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:15.915614910 (591h12m15.786697945s) [noise=true , address=81.171.14.70:4039 , piece_id=DEJDXFDHOSAMIP2FN5LJBJNBB4QLJSE2AUDW6IMA6IC4VRY7VAGQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:15.967820176 (591h12m15.838903211s) [noise=true , address=81.171.14.70:4046 , piece_id=B2MTUNKMUP5S7XW4SVQ5QI2ZO3VFB5QVAKBBY4BIZIUILM6O7VYQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:16.172116128 (591h12m16.043199163s) [noise=true , address=89.149.201.147:4007 , piece_id=HZLSYGSBP4TJH5V5XF3Q3AETJU625ERNDBD3VXW2HUZTTPAHLBMA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:16.172137559 (591h12m16.043220604s) [seg_pos={0 0}, upload=518292]: Segment reached optimal threshold of 26 pieces. Dec 30 06:41:16.172336949 (591h12m16.043419994s) [noise=true , address=89.149.201.147:4008 , piece_id=XYFRNJVIFFZA5S36KFPGZPVCPLRL2OW5QBJD3UR37VPPA2RX3RJA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QN8VP6o7qGw9Dd8EafU9xe7s7Cuf1FyqPMbnRaznrfPxfEUUn): protocol: expected piece hash; context canceled; EOF Dec 30 06:41:16.172344213 (591h12m16.043427258s) [noise=true , address=89.149.201.147:4008 , piece_id=XYFRNJVIFFZA5S36KFPGZPVCPLRL2OW5QBJD3UR37VPPA2RX3RJA, seg_pos={0 0}, upload=518292]: Long tail cancellation due to optimal threshold Dec 30 06:41:16.172554002 (591h12m16.043637037s) [noise=true , address=89.149.201.147:4020 , piece_id=N7G5DGEDYVKBTO5S6BYW3IYEOK7HAMK5JQIBMVPMPYRDHXODWEGA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YeKHJH7CoouLFJYENJhzVLNkA32hJaqjLz3dXhr78eaSGtcic): protocol: expected piece hash; context canceled; EOF Dec 30 06:41:16.172565233 (591h12m16.043648278s) [noise=true , address=89.149.201.147:4020 , piece_id=N7G5DGEDYVKBTO5S6BYW3IYEOK7HAMK5JQIBMVPMPYRDHXODWEGA, seg_pos={0 0}, upload=518292]: Long tail cancellation due to optimal threshold Dec 30 06:41:16.172587506 (591h12m16.043670541s) [seg_pos={0 0}, upload=518292]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 30 06:41:16.172717945 (591h12m16.04380098s) [upload=518292]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 06:41:16.172724567 (591h12m16.043807602s) [upload=518292]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 06:41:16.404092237 (591h12m16.275175292s) [upload=518292]: Done uploading segments... Dec 30 06:41:16.404119940 (591h12m16.275202975s) [upload=518292]: Upload finished. err: <nil> Duration: 8.417614766s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 20 out of 60 is failed (threshold=6) [.X.....X.....X....XXX...XXXX.........X.X.....X..XXXXXX.X....] duration: 3.495444012s trace: https://jaeger.dev.storj.io/trace/14547861cf082dcb FAIL (2025-12-30 07:20:29 UTC) orbiter warning: context deadline exceeded Duration: 3.495455283s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 37.610219958s trace: https://jaeger.dev.storj.io/trace/1d0a2f93cb06aaa4 FAIL (2025-12-30 07:20:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== loading piece... Dec 30 07:06:00.467281687 (591h37m0.338364722s) [noise=true , address=77.90.31.14:4693 , piece_id=CQ6A6DJJDCKQAAMCXW7FELWCWKAOI7YCCWKVDSTZEBXNHVKQRKPQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.467716555 (591h37m0.33879959s) [noise=true , address=145.79.227.40:4082 , piece_id=7UFAIMVNINXXXRPNXLGJ34YEO3ILVN2O7UFG3HHTIT2DKWVZSUPA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468044320 (591h37m0.339127365s) [noise=true , address=145.79.227.29:4955 , piece_id=CBV7RUAZ6QAZE3VDTZKOGTIMOY66VUJTSYQSZRC7WLUD53UWVJTQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468308362 (591h37m0.339391407s) [noise=true , address=145.79.227.17:4000 , piece_id=KN3NDKB4NODHNXUWWQQ4YBWBPJCNE7YHYEMLQ4UITRCFZB4BRSMA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468565703 (591h37m0.339648748s) [noise=true , address=145.79.227.46:4604 , piece_id=NY5A6JZW4F6E4CSX3ERMKSAL2R3IYSKQ2EVHZSPF6ZOHWQFXBYWA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.468843542 (591h37m0.339926587s) [noise=true , address=77.90.31.17:4781 , piece_id=TFUDJVQZFLS2XTZEGBPTV23WYLNHZMGK5ZXNDJSMW5HSNZXWXBBA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.469105310 (591h37m0.340188345s) [noise=true , address=145.79.227.35:4494 , piece_id=U4P6QB4ZRFGAJOHVSSTZHCG4PTBGII4LXAAJXXVN5RVR4WUWUBBQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.470034750 (591h37m0.341117795s) [noise=true , address=145.79.227.49:4365 , piece_id=Z357V6BZEGQOFLFH32RQLYJJZDLURVHZAHKW3H3ZIJXMQC3ZIFBQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.470487622 (591h37m0.341570657s) [noise=true , address=64.49.13.169:4713 , piece_id=YS6FRVS5V3BECH3TJQRJBGB5SSYYGUBK2HMO25PZEUOUSC7ENNZQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.470809334 (591h37m0.341892379s) [noise=true , address=77.90.31.10:4530 , piece_id=LNRM5YW2MZLXHWHGNUMP5I7Y5RRNRESBUA2M3SQKDFEQXZ25RJ3A, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.471118874 (591h37m0.342201919s) [noise=true , address=145.79.227.44:4221 , piece_id=GDKGEN3QCGFIICNST42FA2ZLLTQCHL2QAZ7AMUNPJIGYTSHDMOJQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.471431078 (591h37m0.342514113s) [noise=true , address=45.91.116.4:4963 , piece_id=FUGWNAY33ZGP6J57FIODDQQJ2UH46ABT52AU4Y7Y74Q2AOTCXWFA, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.471741530 (591h37m0.342824575s) [noise=true , address=193.149.164.143:4704 , piece_id=PQQZ6N4A7TZGGX5DVRE6NT6TSDJVQARR22F6MZ5Z6LYXLUKF4V7A, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:00.472073311 (591h37m0.343156356s) [noise=true , address=145.79.227.33:4901 , piece_id=FFTUVNIKVKRAZ5R44V6WLWNKKC5MQYA66RZPSMW6N5EQKY7NSGZQ, seg_pos={0 0}, upload=518641]: Uploading piece... Dec 30 07:06:01.899399345 (591h37m1.77048239s) [upload=518641]: Next returned nil segment Dec 30 07:06:01.899418712 (591h37m1.770501747s) [upload=518641]: Waiting for error group managing segments... Dec 30 07:06:02.323303785 (591h37m2.194386821s) [noise=true , address=145.79.227.46:4604 , piece_id=NY5A6JZW4F6E4CSX3ERMKSAL2R3IYSKQ2EVHZSPF6ZOHWQFXBYWA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.339689637 (591h37m2.210772682s) [noise=true , address=145.79.227.17:4000 , piece_id=KN3NDKB4NODHNXUWWQQ4YBWBPJCNE7YHYEMLQ4UITRCFZB4BRSMA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.346460570 (591h37m2.217543615s) [noise=true , address=145.79.227.29:4955 , piece_id=CBV7RUAZ6QAZE3VDTZKOGTIMOY66VUJTSYQSZRC7WLUD53UWVJTQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.347028050 (591h37m2.218111095s) [noise=true , address=145.79.227.34:4931 , piece_id=TJ4NTHHJQ4VLPBIC7RW42DBFIBI5G7LJWBN53TPSEVAWHANHMTBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.427966770 (591h37m2.299049815s) [noise=true , address=145.79.227.5:4226 , piece_id=QKDFEH6UKIH4QYN6DE3YNH3FZPAYGNYMQSBCYGM7EAW7F4MDA66A, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.446440007 (591h37m2.317523052s) [noise=true , address=145.79.227.35:4494 , piece_id=U4P6QB4ZRFGAJOHVSSTZHCG4PTBGII4LXAAJXXVN5RVR4WUWUBBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.583477598 (591h37m2.454560643s) [noise=true , address=77.90.31.19:4859 , piece_id=XHNZMPC2MF4RALF2FZJY73UYN4XWOYJUL66RFRUXNPT64JZQE5KQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.613112418 (591h37m2.484195463s) [noise=true , address=77.90.31.17:4781 , piece_id=TFUDJVQZFLS2XTZEGBPTV23WYLNHZMGK5ZXNDJSMW5HSNZXWXBBA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.631421591 (591h37m2.502504626s) [noise=true , address=77.90.31.10:4530 , piece_id=LNRM5YW2MZLXHWHGNUMP5I7Y5RRNRESBUA2M3SQKDFEQXZ25RJ3A, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:02.944663583 (591h37m2.815746628s) [noise=true , address=145.79.227.49:4365 , piece_id=Z357V6BZEGQOFLFH32RQLYJJZDLURVHZAHKW3H3ZIJXMQC3ZIFBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.195837408 (591h37m3.066920453s) [noise=true , address=193.149.164.143:4704 , piece_id=PQQZ6N4A7TZGGX5DVRE6NT6TSDJVQARR22F6MZ5Z6LYXLUKF4V7A, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.269365478 (591h37m3.140448523s) [noise=true , address=145.79.227.44:4221 , piece_id=GDKGEN3QCGFIICNST42FA2ZLLTQCHL2QAZ7AMUNPJIGYTSHDMOJQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.272362485 (591h37m3.14344553s) [noise=true , address=145.79.227.30:4778 , piece_id=EVWGEZL3VCMHEM6ALESBLMU65XBE74CKVMFXMSHK2GCXFPI35RBQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.340891504 (591h37m3.211974549s) [noise=true , address=145.79.227.40:4082 , piece_id=7UFAIMVNINXXXRPNXLGJ34YEO3ILVN2O7UFG3HHTIT2DKWVZSUPA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: <nil> Dec 30 07:06:03.340922954 (591h37m3.212005999s) [seg_pos={0 0}, upload=518641]: Segment reached optimal threshold of 14 pieces. Dec 30 07:06:03.341286095 (591h37m3.21236913s) [noise=true , address=77.90.31.14:4693 , piece_id=CQ6A6DJJDCKQAAMCXW7FELWCWKAOI7YCCWKVDSTZEBXNHVKQRKPQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RASrnvpAj4ToD4xNvbma1iKZEe2uHK8WosjfZwDDom8Qd17Ey): protocol: write tcp 188.166.225.170:47364->77.90.31.14:4693: use of closed network connection; write tcp 188.166.225.170:47364->77.90.31.14:4693: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47364->77.90.31.14:4693: use of closed network connection Dec 30 07:06:03.341317094 (591h37m3.212400139s) [noise=true , address=77.90.31.14:4693 , piece_id=CQ6A6DJJDCKQAAMCXW7FELWCWKAOI7YCCWKVDSTZEBXNHVKQRKPQ, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341322114 (591h37m3.212405159s) [noise=true , address=145.79.227.33:4901 , piece_id=FFTUVNIKVKRAZ5R44V6WLWNKKC5MQYA66RZPSMW6N5EQKY7NSGZQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1): protocol: expected piece hash; context canceled; EOF Dec 30 07:06:03.341336261 (591h37m3.212419306s) [noise=true , address=145.79.227.33:4901 , piece_id=FFTUVNIKVKRAZ5R44V6WLWNKKC5MQYA66RZPSMW6N5EQKY7NSGZQ, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341546141 (591h37m3.212629196s) [noise=true , address=45.91.116.4:4963 , piece_id=FUGWNAY33ZGP6J57FIODDQQJ2UH46ABT52AU4Y7Y74Q2AOTCXWFA, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Eki7jYegwvkdixR5pznQBbSeJPn9GjGofxnUApidQF4kRNBK5): protocol: expected piece hash; write tcp 188.166.225.170:51728->45.91.116.4:4963: use of closed network connection; write tcp 188.166.225.170:51728->45.91.116.4:4963: use of closed network connection; EOF Dec 30 07:06:03.341555549 (591h37m3.212638594s) [noise=true , address=45.91.116.4:4963 , piece_id=FUGWNAY33ZGP6J57FIODDQQJ2UH46ABT52AU4Y7Y74Q2AOTCXWFA, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341759577 (591h37m3.212842622s) [noise=true , address=64.49.13.169:4713 , piece_id=YS6FRVS5V3BECH3TJQRJBGB5SSYYGUBK2HMO25PZEUOUSC7ENNZQ, seg_pos={0 0}, upload=518641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17z3AiUfbxBX2eMDnUnqo1bbgojuQxJro7PsEaPjYzQjHQ3D6K): protocol: write tcp 188.166.225.170:42002->64.49.13.169:4713: use of closed network connection; write tcp 188.166.225.170:42002->64.49.13.169:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42002->64.49.13.169:4713: use of closed network connection Dec 30 07:06:03.341769897 (591h37m3.212852942s) [noise=true , address=64.49.13.169:4713 , piece_id=YS6FRVS5V3BECH3TJQRJBGB5SSYYGUBK2HMO25PZEUOUSC7ENNZQ, seg_pos={0 0}, upload=518641]: Long tail cancellation due to optimal threshold Dec 30 07:06:03.341869927 (591h37m3.212952972s) [seg_pos={0 0}, upload=518641]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 07:06:03.342018730 (591h37m3.213101775s) [upload=518641]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 07:06:03.342025634 (591h37m3.213108679s) [upload=518641]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 07:06:03.635315934 (591h37m3.506398979s) [upload=518641]: Done uploading segments... Dec 30 07:06:03.635343146 (591h37m3.506426191s) [upload=518641]: Upload finished. err: <nil> Duration: 37.610240406s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 20 out of 60 is failed (threshold=6) [..X.....X.....X....XXX...XXXX.........X.X.....X..XXXXXX.X...] duration: 3.636513s trace: https://jaeger.dev.storj.io/trace/14547861cf082dcb FAIL (2025-12-30 07:19:41 UTC) orbiter warning: context deadline exceeded Duration: 3.636536565s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 6.544049575s trace: https://jaeger.dev.storj.io/trace/4c2f4efaff0e0ea0 FAIL (2025-12-30 07:19:38 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== load=518292]: Uploading piece... Dec 30 06:41:13.598100152 (591h12m13.469183187s) [noise=true , address=81.171.14.70:4041 , piece_id=NORNA6WFTGPNOHU7MPTNNB5ONTBS5DJGCN2NWIZAEPYSUSIOSGEQ, seg_pos={0 0}, upload=518292]: Uploading piece... Dec 30 06:41:13.598587971 (591h12m13.469671016s) [noise=true , address=89.149.201.147:4007 , piece_id=HZLSYGSBP4TJH5V5XF3Q3AETJU625ERNDBD3VXW2HUZTTPAHLBMA, seg_pos={0 0}, upload=518292]: Uploading piece... Dec 30 06:41:13.599093143 (591h12m13.470176178s) [noise=true , address=89.149.201.147:4008 , piece_id=XYFRNJVIFFZA5S36KFPGZPVCPLRL2OW5QBJD3UR37VPPA2RX3RJA, seg_pos={0 0}, upload=518292]: Uploading piece... Dec 30 06:41:13.599699999 (591h12m13.470783034s) [noise=true , address=81.171.14.70:4039 , piece_id=DEJDXFDHOSAMIP2FN5LJBJNBB4QLJSE2AUDW6IMA6IC4VRY7VAGQ, seg_pos={0 0}, upload=518292]: Uploading piece... Dec 30 06:41:13.600136169 (591h12m13.471219204s) [noise=true , address=185.167.204.205:4034 , piece_id=OWZHQB5ZDT2QZ2QAHZJVRWQ222MX6CWLKJUOIAAGSMAVLWGJAOBA, seg_pos={0 0}, upload=518292]: Uploading piece... Dec 30 06:41:13.600543184 (591h12m13.471626229s) [noise=true , address=185.167.204.206:4058 , piece_id=ROOCGK64M5RHIWNAVCSZNYTZ6M2ORWB7MJ27XJVDTFVQIKJZHY4A, seg_pos={0 0}, upload=518292]: Uploading piece... Dec 30 06:41:13.601048286 (591h12m13.472131332s) [noise=true , address=81.171.14.70:4046 , piece_id=B2MTUNKMUP5S7XW4SVQ5QI2ZO3VFB5QVAKBBY4BIZIUILM6O7VYQ, seg_pos={0 0}, upload=518292]: Uploading piece... Dec 30 06:41:13.601698784 (591h12m13.472781819s) [noise=true , address=185.167.204.206:4064 , piece_id=H4PP7HPHXRX2LOZ4776AMWAJDN7OC435E2K4BXI7XOZYKTB4YUNA, seg_pos={0 0}, upload=518292]: Uploading piece... Dec 30 06:41:13.602168259 (591h12m13.473251304s) [noise=true , address=185.167.204.206:4065 , piece_id=LD3J4656Q5QEUCMJ5OKEYATEVRIKSXO7XGARTN25BVM7FDIYOWKA, seg_pos={0 0}, upload=518292]: Uploading piece... Dec 30 06:41:13.602782708 (591h12m13.473865753s) [noise=true , address=81.171.14.70:4027 , piece_id=6WEORWX5QOMFSJUEADIIBHHIPKNFP4NJPZAJADBUTPUSG7MZJMOQ, seg_pos={0 0}, upload=518292]: Uploading piece... Dec 30 06:41:14.423395002 (591h12m14.294478047s) [upload=518292]: Next returned nil segment Dec 30 06:41:14.423414068 (591h12m14.294497103s) [upload=518292]: Waiting for error group managing segments... Dec 30 06:41:14.817650316 (591h12m14.688733351s) [noise=true , address=185.167.204.205:4029 , piece_id=KUOMMYJ2F4NIK6VN4C5M3Y3TMHYEZH7QFNJ6K3CLOAVI5R6VKTHQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:14.817870054 (591h12m14.688953099s) [noise=true , address=185.167.204.205:4019 , piece_id=H6AH654VGX6NHBUKODIWGL7I2RKE3DDMY4UM6VDONP2EP3NPDJBA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:14.818623698 (591h12m14.689706744s) [noise=true , address=185.167.204.206:4065 , piece_id=LD3J4656Q5QEUCMJ5OKEYATEVRIKSXO7XGARTN25BVM7FDIYOWKA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:14.818932436 (591h12m14.690015481s) [noise=true , address=185.167.204.206:4061 , piece_id=T2XS4IFPQSVS3T45SPQ5ZDDVT2VMTMXHKL2ZMX4OIOPRR67SCD2Q, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:14.819147046 (591h12m14.690230091s) [noise=true , address=185.167.204.206:4071 , piece_id=QI2TOY5KCDZVWCCLMAJXVUWD2HB7S7IOTEAV2EHAMUNHKOFKTNYA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:14.819678578 (591h12m14.690761623s) [noise=true , address=185.167.204.206:4059 , piece_id=LMOGTWX2YIRR3EKNROK43O2B27S5L5U6GYB7HBVRBPQPH4WDDWZQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:14.819905028 (591h12m14.690988063s) [noise=true , address=185.167.204.206:4058 , piece_id=ROOCGK64M5RHIWNAVCSZNYTZ6M2ORWB7MJ27XJVDTFVQIKJZHY4A, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:14.820479312 (591h12m14.691562357s) [noise=true , address=185.167.204.205:4021 , piece_id=KP7J5XSY4SKSKZG7XXFZMNKMRP4472PXI7CW2UQBZICN4MAFO5RQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:14.823149296 (591h12m14.694232331s) [noise=true , address=185.167.204.205:4034 , piece_id=OWZHQB5ZDT2QZ2QAHZJVRWQ222MX6CWLKJUOIAAGSMAVLWGJAOBA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:14.827906996 (591h12m14.698990041s) [noise=true , address=185.167.204.206:4055 , piece_id=VYEDR6XHZBZMTVJ5U5OF67WZOGVMCMQLXXBQUI7DHBCCVQ2RRERQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:14.828003769 (591h12m14.699086814s) [noise=true , address=185.167.204.205:4016 , piece_id=I6MUDLAXUUDJL2C4HFE22C2L56A7D6HSCNWXKWJ7OKO2N4RA4W7Q, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:14.828893203 (591h12m14.699976238s) [noise=true , address=185.167.204.206:4064 , piece_id=H4PP7HPHXRX2LOZ4776AMWAJDN7OC435E2K4BXI7XOZYKTB4YUNA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:14.829477074 (591h12m14.700560109s) [noise=true , address=185.167.204.205:4017 , piece_id=CZP65KETMGOHFDMV7KENRL4V36UHBPWRDCBBPUM5NE5L2PV4AQCQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:14.829774972 (591h12m14.700858007s) [noise=true , address=185.167.204.206:4043 , piece_id=G7VZYID4TXGNPDN63NQULXOVSX42WMFFNPBX6KQJ5L2WSW4D2YGQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:14.831734493 (591h12m14.702817528s) [noise=true , address=185.167.204.206:4049 , piece_id=BQ7JEGSGPIFJUGZFBYRPDMBNPZ24BG6TEG4VUM4JUBTE4T2RF4SA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:15.405688561 (591h12m15.276771606s) [noise=true , address=81.171.14.70:4027 , piece_id=6WEORWX5QOMFSJUEADIIBHHIPKNFP4NJPZAJADBUTPUSG7MZJMOQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:15.417110440 (591h12m15.288193485s) [noise=true , address=81.171.14.70:4031 , piece_id=U6Z4SNK32ZSWQ7NDR7ACXGRC4BADDOPR65ILBR64QLGVNDBADFFQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:15.425600446 (591h12m15.296683481s) [noise=true , address=81.171.14.70:4037 , piece_id=6A25LEIAQEJB3DNCTXQT6K7ZSKIGOW4QUGMNKLOG3CT2XXH25PSA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:15.499056228 (591h12m15.370139263s) [noise=true , address=81.171.14.70:4038 , piece_id=3WRHW3VGXWHOEOV5Q6CLYSF7MUVWS7QHTNHLSTCKAUERPHAH3I3Q, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:15.514998925 (591h12m15.38608197s) [noise=true , address=81.171.14.70:4041 , piece_id=NORNA6WFTGPNOHU7MPTNNB5ONTBS5DJGCN2NWIZAEPYSUSIOSGEQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:15.528579544 (591h12m15.399662589s) [noise=true , address=81.171.14.70:4030 , piece_id=KITOJIT34CHS2QZBXCW2SJPZB4RHEHCMJ6IYQ4Z5KLL24YAWBIUQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:15.528991398 (591h12m15.400074434s) [noise=true , address=81.171.14.70:4047 , piece_id=ZWV5DP4YC3MSYDWEQH4ZK3VR4WZ4DUTI4VL3WVLQKFIEKIW44LOQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:15.534844544 (591h12m15.405927579s) [noise=true , address=81.171.14.70:4035 , piece_id=XCLL5SOAPJSSGF5YYQH6ZKGOHMLS6RI6H26FOW5XIJRY3TWYNEGA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:15.915614910 (591h12m15.786697945s) [noise=true , address=81.171.14.70:4039 , piece_id=DEJDXFDHOSAMIP2FN5LJBJNBB4QLJSE2AUDW6IMA6IC4VRY7VAGQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:15.967820176 (591h12m15.838903211s) [noise=true , address=81.171.14.70:4046 , piece_id=B2MTUNKMUP5S7XW4SVQ5QI2ZO3VFB5QVAKBBY4BIZIUILM6O7VYQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:16.172116128 (591h12m16.043199163s) [noise=true , address=89.149.201.147:4007 , piece_id=HZLSYGSBP4TJH5V5XF3Q3AETJU625ERNDBD3VXW2HUZTTPAHLBMA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil> Dec 30 06:41:16.172137559 (591h12m16.043220604s) [seg_pos={0 0}, upload=518292]: Segment reached optimal threshold of 26 pieces. Dec 30 06:41:16.172336949 (591h12m16.043419994s) [noise=true , address=89.149.201.147:4008 , piece_id=XYFRNJVIFFZA5S36KFPGZPVCPLRL2OW5QBJD3UR37VPPA2RX3RJA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QN8VP6o7qGw9Dd8EafU9xe7s7Cuf1FyqPMbnRaznrfPxfEUUn): protocol: expected piece hash; context canceled; EOF Dec 30 06:41:16.172344213 (591h12m16.043427258s) [noise=true , address=89.149.201.147:4008 , piece_id=XYFRNJVIFFZA5S36KFPGZPVCPLRL2OW5QBJD3UR37VPPA2RX3RJA, seg_pos={0 0}, upload=518292]: Long tail cancellation due to optimal threshold Dec 30 06:41:16.172554002 (591h12m16.043637037s) [noise=true , address=89.149.201.147:4020 , piece_id=N7G5DGEDYVKBTO5S6BYW3IYEOK7HAMK5JQIBMVPMPYRDHXODWEGA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YeKHJH7CoouLFJYENJhzVLNkA32hJaqjLz3dXhr78eaSGtcic): protocol: expected piece hash; context canceled; EOF Dec 30 06:41:16.172565233 (591h12m16.043648278s) [noise=true , address=89.149.201.147:4020 , piece_id=N7G5DGEDYVKBTO5S6BYW3IYEOK7HAMK5JQIBMVPMPYRDHXODWEGA, seg_pos={0 0}, upload=518292]: Long tail cancellation due to optimal threshold Dec 30 06:41:16.172587506 (591h12m16.043670541s) [seg_pos={0 0}, upload=518292]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 30 06:41:16.172717945 (591h12m16.04380098s) [upload=518292]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 06:41:16.172724567 (591h12m16.043807602s) [upload=518292]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 06:41:16.404092237 (591h12m16.275175292s) [upload=518292]: Done uploading segments... Dec 30 06:41:16.404119940 (591h12m16.275202975s) [upload=518292]: Upload finished. err: <nil> Duration: 6.54407881s core uplink eu-select-1 eu1-placement-15

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)