Orbiter History

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 8.481645391s trace: https://jaeger.dev.storj.io/trace/718f722b1bda7fe2 FAIL (2025-12-22 15:00:34 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== VMAC4CKPNLATBLZLKAGDGZHVT7BHEPN3Q, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.367093115 (407h9m14.23817616s) [noise=true , address=185.167.204.206:4043 , piece_id=HL32YEZF67X2UT6SRMXJVJ4W5A3PC5SXOCM7BYQVMEC76TKEVJRA, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.367396732 (407h9m14.238479767s) [noise=true , address=185.167.204.205:4032 , piece_id=FJMXJG2PQ4WZDSDCPJ6LLSVQKI7JJHUCDUZLU2BJZV7RZTLETM5A, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.367549273 (407h9m14.238632308s) [noise=true , address=185.167.204.206:4070 , piece_id=Y7IRY4E2GCF7TCFOIQJ7QH6IFEJBMW4WM6QBIS5SCUJ7E7ASV4HA, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.367749263 (407h9m14.238832308s) [noise=true , address=81.171.14.70:4046 , piece_id=NRMV6FG5CZSTLUNP2EHNCK6VM2Z2UHKBLCHLKEPQ225Q5DGYEBPA, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.368034046 (407h9m14.239117091s) [noise=true , address=185.167.204.206:4060 , piece_id=WN6T5TUBVMURPSBBXGXCTLPZZ74PFNSRTTWXI2JPUTY74DPH3YSQ, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.368552724 (407h9m14.239635759s) [noise=true , address=81.171.14.70:4029 , piece_id=APDMOYEMDCLOREXY2BBOK73RSQVWKIXUX4SCHRVFDSM7ZLWBEQXA, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.369160691 (407h9m14.240243736s) [noise=true , address=185.167.204.206:4057 , piece_id=ONW7YS5JG44B2STNFJGJWFEZ3WYXO74N3CXI3C6T5OTJYHGQPQZQ, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.369714496 (407h9m14.240797541s) [noise=true , address=185.167.204.206:4071 , piece_id=DVJAKIMPAUGWQPHDPGTXASQDONAKOZIGX5PQDBGBXNQAPMBPUFBA, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.370249614 (407h9m14.241332649s) [noise=true , address=89.149.201.147:4019 , piece_id=ILE7ZX5X6HBPUVYGCOQ5SNG3AA6B3YTXIF47CPLJ3MN5EXVOS2OA, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.372019455 (407h9m14.24310249s) [noise=true , address=89.149.201.147:4015 , piece_id=ZIWRTD7HSP7YMRJET2TTJQFAQZM7KGDKJDLMYCWWY2W6SV2UVQBQ, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:15.199398795 (407h9m15.07048184s) [upload=357453]: Next returned nil segment Dec 22 14:38:15.199414966 (407h9m15.070498011s) [upload=357453]: Waiting for error group managing segments... Dec 22 14:38:15.591187572 (407h9m15.462270607s) [noise=true , address=185.167.204.205:4009 , piece_id=JTHX2O6T5HRENGEMAFJUWQUEUYAIG2KQ5HDEHF4BK42IT2ER7EIA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.591766795 (407h9m15.46284984s) [noise=true , address=185.167.204.205:4006 , piece_id=XHNFS6G7EX4BQWKOK4JUWYV3OTVJ36A6VOP4N7G2XPZKNNCFFKXA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.592074521 (407h9m15.463157566s) [noise=true , address=185.167.204.206:4060 , piece_id=WN6T5TUBVMURPSBBXGXCTLPZZ74PFNSRTTWXI2JPUTY74DPH3YSQ, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.593304694 (407h9m15.464387729s) [noise=true , address=185.167.204.205:4036 , piece_id=5QWJTTL6TZIZSSYHCBEJBKGX7BS76VVTELWTKEAKTOVYKINXVWDQ, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.593638659 (407h9m15.464721694s) [noise=true , address=185.167.204.205:4004 , piece_id=PYGDQZMAKOVJWB7AWEPFMPMB6C24FMS4HT2UP2AKZNYGG4DGZR3Q, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.593818251 (407h9m15.464901286s) [noise=true , address=185.167.204.206:4048 , piece_id=52WC3XH2K2BHG33BQPRENPNYDT7EOKCNBCTZKE3LIONDJHVATEYA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.594252549 (407h9m15.465335594s) [noise=true , address=185.167.204.205:4034 , piece_id=OS5H57TG2REFE535DK4T6N75VNIXPLQWW6DM55QCC2S7QBVZM3UQ, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.597113216 (407h9m15.468196261s) [noise=true , address=185.167.204.205:4032 , piece_id=FJMXJG2PQ4WZDSDCPJ6LLSVQKI7JJHUCDUZLU2BJZV7RZTLETM5A, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.597394541 (407h9m15.468477586s) [noise=true , address=185.167.204.205:4035 , piece_id=3D7T4SGPAJQZCA74CUTR4GJRE23R4JD5KBNW3BLEASSP4GSR5PHQ, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.597612756 (407h9m15.468695792s) [noise=true , address=185.167.204.206:4043 , piece_id=HL32YEZF67X2UT6SRMXJVJ4W5A3PC5SXOCM7BYQVMEC76TKEVJRA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.598018900 (407h9m15.469101945s) [noise=true , address=185.167.204.205:4012 , piece_id=A5G37BVL2GZ7LAY3FZWIZV4R4M7M6UPYDYACJFRD5HKDWWKB4Q6A, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.598254719 (407h9m15.469337765s) [noise=true , address=185.167.204.206:4052 , piece_id=V2DVAAXN5A4C3NQW556VMAC4CKPNLATBLZLKAGDGZHVT7BHEPN3Q, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.598480649 (407h9m15.469563694s) [noise=true , address=185.167.204.205:4018 , piece_id=NUYRZ7PXKJQ3PIR44O3E6N6COZZPRDNSMTQCFI5RN7FBRV2CBALA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.598968278 (407h9m15.470051323s) [noise=true , address=185.167.204.205:4033 , piece_id=A32ZTBDBWMCNBMZPD4DUWS5NKUKQ22ACQH6HAW5AKB3FNHBC56VA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.599287886 (407h9m15.470370931s) [noise=true , address=185.167.204.206:4072 , piece_id=I6PCSC5JYTJXLT6B34TTIVZQVX52R7TRHSRLRP4ZHG5J63XDZN6Q, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.599632252 (407h9m15.470715297s) [noise=true , address=185.167.204.206:4070 , piece_id=Y7IRY4E2GCF7TCFOIQJ7QH6IFEJBMW4WM6QBIS5SCUJ7E7ASV4HA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.599676297 (407h9m15.470759332s) [noise=true , address=185.167.204.206:4071 , piece_id=DVJAKIMPAUGWQPHDPGTXASQDONAKOZIGX5PQDBGBXNQAPMBPUFBA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.601435877 (407h9m15.472518912s) [noise=true , address=185.167.204.206:4057 , piece_id=ONW7YS5JG44B2STNFJGJWFEZ3WYXO74N3CXI3C6T5OTJYHGQPQZQ, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:17.299689191 (407h9m17.170772226s) [noise=true , address=89.149.201.147:4019 , piece_id=ILE7ZX5X6HBPUVYGCOQ5SNG3AA6B3YTXIF47CPLJ3MN5EXVOS2OA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:17.526601953 (407h9m17.397684998s) [noise=true , address=89.149.201.147:4015 , piece_id=ZIWRTD7HSP7YMRJET2TTJQFAQZM7KGDKJDLMYCWWY2W6SV2UVQBQ, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:18.294911214 (407h9m18.165994249s) [noise=true , address=89.149.201.147:4017 , piece_id=ZOSNSMZOBQVIKD2ZGJRAUWJEJ4JXJK452YWQQSDL77E77F6LRDZA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:18.402191355 (407h9m18.2732744s) [noise=true , address=89.149.201.147:4011 , piece_id=ATMM77IGBDSRHNJ5FQ2VKP7W27JUA2F6HYJYSQ62IL5JQN5PF5OA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:22.436508909 (407h9m22.307591944s) [noise=true , address=81.171.14.70:4037 , piece_id=NYGC2ZLUSBUAGFULQ46LXCVNQUJ5M4XT3YICVA2ZMBR2D6WPJ27Q, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:25.935108368 (407h9m25.806191413s) [noise=true , address=81.171.14.70:4046 , piece_id=NRMV6FG5CZSTLUNP2EHNCK6VM2Z2UHKBLCHLKEPQ225Q5DGYEBPA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:30.904793996 (407h9m30.775877041s) [noise=true , address=81.171.14.70:4029 , piece_id=APDMOYEMDCLOREXY2BBOK73RSQVWKIXUX4SCHRVFDSM7ZLWBEQXA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:32.926010512 (407h9m32.797093567s) [noise=true , address=81.171.14.70:4044 , piece_id=QCWQO6JQRRKSG5U2KVH3Y6I3UI2F7G3DOW4CH3PQZKUVQ6Q2XOHA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:32.926036552 (407h9m32.797119587s) [seg_pos={0 0}, upload=357453]: Segment reached optimal threshold of 26 pieces. Dec 22 14:38:32.926196857 (407h9m32.797279902s) [noise=true , address=81.171.14.70:4033 , piece_id=MKNAJOFD5CIHTA55XO476LOJJGIWSA7HBQFQND2YNM562W74IKJA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BnjusiXEfHYwTDL8soMkm8GYnhbkwy2oCAxVA7oiDyhFXozZ8): protocol: expected piece hash; context canceled; EOF Dec 22 14:38:32.926203058 (407h9m32.797286093s) [noise=true , address=81.171.14.70:4033 , piece_id=MKNAJOFD5CIHTA55XO476LOJJGIWSA7HBQFQND2YNM562W74IKJA, seg_pos={0 0}, upload=357453]: Long tail cancellation due to optimal threshold Dec 22 14:38:32.926237574 (407h9m32.797320609s) [noise=true , address=81.171.14.70:4031 , piece_id=7MD5N7PPKFPCFMOIXAGI3MPBDDX45GTLMA3CFGNM7BBGI6TSL6RA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12drdYnnWsuMyFo6LkcewAGGWnorRCAhxPpNqK41LvGy2zkBvng): protocol: expected piece hash; context canceled; EOF Dec 22 14:38:32.926250028 (407h9m32.797333063s) [noise=true , address=81.171.14.70:4031 , piece_id=7MD5N7PPKFPCFMOIXAGI3MPBDDX45GTLMA3CFGNM7BBGI6TSL6RA, seg_pos={0 0}, upload=357453]: Long tail cancellation due to optimal threshold Dec 22 14:38:32.926299612 (407h9m32.797382647s) [seg_pos={0 0}, upload=357453]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 22 14:38:32.926403941 (407h9m32.797486986s) [upload=357453]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 14:38:32.926409692 (407h9m32.797492737s) [upload=357453]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 14:39:04.793379844 (407h10m4.664462889s) [upload=357453]: Done uploading segments... Dec 22 14:39:04.793413608 (407h10m4.664496643s) [upload=357453]: Upload finished. err: <nil> Duration: 8.481673366s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........X........................X..........................] duration: 6.390506714s trace: https://jaeger.dev.storj.io/trace/510cfc1b6c68e873 FAIL (2025-12-22 15:00:18 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 6.390536871s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 1.492941404s trace: https://jaeger.dev.storj.io/trace/62b55ec615ab6123 FAIL (2025-12-22 15:00:17 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.492974386s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...X........................................................] duration: 3.977140156s trace: https://jaeger.dev.storj.io/trace/1c84c110f15266bb FAIL (2025-12-22 15:00:11 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.977382777s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [X.....................................X.....XX.....XX.XXXXX.] duration: 15.057871597s trace: https://jaeger.dev.storj.io/trace/64d5998b4085076f FAIL (2025-12-22 14:59:48 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 04866578s) [upload=357761]: Uploading segments... Dec 22 14:59:03.533960140 (407h30m3.405043185s) [upload=357761]: Got next segment. Inline: false Dec 22 14:59:03.534029151 (407h30m3.405112196s) [upload=357761]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 14:59:03.534039550 (407h30m3.405122596s) [upload=357761]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 14:59:03.784341119 (407h30m3.655424154s) [seg_pos={0 0}, upload=357761]: Begin upload segment... Dec 22 14:59:03.784408697 (407h30m3.655491732s) [seg_pos={0 0}, upload=357761]: Done begin upload segment. Dec 22 14:59:03.784430288 (407h30m3.655513333s) [noise=true , address=145.79.227.46:4603 , piece_id=WNW2PJCFNITKKYDHQLRO233C5IYTOQ6ET2NXN5MJBRMOMV5OZC3Q, seg_pos={0 0}, upload=357761]: Uploading piece... Dec 22 14:59:03.784602646 (407h30m3.655685691s) [noise=true , address=45.91.116.13:4156 , piece_id=J2RDO4SWP6G5GDCOK4255GDBWPOGO6OZZDAUNPEDHC625Q6MF4FA, seg_pos={0 0}, upload=357761]: Uploading piece... Dec 22 14:59:03.784841210 (407h30m3.655924245s) [noise=true , address=145.79.227.47:4267 , piece_id=HYVGAK4WVG3EUHWRP6EENWJLC4HLGAPVFKXZFHCB5O7ERFTLCKXQ, seg_pos={0 0}, upload=357761]: Uploading piece... Dec 22 14:59:03.785300545 (407h30m3.65638359s) [noise=true , address=145.79.227.32:4834 , piece_id=RPB5L4CGS3QG5KKYCZVYLLHHKDCWY2HWREOLDHCINX4ES3YCY4MA, seg_pos={0 0}, upload=357761]: Uploading piece... Dec 22 14:59:03.785695958 (407h30m3.656779004s) [noise=true , address=145.79.227.51:4411 , piece_id=WQCVZGIHJ7P5GXXJKEONTIOCJF6XHAS6SORDWQP45XX4LIZT55HA, seg_pos={0 0}, upload=357761]: Uploading piece... Dec 22 14:59:03.786218303 (407h30m3.657301348s) [noise=true , address=145.79.227.37:4566 , piece_id=BTMRQTCB3O6LLSWFCXUBLCRDG4AB3JATJ6IDTPNYMSJW2GVG5S6Q, seg_pos={0 0}, upload=357761]: Uploading piece... Dec 22 14:59:03.786649133 (407h30m3.657732178s) [noise=true , address=45.91.116.12:4226 , piece_id=TQSAYNCCDP6FJMBXFW3CDQXAUHKHLTRUEHPEYRF6DRIBRW2HI7TQ, seg_pos={0 0}, upload=357761]: Uploading piece... Dec 22 14:59:03.787867483 (407h30m3.658950518s) [noise=true , address=77.90.31.12:4590 , piece_id=GU7B34FTJN3A5BA76Y3QSFRDX3SVUSVFZZJR3K4WSG6V6GJTWXVQ, seg_pos={0 0}, upload=357761]: Uploading piece... Dec 22 14:59:03.788662247 (407h30m3.659745292s) [noise=true , address=145.79.227.52:4446 , piece_id=IHX5HJ6DTFTLXKZNXYRGPAJQHRHS6GKHNM2DV5SGCBYRXDW5AJPA, seg_pos={0 0}, upload=357761]: Uploading piece... Dec 22 14:59:03.788926289 (407h30m3.660009334s) [noise=true , address=145.79.227.36:4518 , piece_id=HJWLGPPWRQWISZDODAK3XZEEP3K3DDYBLLE6Y7M2REJUNXUT7IXQ, seg_pos={0 0}, upload=357761]: Uploading piece... Dec 22 14:59:03.789297045 (407h30m3.66038009s) [noise=true , address=145.79.227.50:4374 , piece_id=NLTMPH6FVBMCZQKO43M3MR2P2YB5QCDLQJ5AKKPY5HITBYU32TIA, seg_pos={0 0}, upload=357761]: Uploading piece... Dec 22 14:59:03.789970738 (407h30m3.661053783s) [noise=true , address=145.79.227.33:4871 , piece_id=7XNFYDVVQ73VBICY2TNZ2OMHTBBYVBYY3VVNEE7TKOKA7QRSNZNA, seg_pos={0 0}, upload=357761]: Uploading piece... Dec 22 14:59:03.790629252 (407h30m3.661712297s) [noise=true , address=145.79.227.5:4223 , piece_id=PCJNSCIQWFYH3ZK4U2YAFRTMBWX63VPZELPB6SUEVOQNKUKW6QIA, seg_pos={0 0}, upload=357761]: Uploading piece... Dec 22 14:59:03.791282937 (407h30m3.662365982s) [noise=true , address=145.79.227.11:4732 , piece_id=MTFAMAYYBIJK6A2PJKZOYGASG37ZDNCZOHV4O7KEEPPTPW5LW4AA, seg_pos={0 0}, upload=357761]: Uploading piece... Dec 22 14:59:03.791831692 (407h30m3.662914747s) [noise=true , address=145.79.227.17:4972 , piece_id=JKUT6AA655S24EBFCQ47FNCDDLJ25OXSWVDKHOARE4XKDXCNKZGQ, seg_pos={0 0}, upload=357761]: Uploading piece... Dec 22 14:59:03.792125310 (407h30m3.663208365s) [noise=true , address=145.79.227.38:4977 , piece_id=L2J7RJZEEL23SI6L7YBSVEVFARV77IWWTRNPYWIFG2NWNQGYTXDQ, seg_pos={0 0}, upload=357761]: Uploading piece... Dec 22 14:59:05.653337738 (407h30m5.524420783s) [upload=357761]: Next returned nil segment Dec 22 14:59:05.653363587 (407h30m5.524446632s) [upload=357761]: Waiting for error group managing segments... Dec 22 14:59:06.060881413 (407h30m5.931964448s) [noise=true , address=77.90.31.12:4590 , piece_id=GU7B34FTJN3A5BA76Y3QSFRDX3SVUSVFZZJR3K4WSG6V6GJTWXVQ, seg_pos={0 0}, upload=357761]: Done uploading piece. err: <nil> Dec 22 14:59:06.063869092 (407h30m5.934952137s) [noise=true , address=45.91.116.13:4156 , piece_id=J2RDO4SWP6G5GDCOK4255GDBWPOGO6OZZDAUNPEDHC625Q6MF4FA, seg_pos={0 0}, upload=357761]: Done uploading piece. err: <nil> Dec 22 14:59:06.263490436 (407h30m6.134573481s) [noise=true , address=45.91.116.12:4226 , piece_id=TQSAYNCCDP6FJMBXFW3CDQXAUHKHLTRUEHPEYRF6DRIBRW2HI7TQ, seg_pos={0 0}, upload=357761]: Done uploading piece. err: <nil> Dec 22 14:59:06.675418780 (407h30m6.546501815s) [noise=true , address=145.79.227.47:4267 , piece_id=HYVGAK4WVG3EUHWRP6EENWJLC4HLGAPVFKXZFHCB5O7ERFTLCKXQ, seg_pos={0 0}, upload=357761]: Done uploading piece. err: <nil> Dec 22 14:59:06.677670729 (407h30m6.548753774s) [noise=true , address=145.79.227.51:4411 , piece_id=WQCVZGIHJ7P5GXXJKEONTIOCJF6XHAS6SORDWQP45XX4LIZT55HA, seg_pos={0 0}, upload=357761]: Done uploading piece. err: <nil> Dec 22 14:59:06.690093824 (407h30m6.561176869s) [noise=true , address=145.79.227.37:4566 , piece_id=BTMRQTCB3O6LLSWFCXUBLCRDG4AB3JATJ6IDTPNYMSJW2GVG5S6Q, seg_pos={0 0}, upload=357761]: Done uploading piece. err: <nil> Dec 22 14:59:06.695236277 (407h30m6.566319322s) [noise=true , address=145.79.227.5:4223 , piece_id=PCJNSCIQWFYH3ZK4U2YAFRTMBWX63VPZELPB6SUEVOQNKUKW6QIA, seg_pos={0 0}, upload=357761]: Done uploading piece. err: <nil> Dec 22 14:59:06.750737611 (407h30m6.621820646s) [noise=true , address=145.79.227.38:4977 , piece_id=L2J7RJZEEL23SI6L7YBSVEVFARV77IWWTRNPYWIFG2NWNQGYTXDQ, seg_pos={0 0}, upload=357761]: Done uploading piece. err: <nil> Dec 22 14:59:06.761387720 (407h30m6.632470765s) [noise=true , address=145.79.227.11:4732 , piece_id=MTFAMAYYBIJK6A2PJKZOYGASG37ZDNCZOHV4O7KEEPPTPW5LW4AA, seg_pos={0 0}, upload=357761]: Done uploading piece. err: <nil> Dec 22 14:59:06.769328340 (407h30m6.640411386s) [noise=true , address=145.79.227.46:4603 , piece_id=WNW2PJCFNITKKYDHQLRO233C5IYTOQ6ET2NXN5MJBRMOMV5OZC3Q, seg_pos={0 0}, upload=357761]: Done uploading piece. err: <nil> Dec 22 14:59:06.846216479 (407h30m6.717299514s) [noise=true , address=145.79.227.50:4374 , piece_id=NLTMPH6FVBMCZQKO43M3MR2P2YB5QCDLQJ5AKKPY5HITBYU32TIA, seg_pos={0 0}, upload=357761]: Done uploading piece. err: <nil> Dec 22 14:59:06.846584721 (407h30m6.717667756s) [noise=true , address=145.79.227.17:4972 , piece_id=JKUT6AA655S24EBFCQ47FNCDDLJ25OXSWVDKHOARE4XKDXCNKZGQ, seg_pos={0 0}, upload=357761]: Done uploading piece. err: <nil> Dec 22 14:59:07.137793744 (407h30m7.008876789s) [noise=true , address=145.79.227.33:4871 , piece_id=7XNFYDVVQ73VBICY2TNZ2OMHTBBYVBYY3VVNEE7TKOKA7QRSNZNA, seg_pos={0 0}, upload=357761]: Done uploading piece. err: <nil> Dec 22 14:59:07.137859660 (407h30m7.008942695s) [seg_pos={0 0}, upload=357761]: Segment reached optimal threshold of 13 pieces. Dec 22 14:59:07.138352869 (407h30m7.009435914s) [noise=true , address=145.79.227.52:4446 , piece_id=IHX5HJ6DTFTLXKZNXYRGPAJQHRHS6GKHNM2DV5SGCBYRXDW5AJPA, seg_pos={0 0}, upload=357761]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RNJPWzogZpsQfuFzJYZiz8bbHSRSrgSE5FLyhCynSwEDfMid7): protocol: write tcp 188.166.225.170:54006->145.79.227.52:4446: use of closed network connection; write tcp 188.166.225.170:54006->145.79.227.52:4446: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54006->145.79.227.52:4446: use of closed network connection Dec 22 14:59:07.138366365 (407h30m7.0094494s) [noise=true , address=145.79.227.52:4446 , piece_id=IHX5HJ6DTFTLXKZNXYRGPAJQHRHS6GKHNM2DV5SGCBYRXDW5AJPA, seg_pos={0 0}, upload=357761]: Long tail cancellation due to optimal threshold Dec 22 14:59:07.138881896 (407h30m7.009964931s) [noise=true , address=145.79.227.36:4518 , piece_id=HJWLGPPWRQWISZDODAK3XZEEP3K3DDYBLLE6Y7M2REJUNXUT7IXQ, seg_pos={0 0}, upload=357761]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17TZ9JLdCivFF46guhSTW6hcoUmTz5uo5wvLkDqC4BGmpm9eLb): protocol: write tcp 188.166.225.170:38088->145.79.227.36:4518: use of closed network connection; write tcp 188.166.225.170:38088->145.79.227.36:4518: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38088->145.79.227.36:4518: use of closed network connection Dec 22 14:59:07.138890994 (407h30m7.009974039s) [noise=true , address=145.79.227.36:4518 , piece_id=HJWLGPPWRQWISZDODAK3XZEEP3K3DDYBLLE6Y7M2REJUNXUT7IXQ, seg_pos={0 0}, upload=357761]: Long tail cancellation due to optimal threshold Dec 22 14:59:07.139329078 (407h30m7.010412123s) [noise=true , address=145.79.227.32:4834 , piece_id=RPB5L4CGS3QG5KKYCZVYLLHHKDCWY2HWREOLDHCINX4ES3YCY4MA, seg_pos={0 0}, upload=357761]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk): protocol: expected piece hash; context canceled; EOF Dec 22 14:59:07.139356270 (407h30m7.010439315s) [noise=true , address=145.79.227.32:4834 , piece_id=RPB5L4CGS3QG5KKYCZVYLLHHKDCWY2HWREOLDHCINX4ES3YCY4MA, seg_pos={0 0}, upload=357761]: Long tail cancellation due to optimal threshold Dec 22 14:59:07.139389403 (407h30m7.010472438s) [seg_pos={0 0}, upload=357761]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 14:59:07.139543817 (407h30m7.010626872s) [upload=357761]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 14:59:07.139552013 (407h30m7.010635078s) [upload=357761]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 14:59:07.446007818 (407h30m7.317090863s) [upload=357761]: Done uploading segments... Dec 22 14:59:07.446031644 (407h30m7.317114679s) [upload=357761]: Upload finished. err: <nil> Duration: 15.057888339s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [.X.....................................X.....XX.....XX.XXXXX] duration: 30.002186362s trace: https://jaeger.dev.storj.io/trace/64d5998b4085076f FAIL (2025-12-22 14:59:32 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 04866578s) [upload=357761]: Uploading segments... Dec 22 14:59:03.533960140 (407h30m3.405043185s) [upload=357761]: Got next segment. Inline: false Dec 22 14:59:03.534029151 (407h30m3.405112196s) [upload=357761]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 14:59:03.534039550 (407h30m3.405122596s) [upload=357761]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 14:59:03.784341119 (407h30m3.655424154s) [seg_pos={0 0}, upload=357761]: Begin upload segment... Dec 22 14:59:03.784408697 (407h30m3.655491732s) [seg_pos={0 0}, upload=357761]: Done begin upload segment. Dec 22 14:59:03.784430288 (407h30m3.655513333s) [noise=true , address=145.79.227.46:4603 , piece_id=WNW2PJCFNITKKYDHQLRO233C5IYTOQ6ET2NXN5MJBRMOMV5OZC3Q, seg_pos={0 0}, upload=357761]: Uploading piece... Dec 22 14:59:03.784602646 (407h30m3.655685691s) [noise=true , address=45.91.116.13:4156 , piece_id=J2RDO4SWP6G5GDCOK4255GDBWPOGO6OZZDAUNPEDHC625Q6MF4FA, seg_pos={0 0}, upload=357761]: Uploading piece... Dec 22 14:59:03.784841210 (407h30m3.655924245s) [noise=true , address=145.79.227.47:4267 , piece_id=HYVGAK4WVG3EUHWRP6EENWJLC4HLGAPVFKXZFHCB5O7ERFTLCKXQ, seg_pos={0 0}, upload=357761]: Uploading piece... Dec 22 14:59:03.785300545 (407h30m3.65638359s) [noise=true , address=145.79.227.32:4834 , piece_id=RPB5L4CGS3QG5KKYCZVYLLHHKDCWY2HWREOLDHCINX4ES3YCY4MA, seg_pos={0 0}, upload=357761]: Uploading piece... Dec 22 14:59:03.785695958 (407h30m3.656779004s) [noise=true , address=145.79.227.51:4411 , piece_id=WQCVZGIHJ7P5GXXJKEONTIOCJF6XHAS6SORDWQP45XX4LIZT55HA, seg_pos={0 0}, upload=357761]: Uploading piece... Dec 22 14:59:03.786218303 (407h30m3.657301348s) [noise=true , address=145.79.227.37:4566 , piece_id=BTMRQTCB3O6LLSWFCXUBLCRDG4AB3JATJ6IDTPNYMSJW2GVG5S6Q, seg_pos={0 0}, upload=357761]: Uploading piece... Dec 22 14:59:03.786649133 (407h30m3.657732178s) [noise=true , address=45.91.116.12:4226 , piece_id=TQSAYNCCDP6FJMBXFW3CDQXAUHKHLTRUEHPEYRF6DRIBRW2HI7TQ, seg_pos={0 0}, upload=357761]: Uploading piece... Dec 22 14:59:03.787867483 (407h30m3.658950518s) [noise=true , address=77.90.31.12:4590 , piece_id=GU7B34FTJN3A5BA76Y3QSFRDX3SVUSVFZZJR3K4WSG6V6GJTWXVQ, seg_pos={0 0}, upload=357761]: Uploading piece... Dec 22 14:59:03.788662247 (407h30m3.659745292s) [noise=true , address=145.79.227.52:4446 , piece_id=IHX5HJ6DTFTLXKZNXYRGPAJQHRHS6GKHNM2DV5SGCBYRXDW5AJPA, seg_pos={0 0}, upload=357761]: Uploading piece... Dec 22 14:59:03.788926289 (407h30m3.660009334s) [noise=true , address=145.79.227.36:4518 , piece_id=HJWLGPPWRQWISZDODAK3XZEEP3K3DDYBLLE6Y7M2REJUNXUT7IXQ, seg_pos={0 0}, upload=357761]: Uploading piece... Dec 22 14:59:03.789297045 (407h30m3.66038009s) [noise=true , address=145.79.227.50:4374 , piece_id=NLTMPH6FVBMCZQKO43M3MR2P2YB5QCDLQJ5AKKPY5HITBYU32TIA, seg_pos={0 0}, upload=357761]: Uploading piece... Dec 22 14:59:03.789970738 (407h30m3.661053783s) [noise=true , address=145.79.227.33:4871 , piece_id=7XNFYDVVQ73VBICY2TNZ2OMHTBBYVBYY3VVNEE7TKOKA7QRSNZNA, seg_pos={0 0}, upload=357761]: Uploading piece... Dec 22 14:59:03.790629252 (407h30m3.661712297s) [noise=true , address=145.79.227.5:4223 , piece_id=PCJNSCIQWFYH3ZK4U2YAFRTMBWX63VPZELPB6SUEVOQNKUKW6QIA, seg_pos={0 0}, upload=357761]: Uploading piece... Dec 22 14:59:03.791282937 (407h30m3.662365982s) [noise=true , address=145.79.227.11:4732 , piece_id=MTFAMAYYBIJK6A2PJKZOYGASG37ZDNCZOHV4O7KEEPPTPW5LW4AA, seg_pos={0 0}, upload=357761]: Uploading piece... Dec 22 14:59:03.791831692 (407h30m3.662914747s) [noise=true , address=145.79.227.17:4972 , piece_id=JKUT6AA655S24EBFCQ47FNCDDLJ25OXSWVDKHOARE4XKDXCNKZGQ, seg_pos={0 0}, upload=357761]: Uploading piece... Dec 22 14:59:03.792125310 (407h30m3.663208365s) [noise=true , address=145.79.227.38:4977 , piece_id=L2J7RJZEEL23SI6L7YBSVEVFARV77IWWTRNPYWIFG2NWNQGYTXDQ, seg_pos={0 0}, upload=357761]: Uploading piece... Dec 22 14:59:05.653337738 (407h30m5.524420783s) [upload=357761]: Next returned nil segment Dec 22 14:59:05.653363587 (407h30m5.524446632s) [upload=357761]: Waiting for error group managing segments... Dec 22 14:59:06.060881413 (407h30m5.931964448s) [noise=true , address=77.90.31.12:4590 , piece_id=GU7B34FTJN3A5BA76Y3QSFRDX3SVUSVFZZJR3K4WSG6V6GJTWXVQ, seg_pos={0 0}, upload=357761]: Done uploading piece. err: <nil> Dec 22 14:59:06.063869092 (407h30m5.934952137s) [noise=true , address=45.91.116.13:4156 , piece_id=J2RDO4SWP6G5GDCOK4255GDBWPOGO6OZZDAUNPEDHC625Q6MF4FA, seg_pos={0 0}, upload=357761]: Done uploading piece. err: <nil> Dec 22 14:59:06.263490436 (407h30m6.134573481s) [noise=true , address=45.91.116.12:4226 , piece_id=TQSAYNCCDP6FJMBXFW3CDQXAUHKHLTRUEHPEYRF6DRIBRW2HI7TQ, seg_pos={0 0}, upload=357761]: Done uploading piece. err: <nil> Dec 22 14:59:06.675418780 (407h30m6.546501815s) [noise=true , address=145.79.227.47:4267 , piece_id=HYVGAK4WVG3EUHWRP6EENWJLC4HLGAPVFKXZFHCB5O7ERFTLCKXQ, seg_pos={0 0}, upload=357761]: Done uploading piece. err: <nil> Dec 22 14:59:06.677670729 (407h30m6.548753774s) [noise=true , address=145.79.227.51:4411 , piece_id=WQCVZGIHJ7P5GXXJKEONTIOCJF6XHAS6SORDWQP45XX4LIZT55HA, seg_pos={0 0}, upload=357761]: Done uploading piece. err: <nil> Dec 22 14:59:06.690093824 (407h30m6.561176869s) [noise=true , address=145.79.227.37:4566 , piece_id=BTMRQTCB3O6LLSWFCXUBLCRDG4AB3JATJ6IDTPNYMSJW2GVG5S6Q, seg_pos={0 0}, upload=357761]: Done uploading piece. err: <nil> Dec 22 14:59:06.695236277 (407h30m6.566319322s) [noise=true , address=145.79.227.5:4223 , piece_id=PCJNSCIQWFYH3ZK4U2YAFRTMBWX63VPZELPB6SUEVOQNKUKW6QIA, seg_pos={0 0}, upload=357761]: Done uploading piece. err: <nil> Dec 22 14:59:06.750737611 (407h30m6.621820646s) [noise=true , address=145.79.227.38:4977 , piece_id=L2J7RJZEEL23SI6L7YBSVEVFARV77IWWTRNPYWIFG2NWNQGYTXDQ, seg_pos={0 0}, upload=357761]: Done uploading piece. err: <nil> Dec 22 14:59:06.761387720 (407h30m6.632470765s) [noise=true , address=145.79.227.11:4732 , piece_id=MTFAMAYYBIJK6A2PJKZOYGASG37ZDNCZOHV4O7KEEPPTPW5LW4AA, seg_pos={0 0}, upload=357761]: Done uploading piece. err: <nil> Dec 22 14:59:06.769328340 (407h30m6.640411386s) [noise=true , address=145.79.227.46:4603 , piece_id=WNW2PJCFNITKKYDHQLRO233C5IYTOQ6ET2NXN5MJBRMOMV5OZC3Q, seg_pos={0 0}, upload=357761]: Done uploading piece. err: <nil> Dec 22 14:59:06.846216479 (407h30m6.717299514s) [noise=true , address=145.79.227.50:4374 , piece_id=NLTMPH6FVBMCZQKO43M3MR2P2YB5QCDLQJ5AKKPY5HITBYU32TIA, seg_pos={0 0}, upload=357761]: Done uploading piece. err: <nil> Dec 22 14:59:06.846584721 (407h30m6.717667756s) [noise=true , address=145.79.227.17:4972 , piece_id=JKUT6AA655S24EBFCQ47FNCDDLJ25OXSWVDKHOARE4XKDXCNKZGQ, seg_pos={0 0}, upload=357761]: Done uploading piece. err: <nil> Dec 22 14:59:07.137793744 (407h30m7.008876789s) [noise=true , address=145.79.227.33:4871 , piece_id=7XNFYDVVQ73VBICY2TNZ2OMHTBBYVBYY3VVNEE7TKOKA7QRSNZNA, seg_pos={0 0}, upload=357761]: Done uploading piece. err: <nil> Dec 22 14:59:07.137859660 (407h30m7.008942695s) [seg_pos={0 0}, upload=357761]: Segment reached optimal threshold of 13 pieces. Dec 22 14:59:07.138352869 (407h30m7.009435914s) [noise=true , address=145.79.227.52:4446 , piece_id=IHX5HJ6DTFTLXKZNXYRGPAJQHRHS6GKHNM2DV5SGCBYRXDW5AJPA, seg_pos={0 0}, upload=357761]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RNJPWzogZpsQfuFzJYZiz8bbHSRSrgSE5FLyhCynSwEDfMid7): protocol: write tcp 188.166.225.170:54006->145.79.227.52:4446: use of closed network connection; write tcp 188.166.225.170:54006->145.79.227.52:4446: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54006->145.79.227.52:4446: use of closed network connection Dec 22 14:59:07.138366365 (407h30m7.0094494s) [noise=true , address=145.79.227.52:4446 , piece_id=IHX5HJ6DTFTLXKZNXYRGPAJQHRHS6GKHNM2DV5SGCBYRXDW5AJPA, seg_pos={0 0}, upload=357761]: Long tail cancellation due to optimal threshold Dec 22 14:59:07.138881896 (407h30m7.009964931s) [noise=true , address=145.79.227.36:4518 , piece_id=HJWLGPPWRQWISZDODAK3XZEEP3K3DDYBLLE6Y7M2REJUNXUT7IXQ, seg_pos={0 0}, upload=357761]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17TZ9JLdCivFF46guhSTW6hcoUmTz5uo5wvLkDqC4BGmpm9eLb): protocol: write tcp 188.166.225.170:38088->145.79.227.36:4518: use of closed network connection; write tcp 188.166.225.170:38088->145.79.227.36:4518: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38088->145.79.227.36:4518: use of closed network connection Dec 22 14:59:07.138890994 (407h30m7.009974039s) [noise=true , address=145.79.227.36:4518 , piece_id=HJWLGPPWRQWISZDODAK3XZEEP3K3DDYBLLE6Y7M2REJUNXUT7IXQ, seg_pos={0 0}, upload=357761]: Long tail cancellation due to optimal threshold Dec 22 14:59:07.139329078 (407h30m7.010412123s) [noise=true , address=145.79.227.32:4834 , piece_id=RPB5L4CGS3QG5KKYCZVYLLHHKDCWY2HWREOLDHCINX4ES3YCY4MA, seg_pos={0 0}, upload=357761]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk): protocol: expected piece hash; context canceled; EOF Dec 22 14:59:07.139356270 (407h30m7.010439315s) [noise=true , address=145.79.227.32:4834 , piece_id=RPB5L4CGS3QG5KKYCZVYLLHHKDCWY2HWREOLDHCINX4ES3YCY4MA, seg_pos={0 0}, upload=357761]: Long tail cancellation due to optimal threshold Dec 22 14:59:07.139389403 (407h30m7.010472438s) [seg_pos={0 0}, upload=357761]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 14:59:07.139543817 (407h30m7.010626872s) [upload=357761]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 14:59:07.139552013 (407h30m7.010635078s) [upload=357761]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 14:59:07.446007818 (407h30m7.317090863s) [upload=357761]: Done uploading segments... Dec 22 14:59:07.446031644 (407h30m7.317114679s) [upload=357761]: Upload finished. err: <nil> Duration: 30.002205721s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 7.34523694s trace: https://jaeger.dev.storj.io/trace/718f722b1bda7fe2 FAIL (2025-12-22 14:59:29 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== VMAC4CKPNLATBLZLKAGDGZHVT7BHEPN3Q, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.367093115 (407h9m14.23817616s) [noise=true , address=185.167.204.206:4043 , piece_id=HL32YEZF67X2UT6SRMXJVJ4W5A3PC5SXOCM7BYQVMEC76TKEVJRA, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.367396732 (407h9m14.238479767s) [noise=true , address=185.167.204.205:4032 , piece_id=FJMXJG2PQ4WZDSDCPJ6LLSVQKI7JJHUCDUZLU2BJZV7RZTLETM5A, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.367549273 (407h9m14.238632308s) [noise=true , address=185.167.204.206:4070 , piece_id=Y7IRY4E2GCF7TCFOIQJ7QH6IFEJBMW4WM6QBIS5SCUJ7E7ASV4HA, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.367749263 (407h9m14.238832308s) [noise=true , address=81.171.14.70:4046 , piece_id=NRMV6FG5CZSTLUNP2EHNCK6VM2Z2UHKBLCHLKEPQ225Q5DGYEBPA, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.368034046 (407h9m14.239117091s) [noise=true , address=185.167.204.206:4060 , piece_id=WN6T5TUBVMURPSBBXGXCTLPZZ74PFNSRTTWXI2JPUTY74DPH3YSQ, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.368552724 (407h9m14.239635759s) [noise=true , address=81.171.14.70:4029 , piece_id=APDMOYEMDCLOREXY2BBOK73RSQVWKIXUX4SCHRVFDSM7ZLWBEQXA, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.369160691 (407h9m14.240243736s) [noise=true , address=185.167.204.206:4057 , piece_id=ONW7YS5JG44B2STNFJGJWFEZ3WYXO74N3CXI3C6T5OTJYHGQPQZQ, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.369714496 (407h9m14.240797541s) [noise=true , address=185.167.204.206:4071 , piece_id=DVJAKIMPAUGWQPHDPGTXASQDONAKOZIGX5PQDBGBXNQAPMBPUFBA, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.370249614 (407h9m14.241332649s) [noise=true , address=89.149.201.147:4019 , piece_id=ILE7ZX5X6HBPUVYGCOQ5SNG3AA6B3YTXIF47CPLJ3MN5EXVOS2OA, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.372019455 (407h9m14.24310249s) [noise=true , address=89.149.201.147:4015 , piece_id=ZIWRTD7HSP7YMRJET2TTJQFAQZM7KGDKJDLMYCWWY2W6SV2UVQBQ, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:15.199398795 (407h9m15.07048184s) [upload=357453]: Next returned nil segment Dec 22 14:38:15.199414966 (407h9m15.070498011s) [upload=357453]: Waiting for error group managing segments... Dec 22 14:38:15.591187572 (407h9m15.462270607s) [noise=true , address=185.167.204.205:4009 , piece_id=JTHX2O6T5HRENGEMAFJUWQUEUYAIG2KQ5HDEHF4BK42IT2ER7EIA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.591766795 (407h9m15.46284984s) [noise=true , address=185.167.204.205:4006 , piece_id=XHNFS6G7EX4BQWKOK4JUWYV3OTVJ36A6VOP4N7G2XPZKNNCFFKXA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.592074521 (407h9m15.463157566s) [noise=true , address=185.167.204.206:4060 , piece_id=WN6T5TUBVMURPSBBXGXCTLPZZ74PFNSRTTWXI2JPUTY74DPH3YSQ, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.593304694 (407h9m15.464387729s) [noise=true , address=185.167.204.205:4036 , piece_id=5QWJTTL6TZIZSSYHCBEJBKGX7BS76VVTELWTKEAKTOVYKINXVWDQ, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.593638659 (407h9m15.464721694s) [noise=true , address=185.167.204.205:4004 , piece_id=PYGDQZMAKOVJWB7AWEPFMPMB6C24FMS4HT2UP2AKZNYGG4DGZR3Q, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.593818251 (407h9m15.464901286s) [noise=true , address=185.167.204.206:4048 , piece_id=52WC3XH2K2BHG33BQPRENPNYDT7EOKCNBCTZKE3LIONDJHVATEYA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.594252549 (407h9m15.465335594s) [noise=true , address=185.167.204.205:4034 , piece_id=OS5H57TG2REFE535DK4T6N75VNIXPLQWW6DM55QCC2S7QBVZM3UQ, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.597113216 (407h9m15.468196261s) [noise=true , address=185.167.204.205:4032 , piece_id=FJMXJG2PQ4WZDSDCPJ6LLSVQKI7JJHUCDUZLU2BJZV7RZTLETM5A, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.597394541 (407h9m15.468477586s) [noise=true , address=185.167.204.205:4035 , piece_id=3D7T4SGPAJQZCA74CUTR4GJRE23R4JD5KBNW3BLEASSP4GSR5PHQ, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.597612756 (407h9m15.468695792s) [noise=true , address=185.167.204.206:4043 , piece_id=HL32YEZF67X2UT6SRMXJVJ4W5A3PC5SXOCM7BYQVMEC76TKEVJRA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.598018900 (407h9m15.469101945s) [noise=true , address=185.167.204.205:4012 , piece_id=A5G37BVL2GZ7LAY3FZWIZV4R4M7M6UPYDYACJFRD5HKDWWKB4Q6A, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.598254719 (407h9m15.469337765s) [noise=true , address=185.167.204.206:4052 , piece_id=V2DVAAXN5A4C3NQW556VMAC4CKPNLATBLZLKAGDGZHVT7BHEPN3Q, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.598480649 (407h9m15.469563694s) [noise=true , address=185.167.204.205:4018 , piece_id=NUYRZ7PXKJQ3PIR44O3E6N6COZZPRDNSMTQCFI5RN7FBRV2CBALA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.598968278 (407h9m15.470051323s) [noise=true , address=185.167.204.205:4033 , piece_id=A32ZTBDBWMCNBMZPD4DUWS5NKUKQ22ACQH6HAW5AKB3FNHBC56VA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.599287886 (407h9m15.470370931s) [noise=true , address=185.167.204.206:4072 , piece_id=I6PCSC5JYTJXLT6B34TTIVZQVX52R7TRHSRLRP4ZHG5J63XDZN6Q, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.599632252 (407h9m15.470715297s) [noise=true , address=185.167.204.206:4070 , piece_id=Y7IRY4E2GCF7TCFOIQJ7QH6IFEJBMW4WM6QBIS5SCUJ7E7ASV4HA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.599676297 (407h9m15.470759332s) [noise=true , address=185.167.204.206:4071 , piece_id=DVJAKIMPAUGWQPHDPGTXASQDONAKOZIGX5PQDBGBXNQAPMBPUFBA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.601435877 (407h9m15.472518912s) [noise=true , address=185.167.204.206:4057 , piece_id=ONW7YS5JG44B2STNFJGJWFEZ3WYXO74N3CXI3C6T5OTJYHGQPQZQ, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:17.299689191 (407h9m17.170772226s) [noise=true , address=89.149.201.147:4019 , piece_id=ILE7ZX5X6HBPUVYGCOQ5SNG3AA6B3YTXIF47CPLJ3MN5EXVOS2OA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:17.526601953 (407h9m17.397684998s) [noise=true , address=89.149.201.147:4015 , piece_id=ZIWRTD7HSP7YMRJET2TTJQFAQZM7KGDKJDLMYCWWY2W6SV2UVQBQ, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:18.294911214 (407h9m18.165994249s) [noise=true , address=89.149.201.147:4017 , piece_id=ZOSNSMZOBQVIKD2ZGJRAUWJEJ4JXJK452YWQQSDL77E77F6LRDZA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:18.402191355 (407h9m18.2732744s) [noise=true , address=89.149.201.147:4011 , piece_id=ATMM77IGBDSRHNJ5FQ2VKP7W27JUA2F6HYJYSQ62IL5JQN5PF5OA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:22.436508909 (407h9m22.307591944s) [noise=true , address=81.171.14.70:4037 , piece_id=NYGC2ZLUSBUAGFULQ46LXCVNQUJ5M4XT3YICVA2ZMBR2D6WPJ27Q, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:25.935108368 (407h9m25.806191413s) [noise=true , address=81.171.14.70:4046 , piece_id=NRMV6FG5CZSTLUNP2EHNCK6VM2Z2UHKBLCHLKEPQ225Q5DGYEBPA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:30.904793996 (407h9m30.775877041s) [noise=true , address=81.171.14.70:4029 , piece_id=APDMOYEMDCLOREXY2BBOK73RSQVWKIXUX4SCHRVFDSM7ZLWBEQXA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:32.926010512 (407h9m32.797093567s) [noise=true , address=81.171.14.70:4044 , piece_id=QCWQO6JQRRKSG5U2KVH3Y6I3UI2F7G3DOW4CH3PQZKUVQ6Q2XOHA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:32.926036552 (407h9m32.797119587s) [seg_pos={0 0}, upload=357453]: Segment reached optimal threshold of 26 pieces. Dec 22 14:38:32.926196857 (407h9m32.797279902s) [noise=true , address=81.171.14.70:4033 , piece_id=MKNAJOFD5CIHTA55XO476LOJJGIWSA7HBQFQND2YNM562W74IKJA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BnjusiXEfHYwTDL8soMkm8GYnhbkwy2oCAxVA7oiDyhFXozZ8): protocol: expected piece hash; context canceled; EOF Dec 22 14:38:32.926203058 (407h9m32.797286093s) [noise=true , address=81.171.14.70:4033 , piece_id=MKNAJOFD5CIHTA55XO476LOJJGIWSA7HBQFQND2YNM562W74IKJA, seg_pos={0 0}, upload=357453]: Long tail cancellation due to optimal threshold Dec 22 14:38:32.926237574 (407h9m32.797320609s) [noise=true , address=81.171.14.70:4031 , piece_id=7MD5N7PPKFPCFMOIXAGI3MPBDDX45GTLMA3CFGNM7BBGI6TSL6RA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12drdYnnWsuMyFo6LkcewAGGWnorRCAhxPpNqK41LvGy2zkBvng): protocol: expected piece hash; context canceled; EOF Dec 22 14:38:32.926250028 (407h9m32.797333063s) [noise=true , address=81.171.14.70:4031 , piece_id=7MD5N7PPKFPCFMOIXAGI3MPBDDX45GTLMA3CFGNM7BBGI6TSL6RA, seg_pos={0 0}, upload=357453]: Long tail cancellation due to optimal threshold Dec 22 14:38:32.926299612 (407h9m32.797382647s) [seg_pos={0 0}, upload=357453]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 22 14:38:32.926403941 (407h9m32.797486986s) [upload=357453]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 14:38:32.926409692 (407h9m32.797492737s) [upload=357453]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 14:39:04.793379844 (407h10m4.664462889s) [upload=357453]: Done uploading segments... Dec 22 14:39:04.793413608 (407h10m4.664496643s) [upload=357453]: Upload finished. err: <nil> Duration: 7.34526927s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........X........................X.........................] duration: 5.675535571s trace: https://jaeger.dev.storj.io/trace/510cfc1b6c68e873 FAIL (2025-12-22 14:59:21 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.675560929s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 1.468019588s trace: https://jaeger.dev.storj.io/trace/62b55ec615ab6123 FAIL (2025-12-22 14:59:13 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.468034637s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....X.......................................................] duration: 3.746592874s trace: https://jaeger.dev.storj.io/trace/1c84c110f15266bb FAIL (2025-12-22 14:59:07 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.746610327s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 3 out of 60 is failed (threshold=3) [X.........X........................X........................] duration: 5.439680771s trace: https://jaeger.dev.storj.io/trace/510cfc1b6c68e873 FAIL (2025-12-22 14:59:05 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.439717571s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 6.33056322s trace: https://jaeger.dev.storj.io/trace/718f722b1bda7fe2 FAIL (2025-12-22 14:58:53 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== VMAC4CKPNLATBLZLKAGDGZHVT7BHEPN3Q, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.367093115 (407h9m14.23817616s) [noise=true , address=185.167.204.206:4043 , piece_id=HL32YEZF67X2UT6SRMXJVJ4W5A3PC5SXOCM7BYQVMEC76TKEVJRA, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.367396732 (407h9m14.238479767s) [noise=true , address=185.167.204.205:4032 , piece_id=FJMXJG2PQ4WZDSDCPJ6LLSVQKI7JJHUCDUZLU2BJZV7RZTLETM5A, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.367549273 (407h9m14.238632308s) [noise=true , address=185.167.204.206:4070 , piece_id=Y7IRY4E2GCF7TCFOIQJ7QH6IFEJBMW4WM6QBIS5SCUJ7E7ASV4HA, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.367749263 (407h9m14.238832308s) [noise=true , address=81.171.14.70:4046 , piece_id=NRMV6FG5CZSTLUNP2EHNCK6VM2Z2UHKBLCHLKEPQ225Q5DGYEBPA, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.368034046 (407h9m14.239117091s) [noise=true , address=185.167.204.206:4060 , piece_id=WN6T5TUBVMURPSBBXGXCTLPZZ74PFNSRTTWXI2JPUTY74DPH3YSQ, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.368552724 (407h9m14.239635759s) [noise=true , address=81.171.14.70:4029 , piece_id=APDMOYEMDCLOREXY2BBOK73RSQVWKIXUX4SCHRVFDSM7ZLWBEQXA, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.369160691 (407h9m14.240243736s) [noise=true , address=185.167.204.206:4057 , piece_id=ONW7YS5JG44B2STNFJGJWFEZ3WYXO74N3CXI3C6T5OTJYHGQPQZQ, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.369714496 (407h9m14.240797541s) [noise=true , address=185.167.204.206:4071 , piece_id=DVJAKIMPAUGWQPHDPGTXASQDONAKOZIGX5PQDBGBXNQAPMBPUFBA, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.370249614 (407h9m14.241332649s) [noise=true , address=89.149.201.147:4019 , piece_id=ILE7ZX5X6HBPUVYGCOQ5SNG3AA6B3YTXIF47CPLJ3MN5EXVOS2OA, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.372019455 (407h9m14.24310249s) [noise=true , address=89.149.201.147:4015 , piece_id=ZIWRTD7HSP7YMRJET2TTJQFAQZM7KGDKJDLMYCWWY2W6SV2UVQBQ, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:15.199398795 (407h9m15.07048184s) [upload=357453]: Next returned nil segment Dec 22 14:38:15.199414966 (407h9m15.070498011s) [upload=357453]: Waiting for error group managing segments... Dec 22 14:38:15.591187572 (407h9m15.462270607s) [noise=true , address=185.167.204.205:4009 , piece_id=JTHX2O6T5HRENGEMAFJUWQUEUYAIG2KQ5HDEHF4BK42IT2ER7EIA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.591766795 (407h9m15.46284984s) [noise=true , address=185.167.204.205:4006 , piece_id=XHNFS6G7EX4BQWKOK4JUWYV3OTVJ36A6VOP4N7G2XPZKNNCFFKXA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.592074521 (407h9m15.463157566s) [noise=true , address=185.167.204.206:4060 , piece_id=WN6T5TUBVMURPSBBXGXCTLPZZ74PFNSRTTWXI2JPUTY74DPH3YSQ, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.593304694 (407h9m15.464387729s) [noise=true , address=185.167.204.205:4036 , piece_id=5QWJTTL6TZIZSSYHCBEJBKGX7BS76VVTELWTKEAKTOVYKINXVWDQ, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.593638659 (407h9m15.464721694s) [noise=true , address=185.167.204.205:4004 , piece_id=PYGDQZMAKOVJWB7AWEPFMPMB6C24FMS4HT2UP2AKZNYGG4DGZR3Q, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.593818251 (407h9m15.464901286s) [noise=true , address=185.167.204.206:4048 , piece_id=52WC3XH2K2BHG33BQPRENPNYDT7EOKCNBCTZKE3LIONDJHVATEYA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.594252549 (407h9m15.465335594s) [noise=true , address=185.167.204.205:4034 , piece_id=OS5H57TG2REFE535DK4T6N75VNIXPLQWW6DM55QCC2S7QBVZM3UQ, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.597113216 (407h9m15.468196261s) [noise=true , address=185.167.204.205:4032 , piece_id=FJMXJG2PQ4WZDSDCPJ6LLSVQKI7JJHUCDUZLU2BJZV7RZTLETM5A, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.597394541 (407h9m15.468477586s) [noise=true , address=185.167.204.205:4035 , piece_id=3D7T4SGPAJQZCA74CUTR4GJRE23R4JD5KBNW3BLEASSP4GSR5PHQ, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.597612756 (407h9m15.468695792s) [noise=true , address=185.167.204.206:4043 , piece_id=HL32YEZF67X2UT6SRMXJVJ4W5A3PC5SXOCM7BYQVMEC76TKEVJRA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.598018900 (407h9m15.469101945s) [noise=true , address=185.167.204.205:4012 , piece_id=A5G37BVL2GZ7LAY3FZWIZV4R4M7M6UPYDYACJFRD5HKDWWKB4Q6A, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.598254719 (407h9m15.469337765s) [noise=true , address=185.167.204.206:4052 , piece_id=V2DVAAXN5A4C3NQW556VMAC4CKPNLATBLZLKAGDGZHVT7BHEPN3Q, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.598480649 (407h9m15.469563694s) [noise=true , address=185.167.204.205:4018 , piece_id=NUYRZ7PXKJQ3PIR44O3E6N6COZZPRDNSMTQCFI5RN7FBRV2CBALA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.598968278 (407h9m15.470051323s) [noise=true , address=185.167.204.205:4033 , piece_id=A32ZTBDBWMCNBMZPD4DUWS5NKUKQ22ACQH6HAW5AKB3FNHBC56VA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.599287886 (407h9m15.470370931s) [noise=true , address=185.167.204.206:4072 , piece_id=I6PCSC5JYTJXLT6B34TTIVZQVX52R7TRHSRLRP4ZHG5J63XDZN6Q, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.599632252 (407h9m15.470715297s) [noise=true , address=185.167.204.206:4070 , piece_id=Y7IRY4E2GCF7TCFOIQJ7QH6IFEJBMW4WM6QBIS5SCUJ7E7ASV4HA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.599676297 (407h9m15.470759332s) [noise=true , address=185.167.204.206:4071 , piece_id=DVJAKIMPAUGWQPHDPGTXASQDONAKOZIGX5PQDBGBXNQAPMBPUFBA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.601435877 (407h9m15.472518912s) [noise=true , address=185.167.204.206:4057 , piece_id=ONW7YS5JG44B2STNFJGJWFEZ3WYXO74N3CXI3C6T5OTJYHGQPQZQ, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:17.299689191 (407h9m17.170772226s) [noise=true , address=89.149.201.147:4019 , piece_id=ILE7ZX5X6HBPUVYGCOQ5SNG3AA6B3YTXIF47CPLJ3MN5EXVOS2OA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:17.526601953 (407h9m17.397684998s) [noise=true , address=89.149.201.147:4015 , piece_id=ZIWRTD7HSP7YMRJET2TTJQFAQZM7KGDKJDLMYCWWY2W6SV2UVQBQ, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:18.294911214 (407h9m18.165994249s) [noise=true , address=89.149.201.147:4017 , piece_id=ZOSNSMZOBQVIKD2ZGJRAUWJEJ4JXJK452YWQQSDL77E77F6LRDZA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:18.402191355 (407h9m18.2732744s) [noise=true , address=89.149.201.147:4011 , piece_id=ATMM77IGBDSRHNJ5FQ2VKP7W27JUA2F6HYJYSQ62IL5JQN5PF5OA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:22.436508909 (407h9m22.307591944s) [noise=true , address=81.171.14.70:4037 , piece_id=NYGC2ZLUSBUAGFULQ46LXCVNQUJ5M4XT3YICVA2ZMBR2D6WPJ27Q, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:25.935108368 (407h9m25.806191413s) [noise=true , address=81.171.14.70:4046 , piece_id=NRMV6FG5CZSTLUNP2EHNCK6VM2Z2UHKBLCHLKEPQ225Q5DGYEBPA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:30.904793996 (407h9m30.775877041s) [noise=true , address=81.171.14.70:4029 , piece_id=APDMOYEMDCLOREXY2BBOK73RSQVWKIXUX4SCHRVFDSM7ZLWBEQXA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:32.926010512 (407h9m32.797093567s) [noise=true , address=81.171.14.70:4044 , piece_id=QCWQO6JQRRKSG5U2KVH3Y6I3UI2F7G3DOW4CH3PQZKUVQ6Q2XOHA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:32.926036552 (407h9m32.797119587s) [seg_pos={0 0}, upload=357453]: Segment reached optimal threshold of 26 pieces. Dec 22 14:38:32.926196857 (407h9m32.797279902s) [noise=true , address=81.171.14.70:4033 , piece_id=MKNAJOFD5CIHTA55XO476LOJJGIWSA7HBQFQND2YNM562W74IKJA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BnjusiXEfHYwTDL8soMkm8GYnhbkwy2oCAxVA7oiDyhFXozZ8): protocol: expected piece hash; context canceled; EOF Dec 22 14:38:32.926203058 (407h9m32.797286093s) [noise=true , address=81.171.14.70:4033 , piece_id=MKNAJOFD5CIHTA55XO476LOJJGIWSA7HBQFQND2YNM562W74IKJA, seg_pos={0 0}, upload=357453]: Long tail cancellation due to optimal threshold Dec 22 14:38:32.926237574 (407h9m32.797320609s) [noise=true , address=81.171.14.70:4031 , piece_id=7MD5N7PPKFPCFMOIXAGI3MPBDDX45GTLMA3CFGNM7BBGI6TSL6RA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12drdYnnWsuMyFo6LkcewAGGWnorRCAhxPpNqK41LvGy2zkBvng): protocol: expected piece hash; context canceled; EOF Dec 22 14:38:32.926250028 (407h9m32.797333063s) [noise=true , address=81.171.14.70:4031 , piece_id=7MD5N7PPKFPCFMOIXAGI3MPBDDX45GTLMA3CFGNM7BBGI6TSL6RA, seg_pos={0 0}, upload=357453]: Long tail cancellation due to optimal threshold Dec 22 14:38:32.926299612 (407h9m32.797382647s) [seg_pos={0 0}, upload=357453]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 22 14:38:32.926403941 (407h9m32.797486986s) [upload=357453]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 14:38:32.926409692 (407h9m32.797492737s) [upload=357453]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 14:39:04.793379844 (407h10m4.664462889s) [upload=357453]: Done uploading segments... Dec 22 14:39:04.793413608 (407h10m4.664496643s) [upload=357453]: Upload finished. err: <nil> Duration: 6.33059384s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [..X.....................................X.....XX.....XX.XXXX] duration: 30.003476127s trace: https://jaeger.dev.storj.io/trace/6be5330f9c1803b0 FAIL (2025-12-22 14:58:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 22 14:58:16.596180199 (407h29m16.467263235s) [upload=357748]: Starting upload Dec 22 14:58:16.596254150 (407h29m16.467337195s) [upload=357748]: Done starting upload Dec 22 14:58:16.596354992 (407h29m16.467438037s) [upload=357748]: Uploading segments... Dec 22 14:58:16.596430937 (407h29m16.467513982s) [upload=357748]: Got next segment. Inline: false Dec 22 14:58:16.596527571 (407h29m16.467610616s) [upload=357748]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 14:58:16.596533863 (407h29m16.467616908s) [upload=357748]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 14:58:16.944351569 (407h29m16.815434614s) [seg_pos={0 0}, upload=357748]: Begin upload segment... Dec 22 14:58:16.944394300 (407h29m16.815477335s) [seg_pos={0 0}, upload=357748]: Done begin upload segment. Dec 22 14:58:16.944436060 (407h29m16.815519106s) [noise=true , address=45.91.116.12:4225 , piece_id=YEA6PPII4J2XGG32TZSZKSXOZ7C2NU45YR7OGOB3ZMTFT7TJZKFQ, seg_pos={0 0}, upload=357748]: Uploading piece... Dec 22 14:58:16.944537192 (407h29m16.815620227s) [noise=true , address=145.79.227.36:4519 , piece_id=4N6UXPUGTS3G7K7FYU2RE6XNASX7V6CUR53NERK4DE755GRB6QCA, seg_pos={0 0}, upload=357748]: Uploading piece... Dec 22 14:58:16.945028599 (407h29m16.816111644s) [noise=true , address=145.79.227.48:4301 , piece_id=BCFJSIEAYW5SFTDFHWNF34ZAGNFMHGU7RYUBK57CSQOYP3QN27JA, seg_pos={0 0}, upload=357748]: Uploading piece... Dec 22 14:58:16.945352856 (407h29m16.816435891s) [noise=true , address=45.91.116.6:4974 , piece_id=N5HQOMNXUKTRESQ6FFQCOTM4GBM2DMNX2OHY2EKLRIHNTZPWIM6Q, seg_pos={0 0}, upload=357748]: Uploading piece... Dec 22 14:58:16.946379500 (407h29m16.817462535s) [noise=true , address=77.90.31.13:4627 , piece_id=3LZFDZRBEEH6PKW3BOEAH27OKV4Z3BI26OLFLTDXPOO5HVW4VAJA, seg_pos={0 0}, upload=357748]: Uploading piece... Dec 22 14:58:16.946754945 (407h29m16.81783798s) [noise=true , address=145.79.227.35:4483 , piece_id=ULZFO2TD7VG542UKD4RB5MEDGOT2DO5KUROXLZCG6N7U442BASAQ, seg_pos={0 0}, upload=357748]: Uploading piece... Dec 22 14:58:16.947224310 (407h29m16.818307355s) [noise=true , address=45.91.116.4:4937 , piece_id=PEBSREXQI7YQ3KSOI5ZO47VB74Q64BJQWFDG3JRZ752KP6NQAPVA, seg_pos={0 0}, upload=357748]: Uploading piece... Dec 22 14:58:16.950812502 (407h29m16.821895547s) [noise=true , address=145.79.227.43:4158 , piece_id=ZTOI6WLOZ5QGJIX2CL7DO4JY4JMXBJBVTPTESVRSEPL54V5DXQWQ, seg_pos={0 0}, upload=357748]: Uploading piece... Dec 22 14:58:16.952302319 (407h29m16.823385394s) [noise=true , address=145.79.227.10:4401 , piece_id=LDI6J5OIJDJLDOARRGUOIZDOTBKNNB4CKQ77XFDLJZ7MVL5GPABA, seg_pos={0 0}, upload=357748]: Uploading piece... Dec 22 14:58:16.953503977 (407h29m16.824587022s) [noise=true , address=45.91.116.8:4047 , piece_id=I7PDY6HAL775GPWVU5MA3SQ3DIWJ6EBKCQAEV4RCICBQ4CESQP7Q, seg_pos={0 0}, upload=357748]: Uploading piece... Dec 22 14:58:16.954002706 (407h29m16.825085741s) [noise=true , address=145.79.227.45:4231 , piece_id=3VRDFO3JCGVP5PA52RCQUPSA6GCFYKEGFFBZ6KK54MMZNVSFBR3Q, seg_pos={0 0}, upload=357748]: Uploading piece... Dec 22 14:58:16.955610859 (407h29m16.826693894s) [noise=true , address=45.91.116.11:4117 , piece_id=C5TT7UELQXRYWZH5YAFM3GZEMVZIOIXRRTNEYALVRGD7GIVWKE7Q, seg_pos={0 0}, upload=357748]: Uploading piece... Dec 22 14:58:16.957012979 (407h29m16.828096024s) [noise=true , address=145.79.227.33:4870 , piece_id=PIAYMWZPJKZ3GNY6B6OBL7VZCZMLBKCWZO224VCGR2WCN3YSSR6A, seg_pos={0 0}, upload=357748]: Uploading piece... Dec 22 14:58:16.958231699 (407h29m16.829314735s) [noise=true , address=145.79.227.50:4375 , piece_id=EK4CQT5FH2SBSJCRP7IA4IZZLJOQKXUMW2K7KV2G7V3RKBGDCASA, seg_pos={0 0}, upload=357748]: Uploading piece... Dec 22 14:58:16.958412964 (407h29m16.829495999s) [noise=true , address=145.79.227.42:4123 , piece_id=T7A5RAOUANUJAQ4RICY7ZVTMATYPZLP2TTUZJTLVPGTGTV4BVSFQ, seg_pos={0 0}, upload=357748]: Uploading piece... Dec 22 14:58:16.958824348 (407h29m16.829907383s) [noise=true , address=145.79.227.38:4979 , piece_id=PAQHZH46W5QLJQNQ2NVSHE5AUDYWSZNARZA3IMQ2S6LOTYAICNBQ, seg_pos={0 0}, upload=357748]: Uploading piece... Dec 22 14:58:18.845152100 (407h29m18.716235135s) [upload=357748]: Next returned nil segment Dec 22 14:58:18.845167118 (407h29m18.716250163s) [upload=357748]: Waiting for error group managing segments... Dec 22 14:58:19.146771175 (407h29m19.01785422s) [noise=true , address=77.90.31.13:4627 , piece_id=3LZFDZRBEEH6PKW3BOEAH27OKV4Z3BI26OLFLTDXPOO5HVW4VAJA, seg_pos={0 0}, upload=357748]: Done uploading piece. err: <nil> Dec 22 14:58:19.202644898 (407h29m19.073727943s) [noise=true , address=45.91.116.12:4225 , piece_id=YEA6PPII4J2XGG32TZSZKSXOZ7C2NU45YR7OGOB3ZMTFT7TJZKFQ, seg_pos={0 0}, upload=357748]: Done uploading piece. err: <nil> Dec 22 14:58:19.208581843 (407h29m19.079664888s) [noise=true , address=45.91.116.11:4117 , piece_id=C5TT7UELQXRYWZH5YAFM3GZEMVZIOIXRRTNEYALVRGD7GIVWKE7Q, seg_pos={0 0}, upload=357748]: Done uploading piece. err: <nil> Dec 22 14:58:19.254096141 (407h29m19.125179186s) [noise=true , address=45.91.116.6:4974 , piece_id=N5HQOMNXUKTRESQ6FFQCOTM4GBM2DMNX2OHY2EKLRIHNTZPWIM6Q, seg_pos={0 0}, upload=357748]: Done uploading piece. err: <nil> Dec 22 14:58:19.274563605 (407h29m19.14564664s) [noise=true , address=45.91.116.4:4937 , piece_id=PEBSREXQI7YQ3KSOI5ZO47VB74Q64BJQWFDG3JRZ752KP6NQAPVA, seg_pos={0 0}, upload=357748]: Done uploading piece. err: <nil> Dec 22 14:58:19.855589400 (407h29m19.726672435s) [noise=true , address=145.79.227.35:4483 , piece_id=ULZFO2TD7VG542UKD4RB5MEDGOT2DO5KUROXLZCG6N7U442BASAQ, seg_pos={0 0}, upload=357748]: Done uploading piece. err: <nil> Dec 22 14:58:19.868933209 (407h29m19.740016244s) [noise=true , address=145.79.227.43:4158 , piece_id=ZTOI6WLOZ5QGJIX2CL7DO4JY4JMXBJBVTPTESVRSEPL54V5DXQWQ, seg_pos={0 0}, upload=357748]: Done uploading piece. err: <nil> Dec 22 14:58:19.887079323 (407h29m19.758162378s) [noise=true , address=145.79.227.50:4375 , piece_id=EK4CQT5FH2SBSJCRP7IA4IZZLJOQKXUMW2K7KV2G7V3RKBGDCASA, seg_pos={0 0}, upload=357748]: Done uploading piece. err: <nil> Dec 22 14:58:19.901268060 (407h29m19.772351095s) [noise=true , address=145.79.227.10:4401 , piece_id=LDI6J5OIJDJLDOARRGUOIZDOTBKNNB4CKQ77XFDLJZ7MVL5GPABA, seg_pos={0 0}, upload=357748]: Done uploading piece. err: <nil> Dec 22 14:58:19.904812880 (407h29m19.775895915s) [noise=true , address=145.79.227.36:4519 , piece_id=4N6UXPUGTS3G7K7FYU2RE6XNASX7V6CUR53NERK4DE755GRB6QCA, seg_pos={0 0}, upload=357748]: Done uploading piece. err: <nil> Dec 22 14:58:19.911566500 (407h29m19.782649545s) [noise=true , address=145.79.227.38:4979 , piece_id=PAQHZH46W5QLJQNQ2NVSHE5AUDYWSZNARZA3IMQ2S6LOTYAICNBQ, seg_pos={0 0}, upload=357748]: Done uploading piece. err: <nil> Dec 22 14:58:19.915429336 (407h29m19.786512381s) [noise=true , address=145.79.227.42:4123 , piece_id=T7A5RAOUANUJAQ4RICY7ZVTMATYPZLP2TTUZJTLVPGTGTV4BVSFQ, seg_pos={0 0}, upload=357748]: Done uploading piece. err: <nil> Dec 22 14:58:19.928050769 (407h29m19.799133804s) [noise=true , address=145.79.227.33:4870 , piece_id=PIAYMWZPJKZ3GNY6B6OBL7VZCZMLBKCWZO224VCGR2WCN3YSSR6A, seg_pos={0 0}, upload=357748]: Done uploading piece. err: <nil> Dec 22 14:58:19.928076579 (407h29m19.799159624s) [seg_pos={0 0}, upload=357748]: Segment reached optimal threshold of 13 pieces. Dec 22 14:58:19.928303090 (407h29m19.799386125s) [noise=true , address=145.79.227.45:4231 , piece_id=3VRDFO3JCGVP5PA52RCQUPSA6GCFYKEGFFBZ6KK54MMZNVSFBR3Q, seg_pos={0 0}, upload=357748]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129WWyyn7SrJFbFJ2kjPVhwDNRoYM7Kj449TZiZLzUbqpwK7qJ4): protocol: expected piece hash; context canceled; EOF Dec 22 14:58:19.928311686 (407h29m19.799394731s) [noise=true , address=145.79.227.45:4231 , piece_id=3VRDFO3JCGVP5PA52RCQUPSA6GCFYKEGFFBZ6KK54MMZNVSFBR3Q, seg_pos={0 0}, upload=357748]: Long tail cancellation due to optimal threshold Dec 22 14:58:19.928520684 (407h29m19.799603749s) [noise=true , address=45.91.116.8:4047 , piece_id=I7PDY6HAL775GPWVU5MA3SQ3DIWJ6EBKCQAEV4RCICBQ4CESQP7Q, seg_pos={0 0}, upload=357748]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 22 14:58:19.928531355 (407h29m19.7996144s) [noise=true , address=45.91.116.8:4047 , piece_id=I7PDY6HAL775GPWVU5MA3SQ3DIWJ6EBKCQAEV4RCICBQ4CESQP7Q, seg_pos={0 0}, upload=357748]: Long tail cancellation due to optimal threshold Dec 22 14:58:19.928598903 (407h29m19.799681938s) [noise=true , address=145.79.227.48:4301 , piece_id=BCFJSIEAYW5SFTDFHWNF34ZAGNFMHGU7RYUBK57CSQOYP3QN27JA, seg_pos={0 0}, upload=357748]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18HkaSrdHHnvmeNz1ormfcSBZH8bMUgkX7X3JmGuqkmtBoak29): protocol: expected piece hash; context canceled; EOF Dec 22 14:58:19.928607730 (407h29m19.799690775s) [noise=true , address=145.79.227.48:4301 , piece_id=BCFJSIEAYW5SFTDFHWNF34ZAGNFMHGU7RYUBK57CSQOYP3QN27JA, seg_pos={0 0}, upload=357748]: Long tail cancellation due to optimal threshold Dec 22 14:58:19.928625603 (407h29m19.799708648s) [seg_pos={0 0}, upload=357748]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 14:58:19.928701398 (407h29m19.799784444s) [upload=357748]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 14:58:19.928706889 (407h29m19.799789934s) [upload=357748]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 14:58:20.338825506 (407h29m20.209908541s) [upload=357748]: Done uploading segments... Dec 22 14:58:20.338848320 (407h29m20.209931355s) [upload=357748]: Upload finished. err: <nil> Duration: 30.003500012s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.X.........X........................X.......................] duration: 5.806280373s trace: https://jaeger.dev.storj.io/trace/510cfc1b6c68e873 FAIL (2025-12-22 14:58:25 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.806306451s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....X......................................................] duration: 3.54979624s trace: https://jaeger.dev.storj.io/trace/1c84c110f15266bb FAIL (2025-12-22 14:58:16 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.549851023s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 6.701530637s trace: https://jaeger.dev.storj.io/trace/718f722b1bda7fe2 FAIL (2025-12-22 14:58:13 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== VMAC4CKPNLATBLZLKAGDGZHVT7BHEPN3Q, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.367093115 (407h9m14.23817616s) [noise=true , address=185.167.204.206:4043 , piece_id=HL32YEZF67X2UT6SRMXJVJ4W5A3PC5SXOCM7BYQVMEC76TKEVJRA, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.367396732 (407h9m14.238479767s) [noise=true , address=185.167.204.205:4032 , piece_id=FJMXJG2PQ4WZDSDCPJ6LLSVQKI7JJHUCDUZLU2BJZV7RZTLETM5A, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.367549273 (407h9m14.238632308s) [noise=true , address=185.167.204.206:4070 , piece_id=Y7IRY4E2GCF7TCFOIQJ7QH6IFEJBMW4WM6QBIS5SCUJ7E7ASV4HA, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.367749263 (407h9m14.238832308s) [noise=true , address=81.171.14.70:4046 , piece_id=NRMV6FG5CZSTLUNP2EHNCK6VM2Z2UHKBLCHLKEPQ225Q5DGYEBPA, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.368034046 (407h9m14.239117091s) [noise=true , address=185.167.204.206:4060 , piece_id=WN6T5TUBVMURPSBBXGXCTLPZZ74PFNSRTTWXI2JPUTY74DPH3YSQ, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.368552724 (407h9m14.239635759s) [noise=true , address=81.171.14.70:4029 , piece_id=APDMOYEMDCLOREXY2BBOK73RSQVWKIXUX4SCHRVFDSM7ZLWBEQXA, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.369160691 (407h9m14.240243736s) [noise=true , address=185.167.204.206:4057 , piece_id=ONW7YS5JG44B2STNFJGJWFEZ3WYXO74N3CXI3C6T5OTJYHGQPQZQ, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.369714496 (407h9m14.240797541s) [noise=true , address=185.167.204.206:4071 , piece_id=DVJAKIMPAUGWQPHDPGTXASQDONAKOZIGX5PQDBGBXNQAPMBPUFBA, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.370249614 (407h9m14.241332649s) [noise=true , address=89.149.201.147:4019 , piece_id=ILE7ZX5X6HBPUVYGCOQ5SNG3AA6B3YTXIF47CPLJ3MN5EXVOS2OA, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.372019455 (407h9m14.24310249s) [noise=true , address=89.149.201.147:4015 , piece_id=ZIWRTD7HSP7YMRJET2TTJQFAQZM7KGDKJDLMYCWWY2W6SV2UVQBQ, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:15.199398795 (407h9m15.07048184s) [upload=357453]: Next returned nil segment Dec 22 14:38:15.199414966 (407h9m15.070498011s) [upload=357453]: Waiting for error group managing segments... Dec 22 14:38:15.591187572 (407h9m15.462270607s) [noise=true , address=185.167.204.205:4009 , piece_id=JTHX2O6T5HRENGEMAFJUWQUEUYAIG2KQ5HDEHF4BK42IT2ER7EIA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.591766795 (407h9m15.46284984s) [noise=true , address=185.167.204.205:4006 , piece_id=XHNFS6G7EX4BQWKOK4JUWYV3OTVJ36A6VOP4N7G2XPZKNNCFFKXA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.592074521 (407h9m15.463157566s) [noise=true , address=185.167.204.206:4060 , piece_id=WN6T5TUBVMURPSBBXGXCTLPZZ74PFNSRTTWXI2JPUTY74DPH3YSQ, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.593304694 (407h9m15.464387729s) [noise=true , address=185.167.204.205:4036 , piece_id=5QWJTTL6TZIZSSYHCBEJBKGX7BS76VVTELWTKEAKTOVYKINXVWDQ, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.593638659 (407h9m15.464721694s) [noise=true , address=185.167.204.205:4004 , piece_id=PYGDQZMAKOVJWB7AWEPFMPMB6C24FMS4HT2UP2AKZNYGG4DGZR3Q, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.593818251 (407h9m15.464901286s) [noise=true , address=185.167.204.206:4048 , piece_id=52WC3XH2K2BHG33BQPRENPNYDT7EOKCNBCTZKE3LIONDJHVATEYA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.594252549 (407h9m15.465335594s) [noise=true , address=185.167.204.205:4034 , piece_id=OS5H57TG2REFE535DK4T6N75VNIXPLQWW6DM55QCC2S7QBVZM3UQ, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.597113216 (407h9m15.468196261s) [noise=true , address=185.167.204.205:4032 , piece_id=FJMXJG2PQ4WZDSDCPJ6LLSVQKI7JJHUCDUZLU2BJZV7RZTLETM5A, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.597394541 (407h9m15.468477586s) [noise=true , address=185.167.204.205:4035 , piece_id=3D7T4SGPAJQZCA74CUTR4GJRE23R4JD5KBNW3BLEASSP4GSR5PHQ, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.597612756 (407h9m15.468695792s) [noise=true , address=185.167.204.206:4043 , piece_id=HL32YEZF67X2UT6SRMXJVJ4W5A3PC5SXOCM7BYQVMEC76TKEVJRA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.598018900 (407h9m15.469101945s) [noise=true , address=185.167.204.205:4012 , piece_id=A5G37BVL2GZ7LAY3FZWIZV4R4M7M6UPYDYACJFRD5HKDWWKB4Q6A, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.598254719 (407h9m15.469337765s) [noise=true , address=185.167.204.206:4052 , piece_id=V2DVAAXN5A4C3NQW556VMAC4CKPNLATBLZLKAGDGZHVT7BHEPN3Q, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.598480649 (407h9m15.469563694s) [noise=true , address=185.167.204.205:4018 , piece_id=NUYRZ7PXKJQ3PIR44O3E6N6COZZPRDNSMTQCFI5RN7FBRV2CBALA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.598968278 (407h9m15.470051323s) [noise=true , address=185.167.204.205:4033 , piece_id=A32ZTBDBWMCNBMZPD4DUWS5NKUKQ22ACQH6HAW5AKB3FNHBC56VA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.599287886 (407h9m15.470370931s) [noise=true , address=185.167.204.206:4072 , piece_id=I6PCSC5JYTJXLT6B34TTIVZQVX52R7TRHSRLRP4ZHG5J63XDZN6Q, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.599632252 (407h9m15.470715297s) [noise=true , address=185.167.204.206:4070 , piece_id=Y7IRY4E2GCF7TCFOIQJ7QH6IFEJBMW4WM6QBIS5SCUJ7E7ASV4HA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.599676297 (407h9m15.470759332s) [noise=true , address=185.167.204.206:4071 , piece_id=DVJAKIMPAUGWQPHDPGTXASQDONAKOZIGX5PQDBGBXNQAPMBPUFBA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.601435877 (407h9m15.472518912s) [noise=true , address=185.167.204.206:4057 , piece_id=ONW7YS5JG44B2STNFJGJWFEZ3WYXO74N3CXI3C6T5OTJYHGQPQZQ, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:17.299689191 (407h9m17.170772226s) [noise=true , address=89.149.201.147:4019 , piece_id=ILE7ZX5X6HBPUVYGCOQ5SNG3AA6B3YTXIF47CPLJ3MN5EXVOS2OA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:17.526601953 (407h9m17.397684998s) [noise=true , address=89.149.201.147:4015 , piece_id=ZIWRTD7HSP7YMRJET2TTJQFAQZM7KGDKJDLMYCWWY2W6SV2UVQBQ, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:18.294911214 (407h9m18.165994249s) [noise=true , address=89.149.201.147:4017 , piece_id=ZOSNSMZOBQVIKD2ZGJRAUWJEJ4JXJK452YWQQSDL77E77F6LRDZA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:18.402191355 (407h9m18.2732744s) [noise=true , address=89.149.201.147:4011 , piece_id=ATMM77IGBDSRHNJ5FQ2VKP7W27JUA2F6HYJYSQ62IL5JQN5PF5OA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:22.436508909 (407h9m22.307591944s) [noise=true , address=81.171.14.70:4037 , piece_id=NYGC2ZLUSBUAGFULQ46LXCVNQUJ5M4XT3YICVA2ZMBR2D6WPJ27Q, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:25.935108368 (407h9m25.806191413s) [noise=true , address=81.171.14.70:4046 , piece_id=NRMV6FG5CZSTLUNP2EHNCK6VM2Z2UHKBLCHLKEPQ225Q5DGYEBPA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:30.904793996 (407h9m30.775877041s) [noise=true , address=81.171.14.70:4029 , piece_id=APDMOYEMDCLOREXY2BBOK73RSQVWKIXUX4SCHRVFDSM7ZLWBEQXA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:32.926010512 (407h9m32.797093567s) [noise=true , address=81.171.14.70:4044 , piece_id=QCWQO6JQRRKSG5U2KVH3Y6I3UI2F7G3DOW4CH3PQZKUVQ6Q2XOHA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:32.926036552 (407h9m32.797119587s) [seg_pos={0 0}, upload=357453]: Segment reached optimal threshold of 26 pieces. Dec 22 14:38:32.926196857 (407h9m32.797279902s) [noise=true , address=81.171.14.70:4033 , piece_id=MKNAJOFD5CIHTA55XO476LOJJGIWSA7HBQFQND2YNM562W74IKJA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BnjusiXEfHYwTDL8soMkm8GYnhbkwy2oCAxVA7oiDyhFXozZ8): protocol: expected piece hash; context canceled; EOF Dec 22 14:38:32.926203058 (407h9m32.797286093s) [noise=true , address=81.171.14.70:4033 , piece_id=MKNAJOFD5CIHTA55XO476LOJJGIWSA7HBQFQND2YNM562W74IKJA, seg_pos={0 0}, upload=357453]: Long tail cancellation due to optimal threshold Dec 22 14:38:32.926237574 (407h9m32.797320609s) [noise=true , address=81.171.14.70:4031 , piece_id=7MD5N7PPKFPCFMOIXAGI3MPBDDX45GTLMA3CFGNM7BBGI6TSL6RA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12drdYnnWsuMyFo6LkcewAGGWnorRCAhxPpNqK41LvGy2zkBvng): protocol: expected piece hash; context canceled; EOF Dec 22 14:38:32.926250028 (407h9m32.797333063s) [noise=true , address=81.171.14.70:4031 , piece_id=7MD5N7PPKFPCFMOIXAGI3MPBDDX45GTLMA3CFGNM7BBGI6TSL6RA, seg_pos={0 0}, upload=357453]: Long tail cancellation due to optimal threshold Dec 22 14:38:32.926299612 (407h9m32.797382647s) [seg_pos={0 0}, upload=357453]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 22 14:38:32.926403941 (407h9m32.797486986s) [upload=357453]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 14:38:32.926409692 (407h9m32.797492737s) [upload=357453]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 14:39:04.793379844 (407h10m4.664462889s) [upload=357453]: Done uploading segments... Dec 22 14:39:04.793413608 (407h10m4.664496643s) [upload=357453]: Upload finished. err: <nil> Duration: 6.7015587s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 1.462933083s trace: https://jaeger.dev.storj.io/trace/62b55ec615ab6123 FAIL (2025-12-22 14:58:09 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.462956367s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 1.43789209s trace: https://jaeger.dev.storj.io/trace/62b55ec615ab6123 FAIL (2025-12-22 14:57:51 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.437921926s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..X.........X........................X......................] duration: 5.442250717s trace: https://jaeger.dev.storj.io/trace/510cfc1b6c68e873 FAIL (2025-12-22 14:57:48 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.44226782s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 6.727203874s trace: https://jaeger.dev.storj.io/trace/718f722b1bda7fe2 FAIL (2025-12-22 14:57:48 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== VMAC4CKPNLATBLZLKAGDGZHVT7BHEPN3Q, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.367093115 (407h9m14.23817616s) [noise=true , address=185.167.204.206:4043 , piece_id=HL32YEZF67X2UT6SRMXJVJ4W5A3PC5SXOCM7BYQVMEC76TKEVJRA, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.367396732 (407h9m14.238479767s) [noise=true , address=185.167.204.205:4032 , piece_id=FJMXJG2PQ4WZDSDCPJ6LLSVQKI7JJHUCDUZLU2BJZV7RZTLETM5A, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.367549273 (407h9m14.238632308s) [noise=true , address=185.167.204.206:4070 , piece_id=Y7IRY4E2GCF7TCFOIQJ7QH6IFEJBMW4WM6QBIS5SCUJ7E7ASV4HA, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.367749263 (407h9m14.238832308s) [noise=true , address=81.171.14.70:4046 , piece_id=NRMV6FG5CZSTLUNP2EHNCK6VM2Z2UHKBLCHLKEPQ225Q5DGYEBPA, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.368034046 (407h9m14.239117091s) [noise=true , address=185.167.204.206:4060 , piece_id=WN6T5TUBVMURPSBBXGXCTLPZZ74PFNSRTTWXI2JPUTY74DPH3YSQ, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.368552724 (407h9m14.239635759s) [noise=true , address=81.171.14.70:4029 , piece_id=APDMOYEMDCLOREXY2BBOK73RSQVWKIXUX4SCHRVFDSM7ZLWBEQXA, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.369160691 (407h9m14.240243736s) [noise=true , address=185.167.204.206:4057 , piece_id=ONW7YS5JG44B2STNFJGJWFEZ3WYXO74N3CXI3C6T5OTJYHGQPQZQ, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.369714496 (407h9m14.240797541s) [noise=true , address=185.167.204.206:4071 , piece_id=DVJAKIMPAUGWQPHDPGTXASQDONAKOZIGX5PQDBGBXNQAPMBPUFBA, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.370249614 (407h9m14.241332649s) [noise=true , address=89.149.201.147:4019 , piece_id=ILE7ZX5X6HBPUVYGCOQ5SNG3AA6B3YTXIF47CPLJ3MN5EXVOS2OA, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.372019455 (407h9m14.24310249s) [noise=true , address=89.149.201.147:4015 , piece_id=ZIWRTD7HSP7YMRJET2TTJQFAQZM7KGDKJDLMYCWWY2W6SV2UVQBQ, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:15.199398795 (407h9m15.07048184s) [upload=357453]: Next returned nil segment Dec 22 14:38:15.199414966 (407h9m15.070498011s) [upload=357453]: Waiting for error group managing segments... Dec 22 14:38:15.591187572 (407h9m15.462270607s) [noise=true , address=185.167.204.205:4009 , piece_id=JTHX2O6T5HRENGEMAFJUWQUEUYAIG2KQ5HDEHF4BK42IT2ER7EIA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.591766795 (407h9m15.46284984s) [noise=true , address=185.167.204.205:4006 , piece_id=XHNFS6G7EX4BQWKOK4JUWYV3OTVJ36A6VOP4N7G2XPZKNNCFFKXA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.592074521 (407h9m15.463157566s) [noise=true , address=185.167.204.206:4060 , piece_id=WN6T5TUBVMURPSBBXGXCTLPZZ74PFNSRTTWXI2JPUTY74DPH3YSQ, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.593304694 (407h9m15.464387729s) [noise=true , address=185.167.204.205:4036 , piece_id=5QWJTTL6TZIZSSYHCBEJBKGX7BS76VVTELWTKEAKTOVYKINXVWDQ, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.593638659 (407h9m15.464721694s) [noise=true , address=185.167.204.205:4004 , piece_id=PYGDQZMAKOVJWB7AWEPFMPMB6C24FMS4HT2UP2AKZNYGG4DGZR3Q, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.593818251 (407h9m15.464901286s) [noise=true , address=185.167.204.206:4048 , piece_id=52WC3XH2K2BHG33BQPRENPNYDT7EOKCNBCTZKE3LIONDJHVATEYA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.594252549 (407h9m15.465335594s) [noise=true , address=185.167.204.205:4034 , piece_id=OS5H57TG2REFE535DK4T6N75VNIXPLQWW6DM55QCC2S7QBVZM3UQ, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.597113216 (407h9m15.468196261s) [noise=true , address=185.167.204.205:4032 , piece_id=FJMXJG2PQ4WZDSDCPJ6LLSVQKI7JJHUCDUZLU2BJZV7RZTLETM5A, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.597394541 (407h9m15.468477586s) [noise=true , address=185.167.204.205:4035 , piece_id=3D7T4SGPAJQZCA74CUTR4GJRE23R4JD5KBNW3BLEASSP4GSR5PHQ, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.597612756 (407h9m15.468695792s) [noise=true , address=185.167.204.206:4043 , piece_id=HL32YEZF67X2UT6SRMXJVJ4W5A3PC5SXOCM7BYQVMEC76TKEVJRA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.598018900 (407h9m15.469101945s) [noise=true , address=185.167.204.205:4012 , piece_id=A5G37BVL2GZ7LAY3FZWIZV4R4M7M6UPYDYACJFRD5HKDWWKB4Q6A, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.598254719 (407h9m15.469337765s) [noise=true , address=185.167.204.206:4052 , piece_id=V2DVAAXN5A4C3NQW556VMAC4CKPNLATBLZLKAGDGZHVT7BHEPN3Q, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.598480649 (407h9m15.469563694s) [noise=true , address=185.167.204.205:4018 , piece_id=NUYRZ7PXKJQ3PIR44O3E6N6COZZPRDNSMTQCFI5RN7FBRV2CBALA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.598968278 (407h9m15.470051323s) [noise=true , address=185.167.204.205:4033 , piece_id=A32ZTBDBWMCNBMZPD4DUWS5NKUKQ22ACQH6HAW5AKB3FNHBC56VA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.599287886 (407h9m15.470370931s) [noise=true , address=185.167.204.206:4072 , piece_id=I6PCSC5JYTJXLT6B34TTIVZQVX52R7TRHSRLRP4ZHG5J63XDZN6Q, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.599632252 (407h9m15.470715297s) [noise=true , address=185.167.204.206:4070 , piece_id=Y7IRY4E2GCF7TCFOIQJ7QH6IFEJBMW4WM6QBIS5SCUJ7E7ASV4HA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.599676297 (407h9m15.470759332s) [noise=true , address=185.167.204.206:4071 , piece_id=DVJAKIMPAUGWQPHDPGTXASQDONAKOZIGX5PQDBGBXNQAPMBPUFBA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.601435877 (407h9m15.472518912s) [noise=true , address=185.167.204.206:4057 , piece_id=ONW7YS5JG44B2STNFJGJWFEZ3WYXO74N3CXI3C6T5OTJYHGQPQZQ, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:17.299689191 (407h9m17.170772226s) [noise=true , address=89.149.201.147:4019 , piece_id=ILE7ZX5X6HBPUVYGCOQ5SNG3AA6B3YTXIF47CPLJ3MN5EXVOS2OA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:17.526601953 (407h9m17.397684998s) [noise=true , address=89.149.201.147:4015 , piece_id=ZIWRTD7HSP7YMRJET2TTJQFAQZM7KGDKJDLMYCWWY2W6SV2UVQBQ, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:18.294911214 (407h9m18.165994249s) [noise=true , address=89.149.201.147:4017 , piece_id=ZOSNSMZOBQVIKD2ZGJRAUWJEJ4JXJK452YWQQSDL77E77F6LRDZA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:18.402191355 (407h9m18.2732744s) [noise=true , address=89.149.201.147:4011 , piece_id=ATMM77IGBDSRHNJ5FQ2VKP7W27JUA2F6HYJYSQ62IL5JQN5PF5OA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:22.436508909 (407h9m22.307591944s) [noise=true , address=81.171.14.70:4037 , piece_id=NYGC2ZLUSBUAGFULQ46LXCVNQUJ5M4XT3YICVA2ZMBR2D6WPJ27Q, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:25.935108368 (407h9m25.806191413s) [noise=true , address=81.171.14.70:4046 , piece_id=NRMV6FG5CZSTLUNP2EHNCK6VM2Z2UHKBLCHLKEPQ225Q5DGYEBPA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:30.904793996 (407h9m30.775877041s) [noise=true , address=81.171.14.70:4029 , piece_id=APDMOYEMDCLOREXY2BBOK73RSQVWKIXUX4SCHRVFDSM7ZLWBEQXA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:32.926010512 (407h9m32.797093567s) [noise=true , address=81.171.14.70:4044 , piece_id=QCWQO6JQRRKSG5U2KVH3Y6I3UI2F7G3DOW4CH3PQZKUVQ6Q2XOHA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:32.926036552 (407h9m32.797119587s) [seg_pos={0 0}, upload=357453]: Segment reached optimal threshold of 26 pieces. Dec 22 14:38:32.926196857 (407h9m32.797279902s) [noise=true , address=81.171.14.70:4033 , piece_id=MKNAJOFD5CIHTA55XO476LOJJGIWSA7HBQFQND2YNM562W74IKJA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BnjusiXEfHYwTDL8soMkm8GYnhbkwy2oCAxVA7oiDyhFXozZ8): protocol: expected piece hash; context canceled; EOF Dec 22 14:38:32.926203058 (407h9m32.797286093s) [noise=true , address=81.171.14.70:4033 , piece_id=MKNAJOFD5CIHTA55XO476LOJJGIWSA7HBQFQND2YNM562W74IKJA, seg_pos={0 0}, upload=357453]: Long tail cancellation due to optimal threshold Dec 22 14:38:32.926237574 (407h9m32.797320609s) [noise=true , address=81.171.14.70:4031 , piece_id=7MD5N7PPKFPCFMOIXAGI3MPBDDX45GTLMA3CFGNM7BBGI6TSL6RA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12drdYnnWsuMyFo6LkcewAGGWnorRCAhxPpNqK41LvGy2zkBvng): protocol: expected piece hash; context canceled; EOF Dec 22 14:38:32.926250028 (407h9m32.797333063s) [noise=true , address=81.171.14.70:4031 , piece_id=7MD5N7PPKFPCFMOIXAGI3MPBDDX45GTLMA3CFGNM7BBGI6TSL6RA, seg_pos={0 0}, upload=357453]: Long tail cancellation due to optimal threshold Dec 22 14:38:32.926299612 (407h9m32.797382647s) [seg_pos={0 0}, upload=357453]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 22 14:38:32.926403941 (407h9m32.797486986s) [upload=357453]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 14:38:32.926409692 (407h9m32.797492737s) [upload=357453]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 14:39:04.793379844 (407h10m4.664462889s) [upload=357453]: Done uploading segments... Dec 22 14:39:04.793413608 (407h10m4.664496643s) [upload=357453]: Upload finished. err: <nil> Duration: 6.727257847s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [...X.....................................X.....XX.....XX.XXX] duration: 30.003636593s trace: https://jaeger.dev.storj.io/trace/8346ee2848b59a4 FAIL (2025-12-22 14:57:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: GUBSGVDKOQCI3XVHXF4AQKVK4Z45LQ6PUDU4RP5DVMKTOYSQSEFQ) context deadline exceeded; piecestore: (Node ID: 124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R, Piece ID: PREHHEY3GCAUFBRICMYX3ZXLSUFX47XQJA6DSMYG47ISZINDLSMA) context deadline exceeded ======LOGS====== gin Dec 22 14:57:06.585666389 (407h28m6.456749434s) [upload=357730]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 14:57:06.826648071 (407h28m6.697731106s) [seg_pos={0 0}, upload=357730]: Begin upload segment... Dec 22 14:57:06.826703417 (407h28m6.697786462s) [seg_pos={0 0}, upload=357730]: Done begin upload segment. Dec 22 14:57:06.826728024 (407h28m6.697811069s) [noise=true , address=45.91.116.10:4082 , piece_id=ZPISC2FI24MFSAMCJ6W2WSF7DFHEXK52Y76ZUFMMEWUIB4BKH6QQ, seg_pos={0 0}, upload=357730]: Uploading piece... Dec 22 14:57:06.826812735 (407h28m6.69789578s) [noise=true , address=77.90.31.13:4627 , piece_id=C6UIUBQLI4NGMO4QHM2WNCAYBOS75WG6ALGPYJQALTOFETGN4KQA, seg_pos={0 0}, upload=357730]: Uploading piece... Dec 22 14:57:06.827296637 (407h28m6.698379682s) [noise=true , address=145.79.227.16:4909 , piece_id=OY7OHYQMAM5XJUKD7YYRXJ2AJ7C7HFCPOOHHSPIF6Y7IWIWFV6TQ, seg_pos={0 0}, upload=357730]: Uploading piece... Dec 22 14:57:06.828610479 (407h28m6.699693514s) [noise=true , address=145.79.227.37:4565 , piece_id=2KB6QBEMXE2THRSHYXAKADRUW2WQXV7SJTLRAMZ5ZN32G7SAJ2KQ, seg_pos={0 0}, upload=357730]: Uploading piece... Dec 22 14:57:06.829088269 (407h28m6.700171314s) [noise=true , address=45.91.116.8:4046 , piece_id=GUBSGVDKOQCI3XVHXF4AQKVK4Z45LQ6PUDU4RP5DVMKTOYSQSEFQ, seg_pos={0 0}, upload=357730]: Uploading piece... Dec 22 14:57:06.829611174 (407h28m6.700694209s) [noise=true , address=145.79.227.7:4295 , piece_id=EG4RYR3JQTCQDKVBSZFBIITEOPFXRDJIXOFNY5KHHTESKXLWA5RQ, seg_pos={0 0}, upload=357730]: Uploading piece... Dec 22 14:57:06.830259458 (407h28m6.701342503s) [noise=true , address=145.79.227.49:4339 , piece_id=PREHHEY3GCAUFBRICMYX3ZXLSUFX47XQJA6DSMYG47ISZINDLSMA, seg_pos={0 0}, upload=357730]: Uploading piece... Dec 22 14:57:06.830281511 (407h28m6.701364546s) [noise=true , address=77.90.31.15:4700 , piece_id=QGXQQHCDZQVN7ORWP6GV4GUZLI2KPUVP6FP7WOR25EAMBUS55ZEA, seg_pos={0 0}, upload=357730]: Uploading piece... Dec 22 14:57:06.830732409 (407h28m6.701815454s) [noise=true , address=145.79.227.50:4375 , piece_id=YLJ2ZZWOV3ZMO4TRCIIVAGA2G2L4YDUT2GPB6V74PVRWDFOVI2SA, seg_pos={0 0}, upload=357730]: Uploading piece... Dec 22 14:57:06.830857817 (407h28m6.701940852s) [noise=true , address=145.79.227.45:4231 , piece_id=LLMQDXWUQ5U4CLLXWGUOD3KWHBIKKNGBFX3DFKKEGXAB42ERBLUQ, seg_pos={0 0}, upload=357730]: Uploading piece... Dec 22 14:57:06.831226019 (407h28m6.702309064s) [noise=true , address=145.79.227.14:4838 , piece_id=NN23H43OMVAYGBUFBYK53OMA5WSWVRXSBTDK3LENI6AJARDTLEUA, seg_pos={0 0}, upload=357730]: Uploading piece... Dec 22 14:57:06.831747191 (407h28m6.702830226s) [noise=true , address=45.91.116.11:4117 , piece_id=IMTI3EW3SGIYESZFOCHAZAESALCJX4WJ7RCZDF3YNG47THT67NVA, seg_pos={0 0}, upload=357730]: Uploading piece... Dec 22 14:57:06.837535503 (407h28m6.708618548s) [noise=true , address=145.79.227.39:4013 , piece_id=HXOXQKXI3GXZOI6Z25QUEFNELIW76X7YXSRR35FXXSJZY7XHOCKA, seg_pos={0 0}, upload=357730]: Uploading piece... Dec 22 14:57:06.840676564 (407h28m6.711759609s) [noise=true , address=145.79.227.10:4401 , piece_id=OYTVFOFZ3MHRM222G5RCV4PWKWBL4CFGRTMTQGQE2GPSYDWCLWPA, seg_pos={0 0}, upload=357730]: Uploading piece... Dec 22 14:57:06.843751480 (407h28m6.714834525s) [noise=true , address=145.79.227.42:4123 , piece_id=KIU6AZIZFB3A4V2E3VVII3IEXZCKPEXTDPGC6VLV2XCCINF464SA, seg_pos={0 0}, upload=357730]: Uploading piece... Dec 22 14:57:06.851435402 (407h28m6.722518447s) [noise=true , address=145.79.227.38:4978 , piece_id=37JIEZP3QPQFX42ENPPFYXB6PV3R26EKVNOQJRSZ4G43F2AXWEPQ, seg_pos={0 0}, upload=357730]: Uploading piece... Dec 22 14:57:08.704317683 (407h28m8.575400718s) [upload=357730]: Next returned nil segment Dec 22 14:57:08.704339764 (407h28m8.575422809s) [upload=357730]: Waiting for error group managing segments... Dec 22 14:57:09.053580882 (407h28m8.924663917s) [noise=true , address=77.90.31.13:4627 , piece_id=C6UIUBQLI4NGMO4QHM2WNCAYBOS75WG6ALGPYJQALTOFETGN4KQA, seg_pos={0 0}, upload=357730]: Done uploading piece. err: <nil> Dec 22 14:57:09.093528309 (407h28m8.964611344s) [noise=true , address=77.90.31.15:4700 , piece_id=QGXQQHCDZQVN7ORWP6GV4GUZLI2KPUVP6FP7WOR25EAMBUS55ZEA, seg_pos={0 0}, upload=357730]: Done uploading piece. err: <nil> Dec 22 14:57:09.223596563 (407h28m9.094679608s) [noise=true , address=45.91.116.10:4082 , piece_id=ZPISC2FI24MFSAMCJ6W2WSF7DFHEXK52Y76ZUFMMEWUIB4BKH6QQ, seg_pos={0 0}, upload=357730]: Done uploading piece. err: <nil> Dec 22 14:57:09.711125384 (407h28m9.582208419s) [noise=true , address=145.79.227.16:4909 , piece_id=OY7OHYQMAM5XJUKD7YYRXJ2AJ7C7HFCPOOHHSPIF6Y7IWIWFV6TQ, seg_pos={0 0}, upload=357730]: Done uploading piece. err: <nil> Dec 22 14:57:09.714556127 (407h28m9.585639163s) [noise=true , address=145.79.227.42:4123 , piece_id=KIU6AZIZFB3A4V2E3VVII3IEXZCKPEXTDPGC6VLV2XCCINF464SA, seg_pos={0 0}, upload=357730]: Done uploading piece. err: <nil> Dec 22 14:57:09.739027455 (407h28m9.61011049s) [noise=true , address=145.79.227.50:4375 , piece_id=YLJ2ZZWOV3ZMO4TRCIIVAGA2G2L4YDUT2GPB6V74PVRWDFOVI2SA, seg_pos={0 0}, upload=357730]: Done uploading piece. err: <nil> Dec 22 14:57:09.759075340 (407h28m9.630158385s) [noise=true , address=145.79.227.14:4838 , piece_id=NN23H43OMVAYGBUFBYK53OMA5WSWVRXSBTDK3LENI6AJARDTLEUA, seg_pos={0 0}, upload=357730]: Done uploading piece. err: <nil> Dec 22 14:57:09.763346002 (407h28m9.634429037s) [noise=true , address=145.79.227.38:4978 , piece_id=37JIEZP3QPQFX42ENPPFYXB6PV3R26EKVNOQJRSZ4G43F2AXWEPQ, seg_pos={0 0}, upload=357730]: Done uploading piece. err: <nil> Dec 22 14:57:09.819060453 (407h28m9.690143498s) [noise=true , address=145.79.227.49:4339 , piece_id=PREHHEY3GCAUFBRICMYX3ZXLSUFX47XQJA6DSMYG47ISZINDLSMA, seg_pos={0 0}, upload=357730]: Done uploading piece. err: <nil> Dec 22 14:57:09.833711351 (407h28m9.704794386s) [noise=true , address=145.79.227.37:4565 , piece_id=2KB6QBEMXE2THRSHYXAKADRUW2WQXV7SJTLRAMZ5ZN32G7SAJ2KQ, seg_pos={0 0}, upload=357730]: Done uploading piece. err: <nil> Dec 22 14:57:09.877554866 (407h28m9.748637911s) [noise=true , address=45.91.116.11:4117 , piece_id=IMTI3EW3SGIYESZFOCHAZAESALCJX4WJ7RCZDF3YNG47THT67NVA, seg_pos={0 0}, upload=357730]: Done uploading piece. err: <nil> Dec 22 14:57:10.295047255 (407h28m10.16613029s) [noise=true , address=45.91.116.8:4046 , piece_id=GUBSGVDKOQCI3XVHXF4AQKVK4Z45LQ6PUDU4RP5DVMKTOYSQSEFQ, seg_pos={0 0}, upload=357730]: Done uploading piece. err: <nil> Dec 22 14:57:10.556811611 (407h28m10.427894646s) [noise=true , address=145.79.227.45:4231 , piece_id=LLMQDXWUQ5U4CLLXWGUOD3KWHBIKKNGBFX3DFKKEGXAB42ERBLUQ, seg_pos={0 0}, upload=357730]: Done uploading piece. err: <nil> Dec 22 14:57:10.556835938 (407h28m10.427919003s) [seg_pos={0 0}, upload=357730]: Segment reached optimal threshold of 13 pieces. Dec 22 14:57:10.557058792 (407h28m10.428141827s) [noise=true , address=145.79.227.10:4401 , piece_id=OYTVFOFZ3MHRM222G5RCV4PWKWBL4CFGRTMTQGQE2GPSYDWCLWPA, seg_pos={0 0}, upload=357730]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1): protocol: write tcp 188.166.225.170:45318->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:45318->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45318->145.79.227.10:4401: use of closed network connection Dec 22 14:57:10.557068991 (407h28m10.428152046s) [noise=true , address=145.79.227.10:4401 , piece_id=OYTVFOFZ3MHRM222G5RCV4PWKWBL4CFGRTMTQGQE2GPSYDWCLWPA, seg_pos={0 0}, upload=357730]: Long tail cancellation due to optimal threshold Dec 22 14:57:10.557082677 (407h28m10.428165722s) [noise=true , address=145.79.227.7:4295 , piece_id=EG4RYR3JQTCQDKVBSZFBIITEOPFXRDJIXOFNY5KHHTESKXLWA5RQ, seg_pos={0 0}, upload=357730]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT): protocol: write tcp 188.166.225.170:33400->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:33400->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33400->145.79.227.7:4295: use of closed network connection Dec 22 14:57:10.557091213 (407h28m10.428174279s) [noise=true , address=145.79.227.7:4295 , piece_id=EG4RYR3JQTCQDKVBSZFBIITEOPFXRDJIXOFNY5KHHTESKXLWA5RQ, seg_pos={0 0}, upload=357730]: Long tail cancellation due to optimal threshold Dec 22 14:57:10.557244525 (407h28m10.42832758s) [noise=true , address=145.79.227.39:4013 , piece_id=HXOXQKXI3GXZOI6Z25QUEFNELIW76X7YXSRR35FXXSJZY7XHOCKA, seg_pos={0 0}, upload=357730]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b): protocol: write tcp 188.166.225.170:48618->145.79.227.39:4013: use of closed network connection; write tcp 188.166.225.170:48618->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48618->145.79.227.39:4013: use of closed network connection Dec 22 14:57:10.557252080 (407h28m10.428335125s) [noise=true , address=145.79.227.39:4013 , piece_id=HXOXQKXI3GXZOI6Z25QUEFNELIW76X7YXSRR35FXXSJZY7XHOCKA, seg_pos={0 0}, upload=357730]: Long tail cancellation due to optimal threshold Dec 22 14:57:10.557269312 (407h28m10.428352347s) [seg_pos={0 0}, upload=357730]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 14:57:10.557354004 (407h28m10.428437039s) [upload=357730]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 14:57:10.557361398 (407h28m10.428444453s) [upload=357730]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 14:57:10.853245193 (407h28m10.724328238s) [upload=357730]: Done uploading segments... Dec 22 14:57:10.853270732 (407h28m10.724353777s) [upload=357730]: Upload finished. err: <nil> Duration:* 30.003659206s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 6.493791145s trace: https://jaeger.dev.storj.io/trace/718f722b1bda7fe2 FAIL (2025-12-22 14:57:24 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== VMAC4CKPNLATBLZLKAGDGZHVT7BHEPN3Q, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.367093115 (407h9m14.23817616s) [noise=true , address=185.167.204.206:4043 , piece_id=HL32YEZF67X2UT6SRMXJVJ4W5A3PC5SXOCM7BYQVMEC76TKEVJRA, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.367396732 (407h9m14.238479767s) [noise=true , address=185.167.204.205:4032 , piece_id=FJMXJG2PQ4WZDSDCPJ6LLSVQKI7JJHUCDUZLU2BJZV7RZTLETM5A, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.367549273 (407h9m14.238632308s) [noise=true , address=185.167.204.206:4070 , piece_id=Y7IRY4E2GCF7TCFOIQJ7QH6IFEJBMW4WM6QBIS5SCUJ7E7ASV4HA, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.367749263 (407h9m14.238832308s) [noise=true , address=81.171.14.70:4046 , piece_id=NRMV6FG5CZSTLUNP2EHNCK6VM2Z2UHKBLCHLKEPQ225Q5DGYEBPA, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.368034046 (407h9m14.239117091s) [noise=true , address=185.167.204.206:4060 , piece_id=WN6T5TUBVMURPSBBXGXCTLPZZ74PFNSRTTWXI2JPUTY74DPH3YSQ, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.368552724 (407h9m14.239635759s) [noise=true , address=81.171.14.70:4029 , piece_id=APDMOYEMDCLOREXY2BBOK73RSQVWKIXUX4SCHRVFDSM7ZLWBEQXA, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.369160691 (407h9m14.240243736s) [noise=true , address=185.167.204.206:4057 , piece_id=ONW7YS5JG44B2STNFJGJWFEZ3WYXO74N3CXI3C6T5OTJYHGQPQZQ, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.369714496 (407h9m14.240797541s) [noise=true , address=185.167.204.206:4071 , piece_id=DVJAKIMPAUGWQPHDPGTXASQDONAKOZIGX5PQDBGBXNQAPMBPUFBA, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.370249614 (407h9m14.241332649s) [noise=true , address=89.149.201.147:4019 , piece_id=ILE7ZX5X6HBPUVYGCOQ5SNG3AA6B3YTXIF47CPLJ3MN5EXVOS2OA, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.372019455 (407h9m14.24310249s) [noise=true , address=89.149.201.147:4015 , piece_id=ZIWRTD7HSP7YMRJET2TTJQFAQZM7KGDKJDLMYCWWY2W6SV2UVQBQ, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:15.199398795 (407h9m15.07048184s) [upload=357453]: Next returned nil segment Dec 22 14:38:15.199414966 (407h9m15.070498011s) [upload=357453]: Waiting for error group managing segments... Dec 22 14:38:15.591187572 (407h9m15.462270607s) [noise=true , address=185.167.204.205:4009 , piece_id=JTHX2O6T5HRENGEMAFJUWQUEUYAIG2KQ5HDEHF4BK42IT2ER7EIA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.591766795 (407h9m15.46284984s) [noise=true , address=185.167.204.205:4006 , piece_id=XHNFS6G7EX4BQWKOK4JUWYV3OTVJ36A6VOP4N7G2XPZKNNCFFKXA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.592074521 (407h9m15.463157566s) [noise=true , address=185.167.204.206:4060 , piece_id=WN6T5TUBVMURPSBBXGXCTLPZZ74PFNSRTTWXI2JPUTY74DPH3YSQ, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.593304694 (407h9m15.464387729s) [noise=true , address=185.167.204.205:4036 , piece_id=5QWJTTL6TZIZSSYHCBEJBKGX7BS76VVTELWTKEAKTOVYKINXVWDQ, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.593638659 (407h9m15.464721694s) [noise=true , address=185.167.204.205:4004 , piece_id=PYGDQZMAKOVJWB7AWEPFMPMB6C24FMS4HT2UP2AKZNYGG4DGZR3Q, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.593818251 (407h9m15.464901286s) [noise=true , address=185.167.204.206:4048 , piece_id=52WC3XH2K2BHG33BQPRENPNYDT7EOKCNBCTZKE3LIONDJHVATEYA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.594252549 (407h9m15.465335594s) [noise=true , address=185.167.204.205:4034 , piece_id=OS5H57TG2REFE535DK4T6N75VNIXPLQWW6DM55QCC2S7QBVZM3UQ, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.597113216 (407h9m15.468196261s) [noise=true , address=185.167.204.205:4032 , piece_id=FJMXJG2PQ4WZDSDCPJ6LLSVQKI7JJHUCDUZLU2BJZV7RZTLETM5A, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.597394541 (407h9m15.468477586s) [noise=true , address=185.167.204.205:4035 , piece_id=3D7T4SGPAJQZCA74CUTR4GJRE23R4JD5KBNW3BLEASSP4GSR5PHQ, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.597612756 (407h9m15.468695792s) [noise=true , address=185.167.204.206:4043 , piece_id=HL32YEZF67X2UT6SRMXJVJ4W5A3PC5SXOCM7BYQVMEC76TKEVJRA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.598018900 (407h9m15.469101945s) [noise=true , address=185.167.204.205:4012 , piece_id=A5G37BVL2GZ7LAY3FZWIZV4R4M7M6UPYDYACJFRD5HKDWWKB4Q6A, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.598254719 (407h9m15.469337765s) [noise=true , address=185.167.204.206:4052 , piece_id=V2DVAAXN5A4C3NQW556VMAC4CKPNLATBLZLKAGDGZHVT7BHEPN3Q, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.598480649 (407h9m15.469563694s) [noise=true , address=185.167.204.205:4018 , piece_id=NUYRZ7PXKJQ3PIR44O3E6N6COZZPRDNSMTQCFI5RN7FBRV2CBALA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.598968278 (407h9m15.470051323s) [noise=true , address=185.167.204.205:4033 , piece_id=A32ZTBDBWMCNBMZPD4DUWS5NKUKQ22ACQH6HAW5AKB3FNHBC56VA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.599287886 (407h9m15.470370931s) [noise=true , address=185.167.204.206:4072 , piece_id=I6PCSC5JYTJXLT6B34TTIVZQVX52R7TRHSRLRP4ZHG5J63XDZN6Q, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.599632252 (407h9m15.470715297s) [noise=true , address=185.167.204.206:4070 , piece_id=Y7IRY4E2GCF7TCFOIQJ7QH6IFEJBMW4WM6QBIS5SCUJ7E7ASV4HA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.599676297 (407h9m15.470759332s) [noise=true , address=185.167.204.206:4071 , piece_id=DVJAKIMPAUGWQPHDPGTXASQDONAKOZIGX5PQDBGBXNQAPMBPUFBA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.601435877 (407h9m15.472518912s) [noise=true , address=185.167.204.206:4057 , piece_id=ONW7YS5JG44B2STNFJGJWFEZ3WYXO74N3CXI3C6T5OTJYHGQPQZQ, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:17.299689191 (407h9m17.170772226s) [noise=true , address=89.149.201.147:4019 , piece_id=ILE7ZX5X6HBPUVYGCOQ5SNG3AA6B3YTXIF47CPLJ3MN5EXVOS2OA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:17.526601953 (407h9m17.397684998s) [noise=true , address=89.149.201.147:4015 , piece_id=ZIWRTD7HSP7YMRJET2TTJQFAQZM7KGDKJDLMYCWWY2W6SV2UVQBQ, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:18.294911214 (407h9m18.165994249s) [noise=true , address=89.149.201.147:4017 , piece_id=ZOSNSMZOBQVIKD2ZGJRAUWJEJ4JXJK452YWQQSDL77E77F6LRDZA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:18.402191355 (407h9m18.2732744s) [noise=true , address=89.149.201.147:4011 , piece_id=ATMM77IGBDSRHNJ5FQ2VKP7W27JUA2F6HYJYSQ62IL5JQN5PF5OA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:22.436508909 (407h9m22.307591944s) [noise=true , address=81.171.14.70:4037 , piece_id=NYGC2ZLUSBUAGFULQ46LXCVNQUJ5M4XT3YICVA2ZMBR2D6WPJ27Q, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:25.935108368 (407h9m25.806191413s) [noise=true , address=81.171.14.70:4046 , piece_id=NRMV6FG5CZSTLUNP2EHNCK6VM2Z2UHKBLCHLKEPQ225Q5DGYEBPA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:30.904793996 (407h9m30.775877041s) [noise=true , address=81.171.14.70:4029 , piece_id=APDMOYEMDCLOREXY2BBOK73RSQVWKIXUX4SCHRVFDSM7ZLWBEQXA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:32.926010512 (407h9m32.797093567s) [noise=true , address=81.171.14.70:4044 , piece_id=QCWQO6JQRRKSG5U2KVH3Y6I3UI2F7G3DOW4CH3PQZKUVQ6Q2XOHA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:32.926036552 (407h9m32.797119587s) [seg_pos={0 0}, upload=357453]: Segment reached optimal threshold of 26 pieces. Dec 22 14:38:32.926196857 (407h9m32.797279902s) [noise=true , address=81.171.14.70:4033 , piece_id=MKNAJOFD5CIHTA55XO476LOJJGIWSA7HBQFQND2YNM562W74IKJA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BnjusiXEfHYwTDL8soMkm8GYnhbkwy2oCAxVA7oiDyhFXozZ8): protocol: expected piece hash; context canceled; EOF Dec 22 14:38:32.926203058 (407h9m32.797286093s) [noise=true , address=81.171.14.70:4033 , piece_id=MKNAJOFD5CIHTA55XO476LOJJGIWSA7HBQFQND2YNM562W74IKJA, seg_pos={0 0}, upload=357453]: Long tail cancellation due to optimal threshold Dec 22 14:38:32.926237574 (407h9m32.797320609s) [noise=true , address=81.171.14.70:4031 , piece_id=7MD5N7PPKFPCFMOIXAGI3MPBDDX45GTLMA3CFGNM7BBGI6TSL6RA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12drdYnnWsuMyFo6LkcewAGGWnorRCAhxPpNqK41LvGy2zkBvng): protocol: expected piece hash; context canceled; EOF Dec 22 14:38:32.926250028 (407h9m32.797333063s) [noise=true , address=81.171.14.70:4031 , piece_id=7MD5N7PPKFPCFMOIXAGI3MPBDDX45GTLMA3CFGNM7BBGI6TSL6RA, seg_pos={0 0}, upload=357453]: Long tail cancellation due to optimal threshold Dec 22 14:38:32.926299612 (407h9m32.797382647s) [seg_pos={0 0}, upload=357453]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 22 14:38:32.926403941 (407h9m32.797486986s) [upload=357453]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 14:38:32.926409692 (407h9m32.797492737s) [upload=357453]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 14:39:04.793379844 (407h10m4.664462889s) [upload=357453]: Done uploading segments... Dec 22 14:39:04.793413608 (407h10m4.664496643s) [upload=357453]: Upload finished. err: <nil> Duration: 6.493821754s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......X.....................................................] duration: 3.823630068s trace: https://jaeger.dev.storj.io/trace/1c84c110f15266bb FAIL (2025-12-22 14:57:21 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.823653602s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 4.980164412s trace: https://jaeger.dev.storj.io/trace/718f722b1bda7fe2 FAIL (2025-12-22 14:57:06 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== VMAC4CKPNLATBLZLKAGDGZHVT7BHEPN3Q, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.367093115 (407h9m14.23817616s) [noise=true , address=185.167.204.206:4043 , piece_id=HL32YEZF67X2UT6SRMXJVJ4W5A3PC5SXOCM7BYQVMEC76TKEVJRA, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.367396732 (407h9m14.238479767s) [noise=true , address=185.167.204.205:4032 , piece_id=FJMXJG2PQ4WZDSDCPJ6LLSVQKI7JJHUCDUZLU2BJZV7RZTLETM5A, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.367549273 (407h9m14.238632308s) [noise=true , address=185.167.204.206:4070 , piece_id=Y7IRY4E2GCF7TCFOIQJ7QH6IFEJBMW4WM6QBIS5SCUJ7E7ASV4HA, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.367749263 (407h9m14.238832308s) [noise=true , address=81.171.14.70:4046 , piece_id=NRMV6FG5CZSTLUNP2EHNCK6VM2Z2UHKBLCHLKEPQ225Q5DGYEBPA, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.368034046 (407h9m14.239117091s) [noise=true , address=185.167.204.206:4060 , piece_id=WN6T5TUBVMURPSBBXGXCTLPZZ74PFNSRTTWXI2JPUTY74DPH3YSQ, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.368552724 (407h9m14.239635759s) [noise=true , address=81.171.14.70:4029 , piece_id=APDMOYEMDCLOREXY2BBOK73RSQVWKIXUX4SCHRVFDSM7ZLWBEQXA, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.369160691 (407h9m14.240243736s) [noise=true , address=185.167.204.206:4057 , piece_id=ONW7YS5JG44B2STNFJGJWFEZ3WYXO74N3CXI3C6T5OTJYHGQPQZQ, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.369714496 (407h9m14.240797541s) [noise=true , address=185.167.204.206:4071 , piece_id=DVJAKIMPAUGWQPHDPGTXASQDONAKOZIGX5PQDBGBXNQAPMBPUFBA, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.370249614 (407h9m14.241332649s) [noise=true , address=89.149.201.147:4019 , piece_id=ILE7ZX5X6HBPUVYGCOQ5SNG3AA6B3YTXIF47CPLJ3MN5EXVOS2OA, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.372019455 (407h9m14.24310249s) [noise=true , address=89.149.201.147:4015 , piece_id=ZIWRTD7HSP7YMRJET2TTJQFAQZM7KGDKJDLMYCWWY2W6SV2UVQBQ, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:15.199398795 (407h9m15.07048184s) [upload=357453]: Next returned nil segment Dec 22 14:38:15.199414966 (407h9m15.070498011s) [upload=357453]: Waiting for error group managing segments... Dec 22 14:38:15.591187572 (407h9m15.462270607s) [noise=true , address=185.167.204.205:4009 , piece_id=JTHX2O6T5HRENGEMAFJUWQUEUYAIG2KQ5HDEHF4BK42IT2ER7EIA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.591766795 (407h9m15.46284984s) [noise=true , address=185.167.204.205:4006 , piece_id=XHNFS6G7EX4BQWKOK4JUWYV3OTVJ36A6VOP4N7G2XPZKNNCFFKXA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.592074521 (407h9m15.463157566s) [noise=true , address=185.167.204.206:4060 , piece_id=WN6T5TUBVMURPSBBXGXCTLPZZ74PFNSRTTWXI2JPUTY74DPH3YSQ, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.593304694 (407h9m15.464387729s) [noise=true , address=185.167.204.205:4036 , piece_id=5QWJTTL6TZIZSSYHCBEJBKGX7BS76VVTELWTKEAKTOVYKINXVWDQ, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.593638659 (407h9m15.464721694s) [noise=true , address=185.167.204.205:4004 , piece_id=PYGDQZMAKOVJWB7AWEPFMPMB6C24FMS4HT2UP2AKZNYGG4DGZR3Q, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.593818251 (407h9m15.464901286s) [noise=true , address=185.167.204.206:4048 , piece_id=52WC3XH2K2BHG33BQPRENPNYDT7EOKCNBCTZKE3LIONDJHVATEYA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.594252549 (407h9m15.465335594s) [noise=true , address=185.167.204.205:4034 , piece_id=OS5H57TG2REFE535DK4T6N75VNIXPLQWW6DM55QCC2S7QBVZM3UQ, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.597113216 (407h9m15.468196261s) [noise=true , address=185.167.204.205:4032 , piece_id=FJMXJG2PQ4WZDSDCPJ6LLSVQKI7JJHUCDUZLU2BJZV7RZTLETM5A, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.597394541 (407h9m15.468477586s) [noise=true , address=185.167.204.205:4035 , piece_id=3D7T4SGPAJQZCA74CUTR4GJRE23R4JD5KBNW3BLEASSP4GSR5PHQ, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.597612756 (407h9m15.468695792s) [noise=true , address=185.167.204.206:4043 , piece_id=HL32YEZF67X2UT6SRMXJVJ4W5A3PC5SXOCM7BYQVMEC76TKEVJRA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.598018900 (407h9m15.469101945s) [noise=true , address=185.167.204.205:4012 , piece_id=A5G37BVL2GZ7LAY3FZWIZV4R4M7M6UPYDYACJFRD5HKDWWKB4Q6A, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.598254719 (407h9m15.469337765s) [noise=true , address=185.167.204.206:4052 , piece_id=V2DVAAXN5A4C3NQW556VMAC4CKPNLATBLZLKAGDGZHVT7BHEPN3Q, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.598480649 (407h9m15.469563694s) [noise=true , address=185.167.204.205:4018 , piece_id=NUYRZ7PXKJQ3PIR44O3E6N6COZZPRDNSMTQCFI5RN7FBRV2CBALA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.598968278 (407h9m15.470051323s) [noise=true , address=185.167.204.205:4033 , piece_id=A32ZTBDBWMCNBMZPD4DUWS5NKUKQ22ACQH6HAW5AKB3FNHBC56VA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.599287886 (407h9m15.470370931s) [noise=true , address=185.167.204.206:4072 , piece_id=I6PCSC5JYTJXLT6B34TTIVZQVX52R7TRHSRLRP4ZHG5J63XDZN6Q, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.599632252 (407h9m15.470715297s) [noise=true , address=185.167.204.206:4070 , piece_id=Y7IRY4E2GCF7TCFOIQJ7QH6IFEJBMW4WM6QBIS5SCUJ7E7ASV4HA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.599676297 (407h9m15.470759332s) [noise=true , address=185.167.204.206:4071 , piece_id=DVJAKIMPAUGWQPHDPGTXASQDONAKOZIGX5PQDBGBXNQAPMBPUFBA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.601435877 (407h9m15.472518912s) [noise=true , address=185.167.204.206:4057 , piece_id=ONW7YS5JG44B2STNFJGJWFEZ3WYXO74N3CXI3C6T5OTJYHGQPQZQ, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:17.299689191 (407h9m17.170772226s) [noise=true , address=89.149.201.147:4019 , piece_id=ILE7ZX5X6HBPUVYGCOQ5SNG3AA6B3YTXIF47CPLJ3MN5EXVOS2OA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:17.526601953 (407h9m17.397684998s) [noise=true , address=89.149.201.147:4015 , piece_id=ZIWRTD7HSP7YMRJET2TTJQFAQZM7KGDKJDLMYCWWY2W6SV2UVQBQ, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:18.294911214 (407h9m18.165994249s) [noise=true , address=89.149.201.147:4017 , piece_id=ZOSNSMZOBQVIKD2ZGJRAUWJEJ4JXJK452YWQQSDL77E77F6LRDZA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:18.402191355 (407h9m18.2732744s) [noise=true , address=89.149.201.147:4011 , piece_id=ATMM77IGBDSRHNJ5FQ2VKP7W27JUA2F6HYJYSQ62IL5JQN5PF5OA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:22.436508909 (407h9m22.307591944s) [noise=true , address=81.171.14.70:4037 , piece_id=NYGC2ZLUSBUAGFULQ46LXCVNQUJ5M4XT3YICVA2ZMBR2D6WPJ27Q, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:25.935108368 (407h9m25.806191413s) [noise=true , address=81.171.14.70:4046 , piece_id=NRMV6FG5CZSTLUNP2EHNCK6VM2Z2UHKBLCHLKEPQ225Q5DGYEBPA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:30.904793996 (407h9m30.775877041s) [noise=true , address=81.171.14.70:4029 , piece_id=APDMOYEMDCLOREXY2BBOK73RSQVWKIXUX4SCHRVFDSM7ZLWBEQXA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:32.926010512 (407h9m32.797093567s) [noise=true , address=81.171.14.70:4044 , piece_id=QCWQO6JQRRKSG5U2KVH3Y6I3UI2F7G3DOW4CH3PQZKUVQ6Q2XOHA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:32.926036552 (407h9m32.797119587s) [seg_pos={0 0}, upload=357453]: Segment reached optimal threshold of 26 pieces. Dec 22 14:38:32.926196857 (407h9m32.797279902s) [noise=true , address=81.171.14.70:4033 , piece_id=MKNAJOFD5CIHTA55XO476LOJJGIWSA7HBQFQND2YNM562W74IKJA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BnjusiXEfHYwTDL8soMkm8GYnhbkwy2oCAxVA7oiDyhFXozZ8): protocol: expected piece hash; context canceled; EOF Dec 22 14:38:32.926203058 (407h9m32.797286093s) [noise=true , address=81.171.14.70:4033 , piece_id=MKNAJOFD5CIHTA55XO476LOJJGIWSA7HBQFQND2YNM562W74IKJA, seg_pos={0 0}, upload=357453]: Long tail cancellation due to optimal threshold Dec 22 14:38:32.926237574 (407h9m32.797320609s) [noise=true , address=81.171.14.70:4031 , piece_id=7MD5N7PPKFPCFMOIXAGI3MPBDDX45GTLMA3CFGNM7BBGI6TSL6RA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12drdYnnWsuMyFo6LkcewAGGWnorRCAhxPpNqK41LvGy2zkBvng): protocol: expected piece hash; context canceled; EOF Dec 22 14:38:32.926250028 (407h9m32.797333063s) [noise=true , address=81.171.14.70:4031 , piece_id=7MD5N7PPKFPCFMOIXAGI3MPBDDX45GTLMA3CFGNM7BBGI6TSL6RA, seg_pos={0 0}, upload=357453]: Long tail cancellation due to optimal threshold Dec 22 14:38:32.926299612 (407h9m32.797382647s) [seg_pos={0 0}, upload=357453]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 22 14:38:32.926403941 (407h9m32.797486986s) [upload=357453]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 14:38:32.926409692 (407h9m32.797492737s) [upload=357453]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 14:39:04.793379844 (407h10m4.664462889s) [upload=357453]: Done uploading segments... Dec 22 14:39:04.793413608 (407h10m4.664496643s) [upload=357453]: Upload finished. err: <nil> Duration: 4.980193857s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 1.474320942s trace: https://jaeger.dev.storj.io/trace/62b55ec615ab6123 FAIL (2025-12-22 14:56:40 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.474348234s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [....X.....................................X.....XX.....XX.XX] duration: 30.002886418s trace: https://jaeger.dev.storj.io/trace/3d081604108f2f35 FAIL (2025-12-22 14:56:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 22 14:56:09.117361928 (407h27m8.988444963s) [upload=357718]: Starting upload Dec 22 14:56:09.117472889 (407h27m8.988555934s) [upload=357718]: Done starting upload Dec 22 14:56:09.118063183 (407h27m8.989146228s) [upload=357718]: Uploading segments... Dec 22 14:56:09.118241082 (407h27m8.989324127s) [upload=357718]: Got next segment. Inline: false Dec 22 14:56:09.118429421 (407h27m8.989512456s) [upload=357718]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 14:56:09.118438207 (407h27m8.989521252s) [upload=357718]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 14:56:09.370174899 (407h27m9.241257954s) [seg_pos={0 0}, upload=357718]: Begin upload segment... Dec 22 14:56:09.370226297 (407h27m9.241309332s) [seg_pos={0 0}, upload=357718]: Done begin upload segment. Dec 22 14:56:09.370258268 (407h27m9.241341313s) [noise=true , address=145.79.227.42:4123 , piece_id=ACMXRCHK3TWI3AM4M5P5M3YJYWIHSIG2FXY3FD74LWGZHFI7K7FA, seg_pos={0 0}, upload=357718]: Uploading piece... Dec 22 14:56:09.370298244 (407h27m9.241381289s) [noise=true , address=145.79.227.14:4838 , piece_id=XDGSRHEIBPOZHP6SJHIE35WSTZ7YINTA2RSW5E42FA7QSMRX6WYQ, seg_pos={0 0}, upload=357718]: Uploading piece... Dec 22 14:56:09.371000832 (407h27m9.242083867s) [noise=true , address=145.79.227.7:4295 , piece_id=2QVC5GNZTVO4DGAUP3F2FQLXFFOTJB7HSZ7ZSSFO4W7PDEXUOBSA, seg_pos={0 0}, upload=357718]: Uploading piece... Dec 22 14:56:09.371319729 (407h27m9.242402774s) [noise=true , address=145.79.227.16:4911 , piece_id=RBVWQIHF4NOSK6SQACTY4THY2JC72G6K7IPQXWLFT72OAJZYPBTA, seg_pos={0 0}, upload=357718]: Uploading piece... Dec 22 14:56:09.371578693 (407h27m9.242661728s) [noise=true , address=45.91.116.8:4045 , piece_id=HMORPEQJ7MO5CIK454KMODCRSG2UGW62AGJYBT5EDQJ7QXSEHVXQ, seg_pos={0 0}, upload=357718]: Uploading piece... Dec 22 14:56:09.371841713 (407h27m9.242924758s) [noise=true , address=77.90.31.10:4520 , piece_id=MA3UM2GEW6LAY4QJGMYB3KV3M5I2HUIUSCSM3DQ2MDTEA4SQ3CFQ, seg_pos={0 0}, upload=357718]: Uploading piece... Dec 22 14:56:09.372146143 (407h27m9.243229178s) [noise=true , address=45.91.116.6:4976 , piece_id=IEURRFPVBADYQYZ7HADT6RE5KWNBVQ4YVE45WHH3VVYMDTUEX2SA, seg_pos={0 0}, upload=357718]: Uploading piece... Dec 22 14:56:09.372241455 (407h27m9.2433245s) [noise=true , address=145.79.227.41:4085 , piece_id=ZU3LEQSNQPH2FL7ENMWF24XNABLFJVR3U5CRA7SYW5MWL35YS4HQ, seg_pos={0 0}, upload=357718]: Uploading piece... Dec 22 14:56:09.372806801 (407h27m9.243889846s) [noise=true , address=145.79.227.40:4050 , piece_id=26QD2JQDDWPG2N7MSB4H5YPAOQC4QU5QEMNSPHGD6K2XLV5JN5DQ, seg_pos={0 0}, upload=357718]: Uploading piece... Dec 22 14:56:09.372856456 (407h27m9.243939501s) [noise=true , address=145.79.227.52:4446 , piece_id=75PSFFAM42KKWILRZOCADP7VUM6ECNVKD2IANCHR6DYUNWXJFPXQ, seg_pos={0 0}, upload=357718]: Uploading piece... Dec 22 14:56:09.373356598 (407h27m9.244439643s) [noise=true , address=145.79.227.50:4374 , piece_id=SZO6INZ3OWCPGTRRCFU5FHK7ZXHFAVP5TWHQ2GS2YKZRWGYOXVJA, seg_pos={0 0}, upload=357718]: Uploading piece... Dec 22 14:56:09.373400171 (407h27m9.244483216s) [noise=true , address=145.79.227.36:4519 , piece_id=MO4MLWPNYTNCX2MGQX6CSR4AQYOHTB53GVROXP4EFUNJAOYETDNA, seg_pos={0 0}, upload=357718]: Uploading piece... Dec 22 14:56:09.374438558 (407h27m9.245521603s) [noise=true , address=45.91.116.4:4939 , piece_id=2ZTAATX2RTT6XBSZ3GP4NWWR6JBSWG7XDRDJVL36WXEQQKT6VWFQ, seg_pos={0 0}, upload=357718]: Uploading piece... Dec 22 14:56:09.374638088 (407h27m9.245721133s) [noise=true , address=145.79.227.37:4566 , piece_id=HPMHQQ2PTGSHPDH7WO3DVQBQ3GFTR77M54ERNJ6F7Z2B667EWXSA, seg_pos={0 0}, upload=357718]: Uploading piece... Dec 22 14:56:09.375451677 (407h27m9.246534722s) [noise=true , address=145.79.227.35:4482 , piece_id=NL5NKEHKD4SS7SMWFII5H4FYVQYTL4YELRTAMWMHHIYD26PEHE2Q, seg_pos={0 0}, upload=357718]: Uploading piece... Dec 22 14:56:09.376229057 (407h27m9.247312092s) [noise=true , address=145.79.227.9:4367 , piece_id=N4EEOPRT4F6ESC2Y55RMDVU22I4UDO5DBG2NSX7PMHX5U3MZDHRA, seg_pos={0 0}, upload=357718]: Uploading piece... Dec 22 14:56:11.302705716 (407h27m11.173788761s) [upload=357718]: Next returned nil segment Dec 22 14:56:11.302722377 (407h27m11.173805422s) [upload=357718]: Waiting for error group managing segments... Dec 22 14:56:11.645011558 (407h27m11.516094593s) [noise=true , address=45.91.116.6:4976 , piece_id=IEURRFPVBADYQYZ7HADT6RE5KWNBVQ4YVE45WHH3VVYMDTUEX2SA, seg_pos={0 0}, upload=357718]: Done uploading piece. err: <nil> Dec 22 14:56:11.673643307 (407h27m11.544726342s) [noise=true , address=45.91.116.4:4939 , piece_id=2ZTAATX2RTT6XBSZ3GP4NWWR6JBSWG7XDRDJVL36WXEQQKT6VWFQ, seg_pos={0 0}, upload=357718]: Done uploading piece. err: <nil> Dec 22 14:56:11.756381115 (407h27m11.62746415s) [noise=true , address=45.91.116.8:4045 , piece_id=HMORPEQJ7MO5CIK454KMODCRSG2UGW62AGJYBT5EDQJ7QXSEHVXQ, seg_pos={0 0}, upload=357718]: Done uploading piece. err: <nil> Dec 22 14:56:12.248227470 (407h27m12.119310505s) [noise=true , address=145.79.227.41:4085 , piece_id=ZU3LEQSNQPH2FL7ENMWF24XNABLFJVR3U5CRA7SYW5MWL35YS4HQ, seg_pos={0 0}, upload=357718]: Done uploading piece. err: <nil> Dec 22 14:56:12.280967482 (407h27m12.152050527s) [noise=true , address=145.79.227.16:4911 , piece_id=RBVWQIHF4NOSK6SQACTY4THY2JC72G6K7IPQXWLFT72OAJZYPBTA, seg_pos={0 0}, upload=357718]: Done uploading piece. err: <nil> Dec 22 14:56:12.287848966 (407h27m12.158932001s) [noise=true , address=145.79.227.36:4519 , piece_id=MO4MLWPNYTNCX2MGQX6CSR4AQYOHTB53GVROXP4EFUNJAOYETDNA, seg_pos={0 0}, upload=357718]: Done uploading piece. err: <nil> Dec 22 14:56:12.330391805 (407h27m12.20147485s) [noise=true , address=145.79.227.14:4838 , piece_id=XDGSRHEIBPOZHP6SJHIE35WSTZ7YINTA2RSW5E42FA7QSMRX6WYQ, seg_pos={0 0}, upload=357718]: Done uploading piece. err: <nil> Dec 22 14:56:12.348344981 (407h27m12.219428016s) [noise=true , address=145.79.227.7:4295 , piece_id=2QVC5GNZTVO4DGAUP3F2FQLXFFOTJB7HSZ7ZSSFO4W7PDEXUOBSA, seg_pos={0 0}, upload=357718]: Done uploading piece. err: <nil> Dec 22 14:56:12.372610005 (407h27m12.24369305s) [noise=true , address=145.79.227.50:4374 , piece_id=SZO6INZ3OWCPGTRRCFU5FHK7ZXHFAVP5TWHQ2GS2YKZRWGYOXVJA, seg_pos={0 0}, upload=357718]: Done uploading piece. err: <nil> Dec 22 14:56:12.398830745 (407h27m12.26991379s) [noise=true , address=145.79.227.52:4446 , piece_id=75PSFFAM42KKWILRZOCADP7VUM6ECNVKD2IANCHR6DYUNWXJFPXQ, seg_pos={0 0}, upload=357718]: Done uploading piece. err: <nil> Dec 22 14:56:13.061270499 (407h27m12.932353534s) [noise=true , address=77.90.31.10:4520 , piece_id=MA3UM2GEW6LAY4QJGMYB3KV3M5I2HUIUSCSM3DQ2MDTEA4SQ3CFQ, seg_pos={0 0}, upload=357718]: Done uploading piece. err: <nil> Dec 22 14:56:13.237472439 (407h27m13.108555484s) [noise=true , address=145.79.227.9:4367 , piece_id=N4EEOPRT4F6ESC2Y55RMDVU22I4UDO5DBG2NSX7PMHX5U3MZDHRA, seg_pos={0 0}, upload=357718]: Done uploading piece. err: <nil> Dec 22 14:56:13.246477807 (407h27m13.117560852s) [noise=true , address=145.79.227.35:4482 , piece_id=NL5NKEHKD4SS7SMWFII5H4FYVQYTL4YELRTAMWMHHIYD26PEHE2Q, seg_pos={0 0}, upload=357718]: Done uploading piece. err: <nil> Dec 22 14:56:13.246508305 (407h27m13.11759135s) [seg_pos={0 0}, upload=357718]: Segment reached optimal threshold of 13 pieces. Dec 22 14:56:13.246771507 (407h27m13.117854552s) [noise=true , address=145.79.227.40:4050 , piece_id=26QD2JQDDWPG2N7MSB4H5YPAOQC4QU5QEMNSPHGD6K2XLV5JN5DQ, seg_pos={0 0}, upload=357718]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 22 14:56:13.246778851 (407h27m13.117861896s) [noise=true , address=145.79.227.40:4050 , piece_id=26QD2JQDDWPG2N7MSB4H5YPAOQC4QU5QEMNSPHGD6K2XLV5JN5DQ, seg_pos={0 0}, upload=357718]: Long tail cancellation due to optimal threshold Dec 22 14:56:13.247016934 (407h27m13.118099969s) [noise=true , address=145.79.227.37:4566 , piece_id=HPMHQQ2PTGSHPDH7WO3DVQBQ3GFTR77M54ERNJ6F7Z2B667EWXSA, seg_pos={0 0}, upload=357718]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SaNJZaVix7BSbEWhu2Vmt3XjDDWCsQ8wTSiFR7MQ8oGwD9N2F): protocol: expected piece hash; context canceled; EOF Dec 22 14:56:13.247042433 (407h27m13.118125478s) [noise=true , address=145.79.227.37:4566 , piece_id=HPMHQQ2PTGSHPDH7WO3DVQBQ3GFTR77M54ERNJ6F7Z2B667EWXSA, seg_pos={0 0}, upload=357718]: Long tail cancellation due to optimal threshold Dec 22 14:56:13.247075085 (407h27m13.11815813s) [noise=true , address=145.79.227.42:4123 , piece_id=ACMXRCHK3TWI3AM4M5P5M3YJYWIHSIG2FXY3FD74LWGZHFI7K7FA, seg_pos={0 0}, upload=357718]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM): protocol: expected piece hash; context canceled; EOF Dec 22 14:56:13.247083691 (407h27m13.118166736s) [noise=true , address=145.79.227.42:4123 , piece_id=ACMXRCHK3TWI3AM4M5P5M3YJYWIHSIG2FXY3FD74LWGZHFI7K7FA, seg_pos={0 0}, upload=357718]: Long tail cancellation due to optimal threshold Dec 22 14:56:13.247167831 (407h27m13.118250876s) [seg_pos={0 0}, upload=357718]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 14:56:13.247257472 (407h27m13.118340517s) [upload=357718]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 14:56:13.247263464 (407h27m13.118346509s) [upload=357718]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 14:56:13.553468693 (407h27m13.424551738s) [upload=357718]: Done uploading segments... Dec 22 14:56:13.553494633 (407h27m13.424577668s) [upload=357718]: Upload finished. err: <nil> Duration: 30.002907759s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...X.........X........................X.....................] duration: 5.879712553s trace: https://jaeger.dev.storj.io/trace/510cfc1b6c68e873 FAIL (2025-12-22 14:56:35 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.879731218s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......X....................................................] duration: 3.46123263s trace: https://jaeger.dev.storj.io/trace/1c84c110f15266bb FAIL (2025-12-22 14:56:22 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.4612481s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 1.480310708s trace: https://jaeger.dev.storj.io/trace/62b55ec615ab6123 FAIL (2025-12-22 14:56:19 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.480339122s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 6.570252333s trace: https://jaeger.dev.storj.io/trace/718f722b1bda7fe2 FAIL (2025-12-22 14:55:53 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== VMAC4CKPNLATBLZLKAGDGZHVT7BHEPN3Q, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.367093115 (407h9m14.23817616s) [noise=true , address=185.167.204.206:4043 , piece_id=HL32YEZF67X2UT6SRMXJVJ4W5A3PC5SXOCM7BYQVMEC76TKEVJRA, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.367396732 (407h9m14.238479767s) [noise=true , address=185.167.204.205:4032 , piece_id=FJMXJG2PQ4WZDSDCPJ6LLSVQKI7JJHUCDUZLU2BJZV7RZTLETM5A, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.367549273 (407h9m14.238632308s) [noise=true , address=185.167.204.206:4070 , piece_id=Y7IRY4E2GCF7TCFOIQJ7QH6IFEJBMW4WM6QBIS5SCUJ7E7ASV4HA, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.367749263 (407h9m14.238832308s) [noise=true , address=81.171.14.70:4046 , piece_id=NRMV6FG5CZSTLUNP2EHNCK6VM2Z2UHKBLCHLKEPQ225Q5DGYEBPA, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.368034046 (407h9m14.239117091s) [noise=true , address=185.167.204.206:4060 , piece_id=WN6T5TUBVMURPSBBXGXCTLPZZ74PFNSRTTWXI2JPUTY74DPH3YSQ, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.368552724 (407h9m14.239635759s) [noise=true , address=81.171.14.70:4029 , piece_id=APDMOYEMDCLOREXY2BBOK73RSQVWKIXUX4SCHRVFDSM7ZLWBEQXA, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.369160691 (407h9m14.240243736s) [noise=true , address=185.167.204.206:4057 , piece_id=ONW7YS5JG44B2STNFJGJWFEZ3WYXO74N3CXI3C6T5OTJYHGQPQZQ, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.369714496 (407h9m14.240797541s) [noise=true , address=185.167.204.206:4071 , piece_id=DVJAKIMPAUGWQPHDPGTXASQDONAKOZIGX5PQDBGBXNQAPMBPUFBA, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.370249614 (407h9m14.241332649s) [noise=true , address=89.149.201.147:4019 , piece_id=ILE7ZX5X6HBPUVYGCOQ5SNG3AA6B3YTXIF47CPLJ3MN5EXVOS2OA, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.372019455 (407h9m14.24310249s) [noise=true , address=89.149.201.147:4015 , piece_id=ZIWRTD7HSP7YMRJET2TTJQFAQZM7KGDKJDLMYCWWY2W6SV2UVQBQ, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:15.199398795 (407h9m15.07048184s) [upload=357453]: Next returned nil segment Dec 22 14:38:15.199414966 (407h9m15.070498011s) [upload=357453]: Waiting for error group managing segments... Dec 22 14:38:15.591187572 (407h9m15.462270607s) [noise=true , address=185.167.204.205:4009 , piece_id=JTHX2O6T5HRENGEMAFJUWQUEUYAIG2KQ5HDEHF4BK42IT2ER7EIA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.591766795 (407h9m15.46284984s) [noise=true , address=185.167.204.205:4006 , piece_id=XHNFS6G7EX4BQWKOK4JUWYV3OTVJ36A6VOP4N7G2XPZKNNCFFKXA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.592074521 (407h9m15.463157566s) [noise=true , address=185.167.204.206:4060 , piece_id=WN6T5TUBVMURPSBBXGXCTLPZZ74PFNSRTTWXI2JPUTY74DPH3YSQ, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.593304694 (407h9m15.464387729s) [noise=true , address=185.167.204.205:4036 , piece_id=5QWJTTL6TZIZSSYHCBEJBKGX7BS76VVTELWTKEAKTOVYKINXVWDQ, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.593638659 (407h9m15.464721694s) [noise=true , address=185.167.204.205:4004 , piece_id=PYGDQZMAKOVJWB7AWEPFMPMB6C24FMS4HT2UP2AKZNYGG4DGZR3Q, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.593818251 (407h9m15.464901286s) [noise=true , address=185.167.204.206:4048 , piece_id=52WC3XH2K2BHG33BQPRENPNYDT7EOKCNBCTZKE3LIONDJHVATEYA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.594252549 (407h9m15.465335594s) [noise=true , address=185.167.204.205:4034 , piece_id=OS5H57TG2REFE535DK4T6N75VNIXPLQWW6DM55QCC2S7QBVZM3UQ, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.597113216 (407h9m15.468196261s) [noise=true , address=185.167.204.205:4032 , piece_id=FJMXJG2PQ4WZDSDCPJ6LLSVQKI7JJHUCDUZLU2BJZV7RZTLETM5A, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.597394541 (407h9m15.468477586s) [noise=true , address=185.167.204.205:4035 , piece_id=3D7T4SGPAJQZCA74CUTR4GJRE23R4JD5KBNW3BLEASSP4GSR5PHQ, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.597612756 (407h9m15.468695792s) [noise=true , address=185.167.204.206:4043 , piece_id=HL32YEZF67X2UT6SRMXJVJ4W5A3PC5SXOCM7BYQVMEC76TKEVJRA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.598018900 (407h9m15.469101945s) [noise=true , address=185.167.204.205:4012 , piece_id=A5G37BVL2GZ7LAY3FZWIZV4R4M7M6UPYDYACJFRD5HKDWWKB4Q6A, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.598254719 (407h9m15.469337765s) [noise=true , address=185.167.204.206:4052 , piece_id=V2DVAAXN5A4C3NQW556VMAC4CKPNLATBLZLKAGDGZHVT7BHEPN3Q, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.598480649 (407h9m15.469563694s) [noise=true , address=185.167.204.205:4018 , piece_id=NUYRZ7PXKJQ3PIR44O3E6N6COZZPRDNSMTQCFI5RN7FBRV2CBALA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.598968278 (407h9m15.470051323s) [noise=true , address=185.167.204.205:4033 , piece_id=A32ZTBDBWMCNBMZPD4DUWS5NKUKQ22ACQH6HAW5AKB3FNHBC56VA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.599287886 (407h9m15.470370931s) [noise=true , address=185.167.204.206:4072 , piece_id=I6PCSC5JYTJXLT6B34TTIVZQVX52R7TRHSRLRP4ZHG5J63XDZN6Q, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.599632252 (407h9m15.470715297s) [noise=true , address=185.167.204.206:4070 , piece_id=Y7IRY4E2GCF7TCFOIQJ7QH6IFEJBMW4WM6QBIS5SCUJ7E7ASV4HA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.599676297 (407h9m15.470759332s) [noise=true , address=185.167.204.206:4071 , piece_id=DVJAKIMPAUGWQPHDPGTXASQDONAKOZIGX5PQDBGBXNQAPMBPUFBA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.601435877 (407h9m15.472518912s) [noise=true , address=185.167.204.206:4057 , piece_id=ONW7YS5JG44B2STNFJGJWFEZ3WYXO74N3CXI3C6T5OTJYHGQPQZQ, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:17.299689191 (407h9m17.170772226s) [noise=true , address=89.149.201.147:4019 , piece_id=ILE7ZX5X6HBPUVYGCOQ5SNG3AA6B3YTXIF47CPLJ3MN5EXVOS2OA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:17.526601953 (407h9m17.397684998s) [noise=true , address=89.149.201.147:4015 , piece_id=ZIWRTD7HSP7YMRJET2TTJQFAQZM7KGDKJDLMYCWWY2W6SV2UVQBQ, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:18.294911214 (407h9m18.165994249s) [noise=true , address=89.149.201.147:4017 , piece_id=ZOSNSMZOBQVIKD2ZGJRAUWJEJ4JXJK452YWQQSDL77E77F6LRDZA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:18.402191355 (407h9m18.2732744s) [noise=true , address=89.149.201.147:4011 , piece_id=ATMM77IGBDSRHNJ5FQ2VKP7W27JUA2F6HYJYSQ62IL5JQN5PF5OA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:22.436508909 (407h9m22.307591944s) [noise=true , address=81.171.14.70:4037 , piece_id=NYGC2ZLUSBUAGFULQ46LXCVNQUJ5M4XT3YICVA2ZMBR2D6WPJ27Q, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:25.935108368 (407h9m25.806191413s) [noise=true , address=81.171.14.70:4046 , piece_id=NRMV6FG5CZSTLUNP2EHNCK6VM2Z2UHKBLCHLKEPQ225Q5DGYEBPA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:30.904793996 (407h9m30.775877041s) [noise=true , address=81.171.14.70:4029 , piece_id=APDMOYEMDCLOREXY2BBOK73RSQVWKIXUX4SCHRVFDSM7ZLWBEQXA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:32.926010512 (407h9m32.797093567s) [noise=true , address=81.171.14.70:4044 , piece_id=QCWQO6JQRRKSG5U2KVH3Y6I3UI2F7G3DOW4CH3PQZKUVQ6Q2XOHA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:32.926036552 (407h9m32.797119587s) [seg_pos={0 0}, upload=357453]: Segment reached optimal threshold of 26 pieces. Dec 22 14:38:32.926196857 (407h9m32.797279902s) [noise=true , address=81.171.14.70:4033 , piece_id=MKNAJOFD5CIHTA55XO476LOJJGIWSA7HBQFQND2YNM562W74IKJA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BnjusiXEfHYwTDL8soMkm8GYnhbkwy2oCAxVA7oiDyhFXozZ8): protocol: expected piece hash; context canceled; EOF Dec 22 14:38:32.926203058 (407h9m32.797286093s) [noise=true , address=81.171.14.70:4033 , piece_id=MKNAJOFD5CIHTA55XO476LOJJGIWSA7HBQFQND2YNM562W74IKJA, seg_pos={0 0}, upload=357453]: Long tail cancellation due to optimal threshold Dec 22 14:38:32.926237574 (407h9m32.797320609s) [noise=true , address=81.171.14.70:4031 , piece_id=7MD5N7PPKFPCFMOIXAGI3MPBDDX45GTLMA3CFGNM7BBGI6TSL6RA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12drdYnnWsuMyFo6LkcewAGGWnorRCAhxPpNqK41LvGy2zkBvng): protocol: expected piece hash; context canceled; EOF Dec 22 14:38:32.926250028 (407h9m32.797333063s) [noise=true , address=81.171.14.70:4031 , piece_id=7MD5N7PPKFPCFMOIXAGI3MPBDDX45GTLMA3CFGNM7BBGI6TSL6RA, seg_pos={0 0}, upload=357453]: Long tail cancellation due to optimal threshold Dec 22 14:38:32.926299612 (407h9m32.797382647s) [seg_pos={0 0}, upload=357453]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 22 14:38:32.926403941 (407h9m32.797486986s) [upload=357453]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 14:38:32.926409692 (407h9m32.797492737s) [upload=357453]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 14:39:04.793379844 (407h10m4.664462889s) [upload=357453]: Done uploading segments... Dec 22 14:39:04.793413608 (407h10m4.664496643s) [upload=357453]: Upload finished. err: <nil> Duration: 6.570336704s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 1.480031417s trace: https://jaeger.dev.storj.io/trace/62b55ec615ab6123 FAIL (2025-12-22 14:55:47 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.480055943s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [X.......X...................................................] duration: 3.51830193s trace: https://jaeger.dev.storj.io/trace/1c84c110f15266bb FAIL (2025-12-22 14:55:44 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.518329833s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....X.........X........................X....................] duration: 6.911986716s trace: https://jaeger.dev.storj.io/trace/510cfc1b6c68e873 FAIL (2025-12-22 14:55:33 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 6.91201071s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.....X.....................................X.....XX.....XX.X] duration: 30.00406486s trace: https://jaeger.dev.storj.io/trace/bcd993425e39ff7 FAIL (2025-12-22 14:55:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12QRixhWaUwXPdjcWKYHbiknME5JVomWYABY6egDD6VKwaTPB8K, Piece ID: 4YE7OQ2GTCBD6CN3WFD5IBGQAXNCQDB4TG33SN25G7IQQLG5IPPQ) context deadline exceeded; piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: HOUHOC73KMD4TM62QI7CL6Q7FWBPVLQWNZWEA65SKIS5JC2WMBBQ) context deadline exceeded ======LOGS====== 54:56.308785435 (407h25m56.17986847s) [upload=357700]: Done starting upload Dec 22 14:54:56.308877360 (407h25m56.179960395s) [upload=357700]: Uploading segments... Dec 22 14:54:56.308957483 (407h25m56.180040518s) [upload=357700]: Got next segment. Inline: false Dec 22 14:54:56.308990676 (407h25m56.180073711s) [upload=357700]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 14:54:56.308995415 (407h25m56.18007845s) [upload=357700]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 14:54:56.662484412 (407h25m56.533567447s) [seg_pos={0 0}, upload=357700]: Begin upload segment... Dec 22 14:54:56.662529327 (407h25m56.533612373s) [seg_pos={0 0}, upload=357700]: Done begin upload segment. Dec 22 14:54:56.662555357 (407h25m56.533638402s) [noise=true , address=145.79.227.44:4193 , piece_id=LUKRVULQBAH533ND3ESTSXLTJMIHZTMM7EC6LRUEYH2HK7LCL3OQ, seg_pos={0 0}, upload=357700]: Uploading piece... Dec 22 14:54:56.662562871 (407h25m56.533645906s) [noise=true , address=77.90.31.17:4773 , piece_id=W34N3DSBNZT2LMPFJBJE54XWP3FZXMJI2IJJT6KMZE2MVQ6HKVNA, seg_pos={0 0}, upload=357700]: Uploading piece... Dec 22 14:54:56.662615030 (407h25m56.533698296s) [noise=true , address=145.79.227.48:4301 , piece_id=D2PKSU3IIMRQPQI34LODAHP2YOCPUQTQNRK6FG3MH3CXTN6GMPJA, seg_pos={0 0}, upload=357700]: Uploading piece... Dec 22 14:54:56.662974105 (407h25m56.53405714s) [noise=true , address=145.79.227.19:4078 , piece_id=Z3HDQI74WENASMIC56I3C4GHVDK3G7GYAFUCTP66WZRSRFMSHZ3A, seg_pos={0 0}, upload=357700]: Uploading piece... Dec 22 14:54:56.663737178 (407h25m56.534820213s) [noise=true , address=145.79.227.35:4482 , piece_id=PKKSXQQF27HIDB4TG2B7LFYLA5272WVFZ675MODAJZSOKBVO67GQ, seg_pos={0 0}, upload=357700]: Uploading piece... Dec 22 14:54:56.664849105 (407h25m56.53593215s) [noise=true , address=145.79.227.50:4374 , piece_id=4YE7OQ2GTCBD6CN3WFD5IBGQAXNCQDB4TG33SN25G7IQQLG5IPPQ, seg_pos={0 0}, upload=357700]: Uploading piece... Dec 22 14:54:56.664981356 (407h25m56.536064401s) [noise=true , address=145.79.227.43:4157 , piece_id=KMMXFBNH45FQ5G7ZXON4LGNIEVFNG3HRPOK2R66DEGGBRG3FSRVA, seg_pos={0 0}, upload=357700]: Uploading piece... Dec 22 14:54:56.666870955 (407h25m56.537954s) [noise=true , address=45.91.116.10:4082 , piece_id=IY2BGGV5EK7Q7GUZTARD24NNVVPGAXKRJMGOUPR5VO4BZ73SJC7Q, seg_pos={0 0}, upload=357700]: Uploading piece... Dec 22 14:54:56.667855379 (407h25m56.538938414s) [noise=true , address=145.79.227.34:4906 , piece_id=L7OY37WIR2MKFDWXTPCZNMQ4N7N5P4TDULX3JD6Z67KBE3KNWEGQ, seg_pos={0 0}, upload=357700]: Uploading piece... Dec 22 14:54:56.668693124 (407h25m56.539776159s) [noise=true , address=45.91.116.9:4262 , piece_id=WC6BWZXJW4DGCJ2A2TCCQQ57R3R2J6VQJI4WU4MNO37SHHV55QVQ, seg_pos={0 0}, upload=357700]: Uploading piece... Dec 22 14:54:56.670197489 (407h25m56.541280524s) [noise=true , address=145.79.227.33:4869 , piece_id=QJAOEGRNOT2OVXHLQP6KKGT6AAIQMX5LCBVYXNGOU6XDJZIATKHA, seg_pos={0 0}, upload=357700]: Uploading piece... Dec 22 14:54:56.670804014 (407h25m56.541887049s) [noise=true , address=77.90.31.16:4736 , piece_id=OCWPCLYFEHCN667JDOUSISUDSIWBXY26S573GMWWTXFFN3OBTIEA, seg_pos={0 0}, upload=357700]: Uploading piece... Dec 22 14:54:56.671127139 (407h25m56.542210184s) [noise=true , address=145.79.227.45:4229 , piece_id=GJXQ5FTZGHFSFCVY43LT7GCBVBW7KOGSZI5L7BQWIOHGAV356FDQ, seg_pos={0 0}, upload=357700]: Uploading piece... Dec 22 14:54:56.671575162 (407h25m56.542658207s) [noise=true , address=45.91.116.13:4156 , piece_id=HOUHOC73KMD4TM62QI7CL6Q7FWBPVLQWNZWEA65SKIS5JC2WMBBQ, seg_pos={0 0}, upload=357700]: Uploading piece... Dec 22 14:54:56.672846132 (407h25m56.543929177s) [noise=true , address=145.79.227.51:4410 , piece_id=TVCHOF5SFG4YN3YK6F5HAHKU2HMNW6GKRXEVESQQKUCLMCRBTJCQ, seg_pos={0 0}, upload=357700]: Uploading piece... Dec 22 14:54:56.673731999 (407h25m56.544815034s) [noise=true , address=145.79.227.8:4331 , piece_id=3U4MWTHDJVIZYNX4NISBJGDBDRT2SHCXTCEPE34PH777KRFECUWQ, seg_pos={0 0}, upload=357700]: Uploading piece... Dec 22 14:54:58.489610478 (407h25m58.360693513s) [upload=357700]: Next returned nil segment Dec 22 14:54:58.489624345 (407h25m58.36070738s) [upload=357700]: Waiting for error group managing segments... Dec 22 14:54:58.946631900 (407h25m58.817714945s) [noise=true , address=45.91.116.13:4156 , piece_id=HOUHOC73KMD4TM62QI7CL6Q7FWBPVLQWNZWEA65SKIS5JC2WMBBQ, seg_pos={0 0}, upload=357700]: Done uploading piece. err: <nil> Dec 22 14:54:58.950265088 (407h25m58.821348123s) [noise=true , address=45.91.116.9:4262 , piece_id=WC6BWZXJW4DGCJ2A2TCCQQ57R3R2J6VQJI4WU4MNO37SHHV55QVQ, seg_pos={0 0}, upload=357700]: Done uploading piece. err: <nil> Dec 22 14:54:58.978693962 (407h25m58.849776997s) [noise=true , address=45.91.116.10:4082 , piece_id=IY2BGGV5EK7Q7GUZTARD24NNVVPGAXKRJMGOUPR5VO4BZ73SJC7Q, seg_pos={0 0}, upload=357700]: Done uploading piece. err: <nil> Dec 22 14:54:58.988690076 (407h25m58.859773111s) [noise=true , address=77.90.31.17:4773 , piece_id=W34N3DSBNZT2LMPFJBJE54XWP3FZXMJI2IJJT6KMZE2MVQ6HKVNA, seg_pos={0 0}, upload=357700]: Done uploading piece. err: <nil> Dec 22 14:54:59.592900742 (407h25m59.463983777s) [noise=true , address=145.79.227.43:4157 , piece_id=KMMXFBNH45FQ5G7ZXON4LGNIEVFNG3HRPOK2R66DEGGBRG3FSRVA, seg_pos={0 0}, upload=357700]: Done uploading piece. err: <nil> Dec 22 14:54:59.599278016 (407h25m59.470361091s) [noise=true , address=145.79.227.51:4410 , piece_id=TVCHOF5SFG4YN3YK6F5HAHKU2HMNW6GKRXEVESQQKUCLMCRBTJCQ, seg_pos={0 0}, upload=357700]: Done uploading piece. err: <nil> Dec 22 14:54:59.614172758 (407h25m59.485255803s) [noise=true , address=145.79.227.45:4229 , piece_id=GJXQ5FTZGHFSFCVY43LT7GCBVBW7KOGSZI5L7BQWIOHGAV356FDQ, seg_pos={0 0}, upload=357700]: Done uploading piece. err: <nil> Dec 22 14:54:59.622464557 (407h25m59.493547592s) [noise=true , address=145.79.227.34:4906 , piece_id=L7OY37WIR2MKFDWXTPCZNMQ4N7N5P4TDULX3JD6Z67KBE3KNWEGQ, seg_pos={0 0}, upload=357700]: Done uploading piece. err: <nil> Dec 22 14:54:59.657509489 (407h25m59.528592524s) [noise=true , address=145.79.227.19:4078 , piece_id=Z3HDQI74WENASMIC56I3C4GHVDK3G7GYAFUCTP66WZRSRFMSHZ3A, seg_pos={0 0}, upload=357700]: Done uploading piece. err: <nil> Dec 22 14:54:59.662526572 (407h25m59.533609617s) [noise=true , address=145.79.227.50:4374 , piece_id=4YE7OQ2GTCBD6CN3WFD5IBGQAXNCQDB4TG33SN25G7IQQLG5IPPQ, seg_pos={0 0}, upload=357700]: Done uploading piece. err: <nil> Dec 22 14:54:59.664524858 (407h25m59.535607893s) [noise=true , address=145.79.227.48:4301 , piece_id=D2PKSU3IIMRQPQI34LODAHP2YOCPUQTQNRK6FG3MH3CXTN6GMPJA, seg_pos={0 0}, upload=357700]: Done uploading piece. err: <nil> Dec 22 14:55:00.166128680 (407h26m0.037211715s) [noise=true , address=77.90.31.16:4736 , piece_id=OCWPCLYFEHCN667JDOUSISUDSIWBXY26S573GMWWTXFFN3OBTIEA, seg_pos={0 0}, upload=357700]: Done uploading piece. err: <nil> Dec 22 14:55:00.554447339 (407h26m0.425530384s) [noise=true , address=145.79.227.35:4482 , piece_id=PKKSXQQF27HIDB4TG2B7LFYLA5272WVFZ675MODAJZSOKBVO67GQ, seg_pos={0 0}, upload=357700]: Done uploading piece. err: <nil> Dec 22 14:55:00.554470433 (407h26m0.425553468s) [seg_pos={0 0}, upload=357700]: Segment reached optimal threshold of 13 pieces. Dec 22 14:55:00.554718165 (407h26m0.42580121s) [noise=true , address=145.79.227.44:4193 , piece_id=LUKRVULQBAH533ND3ESTSXLTJMIHZTMM7EC6LRUEYH2HK7LCL3OQ, seg_pos={0 0}, upload=357700]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 22 14:55:00.554728895 (407h26m0.42581193s) [noise=true , address=145.79.227.44:4193 , piece_id=LUKRVULQBAH533ND3ESTSXLTJMIHZTMM7EC6LRUEYH2HK7LCL3OQ, seg_pos={0 0}, upload=357700]: Long tail cancellation due to optimal threshold Dec 22 14:55:00.554738894 (407h26m0.425821939s) [noise=true , address=145.79.227.33:4869 , piece_id=QJAOEGRNOT2OVXHLQP6KKGT6AAIQMX5LCBVYXNGOU6XDJZIATKHA, seg_pos={0 0}, upload=357700]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: write tcp 188.166.225.170:46634->145.79.227.33:4869: use of closed network connection; write tcp 188.166.225.170:46634->145.79.227.33:4869: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46634->145.79.227.33:4869: use of closed network connection Dec 22 14:55:00.554745547 (407h26m0.425828592s) [noise=true , address=145.79.227.33:4869 , piece_id=QJAOEGRNOT2OVXHLQP6KKGT6AAIQMX5LCBVYXNGOU6XDJZIATKHA, seg_pos={0 0}, upload=357700]: Long tail cancellation due to optimal threshold Dec 22 14:55:00.555056900 (407h26m0.426139935s) [noise=true , address=145.79.227.8:4331 , piece_id=3U4MWTHDJVIZYNX4NISBJGDBDRT2SHCXTCEPE34PH777KRFECUWQ, seg_pos={0 0}, upload=357700]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU): protocol: read tcp 188.166.225.170:52740->145.79.227.8:4331: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 22 14:55:00.555064865 (407h26m0.42614791s) [noise=true , address=145.79.227.8:4331 , piece_id=3U4MWTHDJVIZYNX4NISBJGDBDRT2SHCXTCEPE34PH777KRFECUWQ, seg_pos={0 0}, upload=357700]: Long tail cancellation due to optimal threshold Dec 22 14:55:00.555116603 (407h26m0.426199648s) [seg_pos={0 0}, upload=357700]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 14:55:00.555207637 (407h26m0.426290682s) [upload=357700]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 14:55:00.555212937 (407h26m0.426295972s) [upload=357700]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 14:55:01.205776598 (407h26m1.076859643s) [upload=357700]: Done uploading segments... Dec 22 14:55:01.205897257 (407h26m1.076980302s) [upload=357700]: Upload finished. err: <nil> Duration: 30.004077884s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 6.723007707s trace: https://jaeger.dev.storj.io/trace/718f722b1bda7fe2 FAIL (2025-12-22 14:55:17 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== VMAC4CKPNLATBLZLKAGDGZHVT7BHEPN3Q, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.367093115 (407h9m14.23817616s) [noise=true , address=185.167.204.206:4043 , piece_id=HL32YEZF67X2UT6SRMXJVJ4W5A3PC5SXOCM7BYQVMEC76TKEVJRA, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.367396732 (407h9m14.238479767s) [noise=true , address=185.167.204.205:4032 , piece_id=FJMXJG2PQ4WZDSDCPJ6LLSVQKI7JJHUCDUZLU2BJZV7RZTLETM5A, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.367549273 (407h9m14.238632308s) [noise=true , address=185.167.204.206:4070 , piece_id=Y7IRY4E2GCF7TCFOIQJ7QH6IFEJBMW4WM6QBIS5SCUJ7E7ASV4HA, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.367749263 (407h9m14.238832308s) [noise=true , address=81.171.14.70:4046 , piece_id=NRMV6FG5CZSTLUNP2EHNCK6VM2Z2UHKBLCHLKEPQ225Q5DGYEBPA, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.368034046 (407h9m14.239117091s) [noise=true , address=185.167.204.206:4060 , piece_id=WN6T5TUBVMURPSBBXGXCTLPZZ74PFNSRTTWXI2JPUTY74DPH3YSQ, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.368552724 (407h9m14.239635759s) [noise=true , address=81.171.14.70:4029 , piece_id=APDMOYEMDCLOREXY2BBOK73RSQVWKIXUX4SCHRVFDSM7ZLWBEQXA, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.369160691 (407h9m14.240243736s) [noise=true , address=185.167.204.206:4057 , piece_id=ONW7YS5JG44B2STNFJGJWFEZ3WYXO74N3CXI3C6T5OTJYHGQPQZQ, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.369714496 (407h9m14.240797541s) [noise=true , address=185.167.204.206:4071 , piece_id=DVJAKIMPAUGWQPHDPGTXASQDONAKOZIGX5PQDBGBXNQAPMBPUFBA, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.370249614 (407h9m14.241332649s) [noise=true , address=89.149.201.147:4019 , piece_id=ILE7ZX5X6HBPUVYGCOQ5SNG3AA6B3YTXIF47CPLJ3MN5EXVOS2OA, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.372019455 (407h9m14.24310249s) [noise=true , address=89.149.201.147:4015 , piece_id=ZIWRTD7HSP7YMRJET2TTJQFAQZM7KGDKJDLMYCWWY2W6SV2UVQBQ, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:15.199398795 (407h9m15.07048184s) [upload=357453]: Next returned nil segment Dec 22 14:38:15.199414966 (407h9m15.070498011s) [upload=357453]: Waiting for error group managing segments... Dec 22 14:38:15.591187572 (407h9m15.462270607s) [noise=true , address=185.167.204.205:4009 , piece_id=JTHX2O6T5HRENGEMAFJUWQUEUYAIG2KQ5HDEHF4BK42IT2ER7EIA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.591766795 (407h9m15.46284984s) [noise=true , address=185.167.204.205:4006 , piece_id=XHNFS6G7EX4BQWKOK4JUWYV3OTVJ36A6VOP4N7G2XPZKNNCFFKXA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.592074521 (407h9m15.463157566s) [noise=true , address=185.167.204.206:4060 , piece_id=WN6T5TUBVMURPSBBXGXCTLPZZ74PFNSRTTWXI2JPUTY74DPH3YSQ, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.593304694 (407h9m15.464387729s) [noise=true , address=185.167.204.205:4036 , piece_id=5QWJTTL6TZIZSSYHCBEJBKGX7BS76VVTELWTKEAKTOVYKINXVWDQ, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.593638659 (407h9m15.464721694s) [noise=true , address=185.167.204.205:4004 , piece_id=PYGDQZMAKOVJWB7AWEPFMPMB6C24FMS4HT2UP2AKZNYGG4DGZR3Q, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.593818251 (407h9m15.464901286s) [noise=true , address=185.167.204.206:4048 , piece_id=52WC3XH2K2BHG33BQPRENPNYDT7EOKCNBCTZKE3LIONDJHVATEYA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.594252549 (407h9m15.465335594s) [noise=true , address=185.167.204.205:4034 , piece_id=OS5H57TG2REFE535DK4T6N75VNIXPLQWW6DM55QCC2S7QBVZM3UQ, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.597113216 (407h9m15.468196261s) [noise=true , address=185.167.204.205:4032 , piece_id=FJMXJG2PQ4WZDSDCPJ6LLSVQKI7JJHUCDUZLU2BJZV7RZTLETM5A, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.597394541 (407h9m15.468477586s) [noise=true , address=185.167.204.205:4035 , piece_id=3D7T4SGPAJQZCA74CUTR4GJRE23R4JD5KBNW3BLEASSP4GSR5PHQ, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.597612756 (407h9m15.468695792s) [noise=true , address=185.167.204.206:4043 , piece_id=HL32YEZF67X2UT6SRMXJVJ4W5A3PC5SXOCM7BYQVMEC76TKEVJRA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.598018900 (407h9m15.469101945s) [noise=true , address=185.167.204.205:4012 , piece_id=A5G37BVL2GZ7LAY3FZWIZV4R4M7M6UPYDYACJFRD5HKDWWKB4Q6A, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.598254719 (407h9m15.469337765s) [noise=true , address=185.167.204.206:4052 , piece_id=V2DVAAXN5A4C3NQW556VMAC4CKPNLATBLZLKAGDGZHVT7BHEPN3Q, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.598480649 (407h9m15.469563694s) [noise=true , address=185.167.204.205:4018 , piece_id=NUYRZ7PXKJQ3PIR44O3E6N6COZZPRDNSMTQCFI5RN7FBRV2CBALA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.598968278 (407h9m15.470051323s) [noise=true , address=185.167.204.205:4033 , piece_id=A32ZTBDBWMCNBMZPD4DUWS5NKUKQ22ACQH6HAW5AKB3FNHBC56VA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.599287886 (407h9m15.470370931s) [noise=true , address=185.167.204.206:4072 , piece_id=I6PCSC5JYTJXLT6B34TTIVZQVX52R7TRHSRLRP4ZHG5J63XDZN6Q, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.599632252 (407h9m15.470715297s) [noise=true , address=185.167.204.206:4070 , piece_id=Y7IRY4E2GCF7TCFOIQJ7QH6IFEJBMW4WM6QBIS5SCUJ7E7ASV4HA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.599676297 (407h9m15.470759332s) [noise=true , address=185.167.204.206:4071 , piece_id=DVJAKIMPAUGWQPHDPGTXASQDONAKOZIGX5PQDBGBXNQAPMBPUFBA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.601435877 (407h9m15.472518912s) [noise=true , address=185.167.204.206:4057 , piece_id=ONW7YS5JG44B2STNFJGJWFEZ3WYXO74N3CXI3C6T5OTJYHGQPQZQ, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:17.299689191 (407h9m17.170772226s) [noise=true , address=89.149.201.147:4019 , piece_id=ILE7ZX5X6HBPUVYGCOQ5SNG3AA6B3YTXIF47CPLJ3MN5EXVOS2OA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:17.526601953 (407h9m17.397684998s) [noise=true , address=89.149.201.147:4015 , piece_id=ZIWRTD7HSP7YMRJET2TTJQFAQZM7KGDKJDLMYCWWY2W6SV2UVQBQ, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:18.294911214 (407h9m18.165994249s) [noise=true , address=89.149.201.147:4017 , piece_id=ZOSNSMZOBQVIKD2ZGJRAUWJEJ4JXJK452YWQQSDL77E77F6LRDZA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:18.402191355 (407h9m18.2732744s) [noise=true , address=89.149.201.147:4011 , piece_id=ATMM77IGBDSRHNJ5FQ2VKP7W27JUA2F6HYJYSQ62IL5JQN5PF5OA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:22.436508909 (407h9m22.307591944s) [noise=true , address=81.171.14.70:4037 , piece_id=NYGC2ZLUSBUAGFULQ46LXCVNQUJ5M4XT3YICVA2ZMBR2D6WPJ27Q, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:25.935108368 (407h9m25.806191413s) [noise=true , address=81.171.14.70:4046 , piece_id=NRMV6FG5CZSTLUNP2EHNCK6VM2Z2UHKBLCHLKEPQ225Q5DGYEBPA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:30.904793996 (407h9m30.775877041s) [noise=true , address=81.171.14.70:4029 , piece_id=APDMOYEMDCLOREXY2BBOK73RSQVWKIXUX4SCHRVFDSM7ZLWBEQXA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:32.926010512 (407h9m32.797093567s) [noise=true , address=81.171.14.70:4044 , piece_id=QCWQO6JQRRKSG5U2KVH3Y6I3UI2F7G3DOW4CH3PQZKUVQ6Q2XOHA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:32.926036552 (407h9m32.797119587s) [seg_pos={0 0}, upload=357453]: Segment reached optimal threshold of 26 pieces. Dec 22 14:38:32.926196857 (407h9m32.797279902s) [noise=true , address=81.171.14.70:4033 , piece_id=MKNAJOFD5CIHTA55XO476LOJJGIWSA7HBQFQND2YNM562W74IKJA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BnjusiXEfHYwTDL8soMkm8GYnhbkwy2oCAxVA7oiDyhFXozZ8): protocol: expected piece hash; context canceled; EOF Dec 22 14:38:32.926203058 (407h9m32.797286093s) [noise=true , address=81.171.14.70:4033 , piece_id=MKNAJOFD5CIHTA55XO476LOJJGIWSA7HBQFQND2YNM562W74IKJA, seg_pos={0 0}, upload=357453]: Long tail cancellation due to optimal threshold Dec 22 14:38:32.926237574 (407h9m32.797320609s) [noise=true , address=81.171.14.70:4031 , piece_id=7MD5N7PPKFPCFMOIXAGI3MPBDDX45GTLMA3CFGNM7BBGI6TSL6RA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12drdYnnWsuMyFo6LkcewAGGWnorRCAhxPpNqK41LvGy2zkBvng): protocol: expected piece hash; context canceled; EOF Dec 22 14:38:32.926250028 (407h9m32.797333063s) [noise=true , address=81.171.14.70:4031 , piece_id=7MD5N7PPKFPCFMOIXAGI3MPBDDX45GTLMA3CFGNM7BBGI6TSL6RA, seg_pos={0 0}, upload=357453]: Long tail cancellation due to optimal threshold Dec 22 14:38:32.926299612 (407h9m32.797382647s) [seg_pos={0 0}, upload=357453]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 22 14:38:32.926403941 (407h9m32.797486986s) [upload=357453]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 14:38:32.926409692 (407h9m32.797492737s) [upload=357453]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 14:39:04.793379844 (407h10m4.664462889s) [upload=357453]: Done uploading segments... Dec 22 14:39:04.793413608 (407h10m4.664496643s) [upload=357453]: Upload finished. err: <nil> Duration: 6.723035951s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.X.......X..................................................] duration: 4.439849122s trace: https://jaeger.dev.storj.io/trace/1c84c110f15266bb FAIL (2025-12-22 14:55:05 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.439901702s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....X.........X........................X...................] duration: 5.40280401s trace: https://jaeger.dev.storj.io/trace/510cfc1b6c68e873 FAIL (2025-12-22 14:55:01 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.402842926s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 1.42688752s trace: https://jaeger.dev.storj.io/trace/62b55ec615ab6123 FAIL (2025-12-22 14:54:55 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.426916356s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [......X.....................................X.....XX.....XX.] duration: 29.947383649s trace: https://jaeger.dev.storj.io/trace/68b9fb69cae779e3 FAIL (2025-12-22 14:54:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 22 14:53:15.872483574 (407h24m15.743566619s) [upload=357672]: Starting upload Dec 22 14:53:15.872562234 (407h24m15.743645279s) [upload=357672]: Done starting upload Dec 22 14:53:15.872675049 (407h24m15.743758084s) [upload=357672]: Uploading segments... Dec 22 14:53:15.872836616 (407h24m15.743919661s) [upload=357672]: Got next segment. Inline: false Dec 22 14:53:15.872878216 (407h24m15.743961261s) [upload=357672]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 14:53:15.872883236 (407h24m15.743966281s) [upload=357672]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 14:53:16.116320242 (407h24m15.987403278s) [seg_pos={0 0}, upload=357672]: Begin upload segment... Dec 22 14:53:16.116375688 (407h24m15.987458734s) [seg_pos={0 0}, upload=357672]: Done begin upload segment. Dec 22 14:53:16.116449589 (407h24m15.987532624s) [noise=true , address=145.79.227.41:4086 , piece_id=DCJWIN7CZUNVYLM226VNVECAKBGV2YHCO5JHRZ42JPP6IHOGZYBA, seg_pos={0 0}, upload=357672]: Uploading piece... Dec 22 14:53:16.116464417 (407h24m15.987547452s) [noise=true , address=145.79.227.51:4409 , piece_id=I3KN23DTGZIBTJ4UNXPTKWH2WX6PW6ONJ5EDIUKFHYSQRRRH62DQ, seg_pos={0 0}, upload=357672]: Uploading piece... Dec 22 14:53:16.116513631 (407h24m15.987596666s) [noise=true , address=145.79.227.28:4727 , piece_id=EZDGF2JWKQVWQLNIB7YLEXMCH74YH2YU7AIV5IYG7XLUFNU3IUCQ, seg_pos={0 0}, upload=357672]: Uploading piece... Dec 22 14:53:16.116936787 (407h24m15.988019822s) [noise=true , address=145.79.227.32:4836 , piece_id=MV2O2TDI6M66B7TJ4PXUWDHL5AGMVW4SCW2HIXIXULB3SYBROG2A, seg_pos={0 0}, upload=357672]: Uploading piece... Dec 22 14:53:16.118274774 (407h24m15.989357819s) [noise=true , address=145.79.227.47:4267 , piece_id=24DFW2AO4Z2IDDUOS6VJST3BW4MLQ5WWFET3P745D4FSTNH6D3QA, seg_pos={0 0}, upload=357672]: Uploading piece... Dec 22 14:53:16.118665860 (407h24m15.989748895s) [noise=true , address=145.79.227.52:4446 , piece_id=HSBTVW4NWNETD4GKIL5XTLX5RJNTPGADGIXNEOJ363JTTSW27G4A, seg_pos={0 0}, upload=357672]: Uploading piece... Dec 22 14:53:16.118966682 (407h24m15.990049727s) [noise=true , address=145.79.227.42:4122 , piece_id=XNDR6IHTE7QYOCDWQ7D7HLFK7GU6JGHHGVR24J2NBUO5U6BHVGFA, seg_pos={0 0}, upload=357672]: Uploading piece... Dec 22 14:53:16.119238039 (407h24m15.990321084s) [noise=true , address=145.79.227.19:4078 , piece_id=Z5IPTK3VHFNNMXCGDYV3URHSAXDQ2QSMNX3GB6W6SYE44J32XYJQ, seg_pos={0 0}, upload=357672]: Uploading piece... Dec 22 14:53:16.120036118 (407h24m15.991119153s) [noise=true , address=45.91.116.5:4297 , piece_id=LC5B5TSEX22ZC3NHFXCUJAA4QKPSYFIILKB2GEGM5LNLWWZ6TDRQ, seg_pos={0 0}, upload=357672]: Uploading piece... Dec 22 14:53:16.120276667 (407h24m15.991359702s) [noise=true , address=145.79.227.33:4871 , piece_id=QSUTGFPM62CNCXVCOCCG7WWQKNXHM2LXP3QIDFJGTYV5B35SNWQA, seg_pos={0 0}, upload=357672]: Uploading piece... Dec 22 14:53:16.121076990 (407h24m15.992160026s) [noise=true , address=145.79.227.48:4302 , piece_id=B6T6N422YL3IAWPANZXH7737ZTGOMACWU3SEBTL4L2A4HKZLPOUA, seg_pos={0 0}, upload=357672]: Uploading piece... Dec 22 14:53:16.121831948 (407h24m15.992914983s) [noise=true , address=77.90.31.19:4844 , piece_id=5B5AD7LA6AK3HC7HVUGCFWHZN27KXPS5B5IC5BFA57VPDBW5TWSA, seg_pos={0 0}, upload=357672]: Uploading piece... Dec 22 14:53:16.124225024 (407h24m15.99530806s) [noise=true , address=145.79.227.14:4837 , piece_id=D42BRGVZKM6HDOQ3V6RQJ5TI7EEUIXKN7GCURO52DOH7VFDIRPBQ, seg_pos={0 0}, upload=357672]: Uploading piece... Dec 22 14:53:16.124355643 (407h24m15.995438678s) [noise=true , address=45.91.116.14:4189 , piece_id=TZ4LCWH555BEQGYONHYUUHQOH2YOULWOZ6DZV7N5K3TXQDTNQQJA, seg_pos={0 0}, upload=357672]: Uploading piece... Dec 22 14:53:16.127407014 (407h24m15.998490049s) [noise=true , address=145.79.227.49:4339 , piece_id=NGK4DL77FPBWQ3GV7FW5IMD5VECIZGWP7LVC6AZ7KJYBXSSBXR5A, seg_pos={0 0}, upload=357672]: Uploading piece... Dec 22 14:53:16.129145695 (407h24m16.00022873s) [noise=true , address=145.79.227.39:4015 , piece_id=LFC5R4LMPOV5OLTL6MP2ISE6XXMD2KFHGLV4CTYUIWBGU52VHTAA, seg_pos={0 0}, upload=357672]: Uploading piece... Dec 22 14:53:18.044280827 (407h24m17.915363872s) [upload=357672]: Next returned nil segment Dec 22 14:53:18.044300604 (407h24m17.915383649s) [upload=357672]: Waiting for error group managing segments... Dec 22 14:53:18.356993368 (407h24m18.228076423s) [noise=true , address=45.91.116.5:4297 , piece_id=LC5B5TSEX22ZC3NHFXCUJAA4QKPSYFIILKB2GEGM5LNLWWZ6TDRQ, seg_pos={0 0}, upload=357672]: Done uploading piece. err: <nil> Dec 22 14:53:18.403809904 (407h24m18.27489294s) [noise=true , address=77.90.31.19:4844 , piece_id=5B5AD7LA6AK3HC7HVUGCFWHZN27KXPS5B5IC5BFA57VPDBW5TWSA, seg_pos={0 0}, upload=357672]: Done uploading piece. err: <nil> Dec 22 14:53:18.414093357 (407h24m18.285176392s) [noise=true , address=45.91.116.14:4189 , piece_id=TZ4LCWH555BEQGYONHYUUHQOH2YOULWOZ6DZV7N5K3TXQDTNQQJA, seg_pos={0 0}, upload=357672]: Done uploading piece. err: <nil> Dec 22 14:53:19.088967038 (407h24m18.960050073s) [noise=true , address=145.79.227.51:4409 , piece_id=I3KN23DTGZIBTJ4UNXPTKWH2WX6PW6ONJ5EDIUKFHYSQRRRH62DQ, seg_pos={0 0}, upload=357672]: Done uploading piece. err: <nil> Dec 22 14:53:19.091720000 (407h24m18.962803035s) [noise=true , address=145.79.227.28:4727 , piece_id=EZDGF2JWKQVWQLNIB7YLEXMCH74YH2YU7AIV5IYG7XLUFNU3IUCQ, seg_pos={0 0}, upload=357672]: Done uploading piece. err: <nil> Dec 22 14:53:19.096481436 (407h24m18.967564482s) [noise=true , address=145.79.227.39:4015 , piece_id=LFC5R4LMPOV5OLTL6MP2ISE6XXMD2KFHGLV4CTYUIWBGU52VHTAA, seg_pos={0 0}, upload=357672]: Done uploading piece. err: <nil> Dec 22 14:53:19.097130121 (407h24m18.968213166s) [noise=true , address=145.79.227.48:4302 , piece_id=B6T6N422YL3IAWPANZXH7737ZTGOMACWU3SEBTL4L2A4HKZLPOUA, seg_pos={0 0}, upload=357672]: Done uploading piece. err: <nil> Dec 22 14:53:19.100241636 (407h24m18.971324672s) [noise=true , address=145.79.227.47:4267 , piece_id=24DFW2AO4Z2IDDUOS6VJST3BW4MLQ5WWFET3P745D4FSTNH6D3QA, seg_pos={0 0}, upload=357672]: Done uploading piece. err: <nil> Dec 22 14:53:19.101469274 (407h24m18.972552309s) [noise=true , address=145.79.227.32:4836 , piece_id=MV2O2TDI6M66B7TJ4PXUWDHL5AGMVW4SCW2HIXIXULB3SYBROG2A, seg_pos={0 0}, upload=357672]: Done uploading piece. err: <nil> Dec 22 14:53:19.112910651 (407h24m18.983993696s) [noise=true , address=145.79.227.19:4078 , piece_id=Z5IPTK3VHFNNMXCGDYV3URHSAXDQ2QSMNX3GB6W6SYE44J32XYJQ, seg_pos={0 0}, upload=357672]: Done uploading piece. err: <nil> Dec 22 14:53:19.134326860 (407h24m19.005409906s) [noise=true , address=145.79.227.33:4871 , piece_id=QSUTGFPM62CNCXVCOCCG7WWQKNXHM2LXP3QIDFJGTYV5B35SNWQA, seg_pos={0 0}, upload=357672]: Done uploading piece. err: <nil> Dec 22 14:53:20.103593870 (407h24m19.974676915s) [noise=true , address=145.79.227.42:4122 , piece_id=XNDR6IHTE7QYOCDWQ7D7HLFK7GU6JGHHGVR24J2NBUO5U6BHVGFA, seg_pos={0 0}, upload=357672]: Done uploading piece. err: <nil> Dec 22 14:53:20.103899042 (407h24m19.974982077s) [noise=true , address=145.79.227.52:4446 , piece_id=HSBTVW4NWNETD4GKIL5XTLX5RJNTPGADGIXNEOJ363JTTSW27G4A, seg_pos={0 0}, upload=357672]: Done uploading piece. err: <nil> Dec 22 14:53:20.103922756 (407h24m19.975005801s) [seg_pos={0 0}, upload=357672]: Segment reached optimal threshold of 13 pieces. Dec 22 14:53:20.104126093 (407h24m19.975209138s) [noise=true , address=145.79.227.49:4339 , piece_id=NGK4DL77FPBWQ3GV7FW5IMD5VECIZGWP7LVC6AZ7KJYBXSSBXR5A, seg_pos={0 0}, upload=357672]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF Dec 22 14:53:20.104133728 (407h24m19.975216783s) [noise=true , address=145.79.227.49:4339 , piece_id=NGK4DL77FPBWQ3GV7FW5IMD5VECIZGWP7LVC6AZ7KJYBXSSBXR5A, seg_pos={0 0}, upload=357672]: Long tail cancellation due to optimal threshold Dec 22 14:53:20.104171820 (407h24m19.975254865s) [noise=true , address=145.79.227.41:4086 , piece_id=DCJWIN7CZUNVYLM226VNVECAKBGV2YHCO5JHRZ42JPP6IHOGZYBA, seg_pos={0 0}, upload=357672]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm): protocol: expected piece hash; context canceled; EOF Dec 22 14:53:20.104180938 (407h24m19.975263973s) [noise=true , address=145.79.227.41:4086 , piece_id=DCJWIN7CZUNVYLM226VNVECAKBGV2YHCO5JHRZ42JPP6IHOGZYBA, seg_pos={0 0}, upload=357672]: Long tail cancellation due to optimal threshold Dec 22 14:53:20.104513200 (407h24m19.975596266s) [noise=true , address=145.79.227.14:4837 , piece_id=D42BRGVZKM6HDOQ3V6RQJ5TI7EEUIXKN7GCURO52DOH7VFDIRPBQ, seg_pos={0 0}, upload=357672]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV): protocol: expected piece hash; context canceled; EOF Dec 22 14:53:20.104523630 (407h24m19.975606665s) [noise=true , address=145.79.227.14:4837 , piece_id=D42BRGVZKM6HDOQ3V6RQJ5TI7EEUIXKN7GCURO52DOH7VFDIRPBQ, seg_pos={0 0}, upload=357672]: Long tail cancellation due to optimal threshold Dec 22 14:53:20.104548979 (407h24m19.975632014s) [seg_pos={0 0}, upload=357672]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 14:53:20.104653798 (407h24m19.975736833s) [upload=357672]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 14:53:20.104658807 (407h24m19.975741853s) [upload=357672]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 14:53:20.403876141 (407h24m20.274959186s) [upload=357672]: Done uploading segments... Dec 22 14:53:20.403898104 (407h24m20.274981149s) [upload=357672]: Upload finished. err: <nil> Duration: 29.947418066s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 6.848712672s trace: https://jaeger.dev.storj.io/trace/718f722b1bda7fe2 FAIL (2025-12-22 14:54:45 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== VMAC4CKPNLATBLZLKAGDGZHVT7BHEPN3Q, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.367093115 (407h9m14.23817616s) [noise=true , address=185.167.204.206:4043 , piece_id=HL32YEZF67X2UT6SRMXJVJ4W5A3PC5SXOCM7BYQVMEC76TKEVJRA, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.367396732 (407h9m14.238479767s) [noise=true , address=185.167.204.205:4032 , piece_id=FJMXJG2PQ4WZDSDCPJ6LLSVQKI7JJHUCDUZLU2BJZV7RZTLETM5A, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.367549273 (407h9m14.238632308s) [noise=true , address=185.167.204.206:4070 , piece_id=Y7IRY4E2GCF7TCFOIQJ7QH6IFEJBMW4WM6QBIS5SCUJ7E7ASV4HA, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.367749263 (407h9m14.238832308s) [noise=true , address=81.171.14.70:4046 , piece_id=NRMV6FG5CZSTLUNP2EHNCK6VM2Z2UHKBLCHLKEPQ225Q5DGYEBPA, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.368034046 (407h9m14.239117091s) [noise=true , address=185.167.204.206:4060 , piece_id=WN6T5TUBVMURPSBBXGXCTLPZZ74PFNSRTTWXI2JPUTY74DPH3YSQ, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.368552724 (407h9m14.239635759s) [noise=true , address=81.171.14.70:4029 , piece_id=APDMOYEMDCLOREXY2BBOK73RSQVWKIXUX4SCHRVFDSM7ZLWBEQXA, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.369160691 (407h9m14.240243736s) [noise=true , address=185.167.204.206:4057 , piece_id=ONW7YS5JG44B2STNFJGJWFEZ3WYXO74N3CXI3C6T5OTJYHGQPQZQ, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.369714496 (407h9m14.240797541s) [noise=true , address=185.167.204.206:4071 , piece_id=DVJAKIMPAUGWQPHDPGTXASQDONAKOZIGX5PQDBGBXNQAPMBPUFBA, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.370249614 (407h9m14.241332649s) [noise=true , address=89.149.201.147:4019 , piece_id=ILE7ZX5X6HBPUVYGCOQ5SNG3AA6B3YTXIF47CPLJ3MN5EXVOS2OA, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.372019455 (407h9m14.24310249s) [noise=true , address=89.149.201.147:4015 , piece_id=ZIWRTD7HSP7YMRJET2TTJQFAQZM7KGDKJDLMYCWWY2W6SV2UVQBQ, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:15.199398795 (407h9m15.07048184s) [upload=357453]: Next returned nil segment Dec 22 14:38:15.199414966 (407h9m15.070498011s) [upload=357453]: Waiting for error group managing segments... Dec 22 14:38:15.591187572 (407h9m15.462270607s) [noise=true , address=185.167.204.205:4009 , piece_id=JTHX2O6T5HRENGEMAFJUWQUEUYAIG2KQ5HDEHF4BK42IT2ER7EIA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.591766795 (407h9m15.46284984s) [noise=true , address=185.167.204.205:4006 , piece_id=XHNFS6G7EX4BQWKOK4JUWYV3OTVJ36A6VOP4N7G2XPZKNNCFFKXA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.592074521 (407h9m15.463157566s) [noise=true , address=185.167.204.206:4060 , piece_id=WN6T5TUBVMURPSBBXGXCTLPZZ74PFNSRTTWXI2JPUTY74DPH3YSQ, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.593304694 (407h9m15.464387729s) [noise=true , address=185.167.204.205:4036 , piece_id=5QWJTTL6TZIZSSYHCBEJBKGX7BS76VVTELWTKEAKTOVYKINXVWDQ, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.593638659 (407h9m15.464721694s) [noise=true , address=185.167.204.205:4004 , piece_id=PYGDQZMAKOVJWB7AWEPFMPMB6C24FMS4HT2UP2AKZNYGG4DGZR3Q, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.593818251 (407h9m15.464901286s) [noise=true , address=185.167.204.206:4048 , piece_id=52WC3XH2K2BHG33BQPRENPNYDT7EOKCNBCTZKE3LIONDJHVATEYA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.594252549 (407h9m15.465335594s) [noise=true , address=185.167.204.205:4034 , piece_id=OS5H57TG2REFE535DK4T6N75VNIXPLQWW6DM55QCC2S7QBVZM3UQ, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.597113216 (407h9m15.468196261s) [noise=true , address=185.167.204.205:4032 , piece_id=FJMXJG2PQ4WZDSDCPJ6LLSVQKI7JJHUCDUZLU2BJZV7RZTLETM5A, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.597394541 (407h9m15.468477586s) [noise=true , address=185.167.204.205:4035 , piece_id=3D7T4SGPAJQZCA74CUTR4GJRE23R4JD5KBNW3BLEASSP4GSR5PHQ, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.597612756 (407h9m15.468695792s) [noise=true , address=185.167.204.206:4043 , piece_id=HL32YEZF67X2UT6SRMXJVJ4W5A3PC5SXOCM7BYQVMEC76TKEVJRA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.598018900 (407h9m15.469101945s) [noise=true , address=185.167.204.205:4012 , piece_id=A5G37BVL2GZ7LAY3FZWIZV4R4M7M6UPYDYACJFRD5HKDWWKB4Q6A, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.598254719 (407h9m15.469337765s) [noise=true , address=185.167.204.206:4052 , piece_id=V2DVAAXN5A4C3NQW556VMAC4CKPNLATBLZLKAGDGZHVT7BHEPN3Q, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.598480649 (407h9m15.469563694s) [noise=true , address=185.167.204.205:4018 , piece_id=NUYRZ7PXKJQ3PIR44O3E6N6COZZPRDNSMTQCFI5RN7FBRV2CBALA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.598968278 (407h9m15.470051323s) [noise=true , address=185.167.204.205:4033 , piece_id=A32ZTBDBWMCNBMZPD4DUWS5NKUKQ22ACQH6HAW5AKB3FNHBC56VA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.599287886 (407h9m15.470370931s) [noise=true , address=185.167.204.206:4072 , piece_id=I6PCSC5JYTJXLT6B34TTIVZQVX52R7TRHSRLRP4ZHG5J63XDZN6Q, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.599632252 (407h9m15.470715297s) [noise=true , address=185.167.204.206:4070 , piece_id=Y7IRY4E2GCF7TCFOIQJ7QH6IFEJBMW4WM6QBIS5SCUJ7E7ASV4HA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.599676297 (407h9m15.470759332s) [noise=true , address=185.167.204.206:4071 , piece_id=DVJAKIMPAUGWQPHDPGTXASQDONAKOZIGX5PQDBGBXNQAPMBPUFBA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.601435877 (407h9m15.472518912s) [noise=true , address=185.167.204.206:4057 , piece_id=ONW7YS5JG44B2STNFJGJWFEZ3WYXO74N3CXI3C6T5OTJYHGQPQZQ, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:17.299689191 (407h9m17.170772226s) [noise=true , address=89.149.201.147:4019 , piece_id=ILE7ZX5X6HBPUVYGCOQ5SNG3AA6B3YTXIF47CPLJ3MN5EXVOS2OA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:17.526601953 (407h9m17.397684998s) [noise=true , address=89.149.201.147:4015 , piece_id=ZIWRTD7HSP7YMRJET2TTJQFAQZM7KGDKJDLMYCWWY2W6SV2UVQBQ, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:18.294911214 (407h9m18.165994249s) [noise=true , address=89.149.201.147:4017 , piece_id=ZOSNSMZOBQVIKD2ZGJRAUWJEJ4JXJK452YWQQSDL77E77F6LRDZA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:18.402191355 (407h9m18.2732744s) [noise=true , address=89.149.201.147:4011 , piece_id=ATMM77IGBDSRHNJ5FQ2VKP7W27JUA2F6HYJYSQ62IL5JQN5PF5OA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:22.436508909 (407h9m22.307591944s) [noise=true , address=81.171.14.70:4037 , piece_id=NYGC2ZLUSBUAGFULQ46LXCVNQUJ5M4XT3YICVA2ZMBR2D6WPJ27Q, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:25.935108368 (407h9m25.806191413s) [noise=true , address=81.171.14.70:4046 , piece_id=NRMV6FG5CZSTLUNP2EHNCK6VM2Z2UHKBLCHLKEPQ225Q5DGYEBPA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:30.904793996 (407h9m30.775877041s) [noise=true , address=81.171.14.70:4029 , piece_id=APDMOYEMDCLOREXY2BBOK73RSQVWKIXUX4SCHRVFDSM7ZLWBEQXA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:32.926010512 (407h9m32.797093567s) [noise=true , address=81.171.14.70:4044 , piece_id=QCWQO6JQRRKSG5U2KVH3Y6I3UI2F7G3DOW4CH3PQZKUVQ6Q2XOHA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:32.926036552 (407h9m32.797119587s) [seg_pos={0 0}, upload=357453]: Segment reached optimal threshold of 26 pieces. Dec 22 14:38:32.926196857 (407h9m32.797279902s) [noise=true , address=81.171.14.70:4033 , piece_id=MKNAJOFD5CIHTA55XO476LOJJGIWSA7HBQFQND2YNM562W74IKJA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BnjusiXEfHYwTDL8soMkm8GYnhbkwy2oCAxVA7oiDyhFXozZ8): protocol: expected piece hash; context canceled; EOF Dec 22 14:38:32.926203058 (407h9m32.797286093s) [noise=true , address=81.171.14.70:4033 , piece_id=MKNAJOFD5CIHTA55XO476LOJJGIWSA7HBQFQND2YNM562W74IKJA, seg_pos={0 0}, upload=357453]: Long tail cancellation due to optimal threshold Dec 22 14:38:32.926237574 (407h9m32.797320609s) [noise=true , address=81.171.14.70:4031 , piece_id=7MD5N7PPKFPCFMOIXAGI3MPBDDX45GTLMA3CFGNM7BBGI6TSL6RA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12drdYnnWsuMyFo6LkcewAGGWnorRCAhxPpNqK41LvGy2zkBvng): protocol: expected piece hash; context canceled; EOF Dec 22 14:38:32.926250028 (407h9m32.797333063s) [noise=true , address=81.171.14.70:4031 , piece_id=7MD5N7PPKFPCFMOIXAGI3MPBDDX45GTLMA3CFGNM7BBGI6TSL6RA, seg_pos={0 0}, upload=357453]: Long tail cancellation due to optimal threshold Dec 22 14:38:32.926299612 (407h9m32.797382647s) [seg_pos={0 0}, upload=357453]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 22 14:38:32.926403941 (407h9m32.797486986s) [upload=357453]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 14:38:32.926409692 (407h9m32.797492737s) [upload=357453]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 14:39:04.793379844 (407h10m4.664462889s) [upload=357453]: Done uploading segments... Dec 22 14:39:04.793413608 (407h10m4.664496643s) [upload=357453]: Upload finished. err: <nil> Duration: 6.848756696s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 2 out of 60 is failed (threshold=3) [X..................................X........................] duration: 1.386904015s trace: https://jaeger.dev.storj.io/trace/62b55ec615ab6123 FAIL (2025-12-22 14:54:13 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.386920225s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [X.X.......X.................................................] duration: 3.752301486s trace: https://jaeger.dev.storj.io/trace/1c84c110f15266bb FAIL (2025-12-22 14:54:11 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.752328758s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 3 out of 60 is failed (threshold=3) [......X.........X........................X..................] duration: 5.365802102s trace: https://jaeger.dev.storj.io/trace/510cfc1b6c68e873 FAIL (2025-12-22 14:53:56 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.365827381s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.......X.....................................X.....XX.....XX] duration: 30.003839955s trace: https://jaeger.dev.storj.io/trace/68b9fb69cae779e3 FAIL (2025-12-22 14:53:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 22 14:53:15.872483574 (407h24m15.743566619s) [upload=357672]: Starting upload Dec 22 14:53:15.872562234 (407h24m15.743645279s) [upload=357672]: Done starting upload Dec 22 14:53:15.872675049 (407h24m15.743758084s) [upload=357672]: Uploading segments... Dec 22 14:53:15.872836616 (407h24m15.743919661s) [upload=357672]: Got next segment. Inline: false Dec 22 14:53:15.872878216 (407h24m15.743961261s) [upload=357672]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 14:53:15.872883236 (407h24m15.743966281s) [upload=357672]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 14:53:16.116320242 (407h24m15.987403278s) [seg_pos={0 0}, upload=357672]: Begin upload segment... Dec 22 14:53:16.116375688 (407h24m15.987458734s) [seg_pos={0 0}, upload=357672]: Done begin upload segment. Dec 22 14:53:16.116449589 (407h24m15.987532624s) [noise=true , address=145.79.227.41:4086 , piece_id=DCJWIN7CZUNVYLM226VNVECAKBGV2YHCO5JHRZ42JPP6IHOGZYBA, seg_pos={0 0}, upload=357672]: Uploading piece... Dec 22 14:53:16.116464417 (407h24m15.987547452s) [noise=true , address=145.79.227.51:4409 , piece_id=I3KN23DTGZIBTJ4UNXPTKWH2WX6PW6ONJ5EDIUKFHYSQRRRH62DQ, seg_pos={0 0}, upload=357672]: Uploading piece... Dec 22 14:53:16.116513631 (407h24m15.987596666s) [noise=true , address=145.79.227.28:4727 , piece_id=EZDGF2JWKQVWQLNIB7YLEXMCH74YH2YU7AIV5IYG7XLUFNU3IUCQ, seg_pos={0 0}, upload=357672]: Uploading piece... Dec 22 14:53:16.116936787 (407h24m15.988019822s) [noise=true , address=145.79.227.32:4836 , piece_id=MV2O2TDI6M66B7TJ4PXUWDHL5AGMVW4SCW2HIXIXULB3SYBROG2A, seg_pos={0 0}, upload=357672]: Uploading piece... Dec 22 14:53:16.118274774 (407h24m15.989357819s) [noise=true , address=145.79.227.47:4267 , piece_id=24DFW2AO4Z2IDDUOS6VJST3BW4MLQ5WWFET3P745D4FSTNH6D3QA, seg_pos={0 0}, upload=357672]: Uploading piece... Dec 22 14:53:16.118665860 (407h24m15.989748895s) [noise=true , address=145.79.227.52:4446 , piece_id=HSBTVW4NWNETD4GKIL5XTLX5RJNTPGADGIXNEOJ363JTTSW27G4A, seg_pos={0 0}, upload=357672]: Uploading piece... Dec 22 14:53:16.118966682 (407h24m15.990049727s) [noise=true , address=145.79.227.42:4122 , piece_id=XNDR6IHTE7QYOCDWQ7D7HLFK7GU6JGHHGVR24J2NBUO5U6BHVGFA, seg_pos={0 0}, upload=357672]: Uploading piece... Dec 22 14:53:16.119238039 (407h24m15.990321084s) [noise=true , address=145.79.227.19:4078 , piece_id=Z5IPTK3VHFNNMXCGDYV3URHSAXDQ2QSMNX3GB6W6SYE44J32XYJQ, seg_pos={0 0}, upload=357672]: Uploading piece... Dec 22 14:53:16.120036118 (407h24m15.991119153s) [noise=true , address=45.91.116.5:4297 , piece_id=LC5B5TSEX22ZC3NHFXCUJAA4QKPSYFIILKB2GEGM5LNLWWZ6TDRQ, seg_pos={0 0}, upload=357672]: Uploading piece... Dec 22 14:53:16.120276667 (407h24m15.991359702s) [noise=true , address=145.79.227.33:4871 , piece_id=QSUTGFPM62CNCXVCOCCG7WWQKNXHM2LXP3QIDFJGTYV5B35SNWQA, seg_pos={0 0}, upload=357672]: Uploading piece... Dec 22 14:53:16.121076990 (407h24m15.992160026s) [noise=true , address=145.79.227.48:4302 , piece_id=B6T6N422YL3IAWPANZXH7737ZTGOMACWU3SEBTL4L2A4HKZLPOUA, seg_pos={0 0}, upload=357672]: Uploading piece... Dec 22 14:53:16.121831948 (407h24m15.992914983s) [noise=true , address=77.90.31.19:4844 , piece_id=5B5AD7LA6AK3HC7HVUGCFWHZN27KXPS5B5IC5BFA57VPDBW5TWSA, seg_pos={0 0}, upload=357672]: Uploading piece... Dec 22 14:53:16.124225024 (407h24m15.99530806s) [noise=true , address=145.79.227.14:4837 , piece_id=D42BRGVZKM6HDOQ3V6RQJ5TI7EEUIXKN7GCURO52DOH7VFDIRPBQ, seg_pos={0 0}, upload=357672]: Uploading piece... Dec 22 14:53:16.124355643 (407h24m15.995438678s) [noise=true , address=45.91.116.14:4189 , piece_id=TZ4LCWH555BEQGYONHYUUHQOH2YOULWOZ6DZV7N5K3TXQDTNQQJA, seg_pos={0 0}, upload=357672]: Uploading piece... Dec 22 14:53:16.127407014 (407h24m15.998490049s) [noise=true , address=145.79.227.49:4339 , piece_id=NGK4DL77FPBWQ3GV7FW5IMD5VECIZGWP7LVC6AZ7KJYBXSSBXR5A, seg_pos={0 0}, upload=357672]: Uploading piece... Dec 22 14:53:16.129145695 (407h24m16.00022873s) [noise=true , address=145.79.227.39:4015 , piece_id=LFC5R4LMPOV5OLTL6MP2ISE6XXMD2KFHGLV4CTYUIWBGU52VHTAA, seg_pos={0 0}, upload=357672]: Uploading piece... Dec 22 14:53:18.044280827 (407h24m17.915363872s) [upload=357672]: Next returned nil segment Dec 22 14:53:18.044300604 (407h24m17.915383649s) [upload=357672]: Waiting for error group managing segments... Dec 22 14:53:18.356993368 (407h24m18.228076423s) [noise=true , address=45.91.116.5:4297 , piece_id=LC5B5TSEX22ZC3NHFXCUJAA4QKPSYFIILKB2GEGM5LNLWWZ6TDRQ, seg_pos={0 0}, upload=357672]: Done uploading piece. err: <nil> Dec 22 14:53:18.403809904 (407h24m18.27489294s) [noise=true , address=77.90.31.19:4844 , piece_id=5B5AD7LA6AK3HC7HVUGCFWHZN27KXPS5B5IC5BFA57VPDBW5TWSA, seg_pos={0 0}, upload=357672]: Done uploading piece. err: <nil> Dec 22 14:53:18.414093357 (407h24m18.285176392s) [noise=true , address=45.91.116.14:4189 , piece_id=TZ4LCWH555BEQGYONHYUUHQOH2YOULWOZ6DZV7N5K3TXQDTNQQJA, seg_pos={0 0}, upload=357672]: Done uploading piece. err: <nil> Dec 22 14:53:19.088967038 (407h24m18.960050073s) [noise=true , address=145.79.227.51:4409 , piece_id=I3KN23DTGZIBTJ4UNXPTKWH2WX6PW6ONJ5EDIUKFHYSQRRRH62DQ, seg_pos={0 0}, upload=357672]: Done uploading piece. err: <nil> Dec 22 14:53:19.091720000 (407h24m18.962803035s) [noise=true , address=145.79.227.28:4727 , piece_id=EZDGF2JWKQVWQLNIB7YLEXMCH74YH2YU7AIV5IYG7XLUFNU3IUCQ, seg_pos={0 0}, upload=357672]: Done uploading piece. err: <nil> Dec 22 14:53:19.096481436 (407h24m18.967564482s) [noise=true , address=145.79.227.39:4015 , piece_id=LFC5R4LMPOV5OLTL6MP2ISE6XXMD2KFHGLV4CTYUIWBGU52VHTAA, seg_pos={0 0}, upload=357672]: Done uploading piece. err: <nil> Dec 22 14:53:19.097130121 (407h24m18.968213166s) [noise=true , address=145.79.227.48:4302 , piece_id=B6T6N422YL3IAWPANZXH7737ZTGOMACWU3SEBTL4L2A4HKZLPOUA, seg_pos={0 0}, upload=357672]: Done uploading piece. err: <nil> Dec 22 14:53:19.100241636 (407h24m18.971324672s) [noise=true , address=145.79.227.47:4267 , piece_id=24DFW2AO4Z2IDDUOS6VJST3BW4MLQ5WWFET3P745D4FSTNH6D3QA, seg_pos={0 0}, upload=357672]: Done uploading piece. err: <nil> Dec 22 14:53:19.101469274 (407h24m18.972552309s) [noise=true , address=145.79.227.32:4836 , piece_id=MV2O2TDI6M66B7TJ4PXUWDHL5AGMVW4SCW2HIXIXULB3SYBROG2A, seg_pos={0 0}, upload=357672]: Done uploading piece. err: <nil> Dec 22 14:53:19.112910651 (407h24m18.983993696s) [noise=true , address=145.79.227.19:4078 , piece_id=Z5IPTK3VHFNNMXCGDYV3URHSAXDQ2QSMNX3GB6W6SYE44J32XYJQ, seg_pos={0 0}, upload=357672]: Done uploading piece. err: <nil> Dec 22 14:53:19.134326860 (407h24m19.005409906s) [noise=true , address=145.79.227.33:4871 , piece_id=QSUTGFPM62CNCXVCOCCG7WWQKNXHM2LXP3QIDFJGTYV5B35SNWQA, seg_pos={0 0}, upload=357672]: Done uploading piece. err: <nil> Dec 22 14:53:20.103593870 (407h24m19.974676915s) [noise=true , address=145.79.227.42:4122 , piece_id=XNDR6IHTE7QYOCDWQ7D7HLFK7GU6JGHHGVR24J2NBUO5U6BHVGFA, seg_pos={0 0}, upload=357672]: Done uploading piece. err: <nil> Dec 22 14:53:20.103899042 (407h24m19.974982077s) [noise=true , address=145.79.227.52:4446 , piece_id=HSBTVW4NWNETD4GKIL5XTLX5RJNTPGADGIXNEOJ363JTTSW27G4A, seg_pos={0 0}, upload=357672]: Done uploading piece. err: <nil> Dec 22 14:53:20.103922756 (407h24m19.975005801s) [seg_pos={0 0}, upload=357672]: Segment reached optimal threshold of 13 pieces. Dec 22 14:53:20.104126093 (407h24m19.975209138s) [noise=true , address=145.79.227.49:4339 , piece_id=NGK4DL77FPBWQ3GV7FW5IMD5VECIZGWP7LVC6AZ7KJYBXSSBXR5A, seg_pos={0 0}, upload=357672]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF Dec 22 14:53:20.104133728 (407h24m19.975216783s) [noise=true , address=145.79.227.49:4339 , piece_id=NGK4DL77FPBWQ3GV7FW5IMD5VECIZGWP7LVC6AZ7KJYBXSSBXR5A, seg_pos={0 0}, upload=357672]: Long tail cancellation due to optimal threshold Dec 22 14:53:20.104171820 (407h24m19.975254865s) [noise=true , address=145.79.227.41:4086 , piece_id=DCJWIN7CZUNVYLM226VNVECAKBGV2YHCO5JHRZ42JPP6IHOGZYBA, seg_pos={0 0}, upload=357672]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm): protocol: expected piece hash; context canceled; EOF Dec 22 14:53:20.104180938 (407h24m19.975263973s) [noise=true , address=145.79.227.41:4086 , piece_id=DCJWIN7CZUNVYLM226VNVECAKBGV2YHCO5JHRZ42JPP6IHOGZYBA, seg_pos={0 0}, upload=357672]: Long tail cancellation due to optimal threshold Dec 22 14:53:20.104513200 (407h24m19.975596266s) [noise=true , address=145.79.227.14:4837 , piece_id=D42BRGVZKM6HDOQ3V6RQJ5TI7EEUIXKN7GCURO52DOH7VFDIRPBQ, seg_pos={0 0}, upload=357672]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV): protocol: expected piece hash; context canceled; EOF Dec 22 14:53:20.104523630 (407h24m19.975606665s) [noise=true , address=145.79.227.14:4837 , piece_id=D42BRGVZKM6HDOQ3V6RQJ5TI7EEUIXKN7GCURO52DOH7VFDIRPBQ, seg_pos={0 0}, upload=357672]: Long tail cancellation due to optimal threshold Dec 22 14:53:20.104548979 (407h24m19.975632014s) [seg_pos={0 0}, upload=357672]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 14:53:20.104653798 (407h24m19.975736833s) [upload=357672]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 14:53:20.104658807 (407h24m19.975741853s) [upload=357672]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 14:53:20.403876141 (407h24m20.274959186s) [upload=357672]: Done uploading segments... Dec 22 14:53:20.403898104 (407h24m20.274981149s) [upload=357672]: Upload finished. err: <nil> Duration: 30.00386295s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 6.338869076s trace: https://jaeger.dev.storj.io/trace/718f722b1bda7fe2 FAIL (2025-12-22 14:53:40 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== VMAC4CKPNLATBLZLKAGDGZHVT7BHEPN3Q, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.367093115 (407h9m14.23817616s) [noise=true , address=185.167.204.206:4043 , piece_id=HL32YEZF67X2UT6SRMXJVJ4W5A3PC5SXOCM7BYQVMEC76TKEVJRA, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.367396732 (407h9m14.238479767s) [noise=true , address=185.167.204.205:4032 , piece_id=FJMXJG2PQ4WZDSDCPJ6LLSVQKI7JJHUCDUZLU2BJZV7RZTLETM5A, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.367549273 (407h9m14.238632308s) [noise=true , address=185.167.204.206:4070 , piece_id=Y7IRY4E2GCF7TCFOIQJ7QH6IFEJBMW4WM6QBIS5SCUJ7E7ASV4HA, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.367749263 (407h9m14.238832308s) [noise=true , address=81.171.14.70:4046 , piece_id=NRMV6FG5CZSTLUNP2EHNCK6VM2Z2UHKBLCHLKEPQ225Q5DGYEBPA, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.368034046 (407h9m14.239117091s) [noise=true , address=185.167.204.206:4060 , piece_id=WN6T5TUBVMURPSBBXGXCTLPZZ74PFNSRTTWXI2JPUTY74DPH3YSQ, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.368552724 (407h9m14.239635759s) [noise=true , address=81.171.14.70:4029 , piece_id=APDMOYEMDCLOREXY2BBOK73RSQVWKIXUX4SCHRVFDSM7ZLWBEQXA, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.369160691 (407h9m14.240243736s) [noise=true , address=185.167.204.206:4057 , piece_id=ONW7YS5JG44B2STNFJGJWFEZ3WYXO74N3CXI3C6T5OTJYHGQPQZQ, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.369714496 (407h9m14.240797541s) [noise=true , address=185.167.204.206:4071 , piece_id=DVJAKIMPAUGWQPHDPGTXASQDONAKOZIGX5PQDBGBXNQAPMBPUFBA, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.370249614 (407h9m14.241332649s) [noise=true , address=89.149.201.147:4019 , piece_id=ILE7ZX5X6HBPUVYGCOQ5SNG3AA6B3YTXIF47CPLJ3MN5EXVOS2OA, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.372019455 (407h9m14.24310249s) [noise=true , address=89.149.201.147:4015 , piece_id=ZIWRTD7HSP7YMRJET2TTJQFAQZM7KGDKJDLMYCWWY2W6SV2UVQBQ, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:15.199398795 (407h9m15.07048184s) [upload=357453]: Next returned nil segment Dec 22 14:38:15.199414966 (407h9m15.070498011s) [upload=357453]: Waiting for error group managing segments... Dec 22 14:38:15.591187572 (407h9m15.462270607s) [noise=true , address=185.167.204.205:4009 , piece_id=JTHX2O6T5HRENGEMAFJUWQUEUYAIG2KQ5HDEHF4BK42IT2ER7EIA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.591766795 (407h9m15.46284984s) [noise=true , address=185.167.204.205:4006 , piece_id=XHNFS6G7EX4BQWKOK4JUWYV3OTVJ36A6VOP4N7G2XPZKNNCFFKXA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.592074521 (407h9m15.463157566s) [noise=true , address=185.167.204.206:4060 , piece_id=WN6T5TUBVMURPSBBXGXCTLPZZ74PFNSRTTWXI2JPUTY74DPH3YSQ, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.593304694 (407h9m15.464387729s) [noise=true , address=185.167.204.205:4036 , piece_id=5QWJTTL6TZIZSSYHCBEJBKGX7BS76VVTELWTKEAKTOVYKINXVWDQ, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.593638659 (407h9m15.464721694s) [noise=true , address=185.167.204.205:4004 , piece_id=PYGDQZMAKOVJWB7AWEPFMPMB6C24FMS4HT2UP2AKZNYGG4DGZR3Q, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.593818251 (407h9m15.464901286s) [noise=true , address=185.167.204.206:4048 , piece_id=52WC3XH2K2BHG33BQPRENPNYDT7EOKCNBCTZKE3LIONDJHVATEYA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.594252549 (407h9m15.465335594s) [noise=true , address=185.167.204.205:4034 , piece_id=OS5H57TG2REFE535DK4T6N75VNIXPLQWW6DM55QCC2S7QBVZM3UQ, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.597113216 (407h9m15.468196261s) [noise=true , address=185.167.204.205:4032 , piece_id=FJMXJG2PQ4WZDSDCPJ6LLSVQKI7JJHUCDUZLU2BJZV7RZTLETM5A, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.597394541 (407h9m15.468477586s) [noise=true , address=185.167.204.205:4035 , piece_id=3D7T4SGPAJQZCA74CUTR4GJRE23R4JD5KBNW3BLEASSP4GSR5PHQ, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.597612756 (407h9m15.468695792s) [noise=true , address=185.167.204.206:4043 , piece_id=HL32YEZF67X2UT6SRMXJVJ4W5A3PC5SXOCM7BYQVMEC76TKEVJRA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.598018900 (407h9m15.469101945s) [noise=true , address=185.167.204.205:4012 , piece_id=A5G37BVL2GZ7LAY3FZWIZV4R4M7M6UPYDYACJFRD5HKDWWKB4Q6A, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.598254719 (407h9m15.469337765s) [noise=true , address=185.167.204.206:4052 , piece_id=V2DVAAXN5A4C3NQW556VMAC4CKPNLATBLZLKAGDGZHVT7BHEPN3Q, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.598480649 (407h9m15.469563694s) [noise=true , address=185.167.204.205:4018 , piece_id=NUYRZ7PXKJQ3PIR44O3E6N6COZZPRDNSMTQCFI5RN7FBRV2CBALA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.598968278 (407h9m15.470051323s) [noise=true , address=185.167.204.205:4033 , piece_id=A32ZTBDBWMCNBMZPD4DUWS5NKUKQ22ACQH6HAW5AKB3FNHBC56VA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.599287886 (407h9m15.470370931s) [noise=true , address=185.167.204.206:4072 , piece_id=I6PCSC5JYTJXLT6B34TTIVZQVX52R7TRHSRLRP4ZHG5J63XDZN6Q, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.599632252 (407h9m15.470715297s) [noise=true , address=185.167.204.206:4070 , piece_id=Y7IRY4E2GCF7TCFOIQJ7QH6IFEJBMW4WM6QBIS5SCUJ7E7ASV4HA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.599676297 (407h9m15.470759332s) [noise=true , address=185.167.204.206:4071 , piece_id=DVJAKIMPAUGWQPHDPGTXASQDONAKOZIGX5PQDBGBXNQAPMBPUFBA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.601435877 (407h9m15.472518912s) [noise=true , address=185.167.204.206:4057 , piece_id=ONW7YS5JG44B2STNFJGJWFEZ3WYXO74N3CXI3C6T5OTJYHGQPQZQ, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:17.299689191 (407h9m17.170772226s) [noise=true , address=89.149.201.147:4019 , piece_id=ILE7ZX5X6HBPUVYGCOQ5SNG3AA6B3YTXIF47CPLJ3MN5EXVOS2OA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:17.526601953 (407h9m17.397684998s) [noise=true , address=89.149.201.147:4015 , piece_id=ZIWRTD7HSP7YMRJET2TTJQFAQZM7KGDKJDLMYCWWY2W6SV2UVQBQ, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:18.294911214 (407h9m18.165994249s) [noise=true , address=89.149.201.147:4017 , piece_id=ZOSNSMZOBQVIKD2ZGJRAUWJEJ4JXJK452YWQQSDL77E77F6LRDZA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:18.402191355 (407h9m18.2732744s) [noise=true , address=89.149.201.147:4011 , piece_id=ATMM77IGBDSRHNJ5FQ2VKP7W27JUA2F6HYJYSQ62IL5JQN5PF5OA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:22.436508909 (407h9m22.307591944s) [noise=true , address=81.171.14.70:4037 , piece_id=NYGC2ZLUSBUAGFULQ46LXCVNQUJ5M4XT3YICVA2ZMBR2D6WPJ27Q, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:25.935108368 (407h9m25.806191413s) [noise=true , address=81.171.14.70:4046 , piece_id=NRMV6FG5CZSTLUNP2EHNCK6VM2Z2UHKBLCHLKEPQ225Q5DGYEBPA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:30.904793996 (407h9m30.775877041s) [noise=true , address=81.171.14.70:4029 , piece_id=APDMOYEMDCLOREXY2BBOK73RSQVWKIXUX4SCHRVFDSM7ZLWBEQXA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:32.926010512 (407h9m32.797093567s) [noise=true , address=81.171.14.70:4044 , piece_id=QCWQO6JQRRKSG5U2KVH3Y6I3UI2F7G3DOW4CH3PQZKUVQ6Q2XOHA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:32.926036552 (407h9m32.797119587s) [seg_pos={0 0}, upload=357453]: Segment reached optimal threshold of 26 pieces. Dec 22 14:38:32.926196857 (407h9m32.797279902s) [noise=true , address=81.171.14.70:4033 , piece_id=MKNAJOFD5CIHTA55XO476LOJJGIWSA7HBQFQND2YNM562W74IKJA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BnjusiXEfHYwTDL8soMkm8GYnhbkwy2oCAxVA7oiDyhFXozZ8): protocol: expected piece hash; context canceled; EOF Dec 22 14:38:32.926203058 (407h9m32.797286093s) [noise=true , address=81.171.14.70:4033 , piece_id=MKNAJOFD5CIHTA55XO476LOJJGIWSA7HBQFQND2YNM562W74IKJA, seg_pos={0 0}, upload=357453]: Long tail cancellation due to optimal threshold Dec 22 14:38:32.926237574 (407h9m32.797320609s) [noise=true , address=81.171.14.70:4031 , piece_id=7MD5N7PPKFPCFMOIXAGI3MPBDDX45GTLMA3CFGNM7BBGI6TSL6RA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12drdYnnWsuMyFo6LkcewAGGWnorRCAhxPpNqK41LvGy2zkBvng): protocol: expected piece hash; context canceled; EOF Dec 22 14:38:32.926250028 (407h9m32.797333063s) [noise=true , address=81.171.14.70:4031 , piece_id=7MD5N7PPKFPCFMOIXAGI3MPBDDX45GTLMA3CFGNM7BBGI6TSL6RA, seg_pos={0 0}, upload=357453]: Long tail cancellation due to optimal threshold Dec 22 14:38:32.926299612 (407h9m32.797382647s) [seg_pos={0 0}, upload=357453]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 22 14:38:32.926403941 (407h9m32.797486986s) [upload=357453]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 14:38:32.926409692 (407h9m32.797492737s) [upload=357453]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 14:39:04.793379844 (407h10m4.664462889s) [upload=357453]: Done uploading segments... Dec 22 14:39:04.793413608 (407h10m4.664496643s) [upload=357453]: Upload finished. err: <nil> Duration: 6.338903713s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 2 out of 60 is failed (threshold=3) [.X..................................X.......................] duration: 1.376290465s trace: https://jaeger.dev.storj.io/trace/62b55ec615ab6123 FAIL (2025-12-22 14:53:40 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.376318498s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.X.X.......X................................................] duration: 3.649084676s trace: https://jaeger.dev.storj.io/trace/1c84c110f15266bb FAIL (2025-12-22 14:53:20 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.649110554s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 2 out of 60 is failed (threshold=3) [..X..................................X......................] duration: 1.397718929s trace: https://jaeger.dev.storj.io/trace/62b55ec615ab6123 FAIL (2025-12-22 14:53:08 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.397751571s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......X.........X........................X.................] duration: 6.327769321s trace: https://jaeger.dev.storj.io/trace/510cfc1b6c68e873 FAIL (2025-12-22 14:52:59 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 6.327798367s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 2 out of 60 is failed (threshold=3) [...X..................................X.....................] duration: 1.408929956s trace: https://jaeger.dev.storj.io/trace/62b55ec615ab6123 FAIL (2025-12-22 14:52:52 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.408953702s core uplink eu1

Orbiter Check History

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)