Orbiter History

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 1.470631353s trace: https://jaeger.dev.storj.io/trace/62b55ec615ab6123 FAIL (2025-12-22 15:11:53 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.47065652s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 8.021009649s trace: https://jaeger.dev.storj.io/trace/718f722b1bda7fe2 FAIL (2025-12-22 15:11:52 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== VMAC4CKPNLATBLZLKAGDGZHVT7BHEPN3Q, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.367093115 (407h9m14.23817616s) [noise=true , address=185.167.204.206:4043 , piece_id=HL32YEZF67X2UT6SRMXJVJ4W5A3PC5SXOCM7BYQVMEC76TKEVJRA, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.367396732 (407h9m14.238479767s) [noise=true , address=185.167.204.205:4032 , piece_id=FJMXJG2PQ4WZDSDCPJ6LLSVQKI7JJHUCDUZLU2BJZV7RZTLETM5A, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.367549273 (407h9m14.238632308s) [noise=true , address=185.167.204.206:4070 , piece_id=Y7IRY4E2GCF7TCFOIQJ7QH6IFEJBMW4WM6QBIS5SCUJ7E7ASV4HA, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.367749263 (407h9m14.238832308s) [noise=true , address=81.171.14.70:4046 , piece_id=NRMV6FG5CZSTLUNP2EHNCK6VM2Z2UHKBLCHLKEPQ225Q5DGYEBPA, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.368034046 (407h9m14.239117091s) [noise=true , address=185.167.204.206:4060 , piece_id=WN6T5TUBVMURPSBBXGXCTLPZZ74PFNSRTTWXI2JPUTY74DPH3YSQ, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.368552724 (407h9m14.239635759s) [noise=true , address=81.171.14.70:4029 , piece_id=APDMOYEMDCLOREXY2BBOK73RSQVWKIXUX4SCHRVFDSM7ZLWBEQXA, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.369160691 (407h9m14.240243736s) [noise=true , address=185.167.204.206:4057 , piece_id=ONW7YS5JG44B2STNFJGJWFEZ3WYXO74N3CXI3C6T5OTJYHGQPQZQ, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.369714496 (407h9m14.240797541s) [noise=true , address=185.167.204.206:4071 , piece_id=DVJAKIMPAUGWQPHDPGTXASQDONAKOZIGX5PQDBGBXNQAPMBPUFBA, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.370249614 (407h9m14.241332649s) [noise=true , address=89.149.201.147:4019 , piece_id=ILE7ZX5X6HBPUVYGCOQ5SNG3AA6B3YTXIF47CPLJ3MN5EXVOS2OA, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.372019455 (407h9m14.24310249s) [noise=true , address=89.149.201.147:4015 , piece_id=ZIWRTD7HSP7YMRJET2TTJQFAQZM7KGDKJDLMYCWWY2W6SV2UVQBQ, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:15.199398795 (407h9m15.07048184s) [upload=357453]: Next returned nil segment Dec 22 14:38:15.199414966 (407h9m15.070498011s) [upload=357453]: Waiting for error group managing segments... Dec 22 14:38:15.591187572 (407h9m15.462270607s) [noise=true , address=185.167.204.205:4009 , piece_id=JTHX2O6T5HRENGEMAFJUWQUEUYAIG2KQ5HDEHF4BK42IT2ER7EIA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.591766795 (407h9m15.46284984s) [noise=true , address=185.167.204.205:4006 , piece_id=XHNFS6G7EX4BQWKOK4JUWYV3OTVJ36A6VOP4N7G2XPZKNNCFFKXA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.592074521 (407h9m15.463157566s) [noise=true , address=185.167.204.206:4060 , piece_id=WN6T5TUBVMURPSBBXGXCTLPZZ74PFNSRTTWXI2JPUTY74DPH3YSQ, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.593304694 (407h9m15.464387729s) [noise=true , address=185.167.204.205:4036 , piece_id=5QWJTTL6TZIZSSYHCBEJBKGX7BS76VVTELWTKEAKTOVYKINXVWDQ, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.593638659 (407h9m15.464721694s) [noise=true , address=185.167.204.205:4004 , piece_id=PYGDQZMAKOVJWB7AWEPFMPMB6C24FMS4HT2UP2AKZNYGG4DGZR3Q, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.593818251 (407h9m15.464901286s) [noise=true , address=185.167.204.206:4048 , piece_id=52WC3XH2K2BHG33BQPRENPNYDT7EOKCNBCTZKE3LIONDJHVATEYA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.594252549 (407h9m15.465335594s) [noise=true , address=185.167.204.205:4034 , piece_id=OS5H57TG2REFE535DK4T6N75VNIXPLQWW6DM55QCC2S7QBVZM3UQ, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.597113216 (407h9m15.468196261s) [noise=true , address=185.167.204.205:4032 , piece_id=FJMXJG2PQ4WZDSDCPJ6LLSVQKI7JJHUCDUZLU2BJZV7RZTLETM5A, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.597394541 (407h9m15.468477586s) [noise=true , address=185.167.204.205:4035 , piece_id=3D7T4SGPAJQZCA74CUTR4GJRE23R4JD5KBNW3BLEASSP4GSR5PHQ, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.597612756 (407h9m15.468695792s) [noise=true , address=185.167.204.206:4043 , piece_id=HL32YEZF67X2UT6SRMXJVJ4W5A3PC5SXOCM7BYQVMEC76TKEVJRA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.598018900 (407h9m15.469101945s) [noise=true , address=185.167.204.205:4012 , piece_id=A5G37BVL2GZ7LAY3FZWIZV4R4M7M6UPYDYACJFRD5HKDWWKB4Q6A, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.598254719 (407h9m15.469337765s) [noise=true , address=185.167.204.206:4052 , piece_id=V2DVAAXN5A4C3NQW556VMAC4CKPNLATBLZLKAGDGZHVT7BHEPN3Q, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.598480649 (407h9m15.469563694s) [noise=true , address=185.167.204.205:4018 , piece_id=NUYRZ7PXKJQ3PIR44O3E6N6COZZPRDNSMTQCFI5RN7FBRV2CBALA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.598968278 (407h9m15.470051323s) [noise=true , address=185.167.204.205:4033 , piece_id=A32ZTBDBWMCNBMZPD4DUWS5NKUKQ22ACQH6HAW5AKB3FNHBC56VA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.599287886 (407h9m15.470370931s) [noise=true , address=185.167.204.206:4072 , piece_id=I6PCSC5JYTJXLT6B34TTIVZQVX52R7TRHSRLRP4ZHG5J63XDZN6Q, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.599632252 (407h9m15.470715297s) [noise=true , address=185.167.204.206:4070 , piece_id=Y7IRY4E2GCF7TCFOIQJ7QH6IFEJBMW4WM6QBIS5SCUJ7E7ASV4HA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.599676297 (407h9m15.470759332s) [noise=true , address=185.167.204.206:4071 , piece_id=DVJAKIMPAUGWQPHDPGTXASQDONAKOZIGX5PQDBGBXNQAPMBPUFBA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.601435877 (407h9m15.472518912s) [noise=true , address=185.167.204.206:4057 , piece_id=ONW7YS5JG44B2STNFJGJWFEZ3WYXO74N3CXI3C6T5OTJYHGQPQZQ, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:17.299689191 (407h9m17.170772226s) [noise=true , address=89.149.201.147:4019 , piece_id=ILE7ZX5X6HBPUVYGCOQ5SNG3AA6B3YTXIF47CPLJ3MN5EXVOS2OA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:17.526601953 (407h9m17.397684998s) [noise=true , address=89.149.201.147:4015 , piece_id=ZIWRTD7HSP7YMRJET2TTJQFAQZM7KGDKJDLMYCWWY2W6SV2UVQBQ, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:18.294911214 (407h9m18.165994249s) [noise=true , address=89.149.201.147:4017 , piece_id=ZOSNSMZOBQVIKD2ZGJRAUWJEJ4JXJK452YWQQSDL77E77F6LRDZA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:18.402191355 (407h9m18.2732744s) [noise=true , address=89.149.201.147:4011 , piece_id=ATMM77IGBDSRHNJ5FQ2VKP7W27JUA2F6HYJYSQ62IL5JQN5PF5OA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:22.436508909 (407h9m22.307591944s) [noise=true , address=81.171.14.70:4037 , piece_id=NYGC2ZLUSBUAGFULQ46LXCVNQUJ5M4XT3YICVA2ZMBR2D6WPJ27Q, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:25.935108368 (407h9m25.806191413s) [noise=true , address=81.171.14.70:4046 , piece_id=NRMV6FG5CZSTLUNP2EHNCK6VM2Z2UHKBLCHLKEPQ225Q5DGYEBPA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:30.904793996 (407h9m30.775877041s) [noise=true , address=81.171.14.70:4029 , piece_id=APDMOYEMDCLOREXY2BBOK73RSQVWKIXUX4SCHRVFDSM7ZLWBEQXA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:32.926010512 (407h9m32.797093567s) [noise=true , address=81.171.14.70:4044 , piece_id=QCWQO6JQRRKSG5U2KVH3Y6I3UI2F7G3DOW4CH3PQZKUVQ6Q2XOHA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:32.926036552 (407h9m32.797119587s) [seg_pos={0 0}, upload=357453]: Segment reached optimal threshold of 26 pieces. Dec 22 14:38:32.926196857 (407h9m32.797279902s) [noise=true , address=81.171.14.70:4033 , piece_id=MKNAJOFD5CIHTA55XO476LOJJGIWSA7HBQFQND2YNM562W74IKJA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BnjusiXEfHYwTDL8soMkm8GYnhbkwy2oCAxVA7oiDyhFXozZ8): protocol: expected piece hash; context canceled; EOF Dec 22 14:38:32.926203058 (407h9m32.797286093s) [noise=true , address=81.171.14.70:4033 , piece_id=MKNAJOFD5CIHTA55XO476LOJJGIWSA7HBQFQND2YNM562W74IKJA, seg_pos={0 0}, upload=357453]: Long tail cancellation due to optimal threshold Dec 22 14:38:32.926237574 (407h9m32.797320609s) [noise=true , address=81.171.14.70:4031 , piece_id=7MD5N7PPKFPCFMOIXAGI3MPBDDX45GTLMA3CFGNM7BBGI6TSL6RA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12drdYnnWsuMyFo6LkcewAGGWnorRCAhxPpNqK41LvGy2zkBvng): protocol: expected piece hash; context canceled; EOF Dec 22 14:38:32.926250028 (407h9m32.797333063s) [noise=true , address=81.171.14.70:4031 , piece_id=7MD5N7PPKFPCFMOIXAGI3MPBDDX45GTLMA3CFGNM7BBGI6TSL6RA, seg_pos={0 0}, upload=357453]: Long tail cancellation due to optimal threshold Dec 22 14:38:32.926299612 (407h9m32.797382647s) [seg_pos={0 0}, upload=357453]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 22 14:38:32.926403941 (407h9m32.797486986s) [upload=357453]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 14:38:32.926409692 (407h9m32.797492737s) [upload=357453]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 14:39:04.793379844 (407h10m4.664462889s) [upload=357453]: Done uploading segments... Dec 22 14:39:04.793413608 (407h10m4.664496643s) [upload=357453]: Upload finished. err: <nil> Duration: 8.021036972s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [................................................X...........] duration: 3.747489411s trace: https://jaeger.dev.storj.io/trace/2e095cd265881ac3 FAIL (2025-12-22 15:11:49 UTC) orbiter warning: context deadline exceeded Duration: 3.747515723s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [.........................X.....XX.....XX.XXXXX.X.X.X.XX.XX.X] duration: 30.0026374s trace: https://jaeger.dev.storj.io/trace/3e82886fd6a1c084 FAIL (2025-12-22 15:11:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi, Piece ID: 3FZVJVVRJUEMERAFQQRZLGTQMSDMZLR2LXV4ZT5LOXKXB6ZXC2EA) context deadline exceeded; piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: 33W5BW6CYIRKHBPS7CKPG2EW4U223THJIBXWMBZV355EH5LXDVTQ) context deadline exceeded ======LOGS====== 7h42m18.412357402s) [upload=357936]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 15:11:18.894543383 (407h42m18.765626418s) [seg_pos={0 0}, upload=357936]: Begin upload segment... Dec 22 15:11:18.894615390 (407h42m18.765698435s) [seg_pos={0 0}, upload=357936]: Done begin upload segment. Dec 22 15:11:18.894651320 (407h42m18.765734355s) [noise=true , address=145.79.227.38:4979 , piece_id=RIMHGZ7SMUEFA5G63AABVS3YEIQWX5U2NCTPTRAUFIQ2DPQ36IGA, seg_pos={0 0}, upload=357936]: Uploading piece... Dec 22 15:11:18.894716633 (407h42m18.765799678s) [noise=true , address=145.79.227.48:4302 , piece_id=CXOXOGRPU6YRZVEJ53T5ZPXZNBOVGUBJM32UJMCS4ACN7K5D4WEQ, seg_pos={0 0}, upload=357936]: Uploading piece... Dec 22 15:11:18.896069850 (407h42m18.767152895s) [noise=true , address=145.79.227.11:4732 , piece_id=4NL3FE6UMJIJL3WKOUBMWVU7CY6JFXG7F7AC6DIQWZUKZQKJ5UHQ, seg_pos={0 0}, upload=357936]: Uploading piece... Dec 22 15:11:18.897093550 (407h42m18.768176585s) [noise=true , address=45.91.116.14:4189 , piece_id=VVJAC44BZLHO2TOFIZK46CVN5VU2DQ6Q66JLNWETYWMQS23FTLQA, seg_pos={0 0}, upload=357936]: Uploading piece... Dec 22 15:11:18.897653656 (407h42m18.768736702s) [noise=true , address=145.79.227.50:4373 , piece_id=2DNS3GGVJ2CWFLHRJKA24ZW6RUCKUCSHOTFSA2QKDBDPEEYIEQVQ, seg_pos={0 0}, upload=357936]: Uploading piece... Dec 22 15:11:18.898760935 (407h42m18.76984398s) [noise=true , address=145.79.227.36:4518 , piece_id=HVXLMSCH2FOPGWCFTEWTOVNG74FSI77PPJA4RHRGKGGKE2S5NTUA, seg_pos={0 0}, upload=357936]: Uploading piece... Dec 22 15:11:18.899938537 (407h42m18.771021572s) [noise=true , address=77.90.31.11:4556 , piece_id=IIJHTPY5RN5GXWEKL3FJGZK6IEDQBT66KAZZH4KLDHEADGLYIXKQ, seg_pos={0 0}, upload=357936]: Uploading piece... Dec 22 15:11:18.901069170 (407h42m18.772152215s) [noise=true , address=145.79.227.45:4230 , piece_id=TMYOMFVOHFKHUMILZKN3WXIDVZRMJL6RWQ7ROWEWSRHQL6CUK4CA, seg_pos={0 0}, upload=357936]: Uploading piece... Dec 22 15:11:18.902748177 (407h42m18.773831222s) [noise=true , address=145.79.227.37:4566 , piece_id=NKZAMRAPEG75YXUIERL577YHNLTYOA3DXSEGNV3ZVHKGX4SHYJTA, seg_pos={0 0}, upload=357936]: Uploading piece... Dec 22 15:11:18.904562291 (407h42m18.775645326s) [noise=true , address=45.91.116.9:4262 , piece_id=5Q533VMZE3K7XU6YGLU7CTKN4TUJQHPJVU6NZ65A5JDU672NC2XA, seg_pos={0 0}, upload=357936]: Uploading piece... Dec 22 15:11:18.905492002 (407h42m18.776575047s) [noise=true , address=145.79.227.35:4481 , piece_id=KGJ66J2I3UDCTY7NMRDMSC5MCCYCPSUFHELK2WZMDUAW7D6RK3RA, seg_pos={0 0}, upload=357936]: Uploading piece... Dec 22 15:11:18.906216331 (407h42m18.777299366s) [noise=true , address=145.79.227.43:4157 , piece_id=KTHFLO3UNPHVSE3ZKEYFBXEJIMM5XKG45ZS225O6DGAHJN3IPODA, seg_pos={0 0}, upload=357936]: Uploading piece... Dec 22 15:11:18.908863903 (407h42m18.779946998s) [noise=true , address=145.79.227.42:4122 , piece_id=3FZVJVVRJUEMERAFQQRZLGTQMSDMZLR2LXV4ZT5LOXKXB6ZXC2EA, seg_pos={0 0}, upload=357936]: Uploading piece... Dec 22 15:11:18.909042793 (407h42m18.780125828s) [noise=true , address=145.79.227.47:4267 , piece_id=WZNJNFHG46IDQIKOZMMQGMSSYXYEBN6RQ6IK7URC24WDYKWSEWQA, seg_pos={0 0}, upload=357936]: Uploading piece... Dec 22 15:11:18.910706561 (407h42m18.781789597s) [noise=true , address=45.91.116.13:4156 , piece_id=33W5BW6CYIRKHBPS7CKPG2EW4U223THJIBXWMBZV355EH5LXDVTQ, seg_pos={0 0}, upload=357936]: Uploading piece... Dec 22 15:11:18.916958936 (407h42m18.788041982s) [noise=true , address=45.91.116.5:4298 , piece_id=2SHSVSYA2ZE3UOTV7CFWW7F6N3OP6HKMSL4B3BKHUTK3I2734XFQ, seg_pos={0 0}, upload=357936]: Uploading piece... Dec 22 15:11:20.743404962 (407h42m20.614487997s) [upload=357936]: Next returned nil segment Dec 22 15:11:20.743438837 (407h42m20.614521872s) [upload=357936]: Waiting for error group managing segments... Dec 22 15:11:21.094203174 (407h42m20.965286209s) [noise=true , address=77.90.31.11:4556 , piece_id=IIJHTPY5RN5GXWEKL3FJGZK6IEDQBT66KAZZH4KLDHEADGLYIXKQ, seg_pos={0 0}, upload=357936]: Done uploading piece. err: <nil> Dec 22 15:11:21.168042658 (407h42m21.039125693s) [noise=true , address=45.91.116.13:4156 , piece_id=33W5BW6CYIRKHBPS7CKPG2EW4U223THJIBXWMBZV355EH5LXDVTQ, seg_pos={0 0}, upload=357936]: Done uploading piece. err: <nil> Dec 22 15:11:21.265974607 (407h42m21.137057652s) [noise=true , address=45.91.116.9:4262 , piece_id=5Q533VMZE3K7XU6YGLU7CTKN4TUJQHPJVU6NZ65A5JDU672NC2XA, seg_pos={0 0}, upload=357936]: Done uploading piece. err: <nil> Dec 22 15:11:21.271602143 (407h42m21.142685178s) [noise=true , address=45.91.116.5:4298 , piece_id=2SHSVSYA2ZE3UOTV7CFWW7F6N3OP6HKMSL4B3BKHUTK3I2734XFQ, seg_pos={0 0}, upload=357936]: Done uploading piece. err: <nil> Dec 22 15:11:21.866374715 (407h42m21.73745776s) [noise=true , address=145.79.227.36:4518 , piece_id=HVXLMSCH2FOPGWCFTEWTOVNG74FSI77PPJA4RHRGKGGKE2S5NTUA, seg_pos={0 0}, upload=357936]: Done uploading piece. err: <nil> Dec 22 15:11:21.878682171 (407h42m21.749765216s) [noise=true , address=145.79.227.37:4566 , piece_id=NKZAMRAPEG75YXUIERL577YHNLTYOA3DXSEGNV3ZVHKGX4SHYJTA, seg_pos={0 0}, upload=357936]: Done uploading piece. err: <nil> Dec 22 15:11:21.896856458 (407h42m21.767939493s) [noise=true , address=145.79.227.50:4373 , piece_id=2DNS3GGVJ2CWFLHRJKA24ZW6RUCKUCSHOTFSA2QKDBDPEEYIEQVQ, seg_pos={0 0}, upload=357936]: Done uploading piece. err: <nil> Dec 22 15:11:21.909550730 (407h42m21.780633765s) [noise=true , address=145.79.227.47:4267 , piece_id=WZNJNFHG46IDQIKOZMMQGMSSYXYEBN6RQ6IK7URC24WDYKWSEWQA, seg_pos={0 0}, upload=357936]: Done uploading piece. err: <nil> Dec 22 15:11:21.916424268 (407h42m21.787507313s) [noise=true , address=145.79.227.38:4979 , piece_id=RIMHGZ7SMUEFA5G63AABVS3YEIQWX5U2NCTPTRAUFIQ2DPQ36IGA, seg_pos={0 0}, upload=357936]: Done uploading piece. err: <nil> Dec 22 15:11:21.929206518 (407h42m21.800289563s) [noise=true , address=145.79.227.42:4122 , piece_id=3FZVJVVRJUEMERAFQQRZLGTQMSDMZLR2LXV4ZT5LOXKXB6ZXC2EA, seg_pos={0 0}, upload=357936]: Done uploading piece. err: <nil> Dec 22 15:11:21.945519050 (407h42m21.816602095s) [noise=true , address=145.79.227.35:4481 , piece_id=KGJ66J2I3UDCTY7NMRDMSC5MCCYCPSUFHELK2WZMDUAW7D6RK3RA, seg_pos={0 0}, upload=357936]: Done uploading piece. err: <nil> Dec 22 15:11:22.013704858 (407h42m21.884787903s) [noise=true , address=45.91.116.14:4189 , piece_id=VVJAC44BZLHO2TOFIZK46CVN5VU2DQ6Q66JLNWETYWMQS23FTLQA, seg_pos={0 0}, upload=357936]: Done uploading piece. err: <nil> Dec 22 15:11:22.620403920 (407h42m22.491486965s) [noise=true , address=145.79.227.11:4732 , piece_id=4NL3FE6UMJIJL3WKOUBMWVU7CY6JFXG7F7AC6DIQWZUKZQKJ5UHQ, seg_pos={0 0}, upload=357936]: Done uploading piece. err: <nil> Dec 22 15:11:22.620437534 (407h42m22.491520579s) [seg_pos={0 0}, upload=357936]: Segment reached optimal threshold of 13 pieces. Dec 22 15:11:22.620775738 (407h42m22.491858783s) [noise=true , address=145.79.227.48:4302 , piece_id=CXOXOGRPU6YRZVEJ53T5ZPXZNBOVGUBJM32UJMCS4ACN7K5D4WEQ, seg_pos={0 0}, upload=357936]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tGpuYdPLiy4ncDjxrZbQfp7Bq79NVBHDmr4vjaL4qMpudrvgR): protocol: write tcp 188.166.225.170:60536->145.79.227.48:4302: use of closed network connection; write tcp 188.166.225.170:60536->145.79.227.48:4302: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60536->145.79.227.48:4302: use of closed network connection Dec 22 15:11:22.620784655 (407h42m22.4918677s) [noise=true , address=145.79.227.48:4302 , piece_id=CXOXOGRPU6YRZVEJ53T5ZPXZNBOVGUBJM32UJMCS4ACN7K5D4WEQ, seg_pos={0 0}, upload=357936]: Long tail cancellation due to optimal threshold Dec 22 15:11:22.620813109 (407h42m22.491896144s) [noise=true , address=145.79.227.43:4157 , piece_id=KTHFLO3UNPHVSE3ZKEYFBXEJIMM5XKG45ZS225O6DGAHJN3IPODA, seg_pos={0 0}, upload=357936]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz): protocol: write tcp 188.166.225.170:34468->145.79.227.43:4157: use of closed network connection; write tcp 188.166.225.170:34468->145.79.227.43:4157: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34468->145.79.227.43:4157: use of closed network connection Dec 22 15:11:22.620821395 (407h42m22.49190444s) [noise=true , address=145.79.227.43:4157 , piece_id=KTHFLO3UNPHVSE3ZKEYFBXEJIMM5XKG45ZS225O6DGAHJN3IPODA, seg_pos={0 0}, upload=357936]: Long tail cancellation due to optimal threshold Dec 22 15:11:22.620864397 (407h42m22.491947442s) [noise=true , address=145.79.227.45:4230 , piece_id=TMYOMFVOHFKHUMILZKN3WXIDVZRMJL6RWQ7ROWEWSRHQL6CUK4CA, seg_pos={0 0}, upload=357936]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12S5FyCphEYSuT8Bz2aZhiRkm6PYDnhD8NmdMQuXGscbMfaaeis): protocol: write tcp 188.166.225.170:49496->145.79.227.45:4230: use of closed network connection; write tcp 188.166.225.170:49496->145.79.227.45:4230: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49496->145.79.227.45:4230: use of closed network connection Dec 22 15:11:22.620873013 (407h42m22.491956048s) [noise=true , address=145.79.227.45:4230 , piece_id=TMYOMFVOHFKHUMILZKN3WXIDVZRMJL6RWQ7ROWEWSRHQL6CUK4CA, seg_pos={0 0}, upload=357936]: Long tail cancellation due to optimal threshold Dec 22 15:11:22.620899523 (407h42m22.491982569s) [seg_pos={0 0}, upload=357936]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 15:11:22.621040522 (407h42m22.492123567s) [upload=357936]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 15:11:22.621047695 (407h42m22.49213074s) [upload=357936]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 15:11:23.037599554 (407h42m22.908682599s) [upload=357936]: Done uploading segments... Dec 22 15:11:23.037624171 (407h42m22.908707216s) [upload=357936]: Upload finished. err: <nil> Duration:* 30.00267961s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 20.737418988s trace: https://jaeger.dev.storj.io/trace/718f722b1bda7fe2 FAIL (2025-12-22 15:11:42 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== VMAC4CKPNLATBLZLKAGDGZHVT7BHEPN3Q, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.367093115 (407h9m14.23817616s) [noise=true , address=185.167.204.206:4043 , piece_id=HL32YEZF67X2UT6SRMXJVJ4W5A3PC5SXOCM7BYQVMEC76TKEVJRA, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.367396732 (407h9m14.238479767s) [noise=true , address=185.167.204.205:4032 , piece_id=FJMXJG2PQ4WZDSDCPJ6LLSVQKI7JJHUCDUZLU2BJZV7RZTLETM5A, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.367549273 (407h9m14.238632308s) [noise=true , address=185.167.204.206:4070 , piece_id=Y7IRY4E2GCF7TCFOIQJ7QH6IFEJBMW4WM6QBIS5SCUJ7E7ASV4HA, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.367749263 (407h9m14.238832308s) [noise=true , address=81.171.14.70:4046 , piece_id=NRMV6FG5CZSTLUNP2EHNCK6VM2Z2UHKBLCHLKEPQ225Q5DGYEBPA, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.368034046 (407h9m14.239117091s) [noise=true , address=185.167.204.206:4060 , piece_id=WN6T5TUBVMURPSBBXGXCTLPZZ74PFNSRTTWXI2JPUTY74DPH3YSQ, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.368552724 (407h9m14.239635759s) [noise=true , address=81.171.14.70:4029 , piece_id=APDMOYEMDCLOREXY2BBOK73RSQVWKIXUX4SCHRVFDSM7ZLWBEQXA, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.369160691 (407h9m14.240243736s) [noise=true , address=185.167.204.206:4057 , piece_id=ONW7YS5JG44B2STNFJGJWFEZ3WYXO74N3CXI3C6T5OTJYHGQPQZQ, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.369714496 (407h9m14.240797541s) [noise=true , address=185.167.204.206:4071 , piece_id=DVJAKIMPAUGWQPHDPGTXASQDONAKOZIGX5PQDBGBXNQAPMBPUFBA, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.370249614 (407h9m14.241332649s) [noise=true , address=89.149.201.147:4019 , piece_id=ILE7ZX5X6HBPUVYGCOQ5SNG3AA6B3YTXIF47CPLJ3MN5EXVOS2OA, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.372019455 (407h9m14.24310249s) [noise=true , address=89.149.201.147:4015 , piece_id=ZIWRTD7HSP7YMRJET2TTJQFAQZM7KGDKJDLMYCWWY2W6SV2UVQBQ, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:15.199398795 (407h9m15.07048184s) [upload=357453]: Next returned nil segment Dec 22 14:38:15.199414966 (407h9m15.070498011s) [upload=357453]: Waiting for error group managing segments... Dec 22 14:38:15.591187572 (407h9m15.462270607s) [noise=true , address=185.167.204.205:4009 , piece_id=JTHX2O6T5HRENGEMAFJUWQUEUYAIG2KQ5HDEHF4BK42IT2ER7EIA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.591766795 (407h9m15.46284984s) [noise=true , address=185.167.204.205:4006 , piece_id=XHNFS6G7EX4BQWKOK4JUWYV3OTVJ36A6VOP4N7G2XPZKNNCFFKXA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.592074521 (407h9m15.463157566s) [noise=true , address=185.167.204.206:4060 , piece_id=WN6T5TUBVMURPSBBXGXCTLPZZ74PFNSRTTWXI2JPUTY74DPH3YSQ, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.593304694 (407h9m15.464387729s) [noise=true , address=185.167.204.205:4036 , piece_id=5QWJTTL6TZIZSSYHCBEJBKGX7BS76VVTELWTKEAKTOVYKINXVWDQ, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.593638659 (407h9m15.464721694s) [noise=true , address=185.167.204.205:4004 , piece_id=PYGDQZMAKOVJWB7AWEPFMPMB6C24FMS4HT2UP2AKZNYGG4DGZR3Q, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.593818251 (407h9m15.464901286s) [noise=true , address=185.167.204.206:4048 , piece_id=52WC3XH2K2BHG33BQPRENPNYDT7EOKCNBCTZKE3LIONDJHVATEYA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.594252549 (407h9m15.465335594s) [noise=true , address=185.167.204.205:4034 , piece_id=OS5H57TG2REFE535DK4T6N75VNIXPLQWW6DM55QCC2S7QBVZM3UQ, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.597113216 (407h9m15.468196261s) [noise=true , address=185.167.204.205:4032 , piece_id=FJMXJG2PQ4WZDSDCPJ6LLSVQKI7JJHUCDUZLU2BJZV7RZTLETM5A, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.597394541 (407h9m15.468477586s) [noise=true , address=185.167.204.205:4035 , piece_id=3D7T4SGPAJQZCA74CUTR4GJRE23R4JD5KBNW3BLEASSP4GSR5PHQ, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.597612756 (407h9m15.468695792s) [noise=true , address=185.167.204.206:4043 , piece_id=HL32YEZF67X2UT6SRMXJVJ4W5A3PC5SXOCM7BYQVMEC76TKEVJRA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.598018900 (407h9m15.469101945s) [noise=true , address=185.167.204.205:4012 , piece_id=A5G37BVL2GZ7LAY3FZWIZV4R4M7M6UPYDYACJFRD5HKDWWKB4Q6A, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.598254719 (407h9m15.469337765s) [noise=true , address=185.167.204.206:4052 , piece_id=V2DVAAXN5A4C3NQW556VMAC4CKPNLATBLZLKAGDGZHVT7BHEPN3Q, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.598480649 (407h9m15.469563694s) [noise=true , address=185.167.204.205:4018 , piece_id=NUYRZ7PXKJQ3PIR44O3E6N6COZZPRDNSMTQCFI5RN7FBRV2CBALA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.598968278 (407h9m15.470051323s) [noise=true , address=185.167.204.205:4033 , piece_id=A32ZTBDBWMCNBMZPD4DUWS5NKUKQ22ACQH6HAW5AKB3FNHBC56VA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.599287886 (407h9m15.470370931s) [noise=true , address=185.167.204.206:4072 , piece_id=I6PCSC5JYTJXLT6B34TTIVZQVX52R7TRHSRLRP4ZHG5J63XDZN6Q, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.599632252 (407h9m15.470715297s) [noise=true , address=185.167.204.206:4070 , piece_id=Y7IRY4E2GCF7TCFOIQJ7QH6IFEJBMW4WM6QBIS5SCUJ7E7ASV4HA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.599676297 (407h9m15.470759332s) [noise=true , address=185.167.204.206:4071 , piece_id=DVJAKIMPAUGWQPHDPGTXASQDONAKOZIGX5PQDBGBXNQAPMBPUFBA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.601435877 (407h9m15.472518912s) [noise=true , address=185.167.204.206:4057 , piece_id=ONW7YS5JG44B2STNFJGJWFEZ3WYXO74N3CXI3C6T5OTJYHGQPQZQ, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:17.299689191 (407h9m17.170772226s) [noise=true , address=89.149.201.147:4019 , piece_id=ILE7ZX5X6HBPUVYGCOQ5SNG3AA6B3YTXIF47CPLJ3MN5EXVOS2OA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:17.526601953 (407h9m17.397684998s) [noise=true , address=89.149.201.147:4015 , piece_id=ZIWRTD7HSP7YMRJET2TTJQFAQZM7KGDKJDLMYCWWY2W6SV2UVQBQ, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:18.294911214 (407h9m18.165994249s) [noise=true , address=89.149.201.147:4017 , piece_id=ZOSNSMZOBQVIKD2ZGJRAUWJEJ4JXJK452YWQQSDL77E77F6LRDZA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:18.402191355 (407h9m18.2732744s) [noise=true , address=89.149.201.147:4011 , piece_id=ATMM77IGBDSRHNJ5FQ2VKP7W27JUA2F6HYJYSQ62IL5JQN5PF5OA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:22.436508909 (407h9m22.307591944s) [noise=true , address=81.171.14.70:4037 , piece_id=NYGC2ZLUSBUAGFULQ46LXCVNQUJ5M4XT3YICVA2ZMBR2D6WPJ27Q, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:25.935108368 (407h9m25.806191413s) [noise=true , address=81.171.14.70:4046 , piece_id=NRMV6FG5CZSTLUNP2EHNCK6VM2Z2UHKBLCHLKEPQ225Q5DGYEBPA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:30.904793996 (407h9m30.775877041s) [noise=true , address=81.171.14.70:4029 , piece_id=APDMOYEMDCLOREXY2BBOK73RSQVWKIXUX4SCHRVFDSM7ZLWBEQXA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:32.926010512 (407h9m32.797093567s) [noise=true , address=81.171.14.70:4044 , piece_id=QCWQO6JQRRKSG5U2KVH3Y6I3UI2F7G3DOW4CH3PQZKUVQ6Q2XOHA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:32.926036552 (407h9m32.797119587s) [seg_pos={0 0}, upload=357453]: Segment reached optimal threshold of 26 pieces. Dec 22 14:38:32.926196857 (407h9m32.797279902s) [noise=true , address=81.171.14.70:4033 , piece_id=MKNAJOFD5CIHTA55XO476LOJJGIWSA7HBQFQND2YNM562W74IKJA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BnjusiXEfHYwTDL8soMkm8GYnhbkwy2oCAxVA7oiDyhFXozZ8): protocol: expected piece hash; context canceled; EOF Dec 22 14:38:32.926203058 (407h9m32.797286093s) [noise=true , address=81.171.14.70:4033 , piece_id=MKNAJOFD5CIHTA55XO476LOJJGIWSA7HBQFQND2YNM562W74IKJA, seg_pos={0 0}, upload=357453]: Long tail cancellation due to optimal threshold Dec 22 14:38:32.926237574 (407h9m32.797320609s) [noise=true , address=81.171.14.70:4031 , piece_id=7MD5N7PPKFPCFMOIXAGI3MPBDDX45GTLMA3CFGNM7BBGI6TSL6RA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12drdYnnWsuMyFo6LkcewAGGWnorRCAhxPpNqK41LvGy2zkBvng): protocol: expected piece hash; context canceled; EOF Dec 22 14:38:32.926250028 (407h9m32.797333063s) [noise=true , address=81.171.14.70:4031 , piece_id=7MD5N7PPKFPCFMOIXAGI3MPBDDX45GTLMA3CFGNM7BBGI6TSL6RA, seg_pos={0 0}, upload=357453]: Long tail cancellation due to optimal threshold Dec 22 14:38:32.926299612 (407h9m32.797382647s) [seg_pos={0 0}, upload=357453]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 22 14:38:32.926403941 (407h9m32.797486986s) [upload=357453]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 14:38:32.926409692 (407h9m32.797492737s) [upload=357453]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 14:39:04.793379844 (407h10m4.664462889s) [upload=357453]: Done uploading segments... Dec 22 14:39:04.793413608 (407h10m4.664496643s) [upload=357453]: Upload finished. err: <nil> Duration: 20.737447974s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.................................................X..........] duration: 3.655522601s trace: https://jaeger.dev.storj.io/trace/2e095cd265881ac3 FAIL (2025-12-22 15:11:11 UTC) orbiter warning: context deadline exceeded Duration: 3.65554324s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 6.262786479s trace: https://jaeger.dev.storj.io/trace/510cfc1b6c68e873 FAIL (2025-12-22 15:11:09 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 6.262812738s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 1.502141741s trace: https://jaeger.dev.storj.io/trace/62b55ec615ab6123 FAIL (2025-12-22 15:10:54 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.502179374s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [..........................X.....XX.....XX.XXXXX.X.X.X.XX.XX.] duration: 26.398295035s trace: https://jaeger.dev.storj.io/trace/3265ee8ef95e96c6 FAIL (2025-12-22 15:10:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN, Piece ID: A42WDIQJ5342JCD4CTGUBVX7JP2QFMSSO3U6O5JNHN6AHLBF5AHA) context deadline exceeded; piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: 5O3H2GF6VFEVBAKPIHLPB7WY22VH3EDQBPL7YQLHU4LAFTG7BBBQ) context deadline exceeded ======LOGS====== Dec 22 15:09:21.838939174 (407h40m21.710022219s) [upload=357906]: Starting upload Dec 22 15:09:21.839019637 (407h40m21.710102672s) [upload=357906]: Done starting upload Dec 22 15:09:21.839136389 (407h40m21.710219424s) [upload=357906]: Uploading segments... Dec 22 15:09:21.839294510 (407h40m21.710377545s) [upload=357906]: Got next segment. Inline: false Dec 22 15:09:21.839330519 (407h40m21.710413564s) [upload=357906]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 15:09:21.839340608 (407h40m21.710423673s) [upload=357906]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 15:09:22.089684838 (407h40m21.960767883s) [seg_pos={0 0}, upload=357906]: Begin upload segment... Dec 22 15:09:22.089736596 (407h40m21.960819641s) [seg_pos={0 0}, upload=357906]: Done begin upload segment. Dec 22 15:09:22.089768256 (407h40m21.960851301s) [noise=true , address=145.79.227.14:4838 , piece_id=XC2NMZH7G42IV3JMHZOS4KSCHYQDKGPUDWTFAXMF2AYPRHZ6OE5A, seg_pos={0 0}, upload=357906]: Uploading piece... Dec 22 15:09:22.089829353 (407h40m21.960912398s) [noise=true , address=145.79.227.11:4732 , piece_id=ZMJP72QLGB4JNY5WJ3GAJL3RIQRSLYQ4SSZKB3HUJB26T526GYZQ, seg_pos={0 0}, upload=357906]: Uploading piece... Dec 22 15:09:22.089895218 (407h40m21.960978253s) [noise=true , address=145.79.227.39:4015 , piece_id=DAPJGM5OOEEMX5QWZO4NMGZHSGTTLM7XE73KSI4NKUIJP3QTITEA, seg_pos={0 0}, upload=357906]: Uploading piece... Dec 22 15:09:22.090318665 (407h40m21.96140171s) [noise=true , address=145.79.227.45:4230 , piece_id=MLBZ724R6ZLW5L47XMIWU36EWS2FEHX2X6YL6S2NAN4NSSILJDKA, seg_pos={0 0}, upload=357906]: Uploading piece... Dec 22 15:09:22.090517503 (407h40m21.961600549s) [noise=true , address=77.90.31.10:4519 , piece_id=5O3H2GF6VFEVBAKPIHLPB7WY22VH3EDQBPL7YQLHU4LAFTG7BBBQ, seg_pos={0 0}, upload=357906]: Uploading piece... Dec 22 15:09:22.090731230 (407h40m21.961814275s) [noise=true , address=77.90.31.17:4770 , piece_id=YODZ3OXGCC6Y5DWB4O3B4GT5YR2XYF46KXM3DTNJUZUDCIYBQICQ, seg_pos={0 0}, upload=357906]: Uploading piece... Dec 22 15:09:22.090930470 (407h40m21.962013515s) [noise=true , address=145.79.227.43:4159 , piece_id=JMESWCEA6ECDJKDBCWOTT56NZFGKNGB6UWAVWE5M44DLHSHWNQQA, seg_pos={0 0}, upload=357906]: Uploading piece... Dec 22 15:09:22.091148405 (407h40m21.96223145s) [noise=true , address=145.79.227.48:4301 , piece_id=QFPOY4WER7LUOML4ZL6ULMYIYWCUSNPQZCD2TOUVQQBXBRT4NSJA, seg_pos={0 0}, upload=357906]: Uploading piece... Dec 22 15:09:22.091311997 (407h40m21.962395032s) [noise=true , address=145.79.227.47:4265 , piece_id=NBJXC33D3D2UEW3NU3WRYHTUL5ZQIOXXY3SW7SKS2KPHBVXFK6XQ, seg_pos={0 0}, upload=357906]: Uploading piece... Dec 22 15:09:22.091613239 (407h40m21.962696285s) [noise=true , address=145.79.227.37:4565 , piece_id=ZHQHRJWYMSLYOWEH7R356EGBQ7LZZ3EWRFGVNBCEEUYX5FGG6YRQ, seg_pos={0 0}, upload=357906]: Uploading piece... Dec 22 15:09:22.091698993 (407h40m21.962782028s) [noise=true , address=45.91.116.4:4940 , piece_id=DBCEPFQ2OOYYONSWJ4K6I2Q2EYTTMPRGCQFK3ZRILPMOU7M7RCJQ, seg_pos={0 0}, upload=357906]: Uploading piece... Dec 22 15:09:22.092037617 (407h40m21.963120662s) [noise=true , address=45.91.116.5:4300 , piece_id=RHLGNEGW7224ZF73MLM7RYGFM77QRECILT4BQYD4RGVUWHHPT5QQ, seg_pos={0 0}, upload=357906]: Uploading piece... Dec 22 15:09:22.092126357 (407h40m21.963209392s) [noise=true , address=145.79.227.38:4979 , piece_id=ZD7MRX62QDJUNTS3MVOTGYYIBMEIYSGDFQF5QBISDWF2GESUBCXQ, seg_pos={0 0}, upload=357906]: Uploading piece... Dec 22 15:09:22.092430757 (407h40m21.963513802s) [noise=true , address=45.91.116.6:4976 , piece_id=GFYZEJDSJYETZVI66ECXGBQLFT5L6QS4I2R43P2H2TFKDSJ6PH5A, seg_pos={0 0}, upload=357906]: Uploading piece... Dec 22 15:09:22.092951378 (407h40m21.964034413s) [noise=true , address=145.79.227.49:4338 , piece_id=A42WDIQJ5342JCD4CTGUBVX7JP2QFMSSO3U6O5JNHN6AHLBF5AHA, seg_pos={0 0}, upload=357906]: Uploading piece... Dec 22 15:09:22.093388600 (407h40m21.964471645s) [noise=true , address=145.79.227.30:4762 , piece_id=4GA25BN5GABFBYXKUKFBHOXPETXVWU7WUJ2N2UWGNEJVGOIXVCOA, seg_pos={0 0}, upload=357906]: Uploading piece... Dec 22 15:09:24.024930087 (407h40m23.896013122s) [upload=357906]: Next returned nil segment Dec 22 15:09:24.024948993 (407h40m23.896032028s) [upload=357906]: Waiting for error group managing segments... Dec 22 15:09:24.344748107 (407h40m24.215831142s) [noise=true , address=77.90.31.17:4770 , piece_id=YODZ3OXGCC6Y5DWB4O3B4GT5YR2XYF46KXM3DTNJUZUDCIYBQICQ, seg_pos={0 0}, upload=357906]: Done uploading piece. err: <nil> Dec 22 15:09:24.360909511 (407h40m24.231992546s) [noise=true , address=45.91.116.5:4300 , piece_id=RHLGNEGW7224ZF73MLM7RYGFM77QRECILT4BQYD4RGVUWHHPT5QQ, seg_pos={0 0}, upload=357906]: Done uploading piece. err: <nil> Dec 22 15:09:24.459274315 (407h40m24.33035736s) [noise=true , address=77.90.31.10:4519 , piece_id=5O3H2GF6VFEVBAKPIHLPB7WY22VH3EDQBPL7YQLHU4LAFTG7BBBQ, seg_pos={0 0}, upload=357906]: Done uploading piece. err: <nil> Dec 22 15:09:24.489593938 (407h40m24.360676974s) [noise=true , address=45.91.116.6:4976 , piece_id=GFYZEJDSJYETZVI66ECXGBQLFT5L6QS4I2R43P2H2TFKDSJ6PH5A, seg_pos={0 0}, upload=357906]: Done uploading piece. err: <nil> Dec 22 15:09:25.044856997 (407h40m24.915940032s) [noise=true , address=145.79.227.37:4565 , piece_id=ZHQHRJWYMSLYOWEH7R356EGBQ7LZZ3EWRFGVNBCEEUYX5FGG6YRQ, seg_pos={0 0}, upload=357906]: Done uploading piece. err: <nil> Dec 22 15:09:25.045891217 (407h40m24.916974262s) [noise=true , address=145.79.227.43:4159 , piece_id=JMESWCEA6ECDJKDBCWOTT56NZFGKNGB6UWAVWE5M44DLHSHWNQQA, seg_pos={0 0}, upload=357906]: Done uploading piece. err: <nil> Dec 22 15:09:25.069597367 (407h40m24.940680403s) [noise=true , address=145.79.227.14:4838 , piece_id=XC2NMZH7G42IV3JMHZOS4KSCHYQDKGPUDWTFAXMF2AYPRHZ6OE5A, seg_pos={0 0}, upload=357906]: Done uploading piece. err: <nil> Dec 22 15:09:25.077829733 (407h40m24.948912778s) [noise=true , address=145.79.227.45:4230 , piece_id=MLBZ724R6ZLW5L47XMIWU36EWS2FEHX2X6YL6S2NAN4NSSILJDKA, seg_pos={0 0}, upload=357906]: Done uploading piece. err: <nil> Dec 22 15:09:25.103859138 (407h40m24.974942183s) [noise=true , address=45.91.116.4:4940 , piece_id=DBCEPFQ2OOYYONSWJ4K6I2Q2EYTTMPRGCQFK3ZRILPMOU7M7RCJQ, seg_pos={0 0}, upload=357906]: Done uploading piece. err: <nil> Dec 22 15:09:25.105223886 (407h40m24.976306931s) [noise=true , address=145.79.227.49:4338 , piece_id=A42WDIQJ5342JCD4CTGUBVX7JP2QFMSSO3U6O5JNHN6AHLBF5AHA, seg_pos={0 0}, upload=357906]: Done uploading piece. err: <nil> Dec 22 15:09:25.105887830 (407h40m24.976970865s) [noise=true , address=145.79.227.11:4732 , piece_id=ZMJP72QLGB4JNY5WJ3GAJL3RIQRSLYQ4SSZKB3HUJB26T526GYZQ, seg_pos={0 0}, upload=357906]: Done uploading piece. err: <nil> Dec 22 15:09:25.923336919 (407h40m25.794419954s) [noise=true , address=145.79.227.39:4015 , piece_id=DAPJGM5OOEEMX5QWZO4NMGZHSGTTLM7XE73KSI4NKUIJP3QTITEA, seg_pos={0 0}, upload=357906]: Done uploading piece. err: <nil> Dec 22 15:09:26.448805232 (407h40m26.319888267s) [noise=true , address=145.79.227.30:4762 , piece_id=4GA25BN5GABFBYXKUKFBHOXPETXVWU7WUJ2N2UWGNEJVGOIXVCOA, seg_pos={0 0}, upload=357906]: Done uploading piece. err: <nil> Dec 22 15:09:26.448824459 (407h40m26.319907494s) [seg_pos={0 0}, upload=357906]: Segment reached optimal threshold of 13 pieces. Dec 22 15:09:26.449017176 (407h40m26.320100211s) [noise=true , address=145.79.227.38:4979 , piece_id=ZD7MRX62QDJUNTS3MVOTGYYIBMEIYSGDFQF5QBISDWF2GESUBCXQ, seg_pos={0 0}, upload=357906]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma): protocol: expected piece hash; context canceled; EOF Dec 22 15:09:26.449023829 (407h40m26.320106864s) [noise=true , address=145.79.227.38:4979 , piece_id=ZD7MRX62QDJUNTS3MVOTGYYIBMEIYSGDFQF5QBISDWF2GESUBCXQ, seg_pos={0 0}, upload=357906]: Long tail cancellation due to optimal threshold Dec 22 15:09:26.449224140 (407h40m26.320307175s) [noise=true , address=145.79.227.47:4265 , piece_id=NBJXC33D3D2UEW3NU3WRYHTUL5ZQIOXXY3SW7SKS2KPHBVXFK6XQ, seg_pos={0 0}, upload=357906]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JvnJetuaEkstkPVNChKo8YQEYC68qRXPGQxx2AKZoHftkhajQ): protocol: expected piece hash; context canceled; EOF Dec 22 15:09:26.449230162 (407h40m26.320313197s) [noise=true , address=145.79.227.47:4265 , piece_id=NBJXC33D3D2UEW3NU3WRYHTUL5ZQIOXXY3SW7SKS2KPHBVXFK6XQ, seg_pos={0 0}, upload=357906]: Long tail cancellation due to optimal threshold Dec 22 15:09:26.449269767 (407h40m26.320352802s) [noise=true , address=145.79.227.48:4301 , piece_id=QFPOY4WER7LUOML4ZL6ULMYIYWCUSNPQZCD2TOUVQQBXBRT4NSJA, seg_pos={0 0}, upload=357906]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18HkaSrdHHnvmeNz1ormfcSBZH8bMUgkX7X3JmGuqkmtBoak29): protocol: expected piece hash; context canceled; EOF Dec 22 15:09:26.449292270 (407h40m26.320375315s) [noise=true , address=145.79.227.48:4301 , piece_id=QFPOY4WER7LUOML4ZL6ULMYIYWCUSNPQZCD2TOUVQQBXBRT4NSJA, seg_pos={0 0}, upload=357906]: Long tail cancellation due to optimal threshold Dec 22 15:09:26.449322487 (407h40m26.320405532s) [seg_pos={0 0}, upload=357906]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 15:09:26.449421045 (407h40m26.32050409s) [upload=357906]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 15:09:26.449437968 (407h40m26.320521003s) [upload=357906]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 15:09:26.760974789 (407h40m26.632057834s) [upload=357906]: Done uploading segments... Dec 22 15:09:26.760996400 (407h40m26.632079435s) [upload=357906]: Upload finished. err: <nil> Duration: 26.398376319s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 1.421434432s trace: https://jaeger.dev.storj.io/trace/62b55ec615ab6123 FAIL (2025-12-22 15:10:33 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.421462165s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..................................................X.........] duration: 3.51171774s trace: https://jaeger.dev.storj.io/trace/2e095cd265881ac3 FAIL (2025-12-22 15:10:33 UTC) orbiter warning: context deadline exceeded Duration: 3.511751755s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 6.573538778s trace: https://jaeger.dev.storj.io/trace/718f722b1bda7fe2 FAIL (2025-12-22 15:10:20 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== VMAC4CKPNLATBLZLKAGDGZHVT7BHEPN3Q, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.367093115 (407h9m14.23817616s) [noise=true , address=185.167.204.206:4043 , piece_id=HL32YEZF67X2UT6SRMXJVJ4W5A3PC5SXOCM7BYQVMEC76TKEVJRA, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.367396732 (407h9m14.238479767s) [noise=true , address=185.167.204.205:4032 , piece_id=FJMXJG2PQ4WZDSDCPJ6LLSVQKI7JJHUCDUZLU2BJZV7RZTLETM5A, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.367549273 (407h9m14.238632308s) [noise=true , address=185.167.204.206:4070 , piece_id=Y7IRY4E2GCF7TCFOIQJ7QH6IFEJBMW4WM6QBIS5SCUJ7E7ASV4HA, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.367749263 (407h9m14.238832308s) [noise=true , address=81.171.14.70:4046 , piece_id=NRMV6FG5CZSTLUNP2EHNCK6VM2Z2UHKBLCHLKEPQ225Q5DGYEBPA, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.368034046 (407h9m14.239117091s) [noise=true , address=185.167.204.206:4060 , piece_id=WN6T5TUBVMURPSBBXGXCTLPZZ74PFNSRTTWXI2JPUTY74DPH3YSQ, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.368552724 (407h9m14.239635759s) [noise=true , address=81.171.14.70:4029 , piece_id=APDMOYEMDCLOREXY2BBOK73RSQVWKIXUX4SCHRVFDSM7ZLWBEQXA, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.369160691 (407h9m14.240243736s) [noise=true , address=185.167.204.206:4057 , piece_id=ONW7YS5JG44B2STNFJGJWFEZ3WYXO74N3CXI3C6T5OTJYHGQPQZQ, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.369714496 (407h9m14.240797541s) [noise=true , address=185.167.204.206:4071 , piece_id=DVJAKIMPAUGWQPHDPGTXASQDONAKOZIGX5PQDBGBXNQAPMBPUFBA, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.370249614 (407h9m14.241332649s) [noise=true , address=89.149.201.147:4019 , piece_id=ILE7ZX5X6HBPUVYGCOQ5SNG3AA6B3YTXIF47CPLJ3MN5EXVOS2OA, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.372019455 (407h9m14.24310249s) [noise=true , address=89.149.201.147:4015 , piece_id=ZIWRTD7HSP7YMRJET2TTJQFAQZM7KGDKJDLMYCWWY2W6SV2UVQBQ, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:15.199398795 (407h9m15.07048184s) [upload=357453]: Next returned nil segment Dec 22 14:38:15.199414966 (407h9m15.070498011s) [upload=357453]: Waiting for error group managing segments... Dec 22 14:38:15.591187572 (407h9m15.462270607s) [noise=true , address=185.167.204.205:4009 , piece_id=JTHX2O6T5HRENGEMAFJUWQUEUYAIG2KQ5HDEHF4BK42IT2ER7EIA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.591766795 (407h9m15.46284984s) [noise=true , address=185.167.204.205:4006 , piece_id=XHNFS6G7EX4BQWKOK4JUWYV3OTVJ36A6VOP4N7G2XPZKNNCFFKXA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.592074521 (407h9m15.463157566s) [noise=true , address=185.167.204.206:4060 , piece_id=WN6T5TUBVMURPSBBXGXCTLPZZ74PFNSRTTWXI2JPUTY74DPH3YSQ, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.593304694 (407h9m15.464387729s) [noise=true , address=185.167.204.205:4036 , piece_id=5QWJTTL6TZIZSSYHCBEJBKGX7BS76VVTELWTKEAKTOVYKINXVWDQ, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.593638659 (407h9m15.464721694s) [noise=true , address=185.167.204.205:4004 , piece_id=PYGDQZMAKOVJWB7AWEPFMPMB6C24FMS4HT2UP2AKZNYGG4DGZR3Q, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.593818251 (407h9m15.464901286s) [noise=true , address=185.167.204.206:4048 , piece_id=52WC3XH2K2BHG33BQPRENPNYDT7EOKCNBCTZKE3LIONDJHVATEYA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.594252549 (407h9m15.465335594s) [noise=true , address=185.167.204.205:4034 , piece_id=OS5H57TG2REFE535DK4T6N75VNIXPLQWW6DM55QCC2S7QBVZM3UQ, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.597113216 (407h9m15.468196261s) [noise=true , address=185.167.204.205:4032 , piece_id=FJMXJG2PQ4WZDSDCPJ6LLSVQKI7JJHUCDUZLU2BJZV7RZTLETM5A, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.597394541 (407h9m15.468477586s) [noise=true , address=185.167.204.205:4035 , piece_id=3D7T4SGPAJQZCA74CUTR4GJRE23R4JD5KBNW3BLEASSP4GSR5PHQ, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.597612756 (407h9m15.468695792s) [noise=true , address=185.167.204.206:4043 , piece_id=HL32YEZF67X2UT6SRMXJVJ4W5A3PC5SXOCM7BYQVMEC76TKEVJRA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.598018900 (407h9m15.469101945s) [noise=true , address=185.167.204.205:4012 , piece_id=A5G37BVL2GZ7LAY3FZWIZV4R4M7M6UPYDYACJFRD5HKDWWKB4Q6A, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.598254719 (407h9m15.469337765s) [noise=true , address=185.167.204.206:4052 , piece_id=V2DVAAXN5A4C3NQW556VMAC4CKPNLATBLZLKAGDGZHVT7BHEPN3Q, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.598480649 (407h9m15.469563694s) [noise=true , address=185.167.204.205:4018 , piece_id=NUYRZ7PXKJQ3PIR44O3E6N6COZZPRDNSMTQCFI5RN7FBRV2CBALA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.598968278 (407h9m15.470051323s) [noise=true , address=185.167.204.205:4033 , piece_id=A32ZTBDBWMCNBMZPD4DUWS5NKUKQ22ACQH6HAW5AKB3FNHBC56VA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.599287886 (407h9m15.470370931s) [noise=true , address=185.167.204.206:4072 , piece_id=I6PCSC5JYTJXLT6B34TTIVZQVX52R7TRHSRLRP4ZHG5J63XDZN6Q, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.599632252 (407h9m15.470715297s) [noise=true , address=185.167.204.206:4070 , piece_id=Y7IRY4E2GCF7TCFOIQJ7QH6IFEJBMW4WM6QBIS5SCUJ7E7ASV4HA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.599676297 (407h9m15.470759332s) [noise=true , address=185.167.204.206:4071 , piece_id=DVJAKIMPAUGWQPHDPGTXASQDONAKOZIGX5PQDBGBXNQAPMBPUFBA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.601435877 (407h9m15.472518912s) [noise=true , address=185.167.204.206:4057 , piece_id=ONW7YS5JG44B2STNFJGJWFEZ3WYXO74N3CXI3C6T5OTJYHGQPQZQ, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:17.299689191 (407h9m17.170772226s) [noise=true , address=89.149.201.147:4019 , piece_id=ILE7ZX5X6HBPUVYGCOQ5SNG3AA6B3YTXIF47CPLJ3MN5EXVOS2OA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:17.526601953 (407h9m17.397684998s) [noise=true , address=89.149.201.147:4015 , piece_id=ZIWRTD7HSP7YMRJET2TTJQFAQZM7KGDKJDLMYCWWY2W6SV2UVQBQ, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:18.294911214 (407h9m18.165994249s) [noise=true , address=89.149.201.147:4017 , piece_id=ZOSNSMZOBQVIKD2ZGJRAUWJEJ4JXJK452YWQQSDL77E77F6LRDZA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:18.402191355 (407h9m18.2732744s) [noise=true , address=89.149.201.147:4011 , piece_id=ATMM77IGBDSRHNJ5FQ2VKP7W27JUA2F6HYJYSQ62IL5JQN5PF5OA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:22.436508909 (407h9m22.307591944s) [noise=true , address=81.171.14.70:4037 , piece_id=NYGC2ZLUSBUAGFULQ46LXCVNQUJ5M4XT3YICVA2ZMBR2D6WPJ27Q, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:25.935108368 (407h9m25.806191413s) [noise=true , address=81.171.14.70:4046 , piece_id=NRMV6FG5CZSTLUNP2EHNCK6VM2Z2UHKBLCHLKEPQ225Q5DGYEBPA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:30.904793996 (407h9m30.775877041s) [noise=true , address=81.171.14.70:4029 , piece_id=APDMOYEMDCLOREXY2BBOK73RSQVWKIXUX4SCHRVFDSM7ZLWBEQXA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:32.926010512 (407h9m32.797093567s) [noise=true , address=81.171.14.70:4044 , piece_id=QCWQO6JQRRKSG5U2KVH3Y6I3UI2F7G3DOW4CH3PQZKUVQ6Q2XOHA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:32.926036552 (407h9m32.797119587s) [seg_pos={0 0}, upload=357453]: Segment reached optimal threshold of 26 pieces. Dec 22 14:38:32.926196857 (407h9m32.797279902s) [noise=true , address=81.171.14.70:4033 , piece_id=MKNAJOFD5CIHTA55XO476LOJJGIWSA7HBQFQND2YNM562W74IKJA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BnjusiXEfHYwTDL8soMkm8GYnhbkwy2oCAxVA7oiDyhFXozZ8): protocol: expected piece hash; context canceled; EOF Dec 22 14:38:32.926203058 (407h9m32.797286093s) [noise=true , address=81.171.14.70:4033 , piece_id=MKNAJOFD5CIHTA55XO476LOJJGIWSA7HBQFQND2YNM562W74IKJA, seg_pos={0 0}, upload=357453]: Long tail cancellation due to optimal threshold Dec 22 14:38:32.926237574 (407h9m32.797320609s) [noise=true , address=81.171.14.70:4031 , piece_id=7MD5N7PPKFPCFMOIXAGI3MPBDDX45GTLMA3CFGNM7BBGI6TSL6RA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12drdYnnWsuMyFo6LkcewAGGWnorRCAhxPpNqK41LvGy2zkBvng): protocol: expected piece hash; context canceled; EOF Dec 22 14:38:32.926250028 (407h9m32.797333063s) [noise=true , address=81.171.14.70:4031 , piece_id=7MD5N7PPKFPCFMOIXAGI3MPBDDX45GTLMA3CFGNM7BBGI6TSL6RA, seg_pos={0 0}, upload=357453]: Long tail cancellation due to optimal threshold Dec 22 14:38:32.926299612 (407h9m32.797382647s) [seg_pos={0 0}, upload=357453]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 22 14:38:32.926403941 (407h9m32.797486986s) [upload=357453]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 14:38:32.926409692 (407h9m32.797492737s) [upload=357453]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 14:39:04.793379844 (407h10m4.664462889s) [upload=357453]: Done uploading segments... Dec 22 14:39:04.793413608 (407h10m4.664496643s) [upload=357453]: Upload finished. err: <nil> Duration: 6.573664217s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 5.159454652s trace: https://jaeger.dev.storj.io/trace/510cfc1b6c68e873 FAIL (2025-12-22 15:10:15 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.159486113s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...................................................X........] duration: 3.46689989s trace: https://jaeger.dev.storj.io/trace/2e095cd265881ac3 FAIL (2025-12-22 15:09:56 UTC) orbiter warning: context deadline exceeded Duration: 3.466912794s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [...........................X.....XX.....XX.XXXXX.X.X.X.XX.XX] duration: 30.003583241s trace: https://jaeger.dev.storj.io/trace/3265ee8ef95e96c6 FAIL (2025-12-22 15:09:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN, Piece ID: A42WDIQJ5342JCD4CTGUBVX7JP2QFMSSO3U6O5JNHN6AHLBF5AHA) context deadline exceeded; piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: 5O3H2GF6VFEVBAKPIHLPB7WY22VH3EDQBPL7YQLHU4LAFTG7BBBQ) context deadline exceeded ======LOGS====== Dec 22 15:09:21.838939174 (407h40m21.710022219s) [upload=357906]: Starting upload Dec 22 15:09:21.839019637 (407h40m21.710102672s) [upload=357906]: Done starting upload Dec 22 15:09:21.839136389 (407h40m21.710219424s) [upload=357906]: Uploading segments... Dec 22 15:09:21.839294510 (407h40m21.710377545s) [upload=357906]: Got next segment. Inline: false Dec 22 15:09:21.839330519 (407h40m21.710413564s) [upload=357906]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 15:09:21.839340608 (407h40m21.710423673s) [upload=357906]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 15:09:22.089684838 (407h40m21.960767883s) [seg_pos={0 0}, upload=357906]: Begin upload segment... Dec 22 15:09:22.089736596 (407h40m21.960819641s) [seg_pos={0 0}, upload=357906]: Done begin upload segment. Dec 22 15:09:22.089768256 (407h40m21.960851301s) [noise=true , address=145.79.227.14:4838 , piece_id=XC2NMZH7G42IV3JMHZOS4KSCHYQDKGPUDWTFAXMF2AYPRHZ6OE5A, seg_pos={0 0}, upload=357906]: Uploading piece... Dec 22 15:09:22.089829353 (407h40m21.960912398s) [noise=true , address=145.79.227.11:4732 , piece_id=ZMJP72QLGB4JNY5WJ3GAJL3RIQRSLYQ4SSZKB3HUJB26T526GYZQ, seg_pos={0 0}, upload=357906]: Uploading piece... Dec 22 15:09:22.089895218 (407h40m21.960978253s) [noise=true , address=145.79.227.39:4015 , piece_id=DAPJGM5OOEEMX5QWZO4NMGZHSGTTLM7XE73KSI4NKUIJP3QTITEA, seg_pos={0 0}, upload=357906]: Uploading piece... Dec 22 15:09:22.090318665 (407h40m21.96140171s) [noise=true , address=145.79.227.45:4230 , piece_id=MLBZ724R6ZLW5L47XMIWU36EWS2FEHX2X6YL6S2NAN4NSSILJDKA, seg_pos={0 0}, upload=357906]: Uploading piece... Dec 22 15:09:22.090517503 (407h40m21.961600549s) [noise=true , address=77.90.31.10:4519 , piece_id=5O3H2GF6VFEVBAKPIHLPB7WY22VH3EDQBPL7YQLHU4LAFTG7BBBQ, seg_pos={0 0}, upload=357906]: Uploading piece... Dec 22 15:09:22.090731230 (407h40m21.961814275s) [noise=true , address=77.90.31.17:4770 , piece_id=YODZ3OXGCC6Y5DWB4O3B4GT5YR2XYF46KXM3DTNJUZUDCIYBQICQ, seg_pos={0 0}, upload=357906]: Uploading piece... Dec 22 15:09:22.090930470 (407h40m21.962013515s) [noise=true , address=145.79.227.43:4159 , piece_id=JMESWCEA6ECDJKDBCWOTT56NZFGKNGB6UWAVWE5M44DLHSHWNQQA, seg_pos={0 0}, upload=357906]: Uploading piece... Dec 22 15:09:22.091148405 (407h40m21.96223145s) [noise=true , address=145.79.227.48:4301 , piece_id=QFPOY4WER7LUOML4ZL6ULMYIYWCUSNPQZCD2TOUVQQBXBRT4NSJA, seg_pos={0 0}, upload=357906]: Uploading piece... Dec 22 15:09:22.091311997 (407h40m21.962395032s) [noise=true , address=145.79.227.47:4265 , piece_id=NBJXC33D3D2UEW3NU3WRYHTUL5ZQIOXXY3SW7SKS2KPHBVXFK6XQ, seg_pos={0 0}, upload=357906]: Uploading piece... Dec 22 15:09:22.091613239 (407h40m21.962696285s) [noise=true , address=145.79.227.37:4565 , piece_id=ZHQHRJWYMSLYOWEH7R356EGBQ7LZZ3EWRFGVNBCEEUYX5FGG6YRQ, seg_pos={0 0}, upload=357906]: Uploading piece... Dec 22 15:09:22.091698993 (407h40m21.962782028s) [noise=true , address=45.91.116.4:4940 , piece_id=DBCEPFQ2OOYYONSWJ4K6I2Q2EYTTMPRGCQFK3ZRILPMOU7M7RCJQ, seg_pos={0 0}, upload=357906]: Uploading piece... Dec 22 15:09:22.092037617 (407h40m21.963120662s) [noise=true , address=45.91.116.5:4300 , piece_id=RHLGNEGW7224ZF73MLM7RYGFM77QRECILT4BQYD4RGVUWHHPT5QQ, seg_pos={0 0}, upload=357906]: Uploading piece... Dec 22 15:09:22.092126357 (407h40m21.963209392s) [noise=true , address=145.79.227.38:4979 , piece_id=ZD7MRX62QDJUNTS3MVOTGYYIBMEIYSGDFQF5QBISDWF2GESUBCXQ, seg_pos={0 0}, upload=357906]: Uploading piece... Dec 22 15:09:22.092430757 (407h40m21.963513802s) [noise=true , address=45.91.116.6:4976 , piece_id=GFYZEJDSJYETZVI66ECXGBQLFT5L6QS4I2R43P2H2TFKDSJ6PH5A, seg_pos={0 0}, upload=357906]: Uploading piece... Dec 22 15:09:22.092951378 (407h40m21.964034413s) [noise=true , address=145.79.227.49:4338 , piece_id=A42WDIQJ5342JCD4CTGUBVX7JP2QFMSSO3U6O5JNHN6AHLBF5AHA, seg_pos={0 0}, upload=357906]: Uploading piece... Dec 22 15:09:22.093388600 (407h40m21.964471645s) [noise=true , address=145.79.227.30:4762 , piece_id=4GA25BN5GABFBYXKUKFBHOXPETXVWU7WUJ2N2UWGNEJVGOIXVCOA, seg_pos={0 0}, upload=357906]: Uploading piece... Dec 22 15:09:24.024930087 (407h40m23.896013122s) [upload=357906]: Next returned nil segment Dec 22 15:09:24.024948993 (407h40m23.896032028s) [upload=357906]: Waiting for error group managing segments... Dec 22 15:09:24.344748107 (407h40m24.215831142s) [noise=true , address=77.90.31.17:4770 , piece_id=YODZ3OXGCC6Y5DWB4O3B4GT5YR2XYF46KXM3DTNJUZUDCIYBQICQ, seg_pos={0 0}, upload=357906]: Done uploading piece. err: <nil> Dec 22 15:09:24.360909511 (407h40m24.231992546s) [noise=true , address=45.91.116.5:4300 , piece_id=RHLGNEGW7224ZF73MLM7RYGFM77QRECILT4BQYD4RGVUWHHPT5QQ, seg_pos={0 0}, upload=357906]: Done uploading piece. err: <nil> Dec 22 15:09:24.459274315 (407h40m24.33035736s) [noise=true , address=77.90.31.10:4519 , piece_id=5O3H2GF6VFEVBAKPIHLPB7WY22VH3EDQBPL7YQLHU4LAFTG7BBBQ, seg_pos={0 0}, upload=357906]: Done uploading piece. err: <nil> Dec 22 15:09:24.489593938 (407h40m24.360676974s) [noise=true , address=45.91.116.6:4976 , piece_id=GFYZEJDSJYETZVI66ECXGBQLFT5L6QS4I2R43P2H2TFKDSJ6PH5A, seg_pos={0 0}, upload=357906]: Done uploading piece. err: <nil> Dec 22 15:09:25.044856997 (407h40m24.915940032s) [noise=true , address=145.79.227.37:4565 , piece_id=ZHQHRJWYMSLYOWEH7R356EGBQ7LZZ3EWRFGVNBCEEUYX5FGG6YRQ, seg_pos={0 0}, upload=357906]: Done uploading piece. err: <nil> Dec 22 15:09:25.045891217 (407h40m24.916974262s) [noise=true , address=145.79.227.43:4159 , piece_id=JMESWCEA6ECDJKDBCWOTT56NZFGKNGB6UWAVWE5M44DLHSHWNQQA, seg_pos={0 0}, upload=357906]: Done uploading piece. err: <nil> Dec 22 15:09:25.069597367 (407h40m24.940680403s) [noise=true , address=145.79.227.14:4838 , piece_id=XC2NMZH7G42IV3JMHZOS4KSCHYQDKGPUDWTFAXMF2AYPRHZ6OE5A, seg_pos={0 0}, upload=357906]: Done uploading piece. err: <nil> Dec 22 15:09:25.077829733 (407h40m24.948912778s) [noise=true , address=145.79.227.45:4230 , piece_id=MLBZ724R6ZLW5L47XMIWU36EWS2FEHX2X6YL6S2NAN4NSSILJDKA, seg_pos={0 0}, upload=357906]: Done uploading piece. err: <nil> Dec 22 15:09:25.103859138 (407h40m24.974942183s) [noise=true , address=45.91.116.4:4940 , piece_id=DBCEPFQ2OOYYONSWJ4K6I2Q2EYTTMPRGCQFK3ZRILPMOU7M7RCJQ, seg_pos={0 0}, upload=357906]: Done uploading piece. err: <nil> Dec 22 15:09:25.105223886 (407h40m24.976306931s) [noise=true , address=145.79.227.49:4338 , piece_id=A42WDIQJ5342JCD4CTGUBVX7JP2QFMSSO3U6O5JNHN6AHLBF5AHA, seg_pos={0 0}, upload=357906]: Done uploading piece. err: <nil> Dec 22 15:09:25.105887830 (407h40m24.976970865s) [noise=true , address=145.79.227.11:4732 , piece_id=ZMJP72QLGB4JNY5WJ3GAJL3RIQRSLYQ4SSZKB3HUJB26T526GYZQ, seg_pos={0 0}, upload=357906]: Done uploading piece. err: <nil> Dec 22 15:09:25.923336919 (407h40m25.794419954s) [noise=true , address=145.79.227.39:4015 , piece_id=DAPJGM5OOEEMX5QWZO4NMGZHSGTTLM7XE73KSI4NKUIJP3QTITEA, seg_pos={0 0}, upload=357906]: Done uploading piece. err: <nil> Dec 22 15:09:26.448805232 (407h40m26.319888267s) [noise=true , address=145.79.227.30:4762 , piece_id=4GA25BN5GABFBYXKUKFBHOXPETXVWU7WUJ2N2UWGNEJVGOIXVCOA, seg_pos={0 0}, upload=357906]: Done uploading piece. err: <nil> Dec 22 15:09:26.448824459 (407h40m26.319907494s) [seg_pos={0 0}, upload=357906]: Segment reached optimal threshold of 13 pieces. Dec 22 15:09:26.449017176 (407h40m26.320100211s) [noise=true , address=145.79.227.38:4979 , piece_id=ZD7MRX62QDJUNTS3MVOTGYYIBMEIYSGDFQF5QBISDWF2GESUBCXQ, seg_pos={0 0}, upload=357906]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma): protocol: expected piece hash; context canceled; EOF Dec 22 15:09:26.449023829 (407h40m26.320106864s) [noise=true , address=145.79.227.38:4979 , piece_id=ZD7MRX62QDJUNTS3MVOTGYYIBMEIYSGDFQF5QBISDWF2GESUBCXQ, seg_pos={0 0}, upload=357906]: Long tail cancellation due to optimal threshold Dec 22 15:09:26.449224140 (407h40m26.320307175s) [noise=true , address=145.79.227.47:4265 , piece_id=NBJXC33D3D2UEW3NU3WRYHTUL5ZQIOXXY3SW7SKS2KPHBVXFK6XQ, seg_pos={0 0}, upload=357906]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JvnJetuaEkstkPVNChKo8YQEYC68qRXPGQxx2AKZoHftkhajQ): protocol: expected piece hash; context canceled; EOF Dec 22 15:09:26.449230162 (407h40m26.320313197s) [noise=true , address=145.79.227.47:4265 , piece_id=NBJXC33D3D2UEW3NU3WRYHTUL5ZQIOXXY3SW7SKS2KPHBVXFK6XQ, seg_pos={0 0}, upload=357906]: Long tail cancellation due to optimal threshold Dec 22 15:09:26.449269767 (407h40m26.320352802s) [noise=true , address=145.79.227.48:4301 , piece_id=QFPOY4WER7LUOML4ZL6ULMYIYWCUSNPQZCD2TOUVQQBXBRT4NSJA, seg_pos={0 0}, upload=357906]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18HkaSrdHHnvmeNz1ormfcSBZH8bMUgkX7X3JmGuqkmtBoak29): protocol: expected piece hash; context canceled; EOF Dec 22 15:09:26.449292270 (407h40m26.320375315s) [noise=true , address=145.79.227.48:4301 , piece_id=QFPOY4WER7LUOML4ZL6ULMYIYWCUSNPQZCD2TOUVQQBXBRT4NSJA, seg_pos={0 0}, upload=357906]: Long tail cancellation due to optimal threshold Dec 22 15:09:26.449322487 (407h40m26.320405532s) [seg_pos={0 0}, upload=357906]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 15:09:26.449421045 (407h40m26.32050409s) [upload=357906]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 15:09:26.449437968 (407h40m26.320521003s) [upload=357906]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 15:09:26.760974789 (407h40m26.632057834s) [upload=357906]: Done uploading segments... Dec 22 15:09:26.760996400 (407h40m26.632079435s) [upload=357906]: Upload finished. err: <nil> Duration: 30.003623798s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 1.387814453s trace: https://jaeger.dev.storj.io/trace/62b55ec615ab6123 FAIL (2025-12-22 15:09:37 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.387842618s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 6.361778801s trace: https://jaeger.dev.storj.io/trace/510cfc1b6c68e873 FAIL (2025-12-22 15:09:36 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 6.361806583s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 4.79954017s trace: https://jaeger.dev.storj.io/trace/718f722b1bda7fe2 FAIL (2025-12-22 15:09:32 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== VMAC4CKPNLATBLZLKAGDGZHVT7BHEPN3Q, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.367093115 (407h9m14.23817616s) [noise=true , address=185.167.204.206:4043 , piece_id=HL32YEZF67X2UT6SRMXJVJ4W5A3PC5SXOCM7BYQVMEC76TKEVJRA, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.367396732 (407h9m14.238479767s) [noise=true , address=185.167.204.205:4032 , piece_id=FJMXJG2PQ4WZDSDCPJ6LLSVQKI7JJHUCDUZLU2BJZV7RZTLETM5A, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.367549273 (407h9m14.238632308s) [noise=true , address=185.167.204.206:4070 , piece_id=Y7IRY4E2GCF7TCFOIQJ7QH6IFEJBMW4WM6QBIS5SCUJ7E7ASV4HA, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.367749263 (407h9m14.238832308s) [noise=true , address=81.171.14.70:4046 , piece_id=NRMV6FG5CZSTLUNP2EHNCK6VM2Z2UHKBLCHLKEPQ225Q5DGYEBPA, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.368034046 (407h9m14.239117091s) [noise=true , address=185.167.204.206:4060 , piece_id=WN6T5TUBVMURPSBBXGXCTLPZZ74PFNSRTTWXI2JPUTY74DPH3YSQ, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.368552724 (407h9m14.239635759s) [noise=true , address=81.171.14.70:4029 , piece_id=APDMOYEMDCLOREXY2BBOK73RSQVWKIXUX4SCHRVFDSM7ZLWBEQXA, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.369160691 (407h9m14.240243736s) [noise=true , address=185.167.204.206:4057 , piece_id=ONW7YS5JG44B2STNFJGJWFEZ3WYXO74N3CXI3C6T5OTJYHGQPQZQ, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.369714496 (407h9m14.240797541s) [noise=true , address=185.167.204.206:4071 , piece_id=DVJAKIMPAUGWQPHDPGTXASQDONAKOZIGX5PQDBGBXNQAPMBPUFBA, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.370249614 (407h9m14.241332649s) [noise=true , address=89.149.201.147:4019 , piece_id=ILE7ZX5X6HBPUVYGCOQ5SNG3AA6B3YTXIF47CPLJ3MN5EXVOS2OA, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.372019455 (407h9m14.24310249s) [noise=true , address=89.149.201.147:4015 , piece_id=ZIWRTD7HSP7YMRJET2TTJQFAQZM7KGDKJDLMYCWWY2W6SV2UVQBQ, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:15.199398795 (407h9m15.07048184s) [upload=357453]: Next returned nil segment Dec 22 14:38:15.199414966 (407h9m15.070498011s) [upload=357453]: Waiting for error group managing segments... Dec 22 14:38:15.591187572 (407h9m15.462270607s) [noise=true , address=185.167.204.205:4009 , piece_id=JTHX2O6T5HRENGEMAFJUWQUEUYAIG2KQ5HDEHF4BK42IT2ER7EIA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.591766795 (407h9m15.46284984s) [noise=true , address=185.167.204.205:4006 , piece_id=XHNFS6G7EX4BQWKOK4JUWYV3OTVJ36A6VOP4N7G2XPZKNNCFFKXA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.592074521 (407h9m15.463157566s) [noise=true , address=185.167.204.206:4060 , piece_id=WN6T5TUBVMURPSBBXGXCTLPZZ74PFNSRTTWXI2JPUTY74DPH3YSQ, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.593304694 (407h9m15.464387729s) [noise=true , address=185.167.204.205:4036 , piece_id=5QWJTTL6TZIZSSYHCBEJBKGX7BS76VVTELWTKEAKTOVYKINXVWDQ, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.593638659 (407h9m15.464721694s) [noise=true , address=185.167.204.205:4004 , piece_id=PYGDQZMAKOVJWB7AWEPFMPMB6C24FMS4HT2UP2AKZNYGG4DGZR3Q, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.593818251 (407h9m15.464901286s) [noise=true , address=185.167.204.206:4048 , piece_id=52WC3XH2K2BHG33BQPRENPNYDT7EOKCNBCTZKE3LIONDJHVATEYA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.594252549 (407h9m15.465335594s) [noise=true , address=185.167.204.205:4034 , piece_id=OS5H57TG2REFE535DK4T6N75VNIXPLQWW6DM55QCC2S7QBVZM3UQ, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.597113216 (407h9m15.468196261s) [noise=true , address=185.167.204.205:4032 , piece_id=FJMXJG2PQ4WZDSDCPJ6LLSVQKI7JJHUCDUZLU2BJZV7RZTLETM5A, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.597394541 (407h9m15.468477586s) [noise=true , address=185.167.204.205:4035 , piece_id=3D7T4SGPAJQZCA74CUTR4GJRE23R4JD5KBNW3BLEASSP4GSR5PHQ, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.597612756 (407h9m15.468695792s) [noise=true , address=185.167.204.206:4043 , piece_id=HL32YEZF67X2UT6SRMXJVJ4W5A3PC5SXOCM7BYQVMEC76TKEVJRA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.598018900 (407h9m15.469101945s) [noise=true , address=185.167.204.205:4012 , piece_id=A5G37BVL2GZ7LAY3FZWIZV4R4M7M6UPYDYACJFRD5HKDWWKB4Q6A, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.598254719 (407h9m15.469337765s) [noise=true , address=185.167.204.206:4052 , piece_id=V2DVAAXN5A4C3NQW556VMAC4CKPNLATBLZLKAGDGZHVT7BHEPN3Q, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.598480649 (407h9m15.469563694s) [noise=true , address=185.167.204.205:4018 , piece_id=NUYRZ7PXKJQ3PIR44O3E6N6COZZPRDNSMTQCFI5RN7FBRV2CBALA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.598968278 (407h9m15.470051323s) [noise=true , address=185.167.204.205:4033 , piece_id=A32ZTBDBWMCNBMZPD4DUWS5NKUKQ22ACQH6HAW5AKB3FNHBC56VA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.599287886 (407h9m15.470370931s) [noise=true , address=185.167.204.206:4072 , piece_id=I6PCSC5JYTJXLT6B34TTIVZQVX52R7TRHSRLRP4ZHG5J63XDZN6Q, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.599632252 (407h9m15.470715297s) [noise=true , address=185.167.204.206:4070 , piece_id=Y7IRY4E2GCF7TCFOIQJ7QH6IFEJBMW4WM6QBIS5SCUJ7E7ASV4HA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.599676297 (407h9m15.470759332s) [noise=true , address=185.167.204.206:4071 , piece_id=DVJAKIMPAUGWQPHDPGTXASQDONAKOZIGX5PQDBGBXNQAPMBPUFBA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.601435877 (407h9m15.472518912s) [noise=true , address=185.167.204.206:4057 , piece_id=ONW7YS5JG44B2STNFJGJWFEZ3WYXO74N3CXI3C6T5OTJYHGQPQZQ, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:17.299689191 (407h9m17.170772226s) [noise=true , address=89.149.201.147:4019 , piece_id=ILE7ZX5X6HBPUVYGCOQ5SNG3AA6B3YTXIF47CPLJ3MN5EXVOS2OA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:17.526601953 (407h9m17.397684998s) [noise=true , address=89.149.201.147:4015 , piece_id=ZIWRTD7HSP7YMRJET2TTJQFAQZM7KGDKJDLMYCWWY2W6SV2UVQBQ, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:18.294911214 (407h9m18.165994249s) [noise=true , address=89.149.201.147:4017 , piece_id=ZOSNSMZOBQVIKD2ZGJRAUWJEJ4JXJK452YWQQSDL77E77F6LRDZA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:18.402191355 (407h9m18.2732744s) [noise=true , address=89.149.201.147:4011 , piece_id=ATMM77IGBDSRHNJ5FQ2VKP7W27JUA2F6HYJYSQ62IL5JQN5PF5OA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:22.436508909 (407h9m22.307591944s) [noise=true , address=81.171.14.70:4037 , piece_id=NYGC2ZLUSBUAGFULQ46LXCVNQUJ5M4XT3YICVA2ZMBR2D6WPJ27Q, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:25.935108368 (407h9m25.806191413s) [noise=true , address=81.171.14.70:4046 , piece_id=NRMV6FG5CZSTLUNP2EHNCK6VM2Z2UHKBLCHLKEPQ225Q5DGYEBPA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:30.904793996 (407h9m30.775877041s) [noise=true , address=81.171.14.70:4029 , piece_id=APDMOYEMDCLOREXY2BBOK73RSQVWKIXUX4SCHRVFDSM7ZLWBEQXA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:32.926010512 (407h9m32.797093567s) [noise=true , address=81.171.14.70:4044 , piece_id=QCWQO6JQRRKSG5U2KVH3Y6I3UI2F7G3DOW4CH3PQZKUVQ6Q2XOHA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:32.926036552 (407h9m32.797119587s) [seg_pos={0 0}, upload=357453]: Segment reached optimal threshold of 26 pieces. Dec 22 14:38:32.926196857 (407h9m32.797279902s) [noise=true , address=81.171.14.70:4033 , piece_id=MKNAJOFD5CIHTA55XO476LOJJGIWSA7HBQFQND2YNM562W74IKJA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BnjusiXEfHYwTDL8soMkm8GYnhbkwy2oCAxVA7oiDyhFXozZ8): protocol: expected piece hash; context canceled; EOF Dec 22 14:38:32.926203058 (407h9m32.797286093s) [noise=true , address=81.171.14.70:4033 , piece_id=MKNAJOFD5CIHTA55XO476LOJJGIWSA7HBQFQND2YNM562W74IKJA, seg_pos={0 0}, upload=357453]: Long tail cancellation due to optimal threshold Dec 22 14:38:32.926237574 (407h9m32.797320609s) [noise=true , address=81.171.14.70:4031 , piece_id=7MD5N7PPKFPCFMOIXAGI3MPBDDX45GTLMA3CFGNM7BBGI6TSL6RA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12drdYnnWsuMyFo6LkcewAGGWnorRCAhxPpNqK41LvGy2zkBvng): protocol: expected piece hash; context canceled; EOF Dec 22 14:38:32.926250028 (407h9m32.797333063s) [noise=true , address=81.171.14.70:4031 , piece_id=7MD5N7PPKFPCFMOIXAGI3MPBDDX45GTLMA3CFGNM7BBGI6TSL6RA, seg_pos={0 0}, upload=357453]: Long tail cancellation due to optimal threshold Dec 22 14:38:32.926299612 (407h9m32.797382647s) [seg_pos={0 0}, upload=357453]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 22 14:38:32.926403941 (407h9m32.797486986s) [upload=357453]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 14:38:32.926409692 (407h9m32.797492737s) [upload=357453]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 14:39:04.793379844 (407h10m4.664462889s) [upload=357453]: Done uploading segments... Dec 22 14:39:04.793413608 (407h10m4.664496643s) [upload=357453]: Upload finished. err: <nil> Duration: 4.799569707s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 1.420106262s trace: https://jaeger.dev.storj.io/trace/62b55ec615ab6123 FAIL (2025-12-22 15:09:18 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.420143593s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 5.465150382s trace: https://jaeger.dev.storj.io/trace/510cfc1b6c68e873 FAIL (2025-12-22 15:09:03 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.465176912s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [............................X.....XX.....XX.XXXXX.X.X.X.XX.X] duration: 30.003731816s trace: https://jaeger.dev.storj.io/trace/43e0b77865ae289e FAIL (2025-12-22 15:08:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 22 15:08:30.033593638 (407h39m29.904676673s) [upload=357894]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 15:08:30.033599780 (407h39m29.904682815s) [upload=357894]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 15:08:30.277075089 (407h39m30.148158124s) [seg_pos={0 0}, upload=357894]: Begin upload segment... Dec 22 15:08:30.277152126 (407h39m30.148235171s) [seg_pos={0 0}, upload=357894]: Done begin upload segment. Dec 22 15:08:30.277211910 (407h39m30.148294955s) [noise=true , address=145.79.227.45:4231 , piece_id=DUWBAMUJ2N3ZJAACBAMAIOOTP737CKIMXWWFNEFFHZD4XTCRNUJQ, seg_pos={0 0}, upload=357894]: Uploading piece... Dec 22 15:08:30.277234563 (407h39m30.148317599s) [noise=true , address=45.91.116.5:4297 , piece_id=W4QHX7QNJGON6DYNR27DS6RGC5ITGVR5SIXOOTDF7KT7EPU3F6XA, seg_pos={0 0}, upload=357894]: Uploading piece... Dec 22 15:08:30.277864793 (407h39m30.148947828s) [noise=true , address=45.91.116.6:4973 , piece_id=3GCOONJZ7IAMJF64FLT4365BT6KU5YVZNHD5QZLEZNNTOHAFKYJQ, seg_pos={0 0}, upload=357894]: Uploading piece... Dec 22 15:08:30.277877127 (407h39m30.148960172s) [noise=true , address=145.79.227.35:4483 , piece_id=5O7XNHW2HGZOP3ZP6IZF2K6P2MIFO2UXV4JUQR6YLWTD6B5HY7LQ, seg_pos={0 0}, upload=357894]: Uploading piece... Dec 22 15:08:30.278161668 (407h39m30.149244703s) [noise=true , address=145.79.227.43:4159 , piece_id=AEY55YCEKHZWPIBXF3FZPQ6UT5GINU7JSYOSJSRLSP6AXVGPAZYA, seg_pos={0 0}, upload=357894]: Uploading piece... Dec 22 15:08:30.278375084 (407h39m30.14945812s) [noise=true , address=77.90.31.19:4842 , piece_id=TGWQ4O2UTO7TKO4TRINMKSV5XRU25O2ZUV3U2JVF7KMMCYB2IEAQ, seg_pos={0 0}, upload=357894]: Uploading piece... Dec 22 15:08:30.278433907 (407h39m30.149516942s) [noise=true , address=145.79.227.51:4410 , piece_id=OEVV7JC3OUQFLINX2O4XD3SWYKMETRVG3Q5OKGZZXOI2OAOAWX5A, seg_pos={0 0}, upload=357894]: Uploading piece... Dec 22 15:08:30.278718929 (407h39m30.149801964s) [noise=true , address=145.79.227.16:4909 , piece_id=NFT27DSVMNOSGLL3WS6HJNOA6WD4OQVQFGAPUPQIKVLZFQG4D6EA, seg_pos={0 0}, upload=357894]: Uploading piece... Dec 22 15:08:30.278739128 (407h39m30.149822163s) [noise=true , address=145.79.227.14:4838 , piece_id=IUVFHQ5ENUNQZSOBYBUZS6ABOASRD4E5QYNSPCJMU7R5M2VQQCUA, seg_pos={0 0}, upload=357894]: Uploading piece... Dec 22 15:08:30.278971892 (407h39m30.150054927s) [noise=true , address=145.79.227.39:4015 , piece_id=O3N3UHBWR26SGF3LLTMZIKEEGYNSS645MDUOXERE6EMDJUMZDOSQ, seg_pos={0 0}, upload=357894]: Uploading piece... Dec 22 15:08:30.279137787 (407h39m30.150220832s) [noise=true , address=45.91.116.7:4012 , piece_id=OGCUHIFU4GYGA4OSKRFELFCKQ2OMMUARBDQNPMOK7HDO7SSVSU3A, seg_pos={0 0}, upload=357894]: Uploading piece... Dec 22 15:08:30.279238829 (407h39m30.150321864s) [noise=true , address=45.91.116.11:4117 , piece_id=2HYA7XL6U64V4JHUSTOIMTOIAL2ZFPR7JWSXLZKBHY5HOYYYLG4A, seg_pos={0 0}, upload=357894]: Uploading piece... Dec 22 15:08:30.279518352 (407h39m30.150601417s) [noise=true , address=145.79.227.28:4725 , piece_id=AW6B2OX62O263LTUYJIEUZ6SNRMFMQ5Y6B6THJALITTU7X6WVCKA, seg_pos={0 0}, upload=357894]: Uploading piece... Dec 22 15:08:30.279872947 (407h39m30.150955982s) [noise=true , address=145.79.227.47:4265 , piece_id=PAX7D7TR7XKB2LS6WXQ2LP5EHGPBWWLOR23GOMFDQWBBA6XY7FEQ, seg_pos={0 0}, upload=357894]: Uploading piece... Dec 22 15:08:30.280164633 (407h39m30.151247668s) [noise=true , address=45.91.116.13:4153 , piece_id=Y4UW5ZBQ623DBUSKLVA5BQVJMIDCIWTFNFZPFL4A4UUR4QMGAOMA, seg_pos={0 0}, upload=357894]: Uploading piece... Dec 22 15:08:30.280838165 (407h39m30.1519212s) [noise=true , address=145.79.227.36:4517 , piece_id=T5EJ5S5Y7LVR5BP46WEMRHUJBYMAX3PNXLIO2TWRNE65L2APSYTA, seg_pos={0 0}, upload=357894]: Uploading piece... Dec 22 15:08:32.189169472 (407h39m32.060252527s) [upload=357894]: Next returned nil segment Dec 22 15:08:32.189188178 (407h39m32.060271223s) [upload=357894]: Waiting for error group managing segments... Dec 22 15:08:32.547877155 (407h39m32.41896019s) [noise=true , address=45.91.116.5:4297 , piece_id=W4QHX7QNJGON6DYNR27DS6RGC5ITGVR5SIXOOTDF7KT7EPU3F6XA, seg_pos={0 0}, upload=357894]: Done uploading piece. err: <nil> Dec 22 15:08:32.554294485 (407h39m32.42537753s) [noise=true , address=45.91.116.6:4973 , piece_id=3GCOONJZ7IAMJF64FLT4365BT6KU5YVZNHD5QZLEZNNTOHAFKYJQ, seg_pos={0 0}, upload=357894]: Done uploading piece. err: <nil> Dec 22 15:08:32.570390795 (407h39m32.44147383s) [noise=true , address=77.90.31.19:4842 , piece_id=TGWQ4O2UTO7TKO4TRINMKSV5XRU25O2ZUV3U2JVF7KMMCYB2IEAQ, seg_pos={0 0}, upload=357894]: Done uploading piece. err: <nil> Dec 22 15:08:32.748047384 (407h39m32.619130429s) [noise=true , address=45.91.116.11:4117 , piece_id=2HYA7XL6U64V4JHUSTOIMTOIAL2ZFPR7JWSXLZKBHY5HOYYYLG4A, seg_pos={0 0}, upload=357894]: Done uploading piece. err: <nil> Dec 22 15:08:32.908211806 (407h39m32.779294851s) [noise=true , address=45.91.116.13:4153 , piece_id=Y4UW5ZBQ623DBUSKLVA5BQVJMIDCIWTFNFZPFL4A4UUR4QMGAOMA, seg_pos={0 0}, upload=357894]: Done uploading piece. err: <nil> Dec 22 15:08:33.250849870 (407h39m33.121932915s) [noise=true , address=145.79.227.47:4265 , piece_id=PAX7D7TR7XKB2LS6WXQ2LP5EHGPBWWLOR23GOMFDQWBBA6XY7FEQ, seg_pos={0 0}, upload=357894]: Done uploading piece. err: <nil> Dec 22 15:08:33.253453757 (407h39m33.124536792s) [noise=true , address=145.79.227.45:4231 , piece_id=DUWBAMUJ2N3ZJAACBAMAIOOTP737CKIMXWWFNEFFHZD4XTCRNUJQ, seg_pos={0 0}, upload=357894]: Done uploading piece. err: <nil> Dec 22 15:08:33.253770712 (407h39m33.124853767s) [noise=true , address=145.79.227.35:4483 , piece_id=5O7XNHW2HGZOP3ZP6IZF2K6P2MIFO2UXV4JUQR6YLWTD6B5HY7LQ, seg_pos={0 0}, upload=357894]: Done uploading piece. err: <nil> Dec 22 15:08:33.254714198 (407h39m33.125797243s) [noise=true , address=145.79.227.36:4517 , piece_id=T5EJ5S5Y7LVR5BP46WEMRHUJBYMAX3PNXLIO2TWRNE65L2APSYTA, seg_pos={0 0}, upload=357894]: Done uploading piece. err: <nil> Dec 22 15:08:33.278815972 (407h39m33.149899017s) [noise=true , address=145.79.227.43:4159 , piece_id=AEY55YCEKHZWPIBXF3FZPQ6UT5GINU7JSYOSJSRLSP6AXVGPAZYA, seg_pos={0 0}, upload=357894]: Done uploading piece. err: <nil> Dec 22 15:08:33.281534458 (407h39m33.152617493s) [noise=true , address=145.79.227.39:4015 , piece_id=O3N3UHBWR26SGF3LLTMZIKEEGYNSS645MDUOXERE6EMDJUMZDOSQ, seg_pos={0 0}, upload=357894]: Done uploading piece. err: <nil> Dec 22 15:08:33.319026399 (407h39m33.190109444s) [noise=true , address=145.79.227.51:4410 , piece_id=OEVV7JC3OUQFLINX2O4XD3SWYKMETRVG3Q5OKGZZXOI2OAOAWX5A, seg_pos={0 0}, upload=357894]: Done uploading piece. err: <nil> Dec 22 15:08:33.689277273 (407h39m33.560360308s) [noise=true , address=45.91.116.7:4012 , piece_id=OGCUHIFU4GYGA4OSKRFELFCKQ2OMMUARBDQNPMOK7HDO7SSVSU3A, seg_pos={0 0}, upload=357894]: Done uploading piece. err: <nil> Dec 22 15:08:33.689297281 (407h39m33.560380326s) [seg_pos={0 0}, upload=357894]: Segment reached optimal threshold of 13 pieces. Dec 22 15:08:33.689462625 (407h39m33.560545671s) [noise=true , address=145.79.227.28:4725 , piece_id=AW6B2OX62O263LTUYJIEUZ6SNRMFMQ5Y6B6THJALITTU7X6WVCKA, seg_pos={0 0}, upload=357894]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1): protocol: write tcp 188.166.225.170:48766->145.79.227.28:4725: use of closed network connection; write tcp 188.166.225.170:48766->145.79.227.28:4725: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48766->145.79.227.28:4725: use of closed network connection Dec 22 15:08:33.689468827 (407h39m33.560551862s) [noise=true , address=145.79.227.28:4725 , piece_id=AW6B2OX62O263LTUYJIEUZ6SNRMFMQ5Y6B6THJALITTU7X6WVCKA, seg_pos={0 0}, upload=357894]: Long tail cancellation due to optimal threshold Dec 22 15:08:33.689747187 (407h39m33.560830232s) [noise=true , address=145.79.227.14:4838 , piece_id=IUVFHQ5ENUNQZSOBYBUZS6ABOASRD4E5QYNSPCJMU7R5M2VQQCUA, seg_pos={0 0}, upload=357894]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: write tcp 188.166.225.170:40774->145.79.227.14:4838: use of closed network connection; write tcp 188.166.225.170:40774->145.79.227.14:4838: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40774->145.79.227.14:4838: use of closed network connection Dec 22 15:08:33.689754291 (407h39m33.560837327s) [noise=true , address=145.79.227.14:4838 , piece_id=IUVFHQ5ENUNQZSOBYBUZS6ABOASRD4E5QYNSPCJMU7R5M2VQQCUA, seg_pos={0 0}, upload=357894]: Long tail cancellation due to optimal threshold Dec 22 15:08:33.689990160 (407h39m33.561073205s) [noise=true , address=145.79.227.16:4909 , piece_id=NFT27DSVMNOSGLL3WS6HJNOA6WD4OQVQFGAPUPQIKVLZFQG4D6EA, seg_pos={0 0}, upload=357894]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x): protocol: read tcp 188.166.225.170:42196->145.79.227.16:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 22 15:08:33.689999639 (407h39m33.561082684s) [noise=true , address=145.79.227.16:4909 , piece_id=NFT27DSVMNOSGLL3WS6HJNOA6WD4OQVQFGAPUPQIKVLZFQG4D6EA, seg_pos={0 0}, upload=357894]: Long tail cancellation due to optimal threshold Dec 22 15:08:33.690022071 (407h39m33.561105116s) [seg_pos={0 0}, upload=357894]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 15:08:33.690102474 (407h39m33.56118551s) [upload=357894]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 15:08:33.690107875 (407h39m33.56119092s) [upload=357894]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 15:08:33.980891457 (407h39m33.851974493s) [upload=357894]: Done uploading segments... Dec 22 15:08:33.980909673 (407h39m33.851992708s) [upload=357894]: Upload finished. err: <nil> Duration: 30.003755762s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....................................................X.......] duration: 4.021505777s trace: https://jaeger.dev.storj.io/trace/2e095cd265881ac3 FAIL (2025-12-22 15:08:43 UTC) orbiter warning: context deadline exceeded Duration: 4.021529783s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 7.1438959s trace: https://jaeger.dev.storj.io/trace/718f722b1bda7fe2 FAIL (2025-12-22 15:08:26 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== VMAC4CKPNLATBLZLKAGDGZHVT7BHEPN3Q, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.367093115 (407h9m14.23817616s) [noise=true , address=185.167.204.206:4043 , piece_id=HL32YEZF67X2UT6SRMXJVJ4W5A3PC5SXOCM7BYQVMEC76TKEVJRA, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.367396732 (407h9m14.238479767s) [noise=true , address=185.167.204.205:4032 , piece_id=FJMXJG2PQ4WZDSDCPJ6LLSVQKI7JJHUCDUZLU2BJZV7RZTLETM5A, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.367549273 (407h9m14.238632308s) [noise=true , address=185.167.204.206:4070 , piece_id=Y7IRY4E2GCF7TCFOIQJ7QH6IFEJBMW4WM6QBIS5SCUJ7E7ASV4HA, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.367749263 (407h9m14.238832308s) [noise=true , address=81.171.14.70:4046 , piece_id=NRMV6FG5CZSTLUNP2EHNCK6VM2Z2UHKBLCHLKEPQ225Q5DGYEBPA, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.368034046 (407h9m14.239117091s) [noise=true , address=185.167.204.206:4060 , piece_id=WN6T5TUBVMURPSBBXGXCTLPZZ74PFNSRTTWXI2JPUTY74DPH3YSQ, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.368552724 (407h9m14.239635759s) [noise=true , address=81.171.14.70:4029 , piece_id=APDMOYEMDCLOREXY2BBOK73RSQVWKIXUX4SCHRVFDSM7ZLWBEQXA, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.369160691 (407h9m14.240243736s) [noise=true , address=185.167.204.206:4057 , piece_id=ONW7YS5JG44B2STNFJGJWFEZ3WYXO74N3CXI3C6T5OTJYHGQPQZQ, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.369714496 (407h9m14.240797541s) [noise=true , address=185.167.204.206:4071 , piece_id=DVJAKIMPAUGWQPHDPGTXASQDONAKOZIGX5PQDBGBXNQAPMBPUFBA, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.370249614 (407h9m14.241332649s) [noise=true , address=89.149.201.147:4019 , piece_id=ILE7ZX5X6HBPUVYGCOQ5SNG3AA6B3YTXIF47CPLJ3MN5EXVOS2OA, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.372019455 (407h9m14.24310249s) [noise=true , address=89.149.201.147:4015 , piece_id=ZIWRTD7HSP7YMRJET2TTJQFAQZM7KGDKJDLMYCWWY2W6SV2UVQBQ, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:15.199398795 (407h9m15.07048184s) [upload=357453]: Next returned nil segment Dec 22 14:38:15.199414966 (407h9m15.070498011s) [upload=357453]: Waiting for error group managing segments... Dec 22 14:38:15.591187572 (407h9m15.462270607s) [noise=true , address=185.167.204.205:4009 , piece_id=JTHX2O6T5HRENGEMAFJUWQUEUYAIG2KQ5HDEHF4BK42IT2ER7EIA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.591766795 (407h9m15.46284984s) [noise=true , address=185.167.204.205:4006 , piece_id=XHNFS6G7EX4BQWKOK4JUWYV3OTVJ36A6VOP4N7G2XPZKNNCFFKXA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.592074521 (407h9m15.463157566s) [noise=true , address=185.167.204.206:4060 , piece_id=WN6T5TUBVMURPSBBXGXCTLPZZ74PFNSRTTWXI2JPUTY74DPH3YSQ, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.593304694 (407h9m15.464387729s) [noise=true , address=185.167.204.205:4036 , piece_id=5QWJTTL6TZIZSSYHCBEJBKGX7BS76VVTELWTKEAKTOVYKINXVWDQ, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.593638659 (407h9m15.464721694s) [noise=true , address=185.167.204.205:4004 , piece_id=PYGDQZMAKOVJWB7AWEPFMPMB6C24FMS4HT2UP2AKZNYGG4DGZR3Q, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.593818251 (407h9m15.464901286s) [noise=true , address=185.167.204.206:4048 , piece_id=52WC3XH2K2BHG33BQPRENPNYDT7EOKCNBCTZKE3LIONDJHVATEYA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.594252549 (407h9m15.465335594s) [noise=true , address=185.167.204.205:4034 , piece_id=OS5H57TG2REFE535DK4T6N75VNIXPLQWW6DM55QCC2S7QBVZM3UQ, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.597113216 (407h9m15.468196261s) [noise=true , address=185.167.204.205:4032 , piece_id=FJMXJG2PQ4WZDSDCPJ6LLSVQKI7JJHUCDUZLU2BJZV7RZTLETM5A, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.597394541 (407h9m15.468477586s) [noise=true , address=185.167.204.205:4035 , piece_id=3D7T4SGPAJQZCA74CUTR4GJRE23R4JD5KBNW3BLEASSP4GSR5PHQ, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.597612756 (407h9m15.468695792s) [noise=true , address=185.167.204.206:4043 , piece_id=HL32YEZF67X2UT6SRMXJVJ4W5A3PC5SXOCM7BYQVMEC76TKEVJRA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.598018900 (407h9m15.469101945s) [noise=true , address=185.167.204.205:4012 , piece_id=A5G37BVL2GZ7LAY3FZWIZV4R4M7M6UPYDYACJFRD5HKDWWKB4Q6A, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.598254719 (407h9m15.469337765s) [noise=true , address=185.167.204.206:4052 , piece_id=V2DVAAXN5A4C3NQW556VMAC4CKPNLATBLZLKAGDGZHVT7BHEPN3Q, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.598480649 (407h9m15.469563694s) [noise=true , address=185.167.204.205:4018 , piece_id=NUYRZ7PXKJQ3PIR44O3E6N6COZZPRDNSMTQCFI5RN7FBRV2CBALA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.598968278 (407h9m15.470051323s) [noise=true , address=185.167.204.205:4033 , piece_id=A32ZTBDBWMCNBMZPD4DUWS5NKUKQ22ACQH6HAW5AKB3FNHBC56VA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.599287886 (407h9m15.470370931s) [noise=true , address=185.167.204.206:4072 , piece_id=I6PCSC5JYTJXLT6B34TTIVZQVX52R7TRHSRLRP4ZHG5J63XDZN6Q, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.599632252 (407h9m15.470715297s) [noise=true , address=185.167.204.206:4070 , piece_id=Y7IRY4E2GCF7TCFOIQJ7QH6IFEJBMW4WM6QBIS5SCUJ7E7ASV4HA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.599676297 (407h9m15.470759332s) [noise=true , address=185.167.204.206:4071 , piece_id=DVJAKIMPAUGWQPHDPGTXASQDONAKOZIGX5PQDBGBXNQAPMBPUFBA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.601435877 (407h9m15.472518912s) [noise=true , address=185.167.204.206:4057 , piece_id=ONW7YS5JG44B2STNFJGJWFEZ3WYXO74N3CXI3C6T5OTJYHGQPQZQ, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:17.299689191 (407h9m17.170772226s) [noise=true , address=89.149.201.147:4019 , piece_id=ILE7ZX5X6HBPUVYGCOQ5SNG3AA6B3YTXIF47CPLJ3MN5EXVOS2OA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:17.526601953 (407h9m17.397684998s) [noise=true , address=89.149.201.147:4015 , piece_id=ZIWRTD7HSP7YMRJET2TTJQFAQZM7KGDKJDLMYCWWY2W6SV2UVQBQ, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:18.294911214 (407h9m18.165994249s) [noise=true , address=89.149.201.147:4017 , piece_id=ZOSNSMZOBQVIKD2ZGJRAUWJEJ4JXJK452YWQQSDL77E77F6LRDZA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:18.402191355 (407h9m18.2732744s) [noise=true , address=89.149.201.147:4011 , piece_id=ATMM77IGBDSRHNJ5FQ2VKP7W27JUA2F6HYJYSQ62IL5JQN5PF5OA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:22.436508909 (407h9m22.307591944s) [noise=true , address=81.171.14.70:4037 , piece_id=NYGC2ZLUSBUAGFULQ46LXCVNQUJ5M4XT3YICVA2ZMBR2D6WPJ27Q, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:25.935108368 (407h9m25.806191413s) [noise=true , address=81.171.14.70:4046 , piece_id=NRMV6FG5CZSTLUNP2EHNCK6VM2Z2UHKBLCHLKEPQ225Q5DGYEBPA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:30.904793996 (407h9m30.775877041s) [noise=true , address=81.171.14.70:4029 , piece_id=APDMOYEMDCLOREXY2BBOK73RSQVWKIXUX4SCHRVFDSM7ZLWBEQXA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:32.926010512 (407h9m32.797093567s) [noise=true , address=81.171.14.70:4044 , piece_id=QCWQO6JQRRKSG5U2KVH3Y6I3UI2F7G3DOW4CH3PQZKUVQ6Q2XOHA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:32.926036552 (407h9m32.797119587s) [seg_pos={0 0}, upload=357453]: Segment reached optimal threshold of 26 pieces. Dec 22 14:38:32.926196857 (407h9m32.797279902s) [noise=true , address=81.171.14.70:4033 , piece_id=MKNAJOFD5CIHTA55XO476LOJJGIWSA7HBQFQND2YNM562W74IKJA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BnjusiXEfHYwTDL8soMkm8GYnhbkwy2oCAxVA7oiDyhFXozZ8): protocol: expected piece hash; context canceled; EOF Dec 22 14:38:32.926203058 (407h9m32.797286093s) [noise=true , address=81.171.14.70:4033 , piece_id=MKNAJOFD5CIHTA55XO476LOJJGIWSA7HBQFQND2YNM562W74IKJA, seg_pos={0 0}, upload=357453]: Long tail cancellation due to optimal threshold Dec 22 14:38:32.926237574 (407h9m32.797320609s) [noise=true , address=81.171.14.70:4031 , piece_id=7MD5N7PPKFPCFMOIXAGI3MPBDDX45GTLMA3CFGNM7BBGI6TSL6RA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12drdYnnWsuMyFo6LkcewAGGWnorRCAhxPpNqK41LvGy2zkBvng): protocol: expected piece hash; context canceled; EOF Dec 22 14:38:32.926250028 (407h9m32.797333063s) [noise=true , address=81.171.14.70:4031 , piece_id=7MD5N7PPKFPCFMOIXAGI3MPBDDX45GTLMA3CFGNM7BBGI6TSL6RA, seg_pos={0 0}, upload=357453]: Long tail cancellation due to optimal threshold Dec 22 14:38:32.926299612 (407h9m32.797382647s) [seg_pos={0 0}, upload=357453]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 22 14:38:32.926403941 (407h9m32.797486986s) [upload=357453]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 14:38:32.926409692 (407h9m32.797492737s) [upload=357453]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 14:39:04.793379844 (407h10m4.664462889s) [upload=357453]: Done uploading segments... Dec 22 14:39:04.793413608 (407h10m4.664496643s) [upload=357453]: Upload finished. err: <nil> Duration: 7.143921229s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 1.445605659s trace: https://jaeger.dev.storj.io/trace/62b55ec615ab6123 FAIL (2025-12-22 15:08:26 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.445634823s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 5.029130884s trace: https://jaeger.dev.storj.io/trace/510cfc1b6c68e873 FAIL (2025-12-22 15:07:59 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.029163094s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....................................................X......] duration: 4.540522326s trace: https://jaeger.dev.storj.io/trace/2e095cd265881ac3 FAIL (2025-12-22 15:07:50 UTC) orbiter warning: context deadline exceeded Duration: 4.540555168s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.............................X.....XX.....XX.XXXXX.X.X.X.XX.] duration: 26.724436726s trace: https://jaeger.dev.storj.io/trace/441901c1401a4ab FAIL (2025-12-22 15:07:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, Piece ID: GNNUGRMOXDYZG6LRU7VJU5HZHCMCT3IJQRZQTP2PVNWQY3TO4URQ) context deadline exceeded; piecestore: (Node ID: 12vXrkhLqCofhR4Fmfqb98gxnqWQF6vMV4q3ATbRvPfcqi82WSo, Piece ID: G7PES7HSPJCP3WZZDW43GLCBSOHABXUWRHPBRFAMEBWEHNAUS6KQ) context deadline exceeded ======LOGS====== ents... Dec 22 15:06:47.625653670 (407h37m47.496736715s) [upload=357868]: Got next segment. Inline: false Dec 22 15:06:47.625684469 (407h37m47.496767504s) [upload=357868]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 15:06:47.625690961 (407h37m47.496773996s) [upload=357868]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 15:06:47.973868734 (407h37m47.844951769s) [seg_pos={0 0}, upload=357868]: Begin upload segment... Dec 22 15:06:47.973915943 (407h37m47.844998978s) [seg_pos={0 0}, upload=357868]: Done begin upload segment. Dec 22 15:06:47.973970137 (407h37m47.845053172s) [noise=true , address=145.79.227.50:4373 , piece_id=DLKEK6A7ZTH5PJ7PY2TQKJV6Y4DCRXOX6F74ZAPYPUDC2NASNHRQ, seg_pos={0 0}, upload=357868]: Uploading piece... Dec 22 15:06:47.973975477 (407h37m47.845058512s) [noise=true , address=77.90.31.13:4627 , piece_id=3DHR7CSXFLL3G72EAUC7R77TXGKOB75IASWTBYKXXW4KTE4XTF5Q, seg_pos={0 0}, upload=357868]: Uploading piece... Dec 22 15:06:47.974537667 (407h37m47.845620712s) [noise=true , address=145.79.227.51:4409 , piece_id=KP3SCGRUZ44VUEK4Z2IQGH4AWM2G42R5EJTUXXMR3BG5OWKX55EQ, seg_pos={0 0}, upload=357868]: Uploading piece... Dec 22 15:06:47.974936967 (407h37m47.846020002s) [noise=true , address=145.79.227.19:4078 , piece_id=GNNUGRMOXDYZG6LRU7VJU5HZHCMCT3IJQRZQTP2PVNWQY3TO4URQ, seg_pos={0 0}, upload=357868]: Uploading piece... Dec 22 15:06:47.975340776 (407h37m47.846423821s) [noise=true , address=145.79.227.45:4230 , piece_id=GONHQAHABEILZRNGYJ7BNAT5GJPGTRJJPU6K6DZPT55U6AZG43LQ, seg_pos={0 0}, upload=357868]: Uploading piece... Dec 22 15:06:47.975864694 (407h37m47.846947729s) [noise=true , address=145.79.227.10:4401 , piece_id=Z4UE533U5LO5SE3MMYI2LN2EVRJOCWH6Y6VO64O6NCH3YCSJLKLA, seg_pos={0 0}, upload=357868]: Uploading piece... Dec 22 15:06:47.976380807 (407h37m47.847463842s) [noise=true , address=77.90.31.14:4664 , piece_id=24B32ZXZ7PXE3UWN3HSH3CDII4BCV37D7ADUEN6RELHP5XOURLKQ, seg_pos={0 0}, upload=357868]: Uploading piece... Dec 22 15:06:47.977513683 (407h37m47.848596718s) [noise=true , address=45.91.116.10:4082 , piece_id=RJLLAUKD2PUYKAPV3SINBJ2ZOEFMV2H7GTWB4N6PP3LUVELGIHAQ, seg_pos={0 0}, upload=357868]: Uploading piece... Dec 22 15:06:47.978128144 (407h37m47.849211189s) [noise=true , address=145.79.227.41:4085 , piece_id=DAUW65NPTXXYTRXA22JH3UZLIIJZW3LPH2PRZK37MY3CQ6JHSILQ, seg_pos={0 0}, upload=357868]: Uploading piece... Dec 22 15:06:47.978685886 (407h37m47.849768921s) [noise=true , address=145.79.227.48:4301 , piece_id=62WZGIKO3TQB77FJ736O54GCZIDHO6XZ7MEOG2FJEU7PDDGKB37Q, seg_pos={0 0}, upload=357868]: Uploading piece... Dec 22 15:06:47.979746856 (407h37m47.850829891s) [noise=true , address=45.91.116.8:4045 , piece_id=DFXGNOWWQB2PX5C7WVBNYNI4JITTK36RMUZXXDMGOFI5LXINTHFQ, seg_pos={0 0}, upload=357868]: Uploading piece... Dec 22 15:06:47.979987584 (407h37m47.851070619s) [noise=true , address=145.79.227.46:4601 , piece_id=MHR47USKMQKYRDOQIMOBLREZFBL6MMUA6VAL63JHMCMRHJHI5RVQ, seg_pos={0 0}, upload=357868]: Uploading piece... Dec 22 15:06:47.980846570 (407h37m47.851929615s) [noise=true , address=145.79.227.37:4567 , piece_id=3BMIDEMKR5PA73DSVXMO4RAOYJUX5J2WWQZQNHE4DPXHO7GX4LOA, seg_pos={0 0}, upload=357868]: Uploading piece... Dec 22 15:06:47.982410298 (407h37m47.853493343s) [noise=true , address=145.79.227.52:4445 , piece_id=G7PES7HSPJCP3WZZDW43GLCBSOHABXUWRHPBRFAMEBWEHNAUS6KQ, seg_pos={0 0}, upload=357868]: Uploading piece... Dec 22 15:06:47.991570440 (407h37m47.862653485s) [noise=true , address=145.79.227.49:4338 , piece_id=TCGWOAGODCWUETO6RSRS2MY2RKHORFF4SLOOBRBWHPLZ6LV4L6QQ, seg_pos={0 0}, upload=357868]: Uploading piece... Dec 22 15:06:47.992632723 (407h37m47.863715768s) [noise=true , address=145.79.227.11:4730 , piece_id=PLWIDPQVN52IMCPPWJKOKRYYVC7I2OUOF6MBBEJDVIAZUS3FXM3Q, seg_pos={0 0}, upload=357868]: Uploading piece... Dec 22 15:06:49.855973533 (407h37m49.727056578s) [upload=357868]: Next returned nil segment Dec 22 15:06:49.855991978 (407h37m49.727075033s) [upload=357868]: Waiting for error group managing segments... Dec 22 15:06:50.231347792 (407h37m50.102430837s) [noise=true , address=77.90.31.14:4664 , piece_id=24B32ZXZ7PXE3UWN3HSH3CDII4BCV37D7ADUEN6RELHP5XOURLKQ, seg_pos={0 0}, upload=357868]: Done uploading piece. err: <nil> Dec 22 15:06:50.369762198 (407h37m50.240845243s) [noise=true , address=45.91.116.10:4082 , piece_id=RJLLAUKD2PUYKAPV3SINBJ2ZOEFMV2H7GTWB4N6PP3LUVELGIHAQ, seg_pos={0 0}, upload=357868]: Done uploading piece. err: <nil> Dec 22 15:06:50.643729590 (407h37m50.514812635s) [noise=true , address=45.91.116.8:4045 , piece_id=DFXGNOWWQB2PX5C7WVBNYNI4JITTK36RMUZXXDMGOFI5LXINTHFQ, seg_pos={0 0}, upload=357868]: Done uploading piece. err: <nil> Dec 22 15:06:50.885256630 (407h37m50.756339675s) [noise=true , address=145.79.227.49:4338 , piece_id=TCGWOAGODCWUETO6RSRS2MY2RKHORFF4SLOOBRBWHPLZ6LV4L6QQ, seg_pos={0 0}, upload=357868]: Done uploading piece. err: <nil> Dec 22 15:06:50.886610147 (407h37m50.757693192s) [noise=true , address=145.79.227.41:4085 , piece_id=DAUW65NPTXXYTRXA22JH3UZLIIJZW3LPH2PRZK37MY3CQ6JHSILQ, seg_pos={0 0}, upload=357868]: Done uploading piece. err: <nil> Dec 22 15:06:50.932261685 (407h37m50.80334472s) [noise=true , address=145.79.227.45:4230 , piece_id=GONHQAHABEILZRNGYJ7BNAT5GJPGTRJJPU6K6DZPT55U6AZG43LQ, seg_pos={0 0}, upload=357868]: Done uploading piece. err: <nil> Dec 22 15:06:50.972473486 (407h37m50.843556531s) [noise=true , address=145.79.227.50:4373 , piece_id=DLKEK6A7ZTH5PJ7PY2TQKJV6Y4DCRXOX6F74ZAPYPUDC2NASNHRQ, seg_pos={0 0}, upload=357868]: Done uploading piece. err: <nil> Dec 22 15:06:50.974127956 (407h37m50.845211001s) [noise=true , address=145.79.227.52:4445 , piece_id=G7PES7HSPJCP3WZZDW43GLCBSOHABXUWRHPBRFAMEBWEHNAUS6KQ, seg_pos={0 0}, upload=357868]: Done uploading piece. err: <nil> Dec 22 15:06:50.978938186 (407h37m50.850021221s) [noise=true , address=145.79.227.19:4078 , piece_id=GNNUGRMOXDYZG6LRU7VJU5HZHCMCT3IJQRZQTP2PVNWQY3TO4URQ, seg_pos={0 0}, upload=357868]: Done uploading piece. err: <nil> Dec 22 15:06:51.004101411 (407h37m50.875184446s) [noise=true , address=145.79.227.11:4730 , piece_id=PLWIDPQVN52IMCPPWJKOKRYYVC7I2OUOF6MBBEJDVIAZUS3FXM3Q, seg_pos={0 0}, upload=357868]: Done uploading piece. err: <nil> Dec 22 15:06:51.006173176 (407h37m50.877256221s) [noise=true , address=145.79.227.46:4601 , piece_id=MHR47USKMQKYRDOQIMOBLREZFBL6MMUA6VAL63JHMCMRHJHI5RVQ, seg_pos={0 0}, upload=357868]: Done uploading piece. err: <nil> Dec 22 15:06:51.010208019 (407h37m50.881291065s) [noise=true , address=145.79.227.51:4409 , piece_id=KP3SCGRUZ44VUEK4Z2IQGH4AWM2G42R5EJTUXXMR3BG5OWKX55EQ, seg_pos={0 0}, upload=357868]: Done uploading piece. err: <nil> Dec 22 15:06:51.031126141 (407h37m50.902209176s) [noise=true , address=145.79.227.37:4567 , piece_id=3BMIDEMKR5PA73DSVXMO4RAOYJUX5J2WWQZQNHE4DPXHO7GX4LOA, seg_pos={0 0}, upload=357868]: Done uploading piece. err: <nil> Dec 22 15:06:51.031146661 (407h37m50.902229696s) [seg_pos={0 0}, upload=357868]: Segment reached optimal threshold of 13 pieces. Dec 22 15:06:51.031307857 (407h37m50.902390892s) [noise=true , address=145.79.227.48:4301 , piece_id=62WZGIKO3TQB77FJ736O54GCZIDHO6XZ7MEOG2FJEU7PDDGKB37Q, seg_pos={0 0}, upload=357868]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18HkaSrdHHnvmeNz1ormfcSBZH8bMUgkX7X3JmGuqkmtBoak29): protocol: expected piece hash; context canceled; EOF Dec 22 15:06:51.031313898 (407h37m50.902396933s) [noise=true , address=145.79.227.48:4301 , piece_id=62WZGIKO3TQB77FJ736O54GCZIDHO6XZ7MEOG2FJEU7PDDGKB37Q, seg_pos={0 0}, upload=357868]: Long tail cancellation due to optimal threshold Dec 22 15:06:51.031641743 (407h37m50.902724788s) [noise=true , address=145.79.227.10:4401 , piece_id=Z4UE533U5LO5SE3MMYI2LN2EVRJOCWH6Y6VO64O6NCH3YCSJLKLA, seg_pos={0 0}, upload=357868]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1): protocol: write tcp 188.166.225.170:50220->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:50220->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50220->145.79.227.10:4401: use of closed network connection Dec 22 15:06:51.031649508 (407h37m50.902732553s) [noise=true , address=145.79.227.10:4401 , piece_id=Z4UE533U5LO5SE3MMYI2LN2EVRJOCWH6Y6VO64O6NCH3YCSJLKLA, seg_pos={0 0}, upload=357868]: Long tail cancellation due to optimal threshold Dec 22 15:06:51.031753205 (407h37m50.90283624s) [noise=true , address=77.90.31.13:4627 , piece_id=3DHR7CSXFLL3G72EAUC7R77TXGKOB75IASWTBYKXXW4KTE4XTF5Q, seg_pos={0 0}, upload=357868]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: write tcp 188.166.225.170:35116->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:35116->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35116->77.90.31.13:4627: use of closed network connection Dec 22 15:06:51.031758966 (407h37m50.902842011s) [noise=true , address=77.90.31.13:4627 , piece_id=3DHR7CSXFLL3G72EAUC7R77TXGKOB75IASWTBYKXXW4KTE4XTF5Q, seg_pos={0 0}, upload=357868]: Long tail cancellation due to optimal threshold Dec 22 15:06:51.031773544 (407h37m50.90285659s) [seg_pos={0 0}, upload=357868]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 15:06:51.031853356 (407h37m50.902936391s) [upload=357868]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 15:06:51.031857965 (407h37m50.90294101s) [upload=357868]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 15:06:51.444127860 (407h37m51.315210895s) [upload=357868]: Done uploading segments... Dec 22 15:06:51.444144831 (407h37m51.315227867s) [upload=357868]: Upload finished. err: <nil> Duration: 26.724452668s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 5.142314034s trace: https://jaeger.dev.storj.io/trace/510cfc1b6c68e873 FAIL (2025-12-22 15:07:28 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.14233798s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 1.458797859s trace: https://jaeger.dev.storj.io/trace/62b55ec615ab6123 FAIL (2025-12-22 15:07:26 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.45882536s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [..............................X.....XX.....XX.XXXXX.X.X.X.XX] duration: 30.003826873s trace: https://jaeger.dev.storj.io/trace/441901c1401a4ab FAIL (2025-12-22 15:07:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, Piece ID: GNNUGRMOXDYZG6LRU7VJU5HZHCMCT3IJQRZQTP2PVNWQY3TO4URQ) context deadline exceeded; piecestore: (Node ID: 12vXrkhLqCofhR4Fmfqb98gxnqWQF6vMV4q3ATbRvPfcqi82WSo, Piece ID: G7PES7HSPJCP3WZZDW43GLCBSOHABXUWRHPBRFAMEBWEHNAUS6KQ) context deadline exceeded ======LOGS====== ents... Dec 22 15:06:47.625653670 (407h37m47.496736715s) [upload=357868]: Got next segment. Inline: false Dec 22 15:06:47.625684469 (407h37m47.496767504s) [upload=357868]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 15:06:47.625690961 (407h37m47.496773996s) [upload=357868]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 15:06:47.973868734 (407h37m47.844951769s) [seg_pos={0 0}, upload=357868]: Begin upload segment... Dec 22 15:06:47.973915943 (407h37m47.844998978s) [seg_pos={0 0}, upload=357868]: Done begin upload segment. Dec 22 15:06:47.973970137 (407h37m47.845053172s) [noise=true , address=145.79.227.50:4373 , piece_id=DLKEK6A7ZTH5PJ7PY2TQKJV6Y4DCRXOX6F74ZAPYPUDC2NASNHRQ, seg_pos={0 0}, upload=357868]: Uploading piece... Dec 22 15:06:47.973975477 (407h37m47.845058512s) [noise=true , address=77.90.31.13:4627 , piece_id=3DHR7CSXFLL3G72EAUC7R77TXGKOB75IASWTBYKXXW4KTE4XTF5Q, seg_pos={0 0}, upload=357868]: Uploading piece... Dec 22 15:06:47.974537667 (407h37m47.845620712s) [noise=true , address=145.79.227.51:4409 , piece_id=KP3SCGRUZ44VUEK4Z2IQGH4AWM2G42R5EJTUXXMR3BG5OWKX55EQ, seg_pos={0 0}, upload=357868]: Uploading piece... Dec 22 15:06:47.974936967 (407h37m47.846020002s) [noise=true , address=145.79.227.19:4078 , piece_id=GNNUGRMOXDYZG6LRU7VJU5HZHCMCT3IJQRZQTP2PVNWQY3TO4URQ, seg_pos={0 0}, upload=357868]: Uploading piece... Dec 22 15:06:47.975340776 (407h37m47.846423821s) [noise=true , address=145.79.227.45:4230 , piece_id=GONHQAHABEILZRNGYJ7BNAT5GJPGTRJJPU6K6DZPT55U6AZG43LQ, seg_pos={0 0}, upload=357868]: Uploading piece... Dec 22 15:06:47.975864694 (407h37m47.846947729s) [noise=true , address=145.79.227.10:4401 , piece_id=Z4UE533U5LO5SE3MMYI2LN2EVRJOCWH6Y6VO64O6NCH3YCSJLKLA, seg_pos={0 0}, upload=357868]: Uploading piece... Dec 22 15:06:47.976380807 (407h37m47.847463842s) [noise=true , address=77.90.31.14:4664 , piece_id=24B32ZXZ7PXE3UWN3HSH3CDII4BCV37D7ADUEN6RELHP5XOURLKQ, seg_pos={0 0}, upload=357868]: Uploading piece... Dec 22 15:06:47.977513683 (407h37m47.848596718s) [noise=true , address=45.91.116.10:4082 , piece_id=RJLLAUKD2PUYKAPV3SINBJ2ZOEFMV2H7GTWB4N6PP3LUVELGIHAQ, seg_pos={0 0}, upload=357868]: Uploading piece... Dec 22 15:06:47.978128144 (407h37m47.849211189s) [noise=true , address=145.79.227.41:4085 , piece_id=DAUW65NPTXXYTRXA22JH3UZLIIJZW3LPH2PRZK37MY3CQ6JHSILQ, seg_pos={0 0}, upload=357868]: Uploading piece... Dec 22 15:06:47.978685886 (407h37m47.849768921s) [noise=true , address=145.79.227.48:4301 , piece_id=62WZGIKO3TQB77FJ736O54GCZIDHO6XZ7MEOG2FJEU7PDDGKB37Q, seg_pos={0 0}, upload=357868]: Uploading piece... Dec 22 15:06:47.979746856 (407h37m47.850829891s) [noise=true , address=45.91.116.8:4045 , piece_id=DFXGNOWWQB2PX5C7WVBNYNI4JITTK36RMUZXXDMGOFI5LXINTHFQ, seg_pos={0 0}, upload=357868]: Uploading piece... Dec 22 15:06:47.979987584 (407h37m47.851070619s) [noise=true , address=145.79.227.46:4601 , piece_id=MHR47USKMQKYRDOQIMOBLREZFBL6MMUA6VAL63JHMCMRHJHI5RVQ, seg_pos={0 0}, upload=357868]: Uploading piece... Dec 22 15:06:47.980846570 (407h37m47.851929615s) [noise=true , address=145.79.227.37:4567 , piece_id=3BMIDEMKR5PA73DSVXMO4RAOYJUX5J2WWQZQNHE4DPXHO7GX4LOA, seg_pos={0 0}, upload=357868]: Uploading piece... Dec 22 15:06:47.982410298 (407h37m47.853493343s) [noise=true , address=145.79.227.52:4445 , piece_id=G7PES7HSPJCP3WZZDW43GLCBSOHABXUWRHPBRFAMEBWEHNAUS6KQ, seg_pos={0 0}, upload=357868]: Uploading piece... Dec 22 15:06:47.991570440 (407h37m47.862653485s) [noise=true , address=145.79.227.49:4338 , piece_id=TCGWOAGODCWUETO6RSRS2MY2RKHORFF4SLOOBRBWHPLZ6LV4L6QQ, seg_pos={0 0}, upload=357868]: Uploading piece... Dec 22 15:06:47.992632723 (407h37m47.863715768s) [noise=true , address=145.79.227.11:4730 , piece_id=PLWIDPQVN52IMCPPWJKOKRYYVC7I2OUOF6MBBEJDVIAZUS3FXM3Q, seg_pos={0 0}, upload=357868]: Uploading piece... Dec 22 15:06:49.855973533 (407h37m49.727056578s) [upload=357868]: Next returned nil segment Dec 22 15:06:49.855991978 (407h37m49.727075033s) [upload=357868]: Waiting for error group managing segments... Dec 22 15:06:50.231347792 (407h37m50.102430837s) [noise=true , address=77.90.31.14:4664 , piece_id=24B32ZXZ7PXE3UWN3HSH3CDII4BCV37D7ADUEN6RELHP5XOURLKQ, seg_pos={0 0}, upload=357868]: Done uploading piece. err: <nil> Dec 22 15:06:50.369762198 (407h37m50.240845243s) [noise=true , address=45.91.116.10:4082 , piece_id=RJLLAUKD2PUYKAPV3SINBJ2ZOEFMV2H7GTWB4N6PP3LUVELGIHAQ, seg_pos={0 0}, upload=357868]: Done uploading piece. err: <nil> Dec 22 15:06:50.643729590 (407h37m50.514812635s) [noise=true , address=45.91.116.8:4045 , piece_id=DFXGNOWWQB2PX5C7WVBNYNI4JITTK36RMUZXXDMGOFI5LXINTHFQ, seg_pos={0 0}, upload=357868]: Done uploading piece. err: <nil> Dec 22 15:06:50.885256630 (407h37m50.756339675s) [noise=true , address=145.79.227.49:4338 , piece_id=TCGWOAGODCWUETO6RSRS2MY2RKHORFF4SLOOBRBWHPLZ6LV4L6QQ, seg_pos={0 0}, upload=357868]: Done uploading piece. err: <nil> Dec 22 15:06:50.886610147 (407h37m50.757693192s) [noise=true , address=145.79.227.41:4085 , piece_id=DAUW65NPTXXYTRXA22JH3UZLIIJZW3LPH2PRZK37MY3CQ6JHSILQ, seg_pos={0 0}, upload=357868]: Done uploading piece. err: <nil> Dec 22 15:06:50.932261685 (407h37m50.80334472s) [noise=true , address=145.79.227.45:4230 , piece_id=GONHQAHABEILZRNGYJ7BNAT5GJPGTRJJPU6K6DZPT55U6AZG43LQ, seg_pos={0 0}, upload=357868]: Done uploading piece. err: <nil> Dec 22 15:06:50.972473486 (407h37m50.843556531s) [noise=true , address=145.79.227.50:4373 , piece_id=DLKEK6A7ZTH5PJ7PY2TQKJV6Y4DCRXOX6F74ZAPYPUDC2NASNHRQ, seg_pos={0 0}, upload=357868]: Done uploading piece. err: <nil> Dec 22 15:06:50.974127956 (407h37m50.845211001s) [noise=true , address=145.79.227.52:4445 , piece_id=G7PES7HSPJCP3WZZDW43GLCBSOHABXUWRHPBRFAMEBWEHNAUS6KQ, seg_pos={0 0}, upload=357868]: Done uploading piece. err: <nil> Dec 22 15:06:50.978938186 (407h37m50.850021221s) [noise=true , address=145.79.227.19:4078 , piece_id=GNNUGRMOXDYZG6LRU7VJU5HZHCMCT3IJQRZQTP2PVNWQY3TO4URQ, seg_pos={0 0}, upload=357868]: Done uploading piece. err: <nil> Dec 22 15:06:51.004101411 (407h37m50.875184446s) [noise=true , address=145.79.227.11:4730 , piece_id=PLWIDPQVN52IMCPPWJKOKRYYVC7I2OUOF6MBBEJDVIAZUS3FXM3Q, seg_pos={0 0}, upload=357868]: Done uploading piece. err: <nil> Dec 22 15:06:51.006173176 (407h37m50.877256221s) [noise=true , address=145.79.227.46:4601 , piece_id=MHR47USKMQKYRDOQIMOBLREZFBL6MMUA6VAL63JHMCMRHJHI5RVQ, seg_pos={0 0}, upload=357868]: Done uploading piece. err: <nil> Dec 22 15:06:51.010208019 (407h37m50.881291065s) [noise=true , address=145.79.227.51:4409 , piece_id=KP3SCGRUZ44VUEK4Z2IQGH4AWM2G42R5EJTUXXMR3BG5OWKX55EQ, seg_pos={0 0}, upload=357868]: Done uploading piece. err: <nil> Dec 22 15:06:51.031126141 (407h37m50.902209176s) [noise=true , address=145.79.227.37:4567 , piece_id=3BMIDEMKR5PA73DSVXMO4RAOYJUX5J2WWQZQNHE4DPXHO7GX4LOA, seg_pos={0 0}, upload=357868]: Done uploading piece. err: <nil> Dec 22 15:06:51.031146661 (407h37m50.902229696s) [seg_pos={0 0}, upload=357868]: Segment reached optimal threshold of 13 pieces. Dec 22 15:06:51.031307857 (407h37m50.902390892s) [noise=true , address=145.79.227.48:4301 , piece_id=62WZGIKO3TQB77FJ736O54GCZIDHO6XZ7MEOG2FJEU7PDDGKB37Q, seg_pos={0 0}, upload=357868]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18HkaSrdHHnvmeNz1ormfcSBZH8bMUgkX7X3JmGuqkmtBoak29): protocol: expected piece hash; context canceled; EOF Dec 22 15:06:51.031313898 (407h37m50.902396933s) [noise=true , address=145.79.227.48:4301 , piece_id=62WZGIKO3TQB77FJ736O54GCZIDHO6XZ7MEOG2FJEU7PDDGKB37Q, seg_pos={0 0}, upload=357868]: Long tail cancellation due to optimal threshold Dec 22 15:06:51.031641743 (407h37m50.902724788s) [noise=true , address=145.79.227.10:4401 , piece_id=Z4UE533U5LO5SE3MMYI2LN2EVRJOCWH6Y6VO64O6NCH3YCSJLKLA, seg_pos={0 0}, upload=357868]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1): protocol: write tcp 188.166.225.170:50220->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:50220->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50220->145.79.227.10:4401: use of closed network connection Dec 22 15:06:51.031649508 (407h37m50.902732553s) [noise=true , address=145.79.227.10:4401 , piece_id=Z4UE533U5LO5SE3MMYI2LN2EVRJOCWH6Y6VO64O6NCH3YCSJLKLA, seg_pos={0 0}, upload=357868]: Long tail cancellation due to optimal threshold Dec 22 15:06:51.031753205 (407h37m50.90283624s) [noise=true , address=77.90.31.13:4627 , piece_id=3DHR7CSXFLL3G72EAUC7R77TXGKOB75IASWTBYKXXW4KTE4XTF5Q, seg_pos={0 0}, upload=357868]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: write tcp 188.166.225.170:35116->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:35116->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35116->77.90.31.13:4627: use of closed network connection Dec 22 15:06:51.031758966 (407h37m50.902842011s) [noise=true , address=77.90.31.13:4627 , piece_id=3DHR7CSXFLL3G72EAUC7R77TXGKOB75IASWTBYKXXW4KTE4XTF5Q, seg_pos={0 0}, upload=357868]: Long tail cancellation due to optimal threshold Dec 22 15:06:51.031773544 (407h37m50.90285659s) [seg_pos={0 0}, upload=357868]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 15:06:51.031853356 (407h37m50.902936391s) [upload=357868]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 15:06:51.031857965 (407h37m50.90294101s) [upload=357868]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 15:06:51.444127860 (407h37m51.315210895s) [upload=357868]: Done uploading segments... Dec 22 15:06:51.444144831 (407h37m51.315227867s) [upload=357868]: Upload finished. err: <nil> Duration: 30.003849617s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 8.449306884s trace: https://jaeger.dev.storj.io/trace/718f722b1bda7fe2 FAIL (2025-12-22 15:07:14 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== VMAC4CKPNLATBLZLKAGDGZHVT7BHEPN3Q, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.367093115 (407h9m14.23817616s) [noise=true , address=185.167.204.206:4043 , piece_id=HL32YEZF67X2UT6SRMXJVJ4W5A3PC5SXOCM7BYQVMEC76TKEVJRA, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.367396732 (407h9m14.238479767s) [noise=true , address=185.167.204.205:4032 , piece_id=FJMXJG2PQ4WZDSDCPJ6LLSVQKI7JJHUCDUZLU2BJZV7RZTLETM5A, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.367549273 (407h9m14.238632308s) [noise=true , address=185.167.204.206:4070 , piece_id=Y7IRY4E2GCF7TCFOIQJ7QH6IFEJBMW4WM6QBIS5SCUJ7E7ASV4HA, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.367749263 (407h9m14.238832308s) [noise=true , address=81.171.14.70:4046 , piece_id=NRMV6FG5CZSTLUNP2EHNCK6VM2Z2UHKBLCHLKEPQ225Q5DGYEBPA, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.368034046 (407h9m14.239117091s) [noise=true , address=185.167.204.206:4060 , piece_id=WN6T5TUBVMURPSBBXGXCTLPZZ74PFNSRTTWXI2JPUTY74DPH3YSQ, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.368552724 (407h9m14.239635759s) [noise=true , address=81.171.14.70:4029 , piece_id=APDMOYEMDCLOREXY2BBOK73RSQVWKIXUX4SCHRVFDSM7ZLWBEQXA, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.369160691 (407h9m14.240243736s) [noise=true , address=185.167.204.206:4057 , piece_id=ONW7YS5JG44B2STNFJGJWFEZ3WYXO74N3CXI3C6T5OTJYHGQPQZQ, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.369714496 (407h9m14.240797541s) [noise=true , address=185.167.204.206:4071 , piece_id=DVJAKIMPAUGWQPHDPGTXASQDONAKOZIGX5PQDBGBXNQAPMBPUFBA, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.370249614 (407h9m14.241332649s) [noise=true , address=89.149.201.147:4019 , piece_id=ILE7ZX5X6HBPUVYGCOQ5SNG3AA6B3YTXIF47CPLJ3MN5EXVOS2OA, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.372019455 (407h9m14.24310249s) [noise=true , address=89.149.201.147:4015 , piece_id=ZIWRTD7HSP7YMRJET2TTJQFAQZM7KGDKJDLMYCWWY2W6SV2UVQBQ, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:15.199398795 (407h9m15.07048184s) [upload=357453]: Next returned nil segment Dec 22 14:38:15.199414966 (407h9m15.070498011s) [upload=357453]: Waiting for error group managing segments... Dec 22 14:38:15.591187572 (407h9m15.462270607s) [noise=true , address=185.167.204.205:4009 , piece_id=JTHX2O6T5HRENGEMAFJUWQUEUYAIG2KQ5HDEHF4BK42IT2ER7EIA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.591766795 (407h9m15.46284984s) [noise=true , address=185.167.204.205:4006 , piece_id=XHNFS6G7EX4BQWKOK4JUWYV3OTVJ36A6VOP4N7G2XPZKNNCFFKXA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.592074521 (407h9m15.463157566s) [noise=true , address=185.167.204.206:4060 , piece_id=WN6T5TUBVMURPSBBXGXCTLPZZ74PFNSRTTWXI2JPUTY74DPH3YSQ, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.593304694 (407h9m15.464387729s) [noise=true , address=185.167.204.205:4036 , piece_id=5QWJTTL6TZIZSSYHCBEJBKGX7BS76VVTELWTKEAKTOVYKINXVWDQ, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.593638659 (407h9m15.464721694s) [noise=true , address=185.167.204.205:4004 , piece_id=PYGDQZMAKOVJWB7AWEPFMPMB6C24FMS4HT2UP2AKZNYGG4DGZR3Q, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.593818251 (407h9m15.464901286s) [noise=true , address=185.167.204.206:4048 , piece_id=52WC3XH2K2BHG33BQPRENPNYDT7EOKCNBCTZKE3LIONDJHVATEYA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.594252549 (407h9m15.465335594s) [noise=true , address=185.167.204.205:4034 , piece_id=OS5H57TG2REFE535DK4T6N75VNIXPLQWW6DM55QCC2S7QBVZM3UQ, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.597113216 (407h9m15.468196261s) [noise=true , address=185.167.204.205:4032 , piece_id=FJMXJG2PQ4WZDSDCPJ6LLSVQKI7JJHUCDUZLU2BJZV7RZTLETM5A, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.597394541 (407h9m15.468477586s) [noise=true , address=185.167.204.205:4035 , piece_id=3D7T4SGPAJQZCA74CUTR4GJRE23R4JD5KBNW3BLEASSP4GSR5PHQ, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.597612756 (407h9m15.468695792s) [noise=true , address=185.167.204.206:4043 , piece_id=HL32YEZF67X2UT6SRMXJVJ4W5A3PC5SXOCM7BYQVMEC76TKEVJRA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.598018900 (407h9m15.469101945s) [noise=true , address=185.167.204.205:4012 , piece_id=A5G37BVL2GZ7LAY3FZWIZV4R4M7M6UPYDYACJFRD5HKDWWKB4Q6A, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.598254719 (407h9m15.469337765s) [noise=true , address=185.167.204.206:4052 , piece_id=V2DVAAXN5A4C3NQW556VMAC4CKPNLATBLZLKAGDGZHVT7BHEPN3Q, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.598480649 (407h9m15.469563694s) [noise=true , address=185.167.204.205:4018 , piece_id=NUYRZ7PXKJQ3PIR44O3E6N6COZZPRDNSMTQCFI5RN7FBRV2CBALA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.598968278 (407h9m15.470051323s) [noise=true , address=185.167.204.205:4033 , piece_id=A32ZTBDBWMCNBMZPD4DUWS5NKUKQ22ACQH6HAW5AKB3FNHBC56VA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.599287886 (407h9m15.470370931s) [noise=true , address=185.167.204.206:4072 , piece_id=I6PCSC5JYTJXLT6B34TTIVZQVX52R7TRHSRLRP4ZHG5J63XDZN6Q, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.599632252 (407h9m15.470715297s) [noise=true , address=185.167.204.206:4070 , piece_id=Y7IRY4E2GCF7TCFOIQJ7QH6IFEJBMW4WM6QBIS5SCUJ7E7ASV4HA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.599676297 (407h9m15.470759332s) [noise=true , address=185.167.204.206:4071 , piece_id=DVJAKIMPAUGWQPHDPGTXASQDONAKOZIGX5PQDBGBXNQAPMBPUFBA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.601435877 (407h9m15.472518912s) [noise=true , address=185.167.204.206:4057 , piece_id=ONW7YS5JG44B2STNFJGJWFEZ3WYXO74N3CXI3C6T5OTJYHGQPQZQ, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:17.299689191 (407h9m17.170772226s) [noise=true , address=89.149.201.147:4019 , piece_id=ILE7ZX5X6HBPUVYGCOQ5SNG3AA6B3YTXIF47CPLJ3MN5EXVOS2OA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:17.526601953 (407h9m17.397684998s) [noise=true , address=89.149.201.147:4015 , piece_id=ZIWRTD7HSP7YMRJET2TTJQFAQZM7KGDKJDLMYCWWY2W6SV2UVQBQ, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:18.294911214 (407h9m18.165994249s) [noise=true , address=89.149.201.147:4017 , piece_id=ZOSNSMZOBQVIKD2ZGJRAUWJEJ4JXJK452YWQQSDL77E77F6LRDZA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:18.402191355 (407h9m18.2732744s) [noise=true , address=89.149.201.147:4011 , piece_id=ATMM77IGBDSRHNJ5FQ2VKP7W27JUA2F6HYJYSQ62IL5JQN5PF5OA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:22.436508909 (407h9m22.307591944s) [noise=true , address=81.171.14.70:4037 , piece_id=NYGC2ZLUSBUAGFULQ46LXCVNQUJ5M4XT3YICVA2ZMBR2D6WPJ27Q, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:25.935108368 (407h9m25.806191413s) [noise=true , address=81.171.14.70:4046 , piece_id=NRMV6FG5CZSTLUNP2EHNCK6VM2Z2UHKBLCHLKEPQ225Q5DGYEBPA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:30.904793996 (407h9m30.775877041s) [noise=true , address=81.171.14.70:4029 , piece_id=APDMOYEMDCLOREXY2BBOK73RSQVWKIXUX4SCHRVFDSM7ZLWBEQXA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:32.926010512 (407h9m32.797093567s) [noise=true , address=81.171.14.70:4044 , piece_id=QCWQO6JQRRKSG5U2KVH3Y6I3UI2F7G3DOW4CH3PQZKUVQ6Q2XOHA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:32.926036552 (407h9m32.797119587s) [seg_pos={0 0}, upload=357453]: Segment reached optimal threshold of 26 pieces. Dec 22 14:38:32.926196857 (407h9m32.797279902s) [noise=true , address=81.171.14.70:4033 , piece_id=MKNAJOFD5CIHTA55XO476LOJJGIWSA7HBQFQND2YNM562W74IKJA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BnjusiXEfHYwTDL8soMkm8GYnhbkwy2oCAxVA7oiDyhFXozZ8): protocol: expected piece hash; context canceled; EOF Dec 22 14:38:32.926203058 (407h9m32.797286093s) [noise=true , address=81.171.14.70:4033 , piece_id=MKNAJOFD5CIHTA55XO476LOJJGIWSA7HBQFQND2YNM562W74IKJA, seg_pos={0 0}, upload=357453]: Long tail cancellation due to optimal threshold Dec 22 14:38:32.926237574 (407h9m32.797320609s) [noise=true , address=81.171.14.70:4031 , piece_id=7MD5N7PPKFPCFMOIXAGI3MPBDDX45GTLMA3CFGNM7BBGI6TSL6RA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12drdYnnWsuMyFo6LkcewAGGWnorRCAhxPpNqK41LvGy2zkBvng): protocol: expected piece hash; context canceled; EOF Dec 22 14:38:32.926250028 (407h9m32.797333063s) [noise=true , address=81.171.14.70:4031 , piece_id=7MD5N7PPKFPCFMOIXAGI3MPBDDX45GTLMA3CFGNM7BBGI6TSL6RA, seg_pos={0 0}, upload=357453]: Long tail cancellation due to optimal threshold Dec 22 14:38:32.926299612 (407h9m32.797382647s) [seg_pos={0 0}, upload=357453]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 22 14:38:32.926403941 (407h9m32.797486986s) [upload=357453]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 14:38:32.926409692 (407h9m32.797492737s) [upload=357453]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 14:39:04.793379844 (407h10m4.664462889s) [upload=357453]: Done uploading segments... Dec 22 14:39:04.793413608 (407h10m4.664496643s) [upload=357453]: Upload finished. err: <nil> Duration: 8.449333704s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 1.518615793s trace: https://jaeger.dev.storj.io/trace/62b55ec615ab6123 FAIL (2025-12-22 15:07:11 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.518655669s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 1.404141985s trace: https://jaeger.dev.storj.io/trace/62b55ec615ab6123 FAIL (2025-12-22 15:06:53 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.404176029s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......................................................X.....] duration: 3.752243603s trace: https://jaeger.dev.storj.io/trace/2e095cd265881ac3 FAIL (2025-12-22 15:06:50 UTC) orbiter warning: context deadline exceeded Duration: 3.752281586s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 1.374263459s trace: https://jaeger.dev.storj.io/trace/62b55ec615ab6123 FAIL (2025-12-22 15:06:36 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.374287286s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 2 out of 60 is failed (threshold=3) [X........................X..................................] duration: 5.557789775s trace: https://jaeger.dev.storj.io/trace/510cfc1b6c68e873 FAIL (2025-12-22 15:06:32 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.557820714s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 5.855105707s trace: https://jaeger.dev.storj.io/trace/718f722b1bda7fe2 FAIL (2025-12-22 15:06:27 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== VMAC4CKPNLATBLZLKAGDGZHVT7BHEPN3Q, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.367093115 (407h9m14.23817616s) [noise=true , address=185.167.204.206:4043 , piece_id=HL32YEZF67X2UT6SRMXJVJ4W5A3PC5SXOCM7BYQVMEC76TKEVJRA, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.367396732 (407h9m14.238479767s) [noise=true , address=185.167.204.205:4032 , piece_id=FJMXJG2PQ4WZDSDCPJ6LLSVQKI7JJHUCDUZLU2BJZV7RZTLETM5A, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.367549273 (407h9m14.238632308s) [noise=true , address=185.167.204.206:4070 , piece_id=Y7IRY4E2GCF7TCFOIQJ7QH6IFEJBMW4WM6QBIS5SCUJ7E7ASV4HA, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.367749263 (407h9m14.238832308s) [noise=true , address=81.171.14.70:4046 , piece_id=NRMV6FG5CZSTLUNP2EHNCK6VM2Z2UHKBLCHLKEPQ225Q5DGYEBPA, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.368034046 (407h9m14.239117091s) [noise=true , address=185.167.204.206:4060 , piece_id=WN6T5TUBVMURPSBBXGXCTLPZZ74PFNSRTTWXI2JPUTY74DPH3YSQ, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.368552724 (407h9m14.239635759s) [noise=true , address=81.171.14.70:4029 , piece_id=APDMOYEMDCLOREXY2BBOK73RSQVWKIXUX4SCHRVFDSM7ZLWBEQXA, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.369160691 (407h9m14.240243736s) [noise=true , address=185.167.204.206:4057 , piece_id=ONW7YS5JG44B2STNFJGJWFEZ3WYXO74N3CXI3C6T5OTJYHGQPQZQ, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.369714496 (407h9m14.240797541s) [noise=true , address=185.167.204.206:4071 , piece_id=DVJAKIMPAUGWQPHDPGTXASQDONAKOZIGX5PQDBGBXNQAPMBPUFBA, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.370249614 (407h9m14.241332649s) [noise=true , address=89.149.201.147:4019 , piece_id=ILE7ZX5X6HBPUVYGCOQ5SNG3AA6B3YTXIF47CPLJ3MN5EXVOS2OA, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.372019455 (407h9m14.24310249s) [noise=true , address=89.149.201.147:4015 , piece_id=ZIWRTD7HSP7YMRJET2TTJQFAQZM7KGDKJDLMYCWWY2W6SV2UVQBQ, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:15.199398795 (407h9m15.07048184s) [upload=357453]: Next returned nil segment Dec 22 14:38:15.199414966 (407h9m15.070498011s) [upload=357453]: Waiting for error group managing segments... Dec 22 14:38:15.591187572 (407h9m15.462270607s) [noise=true , address=185.167.204.205:4009 , piece_id=JTHX2O6T5HRENGEMAFJUWQUEUYAIG2KQ5HDEHF4BK42IT2ER7EIA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.591766795 (407h9m15.46284984s) [noise=true , address=185.167.204.205:4006 , piece_id=XHNFS6G7EX4BQWKOK4JUWYV3OTVJ36A6VOP4N7G2XPZKNNCFFKXA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.592074521 (407h9m15.463157566s) [noise=true , address=185.167.204.206:4060 , piece_id=WN6T5TUBVMURPSBBXGXCTLPZZ74PFNSRTTWXI2JPUTY74DPH3YSQ, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.593304694 (407h9m15.464387729s) [noise=true , address=185.167.204.205:4036 , piece_id=5QWJTTL6TZIZSSYHCBEJBKGX7BS76VVTELWTKEAKTOVYKINXVWDQ, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.593638659 (407h9m15.464721694s) [noise=true , address=185.167.204.205:4004 , piece_id=PYGDQZMAKOVJWB7AWEPFMPMB6C24FMS4HT2UP2AKZNYGG4DGZR3Q, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.593818251 (407h9m15.464901286s) [noise=true , address=185.167.204.206:4048 , piece_id=52WC3XH2K2BHG33BQPRENPNYDT7EOKCNBCTZKE3LIONDJHVATEYA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.594252549 (407h9m15.465335594s) [noise=true , address=185.167.204.205:4034 , piece_id=OS5H57TG2REFE535DK4T6N75VNIXPLQWW6DM55QCC2S7QBVZM3UQ, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.597113216 (407h9m15.468196261s) [noise=true , address=185.167.204.205:4032 , piece_id=FJMXJG2PQ4WZDSDCPJ6LLSVQKI7JJHUCDUZLU2BJZV7RZTLETM5A, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.597394541 (407h9m15.468477586s) [noise=true , address=185.167.204.205:4035 , piece_id=3D7T4SGPAJQZCA74CUTR4GJRE23R4JD5KBNW3BLEASSP4GSR5PHQ, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.597612756 (407h9m15.468695792s) [noise=true , address=185.167.204.206:4043 , piece_id=HL32YEZF67X2UT6SRMXJVJ4W5A3PC5SXOCM7BYQVMEC76TKEVJRA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.598018900 (407h9m15.469101945s) [noise=true , address=185.167.204.205:4012 , piece_id=A5G37BVL2GZ7LAY3FZWIZV4R4M7M6UPYDYACJFRD5HKDWWKB4Q6A, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.598254719 (407h9m15.469337765s) [noise=true , address=185.167.204.206:4052 , piece_id=V2DVAAXN5A4C3NQW556VMAC4CKPNLATBLZLKAGDGZHVT7BHEPN3Q, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.598480649 (407h9m15.469563694s) [noise=true , address=185.167.204.205:4018 , piece_id=NUYRZ7PXKJQ3PIR44O3E6N6COZZPRDNSMTQCFI5RN7FBRV2CBALA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.598968278 (407h9m15.470051323s) [noise=true , address=185.167.204.205:4033 , piece_id=A32ZTBDBWMCNBMZPD4DUWS5NKUKQ22ACQH6HAW5AKB3FNHBC56VA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.599287886 (407h9m15.470370931s) [noise=true , address=185.167.204.206:4072 , piece_id=I6PCSC5JYTJXLT6B34TTIVZQVX52R7TRHSRLRP4ZHG5J63XDZN6Q, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.599632252 (407h9m15.470715297s) [noise=true , address=185.167.204.206:4070 , piece_id=Y7IRY4E2GCF7TCFOIQJ7QH6IFEJBMW4WM6QBIS5SCUJ7E7ASV4HA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.599676297 (407h9m15.470759332s) [noise=true , address=185.167.204.206:4071 , piece_id=DVJAKIMPAUGWQPHDPGTXASQDONAKOZIGX5PQDBGBXNQAPMBPUFBA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.601435877 (407h9m15.472518912s) [noise=true , address=185.167.204.206:4057 , piece_id=ONW7YS5JG44B2STNFJGJWFEZ3WYXO74N3CXI3C6T5OTJYHGQPQZQ, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:17.299689191 (407h9m17.170772226s) [noise=true , address=89.149.201.147:4019 , piece_id=ILE7ZX5X6HBPUVYGCOQ5SNG3AA6B3YTXIF47CPLJ3MN5EXVOS2OA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:17.526601953 (407h9m17.397684998s) [noise=true , address=89.149.201.147:4015 , piece_id=ZIWRTD7HSP7YMRJET2TTJQFAQZM7KGDKJDLMYCWWY2W6SV2UVQBQ, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:18.294911214 (407h9m18.165994249s) [noise=true , address=89.149.201.147:4017 , piece_id=ZOSNSMZOBQVIKD2ZGJRAUWJEJ4JXJK452YWQQSDL77E77F6LRDZA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:18.402191355 (407h9m18.2732744s) [noise=true , address=89.149.201.147:4011 , piece_id=ATMM77IGBDSRHNJ5FQ2VKP7W27JUA2F6HYJYSQ62IL5JQN5PF5OA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:22.436508909 (407h9m22.307591944s) [noise=true , address=81.171.14.70:4037 , piece_id=NYGC2ZLUSBUAGFULQ46LXCVNQUJ5M4XT3YICVA2ZMBR2D6WPJ27Q, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:25.935108368 (407h9m25.806191413s) [noise=true , address=81.171.14.70:4046 , piece_id=NRMV6FG5CZSTLUNP2EHNCK6VM2Z2UHKBLCHLKEPQ225Q5DGYEBPA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:30.904793996 (407h9m30.775877041s) [noise=true , address=81.171.14.70:4029 , piece_id=APDMOYEMDCLOREXY2BBOK73RSQVWKIXUX4SCHRVFDSM7ZLWBEQXA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:32.926010512 (407h9m32.797093567s) [noise=true , address=81.171.14.70:4044 , piece_id=QCWQO6JQRRKSG5U2KVH3Y6I3UI2F7G3DOW4CH3PQZKUVQ6Q2XOHA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:32.926036552 (407h9m32.797119587s) [seg_pos={0 0}, upload=357453]: Segment reached optimal threshold of 26 pieces. Dec 22 14:38:32.926196857 (407h9m32.797279902s) [noise=true , address=81.171.14.70:4033 , piece_id=MKNAJOFD5CIHTA55XO476LOJJGIWSA7HBQFQND2YNM562W74IKJA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BnjusiXEfHYwTDL8soMkm8GYnhbkwy2oCAxVA7oiDyhFXozZ8): protocol: expected piece hash; context canceled; EOF Dec 22 14:38:32.926203058 (407h9m32.797286093s) [noise=true , address=81.171.14.70:4033 , piece_id=MKNAJOFD5CIHTA55XO476LOJJGIWSA7HBQFQND2YNM562W74IKJA, seg_pos={0 0}, upload=357453]: Long tail cancellation due to optimal threshold Dec 22 14:38:32.926237574 (407h9m32.797320609s) [noise=true , address=81.171.14.70:4031 , piece_id=7MD5N7PPKFPCFMOIXAGI3MPBDDX45GTLMA3CFGNM7BBGI6TSL6RA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12drdYnnWsuMyFo6LkcewAGGWnorRCAhxPpNqK41LvGy2zkBvng): protocol: expected piece hash; context canceled; EOF Dec 22 14:38:32.926250028 (407h9m32.797333063s) [noise=true , address=81.171.14.70:4031 , piece_id=7MD5N7PPKFPCFMOIXAGI3MPBDDX45GTLMA3CFGNM7BBGI6TSL6RA, seg_pos={0 0}, upload=357453]: Long tail cancellation due to optimal threshold Dec 22 14:38:32.926299612 (407h9m32.797382647s) [seg_pos={0 0}, upload=357453]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 22 14:38:32.926403941 (407h9m32.797486986s) [upload=357453]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 14:38:32.926409692 (407h9m32.797492737s) [upload=357453]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 14:39:04.793379844 (407h10m4.664462889s) [upload=357453]: Done uploading segments... Dec 22 14:39:04.793413608 (407h10m4.664496643s) [upload=357453]: Upload finished. err: <nil> Duration: 5.855135362s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [...............................X.....XX.....XX.XXXXX.X.X.X.X] duration: 30.002375994s trace: https://jaeger.dev.storj.io/trace/4215f1725d2afb7e FAIL (2025-12-22 15:06:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 22 15:05:40.778299167 (407h36m40.649382212s) [upload=357851]: Starting upload Dec 22 15:05:40.778355864 (407h36m40.649438899s) [upload=357851]: Done starting upload Dec 22 15:05:40.778565904 (407h36m40.649648939s) [upload=357851]: Uploading segments... Dec 22 15:05:40.778616681 (407h36m40.649699716s) [upload=357851]: Got next segment. Inline: false Dec 22 15:05:40.778644894 (407h36m40.649727929s) [upload=357851]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 15:05:40.778650025 (407h36m40.64973306s) [upload=357851]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 15:05:41.025784894 (407h36m40.896867939s) [seg_pos={0 0}, upload=357851]: Begin upload segment... Dec 22 15:05:41.025884273 (407h36m40.896967308s) [seg_pos={0 0}, upload=357851]: Done begin upload segment. Dec 22 15:05:41.025902427 (407h36m40.896985472s) [noise=true , address=145.79.227.46:4602 , piece_id=E4AGL2E35ZBXF74PRNLBJZKK7JRUGQUO3FQIA6VGJ5GU2WVJKVIQ, seg_pos={0 0}, upload=357851]: Uploading piece... Dec 22 15:05:41.025915423 (407h36m40.896998458s) [noise=true , address=145.79.227.43:4158 , piece_id=3TFLGIKPQOCIKG2JTAN55CXZZG3ZPFUQB4JHEH7ZQN6ES36MJK5Q, seg_pos={0 0}, upload=357851]: Uploading piece... Dec 22 15:05:41.025984284 (407h36m40.897067329s) [noise=true , address=145.79.227.9:4367 , piece_id=XU3MJLVT2P2OWRRVCGRCGAZOHGA7DY27ODY6CDPTUAG3LX7XL7EQ, seg_pos={0 0}, upload=357851]: Uploading piece... Dec 22 15:05:41.026733410 (407h36m40.897816455s) [noise=true , address=145.79.227.49:4339 , piece_id=7C7P5H6T37TC2A5L4A2MYAMOQBDAWKHM6SIF6IMJS45QEO4455ZQ, seg_pos={0 0}, upload=357851]: Uploading piece... Dec 22 15:05:41.027093897 (407h36m40.898176942s) [noise=true , address=145.79.227.36:4518 , piece_id=CF7Z2P56OQFBLA5ZCTNU37I4XRV7X6MU2CZPGQTUNVNVRKZ3RH4Q, seg_pos={0 0}, upload=357851]: Uploading piece... Dec 22 15:05:41.027684881 (407h36m40.898767917s) [noise=true , address=145.79.227.48:4301 , piece_id=JRTPGIJK5WHS4J734Y6NKMDHF4ORMM77RV5NJ2ZT2T7YGJJ7PFYQ, seg_pos={0 0}, upload=357851]: Uploading piece... Dec 22 15:05:41.028223557 (407h36m40.899306602s) [noise=true , address=145.79.227.14:4837 , piece_id=BOTK2DAFCSHYG7REWZLRL2RQGUYWKK24TNSKUV4QUWRKFY365NHQ, seg_pos={0 0}, upload=357851]: Uploading piece... Dec 22 15:05:41.028239117 (407h36m40.899322162s) [noise=true , address=45.91.116.5:4297 , piece_id=TDSU7WDKZAAKE7MR5DWSZKCUFGRPOZX56QTHQWRTVD2UA6XA422A, seg_pos={0 0}, upload=357851]: Uploading piece... Dec 22 15:05:41.028817659 (407h36m40.899900704s) [noise=true , address=145.79.227.10:4401 , piece_id=IUNU5DBALANIF4Y5PAF4REQQ322SYQ27DGKVZ7YTNN55NWJMQ4NA, seg_pos={0 0}, upload=357851]: Uploading piece... Dec 22 15:05:41.029203123 (407h36m40.900286168s) [noise=true , address=77.90.31.17:4773 , piece_id=5JFT4JHR2RRM4GRF2YXVBPQYPF6CIRZUZJM4OP6T5THY5NFDWFDQ, seg_pos={0 0}, upload=357851]: Uploading piece... Dec 22 15:05:41.029566325 (407h36m40.90064937s) [noise=true , address=77.90.31.14:4664 , piece_id=BCAMBGOM6VDJ4C4TTIP5LWA4PZOEGZTDSU47CREJ5SEABWOV23CQ, seg_pos={0 0}, upload=357851]: Uploading piece... Dec 22 15:05:41.030056558 (407h36m40.901139593s) [noise=true , address=145.79.227.50:4375 , piece_id=FOPIAX3G4E53TSJ42ZINSMWEN7CD4VJKPFLMDIV5FFJ6L46YI5WQ, seg_pos={0 0}, upload=357851]: Uploading piece... Dec 22 15:05:41.030477119 (407h36m40.901560164s) [noise=true , address=145.79.227.51:4411 , piece_id=5BNA256FKBEWA5WKK53IJLBFLEFCHIJ22JCJTL4MDZR4NWIAHWDA, seg_pos={0 0}, upload=357851]: Uploading piece... Dec 22 15:05:41.030853255 (407h36m40.9019363s) [noise=true , address=145.79.227.38:4979 , piece_id=QGZEDPTMVWO6ZAQJUPLRMJJVRGSZ3EL6GGLLAS4UFJFNOIZDL7BA, seg_pos={0 0}, upload=357851]: Uploading piece... Dec 22 15:05:41.031210205 (407h36m40.90229325s) [noise=true , address=77.90.31.10:4518 , piece_id=OENWB37GGDAGKOJVH6DOSLWOZ6PW4BKDVVXUHPSBJQE73NVKWT7Q, seg_pos={0 0}, upload=357851]: Uploading piece... Dec 22 15:05:41.031731658 (407h36m40.902814703s) [noise=true , address=145.79.227.42:4123 , piece_id=X42HC2U6ANMTUVHTRMFZLKFA7IV6DUOVTPF7QOLICTV5SRMFWFAQ, seg_pos={0 0}, upload=357851]: Uploading piece... Dec 22 15:05:42.916893217 (407h36m42.787976252s) [upload=357851]: Next returned nil segment Dec 22 15:05:42.916907264 (407h36m42.78799031s) [upload=357851]: Waiting for error group managing segments... Dec 22 15:05:43.237594990 (407h36m43.108678035s) [noise=true , address=77.90.31.10:4518 , piece_id=OENWB37GGDAGKOJVH6DOSLWOZ6PW4BKDVVXUHPSBJQE73NVKWT7Q, seg_pos={0 0}, upload=357851]: Done uploading piece. err: <nil> Dec 22 15:05:43.276023785 (407h36m43.14710682s) [noise=true , address=77.90.31.17:4773 , piece_id=5JFT4JHR2RRM4GRF2YXVBPQYPF6CIRZUZJM4OP6T5THY5NFDWFDQ, seg_pos={0 0}, upload=357851]: Done uploading piece. err: <nil> Dec 22 15:05:43.306881634 (407h36m43.177964669s) [noise=true , address=45.91.116.5:4297 , piece_id=TDSU7WDKZAAKE7MR5DWSZKCUFGRPOZX56QTHQWRTVD2UA6XA422A, seg_pos={0 0}, upload=357851]: Done uploading piece. err: <nil> Dec 22 15:05:43.425754679 (407h36m43.296837714s) [noise=true , address=77.90.31.14:4664 , piece_id=BCAMBGOM6VDJ4C4TTIP5LWA4PZOEGZTDSU47CREJ5SEABWOV23CQ, seg_pos={0 0}, upload=357851]: Done uploading piece. err: <nil> Dec 22 15:05:43.860445157 (407h36m43.731528192s) [noise=true , address=145.79.227.14:4837 , piece_id=BOTK2DAFCSHYG7REWZLRL2RQGUYWKK24TNSKUV4QUWRKFY365NHQ, seg_pos={0 0}, upload=357851]: Done uploading piece. err: <nil> Dec 22 15:05:43.861690228 (407h36m43.732773273s) [noise=true , address=145.79.227.49:4339 , piece_id=7C7P5H6T37TC2A5L4A2MYAMOQBDAWKHM6SIF6IMJS45QEO4455ZQ, seg_pos={0 0}, upload=357851]: Done uploading piece. err: <nil> Dec 22 15:05:43.863039637 (407h36m43.734122683s) [noise=true , address=145.79.227.42:4123 , piece_id=X42HC2U6ANMTUVHTRMFZLKFA7IV6DUOVTPF7QOLICTV5SRMFWFAQ, seg_pos={0 0}, upload=357851]: Done uploading piece. err: <nil> Dec 22 15:05:43.995734492 (407h36m43.866817537s) [noise=true , address=145.79.227.36:4518 , piece_id=CF7Z2P56OQFBLA5ZCTNU37I4XRV7X6MU2CZPGQTUNVNVRKZ3RH4Q, seg_pos={0 0}, upload=357851]: Done uploading piece. err: <nil> Dec 22 15:05:43.996052106 (407h36m43.867135141s) [noise=true , address=145.79.227.46:4602 , piece_id=E4AGL2E35ZBXF74PRNLBJZKK7JRUGQUO3FQIA6VGJ5GU2WVJKVIQ, seg_pos={0 0}, upload=357851]: Done uploading piece. err: <nil> Dec 22 15:05:43.997917859 (407h36m43.869000894s) [noise=true , address=145.79.227.43:4158 , piece_id=3TFLGIKPQOCIKG2JTAN55CXZZG3ZPFUQB4JHEH7ZQN6ES36MJK5Q, seg_pos={0 0}, upload=357851]: Done uploading piece. err: <nil> Dec 22 15:05:44.002609493 (407h36m43.873692538s) [noise=true , address=145.79.227.51:4411 , piece_id=5BNA256FKBEWA5WKK53IJLBFLEFCHIJ22JCJTL4MDZR4NWIAHWDA, seg_pos={0 0}, upload=357851]: Done uploading piece. err: <nil> Dec 22 15:05:44.034007110 (407h36m43.905090145s) [noise=true , address=145.79.227.48:4301 , piece_id=JRTPGIJK5WHS4J734Y6NKMDHF4ORMM77RV5NJ2ZT2T7YGJJ7PFYQ, seg_pos={0 0}, upload=357851]: Done uploading piece. err: <nil> Dec 22 15:05:44.054685856 (407h36m43.925768891s) [noise=true , address=145.79.227.9:4367 , piece_id=XU3MJLVT2P2OWRRVCGRCGAZOHGA7DY27ODY6CDPTUAG3LX7XL7EQ, seg_pos={0 0}, upload=357851]: Done uploading piece. err: <nil> Dec 22 15:05:44.054707708 (407h36m43.925790743s) [seg_pos={0 0}, upload=357851]: Segment reached optimal threshold of 13 pieces. Dec 22 15:05:44.054926594 (407h36m43.926009629s) [noise=true , address=145.79.227.38:4979 , piece_id=QGZEDPTMVWO6ZAQJUPLRMJJVRGSZ3EL6GGLLAS4UFJFNOIZDL7BA, seg_pos={0 0}, upload=357851]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma): protocol: expected piece hash; context canceled; EOF Dec 22 15:05:44.054934840 (407h36m43.926017875s) [noise=true , address=145.79.227.38:4979 , piece_id=QGZEDPTMVWO6ZAQJUPLRMJJVRGSZ3EL6GGLLAS4UFJFNOIZDL7BA, seg_pos={0 0}, upload=357851]: Long tail cancellation due to optimal threshold Dec 22 15:05:44.055022687 (407h36m43.926105722s) [noise=true , address=145.79.227.10:4401 , piece_id=IUNU5DBALANIF4Y5PAF4REQQ322SYQ27DGKVZ7YTNN55NWJMQ4NA, seg_pos={0 0}, upload=357851]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1): protocol: expected piece hash; context canceled; EOF Dec 22 15:05:44.055045450 (407h36m43.926128485s) [noise=true , address=145.79.227.10:4401 , piece_id=IUNU5DBALANIF4Y5PAF4REQQ322SYQ27DGKVZ7YTNN55NWJMQ4NA, seg_pos={0 0}, upload=357851]: Long tail cancellation due to optimal threshold Dec 22 15:05:44.055194975 (407h36m43.92627801s) [noise=true , address=145.79.227.50:4375 , piece_id=FOPIAX3G4E53TSJ42ZINSMWEN7CD4VJKPFLMDIV5FFJ6L46YI5WQ, seg_pos={0 0}, upload=357851]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121fzX9v4rgEd78xnDAwcQtthyQHV3yUZUQYLXZ9byPDucGK4T5): protocol: expected piece hash; context canceled; EOF Dec 22 15:05:44.055202049 (407h36m43.926285084s) [noise=true , address=145.79.227.50:4375 , piece_id=FOPIAX3G4E53TSJ42ZINSMWEN7CD4VJKPFLMDIV5FFJ6L46YI5WQ, seg_pos={0 0}, upload=357851]: Long tail cancellation due to optimal threshold Dec 22 15:05:44.055221766 (407h36m43.926304801s) [seg_pos={0 0}, upload=357851]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 15:05:44.055301628 (407h36m43.926384664s) [upload=357851]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 15:05:44.055309553 (407h36m43.926392588s) [upload=357851]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 15:05:44.341529647 (407h36m44.212612682s) [upload=357851]: Done uploading segments... Dec 22 15:05:44.341549144 (407h36m44.212632179s) [upload=357851]: Upload finished. err: <nil> Duration: 30.002398747s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 10.338567218s trace: https://jaeger.dev.storj.io/trace/718f722b1bda7fe2 FAIL (2025-12-22 15:06:10 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== VMAC4CKPNLATBLZLKAGDGZHVT7BHEPN3Q, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.367093115 (407h9m14.23817616s) [noise=true , address=185.167.204.206:4043 , piece_id=HL32YEZF67X2UT6SRMXJVJ4W5A3PC5SXOCM7BYQVMEC76TKEVJRA, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.367396732 (407h9m14.238479767s) [noise=true , address=185.167.204.205:4032 , piece_id=FJMXJG2PQ4WZDSDCPJ6LLSVQKI7JJHUCDUZLU2BJZV7RZTLETM5A, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.367549273 (407h9m14.238632308s) [noise=true , address=185.167.204.206:4070 , piece_id=Y7IRY4E2GCF7TCFOIQJ7QH6IFEJBMW4WM6QBIS5SCUJ7E7ASV4HA, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.367749263 (407h9m14.238832308s) [noise=true , address=81.171.14.70:4046 , piece_id=NRMV6FG5CZSTLUNP2EHNCK6VM2Z2UHKBLCHLKEPQ225Q5DGYEBPA, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.368034046 (407h9m14.239117091s) [noise=true , address=185.167.204.206:4060 , piece_id=WN6T5TUBVMURPSBBXGXCTLPZZ74PFNSRTTWXI2JPUTY74DPH3YSQ, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.368552724 (407h9m14.239635759s) [noise=true , address=81.171.14.70:4029 , piece_id=APDMOYEMDCLOREXY2BBOK73RSQVWKIXUX4SCHRVFDSM7ZLWBEQXA, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.369160691 (407h9m14.240243736s) [noise=true , address=185.167.204.206:4057 , piece_id=ONW7YS5JG44B2STNFJGJWFEZ3WYXO74N3CXI3C6T5OTJYHGQPQZQ, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.369714496 (407h9m14.240797541s) [noise=true , address=185.167.204.206:4071 , piece_id=DVJAKIMPAUGWQPHDPGTXASQDONAKOZIGX5PQDBGBXNQAPMBPUFBA, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.370249614 (407h9m14.241332649s) [noise=true , address=89.149.201.147:4019 , piece_id=ILE7ZX5X6HBPUVYGCOQ5SNG3AA6B3YTXIF47CPLJ3MN5EXVOS2OA, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.372019455 (407h9m14.24310249s) [noise=true , address=89.149.201.147:4015 , piece_id=ZIWRTD7HSP7YMRJET2TTJQFAQZM7KGDKJDLMYCWWY2W6SV2UVQBQ, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:15.199398795 (407h9m15.07048184s) [upload=357453]: Next returned nil segment Dec 22 14:38:15.199414966 (407h9m15.070498011s) [upload=357453]: Waiting for error group managing segments... Dec 22 14:38:15.591187572 (407h9m15.462270607s) [noise=true , address=185.167.204.205:4009 , piece_id=JTHX2O6T5HRENGEMAFJUWQUEUYAIG2KQ5HDEHF4BK42IT2ER7EIA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.591766795 (407h9m15.46284984s) [noise=true , address=185.167.204.205:4006 , piece_id=XHNFS6G7EX4BQWKOK4JUWYV3OTVJ36A6VOP4N7G2XPZKNNCFFKXA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.592074521 (407h9m15.463157566s) [noise=true , address=185.167.204.206:4060 , piece_id=WN6T5TUBVMURPSBBXGXCTLPZZ74PFNSRTTWXI2JPUTY74DPH3YSQ, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.593304694 (407h9m15.464387729s) [noise=true , address=185.167.204.205:4036 , piece_id=5QWJTTL6TZIZSSYHCBEJBKGX7BS76VVTELWTKEAKTOVYKINXVWDQ, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.593638659 (407h9m15.464721694s) [noise=true , address=185.167.204.205:4004 , piece_id=PYGDQZMAKOVJWB7AWEPFMPMB6C24FMS4HT2UP2AKZNYGG4DGZR3Q, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.593818251 (407h9m15.464901286s) [noise=true , address=185.167.204.206:4048 , piece_id=52WC3XH2K2BHG33BQPRENPNYDT7EOKCNBCTZKE3LIONDJHVATEYA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.594252549 (407h9m15.465335594s) [noise=true , address=185.167.204.205:4034 , piece_id=OS5H57TG2REFE535DK4T6N75VNIXPLQWW6DM55QCC2S7QBVZM3UQ, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.597113216 (407h9m15.468196261s) [noise=true , address=185.167.204.205:4032 , piece_id=FJMXJG2PQ4WZDSDCPJ6LLSVQKI7JJHUCDUZLU2BJZV7RZTLETM5A, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.597394541 (407h9m15.468477586s) [noise=true , address=185.167.204.205:4035 , piece_id=3D7T4SGPAJQZCA74CUTR4GJRE23R4JD5KBNW3BLEASSP4GSR5PHQ, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.597612756 (407h9m15.468695792s) [noise=true , address=185.167.204.206:4043 , piece_id=HL32YEZF67X2UT6SRMXJVJ4W5A3PC5SXOCM7BYQVMEC76TKEVJRA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.598018900 (407h9m15.469101945s) [noise=true , address=185.167.204.205:4012 , piece_id=A5G37BVL2GZ7LAY3FZWIZV4R4M7M6UPYDYACJFRD5HKDWWKB4Q6A, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.598254719 (407h9m15.469337765s) [noise=true , address=185.167.204.206:4052 , piece_id=V2DVAAXN5A4C3NQW556VMAC4CKPNLATBLZLKAGDGZHVT7BHEPN3Q, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.598480649 (407h9m15.469563694s) [noise=true , address=185.167.204.205:4018 , piece_id=NUYRZ7PXKJQ3PIR44O3E6N6COZZPRDNSMTQCFI5RN7FBRV2CBALA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.598968278 (407h9m15.470051323s) [noise=true , address=185.167.204.205:4033 , piece_id=A32ZTBDBWMCNBMZPD4DUWS5NKUKQ22ACQH6HAW5AKB3FNHBC56VA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.599287886 (407h9m15.470370931s) [noise=true , address=185.167.204.206:4072 , piece_id=I6PCSC5JYTJXLT6B34TTIVZQVX52R7TRHSRLRP4ZHG5J63XDZN6Q, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.599632252 (407h9m15.470715297s) [noise=true , address=185.167.204.206:4070 , piece_id=Y7IRY4E2GCF7TCFOIQJ7QH6IFEJBMW4WM6QBIS5SCUJ7E7ASV4HA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.599676297 (407h9m15.470759332s) [noise=true , address=185.167.204.206:4071 , piece_id=DVJAKIMPAUGWQPHDPGTXASQDONAKOZIGX5PQDBGBXNQAPMBPUFBA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.601435877 (407h9m15.472518912s) [noise=true , address=185.167.204.206:4057 , piece_id=ONW7YS5JG44B2STNFJGJWFEZ3WYXO74N3CXI3C6T5OTJYHGQPQZQ, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:17.299689191 (407h9m17.170772226s) [noise=true , address=89.149.201.147:4019 , piece_id=ILE7ZX5X6HBPUVYGCOQ5SNG3AA6B3YTXIF47CPLJ3MN5EXVOS2OA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:17.526601953 (407h9m17.397684998s) [noise=true , address=89.149.201.147:4015 , piece_id=ZIWRTD7HSP7YMRJET2TTJQFAQZM7KGDKJDLMYCWWY2W6SV2UVQBQ, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:18.294911214 (407h9m18.165994249s) [noise=true , address=89.149.201.147:4017 , piece_id=ZOSNSMZOBQVIKD2ZGJRAUWJEJ4JXJK452YWQQSDL77E77F6LRDZA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:18.402191355 (407h9m18.2732744s) [noise=true , address=89.149.201.147:4011 , piece_id=ATMM77IGBDSRHNJ5FQ2VKP7W27JUA2F6HYJYSQ62IL5JQN5PF5OA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:22.436508909 (407h9m22.307591944s) [noise=true , address=81.171.14.70:4037 , piece_id=NYGC2ZLUSBUAGFULQ46LXCVNQUJ5M4XT3YICVA2ZMBR2D6WPJ27Q, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:25.935108368 (407h9m25.806191413s) [noise=true , address=81.171.14.70:4046 , piece_id=NRMV6FG5CZSTLUNP2EHNCK6VM2Z2UHKBLCHLKEPQ225Q5DGYEBPA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:30.904793996 (407h9m30.775877041s) [noise=true , address=81.171.14.70:4029 , piece_id=APDMOYEMDCLOREXY2BBOK73RSQVWKIXUX4SCHRVFDSM7ZLWBEQXA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:32.926010512 (407h9m32.797093567s) [noise=true , address=81.171.14.70:4044 , piece_id=QCWQO6JQRRKSG5U2KVH3Y6I3UI2F7G3DOW4CH3PQZKUVQ6Q2XOHA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:32.926036552 (407h9m32.797119587s) [seg_pos={0 0}, upload=357453]: Segment reached optimal threshold of 26 pieces. Dec 22 14:38:32.926196857 (407h9m32.797279902s) [noise=true , address=81.171.14.70:4033 , piece_id=MKNAJOFD5CIHTA55XO476LOJJGIWSA7HBQFQND2YNM562W74IKJA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BnjusiXEfHYwTDL8soMkm8GYnhbkwy2oCAxVA7oiDyhFXozZ8): protocol: expected piece hash; context canceled; EOF Dec 22 14:38:32.926203058 (407h9m32.797286093s) [noise=true , address=81.171.14.70:4033 , piece_id=MKNAJOFD5CIHTA55XO476LOJJGIWSA7HBQFQND2YNM562W74IKJA, seg_pos={0 0}, upload=357453]: Long tail cancellation due to optimal threshold Dec 22 14:38:32.926237574 (407h9m32.797320609s) [noise=true , address=81.171.14.70:4031 , piece_id=7MD5N7PPKFPCFMOIXAGI3MPBDDX45GTLMA3CFGNM7BBGI6TSL6RA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12drdYnnWsuMyFo6LkcewAGGWnorRCAhxPpNqK41LvGy2zkBvng): protocol: expected piece hash; context canceled; EOF Dec 22 14:38:32.926250028 (407h9m32.797333063s) [noise=true , address=81.171.14.70:4031 , piece_id=7MD5N7PPKFPCFMOIXAGI3MPBDDX45GTLMA3CFGNM7BBGI6TSL6RA, seg_pos={0 0}, upload=357453]: Long tail cancellation due to optimal threshold Dec 22 14:38:32.926299612 (407h9m32.797382647s) [seg_pos={0 0}, upload=357453]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 22 14:38:32.926403941 (407h9m32.797486986s) [upload=357453]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 14:38:32.926409692 (407h9m32.797492737s) [upload=357453]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 14:39:04.793379844 (407h10m4.664462889s) [upload=357453]: Done uploading segments... Dec 22 14:39:04.793413608 (407h10m4.664496643s) [upload=357453]: Upload finished. err: <nil> Duration: 10.338597726s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.X........................X.................................] duration: 5.713083461s trace: https://jaeger.dev.storj.io/trace/510cfc1b6c68e873 FAIL (2025-12-22 15:05:53 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.713112847s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 1.46500572s trace: https://jaeger.dev.storj.io/trace/62b55ec615ab6123 FAIL (2025-12-22 15:05:41 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.465036529s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......................................................X....] duration: 3.550418903s trace: https://jaeger.dev.storj.io/trace/2e095cd265881ac3 FAIL (2025-12-22 15:05:40 UTC) orbiter warning: context deadline exceeded Duration: 3.550481153s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 7.314417072s trace: https://jaeger.dev.storj.io/trace/718f722b1bda7fe2 FAIL (2025-12-22 15:05:15 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== VMAC4CKPNLATBLZLKAGDGZHVT7BHEPN3Q, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.367093115 (407h9m14.23817616s) [noise=true , address=185.167.204.206:4043 , piece_id=HL32YEZF67X2UT6SRMXJVJ4W5A3PC5SXOCM7BYQVMEC76TKEVJRA, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.367396732 (407h9m14.238479767s) [noise=true , address=185.167.204.205:4032 , piece_id=FJMXJG2PQ4WZDSDCPJ6LLSVQKI7JJHUCDUZLU2BJZV7RZTLETM5A, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.367549273 (407h9m14.238632308s) [noise=true , address=185.167.204.206:4070 , piece_id=Y7IRY4E2GCF7TCFOIQJ7QH6IFEJBMW4WM6QBIS5SCUJ7E7ASV4HA, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.367749263 (407h9m14.238832308s) [noise=true , address=81.171.14.70:4046 , piece_id=NRMV6FG5CZSTLUNP2EHNCK6VM2Z2UHKBLCHLKEPQ225Q5DGYEBPA, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.368034046 (407h9m14.239117091s) [noise=true , address=185.167.204.206:4060 , piece_id=WN6T5TUBVMURPSBBXGXCTLPZZ74PFNSRTTWXI2JPUTY74DPH3YSQ, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.368552724 (407h9m14.239635759s) [noise=true , address=81.171.14.70:4029 , piece_id=APDMOYEMDCLOREXY2BBOK73RSQVWKIXUX4SCHRVFDSM7ZLWBEQXA, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.369160691 (407h9m14.240243736s) [noise=true , address=185.167.204.206:4057 , piece_id=ONW7YS5JG44B2STNFJGJWFEZ3WYXO74N3CXI3C6T5OTJYHGQPQZQ, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.369714496 (407h9m14.240797541s) [noise=true , address=185.167.204.206:4071 , piece_id=DVJAKIMPAUGWQPHDPGTXASQDONAKOZIGX5PQDBGBXNQAPMBPUFBA, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.370249614 (407h9m14.241332649s) [noise=true , address=89.149.201.147:4019 , piece_id=ILE7ZX5X6HBPUVYGCOQ5SNG3AA6B3YTXIF47CPLJ3MN5EXVOS2OA, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.372019455 (407h9m14.24310249s) [noise=true , address=89.149.201.147:4015 , piece_id=ZIWRTD7HSP7YMRJET2TTJQFAQZM7KGDKJDLMYCWWY2W6SV2UVQBQ, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:15.199398795 (407h9m15.07048184s) [upload=357453]: Next returned nil segment Dec 22 14:38:15.199414966 (407h9m15.070498011s) [upload=357453]: Waiting for error group managing segments... Dec 22 14:38:15.591187572 (407h9m15.462270607s) [noise=true , address=185.167.204.205:4009 , piece_id=JTHX2O6T5HRENGEMAFJUWQUEUYAIG2KQ5HDEHF4BK42IT2ER7EIA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.591766795 (407h9m15.46284984s) [noise=true , address=185.167.204.205:4006 , piece_id=XHNFS6G7EX4BQWKOK4JUWYV3OTVJ36A6VOP4N7G2XPZKNNCFFKXA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.592074521 (407h9m15.463157566s) [noise=true , address=185.167.204.206:4060 , piece_id=WN6T5TUBVMURPSBBXGXCTLPZZ74PFNSRTTWXI2JPUTY74DPH3YSQ, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.593304694 (407h9m15.464387729s) [noise=true , address=185.167.204.205:4036 , piece_id=5QWJTTL6TZIZSSYHCBEJBKGX7BS76VVTELWTKEAKTOVYKINXVWDQ, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.593638659 (407h9m15.464721694s) [noise=true , address=185.167.204.205:4004 , piece_id=PYGDQZMAKOVJWB7AWEPFMPMB6C24FMS4HT2UP2AKZNYGG4DGZR3Q, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.593818251 (407h9m15.464901286s) [noise=true , address=185.167.204.206:4048 , piece_id=52WC3XH2K2BHG33BQPRENPNYDT7EOKCNBCTZKE3LIONDJHVATEYA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.594252549 (407h9m15.465335594s) [noise=true , address=185.167.204.205:4034 , piece_id=OS5H57TG2REFE535DK4T6N75VNIXPLQWW6DM55QCC2S7QBVZM3UQ, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.597113216 (407h9m15.468196261s) [noise=true , address=185.167.204.205:4032 , piece_id=FJMXJG2PQ4WZDSDCPJ6LLSVQKI7JJHUCDUZLU2BJZV7RZTLETM5A, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.597394541 (407h9m15.468477586s) [noise=true , address=185.167.204.205:4035 , piece_id=3D7T4SGPAJQZCA74CUTR4GJRE23R4JD5KBNW3BLEASSP4GSR5PHQ, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.597612756 (407h9m15.468695792s) [noise=true , address=185.167.204.206:4043 , piece_id=HL32YEZF67X2UT6SRMXJVJ4W5A3PC5SXOCM7BYQVMEC76TKEVJRA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.598018900 (407h9m15.469101945s) [noise=true , address=185.167.204.205:4012 , piece_id=A5G37BVL2GZ7LAY3FZWIZV4R4M7M6UPYDYACJFRD5HKDWWKB4Q6A, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.598254719 (407h9m15.469337765s) [noise=true , address=185.167.204.206:4052 , piece_id=V2DVAAXN5A4C3NQW556VMAC4CKPNLATBLZLKAGDGZHVT7BHEPN3Q, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.598480649 (407h9m15.469563694s) [noise=true , address=185.167.204.205:4018 , piece_id=NUYRZ7PXKJQ3PIR44O3E6N6COZZPRDNSMTQCFI5RN7FBRV2CBALA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.598968278 (407h9m15.470051323s) [noise=true , address=185.167.204.205:4033 , piece_id=A32ZTBDBWMCNBMZPD4DUWS5NKUKQ22ACQH6HAW5AKB3FNHBC56VA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.599287886 (407h9m15.470370931s) [noise=true , address=185.167.204.206:4072 , piece_id=I6PCSC5JYTJXLT6B34TTIVZQVX52R7TRHSRLRP4ZHG5J63XDZN6Q, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.599632252 (407h9m15.470715297s) [noise=true , address=185.167.204.206:4070 , piece_id=Y7IRY4E2GCF7TCFOIQJ7QH6IFEJBMW4WM6QBIS5SCUJ7E7ASV4HA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.599676297 (407h9m15.470759332s) [noise=true , address=185.167.204.206:4071 , piece_id=DVJAKIMPAUGWQPHDPGTXASQDONAKOZIGX5PQDBGBXNQAPMBPUFBA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.601435877 (407h9m15.472518912s) [noise=true , address=185.167.204.206:4057 , piece_id=ONW7YS5JG44B2STNFJGJWFEZ3WYXO74N3CXI3C6T5OTJYHGQPQZQ, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:17.299689191 (407h9m17.170772226s) [noise=true , address=89.149.201.147:4019 , piece_id=ILE7ZX5X6HBPUVYGCOQ5SNG3AA6B3YTXIF47CPLJ3MN5EXVOS2OA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:17.526601953 (407h9m17.397684998s) [noise=true , address=89.149.201.147:4015 , piece_id=ZIWRTD7HSP7YMRJET2TTJQFAQZM7KGDKJDLMYCWWY2W6SV2UVQBQ, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:18.294911214 (407h9m18.165994249s) [noise=true , address=89.149.201.147:4017 , piece_id=ZOSNSMZOBQVIKD2ZGJRAUWJEJ4JXJK452YWQQSDL77E77F6LRDZA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:18.402191355 (407h9m18.2732744s) [noise=true , address=89.149.201.147:4011 , piece_id=ATMM77IGBDSRHNJ5FQ2VKP7W27JUA2F6HYJYSQ62IL5JQN5PF5OA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:22.436508909 (407h9m22.307591944s) [noise=true , address=81.171.14.70:4037 , piece_id=NYGC2ZLUSBUAGFULQ46LXCVNQUJ5M4XT3YICVA2ZMBR2D6WPJ27Q, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:25.935108368 (407h9m25.806191413s) [noise=true , address=81.171.14.70:4046 , piece_id=NRMV6FG5CZSTLUNP2EHNCK6VM2Z2UHKBLCHLKEPQ225Q5DGYEBPA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:30.904793996 (407h9m30.775877041s) [noise=true , address=81.171.14.70:4029 , piece_id=APDMOYEMDCLOREXY2BBOK73RSQVWKIXUX4SCHRVFDSM7ZLWBEQXA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:32.926010512 (407h9m32.797093567s) [noise=true , address=81.171.14.70:4044 , piece_id=QCWQO6JQRRKSG5U2KVH3Y6I3UI2F7G3DOW4CH3PQZKUVQ6Q2XOHA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:32.926036552 (407h9m32.797119587s) [seg_pos={0 0}, upload=357453]: Segment reached optimal threshold of 26 pieces. Dec 22 14:38:32.926196857 (407h9m32.797279902s) [noise=true , address=81.171.14.70:4033 , piece_id=MKNAJOFD5CIHTA55XO476LOJJGIWSA7HBQFQND2YNM562W74IKJA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BnjusiXEfHYwTDL8soMkm8GYnhbkwy2oCAxVA7oiDyhFXozZ8): protocol: expected piece hash; context canceled; EOF Dec 22 14:38:32.926203058 (407h9m32.797286093s) [noise=true , address=81.171.14.70:4033 , piece_id=MKNAJOFD5CIHTA55XO476LOJJGIWSA7HBQFQND2YNM562W74IKJA, seg_pos={0 0}, upload=357453]: Long tail cancellation due to optimal threshold Dec 22 14:38:32.926237574 (407h9m32.797320609s) [noise=true , address=81.171.14.70:4031 , piece_id=7MD5N7PPKFPCFMOIXAGI3MPBDDX45GTLMA3CFGNM7BBGI6TSL6RA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12drdYnnWsuMyFo6LkcewAGGWnorRCAhxPpNqK41LvGy2zkBvng): protocol: expected piece hash; context canceled; EOF Dec 22 14:38:32.926250028 (407h9m32.797333063s) [noise=true , address=81.171.14.70:4031 , piece_id=7MD5N7PPKFPCFMOIXAGI3MPBDDX45GTLMA3CFGNM7BBGI6TSL6RA, seg_pos={0 0}, upload=357453]: Long tail cancellation due to optimal threshold Dec 22 14:38:32.926299612 (407h9m32.797382647s) [seg_pos={0 0}, upload=357453]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 22 14:38:32.926403941 (407h9m32.797486986s) [upload=357453]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 14:38:32.926409692 (407h9m32.797492737s) [upload=357453]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 14:39:04.793379844 (407h10m4.664462889s) [upload=357453]: Done uploading segments... Dec 22 14:39:04.793413608 (407h10m4.664496643s) [upload=357453]: Upload finished. err: <nil> Duration: 7.314447719s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..X........................X................................] duration: 5.54903439s trace: https://jaeger.dev.storj.io/trace/510cfc1b6c68e873 FAIL (2025-12-22 15:05:07 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.549063948s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 1.553352067s trace: https://jaeger.dev.storj.io/trace/62b55ec615ab6123 FAIL (2025-12-22 15:05:04 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.553374038s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [................................X.....XX.....XX.XXXXX.X.X.X.] duration: 25.704732259s trace: https://jaeger.dev.storj.io/trace/54d49dcbd8df6b3e FAIL (2025-12-22 15:04:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 22 15:03:17.014711182 (407h34m16.885794227s) [upload=357820]: Starting upload Dec 22 15:03:17.014799581 (407h34m16.885882626s) [upload=357820]: Done starting upload Dec 22 15:03:17.014892728 (407h34m16.885975763s) [upload=357820]: Uploading segments... Dec 22 15:03:17.015089282 (407h34m16.886172317s) [upload=357820]: Got next segment. Inline: false Dec 22 15:03:17.015123387 (407h34m16.886206422s) [upload=357820]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 15:03:17.015128787 (407h34m16.886211822s) [upload=357820]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 15:03:17.258400290 (407h34m17.129483335s) [seg_pos={0 0}, upload=357820]: Begin upload segment... Dec 22 15:03:17.258466977 (407h34m17.129550022s) [seg_pos={0 0}, upload=357820]: Done begin upload segment. Dec 22 15:03:17.258545145 (407h34m17.12962819s) [noise=true , address=145.79.227.38:4977 , piece_id=OJ2GNUD25P2THIXQDYBNFGK37KGKJ6E4JQROXL6KADC65OXZINXQ, seg_pos={0 0}, upload=357820]: Uploading piece... Dec 22 15:03:17.258571145 (407h34m17.1296542s) [noise=true , address=145.79.227.47:4267 , piece_id=WCXKBNCPW4RKXWONHMC3ULBAIJ3HSUB4Q6ADM27BYRRUFJWPTPRA, seg_pos={0 0}, upload=357820]: Uploading piece... Dec 22 15:03:17.259402258 (407h34m17.130485303s) [noise=true , address=145.79.227.46:4601 , piece_id=WYJ472TQ5G3OYS44R54RU5EN75KNXWAD3D47TWQY3ZSXJF6PQPLA, seg_pos={0 0}, upload=357820]: Uploading piece... Dec 22 15:03:17.259884044 (407h34m17.130967079s) [noise=true , address=145.79.227.5:4223 , piece_id=77ZX3OMV7UNJV6DQQCWCZMJJIXVTXRKLVR6VUMUKBFQ375EAGORQ, seg_pos={0 0}, upload=357820]: Uploading piece... Dec 22 15:03:17.260306980 (407h34m17.131390025s) [noise=true , address=145.79.227.32:4836 , piece_id=IOQABCPNQNS4DTD3MCDPZDJU3XLGBTVMGPZDSCMZNSQV2KVFNWVQ, seg_pos={0 0}, upload=357820]: Uploading piece... Dec 22 15:03:17.260874431 (407h34m17.131957476s) [noise=true , address=145.79.227.11:4730 , piece_id=ZYIE5LN733BEYDNTVSGK3JFLEAPZ3DFE447CLU2JOTZJVWOPQJKQ, seg_pos={0 0}, upload=357820]: Uploading piece... Dec 22 15:03:17.261325720 (407h34m17.132408765s) [noise=true , address=145.79.227.41:4087 , piece_id=J62C6BASRC7CQWP4ZBQ4SLKXCSDOXEY33E54OXL5ITUTD535P7AQ, seg_pos={0 0}, upload=357820]: Uploading piece... Dec 22 15:03:17.261784463 (407h34m17.132867508s) [noise=true , address=45.91.116.13:4155 , piece_id=VDN753NH3DLAEUK5RKIG27WHVIXPJY26BZSUCELQEALJT3DLE5DQ, seg_pos={0 0}, upload=357820]: Uploading piece... Dec 22 15:03:17.261985867 (407h34m17.133068912s) [noise=true , address=145.79.227.50:4375 , piece_id=OBW4GF2LUUPJHYO7SGQ4MUDDJHR4HO5YO6GFLWJ7GAMS67HUECKQ, seg_pos={0 0}, upload=357820]: Uploading piece... Dec 22 15:03:17.262249839 (407h34m17.133332884s) [noise=true , address=145.79.227.33:4870 , piece_id=GABHOWGAFDT6NOGGIZLCFAV5EMPUBAENF7M62MEFQAHXSQ43UAUQ, seg_pos={0 0}, upload=357820]: Uploading piece... Dec 22 15:03:17.262691731 (407h34m17.133774776s) [noise=true , address=145.79.227.36:4519 , piece_id=7ZVXI4GDQUZ6SC26LATGJBWPQMTFE7ECOL6TDUQDXFIHEMTOXKGA, seg_pos={0 0}, upload=357820]: Uploading piece... Dec 22 15:03:17.262914284 (407h34m17.133997329s) [noise=true , address=77.90.31.11:4556 , piece_id=F56P3JXSD4FO7UIFLGLD4MUHFHVFKFZ2NGV2JPWY7AQIR7ZARXUA, seg_pos={0 0}, upload=357820]: Uploading piece... Dec 22 15:03:17.263131218 (407h34m17.134214263s) [noise=true , address=77.90.31.19:4841 , piece_id=YONIOZPHTGCT2AMZTOWW5QZKRS3JC4ZGMO3M52NLZM4KBYPFDO2Q, seg_pos={0 0}, upload=357820]: Uploading piece... Dec 22 15:03:17.263577357 (407h34m17.134660402s) [noise=true , address=145.79.227.48:4301 , piece_id=3TNPWUQCRBHRL6VOXBND4IKWRUXAVDTRO4ADJ4HF4ULU23RBLKVQ, seg_pos={0 0}, upload=357820]: Uploading piece... Dec 22 15:03:17.264167681 (407h34m17.135250716s) [noise=true , address=145.79.227.17:4972 , piece_id=PQAFB42ZZMBLCCCPDNH3QSVJ5PKLN6EZT4B3N7KXAQXSH5SUAHGA, seg_pos={0 0}, upload=357820]: Uploading piece... Dec 22 15:03:17.265130865 (407h34m17.13621391s) [noise=true , address=145.79.227.51:4411 , piece_id=556BW2WBD4CRNTIYMUBX66GD2SOFPFVLBX6NYWNENQEBRVVP2HZA, seg_pos={0 0}, upload=357820]: Uploading piece... Dec 22 15:03:18.355772570 (407h34m18.226855615s) [noise=true , address=77.90.31.19:4841 , piece_id=YONIOZPHTGCT2AMZTOWW5QZKRS3JC4ZGMO3M52NLZM4KBYPFDO2Q, seg_pos={0 0}, upload=357820]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 22 15:03:19.087573949 (407h34m18.958657014s) [upload=357820]: Next returned nil segment Dec 22 15:03:19.087634214 (407h34m18.958717259s) [upload=357820]: Waiting for error group managing segments... Dec 22 15:03:19.441445984 (407h34m19.312529019s) [noise=true , address=77.90.31.11:4556 , piece_id=F56P3JXSD4FO7UIFLGLD4MUHFHVFKFZ2NGV2JPWY7AQIR7ZARXUA, seg_pos={0 0}, upload=357820]: Done uploading piece. err: <nil> Dec 22 15:03:19.686183157 (407h34m19.557266192s) [noise=true , address=45.91.116.13:4155 , piece_id=VDN753NH3DLAEUK5RKIG27WHVIXPJY26BZSUCELQEALJT3DLE5DQ, seg_pos={0 0}, upload=357820]: Done uploading piece. err: <nil> Dec 22 15:03:20.239474952 (407h34m20.110557987s) [noise=true , address=145.79.227.46:4601 , piece_id=WYJ472TQ5G3OYS44R54RU5EN75KNXWAD3D47TWQY3ZSXJF6PQPLA, seg_pos={0 0}, upload=357820]: Done uploading piece. err: <nil> Dec 22 15:03:20.247651201 (407h34m20.118734236s) [noise=true , address=145.79.227.41:4087 , piece_id=J62C6BASRC7CQWP4ZBQ4SLKXCSDOXEY33E54OXL5ITUTD535P7AQ, seg_pos={0 0}, upload=357820]: Done uploading piece. err: <nil> Dec 22 15:03:20.248052676 (407h34m20.119135721s) [noise=true , address=145.79.227.48:4301 , piece_id=3TNPWUQCRBHRL6VOXBND4IKWRUXAVDTRO4ADJ4HF4ULU23RBLKVQ, seg_pos={0 0}, upload=357820]: Done uploading piece. err: <nil> Dec 22 15:03:20.249332632 (407h34m20.120415677s) [noise=true , address=145.79.227.11:4730 , piece_id=ZYIE5LN733BEYDNTVSGK3JFLEAPZ3DFE447CLU2JOTZJVWOPQJKQ, seg_pos={0 0}, upload=357820]: Done uploading piece. err: <nil> Dec 22 15:03:20.250109913 (407h34m20.121192948s) [noise=true , address=145.79.227.51:4411 , piece_id=556BW2WBD4CRNTIYMUBX66GD2SOFPFVLBX6NYWNENQEBRVVP2HZA, seg_pos={0 0}, upload=357820]: Done uploading piece. err: <nil> Dec 22 15:03:20.252252843 (407h34m20.123335888s) [noise=true , address=145.79.227.17:4972 , piece_id=PQAFB42ZZMBLCCCPDNH3QSVJ5PKLN6EZT4B3N7KXAQXSH5SUAHGA, seg_pos={0 0}, upload=357820]: Done uploading piece. err: <nil> Dec 22 15:03:20.262125703 (407h34m20.133208748s) [noise=true , address=145.79.227.5:4223 , piece_id=77ZX3OMV7UNJV6DQQCWCZMJJIXVTXRKLVR6VUMUKBFQ375EAGORQ, seg_pos={0 0}, upload=357820]: Done uploading piece. err: <nil> Dec 22 15:03:20.279532178 (407h34m20.150615213s) [noise=true , address=145.79.227.47:4267 , piece_id=WCXKBNCPW4RKXWONHMC3ULBAIJ3HSUB4Q6ADM27BYRRUFJWPTPRA, seg_pos={0 0}, upload=357820]: Done uploading piece. err: <nil> Dec 22 15:03:20.285144105 (407h34m20.15622714s) [noise=true , address=145.79.227.38:4977 , piece_id=OJ2GNUD25P2THIXQDYBNFGK37KGKJ6E4JQROXL6KADC65OXZINXQ, seg_pos={0 0}, upload=357820]: Done uploading piece. err: <nil> Dec 22 15:03:20.286234511 (407h34m20.157317556s) [noise=true , address=145.79.227.36:4519 , piece_id=7ZVXI4GDQUZ6SC26LATGJBWPQMTFE7ECOL6TDUQDXFIHEMTOXKGA, seg_pos={0 0}, upload=357820]: Done uploading piece. err: <nil> Dec 22 15:03:20.602075918 (407h34m20.473158963s) [noise=true , address=145.79.227.33:4870 , piece_id=GABHOWGAFDT6NOGGIZLCFAV5EMPUBAENF7M62MEFQAHXSQ43UAUQ, seg_pos={0 0}, upload=357820]: Done uploading piece. err: <nil> Dec 22 15:03:20.602097139 (407h34m20.473180174s) [seg_pos={0 0}, upload=357820]: Segment reached optimal threshold of 13 pieces. Dec 22 15:03:20.602449550 (407h34m20.473532625s) [noise=true , address=145.79.227.50:4375 , piece_id=OBW4GF2LUUPJHYO7SGQ4MUDDJHR4HO5YO6GFLWJ7GAMS67HUECKQ, seg_pos={0 0}, upload=357820]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121fzX9v4rgEd78xnDAwcQtthyQHV3yUZUQYLXZ9byPDucGK4T5): protocol: rpc: dial tcp 145.79.227.50:4375: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 22 15:03:20.602461663 (407h34m20.473544698s) [noise=true , address=145.79.227.50:4375 , piece_id=OBW4GF2LUUPJHYO7SGQ4MUDDJHR4HO5YO6GFLWJ7GAMS67HUECKQ, seg_pos={0 0}, upload=357820]: Long tail cancellation due to optimal threshold Dec 22 15:03:20.603013303 (407h34m20.474096348s) [noise=true , address=145.79.227.32:4836 , piece_id=IOQABCPNQNS4DTD3MCDPZDJU3XLGBTVMGPZDSCMZNSQV2KVFNWVQ, seg_pos={0 0}, upload=357820]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR): protocol: read tcp 188.166.225.170:59548->145.79.227.32:4836: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Dec 22 15:03:20.603031799 (407h34m20.474114834s) [noise=true , address=145.79.227.32:4836 , piece_id=IOQABCPNQNS4DTD3MCDPZDJU3XLGBTVMGPZDSCMZNSQV2KVFNWVQ, seg_pos={0 0}, upload=357820]: Long tail cancellation due to optimal threshold Dec 22 15:03:20.603056475 (407h34m20.47413952s) [seg_pos={0 0}, upload=357820]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 15:03:20.603188296 (407h34m20.474271341s) [upload=357820]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 15:03:20.603196161 (407h34m20.474279196s) [upload=357820]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 15:03:21.002396249 (407h34m20.873479284s) [upload=357820]: Done uploading segments... Dec 22 15:03:21.002414944 (407h34m20.873497979s) [upload=357820]: Upload finished. err: <nil> Duration: 25.704761106s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...X........................X...............................] duration: 5.343805609s trace: https://jaeger.dev.storj.io/trace/510cfc1b6c68e873 FAIL (2025-12-22 15:04:45 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.343832131s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........................................................X...] duration: 3.475418891s trace: https://jaeger.dev.storj.io/trace/2e095cd265881ac3 FAIL (2025-12-22 15:04:37 UTC) orbiter warning: context deadline exceeded Duration: 3.475436655s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 5.876844109s trace: https://jaeger.dev.storj.io/trace/718f722b1bda7fe2 FAIL (2025-12-22 15:04:09 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== VMAC4CKPNLATBLZLKAGDGZHVT7BHEPN3Q, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.367093115 (407h9m14.23817616s) [noise=true , address=185.167.204.206:4043 , piece_id=HL32YEZF67X2UT6SRMXJVJ4W5A3PC5SXOCM7BYQVMEC76TKEVJRA, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.367396732 (407h9m14.238479767s) [noise=true , address=185.167.204.205:4032 , piece_id=FJMXJG2PQ4WZDSDCPJ6LLSVQKI7JJHUCDUZLU2BJZV7RZTLETM5A, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.367549273 (407h9m14.238632308s) [noise=true , address=185.167.204.206:4070 , piece_id=Y7IRY4E2GCF7TCFOIQJ7QH6IFEJBMW4WM6QBIS5SCUJ7E7ASV4HA, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.367749263 (407h9m14.238832308s) [noise=true , address=81.171.14.70:4046 , piece_id=NRMV6FG5CZSTLUNP2EHNCK6VM2Z2UHKBLCHLKEPQ225Q5DGYEBPA, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.368034046 (407h9m14.239117091s) [noise=true , address=185.167.204.206:4060 , piece_id=WN6T5TUBVMURPSBBXGXCTLPZZ74PFNSRTTWXI2JPUTY74DPH3YSQ, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.368552724 (407h9m14.239635759s) [noise=true , address=81.171.14.70:4029 , piece_id=APDMOYEMDCLOREXY2BBOK73RSQVWKIXUX4SCHRVFDSM7ZLWBEQXA, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.369160691 (407h9m14.240243736s) [noise=true , address=185.167.204.206:4057 , piece_id=ONW7YS5JG44B2STNFJGJWFEZ3WYXO74N3CXI3C6T5OTJYHGQPQZQ, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.369714496 (407h9m14.240797541s) [noise=true , address=185.167.204.206:4071 , piece_id=DVJAKIMPAUGWQPHDPGTXASQDONAKOZIGX5PQDBGBXNQAPMBPUFBA, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.370249614 (407h9m14.241332649s) [noise=true , address=89.149.201.147:4019 , piece_id=ILE7ZX5X6HBPUVYGCOQ5SNG3AA6B3YTXIF47CPLJ3MN5EXVOS2OA, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:14.372019455 (407h9m14.24310249s) [noise=true , address=89.149.201.147:4015 , piece_id=ZIWRTD7HSP7YMRJET2TTJQFAQZM7KGDKJDLMYCWWY2W6SV2UVQBQ, seg_pos={0 0}, upload=357453]: Uploading piece... Dec 22 14:38:15.199398795 (407h9m15.07048184s) [upload=357453]: Next returned nil segment Dec 22 14:38:15.199414966 (407h9m15.070498011s) [upload=357453]: Waiting for error group managing segments... Dec 22 14:38:15.591187572 (407h9m15.462270607s) [noise=true , address=185.167.204.205:4009 , piece_id=JTHX2O6T5HRENGEMAFJUWQUEUYAIG2KQ5HDEHF4BK42IT2ER7EIA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.591766795 (407h9m15.46284984s) [noise=true , address=185.167.204.205:4006 , piece_id=XHNFS6G7EX4BQWKOK4JUWYV3OTVJ36A6VOP4N7G2XPZKNNCFFKXA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.592074521 (407h9m15.463157566s) [noise=true , address=185.167.204.206:4060 , piece_id=WN6T5TUBVMURPSBBXGXCTLPZZ74PFNSRTTWXI2JPUTY74DPH3YSQ, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.593304694 (407h9m15.464387729s) [noise=true , address=185.167.204.205:4036 , piece_id=5QWJTTL6TZIZSSYHCBEJBKGX7BS76VVTELWTKEAKTOVYKINXVWDQ, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.593638659 (407h9m15.464721694s) [noise=true , address=185.167.204.205:4004 , piece_id=PYGDQZMAKOVJWB7AWEPFMPMB6C24FMS4HT2UP2AKZNYGG4DGZR3Q, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.593818251 (407h9m15.464901286s) [noise=true , address=185.167.204.206:4048 , piece_id=52WC3XH2K2BHG33BQPRENPNYDT7EOKCNBCTZKE3LIONDJHVATEYA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.594252549 (407h9m15.465335594s) [noise=true , address=185.167.204.205:4034 , piece_id=OS5H57TG2REFE535DK4T6N75VNIXPLQWW6DM55QCC2S7QBVZM3UQ, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.597113216 (407h9m15.468196261s) [noise=true , address=185.167.204.205:4032 , piece_id=FJMXJG2PQ4WZDSDCPJ6LLSVQKI7JJHUCDUZLU2BJZV7RZTLETM5A, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.597394541 (407h9m15.468477586s) [noise=true , address=185.167.204.205:4035 , piece_id=3D7T4SGPAJQZCA74CUTR4GJRE23R4JD5KBNW3BLEASSP4GSR5PHQ, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.597612756 (407h9m15.468695792s) [noise=true , address=185.167.204.206:4043 , piece_id=HL32YEZF67X2UT6SRMXJVJ4W5A3PC5SXOCM7BYQVMEC76TKEVJRA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.598018900 (407h9m15.469101945s) [noise=true , address=185.167.204.205:4012 , piece_id=A5G37BVL2GZ7LAY3FZWIZV4R4M7M6UPYDYACJFRD5HKDWWKB4Q6A, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.598254719 (407h9m15.469337765s) [noise=true , address=185.167.204.206:4052 , piece_id=V2DVAAXN5A4C3NQW556VMAC4CKPNLATBLZLKAGDGZHVT7BHEPN3Q, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.598480649 (407h9m15.469563694s) [noise=true , address=185.167.204.205:4018 , piece_id=NUYRZ7PXKJQ3PIR44O3E6N6COZZPRDNSMTQCFI5RN7FBRV2CBALA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.598968278 (407h9m15.470051323s) [noise=true , address=185.167.204.205:4033 , piece_id=A32ZTBDBWMCNBMZPD4DUWS5NKUKQ22ACQH6HAW5AKB3FNHBC56VA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.599287886 (407h9m15.470370931s) [noise=true , address=185.167.204.206:4072 , piece_id=I6PCSC5JYTJXLT6B34TTIVZQVX52R7TRHSRLRP4ZHG5J63XDZN6Q, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.599632252 (407h9m15.470715297s) [noise=true , address=185.167.204.206:4070 , piece_id=Y7IRY4E2GCF7TCFOIQJ7QH6IFEJBMW4WM6QBIS5SCUJ7E7ASV4HA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.599676297 (407h9m15.470759332s) [noise=true , address=185.167.204.206:4071 , piece_id=DVJAKIMPAUGWQPHDPGTXASQDONAKOZIGX5PQDBGBXNQAPMBPUFBA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:15.601435877 (407h9m15.472518912s) [noise=true , address=185.167.204.206:4057 , piece_id=ONW7YS5JG44B2STNFJGJWFEZ3WYXO74N3CXI3C6T5OTJYHGQPQZQ, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:17.299689191 (407h9m17.170772226s) [noise=true , address=89.149.201.147:4019 , piece_id=ILE7ZX5X6HBPUVYGCOQ5SNG3AA6B3YTXIF47CPLJ3MN5EXVOS2OA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:17.526601953 (407h9m17.397684998s) [noise=true , address=89.149.201.147:4015 , piece_id=ZIWRTD7HSP7YMRJET2TTJQFAQZM7KGDKJDLMYCWWY2W6SV2UVQBQ, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:18.294911214 (407h9m18.165994249s) [noise=true , address=89.149.201.147:4017 , piece_id=ZOSNSMZOBQVIKD2ZGJRAUWJEJ4JXJK452YWQQSDL77E77F6LRDZA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:18.402191355 (407h9m18.2732744s) [noise=true , address=89.149.201.147:4011 , piece_id=ATMM77IGBDSRHNJ5FQ2VKP7W27JUA2F6HYJYSQ62IL5JQN5PF5OA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:22.436508909 (407h9m22.307591944s) [noise=true , address=81.171.14.70:4037 , piece_id=NYGC2ZLUSBUAGFULQ46LXCVNQUJ5M4XT3YICVA2ZMBR2D6WPJ27Q, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:25.935108368 (407h9m25.806191413s) [noise=true , address=81.171.14.70:4046 , piece_id=NRMV6FG5CZSTLUNP2EHNCK6VM2Z2UHKBLCHLKEPQ225Q5DGYEBPA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:30.904793996 (407h9m30.775877041s) [noise=true , address=81.171.14.70:4029 , piece_id=APDMOYEMDCLOREXY2BBOK73RSQVWKIXUX4SCHRVFDSM7ZLWBEQXA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:32.926010512 (407h9m32.797093567s) [noise=true , address=81.171.14.70:4044 , piece_id=QCWQO6JQRRKSG5U2KVH3Y6I3UI2F7G3DOW4CH3PQZKUVQ6Q2XOHA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: <nil> Dec 22 14:38:32.926036552 (407h9m32.797119587s) [seg_pos={0 0}, upload=357453]: Segment reached optimal threshold of 26 pieces. Dec 22 14:38:32.926196857 (407h9m32.797279902s) [noise=true , address=81.171.14.70:4033 , piece_id=MKNAJOFD5CIHTA55XO476LOJJGIWSA7HBQFQND2YNM562W74IKJA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BnjusiXEfHYwTDL8soMkm8GYnhbkwy2oCAxVA7oiDyhFXozZ8): protocol: expected piece hash; context canceled; EOF Dec 22 14:38:32.926203058 (407h9m32.797286093s) [noise=true , address=81.171.14.70:4033 , piece_id=MKNAJOFD5CIHTA55XO476LOJJGIWSA7HBQFQND2YNM562W74IKJA, seg_pos={0 0}, upload=357453]: Long tail cancellation due to optimal threshold Dec 22 14:38:32.926237574 (407h9m32.797320609s) [noise=true , address=81.171.14.70:4031 , piece_id=7MD5N7PPKFPCFMOIXAGI3MPBDDX45GTLMA3CFGNM7BBGI6TSL6RA, seg_pos={0 0}, upload=357453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12drdYnnWsuMyFo6LkcewAGGWnorRCAhxPpNqK41LvGy2zkBvng): protocol: expected piece hash; context canceled; EOF Dec 22 14:38:32.926250028 (407h9m32.797333063s) [noise=true , address=81.171.14.70:4031 , piece_id=7MD5N7PPKFPCFMOIXAGI3MPBDDX45GTLMA3CFGNM7BBGI6TSL6RA, seg_pos={0 0}, upload=357453]: Long tail cancellation due to optimal threshold Dec 22 14:38:32.926299612 (407h9m32.797382647s) [seg_pos={0 0}, upload=357453]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 22 14:38:32.926403941 (407h9m32.797486986s) [upload=357453]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 14:38:32.926409692 (407h9m32.797492737s) [upload=357453]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 14:39:04.793379844 (407h10m4.664462889s) [upload=357453]: Done uploading segments... Dec 22 14:39:04.793413608 (407h10m4.664496643s) [upload=357453]: Upload finished. err: <nil> Duration: 5.876868284s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 1.370457165s trace: https://jaeger.dev.storj.io/trace/62b55ec615ab6123 FAIL (2025-12-22 15:03:56 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.370486991s core uplink eu1

Orbiter Check History

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)