Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [..............X................X....X............X........X.] duration: 4.980456819s trace: https://jaeger.dev.storj.io/trace/308811dfa5452ea9 FAIL (2025-12-28 02:59:55 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.980484573s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [...............X................X....X............X........X] duration: 5.001528935s trace: https://jaeger.dev.storj.io/trace/308811dfa5452ea9 FAIL (2025-12-28 02:59:29 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001541188s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 44 out of 60 is failed (threshold=3) [XX.XXX.X.....XXX.XXXXXX.XXXX.XXXX..X..XXXXX.XXXXXXXXXXXXX.XX] duration: 30.002029274s trace: https://jaeger.dev.storj.io/trace/4dc5beef3e06a02e FAIL (2025-12-28 02:59:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: IT6MVH7ZEVRHVNIFPHEZ7PXRYAEGT7N5SIAB6IDGVWZ7LTSWCCOQ) context deadline exceeded; piecestore: (Node ID: 12enbzhEQp1y2H8Fwdb5UqxcMDdgd5HYb5jYfo2U6Fnycvscaw3, Piece ID: GFYXTOFDT3E2GH5FLHH2EKTDMUINBXQTD7MBGRVMKEXFJDHPOXVA) context deadline exceeded ======LOGS====== 02:58:42.883589775 (539h29m42.75467281s) [upload=473118]: Starting upload Dec 28 02:58:42.883644339 (539h29m42.754727374s) [upload=473118]: Done starting upload Dec 28 02:58:42.883781189 (539h29m42.754864224s) [upload=473118]: Uploading segments... Dec 28 02:58:42.883892913 (539h29m42.754975948s) [upload=473118]: Got next segment. Inline: false Dec 28 02:58:42.883944340 (539h29m42.755027385s) [upload=473118]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 02:58:42.883951654 (539h29m42.755034699s) [upload=473118]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 02:58:43.133624242 (539h29m43.004707287s) [seg_pos={0 0}, upload=473118]: Begin upload segment... Dec 28 02:58:43.133688625 (539h29m43.00477166s) [seg_pos={0 0}, upload=473118]: Done begin upload segment. Dec 28 02:58:43.133734993 (539h29m43.004818028s) [noise=true , address=145.79.227.7:4293 , piece_id=F6KLM4O5T2SQAYCY7GUX4IEF5RVPHZ3A5KLG7ZQMAR26OVHZ4PUA, seg_pos={0 0}, upload=473118]: Uploading piece... Dec 28 02:58:43.133768897 (539h29m43.004851953s) [noise=true , address=145.79.227.27:4691 , piece_id=VNEPECSFEFISDSHKYNXFZ65NBTLW5QQZ7DO2NPD6YGRCAWS3FRZA, seg_pos={0 0}, upload=473118]: Uploading piece... Dec 28 02:58:43.133858187 (539h29m43.004941222s) [noise=true , address=77.90.31.16:4735 , piece_id=RLRI6MGF3RBEAUPD4PT7WNY36ZVT64RZIFNMJMWPGZMTQPGU77YQ, seg_pos={0 0}, upload=473118]: Uploading piece... Dec 28 02:58:43.134489660 (539h29m43.005572705s) [noise=true , address=77.90.31.19:4844 , piece_id=YP3Y4F6KJTBRFRJFACP7GYIZEJOF53F4LD23QHOXB25YUEBOZ5TA, seg_pos={0 0}, upload=473118]: Uploading piece... Dec 28 02:58:43.134503787 (539h29m43.005586842s) [noise=true , address=145.79.227.9:4365 , piece_id=CFLFHFBIRHBXXI6LM3JHATINZHEZAHATMV2N4E3LLPF3O4X2H6EA, seg_pos={0 0}, upload=473118]: Uploading piece... Dec 28 02:58:43.135012035 (539h29m43.00609508s) [noise=true , address=77.90.31.18:4805 , piece_id=IT6MVH7ZEVRHVNIFPHEZ7PXRYAEGT7N5SIAB6IDGVWZ7LTSWCCOQ, seg_pos={0 0}, upload=473118]: Uploading piece... Dec 28 02:58:43.135019228 (539h29m43.006102263s) [noise=true , address=145.79.227.36:4519 , piece_id=GFYXTOFDT3E2GH5FLHH2EKTDMUINBXQTD7MBGRVMKEXFJDHPOXVA, seg_pos={0 0}, upload=473118]: Uploading piece... Dec 28 02:58:43.135393952 (539h29m43.006476987s) [noise=true , address=145.79.227.48:4301 , piece_id=ZMJBADAYCAJZUB2UL3OYJVZZNNETR5UNV5PQAS3Z5JRV4QAOND2A, seg_pos={0 0}, upload=473118]: Uploading piece... Dec 28 02:58:43.135452323 (539h29m43.006535368s) [noise=true , address=77.90.31.13:4629 , piece_id=ZUYD44FGUJO7J6FJ5LDYDTUIBS7ODSCKKU3BOLJE6CSC6EGLNPOA, seg_pos={0 0}, upload=473118]: Uploading piece... Dec 28 02:58:43.135769296 (539h29m43.006852341s) [noise=true , address=145.79.227.50:4374 , piece_id=7G2BE4FBX6A473WKITKCPHXUIMNB65NVNMDA6IXVNOTHJCE25P5A, seg_pos={0 0}, upload=473118]: Uploading piece... Dec 28 02:58:43.135845442 (539h29m43.006928477s) [noise=true , address=45.91.116.7:4010 , piece_id=4EN7WJ35KB4BQPRVHHNZWXE7XCP6YUEP2QDB25UHQH5S3TCGHUPQ, seg_pos={0 0}, upload=473118]: Uploading piece... Dec 28 02:58:43.136164008 (539h29m43.007247043s) [noise=true , address=145.79.227.11:4730 , piece_id=3KDERTIRAIP2CFF2HOPBRFDUXIZIPBGJ7DMDFYUZCR5VRGIFZDSA, seg_pos={0 0}, upload=473118]: Uploading piece... Dec 28 02:58:43.136226226 (539h29m43.007309261s) [noise=true , address=145.79.227.15:4874 , piece_id=7LOIVW6JL4AGM5UGDLM75I67UKR4XK2PRN4TW54DI5LOVDEXGILQ, seg_pos={0 0}, upload=473118]: Uploading piece... Dec 28 02:58:43.136552999 (539h29m43.007636034s) [noise=true , address=145.79.227.51:4409 , piece_id=5RTTNHPT7CQDIGR7RXRFPEGGZ4DLYLA642CODSSMSTUEZPHO3KCQ, seg_pos={0 0}, upload=473118]: Uploading piece... Dec 28 02:58:43.137478771 (539h29m43.008561816s) [noise=true , address=145.79.227.22:4511 , piece_id=Z6PUA2E5NGR5CKFPDSVJOHZNKCHOWL3LH2HE6ATWQG573YO25TYA, seg_pos={0 0}, upload=473118]: Uploading piece... Dec 28 02:58:43.138102669 (539h29m43.009185714s) [noise=true , address=45.91.116.14:4192 , piece_id=RTQ2YWCFUJS3PZ6VPKY3CCC3LDL6KAKPO3DFYFPWD2VAJMHMPVQA, seg_pos={0 0}, upload=473118]: Uploading piece... Dec 28 02:58:44.936473356 (539h29m44.807556411s) [upload=473118]: Next returned nil segment Dec 28 02:58:44.936494757 (539h29m44.807577802s) [upload=473118]: Waiting for error group managing segments... Dec 28 02:58:45.272502572 (539h29m45.143585607s) [noise=true , address=145.79.227.48:4301 , piece_id=ZMJBADAYCAJZUB2UL3OYJVZZNNETR5UNV5PQAS3Z5JRV4QAOND2A, seg_pos={0 0}, upload=473118]: Done uploading piece. err: <nil> Dec 28 02:58:45.292633475 (539h29m45.16371654s) [noise=true , address=77.90.31.19:4844 , piece_id=YP3Y4F6KJTBRFRJFACP7GYIZEJOF53F4LD23QHOXB25YUEBOZ5TA, seg_pos={0 0}, upload=473118]: Done uploading piece. err: <nil> Dec 28 02:58:45.302457151 (539h29m45.173540186s) [noise=true , address=145.79.227.15:4874 , piece_id=7LOIVW6JL4AGM5UGDLM75I67UKR4XK2PRN4TW54DI5LOVDEXGILQ, seg_pos={0 0}, upload=473118]: Done uploading piece. err: <nil> Dec 28 02:58:45.307391006 (539h29m45.178474041s) [noise=true , address=145.79.227.36:4519 , piece_id=GFYXTOFDT3E2GH5FLHH2EKTDMUINBXQTD7MBGRVMKEXFJDHPOXVA, seg_pos={0 0}, upload=473118]: Done uploading piece. err: <nil> Dec 28 02:58:45.307478642 (539h29m45.178561677s) [noise=true , address=145.79.227.11:4730 , piece_id=3KDERTIRAIP2CFF2HOPBRFDUXIZIPBGJ7DMDFYUZCR5VRGIFZDSA, seg_pos={0 0}, upload=473118]: Done uploading piece. err: <nil> Dec 28 02:58:45.307773484 (539h29m45.178856519s) [noise=true , address=145.79.227.50:4374 , piece_id=7G2BE4FBX6A473WKITKCPHXUIMNB65NVNMDA6IXVNOTHJCE25P5A, seg_pos={0 0}, upload=473118]: Done uploading piece. err: <nil> Dec 28 02:58:45.308109023 (539h29m45.179192068s) [noise=true , address=145.79.227.22:4511 , piece_id=Z6PUA2E5NGR5CKFPDSVJOHZNKCHOWL3LH2HE6ATWQG573YO25TYA, seg_pos={0 0}, upload=473118]: Done uploading piece. err: <nil> Dec 28 02:58:45.322915977 (539h29m45.193999012s) [noise=true , address=145.79.227.27:4691 , piece_id=VNEPECSFEFISDSHKYNXFZ65NBTLW5QQZ7DO2NPD6YGRCAWS3FRZA, seg_pos={0 0}, upload=473118]: Done uploading piece. err: <nil> Dec 28 02:58:45.383153159 (539h29m45.254236204s) [noise=true , address=45.91.116.7:4010 , piece_id=4EN7WJ35KB4BQPRVHHNZWXE7XCP6YUEP2QDB25UHQH5S3TCGHUPQ, seg_pos={0 0}, upload=473118]: Done uploading piece. err: <nil> Dec 28 02:58:45.418864570 (539h29m45.289947605s) [noise=true , address=77.90.31.18:4805 , piece_id=IT6MVH7ZEVRHVNIFPHEZ7PXRYAEGT7N5SIAB6IDGVWZ7LTSWCCOQ, seg_pos={0 0}, upload=473118]: Done uploading piece. err: <nil> Dec 28 02:58:45.428454361 (539h29m45.299537406s) [noise=true , address=77.90.31.13:4629 , piece_id=ZUYD44FGUJO7J6FJ5LDYDTUIBS7ODSCKKU3BOLJE6CSC6EGLNPOA, seg_pos={0 0}, upload=473118]: Done uploading piece. err: <nil> Dec 28 02:58:46.033782205 (539h29m45.904865251s) [noise=true , address=145.79.227.9:4365 , piece_id=CFLFHFBIRHBXXI6LM3JHATINZHEZAHATMV2N4E3LLPF3O4X2H6EA, seg_pos={0 0}, upload=473118]: Done uploading piece. err: <nil> Dec 28 02:58:46.132517473 (539h29m46.003600518s) [noise=true , address=145.79.227.51:4409 , piece_id=5RTTNHPT7CQDIGR7RXRFPEGGZ4DLYLA642CODSSMSTUEZPHO3KCQ, seg_pos={0 0}, upload=473118]: Done uploading piece. err: <nil> Dec 28 02:58:46.132540397 (539h29m46.003623432s) [seg_pos={0 0}, upload=473118]: Segment reached optimal threshold of 13 pieces. Dec 28 02:58:46.132725489 (539h29m46.003808554s) [noise=true , address=145.79.227.7:4293 , piece_id=F6KLM4O5T2SQAYCY7GUX4IEF5RVPHZ3A5KLG7ZQMAR26OVHZ4PUA, seg_pos={0 0}, upload=473118]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE): protocol: write tcp 188.166.225.170:52012->145.79.227.7:4293: use of closed network connection; write tcp 188.166.225.170:52012->145.79.227.7:4293: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52012->145.79.227.7:4293: use of closed network connection Dec 28 02:58:46.132734947 (539h29m46.003817992s) [noise=true , address=145.79.227.7:4293 , piece_id=F6KLM4O5T2SQAYCY7GUX4IEF5RVPHZ3A5KLG7ZQMAR26OVHZ4PUA, seg_pos={0 0}, upload=473118]: Long tail cancellation due to optimal threshold Dec 28 02:58:46.132901384 (539h29m46.003984419s) [noise=true , address=77.90.31.16:4735 , piece_id=RLRI6MGF3RBEAUPD4PT7WNY36ZVT64RZIFNMJMWPGZMTQPGU77YQ, seg_pos={0 0}, upload=473118]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 28 02:58:46.132908057 (539h29m46.003991092s) [noise=true , address=77.90.31.16:4735 , piece_id=RLRI6MGF3RBEAUPD4PT7WNY36ZVT64RZIFNMJMWPGZMTQPGU77YQ, seg_pos={0 0}, upload=473118]: Long tail cancellation due to optimal threshold Dec 28 02:58:46.133195845 (539h29m46.00427889s) [noise=true , address=45.91.116.14:4192 , piece_id=RTQ2YWCFUJS3PZ6VPKY3CCC3LDL6KAKPO3DFYFPWD2VAJMHMPVQA, seg_pos={0 0}, upload=473118]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: expected piece hash; context canceled; EOF Dec 28 02:58:46.133214831 (539h29m46.004297876s) [noise=true , address=45.91.116.14:4192 , piece_id=RTQ2YWCFUJS3PZ6VPKY3CCC3LDL6KAKPO3DFYFPWD2VAJMHMPVQA, seg_pos={0 0}, upload=473118]: Long tail cancellation due to optimal threshold Dec 28 02:58:46.133249908 (539h29m46.004332943s) [seg_pos={0 0}, upload=473118]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 02:58:46.133364056 (539h29m46.004447091s) [upload=473118]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 02:58:46.133371590 (539h29m46.004454635s) [upload=473118]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 02:58:46.435703951 (539h29m46.306786996s) [upload=473118]: Done uploading segments... Dec 28 02:58:46.435728738 (539h29m46.306811784s) [upload=473118]: Upload finished. err: <nil> Duration: 30.002050144s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [................X................X....X............X........] duration: 4.391445623s trace: https://jaeger.dev.storj.io/trace/20df15b758999ef2 FAIL (2025-12-28 02:59:08 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.391482384s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............X.....X........................................] duration: 33.446548034s trace: https://jaeger.dev.storj.io/trace/1822be52d200e5b5 FAIL (2025-12-28 02:58:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1ZAfYZ4wgpZVvatj6wtHona3hTaPcLxAmXC1DT2uKQyo8nLQQ7, Piece ID: 5MMQ44MXRT4XEL4X6WO54WRG2XYMX3AN3FNQ2H2XIUTKBXI3EEPA) context deadline exceeded; piecestore: (Node ID: 124guAM82oibYZsMho67Em6Ke6yEZXwT6LPMLLSZZBLqwoFGUkn, Piece ID: 22WJIPX4SM5NTIAW5ZRM2UKX32CTDBFCXPCCHRRWRCUCOF5CRE5Q) context deadline exceeded; piecestore: (Node ID: 1e6p1TD5CqUyWdZjkrxh2kRqMuyDiMUr9eimfeayDAKwhtBrZ8, Piece ID: XRI5EJFN7YYUG4QKLDGEFT6YAUHCUTTSXF735UDR6LZZBGMAZDJQ) context deadline exceeded ======LOGS====== 5.234848204 (538h55m35.105931239s) [noise=true , address=77.90.31.11:4572 , piece_id=5MMQ44MXRT4XEL4X6WO54WRG2XYMX3AN3FNQ2H2XIUTKBXI3EEPA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.235412057 (538h55m35.106495102s) [noise=true , address=145.79.227.22:4512 , piece_id=D46YLTIKGHG36DQ4YBKBSUDV4GGB4WVDWNOJS27IYYLN6AYZ5QFQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.235820946 (538h55m35.106903981s) [noise=true , address=77.90.31.19:4874 , piece_id=RHZUXUYMJU6RKQGSDZM5WDWIN5SQ7YIRSRATNNKT5GEMU2QH444A, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.236192192 (538h55m35.107275237s) [noise=true , address=77.90.31.16:4760 , piece_id=XRI5EJFN7YYUG4QKLDGEFT6YAUHCUTTSXF735UDR6LZZBGMAZDJQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.236564772 (538h55m35.107647817s) [noise=true , address=145.79.227.40:4062 , piece_id=PW6QWIPQWVUETLC6DJ624G6IB4P6EM4SYB72NAMQ4V6NUKNLUYLQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.236929766 (538h55m35.108012812s) [noise=true , address=64.49.13.139:4480 , piece_id=JAJY3F7IDFFMFKCK2QJV6W74RIESLRKBLXMN27Q3C45JWEPLBZLQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.237916846 (538h55m35.108999881s) [noise=true , address=145.79.227.34:4932 , piece_id=UPXSCQ45QDGPKNJXRFJ33EAA3QENVHRVQARU5TFJTEWICZNR253Q, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.238852948 (538h55m35.109935983s) [noise=true , address=45.91.116.12:4257 , piece_id=ZI67LIAHZW4G357GWKDPTB5SMXYGNXZTTTDMNAETDS2B6SWVDAEQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239152910 (538h55m35.110235945s) [noise=true , address=145.79.227.44:4222 , piece_id=CZ2CS62CTHQFAIE3S3JO2N3VJ4XIR5YJ2LMRVLLX7I4VM2VZVQVA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239409969 (538h55m35.110493004s) [noise=true , address=145.79.227.30:4773 , piece_id=KUGJXZCNMUE7WSK3VLY4AUBSOT4H6IY4XB4NCBGAGCNUG7F7IODQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239681136 (538h55m35.110764171s) [noise=true , address=77.90.31.13:4635 , piece_id=IIOY5X2W3XW4C2OFEBRCABMYFBI24RRTYNUSKN6KJU7CTUW4KONQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239954676 (538h55m35.111037711s) [noise=true , address=45.91.116.9:4281 , piece_id=JZLDW777CBX2BP7MLOQV4NFP3H4XJVCNQXTHNSUWPVDOJLPZK36A, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.240189914 (538h55m35.111272949s) [noise=true , address=145.79.227.35:4512 , piece_id=3E3VCCC45XVT5FHFDKXSTGGAGYBEML3BKU6JJTT33LCD2YMOMBZQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.240449147 (538h55m35.111532182s) [noise=true , address=45.91.116.8:4064 , piece_id=22WJIPX4SM5NTIAW5ZRM2UKX32CTDBFCXPCCHRRWRCUCOF5CRE5Q, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.240639640 (538h55m35.111722675s) [noise=true , address=145.79.227.32:4853 , piece_id=MBNCFDBQ4SZZXV4F7J5TRMOUYUQ5IPKQCGYGGM5SOIFJ6ZYCAUYA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.241137798 (538h55m35.112220833s) [noise=true , address=77.90.31.16:4751 , piece_id=N3Z2LLMCIA2F42VXZA7IDRU4XEKGOICH6TRVERODU76MFFE4QSHA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.241383797 (538h55m35.112466842s) [noise=true , address=77.90.31.18:4836 , piece_id=BSVEFI52EKN44CMINFUSZBNOH53IW5TXLVTF5IH6ZUTNII6RRLVA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:36.592880816 (538h55m36.463963871s) [upload=472631]: Next returned nil segment Dec 28 02:24:36.592901625 (538h55m36.46398467s) [upload=472631]: Waiting for error group managing segments... Dec 28 02:24:36.946933741 (538h55m36.818016786s) [noise=true , address=64.49.13.139:4480 , piece_id=JAJY3F7IDFFMFKCK2QJV6W74RIESLRKBLXMN27Q3C45JWEPLBZLQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.128189692 (538h55m36.999272737s) [noise=true , address=145.79.227.35:4512 , piece_id=3E3VCCC45XVT5FHFDKXSTGGAGYBEML3BKU6JJTT33LCD2YMOMBZQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.160760084 (538h55m37.031843119s) [noise=true , address=145.79.227.34:4932 , piece_id=UPXSCQ45QDGPKNJXRFJ33EAA3QENVHRVQARU5TFJTEWICZNR253Q, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.288838207 (538h55m37.159921252s) [noise=true , address=77.90.31.11:4572 , piece_id=5MMQ44MXRT4XEL4X6WO54WRG2XYMX3AN3FNQ2H2XIUTKBXI3EEPA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.319996310 (538h55m37.191079345s) [noise=true , address=145.79.227.22:4512 , piece_id=D46YLTIKGHG36DQ4YBKBSUDV4GGB4WVDWNOJS27IYYLN6AYZ5QFQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.320513273 (538h55m37.191596319s) [noise=true , address=77.90.31.18:4836 , piece_id=BSVEFI52EKN44CMINFUSZBNOH53IW5TXLVTF5IH6ZUTNII6RRLVA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.336522926 (538h55m37.207605971s) [noise=true , address=145.79.227.32:4853 , piece_id=MBNCFDBQ4SZZXV4F7J5TRMOUYUQ5IPKQCGYGGM5SOIFJ6ZYCAUYA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.384868031 (538h55m37.255951066s) [noise=true , address=77.90.31.16:4751 , piece_id=N3Z2LLMCIA2F42VXZA7IDRU4XEKGOICH6TRVERODU76MFFE4QSHA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.453563554 (538h55m37.324646599s) [noise=true , address=45.91.116.8:4064 , piece_id=22WJIPX4SM5NTIAW5ZRM2UKX32CTDBFCXPCCHRRWRCUCOF5CRE5Q, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.479454839 (538h55m37.350537884s) [noise=true , address=45.91.116.12:4257 , piece_id=ZI67LIAHZW4G357GWKDPTB5SMXYGNXZTTTDMNAETDS2B6SWVDAEQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.659710174 (538h55m37.530793219s) [noise=true , address=77.90.31.13:4635 , piece_id=IIOY5X2W3XW4C2OFEBRCABMYFBI24RRTYNUSKN6KJU7CTUW4KONQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.715039128 (538h55m37.586122163s) [noise=true , address=77.90.31.16:4760 , piece_id=XRI5EJFN7YYUG4QKLDGEFT6YAUHCUTTSXF735UDR6LZZBGMAZDJQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.977473406 (538h55m37.848556441s) [noise=true , address=77.90.31.19:4874 , piece_id=RHZUXUYMJU6RKQGSDZM5WDWIN5SQ7YIRSRATNNKT5GEMU2QH444A, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:38.016302745 (538h55m37.88738578s) [noise=true , address=145.79.227.30:4773 , piece_id=KUGJXZCNMUE7WSK3VLY4AUBSOT4H6IY4XB4NCBGAGCNUG7F7IODQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:38.016326982 (538h55m37.887410027s) [seg_pos={0 0}, upload=472631]: Segment reached optimal threshold of 14 pieces. Dec 28 02:24:38.016585353 (538h55m37.887668398s) [noise=true , address=145.79.227.30:4766 , piece_id=67M2DMPAVWIQKQSWUAG73D4E4NRRSG4XEVPOUVIJPEQOUV7A6VVA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1k6zdamuKPkBdzyNZNWx4FCmaTkv33PchcwvTRR8bgcETBs4K7): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016594450 (538h55m37.887677495s) [noise=true , address=145.79.227.30:4766 , piece_id=67M2DMPAVWIQKQSWUAG73D4E4NRRSG4XEVPOUVIJPEQOUV7A6VVA, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016627093 (538h55m37.887710148s) [noise=true , address=45.91.116.9:4281 , piece_id=JZLDW777CBX2BP7MLOQV4NFP3H4XJVCNQXTHNSUWPVDOJLPZK36A, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gxf7DpJZthyR6f1PExX56tJavjhPw9Gxb4h8mtmeYKW3t3tS1): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016635258 (538h55m37.887718293s) [noise=true , address=45.91.116.9:4281 , piece_id=JZLDW777CBX2BP7MLOQV4NFP3H4XJVCNQXTHNSUWPVDOJLPZK36A, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016871298 (538h55m37.887954363s) [noise=true , address=145.79.227.44:4222 , piece_id=CZ2CS62CTHQFAIE3S3JO2N3VJ4XIR5YJ2LMRVLLX7I4VM2VZVQVA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZKW35KvrsoYRefbYeUFZo8bh4XZYycNKGEvTFjzibezjXgj3e): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016880124 (538h55m37.887963169s) [noise=true , address=145.79.227.44:4222 , piece_id=CZ2CS62CTHQFAIE3S3JO2N3VJ4XIR5YJ2LMRVLLX7I4VM2VZVQVA, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016934839 (538h55m37.888017884s) [noise=true , address=145.79.227.40:4062 , piece_id=PW6QWIPQWVUETLC6DJ624G6IB4P6EM4SYB72NAMQ4V6NUKNLUYLQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016943425 (538h55m37.88802649s) [noise=true , address=145.79.227.40:4062 , piece_id=PW6QWIPQWVUETLC6DJ624G6IB4P6EM4SYB72NAMQ4V6NUKNLUYLQ, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016968252 (538h55m37.888051297s) [seg_pos={0 0}, upload=472631]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 02:24:38.017067822 (538h55m37.888150867s) [upload=472631]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 02:24:38.017074244 (538h55m37.888157289s) [upload=472631]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 02:24:38.304306667 (538h55m38.175389712s) [upload=472631]: Done uploading segments... Dec 28 02:24:38.304329641 (538h55m38.175412686s) [upload=472631]: Upload finished. err: <nil> Duration: 33.446578693s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 44 out of 60 is failed (threshold=3) [XXX.XXX.X.....XXX.XXXXXX.XXXX.XXXX..X..XXXXX.XXXXXXXXXXXXX.X] duration: 30.002578058s trace: https://jaeger.dev.storj.io/trace/22b2989c0e14dea9 FAIL (2025-12-28 02:58:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: 6UY3QTTU5V4L7GRUEXUMS3VWDPBF63ED5264FFMWSLUVNCESM3OQ) context deadline exceeded; piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: IBWVRPJM6RV7G4PHNFAUVX5UUHKLBTPAXO6DLYWAZ6BVAPY25XBA) context deadline exceeded ======LOGS====== Dec 28 02:57:51.998103812 (539h28m51.869186857s) [upload=473106]: Starting upload Dec 28 02:57:51.998308662 (539h28m51.869391707s) [upload=473106]: Done starting upload Dec 28 02:57:51.998416297 (539h28m51.869499332s) [upload=473106]: Uploading segments... Dec 28 02:57:51.998606229 (539h28m51.869689264s) [upload=473106]: Got next segment. Inline: false Dec 28 02:57:51.998674930 (539h28m51.869757965s) [upload=473106]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 02:57:51.998684087 (539h28m51.869767122s) [upload=473106]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 02:57:52.243412020 (539h28m52.114495055s) [seg_pos={0 0}, upload=473106]: Begin upload segment... Dec 28 02:57:52.243462696 (539h28m52.114545731s) [seg_pos={0 0}, upload=473106]: Done begin upload segment. Dec 28 02:57:52.243553440 (539h28m52.114636475s) [noise=true , address=45.91.116.8:4045 , piece_id=NZTJJRN23NGM54LHJJPXBDLLXFAPU4Y62H6IDNFUZVIWQLVD5SZQ, seg_pos={0 0}, upload=473106]: Uploading piece... Dec 28 02:57:52.243575672 (539h28m52.114658717s) [noise=true , address=77.90.31.19:4842 , piece_id=IBWVRPJM6RV7G4PHNFAUVX5UUHKLBTPAXO6DLYWAZ6BVAPY25XBA, seg_pos={0 0}, upload=473106]: Uploading piece... Dec 28 02:57:52.244066387 (539h28m52.115149432s) [noise=true , address=145.79.227.30:4762 , piece_id=U7OANMEYGXFXRII4SARKZXTJB2GP34Q6ZZMXMOCPTVSTDQKWBWDA, seg_pos={0 0}, upload=473106]: Uploading piece... Dec 28 02:57:52.244353223 (539h28m52.115436258s) [noise=true , address=145.79.227.36:4519 , piece_id=D6YQZNKL2CTDY3NRJGRYX6N3LGUTI4ZF6JUT7HNR4XJ3EYVKNT2Q, seg_pos={0 0}, upload=473106]: Uploading piece... Dec 28 02:57:52.244648745 (539h28m52.11573179s) [noise=true , address=145.79.227.18:4007 , piece_id=M4VOGWJ5GJJIDAGBKKX47CYAUPZ467627LIHQHUCD57ENBO6YJXA, seg_pos={0 0}, upload=473106]: Uploading piece... Dec 28 02:57:52.244956190 (539h28m52.116039235s) [noise=true , address=145.79.227.52:4446 , piece_id=M7I3MXFZV6JC7CKNJFVKPKVYZPE66HOWN3Z46PFOJGYHD7IR5MRQ, seg_pos={0 0}, upload=473106]: Uploading piece... Dec 28 02:57:52.245231956 (539h28m52.116314991s) [noise=true , address=45.91.116.13:4156 , piece_id=RUWMEBBPSIFCGUNNE55YX73RN6WYKWGDZQK3IZ6ALBABIGCS7RLA, seg_pos={0 0}, upload=473106]: Uploading piece... Dec 28 02:57:52.245645273 (539h28m52.116728308s) [noise=true , address=77.90.31.13:4626 , piece_id=6UY3QTTU5V4L7GRUEXUMS3VWDPBF63ED5264FFMWSLUVNCESM3OQ, seg_pos={0 0}, upload=473106]: Uploading piece... Dec 28 02:57:52.246282486 (539h28m52.117365521s) [noise=true , address=145.79.227.24:4583 , piece_id=NKIBULFPYWSMNU3HDSSE4DLAP4H5AEHDYFGITHSVNK3WN4ZOH55A, seg_pos={0 0}, upload=473106]: Uploading piece... Dec 28 02:57:52.247769898 (539h28m52.118852943s) [noise=true , address=45.91.116.4:4937 , piece_id=SD4LALFAECH77RUNQC5AX4BAFVP5P5F7YU6FFXFRJB2AJRUBGBQQ, seg_pos={0 0}, upload=473106]: Uploading piece... Dec 28 02:57:52.248260242 (539h28m52.119343277s) [noise=true , address=77.90.31.11:4554 , piece_id=NQ5VGUTZ4X32HC2X4Y5KZRI7AM5XKZIS3GDBNPLLVH5XKIX6R7YQ, seg_pos={0 0}, upload=473106]: Uploading piece... Dec 28 02:57:52.250385559 (539h28m52.121468594s) [noise=true , address=145.79.227.7:4293 , piece_id=I6V6E7UVVC7HNWJRYPDP7XDLAFGM6GZ4EPQMPBCJZA4KRCUZJTOQ, seg_pos={0 0}, upload=473106]: Uploading piece... Dec 28 02:57:52.251696125 (539h28m52.12277916s) [noise=true , address=45.91.116.9:4264 , piece_id=P4KQ7VAEI65T55SZ4YBMFCVQVSEMUAYWT4RWSGL5ZHVOXDMFUIOQ, seg_pos={0 0}, upload=473106]: Uploading piece... Dec 28 02:57:52.252833330 (539h28m52.123916366s) [noise=true , address=145.79.227.19:4078 , piece_id=JHJMHTRH3M5BKTKMRD7UCUTAL7ZJX7QR4QX42IZCNZTHCEF4AHKQ, seg_pos={0 0}, upload=473106]: Uploading piece... Dec 28 02:57:52.253919058 (539h28m52.125002093s) [noise=true , address=145.79.227.39:4015 , piece_id=V5OMHQ2FRYK3ISS7QWTDM2CNC2JIXDJETOKLF7RNORT3P3CH6LXQ, seg_pos={0 0}, upload=473106]: Uploading piece... Dec 28 02:57:52.255323091 (539h28m52.126406136s) [noise=true , address=145.79.227.38:4979 , piece_id=THDTI6JW322SQKALUN56AIE3U6UNQDHYTND22HS3NICEFDQVHCUQ, seg_pos={0 0}, upload=473106]: Uploading piece... Dec 28 02:57:53.964828784 (539h28m53.835911829s) [upload=473106]: Next returned nil segment Dec 28 02:57:53.964841969 (539h28m53.835925004s) [upload=473106]: Waiting for error group managing segments... Dec 28 02:57:54.245100858 (539h28m54.116183903s) [noise=true , address=145.79.227.18:4007 , piece_id=M4VOGWJ5GJJIDAGBKKX47CYAUPZ467627LIHQHUCD57ENBO6YJXA, seg_pos={0 0}, upload=473106]: Done uploading piece. err: <nil> Dec 28 02:57:54.254448088 (539h28m54.125531123s) [noise=true , address=145.79.227.7:4293 , piece_id=I6V6E7UVVC7HNWJRYPDP7XDLAFGM6GZ4EPQMPBCJZA4KRCUZJTOQ, seg_pos={0 0}, upload=473106]: Done uploading piece. err: <nil> Dec 28 02:57:54.254887774 (539h28m54.125970809s) [noise=true , address=145.79.227.24:4583 , piece_id=NKIBULFPYWSMNU3HDSSE4DLAP4H5AEHDYFGITHSVNK3WN4ZOH55A, seg_pos={0 0}, upload=473106]: Done uploading piece. err: <nil> Dec 28 02:57:54.257999380 (539h28m54.129082425s) [noise=true , address=145.79.227.38:4979 , piece_id=THDTI6JW322SQKALUN56AIE3U6UNQDHYTND22HS3NICEFDQVHCUQ, seg_pos={0 0}, upload=473106]: Done uploading piece. err: <nil> Dec 28 02:57:54.283624935 (539h28m54.15470797s) [noise=true , address=145.79.227.39:4015 , piece_id=V5OMHQ2FRYK3ISS7QWTDM2CNC2JIXDJETOKLF7RNORT3P3CH6LXQ, seg_pos={0 0}, upload=473106]: Done uploading piece. err: <nil> Dec 28 02:57:54.434713823 (539h28m54.305796858s) [noise=true , address=77.90.31.19:4842 , piece_id=IBWVRPJM6RV7G4PHNFAUVX5UUHKLBTPAXO6DLYWAZ6BVAPY25XBA, seg_pos={0 0}, upload=473106]: Done uploading piece. err: <nil> Dec 28 02:57:54.486712267 (539h28m54.357795302s) [noise=true , address=45.91.116.13:4156 , piece_id=RUWMEBBPSIFCGUNNE55YX73RN6WYKWGDZQK3IZ6ALBABIGCS7RLA, seg_pos={0 0}, upload=473106]: Done uploading piece. err: <nil> Dec 28 02:57:54.594329197 (539h28m54.465412242s) [noise=true , address=45.91.116.9:4264 , piece_id=P4KQ7VAEI65T55SZ4YBMFCVQVSEMUAYWT4RWSGL5ZHVOXDMFUIOQ, seg_pos={0 0}, upload=473106]: Done uploading piece. err: <nil> Dec 28 02:57:54.595722891 (539h28m54.466805956s) [noise=true , address=77.90.31.13:4626 , piece_id=6UY3QTTU5V4L7GRUEXUMS3VWDPBF63ED5264FFMWSLUVNCESM3OQ, seg_pos={0 0}, upload=473106]: Done uploading piece. err: <nil> Dec 28 02:57:54.644428205 (539h28m54.51551125s) [noise=true , address=45.91.116.8:4045 , piece_id=NZTJJRN23NGM54LHJJPXBDLLXFAPU4Y62H6IDNFUZVIWQLVD5SZQ, seg_pos={0 0}, upload=473106]: Done uploading piece. err: <nil> Dec 28 02:57:54.959874286 (539h28m54.830957331s) [noise=true , address=45.91.116.4:4937 , piece_id=SD4LALFAECH77RUNQC5AX4BAFVP5P5F7YU6FFXFRJB2AJRUBGBQQ, seg_pos={0 0}, upload=473106]: Done uploading piece. err: <nil> Dec 28 02:57:55.063610177 (539h28m54.934693222s) [noise=true , address=145.79.227.36:4519 , piece_id=D6YQZNKL2CTDY3NRJGRYX6N3LGUTI4ZF6JUT7HNR4XJ3EYVKNT2Q, seg_pos={0 0}, upload=473106]: Done uploading piece. err: <nil> Dec 28 02:57:55.202397030 (539h28m55.073480075s) [noise=true , address=77.90.31.11:4554 , piece_id=NQ5VGUTZ4X32HC2X4Y5KZRI7AM5XKZIS3GDBNPLLVH5XKIX6R7YQ, seg_pos={0 0}, upload=473106]: Done uploading piece. err: <nil> Dec 28 02:57:55.202414774 (539h28m55.073497809s) [seg_pos={0 0}, upload=473106]: Segment reached optimal threshold of 13 pieces. Dec 28 02:57:55.202720376 (539h28m55.073803411s) [noise=true , address=145.79.227.19:4078 , piece_id=JHJMHTRH3M5BKTKMRD7UCUTAL7ZJX7QR4QX42IZCNZTHCEF4AHKQ, seg_pos={0 0}, upload=473106]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: expected piece hash; context canceled; EOF Dec 28 02:57:55.202728181 (539h28m55.073811216s) [noise=true , address=145.79.227.19:4078 , piece_id=JHJMHTRH3M5BKTKMRD7UCUTAL7ZJX7QR4QX42IZCNZTHCEF4AHKQ, seg_pos={0 0}, upload=473106]: Long tail cancellation due to optimal threshold Dec 28 02:57:55.202809245 (539h28m55.07389229s) [noise=true , address=145.79.227.52:4446 , piece_id=M7I3MXFZV6JC7CKNJFVKPKVYZPE66HOWN3Z46PFOJGYHD7IR5MRQ, seg_pos={0 0}, upload=473106]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RNJPWzogZpsQfuFzJYZiz8bbHSRSrgSE5FLyhCynSwEDfMid7): protocol: expected piece hash; context canceled; EOF Dec 28 02:57:55.202819515 (539h28m55.07390255s) [noise=true , address=145.79.227.52:4446 , piece_id=M7I3MXFZV6JC7CKNJFVKPKVYZPE66HOWN3Z46PFOJGYHD7IR5MRQ, seg_pos={0 0}, upload=473106]: Long tail cancellation due to optimal threshold Dec 28 02:57:55.202966374 (539h28m55.074049429s) [noise=true , address=145.79.227.30:4762 , piece_id=U7OANMEYGXFXRII4SARKZXTJB2GP34Q6ZZMXMOCPTVSTDQKWBWDA, seg_pos={0 0}, upload=473106]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 28 02:57:55.202977365 (539h28m55.0740604s) [noise=true , address=145.79.227.30:4762 , piece_id=U7OANMEYGXFXRII4SARKZXTJB2GP34Q6ZZMXMOCPTVSTDQKWBWDA, seg_pos={0 0}, upload=473106]: Long tail cancellation due to optimal threshold Dec 28 02:57:55.202998375 (539h28m55.07408142s) [seg_pos={0 0}, upload=473106]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 02:57:55.203080992 (539h28m55.074164027s) [upload=473106]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 02:57:55.203087785 (539h28m55.07417082s) [upload=473106]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 02:57:55.491150515 (539h28m55.36223357s) [upload=473106]: Done uploading segments... Dec 28 02:57:55.491173469 (539h28m55.362256504s) [upload=473106]: Upload finished. err: <nil> Duration: 30.002601613s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [.................X................X....X............X.......] duration: 3.474025312s trace: https://jaeger.dev.storj.io/trace/20df15b758999ef2 FAIL (2025-12-28 02:58:20 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.474055239s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............X.....X.......................................] duration: 31.373410676s trace: https://jaeger.dev.storj.io/trace/1822be52d200e5b5 FAIL (2025-12-28 02:58:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1ZAfYZ4wgpZVvatj6wtHona3hTaPcLxAmXC1DT2uKQyo8nLQQ7, Piece ID: 5MMQ44MXRT4XEL4X6WO54WRG2XYMX3AN3FNQ2H2XIUTKBXI3EEPA) context deadline exceeded; piecestore: (Node ID: 124guAM82oibYZsMho67Em6Ke6yEZXwT6LPMLLSZZBLqwoFGUkn, Piece ID: 22WJIPX4SM5NTIAW5ZRM2UKX32CTDBFCXPCCHRRWRCUCOF5CRE5Q) context deadline exceeded; piecestore: (Node ID: 1e6p1TD5CqUyWdZjkrxh2kRqMuyDiMUr9eimfeayDAKwhtBrZ8, Piece ID: XRI5EJFN7YYUG4QKLDGEFT6YAUHCUTTSXF735UDR6LZZBGMAZDJQ) context deadline exceeded ======LOGS====== 5.234848204 (538h55m35.105931239s) [noise=true , address=77.90.31.11:4572 , piece_id=5MMQ44MXRT4XEL4X6WO54WRG2XYMX3AN3FNQ2H2XIUTKBXI3EEPA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.235412057 (538h55m35.106495102s) [noise=true , address=145.79.227.22:4512 , piece_id=D46YLTIKGHG36DQ4YBKBSUDV4GGB4WVDWNOJS27IYYLN6AYZ5QFQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.235820946 (538h55m35.106903981s) [noise=true , address=77.90.31.19:4874 , piece_id=RHZUXUYMJU6RKQGSDZM5WDWIN5SQ7YIRSRATNNKT5GEMU2QH444A, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.236192192 (538h55m35.107275237s) [noise=true , address=77.90.31.16:4760 , piece_id=XRI5EJFN7YYUG4QKLDGEFT6YAUHCUTTSXF735UDR6LZZBGMAZDJQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.236564772 (538h55m35.107647817s) [noise=true , address=145.79.227.40:4062 , piece_id=PW6QWIPQWVUETLC6DJ624G6IB4P6EM4SYB72NAMQ4V6NUKNLUYLQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.236929766 (538h55m35.108012812s) [noise=true , address=64.49.13.139:4480 , piece_id=JAJY3F7IDFFMFKCK2QJV6W74RIESLRKBLXMN27Q3C45JWEPLBZLQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.237916846 (538h55m35.108999881s) [noise=true , address=145.79.227.34:4932 , piece_id=UPXSCQ45QDGPKNJXRFJ33EAA3QENVHRVQARU5TFJTEWICZNR253Q, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.238852948 (538h55m35.109935983s) [noise=true , address=45.91.116.12:4257 , piece_id=ZI67LIAHZW4G357GWKDPTB5SMXYGNXZTTTDMNAETDS2B6SWVDAEQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239152910 (538h55m35.110235945s) [noise=true , address=145.79.227.44:4222 , piece_id=CZ2CS62CTHQFAIE3S3JO2N3VJ4XIR5YJ2LMRVLLX7I4VM2VZVQVA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239409969 (538h55m35.110493004s) [noise=true , address=145.79.227.30:4773 , piece_id=KUGJXZCNMUE7WSK3VLY4AUBSOT4H6IY4XB4NCBGAGCNUG7F7IODQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239681136 (538h55m35.110764171s) [noise=true , address=77.90.31.13:4635 , piece_id=IIOY5X2W3XW4C2OFEBRCABMYFBI24RRTYNUSKN6KJU7CTUW4KONQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239954676 (538h55m35.111037711s) [noise=true , address=45.91.116.9:4281 , piece_id=JZLDW777CBX2BP7MLOQV4NFP3H4XJVCNQXTHNSUWPVDOJLPZK36A, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.240189914 (538h55m35.111272949s) [noise=true , address=145.79.227.35:4512 , piece_id=3E3VCCC45XVT5FHFDKXSTGGAGYBEML3BKU6JJTT33LCD2YMOMBZQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.240449147 (538h55m35.111532182s) [noise=true , address=45.91.116.8:4064 , piece_id=22WJIPX4SM5NTIAW5ZRM2UKX32CTDBFCXPCCHRRWRCUCOF5CRE5Q, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.240639640 (538h55m35.111722675s) [noise=true , address=145.79.227.32:4853 , piece_id=MBNCFDBQ4SZZXV4F7J5TRMOUYUQ5IPKQCGYGGM5SOIFJ6ZYCAUYA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.241137798 (538h55m35.112220833s) [noise=true , address=77.90.31.16:4751 , piece_id=N3Z2LLMCIA2F42VXZA7IDRU4XEKGOICH6TRVERODU76MFFE4QSHA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.241383797 (538h55m35.112466842s) [noise=true , address=77.90.31.18:4836 , piece_id=BSVEFI52EKN44CMINFUSZBNOH53IW5TXLVTF5IH6ZUTNII6RRLVA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:36.592880816 (538h55m36.463963871s) [upload=472631]: Next returned nil segment Dec 28 02:24:36.592901625 (538h55m36.46398467s) [upload=472631]: Waiting for error group managing segments... Dec 28 02:24:36.946933741 (538h55m36.818016786s) [noise=true , address=64.49.13.139:4480 , piece_id=JAJY3F7IDFFMFKCK2QJV6W74RIESLRKBLXMN27Q3C45JWEPLBZLQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.128189692 (538h55m36.999272737s) [noise=true , address=145.79.227.35:4512 , piece_id=3E3VCCC45XVT5FHFDKXSTGGAGYBEML3BKU6JJTT33LCD2YMOMBZQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.160760084 (538h55m37.031843119s) [noise=true , address=145.79.227.34:4932 , piece_id=UPXSCQ45QDGPKNJXRFJ33EAA3QENVHRVQARU5TFJTEWICZNR253Q, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.288838207 (538h55m37.159921252s) [noise=true , address=77.90.31.11:4572 , piece_id=5MMQ44MXRT4XEL4X6WO54WRG2XYMX3AN3FNQ2H2XIUTKBXI3EEPA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.319996310 (538h55m37.191079345s) [noise=true , address=145.79.227.22:4512 , piece_id=D46YLTIKGHG36DQ4YBKBSUDV4GGB4WVDWNOJS27IYYLN6AYZ5QFQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.320513273 (538h55m37.191596319s) [noise=true , address=77.90.31.18:4836 , piece_id=BSVEFI52EKN44CMINFUSZBNOH53IW5TXLVTF5IH6ZUTNII6RRLVA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.336522926 (538h55m37.207605971s) [noise=true , address=145.79.227.32:4853 , piece_id=MBNCFDBQ4SZZXV4F7J5TRMOUYUQ5IPKQCGYGGM5SOIFJ6ZYCAUYA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.384868031 (538h55m37.255951066s) [noise=true , address=77.90.31.16:4751 , piece_id=N3Z2LLMCIA2F42VXZA7IDRU4XEKGOICH6TRVERODU76MFFE4QSHA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.453563554 (538h55m37.324646599s) [noise=true , address=45.91.116.8:4064 , piece_id=22WJIPX4SM5NTIAW5ZRM2UKX32CTDBFCXPCCHRRWRCUCOF5CRE5Q, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.479454839 (538h55m37.350537884s) [noise=true , address=45.91.116.12:4257 , piece_id=ZI67LIAHZW4G357GWKDPTB5SMXYGNXZTTTDMNAETDS2B6SWVDAEQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.659710174 (538h55m37.530793219s) [noise=true , address=77.90.31.13:4635 , piece_id=IIOY5X2W3XW4C2OFEBRCABMYFBI24RRTYNUSKN6KJU7CTUW4KONQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.715039128 (538h55m37.586122163s) [noise=true , address=77.90.31.16:4760 , piece_id=XRI5EJFN7YYUG4QKLDGEFT6YAUHCUTTSXF735UDR6LZZBGMAZDJQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.977473406 (538h55m37.848556441s) [noise=true , address=77.90.31.19:4874 , piece_id=RHZUXUYMJU6RKQGSDZM5WDWIN5SQ7YIRSRATNNKT5GEMU2QH444A, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:38.016302745 (538h55m37.88738578s) [noise=true , address=145.79.227.30:4773 , piece_id=KUGJXZCNMUE7WSK3VLY4AUBSOT4H6IY4XB4NCBGAGCNUG7F7IODQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:38.016326982 (538h55m37.887410027s) [seg_pos={0 0}, upload=472631]: Segment reached optimal threshold of 14 pieces. Dec 28 02:24:38.016585353 (538h55m37.887668398s) [noise=true , address=145.79.227.30:4766 , piece_id=67M2DMPAVWIQKQSWUAG73D4E4NRRSG4XEVPOUVIJPEQOUV7A6VVA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1k6zdamuKPkBdzyNZNWx4FCmaTkv33PchcwvTRR8bgcETBs4K7): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016594450 (538h55m37.887677495s) [noise=true , address=145.79.227.30:4766 , piece_id=67M2DMPAVWIQKQSWUAG73D4E4NRRSG4XEVPOUVIJPEQOUV7A6VVA, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016627093 (538h55m37.887710148s) [noise=true , address=45.91.116.9:4281 , piece_id=JZLDW777CBX2BP7MLOQV4NFP3H4XJVCNQXTHNSUWPVDOJLPZK36A, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gxf7DpJZthyR6f1PExX56tJavjhPw9Gxb4h8mtmeYKW3t3tS1): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016635258 (538h55m37.887718293s) [noise=true , address=45.91.116.9:4281 , piece_id=JZLDW777CBX2BP7MLOQV4NFP3H4XJVCNQXTHNSUWPVDOJLPZK36A, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016871298 (538h55m37.887954363s) [noise=true , address=145.79.227.44:4222 , piece_id=CZ2CS62CTHQFAIE3S3JO2N3VJ4XIR5YJ2LMRVLLX7I4VM2VZVQVA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZKW35KvrsoYRefbYeUFZo8bh4XZYycNKGEvTFjzibezjXgj3e): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016880124 (538h55m37.887963169s) [noise=true , address=145.79.227.44:4222 , piece_id=CZ2CS62CTHQFAIE3S3JO2N3VJ4XIR5YJ2LMRVLLX7I4VM2VZVQVA, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016934839 (538h55m37.888017884s) [noise=true , address=145.79.227.40:4062 , piece_id=PW6QWIPQWVUETLC6DJ624G6IB4P6EM4SYB72NAMQ4V6NUKNLUYLQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016943425 (538h55m37.88802649s) [noise=true , address=145.79.227.40:4062 , piece_id=PW6QWIPQWVUETLC6DJ624G6IB4P6EM4SYB72NAMQ4V6NUKNLUYLQ, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016968252 (538h55m37.888051297s) [seg_pos={0 0}, upload=472631]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 02:24:38.017067822 (538h55m37.888150867s) [upload=472631]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 02:24:38.017074244 (538h55m37.888157289s) [upload=472631]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 02:24:38.304306667 (538h55m38.175389712s) [upload=472631]: Done uploading segments... Dec 28 02:24:38.304329641 (538h55m38.175412686s) [upload=472631]: Upload finished. err: <nil> Duration: 31.373431206s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............X.....X......................................] duration: 44.483537851s trace: https://jaeger.dev.storj.io/trace/1822be52d200e5b5 FAIL (2025-12-28 02:57:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1ZAfYZ4wgpZVvatj6wtHona3hTaPcLxAmXC1DT2uKQyo8nLQQ7, Piece ID: 5MMQ44MXRT4XEL4X6WO54WRG2XYMX3AN3FNQ2H2XIUTKBXI3EEPA) context deadline exceeded; piecestore: (Node ID: 124guAM82oibYZsMho67Em6Ke6yEZXwT6LPMLLSZZBLqwoFGUkn, Piece ID: 22WJIPX4SM5NTIAW5ZRM2UKX32CTDBFCXPCCHRRWRCUCOF5CRE5Q) context deadline exceeded; piecestore: (Node ID: 1e6p1TD5CqUyWdZjkrxh2kRqMuyDiMUr9eimfeayDAKwhtBrZ8, Piece ID: XRI5EJFN7YYUG4QKLDGEFT6YAUHCUTTSXF735UDR6LZZBGMAZDJQ) context deadline exceeded ======LOGS====== 5.234848204 (538h55m35.105931239s) [noise=true , address=77.90.31.11:4572 , piece_id=5MMQ44MXRT4XEL4X6WO54WRG2XYMX3AN3FNQ2H2XIUTKBXI3EEPA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.235412057 (538h55m35.106495102s) [noise=true , address=145.79.227.22:4512 , piece_id=D46YLTIKGHG36DQ4YBKBSUDV4GGB4WVDWNOJS27IYYLN6AYZ5QFQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.235820946 (538h55m35.106903981s) [noise=true , address=77.90.31.19:4874 , piece_id=RHZUXUYMJU6RKQGSDZM5WDWIN5SQ7YIRSRATNNKT5GEMU2QH444A, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.236192192 (538h55m35.107275237s) [noise=true , address=77.90.31.16:4760 , piece_id=XRI5EJFN7YYUG4QKLDGEFT6YAUHCUTTSXF735UDR6LZZBGMAZDJQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.236564772 (538h55m35.107647817s) [noise=true , address=145.79.227.40:4062 , piece_id=PW6QWIPQWVUETLC6DJ624G6IB4P6EM4SYB72NAMQ4V6NUKNLUYLQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.236929766 (538h55m35.108012812s) [noise=true , address=64.49.13.139:4480 , piece_id=JAJY3F7IDFFMFKCK2QJV6W74RIESLRKBLXMN27Q3C45JWEPLBZLQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.237916846 (538h55m35.108999881s) [noise=true , address=145.79.227.34:4932 , piece_id=UPXSCQ45QDGPKNJXRFJ33EAA3QENVHRVQARU5TFJTEWICZNR253Q, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.238852948 (538h55m35.109935983s) [noise=true , address=45.91.116.12:4257 , piece_id=ZI67LIAHZW4G357GWKDPTB5SMXYGNXZTTTDMNAETDS2B6SWVDAEQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239152910 (538h55m35.110235945s) [noise=true , address=145.79.227.44:4222 , piece_id=CZ2CS62CTHQFAIE3S3JO2N3VJ4XIR5YJ2LMRVLLX7I4VM2VZVQVA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239409969 (538h55m35.110493004s) [noise=true , address=145.79.227.30:4773 , piece_id=KUGJXZCNMUE7WSK3VLY4AUBSOT4H6IY4XB4NCBGAGCNUG7F7IODQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239681136 (538h55m35.110764171s) [noise=true , address=77.90.31.13:4635 , piece_id=IIOY5X2W3XW4C2OFEBRCABMYFBI24RRTYNUSKN6KJU7CTUW4KONQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239954676 (538h55m35.111037711s) [noise=true , address=45.91.116.9:4281 , piece_id=JZLDW777CBX2BP7MLOQV4NFP3H4XJVCNQXTHNSUWPVDOJLPZK36A, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.240189914 (538h55m35.111272949s) [noise=true , address=145.79.227.35:4512 , piece_id=3E3VCCC45XVT5FHFDKXSTGGAGYBEML3BKU6JJTT33LCD2YMOMBZQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.240449147 (538h55m35.111532182s) [noise=true , address=45.91.116.8:4064 , piece_id=22WJIPX4SM5NTIAW5ZRM2UKX32CTDBFCXPCCHRRWRCUCOF5CRE5Q, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.240639640 (538h55m35.111722675s) [noise=true , address=145.79.227.32:4853 , piece_id=MBNCFDBQ4SZZXV4F7J5TRMOUYUQ5IPKQCGYGGM5SOIFJ6ZYCAUYA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.241137798 (538h55m35.112220833s) [noise=true , address=77.90.31.16:4751 , piece_id=N3Z2LLMCIA2F42VXZA7IDRU4XEKGOICH6TRVERODU76MFFE4QSHA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.241383797 (538h55m35.112466842s) [noise=true , address=77.90.31.18:4836 , piece_id=BSVEFI52EKN44CMINFUSZBNOH53IW5TXLVTF5IH6ZUTNII6RRLVA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:36.592880816 (538h55m36.463963871s) [upload=472631]: Next returned nil segment Dec 28 02:24:36.592901625 (538h55m36.46398467s) [upload=472631]: Waiting for error group managing segments... Dec 28 02:24:36.946933741 (538h55m36.818016786s) [noise=true , address=64.49.13.139:4480 , piece_id=JAJY3F7IDFFMFKCK2QJV6W74RIESLRKBLXMN27Q3C45JWEPLBZLQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.128189692 (538h55m36.999272737s) [noise=true , address=145.79.227.35:4512 , piece_id=3E3VCCC45XVT5FHFDKXSTGGAGYBEML3BKU6JJTT33LCD2YMOMBZQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.160760084 (538h55m37.031843119s) [noise=true , address=145.79.227.34:4932 , piece_id=UPXSCQ45QDGPKNJXRFJ33EAA3QENVHRVQARU5TFJTEWICZNR253Q, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.288838207 (538h55m37.159921252s) [noise=true , address=77.90.31.11:4572 , piece_id=5MMQ44MXRT4XEL4X6WO54WRG2XYMX3AN3FNQ2H2XIUTKBXI3EEPA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.319996310 (538h55m37.191079345s) [noise=true , address=145.79.227.22:4512 , piece_id=D46YLTIKGHG36DQ4YBKBSUDV4GGB4WVDWNOJS27IYYLN6AYZ5QFQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.320513273 (538h55m37.191596319s) [noise=true , address=77.90.31.18:4836 , piece_id=BSVEFI52EKN44CMINFUSZBNOH53IW5TXLVTF5IH6ZUTNII6RRLVA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.336522926 (538h55m37.207605971s) [noise=true , address=145.79.227.32:4853 , piece_id=MBNCFDBQ4SZZXV4F7J5TRMOUYUQ5IPKQCGYGGM5SOIFJ6ZYCAUYA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.384868031 (538h55m37.255951066s) [noise=true , address=77.90.31.16:4751 , piece_id=N3Z2LLMCIA2F42VXZA7IDRU4XEKGOICH6TRVERODU76MFFE4QSHA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.453563554 (538h55m37.324646599s) [noise=true , address=45.91.116.8:4064 , piece_id=22WJIPX4SM5NTIAW5ZRM2UKX32CTDBFCXPCCHRRWRCUCOF5CRE5Q, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.479454839 (538h55m37.350537884s) [noise=true , address=45.91.116.12:4257 , piece_id=ZI67LIAHZW4G357GWKDPTB5SMXYGNXZTTTDMNAETDS2B6SWVDAEQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.659710174 (538h55m37.530793219s) [noise=true , address=77.90.31.13:4635 , piece_id=IIOY5X2W3XW4C2OFEBRCABMYFBI24RRTYNUSKN6KJU7CTUW4KONQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.715039128 (538h55m37.586122163s) [noise=true , address=77.90.31.16:4760 , piece_id=XRI5EJFN7YYUG4QKLDGEFT6YAUHCUTTSXF735UDR6LZZBGMAZDJQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.977473406 (538h55m37.848556441s) [noise=true , address=77.90.31.19:4874 , piece_id=RHZUXUYMJU6RKQGSDZM5WDWIN5SQ7YIRSRATNNKT5GEMU2QH444A, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:38.016302745 (538h55m37.88738578s) [noise=true , address=145.79.227.30:4773 , piece_id=KUGJXZCNMUE7WSK3VLY4AUBSOT4H6IY4XB4NCBGAGCNUG7F7IODQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:38.016326982 (538h55m37.887410027s) [seg_pos={0 0}, upload=472631]: Segment reached optimal threshold of 14 pieces. Dec 28 02:24:38.016585353 (538h55m37.887668398s) [noise=true , address=145.79.227.30:4766 , piece_id=67M2DMPAVWIQKQSWUAG73D4E4NRRSG4XEVPOUVIJPEQOUV7A6VVA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1k6zdamuKPkBdzyNZNWx4FCmaTkv33PchcwvTRR8bgcETBs4K7): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016594450 (538h55m37.887677495s) [noise=true , address=145.79.227.30:4766 , piece_id=67M2DMPAVWIQKQSWUAG73D4E4NRRSG4XEVPOUVIJPEQOUV7A6VVA, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016627093 (538h55m37.887710148s) [noise=true , address=45.91.116.9:4281 , piece_id=JZLDW777CBX2BP7MLOQV4NFP3H4XJVCNQXTHNSUWPVDOJLPZK36A, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gxf7DpJZthyR6f1PExX56tJavjhPw9Gxb4h8mtmeYKW3t3tS1): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016635258 (538h55m37.887718293s) [noise=true , address=45.91.116.9:4281 , piece_id=JZLDW777CBX2BP7MLOQV4NFP3H4XJVCNQXTHNSUWPVDOJLPZK36A, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016871298 (538h55m37.887954363s) [noise=true , address=145.79.227.44:4222 , piece_id=CZ2CS62CTHQFAIE3S3JO2N3VJ4XIR5YJ2LMRVLLX7I4VM2VZVQVA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZKW35KvrsoYRefbYeUFZo8bh4XZYycNKGEvTFjzibezjXgj3e): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016880124 (538h55m37.887963169s) [noise=true , address=145.79.227.44:4222 , piece_id=CZ2CS62CTHQFAIE3S3JO2N3VJ4XIR5YJ2LMRVLLX7I4VM2VZVQVA, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016934839 (538h55m37.888017884s) [noise=true , address=145.79.227.40:4062 , piece_id=PW6QWIPQWVUETLC6DJ624G6IB4P6EM4SYB72NAMQ4V6NUKNLUYLQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016943425 (538h55m37.88802649s) [noise=true , address=145.79.227.40:4062 , piece_id=PW6QWIPQWVUETLC6DJ624G6IB4P6EM4SYB72NAMQ4V6NUKNLUYLQ, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016968252 (538h55m37.888051297s) [seg_pos={0 0}, upload=472631]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 02:24:38.017067822 (538h55m37.888150867s) [upload=472631]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 02:24:38.017074244 (538h55m37.888157289s) [upload=472631]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 02:24:38.304306667 (538h55m38.175389712s) [upload=472631]: Done uploading segments... Dec 28 02:24:38.304329641 (538h55m38.175412686s) [upload=472631]: Upload finished. err: <nil> Duration: 44.483564632s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [..................X................X....X............X......] duration: 5.001096881s trace: https://jaeger.dev.storj.io/trace/20df15b758999ef2 FAIL (2025-12-28 02:57:22 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001115488s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 43 out of 60 is failed (threshold=3) [.XXX.XXX.X.....XXX.XXXXXX.XXXX.XXXX..X..XXXXX.XXXXXXXXXXXXX.] duration: 6.67098654s trace: https://jaeger.dev.storj.io/trace/6ede6e48b6098cbc FAIL (2025-12-28 02:57:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: UA4IVYA4WQT2Z32RYRMKLP4L3ZU7ZQSLCAR4XZPWLEB2CZHI3JGQ) context deadline exceeded; piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: X3VIV5R6JUWRAGR6MF6QMFWVJDUJQ6VYBFN2Y7473Y3YOJZ35DUQ) context deadline exceeded ======LOGS====== Dec 28 02:56:34.207261982 (539h27m34.078345027s) [upload=473082]: Starting upload Dec 28 02:56:34.207317187 (539h27m34.078400222s) [upload=473082]: Done starting upload Dec 28 02:56:34.207386559 (539h27m34.078469594s) [upload=473082]: Uploading segments... Dec 28 02:56:34.207528460 (539h27m34.078611495s) [upload=473082]: Got next segment. Inline: false Dec 28 02:56:34.207606779 (539h27m34.078689834s) [upload=473082]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 02:56:34.207613892 (539h27m34.078696957s) [upload=473082]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 02:56:34.455905841 (539h27m34.326988876s) [seg_pos={0 0}, upload=473082]: Begin upload segment... Dec 28 02:56:34.455960897 (539h27m34.327043932s) [seg_pos={0 0}, upload=473082]: Done begin upload segment. Dec 28 02:56:34.455995623 (539h27m34.327078658s) [noise=true , address=145.79.227.11:4731 , piece_id=LLDKPJ37IYFHHREFM52EYGXHVFOEGJRVS2CTBRYNOLXAZFQ7QTGQ, seg_pos={0 0}, upload=473082]: Uploading piece... Dec 28 02:56:34.456072940 (539h27m34.327155985s) [noise=true , address=145.79.227.5:4222 , piece_id=FT2T2KRZPCHMSV6MAMY2QVORPOA3GP544PDTFGM62X6PIRRXDY3A, seg_pos={0 0}, upload=473082]: Uploading piece... Dec 28 02:56:34.456878313 (539h27m34.327961348s) [noise=true , address=145.79.227.6:4257 , piece_id=I5A533FIFUAZ7JRH2E4TGSRAU26E6SL2XMWM4JS4F4TOICQ2PIMA, seg_pos={0 0}, upload=473082]: Uploading piece... Dec 28 02:56:34.457125664 (539h27m34.328208709s) [noise=true , address=145.79.227.37:4565 , piece_id=ZKAKQFYIFELEZ6HA55XXA3XRN3QVAB5PLE7TZISWBDWRARC5PTAQ, seg_pos={0 0}, upload=473082]: Uploading piece... Dec 28 02:56:34.457520587 (539h27m34.328603622s) [noise=true , address=77.90.31.11:4556 , piece_id=KAPN4CPMWM4OUQW6AVNEG7SMRZM4GQACJ72OFQSVQPVDES5D6CGA, seg_pos={0 0}, upload=473082]: Uploading piece... Dec 28 02:56:34.457911240 (539h27m34.328994285s) [noise=true , address=145.79.227.16:4912 , piece_id=JQK25B3PCURS5VY4CHCV4J6BY2ITJ45CVMEX3LQ4D4LGDWYL7ORA, seg_pos={0 0}, upload=473082]: Uploading piece... Dec 28 02:56:34.458058992 (539h27m34.329142027s) [noise=true , address=77.90.31.13:4626 , piece_id=X3VIV5R6JUWRAGR6MF6QMFWVJDUJQ6VYBFN2Y7473Y3YOJZ35DUQ, seg_pos={0 0}, upload=473082]: Uploading piece... Dec 28 02:56:34.458218105 (539h27m34.32930114s) [noise=true , address=45.91.116.5:4299 , piece_id=5JHPHTPARWXCFHZB23KGJYPYATO2ANXGHQQNXUDVFGQ6BFK2PL2A, seg_pos={0 0}, upload=473082]: Uploading piece... Dec 28 02:56:34.458465135 (539h27m34.32954817s) [noise=true , address=45.91.116.13:4156 , piece_id=LKWX4FCKPWRDHKGLSRECHYCKJZS77COBNAHB2M2XE2BL2M6VY4WQ, seg_pos={0 0}, upload=473082]: Uploading piece... Dec 28 02:56:34.458738857 (539h27m34.329821892s) [noise=true , address=45.91.116.4:4937 , piece_id=3DGORE2BAPFSTMP6PNEUTKPS6UVNWVLT4NVX7CH3CPDXEBZK4PYA, seg_pos={0 0}, upload=473082]: Uploading piece... Dec 28 02:56:34.459059868 (539h27m34.330142903s) [noise=true , address=77.90.31.10:4518 , piece_id=EVSYIELGBLKMOZO2JJNO4ZUMN2QDS7KJEEFJPEOHA7DUGZ4DYQPQ, seg_pos={0 0}, upload=473082]: Uploading piece... Dec 28 02:56:34.459065288 (539h27m34.330148323s) [noise=true , address=45.91.116.6:4974 , piece_id=JPAJR6HV36LLIPDEGEQQLJ5JBE32PQZJ5USSQD4BG7FFJRI7MWTQ, seg_pos={0 0}, upload=473082]: Uploading piece... Dec 28 02:56:34.459353156 (539h27m34.330436191s) [noise=true , address=145.79.227.46:4602 , piece_id=NGLL5MPL2PUA6W4UDTUKR7GJVZIN2JUJHCKISVPVWIB3ZMTOGUWA, seg_pos={0 0}, upload=473082]: Uploading piece... Dec 28 02:56:34.459477603 (539h27m34.330560648s) [noise=true , address=145.79.227.15:4873 , piece_id=AAJPAAHZOWVUI3C2EWYS2EZ2QUAEQQNTATTZMNAFWMHEGRZJFS4A, seg_pos={0 0}, upload=473082]: Uploading piece... Dec 28 02:56:34.459724503 (539h27m34.330807548s) [noise=true , address=77.90.31.15:4700 , piece_id=UA4IVYA4WQT2Z32RYRMKLP4L3ZU7ZQSLCAR4XZPWLEB2CZHI3JGQ, seg_pos={0 0}, upload=473082]: Uploading piece... Dec 28 02:56:34.459996251 (539h27m34.331079296s) [noise=true , address=145.79.227.51:4409 , piece_id=SB3LIQDXCVDOHZDTW7PB46XMRRWQ6LD3PW3MPHAEKJ3EX42UCLSQ, seg_pos={0 0}, upload=473082]: Uploading piece... Dec 28 02:56:36.141335781 (539h27m36.012418816s) [upload=473082]: Next returned nil segment Dec 28 02:56:36.141352292 (539h27m36.012435337s) [upload=473082]: Waiting for error group managing segments... Dec 28 02:56:36.448609491 (539h27m36.319692536s) [noise=true , address=145.79.227.5:4222 , piece_id=FT2T2KRZPCHMSV6MAMY2QVORPOA3GP544PDTFGM62X6PIRRXDY3A, seg_pos={0 0}, upload=473082]: Done uploading piece. err: <nil> Dec 28 02:56:36.461384086 (539h27m36.332467121s) [noise=true , address=145.79.227.51:4409 , piece_id=SB3LIQDXCVDOHZDTW7PB46XMRRWQ6LD3PW3MPHAEKJ3EX42UCLSQ, seg_pos={0 0}, upload=473082]: Done uploading piece. err: <nil> Dec 28 02:56:36.475594144 (539h27m36.34667718s) [noise=true , address=145.79.227.11:4731 , piece_id=LLDKPJ37IYFHHREFM52EYGXHVFOEGJRVS2CTBRYNOLXAZFQ7QTGQ, seg_pos={0 0}, upload=473082]: Done uploading piece. err: <nil> Dec 28 02:56:36.483557859 (539h27m36.354640894s) [noise=true , address=145.79.227.6:4257 , piece_id=I5A533FIFUAZ7JRH2E4TGSRAU26E6SL2XMWM4JS4F4TOICQ2PIMA, seg_pos={0 0}, upload=473082]: Done uploading piece. err: <nil> Dec 28 02:56:36.488186192 (539h27m36.359269237s) [noise=true , address=145.79.227.16:4912 , piece_id=JQK25B3PCURS5VY4CHCV4J6BY2ITJ45CVMEX3LQ4D4LGDWYL7ORA, seg_pos={0 0}, upload=473082]: Done uploading piece. err: <nil> Dec 28 02:56:36.706859552 (539h27m36.577942587s) [noise=true , address=77.90.31.15:4700 , piece_id=UA4IVYA4WQT2Z32RYRMKLP4L3ZU7ZQSLCAR4XZPWLEB2CZHI3JGQ, seg_pos={0 0}, upload=473082]: Done uploading piece. err: <nil> Dec 28 02:56:36.755690755 (539h27m36.62677379s) [noise=true , address=77.90.31.13:4626 , piece_id=X3VIV5R6JUWRAGR6MF6QMFWVJDUJQ6VYBFN2Y7473Y3YOJZ35DUQ, seg_pos={0 0}, upload=473082]: Done uploading piece. err: <nil> Dec 28 02:56:36.780557556 (539h27m36.651640601s) [noise=true , address=45.91.116.5:4299 , piece_id=5JHPHTPARWXCFHZB23KGJYPYATO2ANXGHQQNXUDVFGQ6BFK2PL2A, seg_pos={0 0}, upload=473082]: Done uploading piece. err: <nil> Dec 28 02:56:36.781411582 (539h27m36.652494617s) [noise=true , address=77.90.31.11:4556 , piece_id=KAPN4CPMWM4OUQW6AVNEG7SMRZM4GQACJ72OFQSVQPVDES5D6CGA, seg_pos={0 0}, upload=473082]: Done uploading piece. err: <nil> Dec 28 02:56:36.785529403 (539h27m36.656612438s) [noise=true , address=45.91.116.13:4156 , piece_id=LKWX4FCKPWRDHKGLSRECHYCKJZS77COBNAHB2M2XE2BL2M6VY4WQ, seg_pos={0 0}, upload=473082]: Done uploading piece. err: <nil> Dec 28 02:56:36.882073178 (539h27m36.753156213s) [noise=true , address=77.90.31.10:4518 , piece_id=EVSYIELGBLKMOZO2JJNO4ZUMN2QDS7KJEEFJPEOHA7DUGZ4DYQPQ, seg_pos={0 0}, upload=473082]: Done uploading piece. err: <nil> Dec 28 02:56:37.535286023 (539h27m37.406369068s) [noise=true , address=145.79.227.15:4873 , piece_id=AAJPAAHZOWVUI3C2EWYS2EZ2QUAEQQNTATTZMNAFWMHEGRZJFS4A, seg_pos={0 0}, upload=473082]: Done uploading piece. err: <nil> Dec 28 02:56:37.542816984 (539h27m37.413900019s) [noise=true , address=45.91.116.4:4937 , piece_id=3DGORE2BAPFSTMP6PNEUTKPS6UVNWVLT4NVX7CH3CPDXEBZK4PYA, seg_pos={0 0}, upload=473082]: Done uploading piece. err: <nil> Dec 28 02:56:37.542835679 (539h27m37.413918714s) [seg_pos={0 0}, upload=473082]: Segment reached optimal threshold of 13 pieces. Dec 28 02:56:37.543013197 (539h27m37.414096232s) [noise=true , address=45.91.116.6:4974 , piece_id=JPAJR6HV36LLIPDEGEQQLJ5JBE32PQZJ5USSQD4BG7FFJRI7MWTQ, seg_pos={0 0}, upload=473082]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Dec 28 02:56:37.543019460 (539h27m37.414102515s) [noise=true , address=45.91.116.6:4974 , piece_id=JPAJR6HV36LLIPDEGEQQLJ5JBE32PQZJ5USSQD4BG7FFJRI7MWTQ, seg_pos={0 0}, upload=473082]: Long tail cancellation due to optimal threshold Dec 28 02:56:37.543031943 (539h27m37.414114978s) [noise=true , address=145.79.227.37:4565 , piece_id=ZKAKQFYIFELEZ6HA55XXA3XRN3QVAB5PLE7TZISWBDWRARC5PTAQ, seg_pos={0 0}, upload=473082]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nvvNczHtQrSEJuJbErnjiAWa4BRDzBrQmBNbB7RrrqN1JWNHT): protocol: expected piece hash; context canceled; EOF Dec 28 02:56:37.543040399 (539h27m37.414123434s) [noise=true , address=145.79.227.37:4565 , piece_id=ZKAKQFYIFELEZ6HA55XXA3XRN3QVAB5PLE7TZISWBDWRARC5PTAQ, seg_pos={0 0}, upload=473082]: Long tail cancellation due to optimal threshold Dec 28 02:56:37.543209812 (539h27m37.414292857s) [noise=true , address=145.79.227.46:4602 , piece_id=NGLL5MPL2PUA6W4UDTUKR7GJVZIN2JUJHCKISVPVWIB3ZMTOGUWA, seg_pos={0 0}, upload=473082]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kgUqvw1r3gSKw1bNjzys3AiAJXdkAnSafmvcPgNVPBbN9JS1d): protocol: expected piece hash; context canceled; EOF Dec 28 02:56:37.543233117 (539h27m37.414316152s) [noise=true , address=145.79.227.46:4602 , piece_id=NGLL5MPL2PUA6W4UDTUKR7GJVZIN2JUJHCKISVPVWIB3ZMTOGUWA, seg_pos={0 0}, upload=473082]: Long tail cancellation due to optimal threshold Dec 28 02:56:37.543248826 (539h27m37.414331861s) [seg_pos={0 0}, upload=473082]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 02:56:37.543312718 (539h27m37.414395753s) [upload=473082]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 02:56:37.543317737 (539h27m37.414400772s) [upload=473082]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 02:56:37.831943248 (539h27m37.703026283s) [upload=473082]: Done uploading segments... Dec 28 02:56:37.831961653 (539h27m37.703044688s) [upload=473082]: Upload finished. err: <nil> Duration: 6.671006198s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 44 out of 60 is failed (threshold=3) [X.XXX.XXX.X.....XXX.XXXXXX.XXXX.XXXX..X..XXXXX.XXXXXXXXXXXXX] duration: 30.003298691s trace: https://jaeger.dev.storj.io/trace/6ede6e48b6098cbc FAIL (2025-12-28 02:57:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: UA4IVYA4WQT2Z32RYRMKLP4L3ZU7ZQSLCAR4XZPWLEB2CZHI3JGQ) context deadline exceeded; piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: X3VIV5R6JUWRAGR6MF6QMFWVJDUJQ6VYBFN2Y7473Y3YOJZ35DUQ) context deadline exceeded ======LOGS====== Dec 28 02:56:34.207261982 (539h27m34.078345027s) [upload=473082]: Starting upload Dec 28 02:56:34.207317187 (539h27m34.078400222s) [upload=473082]: Done starting upload Dec 28 02:56:34.207386559 (539h27m34.078469594s) [upload=473082]: Uploading segments... Dec 28 02:56:34.207528460 (539h27m34.078611495s) [upload=473082]: Got next segment. Inline: false Dec 28 02:56:34.207606779 (539h27m34.078689834s) [upload=473082]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 02:56:34.207613892 (539h27m34.078696957s) [upload=473082]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 02:56:34.455905841 (539h27m34.326988876s) [seg_pos={0 0}, upload=473082]: Begin upload segment... Dec 28 02:56:34.455960897 (539h27m34.327043932s) [seg_pos={0 0}, upload=473082]: Done begin upload segment. Dec 28 02:56:34.455995623 (539h27m34.327078658s) [noise=true , address=145.79.227.11:4731 , piece_id=LLDKPJ37IYFHHREFM52EYGXHVFOEGJRVS2CTBRYNOLXAZFQ7QTGQ, seg_pos={0 0}, upload=473082]: Uploading piece... Dec 28 02:56:34.456072940 (539h27m34.327155985s) [noise=true , address=145.79.227.5:4222 , piece_id=FT2T2KRZPCHMSV6MAMY2QVORPOA3GP544PDTFGM62X6PIRRXDY3A, seg_pos={0 0}, upload=473082]: Uploading piece... Dec 28 02:56:34.456878313 (539h27m34.327961348s) [noise=true , address=145.79.227.6:4257 , piece_id=I5A533FIFUAZ7JRH2E4TGSRAU26E6SL2XMWM4JS4F4TOICQ2PIMA, seg_pos={0 0}, upload=473082]: Uploading piece... Dec 28 02:56:34.457125664 (539h27m34.328208709s) [noise=true , address=145.79.227.37:4565 , piece_id=ZKAKQFYIFELEZ6HA55XXA3XRN3QVAB5PLE7TZISWBDWRARC5PTAQ, seg_pos={0 0}, upload=473082]: Uploading piece... Dec 28 02:56:34.457520587 (539h27m34.328603622s) [noise=true , address=77.90.31.11:4556 , piece_id=KAPN4CPMWM4OUQW6AVNEG7SMRZM4GQACJ72OFQSVQPVDES5D6CGA, seg_pos={0 0}, upload=473082]: Uploading piece... Dec 28 02:56:34.457911240 (539h27m34.328994285s) [noise=true , address=145.79.227.16:4912 , piece_id=JQK25B3PCURS5VY4CHCV4J6BY2ITJ45CVMEX3LQ4D4LGDWYL7ORA, seg_pos={0 0}, upload=473082]: Uploading piece... Dec 28 02:56:34.458058992 (539h27m34.329142027s) [noise=true , address=77.90.31.13:4626 , piece_id=X3VIV5R6JUWRAGR6MF6QMFWVJDUJQ6VYBFN2Y7473Y3YOJZ35DUQ, seg_pos={0 0}, upload=473082]: Uploading piece... Dec 28 02:56:34.458218105 (539h27m34.32930114s) [noise=true , address=45.91.116.5:4299 , piece_id=5JHPHTPARWXCFHZB23KGJYPYATO2ANXGHQQNXUDVFGQ6BFK2PL2A, seg_pos={0 0}, upload=473082]: Uploading piece... Dec 28 02:56:34.458465135 (539h27m34.32954817s) [noise=true , address=45.91.116.13:4156 , piece_id=LKWX4FCKPWRDHKGLSRECHYCKJZS77COBNAHB2M2XE2BL2M6VY4WQ, seg_pos={0 0}, upload=473082]: Uploading piece... Dec 28 02:56:34.458738857 (539h27m34.329821892s) [noise=true , address=45.91.116.4:4937 , piece_id=3DGORE2BAPFSTMP6PNEUTKPS6UVNWVLT4NVX7CH3CPDXEBZK4PYA, seg_pos={0 0}, upload=473082]: Uploading piece... Dec 28 02:56:34.459059868 (539h27m34.330142903s) [noise=true , address=77.90.31.10:4518 , piece_id=EVSYIELGBLKMOZO2JJNO4ZUMN2QDS7KJEEFJPEOHA7DUGZ4DYQPQ, seg_pos={0 0}, upload=473082]: Uploading piece... Dec 28 02:56:34.459065288 (539h27m34.330148323s) [noise=true , address=45.91.116.6:4974 , piece_id=JPAJR6HV36LLIPDEGEQQLJ5JBE32PQZJ5USSQD4BG7FFJRI7MWTQ, seg_pos={0 0}, upload=473082]: Uploading piece... Dec 28 02:56:34.459353156 (539h27m34.330436191s) [noise=true , address=145.79.227.46:4602 , piece_id=NGLL5MPL2PUA6W4UDTUKR7GJVZIN2JUJHCKISVPVWIB3ZMTOGUWA, seg_pos={0 0}, upload=473082]: Uploading piece... Dec 28 02:56:34.459477603 (539h27m34.330560648s) [noise=true , address=145.79.227.15:4873 , piece_id=AAJPAAHZOWVUI3C2EWYS2EZ2QUAEQQNTATTZMNAFWMHEGRZJFS4A, seg_pos={0 0}, upload=473082]: Uploading piece... Dec 28 02:56:34.459724503 (539h27m34.330807548s) [noise=true , address=77.90.31.15:4700 , piece_id=UA4IVYA4WQT2Z32RYRMKLP4L3ZU7ZQSLCAR4XZPWLEB2CZHI3JGQ, seg_pos={0 0}, upload=473082]: Uploading piece... Dec 28 02:56:34.459996251 (539h27m34.331079296s) [noise=true , address=145.79.227.51:4409 , piece_id=SB3LIQDXCVDOHZDTW7PB46XMRRWQ6LD3PW3MPHAEKJ3EX42UCLSQ, seg_pos={0 0}, upload=473082]: Uploading piece... Dec 28 02:56:36.141335781 (539h27m36.012418816s) [upload=473082]: Next returned nil segment Dec 28 02:56:36.141352292 (539h27m36.012435337s) [upload=473082]: Waiting for error group managing segments... Dec 28 02:56:36.448609491 (539h27m36.319692536s) [noise=true , address=145.79.227.5:4222 , piece_id=FT2T2KRZPCHMSV6MAMY2QVORPOA3GP544PDTFGM62X6PIRRXDY3A, seg_pos={0 0}, upload=473082]: Done uploading piece. err: <nil> Dec 28 02:56:36.461384086 (539h27m36.332467121s) [noise=true , address=145.79.227.51:4409 , piece_id=SB3LIQDXCVDOHZDTW7PB46XMRRWQ6LD3PW3MPHAEKJ3EX42UCLSQ, seg_pos={0 0}, upload=473082]: Done uploading piece. err: <nil> Dec 28 02:56:36.475594144 (539h27m36.34667718s) [noise=true , address=145.79.227.11:4731 , piece_id=LLDKPJ37IYFHHREFM52EYGXHVFOEGJRVS2CTBRYNOLXAZFQ7QTGQ, seg_pos={0 0}, upload=473082]: Done uploading piece. err: <nil> Dec 28 02:56:36.483557859 (539h27m36.354640894s) [noise=true , address=145.79.227.6:4257 , piece_id=I5A533FIFUAZ7JRH2E4TGSRAU26E6SL2XMWM4JS4F4TOICQ2PIMA, seg_pos={0 0}, upload=473082]: Done uploading piece. err: <nil> Dec 28 02:56:36.488186192 (539h27m36.359269237s) [noise=true , address=145.79.227.16:4912 , piece_id=JQK25B3PCURS5VY4CHCV4J6BY2ITJ45CVMEX3LQ4D4LGDWYL7ORA, seg_pos={0 0}, upload=473082]: Done uploading piece. err: <nil> Dec 28 02:56:36.706859552 (539h27m36.577942587s) [noise=true , address=77.90.31.15:4700 , piece_id=UA4IVYA4WQT2Z32RYRMKLP4L3ZU7ZQSLCAR4XZPWLEB2CZHI3JGQ, seg_pos={0 0}, upload=473082]: Done uploading piece. err: <nil> Dec 28 02:56:36.755690755 (539h27m36.62677379s) [noise=true , address=77.90.31.13:4626 , piece_id=X3VIV5R6JUWRAGR6MF6QMFWVJDUJQ6VYBFN2Y7473Y3YOJZ35DUQ, seg_pos={0 0}, upload=473082]: Done uploading piece. err: <nil> Dec 28 02:56:36.780557556 (539h27m36.651640601s) [noise=true , address=45.91.116.5:4299 , piece_id=5JHPHTPARWXCFHZB23KGJYPYATO2ANXGHQQNXUDVFGQ6BFK2PL2A, seg_pos={0 0}, upload=473082]: Done uploading piece. err: <nil> Dec 28 02:56:36.781411582 (539h27m36.652494617s) [noise=true , address=77.90.31.11:4556 , piece_id=KAPN4CPMWM4OUQW6AVNEG7SMRZM4GQACJ72OFQSVQPVDES5D6CGA, seg_pos={0 0}, upload=473082]: Done uploading piece. err: <nil> Dec 28 02:56:36.785529403 (539h27m36.656612438s) [noise=true , address=45.91.116.13:4156 , piece_id=LKWX4FCKPWRDHKGLSRECHYCKJZS77COBNAHB2M2XE2BL2M6VY4WQ, seg_pos={0 0}, upload=473082]: Done uploading piece. err: <nil> Dec 28 02:56:36.882073178 (539h27m36.753156213s) [noise=true , address=77.90.31.10:4518 , piece_id=EVSYIELGBLKMOZO2JJNO4ZUMN2QDS7KJEEFJPEOHA7DUGZ4DYQPQ, seg_pos={0 0}, upload=473082]: Done uploading piece. err: <nil> Dec 28 02:56:37.535286023 (539h27m37.406369068s) [noise=true , address=145.79.227.15:4873 , piece_id=AAJPAAHZOWVUI3C2EWYS2EZ2QUAEQQNTATTZMNAFWMHEGRZJFS4A, seg_pos={0 0}, upload=473082]: Done uploading piece. err: <nil> Dec 28 02:56:37.542816984 (539h27m37.413900019s) [noise=true , address=45.91.116.4:4937 , piece_id=3DGORE2BAPFSTMP6PNEUTKPS6UVNWVLT4NVX7CH3CPDXEBZK4PYA, seg_pos={0 0}, upload=473082]: Done uploading piece. err: <nil> Dec 28 02:56:37.542835679 (539h27m37.413918714s) [seg_pos={0 0}, upload=473082]: Segment reached optimal threshold of 13 pieces. Dec 28 02:56:37.543013197 (539h27m37.414096232s) [noise=true , address=45.91.116.6:4974 , piece_id=JPAJR6HV36LLIPDEGEQQLJ5JBE32PQZJ5USSQD4BG7FFJRI7MWTQ, seg_pos={0 0}, upload=473082]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Dec 28 02:56:37.543019460 (539h27m37.414102515s) [noise=true , address=45.91.116.6:4974 , piece_id=JPAJR6HV36LLIPDEGEQQLJ5JBE32PQZJ5USSQD4BG7FFJRI7MWTQ, seg_pos={0 0}, upload=473082]: Long tail cancellation due to optimal threshold Dec 28 02:56:37.543031943 (539h27m37.414114978s) [noise=true , address=145.79.227.37:4565 , piece_id=ZKAKQFYIFELEZ6HA55XXA3XRN3QVAB5PLE7TZISWBDWRARC5PTAQ, seg_pos={0 0}, upload=473082]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nvvNczHtQrSEJuJbErnjiAWa4BRDzBrQmBNbB7RrrqN1JWNHT): protocol: expected piece hash; context canceled; EOF Dec 28 02:56:37.543040399 (539h27m37.414123434s) [noise=true , address=145.79.227.37:4565 , piece_id=ZKAKQFYIFELEZ6HA55XXA3XRN3QVAB5PLE7TZISWBDWRARC5PTAQ, seg_pos={0 0}, upload=473082]: Long tail cancellation due to optimal threshold Dec 28 02:56:37.543209812 (539h27m37.414292857s) [noise=true , address=145.79.227.46:4602 , piece_id=NGLL5MPL2PUA6W4UDTUKR7GJVZIN2JUJHCKISVPVWIB3ZMTOGUWA, seg_pos={0 0}, upload=473082]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kgUqvw1r3gSKw1bNjzys3AiAJXdkAnSafmvcPgNVPBbN9JS1d): protocol: expected piece hash; context canceled; EOF Dec 28 02:56:37.543233117 (539h27m37.414316152s) [noise=true , address=145.79.227.46:4602 , piece_id=NGLL5MPL2PUA6W4UDTUKR7GJVZIN2JUJHCKISVPVWIB3ZMTOGUWA, seg_pos={0 0}, upload=473082]: Long tail cancellation due to optimal threshold Dec 28 02:56:37.543248826 (539h27m37.414331861s) [seg_pos={0 0}, upload=473082]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 02:56:37.543312718 (539h27m37.414395753s) [upload=473082]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 02:56:37.543317737 (539h27m37.414400772s) [upload=473082]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 02:56:37.831943248 (539h27m37.703026283s) [upload=473082]: Done uploading segments... Dec 28 02:56:37.831961653 (539h27m37.703044688s) [upload=473082]: Upload finished. err: <nil> Duration: 30.003320283s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [................X.....X.....................................] duration: 38.717545008s trace: https://jaeger.dev.storj.io/trace/1822be52d200e5b5 FAIL (2025-12-28 02:56:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1ZAfYZ4wgpZVvatj6wtHona3hTaPcLxAmXC1DT2uKQyo8nLQQ7, Piece ID: 5MMQ44MXRT4XEL4X6WO54WRG2XYMX3AN3FNQ2H2XIUTKBXI3EEPA) context deadline exceeded; piecestore: (Node ID: 124guAM82oibYZsMho67Em6Ke6yEZXwT6LPMLLSZZBLqwoFGUkn, Piece ID: 22WJIPX4SM5NTIAW5ZRM2UKX32CTDBFCXPCCHRRWRCUCOF5CRE5Q) context deadline exceeded; piecestore: (Node ID: 1e6p1TD5CqUyWdZjkrxh2kRqMuyDiMUr9eimfeayDAKwhtBrZ8, Piece ID: XRI5EJFN7YYUG4QKLDGEFT6YAUHCUTTSXF735UDR6LZZBGMAZDJQ) context deadline exceeded ======LOGS====== 5.234848204 (538h55m35.105931239s) [noise=true , address=77.90.31.11:4572 , piece_id=5MMQ44MXRT4XEL4X6WO54WRG2XYMX3AN3FNQ2H2XIUTKBXI3EEPA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.235412057 (538h55m35.106495102s) [noise=true , address=145.79.227.22:4512 , piece_id=D46YLTIKGHG36DQ4YBKBSUDV4GGB4WVDWNOJS27IYYLN6AYZ5QFQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.235820946 (538h55m35.106903981s) [noise=true , address=77.90.31.19:4874 , piece_id=RHZUXUYMJU6RKQGSDZM5WDWIN5SQ7YIRSRATNNKT5GEMU2QH444A, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.236192192 (538h55m35.107275237s) [noise=true , address=77.90.31.16:4760 , piece_id=XRI5EJFN7YYUG4QKLDGEFT6YAUHCUTTSXF735UDR6LZZBGMAZDJQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.236564772 (538h55m35.107647817s) [noise=true , address=145.79.227.40:4062 , piece_id=PW6QWIPQWVUETLC6DJ624G6IB4P6EM4SYB72NAMQ4V6NUKNLUYLQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.236929766 (538h55m35.108012812s) [noise=true , address=64.49.13.139:4480 , piece_id=JAJY3F7IDFFMFKCK2QJV6W74RIESLRKBLXMN27Q3C45JWEPLBZLQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.237916846 (538h55m35.108999881s) [noise=true , address=145.79.227.34:4932 , piece_id=UPXSCQ45QDGPKNJXRFJ33EAA3QENVHRVQARU5TFJTEWICZNR253Q, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.238852948 (538h55m35.109935983s) [noise=true , address=45.91.116.12:4257 , piece_id=ZI67LIAHZW4G357GWKDPTB5SMXYGNXZTTTDMNAETDS2B6SWVDAEQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239152910 (538h55m35.110235945s) [noise=true , address=145.79.227.44:4222 , piece_id=CZ2CS62CTHQFAIE3S3JO2N3VJ4XIR5YJ2LMRVLLX7I4VM2VZVQVA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239409969 (538h55m35.110493004s) [noise=true , address=145.79.227.30:4773 , piece_id=KUGJXZCNMUE7WSK3VLY4AUBSOT4H6IY4XB4NCBGAGCNUG7F7IODQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239681136 (538h55m35.110764171s) [noise=true , address=77.90.31.13:4635 , piece_id=IIOY5X2W3XW4C2OFEBRCABMYFBI24RRTYNUSKN6KJU7CTUW4KONQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239954676 (538h55m35.111037711s) [noise=true , address=45.91.116.9:4281 , piece_id=JZLDW777CBX2BP7MLOQV4NFP3H4XJVCNQXTHNSUWPVDOJLPZK36A, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.240189914 (538h55m35.111272949s) [noise=true , address=145.79.227.35:4512 , piece_id=3E3VCCC45XVT5FHFDKXSTGGAGYBEML3BKU6JJTT33LCD2YMOMBZQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.240449147 (538h55m35.111532182s) [noise=true , address=45.91.116.8:4064 , piece_id=22WJIPX4SM5NTIAW5ZRM2UKX32CTDBFCXPCCHRRWRCUCOF5CRE5Q, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.240639640 (538h55m35.111722675s) [noise=true , address=145.79.227.32:4853 , piece_id=MBNCFDBQ4SZZXV4F7J5TRMOUYUQ5IPKQCGYGGM5SOIFJ6ZYCAUYA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.241137798 (538h55m35.112220833s) [noise=true , address=77.90.31.16:4751 , piece_id=N3Z2LLMCIA2F42VXZA7IDRU4XEKGOICH6TRVERODU76MFFE4QSHA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.241383797 (538h55m35.112466842s) [noise=true , address=77.90.31.18:4836 , piece_id=BSVEFI52EKN44CMINFUSZBNOH53IW5TXLVTF5IH6ZUTNII6RRLVA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:36.592880816 (538h55m36.463963871s) [upload=472631]: Next returned nil segment Dec 28 02:24:36.592901625 (538h55m36.46398467s) [upload=472631]: Waiting for error group managing segments... Dec 28 02:24:36.946933741 (538h55m36.818016786s) [noise=true , address=64.49.13.139:4480 , piece_id=JAJY3F7IDFFMFKCK2QJV6W74RIESLRKBLXMN27Q3C45JWEPLBZLQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.128189692 (538h55m36.999272737s) [noise=true , address=145.79.227.35:4512 , piece_id=3E3VCCC45XVT5FHFDKXSTGGAGYBEML3BKU6JJTT33LCD2YMOMBZQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.160760084 (538h55m37.031843119s) [noise=true , address=145.79.227.34:4932 , piece_id=UPXSCQ45QDGPKNJXRFJ33EAA3QENVHRVQARU5TFJTEWICZNR253Q, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.288838207 (538h55m37.159921252s) [noise=true , address=77.90.31.11:4572 , piece_id=5MMQ44MXRT4XEL4X6WO54WRG2XYMX3AN3FNQ2H2XIUTKBXI3EEPA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.319996310 (538h55m37.191079345s) [noise=true , address=145.79.227.22:4512 , piece_id=D46YLTIKGHG36DQ4YBKBSUDV4GGB4WVDWNOJS27IYYLN6AYZ5QFQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.320513273 (538h55m37.191596319s) [noise=true , address=77.90.31.18:4836 , piece_id=BSVEFI52EKN44CMINFUSZBNOH53IW5TXLVTF5IH6ZUTNII6RRLVA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.336522926 (538h55m37.207605971s) [noise=true , address=145.79.227.32:4853 , piece_id=MBNCFDBQ4SZZXV4F7J5TRMOUYUQ5IPKQCGYGGM5SOIFJ6ZYCAUYA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.384868031 (538h55m37.255951066s) [noise=true , address=77.90.31.16:4751 , piece_id=N3Z2LLMCIA2F42VXZA7IDRU4XEKGOICH6TRVERODU76MFFE4QSHA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.453563554 (538h55m37.324646599s) [noise=true , address=45.91.116.8:4064 , piece_id=22WJIPX4SM5NTIAW5ZRM2UKX32CTDBFCXPCCHRRWRCUCOF5CRE5Q, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.479454839 (538h55m37.350537884s) [noise=true , address=45.91.116.12:4257 , piece_id=ZI67LIAHZW4G357GWKDPTB5SMXYGNXZTTTDMNAETDS2B6SWVDAEQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.659710174 (538h55m37.530793219s) [noise=true , address=77.90.31.13:4635 , piece_id=IIOY5X2W3XW4C2OFEBRCABMYFBI24RRTYNUSKN6KJU7CTUW4KONQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.715039128 (538h55m37.586122163s) [noise=true , address=77.90.31.16:4760 , piece_id=XRI5EJFN7YYUG4QKLDGEFT6YAUHCUTTSXF735UDR6LZZBGMAZDJQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.977473406 (538h55m37.848556441s) [noise=true , address=77.90.31.19:4874 , piece_id=RHZUXUYMJU6RKQGSDZM5WDWIN5SQ7YIRSRATNNKT5GEMU2QH444A, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:38.016302745 (538h55m37.88738578s) [noise=true , address=145.79.227.30:4773 , piece_id=KUGJXZCNMUE7WSK3VLY4AUBSOT4H6IY4XB4NCBGAGCNUG7F7IODQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:38.016326982 (538h55m37.887410027s) [seg_pos={0 0}, upload=472631]: Segment reached optimal threshold of 14 pieces. Dec 28 02:24:38.016585353 (538h55m37.887668398s) [noise=true , address=145.79.227.30:4766 , piece_id=67M2DMPAVWIQKQSWUAG73D4E4NRRSG4XEVPOUVIJPEQOUV7A6VVA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1k6zdamuKPkBdzyNZNWx4FCmaTkv33PchcwvTRR8bgcETBs4K7): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016594450 (538h55m37.887677495s) [noise=true , address=145.79.227.30:4766 , piece_id=67M2DMPAVWIQKQSWUAG73D4E4NRRSG4XEVPOUVIJPEQOUV7A6VVA, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016627093 (538h55m37.887710148s) [noise=true , address=45.91.116.9:4281 , piece_id=JZLDW777CBX2BP7MLOQV4NFP3H4XJVCNQXTHNSUWPVDOJLPZK36A, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gxf7DpJZthyR6f1PExX56tJavjhPw9Gxb4h8mtmeYKW3t3tS1): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016635258 (538h55m37.887718293s) [noise=true , address=45.91.116.9:4281 , piece_id=JZLDW777CBX2BP7MLOQV4NFP3H4XJVCNQXTHNSUWPVDOJLPZK36A, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016871298 (538h55m37.887954363s) [noise=true , address=145.79.227.44:4222 , piece_id=CZ2CS62CTHQFAIE3S3JO2N3VJ4XIR5YJ2LMRVLLX7I4VM2VZVQVA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZKW35KvrsoYRefbYeUFZo8bh4XZYycNKGEvTFjzibezjXgj3e): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016880124 (538h55m37.887963169s) [noise=true , address=145.79.227.44:4222 , piece_id=CZ2CS62CTHQFAIE3S3JO2N3VJ4XIR5YJ2LMRVLLX7I4VM2VZVQVA, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016934839 (538h55m37.888017884s) [noise=true , address=145.79.227.40:4062 , piece_id=PW6QWIPQWVUETLC6DJ624G6IB4P6EM4SYB72NAMQ4V6NUKNLUYLQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016943425 (538h55m37.88802649s) [noise=true , address=145.79.227.40:4062 , piece_id=PW6QWIPQWVUETLC6DJ624G6IB4P6EM4SYB72NAMQ4V6NUKNLUYLQ, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016968252 (538h55m37.888051297s) [seg_pos={0 0}, upload=472631]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 02:24:38.017067822 (538h55m37.888150867s) [upload=472631]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 02:24:38.017074244 (538h55m37.888157289s) [upload=472631]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 02:24:38.304306667 (538h55m38.175389712s) [upload=472631]: Done uploading segments... Dec 28 02:24:38.304329641 (538h55m38.175412686s) [upload=472631]: Upload finished. err: <nil> Duration: 38.717561408s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [...................X................X....X............X.....] duration: 4.323086887s trace: https://jaeger.dev.storj.io/trace/20df15b758999ef2 FAIL (2025-12-28 02:56:39 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.323109991s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................X.....X....................................] duration: 37.448259736s trace: https://jaeger.dev.storj.io/trace/1822be52d200e5b5 FAIL (2025-12-28 02:56:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1ZAfYZ4wgpZVvatj6wtHona3hTaPcLxAmXC1DT2uKQyo8nLQQ7, Piece ID: 5MMQ44MXRT4XEL4X6WO54WRG2XYMX3AN3FNQ2H2XIUTKBXI3EEPA) context deadline exceeded; piecestore: (Node ID: 124guAM82oibYZsMho67Em6Ke6yEZXwT6LPMLLSZZBLqwoFGUkn, Piece ID: 22WJIPX4SM5NTIAW5ZRM2UKX32CTDBFCXPCCHRRWRCUCOF5CRE5Q) context deadline exceeded; piecestore: (Node ID: 1e6p1TD5CqUyWdZjkrxh2kRqMuyDiMUr9eimfeayDAKwhtBrZ8, Piece ID: XRI5EJFN7YYUG4QKLDGEFT6YAUHCUTTSXF735UDR6LZZBGMAZDJQ) context deadline exceeded ======LOGS====== 5.234848204 (538h55m35.105931239s) [noise=true , address=77.90.31.11:4572 , piece_id=5MMQ44MXRT4XEL4X6WO54WRG2XYMX3AN3FNQ2H2XIUTKBXI3EEPA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.235412057 (538h55m35.106495102s) [noise=true , address=145.79.227.22:4512 , piece_id=D46YLTIKGHG36DQ4YBKBSUDV4GGB4WVDWNOJS27IYYLN6AYZ5QFQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.235820946 (538h55m35.106903981s) [noise=true , address=77.90.31.19:4874 , piece_id=RHZUXUYMJU6RKQGSDZM5WDWIN5SQ7YIRSRATNNKT5GEMU2QH444A, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.236192192 (538h55m35.107275237s) [noise=true , address=77.90.31.16:4760 , piece_id=XRI5EJFN7YYUG4QKLDGEFT6YAUHCUTTSXF735UDR6LZZBGMAZDJQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.236564772 (538h55m35.107647817s) [noise=true , address=145.79.227.40:4062 , piece_id=PW6QWIPQWVUETLC6DJ624G6IB4P6EM4SYB72NAMQ4V6NUKNLUYLQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.236929766 (538h55m35.108012812s) [noise=true , address=64.49.13.139:4480 , piece_id=JAJY3F7IDFFMFKCK2QJV6W74RIESLRKBLXMN27Q3C45JWEPLBZLQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.237916846 (538h55m35.108999881s) [noise=true , address=145.79.227.34:4932 , piece_id=UPXSCQ45QDGPKNJXRFJ33EAA3QENVHRVQARU5TFJTEWICZNR253Q, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.238852948 (538h55m35.109935983s) [noise=true , address=45.91.116.12:4257 , piece_id=ZI67LIAHZW4G357GWKDPTB5SMXYGNXZTTTDMNAETDS2B6SWVDAEQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239152910 (538h55m35.110235945s) [noise=true , address=145.79.227.44:4222 , piece_id=CZ2CS62CTHQFAIE3S3JO2N3VJ4XIR5YJ2LMRVLLX7I4VM2VZVQVA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239409969 (538h55m35.110493004s) [noise=true , address=145.79.227.30:4773 , piece_id=KUGJXZCNMUE7WSK3VLY4AUBSOT4H6IY4XB4NCBGAGCNUG7F7IODQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239681136 (538h55m35.110764171s) [noise=true , address=77.90.31.13:4635 , piece_id=IIOY5X2W3XW4C2OFEBRCABMYFBI24RRTYNUSKN6KJU7CTUW4KONQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239954676 (538h55m35.111037711s) [noise=true , address=45.91.116.9:4281 , piece_id=JZLDW777CBX2BP7MLOQV4NFP3H4XJVCNQXTHNSUWPVDOJLPZK36A, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.240189914 (538h55m35.111272949s) [noise=true , address=145.79.227.35:4512 , piece_id=3E3VCCC45XVT5FHFDKXSTGGAGYBEML3BKU6JJTT33LCD2YMOMBZQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.240449147 (538h55m35.111532182s) [noise=true , address=45.91.116.8:4064 , piece_id=22WJIPX4SM5NTIAW5ZRM2UKX32CTDBFCXPCCHRRWRCUCOF5CRE5Q, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.240639640 (538h55m35.111722675s) [noise=true , address=145.79.227.32:4853 , piece_id=MBNCFDBQ4SZZXV4F7J5TRMOUYUQ5IPKQCGYGGM5SOIFJ6ZYCAUYA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.241137798 (538h55m35.112220833s) [noise=true , address=77.90.31.16:4751 , piece_id=N3Z2LLMCIA2F42VXZA7IDRU4XEKGOICH6TRVERODU76MFFE4QSHA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.241383797 (538h55m35.112466842s) [noise=true , address=77.90.31.18:4836 , piece_id=BSVEFI52EKN44CMINFUSZBNOH53IW5TXLVTF5IH6ZUTNII6RRLVA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:36.592880816 (538h55m36.463963871s) [upload=472631]: Next returned nil segment Dec 28 02:24:36.592901625 (538h55m36.46398467s) [upload=472631]: Waiting for error group managing segments... Dec 28 02:24:36.946933741 (538h55m36.818016786s) [noise=true , address=64.49.13.139:4480 , piece_id=JAJY3F7IDFFMFKCK2QJV6W74RIESLRKBLXMN27Q3C45JWEPLBZLQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.128189692 (538h55m36.999272737s) [noise=true , address=145.79.227.35:4512 , piece_id=3E3VCCC45XVT5FHFDKXSTGGAGYBEML3BKU6JJTT33LCD2YMOMBZQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.160760084 (538h55m37.031843119s) [noise=true , address=145.79.227.34:4932 , piece_id=UPXSCQ45QDGPKNJXRFJ33EAA3QENVHRVQARU5TFJTEWICZNR253Q, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.288838207 (538h55m37.159921252s) [noise=true , address=77.90.31.11:4572 , piece_id=5MMQ44MXRT4XEL4X6WO54WRG2XYMX3AN3FNQ2H2XIUTKBXI3EEPA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.319996310 (538h55m37.191079345s) [noise=true , address=145.79.227.22:4512 , piece_id=D46YLTIKGHG36DQ4YBKBSUDV4GGB4WVDWNOJS27IYYLN6AYZ5QFQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.320513273 (538h55m37.191596319s) [noise=true , address=77.90.31.18:4836 , piece_id=BSVEFI52EKN44CMINFUSZBNOH53IW5TXLVTF5IH6ZUTNII6RRLVA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.336522926 (538h55m37.207605971s) [noise=true , address=145.79.227.32:4853 , piece_id=MBNCFDBQ4SZZXV4F7J5TRMOUYUQ5IPKQCGYGGM5SOIFJ6ZYCAUYA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.384868031 (538h55m37.255951066s) [noise=true , address=77.90.31.16:4751 , piece_id=N3Z2LLMCIA2F42VXZA7IDRU4XEKGOICH6TRVERODU76MFFE4QSHA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.453563554 (538h55m37.324646599s) [noise=true , address=45.91.116.8:4064 , piece_id=22WJIPX4SM5NTIAW5ZRM2UKX32CTDBFCXPCCHRRWRCUCOF5CRE5Q, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.479454839 (538h55m37.350537884s) [noise=true , address=45.91.116.12:4257 , piece_id=ZI67LIAHZW4G357GWKDPTB5SMXYGNXZTTTDMNAETDS2B6SWVDAEQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.659710174 (538h55m37.530793219s) [noise=true , address=77.90.31.13:4635 , piece_id=IIOY5X2W3XW4C2OFEBRCABMYFBI24RRTYNUSKN6KJU7CTUW4KONQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.715039128 (538h55m37.586122163s) [noise=true , address=77.90.31.16:4760 , piece_id=XRI5EJFN7YYUG4QKLDGEFT6YAUHCUTTSXF735UDR6LZZBGMAZDJQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.977473406 (538h55m37.848556441s) [noise=true , address=77.90.31.19:4874 , piece_id=RHZUXUYMJU6RKQGSDZM5WDWIN5SQ7YIRSRATNNKT5GEMU2QH444A, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:38.016302745 (538h55m37.88738578s) [noise=true , address=145.79.227.30:4773 , piece_id=KUGJXZCNMUE7WSK3VLY4AUBSOT4H6IY4XB4NCBGAGCNUG7F7IODQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:38.016326982 (538h55m37.887410027s) [seg_pos={0 0}, upload=472631]: Segment reached optimal threshold of 14 pieces. Dec 28 02:24:38.016585353 (538h55m37.887668398s) [noise=true , address=145.79.227.30:4766 , piece_id=67M2DMPAVWIQKQSWUAG73D4E4NRRSG4XEVPOUVIJPEQOUV7A6VVA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1k6zdamuKPkBdzyNZNWx4FCmaTkv33PchcwvTRR8bgcETBs4K7): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016594450 (538h55m37.887677495s) [noise=true , address=145.79.227.30:4766 , piece_id=67M2DMPAVWIQKQSWUAG73D4E4NRRSG4XEVPOUVIJPEQOUV7A6VVA, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016627093 (538h55m37.887710148s) [noise=true , address=45.91.116.9:4281 , piece_id=JZLDW777CBX2BP7MLOQV4NFP3H4XJVCNQXTHNSUWPVDOJLPZK36A, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gxf7DpJZthyR6f1PExX56tJavjhPw9Gxb4h8mtmeYKW3t3tS1): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016635258 (538h55m37.887718293s) [noise=true , address=45.91.116.9:4281 , piece_id=JZLDW777CBX2BP7MLOQV4NFP3H4XJVCNQXTHNSUWPVDOJLPZK36A, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016871298 (538h55m37.887954363s) [noise=true , address=145.79.227.44:4222 , piece_id=CZ2CS62CTHQFAIE3S3JO2N3VJ4XIR5YJ2LMRVLLX7I4VM2VZVQVA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZKW35KvrsoYRefbYeUFZo8bh4XZYycNKGEvTFjzibezjXgj3e): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016880124 (538h55m37.887963169s) [noise=true , address=145.79.227.44:4222 , piece_id=CZ2CS62CTHQFAIE3S3JO2N3VJ4XIR5YJ2LMRVLLX7I4VM2VZVQVA, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016934839 (538h55m37.888017884s) [noise=true , address=145.79.227.40:4062 , piece_id=PW6QWIPQWVUETLC6DJ624G6IB4P6EM4SYB72NAMQ4V6NUKNLUYLQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016943425 (538h55m37.88802649s) [noise=true , address=145.79.227.40:4062 , piece_id=PW6QWIPQWVUETLC6DJ624G6IB4P6EM4SYB72NAMQ4V6NUKNLUYLQ, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016968252 (538h55m37.888051297s) [seg_pos={0 0}, upload=472631]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 02:24:38.017067822 (538h55m37.888150867s) [upload=472631]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 02:24:38.017074244 (538h55m37.888157289s) [upload=472631]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 02:24:38.304306667 (538h55m38.175389712s) [upload=472631]: Done uploading segments... Dec 28 02:24:38.304329641 (538h55m38.175412686s) [upload=472631]: Upload finished. err: <nil> Duration: 37.448275966s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [....................X................X....X............X....] duration: 5.000830061s trace: https://jaeger.dev.storj.io/trace/20df15b758999ef2 FAIL (2025-12-28 02:55:55 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000854347s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 43 out of 60 is failed (threshold=3) [.X.XXX.XXX.X.....XXX.XXXXXX.XXXX.XXXX..X..XXXXX.XXXXXXXXXXXX] duration: 30.002649249s trace: https://jaeger.dev.storj.io/trace/12f9a1c4122d88cd FAIL (2025-12-28 02:55:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :55:22.754196588 (539h26m22.625279633s) [upload=473065]: Starting upload Dec 28 02:55:22.754286901 (539h26m22.625369946s) [upload=473065]: Done starting upload Dec 28 02:55:22.754402160 (539h26m22.625485215s) [upload=473065]: Uploading segments... Dec 28 02:55:22.754614193 (539h26m22.625697238s) [upload=473065]: Got next segment. Inline: false Dec 28 02:55:22.754678646 (539h26m22.625761691s) [upload=473065]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 02:55:22.754685108 (539h26m22.625768153s) [upload=473065]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 02:55:22.998652984 (539h26m22.869736019s) [seg_pos={0 0}, upload=473065]: Begin upload segment... Dec 28 02:55:22.998723648 (539h26m22.869806684s) [seg_pos={0 0}, upload=473065]: Done begin upload segment. Dec 28 02:55:22.998784455 (539h26m22.86986749s) [noise=true , address=145.79.227.46:4603 , piece_id=PSBMHXNF3R4YQZ4AIO4SOEHOMPATJSNWR4L723XNAM335BVDK4CA, seg_pos={0 0}, upload=473065]: Uploading piece... Dec 28 02:55:22.998847735 (539h26m22.86993077s) [noise=true , address=45.91.116.8:4048 , piece_id=3OBGCAHFR5P4IB5W434ZE2VNA7NZ6TMNEEYT4OSFRH2DRAOVNIGQ, seg_pos={0 0}, upload=473065]: Uploading piece... Dec 28 02:55:22.999419974 (539h26m22.870503009s) [noise=true , address=45.91.116.6:4976 , piece_id=26LCZDOURUMSUBMGMWZ4SVEJJ7PC547JS3QPNJJWC7JEBCERTPLQ, seg_pos={0 0}, upload=473065]: Uploading piece... Dec 28 02:55:22.999959663 (539h26m22.871042698s) [noise=true , address=145.79.227.42:4122 , piece_id=UK7QOH7IS7TZQYXLLAKWF3WUCLVR3O5OR5Y5IXK4HZFTB43JJOEA, seg_pos={0 0}, upload=473065]: Uploading piece... Dec 28 02:55:23.000382117 (539h26m22.871465172s) [noise=true , address=145.79.227.7:4295 , piece_id=6AKPZDOYLEBZWZ2UKQHE7BUTCIGREEFNCPFC7SSGSK3CIKXK6Y7Q, seg_pos={0 0}, upload=473065]: Uploading piece... Dec 28 02:55:23.000779703 (539h26m22.871862759s) [noise=true , address=145.79.227.3:4150 , piece_id=LSMILR2467YY4O67ZOVAGBG2EPAMVHDQ6KLPBGNNVZ5JCMA47BVA, seg_pos={0 0}, upload=473065]: Uploading piece... Dec 28 02:55:23.001213640 (539h26m22.872296705s) [noise=true , address=145.79.227.22:4511 , piece_id=WZZYGGVYIR4VYYWAYLCGV4R6JQOHU4OE4T3CJ2LM6KVCF7KCLAOA, seg_pos={0 0}, upload=473065]: Uploading piece... Dec 28 02:55:23.001721697 (539h26m22.872804753s) [noise=true , address=77.90.31.10:4517 , piece_id=5EHNHPMFTJL57IZDLYJHJHZCNZXPFYAVA3X3OPJTDN6E4CQLJQUA, seg_pos={0 0}, upload=473065]: Uploading piece... Dec 28 02:55:23.001742417 (539h26m22.872825482s) [noise=true , address=145.79.227.14:4838 , piece_id=EJ5WSD622Y2FAXB6O5OSJ4CPTG4ZG2KHWDWMS6VIBPXOGOTDKQ3Q, seg_pos={0 0}, upload=473065]: Uploading piece... Dec 28 02:55:23.002225106 (539h26m22.873308151s) [noise=true , address=145.79.227.11:4730 , piece_id=DBAZ3RXKY4YKGNNZ4G5SCKUJWWN4YVO7D3SMXQII6QIBYICISFBQ, seg_pos={0 0}, upload=473065]: Uploading piece... Dec 28 02:55:23.002663370 (539h26m22.873746415s) [noise=true , address=145.79.227.4:4185 , piece_id=LVHXXWBFZX22OUJGN3GBGITUPIAJPSWX4HCXZ5AR7X2YEVF5H53A, seg_pos={0 0}, upload=473065]: Uploading piece... Dec 28 02:55:23.002992417 (539h26m22.874075452s) [noise=true , address=145.79.227.48:4302 , piece_id=JO3TZC6XBEJV6MJOHSC5RCLNC4UA2SGMXM3JGXRSW3QRY6TTSPCQ, seg_pos={0 0}, upload=473065]: Uploading piece... Dec 28 02:55:23.003302608 (539h26m22.874385643s) [noise=true , address=145.79.227.50:4374 , piece_id=F4RC32KQHC3C3NTJFUUDKT2YHIWUFVY7JOWQ4HTTFBJMZPYERR2A, seg_pos={0 0}, upload=473065]: Uploading piece... Dec 28 02:55:23.003634009 (539h26m22.874717044s) [noise=true , address=145.79.227.5:4222 , piece_id=2VAGPACU335GH62VHX25ALAA7QMAN7EBNHQAF5LZLHZDFL7LVCNQ, seg_pos={0 0}, upload=473065]: Uploading piece... Dec 28 02:55:23.003971572 (539h26m22.875054607s) [noise=true , address=145.79.227.41:4085 , piece_id=MQOB4H6GFNJDIZXW3UVSF4QPIZPISAK2MPISCQNLDQM7F3IPIETA, seg_pos={0 0}, upload=473065]: Uploading piece... Dec 28 02:55:23.004964031 (539h26m22.876047076s) [noise=true , address=145.79.227.15:4874 , piece_id=YBFXXHAKDS7KLRB273OSS5IFP3LPP2BTQL35P7QTRKNSG2JKLHBQ, seg_pos={0 0}, upload=473065]: Uploading piece... Dec 28 02:55:24.851175935 (539h26m24.722258971s) [upload=473065]: Next returned nil segment Dec 28 02:55:24.851192177 (539h26m24.722275242s) [upload=473065]: Waiting for error group managing segments... Dec 28 02:55:25.153081364 (539h26m25.024164399s) [noise=true , address=145.79.227.46:4603 , piece_id=PSBMHXNF3R4YQZ4AIO4SOEHOMPATJSNWR4L723XNAM335BVDK4CA, seg_pos={0 0}, upload=473065]: Done uploading piece. err: <nil> Dec 28 02:55:25.161906719 (539h26m25.032989754s) [noise=true , address=145.79.227.50:4374 , piece_id=F4RC32KQHC3C3NTJFUUDKT2YHIWUFVY7JOWQ4HTTFBJMZPYERR2A, seg_pos={0 0}, upload=473065]: Done uploading piece. err: <nil> Dec 28 02:55:25.186139232 (539h26m25.057222267s) [noise=true , address=145.79.227.5:4222 , piece_id=2VAGPACU335GH62VHX25ALAA7QMAN7EBNHQAF5LZLHZDFL7LVCNQ, seg_pos={0 0}, upload=473065]: Done uploading piece. err: <nil> Dec 28 02:55:25.191114065 (539h26m25.0621971s) [noise=true , address=145.79.227.22:4511 , piece_id=WZZYGGVYIR4VYYWAYLCGV4R6JQOHU4OE4T3CJ2LM6KVCF7KCLAOA, seg_pos={0 0}, upload=473065]: Done uploading piece. err: <nil> Dec 28 02:55:25.194791628 (539h26m25.065874673s) [noise=true , address=145.79.227.41:4085 , piece_id=MQOB4H6GFNJDIZXW3UVSF4QPIZPISAK2MPISCQNLDQM7F3IPIETA, seg_pos={0 0}, upload=473065]: Done uploading piece. err: <nil> Dec 28 02:55:25.198929707 (539h26m25.070012742s) [noise=true , address=145.79.227.42:4122 , piece_id=UK7QOH7IS7TZQYXLLAKWF3WUCLVR3O5OR5Y5IXK4HZFTB43JJOEA, seg_pos={0 0}, upload=473065]: Done uploading piece. err: <nil> Dec 28 02:55:25.212932701 (539h26m25.084015736s) [noise=true , address=145.79.227.15:4874 , piece_id=YBFXXHAKDS7KLRB273OSS5IFP3LPP2BTQL35P7QTRKNSG2JKLHBQ, seg_pos={0 0}, upload=473065]: Done uploading piece. err: <nil> Dec 28 02:55:25.222074148 (539h26m25.093157183s) [noise=true , address=145.79.227.14:4838 , piece_id=EJ5WSD622Y2FAXB6O5OSJ4CPTG4ZG2KHWDWMS6VIBPXOGOTDKQ3Q, seg_pos={0 0}, upload=473065]: Done uploading piece. err: <nil> Dec 28 02:55:25.311013513 (539h26m25.182096548s) [noise=true , address=77.90.31.10:4517 , piece_id=5EHNHPMFTJL57IZDLYJHJHZCNZXPFYAVA3X3OPJTDN6E4CQLJQUA, seg_pos={0 0}, upload=473065]: Done uploading piece. err: <nil> Dec 28 02:55:25.466214712 (539h26m25.337297757s) [noise=true , address=145.79.227.11:4730 , piece_id=DBAZ3RXKY4YKGNNZ4G5SCKUJWWN4YVO7D3SMXQII6QIBYICISFBQ, seg_pos={0 0}, upload=473065]: Done uploading piece. err: <nil> Dec 28 02:55:25.583372951 (539h26m25.454455986s) [noise=true , address=145.79.227.4:4185 , piece_id=LVHXXWBFZX22OUJGN3GBGITUPIAJPSWX4HCXZ5AR7X2YEVF5H53A, seg_pos={0 0}, upload=473065]: Done uploading piece. err: <nil> Dec 28 02:55:25.954479940 (539h26m25.825562975s) [noise=true , address=45.91.116.8:4048 , piece_id=3OBGCAHFR5P4IB5W434ZE2VNA7NZ6TMNEEYT4OSFRH2DRAOVNIGQ, seg_pos={0 0}, upload=473065]: Done uploading piece. err: <nil> Dec 28 02:55:27.620290501 (539h26m27.491373536s) [noise=true , address=145.79.227.3:4150 , piece_id=LSMILR2467YY4O67ZOVAGBG2EPAMVHDQ6KLPBGNNVZ5JCMA47BVA, seg_pos={0 0}, upload=473065]: Done uploading piece. err: <nil> Dec 28 02:55:27.620309718 (539h26m27.491392753s) [seg_pos={0 0}, upload=473065]: Segment reached optimal threshold of 13 pieces. Dec 28 02:55:27.620459974 (539h26m27.491543009s) [noise=true , address=145.79.227.7:4295 , piece_id=6AKPZDOYLEBZWZ2UKQHE7BUTCIGREEFNCPFC7SSGSK3CIKXK6Y7Q, seg_pos={0 0}, upload=473065]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT): protocol: expected piece hash; context canceled; EOF Dec 28 02:55:27.620465444 (539h26m27.491548479s) [noise=true , address=145.79.227.7:4295 , piece_id=6AKPZDOYLEBZWZ2UKQHE7BUTCIGREEFNCPFC7SSGSK3CIKXK6Y7Q, seg_pos={0 0}, upload=473065]: Long tail cancellation due to optimal threshold Dec 28 02:55:27.620641459 (539h26m27.491724494s) [noise=true , address=45.91.116.6:4976 , piece_id=26LCZDOURUMSUBMGMWZ4SVEJJ7PC547JS3QPNJJWC7JEBCERTPLQ, seg_pos={0 0}, upload=473065]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Dec 28 02:55:27.620647111 (539h26m27.491730146s) [noise=true , address=45.91.116.6:4976 , piece_id=26LCZDOURUMSUBMGMWZ4SVEJJ7PC547JS3QPNJJWC7JEBCERTPLQ, seg_pos={0 0}, upload=473065]: Long tail cancellation due to optimal threshold Dec 28 02:55:27.620741380 (539h26m27.491824415s) [noise=true , address=145.79.227.48:4302 , piece_id=JO3TZC6XBEJV6MJOHSC5RCLNC4UA2SGMXM3JGXRSW3QRY6TTSPCQ, seg_pos={0 0}, upload=473065]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tGpuYdPLiy4ncDjxrZbQfp7Bq79NVBHDmr4vjaL4qMpudrvgR): protocol: write tcp 188.166.225.170:44866->145.79.227.48:4302: use of closed network connection; write tcp 188.166.225.170:44866->145.79.227.48:4302: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44866->145.79.227.48:4302: use of closed network connection Dec 28 02:55:27.620759284 (539h26m27.491842329s) [noise=true , address=145.79.227.48:4302 , piece_id=JO3TZC6XBEJV6MJOHSC5RCLNC4UA2SGMXM3JGXRSW3QRY6TTSPCQ, seg_pos={0 0}, upload=473065]: Long tail cancellation due to optimal threshold Dec 28 02:55:27.620795022 (539h26m27.491878057s) [seg_pos={0 0}, upload=473065]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 02:55:27.620962662 (539h26m27.492045697s) [upload=473065]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 02:55:27.620968272 (539h26m27.492051307s) [upload=473065]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 02:55:27.916071147 (539h26m27.787154182s) [upload=473065]: Done uploading segments... Dec 28 02:55:27.916092147 (539h26m27.787175182s) [upload=473065]: Upload finished. err: <nil> Duration: 30.002672573s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................X.....X...................................] duration: 34.456699029s trace: https://jaeger.dev.storj.io/trace/1822be52d200e5b5 FAIL (2025-12-28 02:55:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1ZAfYZ4wgpZVvatj6wtHona3hTaPcLxAmXC1DT2uKQyo8nLQQ7, Piece ID: 5MMQ44MXRT4XEL4X6WO54WRG2XYMX3AN3FNQ2H2XIUTKBXI3EEPA) context deadline exceeded; piecestore: (Node ID: 124guAM82oibYZsMho67Em6Ke6yEZXwT6LPMLLSZZBLqwoFGUkn, Piece ID: 22WJIPX4SM5NTIAW5ZRM2UKX32CTDBFCXPCCHRRWRCUCOF5CRE5Q) context deadline exceeded; piecestore: (Node ID: 1e6p1TD5CqUyWdZjkrxh2kRqMuyDiMUr9eimfeayDAKwhtBrZ8, Piece ID: XRI5EJFN7YYUG4QKLDGEFT6YAUHCUTTSXF735UDR6LZZBGMAZDJQ) context deadline exceeded ======LOGS====== 5.234848204 (538h55m35.105931239s) [noise=true , address=77.90.31.11:4572 , piece_id=5MMQ44MXRT4XEL4X6WO54WRG2XYMX3AN3FNQ2H2XIUTKBXI3EEPA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.235412057 (538h55m35.106495102s) [noise=true , address=145.79.227.22:4512 , piece_id=D46YLTIKGHG36DQ4YBKBSUDV4GGB4WVDWNOJS27IYYLN6AYZ5QFQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.235820946 (538h55m35.106903981s) [noise=true , address=77.90.31.19:4874 , piece_id=RHZUXUYMJU6RKQGSDZM5WDWIN5SQ7YIRSRATNNKT5GEMU2QH444A, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.236192192 (538h55m35.107275237s) [noise=true , address=77.90.31.16:4760 , piece_id=XRI5EJFN7YYUG4QKLDGEFT6YAUHCUTTSXF735UDR6LZZBGMAZDJQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.236564772 (538h55m35.107647817s) [noise=true , address=145.79.227.40:4062 , piece_id=PW6QWIPQWVUETLC6DJ624G6IB4P6EM4SYB72NAMQ4V6NUKNLUYLQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.236929766 (538h55m35.108012812s) [noise=true , address=64.49.13.139:4480 , piece_id=JAJY3F7IDFFMFKCK2QJV6W74RIESLRKBLXMN27Q3C45JWEPLBZLQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.237916846 (538h55m35.108999881s) [noise=true , address=145.79.227.34:4932 , piece_id=UPXSCQ45QDGPKNJXRFJ33EAA3QENVHRVQARU5TFJTEWICZNR253Q, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.238852948 (538h55m35.109935983s) [noise=true , address=45.91.116.12:4257 , piece_id=ZI67LIAHZW4G357GWKDPTB5SMXYGNXZTTTDMNAETDS2B6SWVDAEQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239152910 (538h55m35.110235945s) [noise=true , address=145.79.227.44:4222 , piece_id=CZ2CS62CTHQFAIE3S3JO2N3VJ4XIR5YJ2LMRVLLX7I4VM2VZVQVA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239409969 (538h55m35.110493004s) [noise=true , address=145.79.227.30:4773 , piece_id=KUGJXZCNMUE7WSK3VLY4AUBSOT4H6IY4XB4NCBGAGCNUG7F7IODQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239681136 (538h55m35.110764171s) [noise=true , address=77.90.31.13:4635 , piece_id=IIOY5X2W3XW4C2OFEBRCABMYFBI24RRTYNUSKN6KJU7CTUW4KONQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239954676 (538h55m35.111037711s) [noise=true , address=45.91.116.9:4281 , piece_id=JZLDW777CBX2BP7MLOQV4NFP3H4XJVCNQXTHNSUWPVDOJLPZK36A, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.240189914 (538h55m35.111272949s) [noise=true , address=145.79.227.35:4512 , piece_id=3E3VCCC45XVT5FHFDKXSTGGAGYBEML3BKU6JJTT33LCD2YMOMBZQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.240449147 (538h55m35.111532182s) [noise=true , address=45.91.116.8:4064 , piece_id=22WJIPX4SM5NTIAW5ZRM2UKX32CTDBFCXPCCHRRWRCUCOF5CRE5Q, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.240639640 (538h55m35.111722675s) [noise=true , address=145.79.227.32:4853 , piece_id=MBNCFDBQ4SZZXV4F7J5TRMOUYUQ5IPKQCGYGGM5SOIFJ6ZYCAUYA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.241137798 (538h55m35.112220833s) [noise=true , address=77.90.31.16:4751 , piece_id=N3Z2LLMCIA2F42VXZA7IDRU4XEKGOICH6TRVERODU76MFFE4QSHA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.241383797 (538h55m35.112466842s) [noise=true , address=77.90.31.18:4836 , piece_id=BSVEFI52EKN44CMINFUSZBNOH53IW5TXLVTF5IH6ZUTNII6RRLVA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:36.592880816 (538h55m36.463963871s) [upload=472631]: Next returned nil segment Dec 28 02:24:36.592901625 (538h55m36.46398467s) [upload=472631]: Waiting for error group managing segments... Dec 28 02:24:36.946933741 (538h55m36.818016786s) [noise=true , address=64.49.13.139:4480 , piece_id=JAJY3F7IDFFMFKCK2QJV6W74RIESLRKBLXMN27Q3C45JWEPLBZLQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.128189692 (538h55m36.999272737s) [noise=true , address=145.79.227.35:4512 , piece_id=3E3VCCC45XVT5FHFDKXSTGGAGYBEML3BKU6JJTT33LCD2YMOMBZQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.160760084 (538h55m37.031843119s) [noise=true , address=145.79.227.34:4932 , piece_id=UPXSCQ45QDGPKNJXRFJ33EAA3QENVHRVQARU5TFJTEWICZNR253Q, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.288838207 (538h55m37.159921252s) [noise=true , address=77.90.31.11:4572 , piece_id=5MMQ44MXRT4XEL4X6WO54WRG2XYMX3AN3FNQ2H2XIUTKBXI3EEPA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.319996310 (538h55m37.191079345s) [noise=true , address=145.79.227.22:4512 , piece_id=D46YLTIKGHG36DQ4YBKBSUDV4GGB4WVDWNOJS27IYYLN6AYZ5QFQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.320513273 (538h55m37.191596319s) [noise=true , address=77.90.31.18:4836 , piece_id=BSVEFI52EKN44CMINFUSZBNOH53IW5TXLVTF5IH6ZUTNII6RRLVA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.336522926 (538h55m37.207605971s) [noise=true , address=145.79.227.32:4853 , piece_id=MBNCFDBQ4SZZXV4F7J5TRMOUYUQ5IPKQCGYGGM5SOIFJ6ZYCAUYA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.384868031 (538h55m37.255951066s) [noise=true , address=77.90.31.16:4751 , piece_id=N3Z2LLMCIA2F42VXZA7IDRU4XEKGOICH6TRVERODU76MFFE4QSHA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.453563554 (538h55m37.324646599s) [noise=true , address=45.91.116.8:4064 , piece_id=22WJIPX4SM5NTIAW5ZRM2UKX32CTDBFCXPCCHRRWRCUCOF5CRE5Q, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.479454839 (538h55m37.350537884s) [noise=true , address=45.91.116.12:4257 , piece_id=ZI67LIAHZW4G357GWKDPTB5SMXYGNXZTTTDMNAETDS2B6SWVDAEQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.659710174 (538h55m37.530793219s) [noise=true , address=77.90.31.13:4635 , piece_id=IIOY5X2W3XW4C2OFEBRCABMYFBI24RRTYNUSKN6KJU7CTUW4KONQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.715039128 (538h55m37.586122163s) [noise=true , address=77.90.31.16:4760 , piece_id=XRI5EJFN7YYUG4QKLDGEFT6YAUHCUTTSXF735UDR6LZZBGMAZDJQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.977473406 (538h55m37.848556441s) [noise=true , address=77.90.31.19:4874 , piece_id=RHZUXUYMJU6RKQGSDZM5WDWIN5SQ7YIRSRATNNKT5GEMU2QH444A, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:38.016302745 (538h55m37.88738578s) [noise=true , address=145.79.227.30:4773 , piece_id=KUGJXZCNMUE7WSK3VLY4AUBSOT4H6IY4XB4NCBGAGCNUG7F7IODQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:38.016326982 (538h55m37.887410027s) [seg_pos={0 0}, upload=472631]: Segment reached optimal threshold of 14 pieces. Dec 28 02:24:38.016585353 (538h55m37.887668398s) [noise=true , address=145.79.227.30:4766 , piece_id=67M2DMPAVWIQKQSWUAG73D4E4NRRSG4XEVPOUVIJPEQOUV7A6VVA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1k6zdamuKPkBdzyNZNWx4FCmaTkv33PchcwvTRR8bgcETBs4K7): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016594450 (538h55m37.887677495s) [noise=true , address=145.79.227.30:4766 , piece_id=67M2DMPAVWIQKQSWUAG73D4E4NRRSG4XEVPOUVIJPEQOUV7A6VVA, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016627093 (538h55m37.887710148s) [noise=true , address=45.91.116.9:4281 , piece_id=JZLDW777CBX2BP7MLOQV4NFP3H4XJVCNQXTHNSUWPVDOJLPZK36A, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gxf7DpJZthyR6f1PExX56tJavjhPw9Gxb4h8mtmeYKW3t3tS1): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016635258 (538h55m37.887718293s) [noise=true , address=45.91.116.9:4281 , piece_id=JZLDW777CBX2BP7MLOQV4NFP3H4XJVCNQXTHNSUWPVDOJLPZK36A, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016871298 (538h55m37.887954363s) [noise=true , address=145.79.227.44:4222 , piece_id=CZ2CS62CTHQFAIE3S3JO2N3VJ4XIR5YJ2LMRVLLX7I4VM2VZVQVA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZKW35KvrsoYRefbYeUFZo8bh4XZYycNKGEvTFjzibezjXgj3e): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016880124 (538h55m37.887963169s) [noise=true , address=145.79.227.44:4222 , piece_id=CZ2CS62CTHQFAIE3S3JO2N3VJ4XIR5YJ2LMRVLLX7I4VM2VZVQVA, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016934839 (538h55m37.888017884s) [noise=true , address=145.79.227.40:4062 , piece_id=PW6QWIPQWVUETLC6DJ624G6IB4P6EM4SYB72NAMQ4V6NUKNLUYLQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016943425 (538h55m37.88802649s) [noise=true , address=145.79.227.40:4062 , piece_id=PW6QWIPQWVUETLC6DJ624G6IB4P6EM4SYB72NAMQ4V6NUKNLUYLQ, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016968252 (538h55m37.888051297s) [seg_pos={0 0}, upload=472631]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 02:24:38.017067822 (538h55m37.888150867s) [upload=472631]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 02:24:38.017074244 (538h55m37.888157289s) [upload=472631]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 02:24:38.304306667 (538h55m38.175389712s) [upload=472631]: Done uploading segments... Dec 28 02:24:38.304329641 (538h55m38.175412686s) [upload=472631]: Upload finished. err: <nil> Duration: 34.456728627s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 42 out of 60 is failed (threshold=3) [..X.XXX.XXX.X.....XXX.XXXXXX.XXXX.XXXX..X..XXXXX.XXXXXXXXXXX] duration: 30.003258128s trace: https://jaeger.dev.storj.io/trace/1aef93a250df4e46 FAIL (2025-12-28 02:55:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: PCDSW5XOOXIE42EUJIOH7TGZSPUNUUEYSLY7HTXDH7HNIQ25WUTQ) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: LDHG7BHFQMTZJ3D6WTFZACABMMFE2MM6LLNOKPVTFDOI7EZLK4VA) context deadline exceeded ======LOGS====== Dec 28 02:54:42.624227998 (539h25m42.495311033s) [upload=473056]: Starting upload Dec 28 02:54:42.624287962 (539h25m42.495370997s) [upload=473056]: Done starting upload Dec 28 02:54:42.624357505 (539h25m42.49544055s) [upload=473056]: Uploading segments... Dec 28 02:54:42.624465220 (539h25m42.495548265s) [upload=473056]: Got next segment. Inline: false Dec 28 02:54:42.624510416 (539h25m42.495593461s) [upload=473056]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 02:54:42.624514755 (539h25m42.4955978s) [upload=473056]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 02:54:42.867292373 (539h25m42.738375408s) [seg_pos={0 0}, upload=473056]: Begin upload segment... Dec 28 02:54:42.867333100 (539h25m42.738416135s) [seg_pos={0 0}, upload=473056]: Done begin upload segment. Dec 28 02:54:42.867355363 (539h25m42.738438398s) [noise=true , address=45.91.116.13:4155 , piece_id=GLR4PNGSR2IPOXAPGZ5OB26YTC427VE2MXMJ56RBM4WHSRN7IPNQ, seg_pos={0 0}, upload=473056]: Uploading piece... Dec 28 02:54:42.867392584 (539h25m42.738475619s) [noise=true , address=45.91.116.4:4937 , piece_id=NRFJOKJWOBQMO3TKLZIJETV3DR5WIRC7V4BUNNIL222M3CFGYPKA, seg_pos={0 0}, upload=473056]: Uploading piece... Dec 28 02:54:42.868044415 (539h25m42.739127451s) [noise=true , address=45.91.116.10:4083 , piece_id=PCDSW5XOOXIE42EUJIOH7TGZSPUNUUEYSLY7HTXDH7HNIQ25WUTQ, seg_pos={0 0}, upload=473056]: Uploading piece... Dec 28 02:54:42.868349647 (539h25m42.739432682s) [noise=true , address=145.79.227.11:4729 , piece_id=ZVGZXARRKVWH3M3O6BMFVXMRQMEQ5I3DYUEJVCW26NKJFJ4SY3EQ, seg_pos={0 0}, upload=473056]: Uploading piece... Dec 28 02:54:42.868359155 (539h25m42.73944229s) [noise=true , address=145.79.227.3:4150 , piece_id=UWTCCZKVCHMGYVYYUV4UWK7SFRIUOQ33TX6X7VMQHCG2JRNWLXXQ, seg_pos={0 0}, upload=473056]: Uploading piece... Dec 28 02:54:42.868647003 (539h25m42.739730038s) [noise=true , address=77.90.31.15:4700 , piece_id=LDHG7BHFQMTZJ3D6WTFZACABMMFE2MM6LLNOKPVTFDOI7EZLK4VA, seg_pos={0 0}, upload=473056]: Uploading piece... Dec 28 02:54:42.868668384 (539h25m42.739751419s) [noise=true , address=145.79.227.18:4007 , piece_id=RJILSRPPK7DO7ZWMYHRX7M6U7YCFQZ3G22L4KDVPHQJX46B2OGBA, seg_pos={0 0}, upload=473056]: Uploading piece... Dec 28 02:54:42.868944049 (539h25m42.740027084s) [noise=true , address=145.79.227.22:4511 , piece_id=T5HIUW6JL3PIY7HKUOCVBOV2ZJ3GCW7BUVTT2TKSLI2RIIP77IZA, seg_pos={0 0}, upload=473056]: Uploading piece... Dec 28 02:54:42.868982291 (539h25m42.740065326s) [noise=true , address=145.79.227.12:4767 , piece_id=QA37B4J4NZ5HR3DQC5LERT55TRTVDMZKCVEYCKQLE7ZUVAUL2PMA, seg_pos={0 0}, upload=473056]: Uploading piece... Dec 28 02:54:42.869214383 (539h25m42.740297418s) [noise=true , address=145.79.227.31:4797 , piece_id=YP5MY7P3HAXLCEGQ2AGQHJNWELTY2YR3VJ3ZWZRWPHUWQFDGGYUQ, seg_pos={0 0}, upload=473056]: Uploading piece... Dec 28 02:54:42.869318902 (539h25m42.740401937s) [noise=true , address=145.79.227.41:4087 , piece_id=JG6LAIGGHSG64FKWOGEWHLM6GS7TVMDWXKI3R7AODZWU6IOLY43A, seg_pos={0 0}, upload=473056]: Uploading piece... Dec 28 02:54:42.869490499 (539h25m42.740573534s) [noise=true , address=145.79.227.6:4257 , piece_id=QYQ7KHQCURJORKSMZMFG2OJ4HORBNCZ3JF2RCRW7HX6VSSHFR3ZA, seg_pos={0 0}, upload=473056]: Uploading piece... Dec 28 02:54:42.869658859 (539h25m42.740741894s) [noise=true , address=145.79.227.46:4602 , piece_id=O3FUV7S6ICCHVTRETSEZOPXT27OZPYZSBPI7PE2XMACPCF6AHEVQ, seg_pos={0 0}, upload=473056]: Uploading piece... Dec 28 02:54:42.869753800 (539h25m42.740836835s) [noise=true , address=77.90.31.16:4736 , piece_id=MBJ4NFIMKMCPMOTDPOWCKXJFJ22ZD7QSTBAQB67BTI43TMORBQOQ, seg_pos={0 0}, upload=473056]: Uploading piece... Dec 28 02:54:42.870044343 (539h25m42.741127378s) [noise=true , address=145.79.227.47:4266 , piece_id=DPDF5MM462LIVUIPMALJNGYT5V6R5AMIBMYOP5WBC3BGZYGDMWZA, seg_pos={0 0}, upload=473056]: Uploading piece... Dec 28 02:54:42.870302494 (539h25m42.741385529s) [noise=true , address=145.79.227.15:4874 , piece_id=YBTET2PVPCZVS3KFWAFXLN4VITE7PZBROHIDVGQODSTN6NAGM4CA, seg_pos={0 0}, upload=473056]: Uploading piece... Dec 28 02:54:44.654322405 (539h25m44.52540545s) [upload=473056]: Next returned nil segment Dec 28 02:54:44.654334538 (539h25m44.525417573s) [upload=473056]: Waiting for error group managing segments... Dec 28 02:54:44.965187384 (539h25m44.836270419s) [noise=true , address=145.79.227.41:4087 , piece_id=JG6LAIGGHSG64FKWOGEWHLM6GS7TVMDWXKI3R7AODZWU6IOLY43A, seg_pos={0 0}, upload=473056]: Done uploading piece. err: <nil> Dec 28 02:54:44.965712804 (539h25m44.836795849s) [noise=true , address=145.79.227.46:4602 , piece_id=O3FUV7S6ICCHVTRETSEZOPXT27OZPYZSBPI7PE2XMACPCF6AHEVQ, seg_pos={0 0}, upload=473056]: Done uploading piece. err: <nil> Dec 28 02:54:44.969763308 (539h25m44.840846353s) [noise=true , address=145.79.227.11:4729 , piece_id=ZVGZXARRKVWH3M3O6BMFVXMRQMEQ5I3DYUEJVCW26NKJFJ4SY3EQ, seg_pos={0 0}, upload=473056]: Done uploading piece. err: <nil> Dec 28 02:54:44.970032008 (539h25m44.841115043s) [noise=true , address=145.79.227.6:4257 , piece_id=QYQ7KHQCURJORKSMZMFG2OJ4HORBNCZ3JF2RCRW7HX6VSSHFR3ZA, seg_pos={0 0}, upload=473056]: Done uploading piece. err: <nil> Dec 28 02:54:44.970608887 (539h25m44.841691922s) [noise=true , address=145.79.227.12:4767 , piece_id=QA37B4J4NZ5HR3DQC5LERT55TRTVDMZKCVEYCKQLE7ZUVAUL2PMA, seg_pos={0 0}, upload=473056]: Done uploading piece. err: <nil> Dec 28 02:54:44.978845011 (539h25m44.849928046s) [noise=true , address=145.79.227.3:4150 , piece_id=UWTCCZKVCHMGYVYYUV4UWK7SFRIUOQ33TX6X7VMQHCG2JRNWLXXQ, seg_pos={0 0}, upload=473056]: Done uploading piece. err: <nil> Dec 28 02:54:44.983442806 (539h25m44.854525841s) [noise=true , address=145.79.227.22:4511 , piece_id=T5HIUW6JL3PIY7HKUOCVBOV2ZJ3GCW7BUVTT2TKSLI2RIIP77IZA, seg_pos={0 0}, upload=473056]: Done uploading piece. err: <nil> Dec 28 02:54:45.054251436 (539h25m44.925334471s) [noise=true , address=77.90.31.15:4700 , piece_id=LDHG7BHFQMTZJ3D6WTFZACABMMFE2MM6LLNOKPVTFDOI7EZLK4VA, seg_pos={0 0}, upload=473056]: Done uploading piece. err: <nil> Dec 28 02:54:45.062821356 (539h25m44.933904391s) [noise=true , address=145.79.227.47:4266 , piece_id=DPDF5MM462LIVUIPMALJNGYT5V6R5AMIBMYOP5WBC3BGZYGDMWZA, seg_pos={0 0}, upload=473056]: Done uploading piece. err: <nil> Dec 28 02:54:45.150207914 (539h25m45.021290959s) [noise=true , address=45.91.116.10:4083 , piece_id=PCDSW5XOOXIE42EUJIOH7TGZSPUNUUEYSLY7HTXDH7HNIQ25WUTQ, seg_pos={0 0}, upload=473056]: Done uploading piece. err: <nil> Dec 28 02:54:45.221885630 (539h25m45.092968675s) [noise=true , address=45.91.116.13:4155 , piece_id=GLR4PNGSR2IPOXAPGZ5OB26YTC427VE2MXMJ56RBM4WHSRN7IPNQ, seg_pos={0 0}, upload=473056]: Done uploading piece. err: <nil> Dec 28 02:54:45.278292098 (539h25m45.149375133s) [noise=true , address=77.90.31.16:4736 , piece_id=MBJ4NFIMKMCPMOTDPOWCKXJFJ22ZD7QSTBAQB67BTI43TMORBQOQ, seg_pos={0 0}, upload=473056]: Done uploading piece. err: <nil> Dec 28 02:54:46.106213968 (539h25m45.977297003s) [noise=true , address=145.79.227.15:4874 , piece_id=YBTET2PVPCZVS3KFWAFXLN4VITE7PZBROHIDVGQODSTN6NAGM4CA, seg_pos={0 0}, upload=473056]: Done uploading piece. err: <nil> Dec 28 02:54:46.106239427 (539h25m45.977322472s) [seg_pos={0 0}, upload=473056]: Segment reached optimal threshold of 13 pieces. Dec 28 02:54:46.106583392 (539h25m45.977666437s) [noise=true , address=145.79.227.31:4797 , piece_id=YP5MY7P3HAXLCEGQ2AGQHJNWELTY2YR3VJ3ZWZRWPHUWQFDGGYUQ, seg_pos={0 0}, upload=473056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9): protocol: expected piece hash; context canceled; EOF Dec 28 02:54:46.106593160 (539h25m45.977676205s) [noise=true , address=145.79.227.31:4797 , piece_id=YP5MY7P3HAXLCEGQ2AGQHJNWELTY2YR3VJ3ZWZRWPHUWQFDGGYUQ, seg_pos={0 0}, upload=473056]: Long tail cancellation due to optimal threshold Dec 28 02:54:46.106624280 (539h25m45.977707325s) [noise=true , address=45.91.116.4:4937 , piece_id=NRFJOKJWOBQMO3TKLZIJETV3DR5WIRC7V4BUNNIL222M3CFGYPKA, seg_pos={0 0}, upload=473056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 28 02:54:46.106639979 (539h25m45.977723014s) [noise=true , address=45.91.116.4:4937 , piece_id=NRFJOKJWOBQMO3TKLZIJETV3DR5WIRC7V4BUNNIL222M3CFGYPKA, seg_pos={0 0}, upload=473056]: Long tail cancellation due to optimal threshold Dec 28 02:54:46.106798081 (539h25m45.977881126s) [noise=true , address=145.79.227.18:4007 , piece_id=RJILSRPPK7DO7ZWMYHRX7M6U7YCFQZ3G22L4KDVPHQJX46B2OGBA, seg_pos={0 0}, upload=473056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp): protocol: expected piece hash; context canceled; EOF Dec 28 02:54:46.106809863 (539h25m45.977892908s) [noise=true , address=145.79.227.18:4007 , piece_id=RJILSRPPK7DO7ZWMYHRX7M6U7YCFQZ3G22L4KDVPHQJX46B2OGBA, seg_pos={0 0}, upload=473056]: Long tail cancellation due to optimal threshold Dec 28 02:54:46.106834951 (539h25m45.977917996s) [seg_pos={0 0}, upload=473056]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 02:54:46.106922678 (539h25m45.978005723s) [upload=473056]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 02:54:46.106946423 (539h25m45.978029478s) [upload=473056]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 02:54:46.400486012 (539h25m46.271569057s) [upload=473056]: Done uploading segments... Dec 28 02:54:46.400510689 (539h25m46.271593734s) [upload=473056]: Upload finished. err: <nil> Duration: 30.003287083s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [.....................X................X....X............X...] duration: 5.001161463s trace: https://jaeger.dev.storj.io/trace/20df15b758999ef2 FAIL (2025-12-28 02:55:05 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001187022s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................X.....X..................................] duration: 20.877419038s trace: https://jaeger.dev.storj.io/trace/1822be52d200e5b5 FAIL (2025-12-28 02:54:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1ZAfYZ4wgpZVvatj6wtHona3hTaPcLxAmXC1DT2uKQyo8nLQQ7, Piece ID: 5MMQ44MXRT4XEL4X6WO54WRG2XYMX3AN3FNQ2H2XIUTKBXI3EEPA) context deadline exceeded; piecestore: (Node ID: 124guAM82oibYZsMho67Em6Ke6yEZXwT6LPMLLSZZBLqwoFGUkn, Piece ID: 22WJIPX4SM5NTIAW5ZRM2UKX32CTDBFCXPCCHRRWRCUCOF5CRE5Q) context deadline exceeded; piecestore: (Node ID: 1e6p1TD5CqUyWdZjkrxh2kRqMuyDiMUr9eimfeayDAKwhtBrZ8, Piece ID: XRI5EJFN7YYUG4QKLDGEFT6YAUHCUTTSXF735UDR6LZZBGMAZDJQ) context deadline exceeded ======LOGS====== 5.234848204 (538h55m35.105931239s) [noise=true , address=77.90.31.11:4572 , piece_id=5MMQ44MXRT4XEL4X6WO54WRG2XYMX3AN3FNQ2H2XIUTKBXI3EEPA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.235412057 (538h55m35.106495102s) [noise=true , address=145.79.227.22:4512 , piece_id=D46YLTIKGHG36DQ4YBKBSUDV4GGB4WVDWNOJS27IYYLN6AYZ5QFQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.235820946 (538h55m35.106903981s) [noise=true , address=77.90.31.19:4874 , piece_id=RHZUXUYMJU6RKQGSDZM5WDWIN5SQ7YIRSRATNNKT5GEMU2QH444A, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.236192192 (538h55m35.107275237s) [noise=true , address=77.90.31.16:4760 , piece_id=XRI5EJFN7YYUG4QKLDGEFT6YAUHCUTTSXF735UDR6LZZBGMAZDJQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.236564772 (538h55m35.107647817s) [noise=true , address=145.79.227.40:4062 , piece_id=PW6QWIPQWVUETLC6DJ624G6IB4P6EM4SYB72NAMQ4V6NUKNLUYLQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.236929766 (538h55m35.108012812s) [noise=true , address=64.49.13.139:4480 , piece_id=JAJY3F7IDFFMFKCK2QJV6W74RIESLRKBLXMN27Q3C45JWEPLBZLQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.237916846 (538h55m35.108999881s) [noise=true , address=145.79.227.34:4932 , piece_id=UPXSCQ45QDGPKNJXRFJ33EAA3QENVHRVQARU5TFJTEWICZNR253Q, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.238852948 (538h55m35.109935983s) [noise=true , address=45.91.116.12:4257 , piece_id=ZI67LIAHZW4G357GWKDPTB5SMXYGNXZTTTDMNAETDS2B6SWVDAEQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239152910 (538h55m35.110235945s) [noise=true , address=145.79.227.44:4222 , piece_id=CZ2CS62CTHQFAIE3S3JO2N3VJ4XIR5YJ2LMRVLLX7I4VM2VZVQVA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239409969 (538h55m35.110493004s) [noise=true , address=145.79.227.30:4773 , piece_id=KUGJXZCNMUE7WSK3VLY4AUBSOT4H6IY4XB4NCBGAGCNUG7F7IODQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239681136 (538h55m35.110764171s) [noise=true , address=77.90.31.13:4635 , piece_id=IIOY5X2W3XW4C2OFEBRCABMYFBI24RRTYNUSKN6KJU7CTUW4KONQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239954676 (538h55m35.111037711s) [noise=true , address=45.91.116.9:4281 , piece_id=JZLDW777CBX2BP7MLOQV4NFP3H4XJVCNQXTHNSUWPVDOJLPZK36A, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.240189914 (538h55m35.111272949s) [noise=true , address=145.79.227.35:4512 , piece_id=3E3VCCC45XVT5FHFDKXSTGGAGYBEML3BKU6JJTT33LCD2YMOMBZQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.240449147 (538h55m35.111532182s) [noise=true , address=45.91.116.8:4064 , piece_id=22WJIPX4SM5NTIAW5ZRM2UKX32CTDBFCXPCCHRRWRCUCOF5CRE5Q, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.240639640 (538h55m35.111722675s) [noise=true , address=145.79.227.32:4853 , piece_id=MBNCFDBQ4SZZXV4F7J5TRMOUYUQ5IPKQCGYGGM5SOIFJ6ZYCAUYA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.241137798 (538h55m35.112220833s) [noise=true , address=77.90.31.16:4751 , piece_id=N3Z2LLMCIA2F42VXZA7IDRU4XEKGOICH6TRVERODU76MFFE4QSHA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.241383797 (538h55m35.112466842s) [noise=true , address=77.90.31.18:4836 , piece_id=BSVEFI52EKN44CMINFUSZBNOH53IW5TXLVTF5IH6ZUTNII6RRLVA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:36.592880816 (538h55m36.463963871s) [upload=472631]: Next returned nil segment Dec 28 02:24:36.592901625 (538h55m36.46398467s) [upload=472631]: Waiting for error group managing segments... Dec 28 02:24:36.946933741 (538h55m36.818016786s) [noise=true , address=64.49.13.139:4480 , piece_id=JAJY3F7IDFFMFKCK2QJV6W74RIESLRKBLXMN27Q3C45JWEPLBZLQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.128189692 (538h55m36.999272737s) [noise=true , address=145.79.227.35:4512 , piece_id=3E3VCCC45XVT5FHFDKXSTGGAGYBEML3BKU6JJTT33LCD2YMOMBZQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.160760084 (538h55m37.031843119s) [noise=true , address=145.79.227.34:4932 , piece_id=UPXSCQ45QDGPKNJXRFJ33EAA3QENVHRVQARU5TFJTEWICZNR253Q, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.288838207 (538h55m37.159921252s) [noise=true , address=77.90.31.11:4572 , piece_id=5MMQ44MXRT4XEL4X6WO54WRG2XYMX3AN3FNQ2H2XIUTKBXI3EEPA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.319996310 (538h55m37.191079345s) [noise=true , address=145.79.227.22:4512 , piece_id=D46YLTIKGHG36DQ4YBKBSUDV4GGB4WVDWNOJS27IYYLN6AYZ5QFQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.320513273 (538h55m37.191596319s) [noise=true , address=77.90.31.18:4836 , piece_id=BSVEFI52EKN44CMINFUSZBNOH53IW5TXLVTF5IH6ZUTNII6RRLVA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.336522926 (538h55m37.207605971s) [noise=true , address=145.79.227.32:4853 , piece_id=MBNCFDBQ4SZZXV4F7J5TRMOUYUQ5IPKQCGYGGM5SOIFJ6ZYCAUYA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.384868031 (538h55m37.255951066s) [noise=true , address=77.90.31.16:4751 , piece_id=N3Z2LLMCIA2F42VXZA7IDRU4XEKGOICH6TRVERODU76MFFE4QSHA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.453563554 (538h55m37.324646599s) [noise=true , address=45.91.116.8:4064 , piece_id=22WJIPX4SM5NTIAW5ZRM2UKX32CTDBFCXPCCHRRWRCUCOF5CRE5Q, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.479454839 (538h55m37.350537884s) [noise=true , address=45.91.116.12:4257 , piece_id=ZI67LIAHZW4G357GWKDPTB5SMXYGNXZTTTDMNAETDS2B6SWVDAEQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.659710174 (538h55m37.530793219s) [noise=true , address=77.90.31.13:4635 , piece_id=IIOY5X2W3XW4C2OFEBRCABMYFBI24RRTYNUSKN6KJU7CTUW4KONQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.715039128 (538h55m37.586122163s) [noise=true , address=77.90.31.16:4760 , piece_id=XRI5EJFN7YYUG4QKLDGEFT6YAUHCUTTSXF735UDR6LZZBGMAZDJQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.977473406 (538h55m37.848556441s) [noise=true , address=77.90.31.19:4874 , piece_id=RHZUXUYMJU6RKQGSDZM5WDWIN5SQ7YIRSRATNNKT5GEMU2QH444A, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:38.016302745 (538h55m37.88738578s) [noise=true , address=145.79.227.30:4773 , piece_id=KUGJXZCNMUE7WSK3VLY4AUBSOT4H6IY4XB4NCBGAGCNUG7F7IODQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:38.016326982 (538h55m37.887410027s) [seg_pos={0 0}, upload=472631]: Segment reached optimal threshold of 14 pieces. Dec 28 02:24:38.016585353 (538h55m37.887668398s) [noise=true , address=145.79.227.30:4766 , piece_id=67M2DMPAVWIQKQSWUAG73D4E4NRRSG4XEVPOUVIJPEQOUV7A6VVA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1k6zdamuKPkBdzyNZNWx4FCmaTkv33PchcwvTRR8bgcETBs4K7): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016594450 (538h55m37.887677495s) [noise=true , address=145.79.227.30:4766 , piece_id=67M2DMPAVWIQKQSWUAG73D4E4NRRSG4XEVPOUVIJPEQOUV7A6VVA, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016627093 (538h55m37.887710148s) [noise=true , address=45.91.116.9:4281 , piece_id=JZLDW777CBX2BP7MLOQV4NFP3H4XJVCNQXTHNSUWPVDOJLPZK36A, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gxf7DpJZthyR6f1PExX56tJavjhPw9Gxb4h8mtmeYKW3t3tS1): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016635258 (538h55m37.887718293s) [noise=true , address=45.91.116.9:4281 , piece_id=JZLDW777CBX2BP7MLOQV4NFP3H4XJVCNQXTHNSUWPVDOJLPZK36A, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016871298 (538h55m37.887954363s) [noise=true , address=145.79.227.44:4222 , piece_id=CZ2CS62CTHQFAIE3S3JO2N3VJ4XIR5YJ2LMRVLLX7I4VM2VZVQVA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZKW35KvrsoYRefbYeUFZo8bh4XZYycNKGEvTFjzibezjXgj3e): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016880124 (538h55m37.887963169s) [noise=true , address=145.79.227.44:4222 , piece_id=CZ2CS62CTHQFAIE3S3JO2N3VJ4XIR5YJ2LMRVLLX7I4VM2VZVQVA, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016934839 (538h55m37.888017884s) [noise=true , address=145.79.227.40:4062 , piece_id=PW6QWIPQWVUETLC6DJ624G6IB4P6EM4SYB72NAMQ4V6NUKNLUYLQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016943425 (538h55m37.88802649s) [noise=true , address=145.79.227.40:4062 , piece_id=PW6QWIPQWVUETLC6DJ624G6IB4P6EM4SYB72NAMQ4V6NUKNLUYLQ, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016968252 (538h55m37.888051297s) [seg_pos={0 0}, upload=472631]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 02:24:38.017067822 (538h55m37.888150867s) [upload=472631]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 02:24:38.017074244 (538h55m37.888157289s) [upload=472631]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 02:24:38.304306667 (538h55m38.175389712s) [upload=472631]: Done uploading segments... Dec 28 02:24:38.304329641 (538h55m38.175412686s) [upload=472631]: Upload finished. err: <nil> Duration: 20.877448584s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 42 out of 60 is failed (threshold=3) [X..X.XXX.XXX.X.....XXX.XXXXXX.XXXX.XXXX..X..XXXXX.XXXXXXXXXX] duration: 30.002878602s trace: https://jaeger.dev.storj.io/trace/18e6284987bed23b FAIL (2025-12-28 02:54:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, Piece ID: BBOYMYLXBUYLXQSR2VHKCXXH2JSZ2P7TF7NUHWYP2HDN7JMK2EHA) context deadline exceeded; piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: JV7HXF6TUOLR5JHRFBUE3NRDQCIJYRHTKPCNZ26TZXCFBZL7KDYQ) context deadline exceeded; piecestore: (Node ID: 14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, Piece ID: HEG5VFBYU4M57ZM6TI7D2QRPVQLY5MCL3J4CDSWVNPWFP57WBQKA) context deadline exceeded ======LOGS====== ... Dec 28 02:53:52.759849418 (539h24m52.630932463s) [upload=473047]: Got next segment. Inline: false Dec 28 02:53:52.759891447 (539h24m52.630974503s) [upload=473047]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 02:53:52.759897459 (539h24m52.630980504s) [upload=473047]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 02:53:53.005490869 (539h24m52.876573914s) [seg_pos={0 0}, upload=473047]: Begin upload segment... Dec 28 02:53:53.005553308 (539h24m52.876636353s) [seg_pos={0 0}, upload=473047]: Done begin upload segment. Dec 28 02:53:53.005612391 (539h24m52.876695426s) [noise=true , address=77.90.31.11:4556 , piece_id=QVSDWJQIQA3RIRE6JJ5FBHHOKOF4YU6WENFLHB4XIKFUYPKOLECA, seg_pos={0 0}, upload=473047]: Uploading piece... Dec 28 02:53:53.005640053 (539h24m52.876723088s) [noise=true , address=145.79.227.2:4114 , piece_id=TN2JJOAYL2PWHMV7C7G6UT4ZNIJJKVORQFR4V23GMZZE5PH63D7A, seg_pos={0 0}, upload=473047]: Uploading piece... Dec 28 02:53:53.006326179 (539h24m52.877409215s) [noise=true , address=45.91.116.12:4226 , piece_id=XAQGUTV6TCQ6RNLQQ73UAVJVRLWZIU3TXPQ4LDTILO7HGTC2RCDA, seg_pos={0 0}, upload=473047]: Uploading piece... Dec 28 02:53:53.007475207 (539h24m52.878558252s) [noise=true , address=45.91.116.8:4047 , piece_id=YXN76T42DN44RQYICBGJ64PAEKJRXIUIGYIUKSO7GLUATIKU3KAQ, seg_pos={0 0}, upload=473047]: Uploading piece... Dec 28 02:53:53.007859559 (539h24m52.878942604s) [noise=true , address=145.79.227.12:4767 , piece_id=SF4SEDXCYL75NJAMY5NY6AQZMRFXIT5L5AA5FCPPPROYEVRK3LZQ, seg_pos={0 0}, upload=473047]: Uploading piece... Dec 28 02:53:53.008191862 (539h24m52.879274897s) [noise=true , address=145.79.227.33:4869 , piece_id=CLVUBQJNWSTSHRGZ5CPRI652SP6H4EMTRJBNRLEGNXFADBIURKYQ, seg_pos={0 0}, upload=473047]: Uploading piece... Dec 28 02:53:53.008258459 (539h24m52.879341504s) [noise=true , address=145.79.227.41:4086 , piece_id=SQY2O6UFZI6ZYJ4FHEJK46E7O4VJTI6HTXFA52Q4MQLHCMPL3LWA, seg_pos={0 0}, upload=473047]: Uploading piece... Dec 28 02:53:53.008465975 (539h24m52.87954901s) [noise=true , address=45.91.116.14:4192 , piece_id=JV7HXF6TUOLR5JHRFBUE3NRDQCIJYRHTKPCNZ26TZXCFBZL7KDYQ, seg_pos={0 0}, upload=473047]: Uploading piece... Dec 28 02:53:53.009057341 (539h24m52.880140386s) [noise=true , address=145.79.227.36:4519 , piece_id=ZZWTXEFRNP6FVUTEADVWO34IMWESR5J6XPMKYR6UOZIQPK7K4GEA, seg_pos={0 0}, upload=473047]: Uploading piece... Dec 28 02:53:53.009477190 (539h24m52.880560225s) [noise=true , address=145.79.227.24:4583 , piece_id=EBK2ZZD3PIIMQPCXEYAW45GDVUPXZ7ATV4NZHD24KNCNXXPERXGQ, seg_pos={0 0}, upload=473047]: Uploading piece... Dec 28 02:53:53.009876019 (539h24m52.880959064s) [noise=true , address=145.79.227.39:4015 , piece_id=KSAXEOQZRK4AP7BQDDYYIW6K2JE2NRIMY7BFHM4ZWIEK6APHU3QA, seg_pos={0 0}, upload=473047]: Uploading piece... Dec 28 02:53:53.010245513 (539h24m52.881328558s) [noise=true , address=45.91.116.11:4119 , piece_id=I4YXOH6HBB6DYN7LVLCQWDW73FTDKAGW4GEERNZGV35EPB5SA72Q, seg_pos={0 0}, upload=473047]: Uploading piece... Dec 28 02:53:53.010591652 (539h24m52.881674687s) [noise=true , address=45.91.116.13:4155 , piece_id=V56C2G4QR3DQRLDXNLMGH75KSOJFMAA25GDK2N2VIPKNOSLZCWMA, seg_pos={0 0}, upload=473047]: Uploading piece... Dec 28 02:53:53.010917642 (539h24m52.882000678s) [noise=true , address=77.90.31.16:4735 , piece_id=HEG5VFBYU4M57ZM6TI7D2QRPVQLY5MCL3J4CDSWVNPWFP57WBQKA, seg_pos={0 0}, upload=473047]: Uploading piece... Dec 28 02:53:53.011054163 (539h24m52.882137208s) [noise=true , address=45.91.116.6:4976 , piece_id=BBOYMYLXBUYLXQSR2VHKCXXH2JSZ2P7TF7NUHWYP2HDN7JMK2EHA, seg_pos={0 0}, upload=473047]: Uploading piece... Dec 28 02:53:53.011757101 (539h24m52.882840136s) [noise=true , address=145.79.227.42:4121 , piece_id=PC42HZFXB7EO5DOAAO2VML4QC7VIPLMWX6DDERIM4OXYJI4TS3DQ, seg_pos={0 0}, upload=473047]: Uploading piece... Dec 28 02:53:54.754811476 (539h24m54.625894521s) [upload=473047]: Next returned nil segment Dec 28 02:53:54.754830132 (539h24m54.625913237s) [upload=473047]: Waiting for error group managing segments... Dec 28 02:53:55.028666933 (539h24m54.899749968s) [noise=true , address=145.79.227.12:4767 , piece_id=SF4SEDXCYL75NJAMY5NY6AQZMRFXIT5L5AA5FCPPPROYEVRK3LZQ, seg_pos={0 0}, upload=473047]: Done uploading piece. err: <nil> Dec 28 02:53:55.045010333 (539h24m54.916093378s) [noise=true , address=145.79.227.39:4015 , piece_id=KSAXEOQZRK4AP7BQDDYYIW6K2JE2NRIMY7BFHM4ZWIEK6APHU3QA, seg_pos={0 0}, upload=473047]: Done uploading piece. err: <nil> Dec 28 02:53:55.045875942 (539h24m54.916958987s) [noise=true , address=145.79.227.36:4519 , piece_id=ZZWTXEFRNP6FVUTEADVWO34IMWESR5J6XPMKYR6UOZIQPK7K4GEA, seg_pos={0 0}, upload=473047]: Done uploading piece. err: <nil> Dec 28 02:53:55.064863066 (539h24m54.935946111s) [noise=true , address=145.79.227.42:4121 , piece_id=PC42HZFXB7EO5DOAAO2VML4QC7VIPLMWX6DDERIM4OXYJI4TS3DQ, seg_pos={0 0}, upload=473047]: Done uploading piece. err: <nil> Dec 28 02:53:55.073087707 (539h24m54.944170742s) [noise=true , address=145.79.227.24:4583 , piece_id=EBK2ZZD3PIIMQPCXEYAW45GDVUPXZ7ATV4NZHD24KNCNXXPERXGQ, seg_pos={0 0}, upload=473047]: Done uploading piece. err: <nil> Dec 28 02:53:55.290281320 (539h24m55.161364365s) [noise=true , address=45.91.116.12:4226 , piece_id=XAQGUTV6TCQ6RNLQQ73UAVJVRLWZIU3TXPQ4LDTILO7HGTC2RCDA, seg_pos={0 0}, upload=473047]: Done uploading piece. err: <nil> Dec 28 02:53:55.299355659 (539h24m55.170438694s) [noise=true , address=45.91.116.8:4047 , piece_id=YXN76T42DN44RQYICBGJ64PAEKJRXIUIGYIUKSO7GLUATIKU3KAQ, seg_pos={0 0}, upload=473047]: Done uploading piece. err: <nil> Dec 28 02:53:55.321638349 (539h24m55.192721384s) [noise=true , address=45.91.116.6:4976 , piece_id=BBOYMYLXBUYLXQSR2VHKCXXH2JSZ2P7TF7NUHWYP2HDN7JMK2EHA, seg_pos={0 0}, upload=473047]: Done uploading piece. err: <nil> Dec 28 02:53:55.984740926 (539h24m55.855823961s) [noise=true , address=45.91.116.13:4155 , piece_id=V56C2G4QR3DQRLDXNLMGH75KSOJFMAA25GDK2N2VIPKNOSLZCWMA, seg_pos={0 0}, upload=473047]: Done uploading piece. err: <nil> Dec 28 02:53:56.007612587 (539h24m55.878695632s) [noise=true , address=77.90.31.16:4735 , piece_id=HEG5VFBYU4M57ZM6TI7D2QRPVQLY5MCL3J4CDSWVNPWFP57WBQKA, seg_pos={0 0}, upload=473047]: Done uploading piece. err: <nil> Dec 28 02:53:56.015362033 (539h24m55.886445078s) [noise=true , address=45.91.116.11:4119 , piece_id=I4YXOH6HBB6DYN7LVLCQWDW73FTDKAGW4GEERNZGV35EPB5SA72Q, seg_pos={0 0}, upload=473047]: Done uploading piece. err: <nil> Dec 28 02:53:56.041646353 (539h24m55.912729398s) [noise=true , address=77.90.31.11:4556 , piece_id=QVSDWJQIQA3RIRE6JJ5FBHHOKOF4YU6WENFLHB4XIKFUYPKOLECA, seg_pos={0 0}, upload=473047]: Done uploading piece. err: <nil> Dec 28 02:53:56.353886751 (539h24m56.224969796s) [noise=true , address=45.91.116.14:4192 , piece_id=JV7HXF6TUOLR5JHRFBUE3NRDQCIJYRHTKPCNZ26TZXCFBZL7KDYQ, seg_pos={0 0}, upload=473047]: Done uploading piece. err: <nil> Dec 28 02:53:56.353905266 (539h24m56.224988301s) [seg_pos={0 0}, upload=473047]: Segment reached optimal threshold of 13 pieces. Dec 28 02:53:56.354077985 (539h24m56.22516102s) [noise=true , address=145.79.227.2:4114 , piece_id=TN2JJOAYL2PWHMV7C7G6UT4ZNIJJKVORQFR4V23GMZZE5PH63D7A, seg_pos={0 0}, upload=473047]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN): protocol: write tcp 188.166.225.170:46170->145.79.227.2:4114: use of closed network connection; write tcp 188.166.225.170:46170->145.79.227.2:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46170->145.79.227.2:4114: use of closed network connection Dec 28 02:53:56.354084428 (539h24m56.225167463s) [noise=true , address=145.79.227.2:4114 , piece_id=TN2JJOAYL2PWHMV7C7G6UT4ZNIJJKVORQFR4V23GMZZE5PH63D7A, seg_pos={0 0}, upload=473047]: Long tail cancellation due to optimal threshold Dec 28 02:53:56.354242057 (539h24m56.225325103s) [noise=true , address=145.79.227.41:4086 , piece_id=SQY2O6UFZI6ZYJ4FHEJK46E7O4VJTI6HTXFA52Q4MQLHCMPL3LWA, seg_pos={0 0}, upload=473047]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm): protocol: write tcp 188.166.225.170:50576->145.79.227.41:4086: use of closed network connection; write tcp 188.166.225.170:50576->145.79.227.41:4086: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50576->145.79.227.41:4086: use of closed network connection Dec 28 02:53:56.354280420 (539h24m56.225363455s) [noise=true , address=145.79.227.41:4086 , piece_id=SQY2O6UFZI6ZYJ4FHEJK46E7O4VJTI6HTXFA52Q4MQLHCMPL3LWA, seg_pos={0 0}, upload=473047]: Long tail cancellation due to optimal threshold Dec 28 02:53:56.354407473 (539h24m56.225490508s) [noise=true , address=145.79.227.33:4869 , piece_id=CLVUBQJNWSTSHRGZ5CPRI652SP6H4EMTRJBNRLEGNXFADBIURKYQ, seg_pos={0 0}, upload=473047]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF Dec 28 02:53:56.354417011 (539h24m56.225500056s) [noise=true , address=145.79.227.33:4869 , piece_id=CLVUBQJNWSTSHRGZ5CPRI652SP6H4EMTRJBNRLEGNXFADBIURKYQ, seg_pos={0 0}, upload=473047]: Long tail cancellation due to optimal threshold Dec 28 02:53:56.354436919 (539h24m56.225519954s) [seg_pos={0 0}, upload=473047]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 02:53:56.354658200 (539h24m56.225741235s) [upload=473047]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 02:53:56.354666315 (539h24m56.22574935s) [upload=473047]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 02:53:56.649709026 (539h24m56.520792061s) [upload=473047]: Done uploading segments... Dec 28 02:53:56.649730788 (539h24m56.520813833s) [upload=473047]: Upload finished. err: <nil> Duration: 30.002890205s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [......................X................X....X............X..] duration: 3.490216022s trace: https://jaeger.dev.storj.io/trace/20df15b758999ef2 FAIL (2025-12-28 02:53:58 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.490231802s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 42 out of 60 is failed (threshold=3) [XX..X.XXX.XXX.X.....XXX.XXXXXX.XXXX.XXXX..X..XXXXX.XXXXXXXXX] duration: 30.00332531s trace: https://jaeger.dev.storj.io/trace/52ce44ae82d96764 FAIL (2025-12-28 02:53:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 02:53:22.534888706 (539h24m22.405971741s) [upload=473040]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 02:53:22.534892413 (539h24m22.405975448s) [upload=473040]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 02:53:22.780160855 (539h24m22.6512439s) [seg_pos={0 0}, upload=473040]: Begin upload segment... Dec 28 02:53:22.780210449 (539h24m22.651293484s) [seg_pos={0 0}, upload=473040]: Done begin upload segment. Dec 28 02:53:22.780254063 (539h24m22.651337098s) [noise=true , address=145.79.227.22:4511 , piece_id=4X6DZE2XJRD6K7F3UDOY524QTOC7AQVYVGUJE4VLYBINO7TZBQNA, seg_pos={0 0}, upload=473040]: Uploading piece... Dec 28 02:53:22.780299559 (539h24m22.651382604s) [noise=true , address=45.91.116.5:4300 , piece_id=AECHJVT4O2X6X3YRBXQUSYYDXUK3DLW5YJQGOCGT7WTK5567RHQA, seg_pos={0 0}, upload=473040]: Uploading piece... Dec 28 02:53:22.780934789 (539h24m22.652017834s) [noise=true , address=45.91.116.9:4264 , piece_id=XP3XDKDYVSGQHLERHVCSQUM3RITNZ3N7VTORLBY2QR342EDBI2XQ, seg_pos={0 0}, upload=473040]: Uploading piece... Dec 28 02:53:22.781499965 (539h24m22.652583s) [noise=true , address=145.79.227.11:4732 , piece_id=WRE4OXRHOS5RUJ5EEKLXJVMXJOHO2BVVN5AHNKC57C7AHZKMKPTQ, seg_pos={0 0}, upload=473040]: Uploading piece... Dec 28 02:53:22.781954069 (539h24m22.653037104s) [noise=true , address=45.91.116.4:4940 , piece_id=HR2KUOP4RJWRQOYJHFBUMNNET6NYTV4TKDFPWDLTDWJDEKGJBLCA, seg_pos={0 0}, upload=473040]: Uploading piece... Dec 28 02:53:22.782029063 (539h24m22.653112108s) [noise=true , address=145.79.227.48:4303 , piece_id=HNVHRLPX3FXNNBDDWVM2UKEJDPLOQQPEKU7UOZJ6C5NAX3SMUOAQ, seg_pos={0 0}, upload=473040]: Uploading piece... Dec 28 02:53:22.782652499 (539h24m22.653735544s) [noise=true , address=77.90.31.16:4735 , piece_id=NM7RFEZ4LL5YN6RI7FKUV36EVKDVYQRF7M7JF7F5PCWBONBGNXIQ, seg_pos={0 0}, upload=473040]: Uploading piece... Dec 28 02:53:22.783173101 (539h24m22.654256146s) [noise=true , address=145.79.227.5:4222 , piece_id=AOZCUABNOGTFUZAZBQNCKQWE4HLYP6BKYF72ONXBIM3VTIUEZSQA, seg_pos={0 0}, upload=473040]: Uploading piece... Dec 28 02:53:22.783755710 (539h24m22.654838755s) [noise=true , address=145.79.227.45:4231 , piece_id=EATJPFP4ZDYZTKKGLPGH5DI2EFVR3VDJ2BNH5OX5MOCNF74NJPEA, seg_pos={0 0}, upload=473040]: Uploading piece... Dec 28 02:53:22.784450553 (539h24m22.655533598s) [noise=true , address=45.91.116.8:4047 , piece_id=C5TGV6KRNN5OMLJBJIRDSXE7GPMIR2OMYNMR5CBU4242YC5KYHLA, seg_pos={0 0}, upload=473040]: Uploading piece... Dec 28 02:53:22.785383300 (539h24m22.656466375s) [noise=true , address=145.79.227.20:4438 , piece_id=ZZRNL6BRHUKOALT3S2MY67YP6YV3IMQN3F5LZMQ7VHRRKM7AXIRQ, seg_pos={0 0}, upload=473040]: Uploading piece... Dec 28 02:53:22.786789126 (539h24m22.657872171s) [noise=true , address=45.91.116.12:4228 , piece_id=XWFTP4F6IBINWHLAIDD3R7YFHEG3WEOWHAOO6WSLAPUY3A6TQH7A, seg_pos={0 0}, upload=473040]: Uploading piece... Dec 28 02:53:22.788003178 (539h24m22.659086223s) [noise=true , address=145.79.227.42:4121 , piece_id=2XFPJK6FTJLSBUIFJGJNQNWCQPNYPJGVOLL7OZE7ILGB3G6HWL7Q, seg_pos={0 0}, upload=473040]: Uploading piece... Dec 28 02:53:22.794448962 (539h24m22.665532007s) [noise=true , address=145.79.227.36:4519 , piece_id=A3O3WM6T5QZKC6VSNO7YMI5ALONQU2JTMSKICJE4BUOSXFO52HMQ, seg_pos={0 0}, upload=473040]: Uploading piece... Dec 28 02:53:22.798425895 (539h24m22.66950894s) [noise=true , address=77.90.31.19:4844 , piece_id=KZ7UHPE7D5Y5FSNZ7SC3LB44OKEJ25FB43E5ZC6BW2LVLHIIZMRQ, seg_pos={0 0}, upload=473040]: Uploading piece... Dec 28 02:53:22.801989581 (539h24m22.673072626s) [noise=true , address=145.79.227.15:4874 , piece_id=CERWGKKY5GOCPLPTWC7JBXP77YVLFYEZVDOL25PMOM3IV2KLLXVQ, seg_pos={0 0}, upload=473040]: Uploading piece... Dec 28 02:53:24.405735090 (539h24m24.276818135s) [upload=473040]: Next returned nil segment Dec 28 02:53:24.405756090 (539h24m24.276839135s) [upload=473040]: Waiting for error group managing segments... Dec 28 02:53:24.799814422 (539h24m24.670897457s) [noise=true , address=145.79.227.15:4874 , piece_id=CERWGKKY5GOCPLPTWC7JBXP77YVLFYEZVDOL25PMOM3IV2KLLXVQ, seg_pos={0 0}, upload=473040]: Done uploading piece. err: <nil> Dec 28 02:53:24.804789165 (539h24m24.67587221s) [noise=true , address=145.79.227.11:4732 , piece_id=WRE4OXRHOS5RUJ5EEKLXJVMXJOHO2BVVN5AHNKC57C7AHZKMKPTQ, seg_pos={0 0}, upload=473040]: Done uploading piece. err: <nil> Dec 28 02:53:24.807963079 (539h24m24.679046124s) [noise=true , address=145.79.227.48:4303 , piece_id=HNVHRLPX3FXNNBDDWVM2UKEJDPLOQQPEKU7UOZJ6C5NAX3SMUOAQ, seg_pos={0 0}, upload=473040]: Done uploading piece. err: <nil> Dec 28 02:53:24.808852012 (539h24m24.679935057s) [noise=true , address=145.79.227.20:4438 , piece_id=ZZRNL6BRHUKOALT3S2MY67YP6YV3IMQN3F5LZMQ7VHRRKM7AXIRQ, seg_pos={0 0}, upload=473040]: Done uploading piece. err: <nil> Dec 28 02:53:24.809437606 (539h24m24.680520651s) [noise=true , address=145.79.227.45:4231 , piece_id=EATJPFP4ZDYZTKKGLPGH5DI2EFVR3VDJ2BNH5OX5MOCNF74NJPEA, seg_pos={0 0}, upload=473040]: Done uploading piece. err: <nil> Dec 28 02:53:24.820661829 (539h24m24.691744864s) [noise=true , address=145.79.227.42:4121 , piece_id=2XFPJK6FTJLSBUIFJGJNQNWCQPNYPJGVOLL7OZE7ILGB3G6HWL7Q, seg_pos={0 0}, upload=473040]: Done uploading piece. err: <nil> Dec 28 02:53:24.824911491 (539h24m24.695994536s) [noise=true , address=145.79.227.36:4519 , piece_id=A3O3WM6T5QZKC6VSNO7YMI5ALONQU2JTMSKICJE4BUOSXFO52HMQ, seg_pos={0 0}, upload=473040]: Done uploading piece. err: <nil> Dec 28 02:53:24.999575079 (539h24m24.870658125s) [noise=true , address=77.90.31.19:4844 , piece_id=KZ7UHPE7D5Y5FSNZ7SC3LB44OKEJ25FB43E5ZC6BW2LVLHIIZMRQ, seg_pos={0 0}, upload=473040]: Done uploading piece. err: <nil> Dec 28 02:53:25.001319221 (539h24m24.872402266s) [noise=true , address=45.91.116.5:4300 , piece_id=AECHJVT4O2X6X3YRBXQUSYYDXUK3DLW5YJQGOCGT7WTK5567RHQA, seg_pos={0 0}, upload=473040]: Done uploading piece. err: <nil> Dec 28 02:53:25.007599911 (539h24m24.878682946s) [noise=true , address=45.91.116.12:4228 , piece_id=XWFTP4F6IBINWHLAIDD3R7YFHEG3WEOWHAOO6WSLAPUY3A6TQH7A, seg_pos={0 0}, upload=473040]: Done uploading piece. err: <nil> Dec 28 02:53:25.039850160 (539h24m24.910933195s) [noise=true , address=45.91.116.4:4940 , piece_id=HR2KUOP4RJWRQOYJHFBUMNNET6NYTV4TKDFPWDLTDWJDEKGJBLCA, seg_pos={0 0}, upload=473040]: Done uploading piece. err: <nil> Dec 28 02:53:25.083300467 (539h24m24.954383502s) [noise=true , address=77.90.31.16:4735 , piece_id=NM7RFEZ4LL5YN6RI7FKUV36EVKDVYQRF7M7JF7F5PCWBONBGNXIQ, seg_pos={0 0}, upload=473040]: Done uploading piece. err: <nil> Dec 28 02:53:25.151918678 (539h24m25.023001713s) [noise=true , address=45.91.116.9:4264 , piece_id=XP3XDKDYVSGQHLERHVCSQUM3RITNZ3N7VTORLBY2QR342EDBI2XQ, seg_pos={0 0}, upload=473040]: Done uploading piece. err: <nil> Dec 28 02:53:25.151939637 (539h24m25.023022672s) [seg_pos={0 0}, upload=473040]: Segment reached optimal threshold of 13 pieces. Dec 28 02:53:25.152238406 (539h24m25.023321451s) [noise=true , address=45.91.116.8:4047 , piece_id=C5TGV6KRNN5OMLJBJIRDSXE7GPMIR2OMYNMR5CBU4242YC5KYHLA, seg_pos={0 0}, upload=473040]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: read tcp 188.166.225.170:50280->45.91.116.8:4047: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 02:53:25.152245920 (539h24m25.023328955s) [noise=true , address=45.91.116.8:4047 , piece_id=C5TGV6KRNN5OMLJBJIRDSXE7GPMIR2OMYNMR5CBU4242YC5KYHLA, seg_pos={0 0}, upload=473040]: Long tail cancellation due to optimal threshold Dec 28 02:53:25.152293130 (539h24m25.023376165s) [noise=true , address=145.79.227.22:4511 , piece_id=4X6DZE2XJRD6K7F3UDOY524QTOC7AQVYVGUJE4VLYBINO7TZBQNA, seg_pos={0 0}, upload=473040]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG): protocol: write tcp 188.166.225.170:53074->145.79.227.22:4511: use of closed network connection; write tcp 188.166.225.170:53074->145.79.227.22:4511: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53074->145.79.227.22:4511: use of closed network connection Dec 28 02:53:25.152312127 (539h24m25.023395182s) [noise=true , address=145.79.227.22:4511 , piece_id=4X6DZE2XJRD6K7F3UDOY524QTOC7AQVYVGUJE4VLYBINO7TZBQNA, seg_pos={0 0}, upload=473040]: Long tail cancellation due to optimal threshold Dec 28 02:53:25.152457663 (539h24m25.023540709s) [noise=true , address=145.79.227.5:4222 , piece_id=AOZCUABNOGTFUZAZBQNCKQWE4HLYP6BKYF72ONXBIM3VTIUEZSQA, seg_pos={0 0}, upload=473040]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1): protocol: write tcp 188.166.225.170:42626->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:42626->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42626->145.79.227.5:4222: use of closed network connection Dec 28 02:53:25.152464276 (539h24m25.023547311s) [noise=true , address=145.79.227.5:4222 , piece_id=AOZCUABNOGTFUZAZBQNCKQWE4HLYP6BKYF72ONXBIM3VTIUEZSQA, seg_pos={0 0}, upload=473040]: Long tail cancellation due to optimal threshold Dec 28 02:53:25.152482551 (539h24m25.023565586s) [seg_pos={0 0}, upload=473040]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 02:53:25.152551192 (539h24m25.023634748s) [upload=473040]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 02:53:25.152561111 (539h24m25.023644146s) [upload=473040]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 02:53:25.446110147 (539h24m25.317193182s) [upload=473040]: Done uploading segments... Dec 28 02:53:25.446137169 (539h24m25.317220214s) [upload=473040]: Upload finished. err: <nil> Duration: 30.00333631s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [X...................X.....X.................................] duration: 38.626489451s trace: https://jaeger.dev.storj.io/trace/1822be52d200e5b5 FAIL (2025-12-28 02:53:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1ZAfYZ4wgpZVvatj6wtHona3hTaPcLxAmXC1DT2uKQyo8nLQQ7, Piece ID: 5MMQ44MXRT4XEL4X6WO54WRG2XYMX3AN3FNQ2H2XIUTKBXI3EEPA) context deadline exceeded; piecestore: (Node ID: 124guAM82oibYZsMho67Em6Ke6yEZXwT6LPMLLSZZBLqwoFGUkn, Piece ID: 22WJIPX4SM5NTIAW5ZRM2UKX32CTDBFCXPCCHRRWRCUCOF5CRE5Q) context deadline exceeded; piecestore: (Node ID: 1e6p1TD5CqUyWdZjkrxh2kRqMuyDiMUr9eimfeayDAKwhtBrZ8, Piece ID: XRI5EJFN7YYUG4QKLDGEFT6YAUHCUTTSXF735UDR6LZZBGMAZDJQ) context deadline exceeded ======LOGS====== 5.234848204 (538h55m35.105931239s) [noise=true , address=77.90.31.11:4572 , piece_id=5MMQ44MXRT4XEL4X6WO54WRG2XYMX3AN3FNQ2H2XIUTKBXI3EEPA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.235412057 (538h55m35.106495102s) [noise=true , address=145.79.227.22:4512 , piece_id=D46YLTIKGHG36DQ4YBKBSUDV4GGB4WVDWNOJS27IYYLN6AYZ5QFQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.235820946 (538h55m35.106903981s) [noise=true , address=77.90.31.19:4874 , piece_id=RHZUXUYMJU6RKQGSDZM5WDWIN5SQ7YIRSRATNNKT5GEMU2QH444A, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.236192192 (538h55m35.107275237s) [noise=true , address=77.90.31.16:4760 , piece_id=XRI5EJFN7YYUG4QKLDGEFT6YAUHCUTTSXF735UDR6LZZBGMAZDJQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.236564772 (538h55m35.107647817s) [noise=true , address=145.79.227.40:4062 , piece_id=PW6QWIPQWVUETLC6DJ624G6IB4P6EM4SYB72NAMQ4V6NUKNLUYLQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.236929766 (538h55m35.108012812s) [noise=true , address=64.49.13.139:4480 , piece_id=JAJY3F7IDFFMFKCK2QJV6W74RIESLRKBLXMN27Q3C45JWEPLBZLQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.237916846 (538h55m35.108999881s) [noise=true , address=145.79.227.34:4932 , piece_id=UPXSCQ45QDGPKNJXRFJ33EAA3QENVHRVQARU5TFJTEWICZNR253Q, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.238852948 (538h55m35.109935983s) [noise=true , address=45.91.116.12:4257 , piece_id=ZI67LIAHZW4G357GWKDPTB5SMXYGNXZTTTDMNAETDS2B6SWVDAEQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239152910 (538h55m35.110235945s) [noise=true , address=145.79.227.44:4222 , piece_id=CZ2CS62CTHQFAIE3S3JO2N3VJ4XIR5YJ2LMRVLLX7I4VM2VZVQVA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239409969 (538h55m35.110493004s) [noise=true , address=145.79.227.30:4773 , piece_id=KUGJXZCNMUE7WSK3VLY4AUBSOT4H6IY4XB4NCBGAGCNUG7F7IODQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239681136 (538h55m35.110764171s) [noise=true , address=77.90.31.13:4635 , piece_id=IIOY5X2W3XW4C2OFEBRCABMYFBI24RRTYNUSKN6KJU7CTUW4KONQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239954676 (538h55m35.111037711s) [noise=true , address=45.91.116.9:4281 , piece_id=JZLDW777CBX2BP7MLOQV4NFP3H4XJVCNQXTHNSUWPVDOJLPZK36A, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.240189914 (538h55m35.111272949s) [noise=true , address=145.79.227.35:4512 , piece_id=3E3VCCC45XVT5FHFDKXSTGGAGYBEML3BKU6JJTT33LCD2YMOMBZQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.240449147 (538h55m35.111532182s) [noise=true , address=45.91.116.8:4064 , piece_id=22WJIPX4SM5NTIAW5ZRM2UKX32CTDBFCXPCCHRRWRCUCOF5CRE5Q, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.240639640 (538h55m35.111722675s) [noise=true , address=145.79.227.32:4853 , piece_id=MBNCFDBQ4SZZXV4F7J5TRMOUYUQ5IPKQCGYGGM5SOIFJ6ZYCAUYA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.241137798 (538h55m35.112220833s) [noise=true , address=77.90.31.16:4751 , piece_id=N3Z2LLMCIA2F42VXZA7IDRU4XEKGOICH6TRVERODU76MFFE4QSHA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.241383797 (538h55m35.112466842s) [noise=true , address=77.90.31.18:4836 , piece_id=BSVEFI52EKN44CMINFUSZBNOH53IW5TXLVTF5IH6ZUTNII6RRLVA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:36.592880816 (538h55m36.463963871s) [upload=472631]: Next returned nil segment Dec 28 02:24:36.592901625 (538h55m36.46398467s) [upload=472631]: Waiting for error group managing segments... Dec 28 02:24:36.946933741 (538h55m36.818016786s) [noise=true , address=64.49.13.139:4480 , piece_id=JAJY3F7IDFFMFKCK2QJV6W74RIESLRKBLXMN27Q3C45JWEPLBZLQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.128189692 (538h55m36.999272737s) [noise=true , address=145.79.227.35:4512 , piece_id=3E3VCCC45XVT5FHFDKXSTGGAGYBEML3BKU6JJTT33LCD2YMOMBZQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.160760084 (538h55m37.031843119s) [noise=true , address=145.79.227.34:4932 , piece_id=UPXSCQ45QDGPKNJXRFJ33EAA3QENVHRVQARU5TFJTEWICZNR253Q, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.288838207 (538h55m37.159921252s) [noise=true , address=77.90.31.11:4572 , piece_id=5MMQ44MXRT4XEL4X6WO54WRG2XYMX3AN3FNQ2H2XIUTKBXI3EEPA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.319996310 (538h55m37.191079345s) [noise=true , address=145.79.227.22:4512 , piece_id=D46YLTIKGHG36DQ4YBKBSUDV4GGB4WVDWNOJS27IYYLN6AYZ5QFQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.320513273 (538h55m37.191596319s) [noise=true , address=77.90.31.18:4836 , piece_id=BSVEFI52EKN44CMINFUSZBNOH53IW5TXLVTF5IH6ZUTNII6RRLVA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.336522926 (538h55m37.207605971s) [noise=true , address=145.79.227.32:4853 , piece_id=MBNCFDBQ4SZZXV4F7J5TRMOUYUQ5IPKQCGYGGM5SOIFJ6ZYCAUYA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.384868031 (538h55m37.255951066s) [noise=true , address=77.90.31.16:4751 , piece_id=N3Z2LLMCIA2F42VXZA7IDRU4XEKGOICH6TRVERODU76MFFE4QSHA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.453563554 (538h55m37.324646599s) [noise=true , address=45.91.116.8:4064 , piece_id=22WJIPX4SM5NTIAW5ZRM2UKX32CTDBFCXPCCHRRWRCUCOF5CRE5Q, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.479454839 (538h55m37.350537884s) [noise=true , address=45.91.116.12:4257 , piece_id=ZI67LIAHZW4G357GWKDPTB5SMXYGNXZTTTDMNAETDS2B6SWVDAEQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.659710174 (538h55m37.530793219s) [noise=true , address=77.90.31.13:4635 , piece_id=IIOY5X2W3XW4C2OFEBRCABMYFBI24RRTYNUSKN6KJU7CTUW4KONQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.715039128 (538h55m37.586122163s) [noise=true , address=77.90.31.16:4760 , piece_id=XRI5EJFN7YYUG4QKLDGEFT6YAUHCUTTSXF735UDR6LZZBGMAZDJQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.977473406 (538h55m37.848556441s) [noise=true , address=77.90.31.19:4874 , piece_id=RHZUXUYMJU6RKQGSDZM5WDWIN5SQ7YIRSRATNNKT5GEMU2QH444A, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:38.016302745 (538h55m37.88738578s) [noise=true , address=145.79.227.30:4773 , piece_id=KUGJXZCNMUE7WSK3VLY4AUBSOT4H6IY4XB4NCBGAGCNUG7F7IODQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:38.016326982 (538h55m37.887410027s) [seg_pos={0 0}, upload=472631]: Segment reached optimal threshold of 14 pieces. Dec 28 02:24:38.016585353 (538h55m37.887668398s) [noise=true , address=145.79.227.30:4766 , piece_id=67M2DMPAVWIQKQSWUAG73D4E4NRRSG4XEVPOUVIJPEQOUV7A6VVA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1k6zdamuKPkBdzyNZNWx4FCmaTkv33PchcwvTRR8bgcETBs4K7): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016594450 (538h55m37.887677495s) [noise=true , address=145.79.227.30:4766 , piece_id=67M2DMPAVWIQKQSWUAG73D4E4NRRSG4XEVPOUVIJPEQOUV7A6VVA, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016627093 (538h55m37.887710148s) [noise=true , address=45.91.116.9:4281 , piece_id=JZLDW777CBX2BP7MLOQV4NFP3H4XJVCNQXTHNSUWPVDOJLPZK36A, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gxf7DpJZthyR6f1PExX56tJavjhPw9Gxb4h8mtmeYKW3t3tS1): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016635258 (538h55m37.887718293s) [noise=true , address=45.91.116.9:4281 , piece_id=JZLDW777CBX2BP7MLOQV4NFP3H4XJVCNQXTHNSUWPVDOJLPZK36A, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016871298 (538h55m37.887954363s) [noise=true , address=145.79.227.44:4222 , piece_id=CZ2CS62CTHQFAIE3S3JO2N3VJ4XIR5YJ2LMRVLLX7I4VM2VZVQVA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZKW35KvrsoYRefbYeUFZo8bh4XZYycNKGEvTFjzibezjXgj3e): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016880124 (538h55m37.887963169s) [noise=true , address=145.79.227.44:4222 , piece_id=CZ2CS62CTHQFAIE3S3JO2N3VJ4XIR5YJ2LMRVLLX7I4VM2VZVQVA, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016934839 (538h55m37.888017884s) [noise=true , address=145.79.227.40:4062 , piece_id=PW6QWIPQWVUETLC6DJ624G6IB4P6EM4SYB72NAMQ4V6NUKNLUYLQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016943425 (538h55m37.88802649s) [noise=true , address=145.79.227.40:4062 , piece_id=PW6QWIPQWVUETLC6DJ624G6IB4P6EM4SYB72NAMQ4V6NUKNLUYLQ, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016968252 (538h55m37.888051297s) [seg_pos={0 0}, upload=472631]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 02:24:38.017067822 (538h55m37.888150867s) [upload=472631]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 02:24:38.017074244 (538h55m37.888157289s) [upload=472631]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 02:24:38.304306667 (538h55m38.175389712s) [upload=472631]: Done uploading segments... Dec 28 02:24:38.304329641 (538h55m38.175412686s) [upload=472631]: Upload finished. err: <nil> Duration: 38.626519558s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [.......................X................X....X............X.] duration: 3.501382475s trace: https://jaeger.dev.storj.io/trace/20df15b758999ef2 FAIL (2025-12-28 02:53:24 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.501409397s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 41 out of 60 is failed (threshold=3) [.XX..X.XXX.XXX.X.....XXX.XXXXXX.XXXX.XXXX..X..XXXXX.XXXXXXXX] duration: 30.003706624s trace: https://jaeger.dev.storj.io/trace/327f5f7f34acc549 FAIL (2025-12-28 02:52:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 02:52:17.065403534 (539h23m16.936486579s) [upload=473024]: Starting upload Dec 28 02:52:17.065463339 (539h23m16.936546384s) [upload=473024]: Done starting upload Dec 28 02:52:17.065528914 (539h23m16.936611949s) [upload=473024]: Uploading segments... Dec 28 02:52:17.065640676 (539h23m16.936723711s) [upload=473024]: Got next segment. Inline: false Dec 28 02:52:17.065685982 (539h23m16.936769017s) [upload=473024]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 02:52:17.065690651 (539h23m16.936773686s) [upload=473024]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 02:52:17.308460235 (539h23m17.17954327s) [seg_pos={0 0}, upload=473024]: Begin upload segment... Dec 28 02:52:17.308520069 (539h23m17.179603104s) [seg_pos={0 0}, upload=473024]: Done begin upload segment. Dec 28 02:52:17.308534156 (539h23m17.179617191s) [noise=true , address=77.90.31.13:4629 , piece_id=V2E7XLYB5MKCIFM3H75YFD355KAGDTFVUQVZ5TB7PHADQWSYLLJA, seg_pos={0 0}, upload=473024]: Uploading piece... Dec 28 02:52:17.308551028 (539h23m17.179634073s) [noise=true , address=145.79.227.46:4601 , piece_id=XI66ZYUIY43WOZAMQXD7KQBCB2EKNFTW7N3TZ2ZZDQIFBWFZHVBQ, seg_pos={0 0}, upload=473024]: Uploading piece... Dec 28 02:52:17.308997619 (539h23m17.180080654s) [noise=true , address=45.91.116.5:4297 , piece_id=THVDDIQQNOBCGYRUMYVB3E2VK3FJKMCUK6SNXQHWJVDOEMXS6B7Q, seg_pos={0 0}, upload=473024]: Uploading piece... Dec 28 02:52:17.309788875 (539h23m17.18087192s) [noise=true , address=45.91.116.10:4083 , piece_id=BCPF4H4SXIT4U4L2XEF3WACBOVSULYJJZYMS5FXKXN4FNXAC7PLQ, seg_pos={0 0}, upload=473024]: Uploading piece... Dec 28 02:52:17.310200759 (539h23m17.181283794s) [noise=true , address=45.91.116.8:4046 , piece_id=OJQK6QEU2GGAC76TFULBQFAW4HZODZD7XTZLYXMXEGAR2FTMKVUQ, seg_pos={0 0}, upload=473024]: Uploading piece... Dec 28 02:52:17.310494910 (539h23m17.181577945s) [noise=true , address=145.79.227.16:4912 , piece_id=2DS2XM752ZRSZ7DT7E2NJBJZX77AZ4O4XKSASANAYBQIZEPKKJRQ, seg_pos={0 0}, upload=473024]: Uploading piece... Dec 28 02:52:17.310887607 (539h23m17.181970643s) [noise=true , address=45.91.116.6:4973 , piece_id=5VT2GCCMZN3J3DUZGY4DT7ZIPWFVE3SW476MNPX5CHH4WYCUZOTQ, seg_pos={0 0}, upload=473024]: Uploading piece... Dec 28 02:52:17.311218537 (539h23m17.182301572s) [noise=true , address=145.79.227.49:4337 , piece_id=VPPVJGU7DOP4Z3WVBNJIDXUIU5WZX47CB3OVJGY3Z6UT5LIIW7NA, seg_pos={0 0}, upload=473024]: Uploading piece... Dec 28 02:52:17.311579756 (539h23m17.182662791s) [noise=true , address=145.79.227.34:4907 , piece_id=BC3PHAQ5QZJJV5POXRGCGFPIR2N6V2EMXEB5NMP377A3KN6JG57Q, seg_pos={0 0}, upload=473024]: Uploading piece... Dec 28 02:52:17.312631197 (539h23m17.183714233s) [noise=true , address=145.79.227.4:4185 , piece_id=IWB52ZMPDPYGKXFNH7JQ5EX6NJH5OFQZLKKI7ZMG5PJE63JHZG6A, seg_pos={0 0}, upload=473024]: Uploading piece... Dec 28 02:52:17.317735678 (539h23m17.188818723s) [noise=true , address=145.79.227.11:4732 , piece_id=WIF3Q2FG4EWS37SVTYEQF43AYS7GAMGFC3BSWWSWIQSOAF2F6FBA, seg_pos={0 0}, upload=473024]: Uploading piece... Dec 28 02:52:17.319364959 (539h23m17.190448004s) [noise=true , address=145.79.227.50:4374 , piece_id=PA3IIORSMPWI3B6N3HSSD35D5KF4XIVLGKQEA4SQB33B77OKSFAA, seg_pos={0 0}, upload=473024]: Uploading piece... Dec 28 02:52:17.321602451 (539h23m17.192685496s) [noise=true , address=45.91.116.12:4225 , piece_id=OFHWALT4JSCJCADL3TKSBGAUHIEYGPVUQTWPDDL4UHV4UC74LOYQ, seg_pos={0 0}, upload=473024]: Uploading piece... Dec 28 02:52:17.324355473 (539h23m17.195438508s) [noise=true , address=77.90.31.21:4880 , piece_id=DL3FPJYGFHRR4KRGNYXDXI3MKCSX75XL4HNWQAOFE6XV2CYOLDTQ, seg_pos={0 0}, upload=473024]: Uploading piece... Dec 28 02:52:17.325927967 (539h23m17.197011012s) [noise=true , address=45.91.116.4:4940 , piece_id=MQX5QIUI6OM3H5LVR3IYAKLEV75TFEB6NQSF6P5SAPLU2MQR53GA, seg_pos={0 0}, upload=473024]: Uploading piece... Dec 28 02:52:17.326481040 (539h23m17.197564085s) [noise=true , address=145.79.227.48:4301 , piece_id=HYA2OI35K2KTELPDU2P7EO34VHTI4XDW45E2D5PENV7V4PDXG6RQ, seg_pos={0 0}, upload=473024]: Uploading piece... Dec 28 02:52:18.941251953 (539h23m18.812334998s) [upload=473024]: Next returned nil segment Dec 28 02:52:18.941269085 (539h23m18.81235212s) [upload=473024]: Waiting for error group managing segments... Dec 28 02:52:19.324629717 (539h23m19.195712762s) [noise=true , address=145.79.227.16:4912 , piece_id=2DS2XM752ZRSZ7DT7E2NJBJZX77AZ4O4XKSASANAYBQIZEPKKJRQ, seg_pos={0 0}, upload=473024]: Done uploading piece. err: <nil> Dec 28 02:52:19.324862240 (539h23m19.195945275s) [noise=true , address=145.79.227.11:4732 , piece_id=WIF3Q2FG4EWS37SVTYEQF43AYS7GAMGFC3BSWWSWIQSOAF2F6FBA, seg_pos={0 0}, upload=473024]: Done uploading piece. err: <nil> Dec 28 02:52:19.327478712 (539h23m19.198561747s) [noise=true , address=145.79.227.50:4374 , piece_id=PA3IIORSMPWI3B6N3HSSD35D5KF4XIVLGKQEA4SQB33B77OKSFAA, seg_pos={0 0}, upload=473024]: Done uploading piece. err: <nil> Dec 28 02:52:19.341039304 (539h23m19.212122349s) [noise=true , address=145.79.227.48:4301 , piece_id=HYA2OI35K2KTELPDU2P7EO34VHTI4XDW45E2D5PENV7V4PDXG6RQ, seg_pos={0 0}, upload=473024]: Done uploading piece. err: <nil> Dec 28 02:52:19.479877105 (539h23m19.35096014s) [noise=true , address=77.90.31.13:4629 , piece_id=V2E7XLYB5MKCIFM3H75YFD355KAGDTFVUQVZ5TB7PHADQWSYLLJA, seg_pos={0 0}, upload=473024]: Done uploading piece. err: <nil> Dec 28 02:52:19.562180724 (539h23m19.433263769s) [noise=true , address=45.91.116.12:4225 , piece_id=OFHWALT4JSCJCADL3TKSBGAUHIEYGPVUQTWPDDL4UHV4UC74LOYQ, seg_pos={0 0}, upload=473024]: Done uploading piece. err: <nil> Dec 28 02:52:19.572919854 (539h23m19.444002899s) [noise=true , address=45.91.116.5:4297 , piece_id=THVDDIQQNOBCGYRUMYVB3E2VK3FJKMCUK6SNXQHWJVDOEMXS6B7Q, seg_pos={0 0}, upload=473024]: Done uploading piece. err: <nil> Dec 28 02:52:19.639353414 (539h23m19.510436459s) [noise=true , address=77.90.31.21:4880 , piece_id=DL3FPJYGFHRR4KRGNYXDXI3MKCSX75XL4HNWQAOFE6XV2CYOLDTQ, seg_pos={0 0}, upload=473024]: Done uploading piece. err: <nil> Dec 28 02:52:19.656275266 (539h23m19.527358351s) [noise=true , address=45.91.116.8:4046 , piece_id=OJQK6QEU2GGAC76TFULBQFAW4HZODZD7XTZLYXMXEGAR2FTMKVUQ, seg_pos={0 0}, upload=473024]: Done uploading piece. err: <nil> Dec 28 02:52:19.672301172 (539h23m19.543384217s) [noise=true , address=45.91.116.4:4940 , piece_id=MQX5QIUI6OM3H5LVR3IYAKLEV75TFEB6NQSF6P5SAPLU2MQR53GA, seg_pos={0 0}, upload=473024]: Done uploading piece. err: <nil> Dec 28 02:52:19.737660316 (539h23m19.608743351s) [noise=true , address=145.79.227.4:4185 , piece_id=IWB52ZMPDPYGKXFNH7JQ5EX6NJH5OFQZLKKI7ZMG5PJE63JHZG6A, seg_pos={0 0}, upload=473024]: Done uploading piece. err: <nil> Dec 28 02:52:19.924372348 (539h23m19.795455383s) [noise=true , address=45.91.116.6:4973 , piece_id=5VT2GCCMZN3J3DUZGY4DT7ZIPWFVE3SW476MNPX5CHH4WYCUZOTQ, seg_pos={0 0}, upload=473024]: Done uploading piece. err: <nil> Dec 28 02:52:20.276715147 (539h23m20.147798182s) [noise=true , address=145.79.227.46:4601 , piece_id=XI66ZYUIY43WOZAMQXD7KQBCB2EKNFTW7N3TZ2ZZDQIFBWFZHVBQ, seg_pos={0 0}, upload=473024]: Done uploading piece. err: <nil> Dec 28 02:52:20.276752357 (539h23m20.147835392s) [seg_pos={0 0}, upload=473024]: Segment reached optimal threshold of 13 pieces. Dec 28 02:52:20.276974941 (539h23m20.148057976s) [noise=true , address=145.79.227.49:4337 , piece_id=VPPVJGU7DOP4Z3WVBNJIDXUIU5WZX47CB3OVJGY3Z6UT5LIIW7NA, seg_pos={0 0}, upload=473024]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nTZFfzNCcVAjut7NXf9seifysJDfvXkS8aZSnXKfw6gh1TrjE): protocol: expected piece hash; context canceled; EOF Dec 28 02:52:20.276983087 (539h23m20.148066132s) [noise=true , address=145.79.227.49:4337 , piece_id=VPPVJGU7DOP4Z3WVBNJIDXUIU5WZX47CB3OVJGY3Z6UT5LIIW7NA, seg_pos={0 0}, upload=473024]: Long tail cancellation due to optimal threshold Dec 28 02:52:20.277029826 (539h23m20.148112861s) [noise=true , address=145.79.227.34:4907 , piece_id=BC3PHAQ5QZJJV5POXRGCGFPIR2N6V2EMXEB5NMP377A3KN6JG57Q, seg_pos={0 0}, upload=473024]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 28 02:52:20.277043902 (539h23m20.148126947s) [noise=true , address=145.79.227.34:4907 , piece_id=BC3PHAQ5QZJJV5POXRGCGFPIR2N6V2EMXEB5NMP377A3KN6JG57Q, seg_pos={0 0}, upload=473024]: Long tail cancellation due to optimal threshold Dec 28 02:52:20.277190021 (539h23m20.148273056s) [noise=true , address=45.91.116.10:4083 , piece_id=BCPF4H4SXIT4U4L2XEF3WACBOVSULYJJZYMS5FXKXN4FNXAC7PLQ, seg_pos={0 0}, upload=473024]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 28 02:52:20.277196323 (539h23m20.148279358s) [noise=true , address=45.91.116.10:4083 , piece_id=BCPF4H4SXIT4U4L2XEF3WACBOVSULYJJZYMS5FXKXN4FNXAC7PLQ, seg_pos={0 0}, upload=473024]: Long tail cancellation due to optimal threshold Dec 28 02:52:20.277212724 (539h23m20.148295759s) [seg_pos={0 0}, upload=473024]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 02:52:20.277275053 (539h23m20.148358088s) [upload=473024]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 02:52:20.277315119 (539h23m20.148398154s) [upload=473024]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 02:52:20.569372775 (539h23m20.44045581s) [upload=473024]: Done uploading segments... Dec 28 02:52:20.569391501 (539h23m20.440474546s) [upload=473024]: Upload finished. err: <nil> Duration: 30.003731421s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.X...................X.....X................................] duration: 14.595189933s trace: https://jaeger.dev.storj.io/trace/1822be52d200e5b5 FAIL (2025-12-28 02:52:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1ZAfYZ4wgpZVvatj6wtHona3hTaPcLxAmXC1DT2uKQyo8nLQQ7, Piece ID: 5MMQ44MXRT4XEL4X6WO54WRG2XYMX3AN3FNQ2H2XIUTKBXI3EEPA) context deadline exceeded; piecestore: (Node ID: 124guAM82oibYZsMho67Em6Ke6yEZXwT6LPMLLSZZBLqwoFGUkn, Piece ID: 22WJIPX4SM5NTIAW5ZRM2UKX32CTDBFCXPCCHRRWRCUCOF5CRE5Q) context deadline exceeded; piecestore: (Node ID: 1e6p1TD5CqUyWdZjkrxh2kRqMuyDiMUr9eimfeayDAKwhtBrZ8, Piece ID: XRI5EJFN7YYUG4QKLDGEFT6YAUHCUTTSXF735UDR6LZZBGMAZDJQ) context deadline exceeded ======LOGS====== 5.234848204 (538h55m35.105931239s) [noise=true , address=77.90.31.11:4572 , piece_id=5MMQ44MXRT4XEL4X6WO54WRG2XYMX3AN3FNQ2H2XIUTKBXI3EEPA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.235412057 (538h55m35.106495102s) [noise=true , address=145.79.227.22:4512 , piece_id=D46YLTIKGHG36DQ4YBKBSUDV4GGB4WVDWNOJS27IYYLN6AYZ5QFQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.235820946 (538h55m35.106903981s) [noise=true , address=77.90.31.19:4874 , piece_id=RHZUXUYMJU6RKQGSDZM5WDWIN5SQ7YIRSRATNNKT5GEMU2QH444A, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.236192192 (538h55m35.107275237s) [noise=true , address=77.90.31.16:4760 , piece_id=XRI5EJFN7YYUG4QKLDGEFT6YAUHCUTTSXF735UDR6LZZBGMAZDJQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.236564772 (538h55m35.107647817s) [noise=true , address=145.79.227.40:4062 , piece_id=PW6QWIPQWVUETLC6DJ624G6IB4P6EM4SYB72NAMQ4V6NUKNLUYLQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.236929766 (538h55m35.108012812s) [noise=true , address=64.49.13.139:4480 , piece_id=JAJY3F7IDFFMFKCK2QJV6W74RIESLRKBLXMN27Q3C45JWEPLBZLQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.237916846 (538h55m35.108999881s) [noise=true , address=145.79.227.34:4932 , piece_id=UPXSCQ45QDGPKNJXRFJ33EAA3QENVHRVQARU5TFJTEWICZNR253Q, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.238852948 (538h55m35.109935983s) [noise=true , address=45.91.116.12:4257 , piece_id=ZI67LIAHZW4G357GWKDPTB5SMXYGNXZTTTDMNAETDS2B6SWVDAEQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239152910 (538h55m35.110235945s) [noise=true , address=145.79.227.44:4222 , piece_id=CZ2CS62CTHQFAIE3S3JO2N3VJ4XIR5YJ2LMRVLLX7I4VM2VZVQVA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239409969 (538h55m35.110493004s) [noise=true , address=145.79.227.30:4773 , piece_id=KUGJXZCNMUE7WSK3VLY4AUBSOT4H6IY4XB4NCBGAGCNUG7F7IODQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239681136 (538h55m35.110764171s) [noise=true , address=77.90.31.13:4635 , piece_id=IIOY5X2W3XW4C2OFEBRCABMYFBI24RRTYNUSKN6KJU7CTUW4KONQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239954676 (538h55m35.111037711s) [noise=true , address=45.91.116.9:4281 , piece_id=JZLDW777CBX2BP7MLOQV4NFP3H4XJVCNQXTHNSUWPVDOJLPZK36A, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.240189914 (538h55m35.111272949s) [noise=true , address=145.79.227.35:4512 , piece_id=3E3VCCC45XVT5FHFDKXSTGGAGYBEML3BKU6JJTT33LCD2YMOMBZQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.240449147 (538h55m35.111532182s) [noise=true , address=45.91.116.8:4064 , piece_id=22WJIPX4SM5NTIAW5ZRM2UKX32CTDBFCXPCCHRRWRCUCOF5CRE5Q, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.240639640 (538h55m35.111722675s) [noise=true , address=145.79.227.32:4853 , piece_id=MBNCFDBQ4SZZXV4F7J5TRMOUYUQ5IPKQCGYGGM5SOIFJ6ZYCAUYA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.241137798 (538h55m35.112220833s) [noise=true , address=77.90.31.16:4751 , piece_id=N3Z2LLMCIA2F42VXZA7IDRU4XEKGOICH6TRVERODU76MFFE4QSHA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.241383797 (538h55m35.112466842s) [noise=true , address=77.90.31.18:4836 , piece_id=BSVEFI52EKN44CMINFUSZBNOH53IW5TXLVTF5IH6ZUTNII6RRLVA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:36.592880816 (538h55m36.463963871s) [upload=472631]: Next returned nil segment Dec 28 02:24:36.592901625 (538h55m36.46398467s) [upload=472631]: Waiting for error group managing segments... Dec 28 02:24:36.946933741 (538h55m36.818016786s) [noise=true , address=64.49.13.139:4480 , piece_id=JAJY3F7IDFFMFKCK2QJV6W74RIESLRKBLXMN27Q3C45JWEPLBZLQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.128189692 (538h55m36.999272737s) [noise=true , address=145.79.227.35:4512 , piece_id=3E3VCCC45XVT5FHFDKXSTGGAGYBEML3BKU6JJTT33LCD2YMOMBZQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.160760084 (538h55m37.031843119s) [noise=true , address=145.79.227.34:4932 , piece_id=UPXSCQ45QDGPKNJXRFJ33EAA3QENVHRVQARU5TFJTEWICZNR253Q, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.288838207 (538h55m37.159921252s) [noise=true , address=77.90.31.11:4572 , piece_id=5MMQ44MXRT4XEL4X6WO54WRG2XYMX3AN3FNQ2H2XIUTKBXI3EEPA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.319996310 (538h55m37.191079345s) [noise=true , address=145.79.227.22:4512 , piece_id=D46YLTIKGHG36DQ4YBKBSUDV4GGB4WVDWNOJS27IYYLN6AYZ5QFQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.320513273 (538h55m37.191596319s) [noise=true , address=77.90.31.18:4836 , piece_id=BSVEFI52EKN44CMINFUSZBNOH53IW5TXLVTF5IH6ZUTNII6RRLVA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.336522926 (538h55m37.207605971s) [noise=true , address=145.79.227.32:4853 , piece_id=MBNCFDBQ4SZZXV4F7J5TRMOUYUQ5IPKQCGYGGM5SOIFJ6ZYCAUYA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.384868031 (538h55m37.255951066s) [noise=true , address=77.90.31.16:4751 , piece_id=N3Z2LLMCIA2F42VXZA7IDRU4XEKGOICH6TRVERODU76MFFE4QSHA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.453563554 (538h55m37.324646599s) [noise=true , address=45.91.116.8:4064 , piece_id=22WJIPX4SM5NTIAW5ZRM2UKX32CTDBFCXPCCHRRWRCUCOF5CRE5Q, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.479454839 (538h55m37.350537884s) [noise=true , address=45.91.116.12:4257 , piece_id=ZI67LIAHZW4G357GWKDPTB5SMXYGNXZTTTDMNAETDS2B6SWVDAEQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.659710174 (538h55m37.530793219s) [noise=true , address=77.90.31.13:4635 , piece_id=IIOY5X2W3XW4C2OFEBRCABMYFBI24RRTYNUSKN6KJU7CTUW4KONQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.715039128 (538h55m37.586122163s) [noise=true , address=77.90.31.16:4760 , piece_id=XRI5EJFN7YYUG4QKLDGEFT6YAUHCUTTSXF735UDR6LZZBGMAZDJQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.977473406 (538h55m37.848556441s) [noise=true , address=77.90.31.19:4874 , piece_id=RHZUXUYMJU6RKQGSDZM5WDWIN5SQ7YIRSRATNNKT5GEMU2QH444A, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:38.016302745 (538h55m37.88738578s) [noise=true , address=145.79.227.30:4773 , piece_id=KUGJXZCNMUE7WSK3VLY4AUBSOT4H6IY4XB4NCBGAGCNUG7F7IODQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:38.016326982 (538h55m37.887410027s) [seg_pos={0 0}, upload=472631]: Segment reached optimal threshold of 14 pieces. Dec 28 02:24:38.016585353 (538h55m37.887668398s) [noise=true , address=145.79.227.30:4766 , piece_id=67M2DMPAVWIQKQSWUAG73D4E4NRRSG4XEVPOUVIJPEQOUV7A6VVA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1k6zdamuKPkBdzyNZNWx4FCmaTkv33PchcwvTRR8bgcETBs4K7): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016594450 (538h55m37.887677495s) [noise=true , address=145.79.227.30:4766 , piece_id=67M2DMPAVWIQKQSWUAG73D4E4NRRSG4XEVPOUVIJPEQOUV7A6VVA, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016627093 (538h55m37.887710148s) [noise=true , address=45.91.116.9:4281 , piece_id=JZLDW777CBX2BP7MLOQV4NFP3H4XJVCNQXTHNSUWPVDOJLPZK36A, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gxf7DpJZthyR6f1PExX56tJavjhPw9Gxb4h8mtmeYKW3t3tS1): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016635258 (538h55m37.887718293s) [noise=true , address=45.91.116.9:4281 , piece_id=JZLDW777CBX2BP7MLOQV4NFP3H4XJVCNQXTHNSUWPVDOJLPZK36A, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016871298 (538h55m37.887954363s) [noise=true , address=145.79.227.44:4222 , piece_id=CZ2CS62CTHQFAIE3S3JO2N3VJ4XIR5YJ2LMRVLLX7I4VM2VZVQVA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZKW35KvrsoYRefbYeUFZo8bh4XZYycNKGEvTFjzibezjXgj3e): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016880124 (538h55m37.887963169s) [noise=true , address=145.79.227.44:4222 , piece_id=CZ2CS62CTHQFAIE3S3JO2N3VJ4XIR5YJ2LMRVLLX7I4VM2VZVQVA, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016934839 (538h55m37.888017884s) [noise=true , address=145.79.227.40:4062 , piece_id=PW6QWIPQWVUETLC6DJ624G6IB4P6EM4SYB72NAMQ4V6NUKNLUYLQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016943425 (538h55m37.88802649s) [noise=true , address=145.79.227.40:4062 , piece_id=PW6QWIPQWVUETLC6DJ624G6IB4P6EM4SYB72NAMQ4V6NUKNLUYLQ, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016968252 (538h55m37.888051297s) [seg_pos={0 0}, upload=472631]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 02:24:38.017067822 (538h55m37.888150867s) [upload=472631]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 02:24:38.017074244 (538h55m37.888157289s) [upload=472631]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 02:24:38.304306667 (538h55m38.175389712s) [upload=472631]: Done uploading segments... Dec 28 02:24:38.304329641 (538h55m38.175412686s) [upload=472631]: Upload finished. err: <nil> Duration: 14.595219648s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [........................X................X....X............X] duration: 5.001949875s trace: https://jaeger.dev.storj.io/trace/20df15b758999ef2 FAIL (2025-12-28 02:52:21 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001973179s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [..X...................X.....X...............................] duration: 33.038083181s trace: https://jaeger.dev.storj.io/trace/1822be52d200e5b5 FAIL (2025-12-28 02:52:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1ZAfYZ4wgpZVvatj6wtHona3hTaPcLxAmXC1DT2uKQyo8nLQQ7, Piece ID: 5MMQ44MXRT4XEL4X6WO54WRG2XYMX3AN3FNQ2H2XIUTKBXI3EEPA) context deadline exceeded; piecestore: (Node ID: 124guAM82oibYZsMho67Em6Ke6yEZXwT6LPMLLSZZBLqwoFGUkn, Piece ID: 22WJIPX4SM5NTIAW5ZRM2UKX32CTDBFCXPCCHRRWRCUCOF5CRE5Q) context deadline exceeded; piecestore: (Node ID: 1e6p1TD5CqUyWdZjkrxh2kRqMuyDiMUr9eimfeayDAKwhtBrZ8, Piece ID: XRI5EJFN7YYUG4QKLDGEFT6YAUHCUTTSXF735UDR6LZZBGMAZDJQ) context deadline exceeded ======LOGS====== 5.234848204 (538h55m35.105931239s) [noise=true , address=77.90.31.11:4572 , piece_id=5MMQ44MXRT4XEL4X6WO54WRG2XYMX3AN3FNQ2H2XIUTKBXI3EEPA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.235412057 (538h55m35.106495102s) [noise=true , address=145.79.227.22:4512 , piece_id=D46YLTIKGHG36DQ4YBKBSUDV4GGB4WVDWNOJS27IYYLN6AYZ5QFQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.235820946 (538h55m35.106903981s) [noise=true , address=77.90.31.19:4874 , piece_id=RHZUXUYMJU6RKQGSDZM5WDWIN5SQ7YIRSRATNNKT5GEMU2QH444A, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.236192192 (538h55m35.107275237s) [noise=true , address=77.90.31.16:4760 , piece_id=XRI5EJFN7YYUG4QKLDGEFT6YAUHCUTTSXF735UDR6LZZBGMAZDJQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.236564772 (538h55m35.107647817s) [noise=true , address=145.79.227.40:4062 , piece_id=PW6QWIPQWVUETLC6DJ624G6IB4P6EM4SYB72NAMQ4V6NUKNLUYLQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.236929766 (538h55m35.108012812s) [noise=true , address=64.49.13.139:4480 , piece_id=JAJY3F7IDFFMFKCK2QJV6W74RIESLRKBLXMN27Q3C45JWEPLBZLQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.237916846 (538h55m35.108999881s) [noise=true , address=145.79.227.34:4932 , piece_id=UPXSCQ45QDGPKNJXRFJ33EAA3QENVHRVQARU5TFJTEWICZNR253Q, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.238852948 (538h55m35.109935983s) [noise=true , address=45.91.116.12:4257 , piece_id=ZI67LIAHZW4G357GWKDPTB5SMXYGNXZTTTDMNAETDS2B6SWVDAEQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239152910 (538h55m35.110235945s) [noise=true , address=145.79.227.44:4222 , piece_id=CZ2CS62CTHQFAIE3S3JO2N3VJ4XIR5YJ2LMRVLLX7I4VM2VZVQVA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239409969 (538h55m35.110493004s) [noise=true , address=145.79.227.30:4773 , piece_id=KUGJXZCNMUE7WSK3VLY4AUBSOT4H6IY4XB4NCBGAGCNUG7F7IODQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239681136 (538h55m35.110764171s) [noise=true , address=77.90.31.13:4635 , piece_id=IIOY5X2W3XW4C2OFEBRCABMYFBI24RRTYNUSKN6KJU7CTUW4KONQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239954676 (538h55m35.111037711s) [noise=true , address=45.91.116.9:4281 , piece_id=JZLDW777CBX2BP7MLOQV4NFP3H4XJVCNQXTHNSUWPVDOJLPZK36A, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.240189914 (538h55m35.111272949s) [noise=true , address=145.79.227.35:4512 , piece_id=3E3VCCC45XVT5FHFDKXSTGGAGYBEML3BKU6JJTT33LCD2YMOMBZQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.240449147 (538h55m35.111532182s) [noise=true , address=45.91.116.8:4064 , piece_id=22WJIPX4SM5NTIAW5ZRM2UKX32CTDBFCXPCCHRRWRCUCOF5CRE5Q, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.240639640 (538h55m35.111722675s) [noise=true , address=145.79.227.32:4853 , piece_id=MBNCFDBQ4SZZXV4F7J5TRMOUYUQ5IPKQCGYGGM5SOIFJ6ZYCAUYA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.241137798 (538h55m35.112220833s) [noise=true , address=77.90.31.16:4751 , piece_id=N3Z2LLMCIA2F42VXZA7IDRU4XEKGOICH6TRVERODU76MFFE4QSHA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.241383797 (538h55m35.112466842s) [noise=true , address=77.90.31.18:4836 , piece_id=BSVEFI52EKN44CMINFUSZBNOH53IW5TXLVTF5IH6ZUTNII6RRLVA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:36.592880816 (538h55m36.463963871s) [upload=472631]: Next returned nil segment Dec 28 02:24:36.592901625 (538h55m36.46398467s) [upload=472631]: Waiting for error group managing segments... Dec 28 02:24:36.946933741 (538h55m36.818016786s) [noise=true , address=64.49.13.139:4480 , piece_id=JAJY3F7IDFFMFKCK2QJV6W74RIESLRKBLXMN27Q3C45JWEPLBZLQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.128189692 (538h55m36.999272737s) [noise=true , address=145.79.227.35:4512 , piece_id=3E3VCCC45XVT5FHFDKXSTGGAGYBEML3BKU6JJTT33LCD2YMOMBZQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.160760084 (538h55m37.031843119s) [noise=true , address=145.79.227.34:4932 , piece_id=UPXSCQ45QDGPKNJXRFJ33EAA3QENVHRVQARU5TFJTEWICZNR253Q, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.288838207 (538h55m37.159921252s) [noise=true , address=77.90.31.11:4572 , piece_id=5MMQ44MXRT4XEL4X6WO54WRG2XYMX3AN3FNQ2H2XIUTKBXI3EEPA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.319996310 (538h55m37.191079345s) [noise=true , address=145.79.227.22:4512 , piece_id=D46YLTIKGHG36DQ4YBKBSUDV4GGB4WVDWNOJS27IYYLN6AYZ5QFQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.320513273 (538h55m37.191596319s) [noise=true , address=77.90.31.18:4836 , piece_id=BSVEFI52EKN44CMINFUSZBNOH53IW5TXLVTF5IH6ZUTNII6RRLVA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.336522926 (538h55m37.207605971s) [noise=true , address=145.79.227.32:4853 , piece_id=MBNCFDBQ4SZZXV4F7J5TRMOUYUQ5IPKQCGYGGM5SOIFJ6ZYCAUYA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.384868031 (538h55m37.255951066s) [noise=true , address=77.90.31.16:4751 , piece_id=N3Z2LLMCIA2F42VXZA7IDRU4XEKGOICH6TRVERODU76MFFE4QSHA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.453563554 (538h55m37.324646599s) [noise=true , address=45.91.116.8:4064 , piece_id=22WJIPX4SM5NTIAW5ZRM2UKX32CTDBFCXPCCHRRWRCUCOF5CRE5Q, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.479454839 (538h55m37.350537884s) [noise=true , address=45.91.116.12:4257 , piece_id=ZI67LIAHZW4G357GWKDPTB5SMXYGNXZTTTDMNAETDS2B6SWVDAEQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.659710174 (538h55m37.530793219s) [noise=true , address=77.90.31.13:4635 , piece_id=IIOY5X2W3XW4C2OFEBRCABMYFBI24RRTYNUSKN6KJU7CTUW4KONQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.715039128 (538h55m37.586122163s) [noise=true , address=77.90.31.16:4760 , piece_id=XRI5EJFN7YYUG4QKLDGEFT6YAUHCUTTSXF735UDR6LZZBGMAZDJQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.977473406 (538h55m37.848556441s) [noise=true , address=77.90.31.19:4874 , piece_id=RHZUXUYMJU6RKQGSDZM5WDWIN5SQ7YIRSRATNNKT5GEMU2QH444A, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:38.016302745 (538h55m37.88738578s) [noise=true , address=145.79.227.30:4773 , piece_id=KUGJXZCNMUE7WSK3VLY4AUBSOT4H6IY4XB4NCBGAGCNUG7F7IODQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:38.016326982 (538h55m37.887410027s) [seg_pos={0 0}, upload=472631]: Segment reached optimal threshold of 14 pieces. Dec 28 02:24:38.016585353 (538h55m37.887668398s) [noise=true , address=145.79.227.30:4766 , piece_id=67M2DMPAVWIQKQSWUAG73D4E4NRRSG4XEVPOUVIJPEQOUV7A6VVA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1k6zdamuKPkBdzyNZNWx4FCmaTkv33PchcwvTRR8bgcETBs4K7): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016594450 (538h55m37.887677495s) [noise=true , address=145.79.227.30:4766 , piece_id=67M2DMPAVWIQKQSWUAG73D4E4NRRSG4XEVPOUVIJPEQOUV7A6VVA, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016627093 (538h55m37.887710148s) [noise=true , address=45.91.116.9:4281 , piece_id=JZLDW777CBX2BP7MLOQV4NFP3H4XJVCNQXTHNSUWPVDOJLPZK36A, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gxf7DpJZthyR6f1PExX56tJavjhPw9Gxb4h8mtmeYKW3t3tS1): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016635258 (538h55m37.887718293s) [noise=true , address=45.91.116.9:4281 , piece_id=JZLDW777CBX2BP7MLOQV4NFP3H4XJVCNQXTHNSUWPVDOJLPZK36A, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016871298 (538h55m37.887954363s) [noise=true , address=145.79.227.44:4222 , piece_id=CZ2CS62CTHQFAIE3S3JO2N3VJ4XIR5YJ2LMRVLLX7I4VM2VZVQVA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZKW35KvrsoYRefbYeUFZo8bh4XZYycNKGEvTFjzibezjXgj3e): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016880124 (538h55m37.887963169s) [noise=true , address=145.79.227.44:4222 , piece_id=CZ2CS62CTHQFAIE3S3JO2N3VJ4XIR5YJ2LMRVLLX7I4VM2VZVQVA, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016934839 (538h55m37.888017884s) [noise=true , address=145.79.227.40:4062 , piece_id=PW6QWIPQWVUETLC6DJ624G6IB4P6EM4SYB72NAMQ4V6NUKNLUYLQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016943425 (538h55m37.88802649s) [noise=true , address=145.79.227.40:4062 , piece_id=PW6QWIPQWVUETLC6DJ624G6IB4P6EM4SYB72NAMQ4V6NUKNLUYLQ, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016968252 (538h55m37.888051297s) [seg_pos={0 0}, upload=472631]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 02:24:38.017067822 (538h55m37.888150867s) [upload=472631]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 02:24:38.017074244 (538h55m37.888157289s) [upload=472631]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 02:24:38.304306667 (538h55m38.175389712s) [upload=472631]: Done uploading segments... Dec 28 02:24:38.304329641 (538h55m38.175412686s) [upload=472631]: Upload finished. err: <nil> Duration: 33.038110742s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 40 out of 60 is failed (threshold=3) [..XX..X.XXX.XXX.X.....XXX.XXXXXX.XXXX.XXXX..X..XXXXX.XXXXXXX] duration: 30.002981064s trace: https://jaeger.dev.storj.io/trace/3c01a7e0c4c76c1a FAIL (2025-12-28 02:51:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 28 02:51:19.781482557 (539h22m19.652565602s) [upload=473011]: Starting upload Dec 28 02:51:19.781566977 (539h22m19.652650022s) [upload=473011]: Done starting upload Dec 28 02:51:19.781672228 (539h22m19.652755263s) [upload=473011]: Uploading segments... Dec 28 02:51:19.781953964 (539h22m19.653036999s) [upload=473011]: Got next segment. Inline: false Dec 28 02:51:19.782010923 (539h22m19.653093958s) [upload=473011]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 02:51:19.782043886 (539h22m19.653126921s) [upload=473011]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 02:51:20.029655970 (539h22m19.900739005s) [seg_pos={0 0}, upload=473011]: Begin upload segment... Dec 28 02:51:20.029713970 (539h22m19.900797005s) [seg_pos={0 0}, upload=473011]: Done begin upload segment. Dec 28 02:51:20.029749398 (539h22m19.900832443s) [noise=true , address=77.90.31.17:4773 , piece_id=YM5XIQ63QJJKBGVCQAUKCMJ7TYP5EQU4653OUBZGMOT2DWM6HL5Q, seg_pos={0 0}, upload=473011]: Uploading piece... Dec 28 02:51:20.029882832 (539h22m19.900965877s) [noise=true , address=145.79.227.18:4007 , piece_id=6EHNO76DHTTKFMOOOHH53LRVNBT7KUTSWVVUQ7LLMSAZJ4K3T5KQ, seg_pos={0 0}, upload=473011]: Uploading piece... Dec 28 02:51:20.030462024 (539h22m19.901545059s) [noise=true , address=145.79.227.45:4230 , piece_id=BOIHA23RRLSZVWPJUXUHNN437ABMU3NPDHGWN7WULZS2ACUOENMA, seg_pos={0 0}, upload=473011]: Uploading piece... Dec 28 02:51:20.030983367 (539h22m19.902066402s) [noise=true , address=45.91.116.14:4192 , piece_id=3UVHDI3W6NKOODBLNJFVLBUN66LB62BUIKZ6FJ2ECEVW2ZGSICZQ, seg_pos={0 0}, upload=473011]: Uploading piece... Dec 28 02:51:20.031297255 (539h22m19.9023803s) [noise=true , address=77.90.31.19:4841 , piece_id=SRUMJXFCZYB2KUPFDMKNMUUL5L35TU5NCBGMKGR5X5BRFLOKX3QA, seg_pos={0 0}, upload=473011]: Uploading piece... Dec 28 02:51:20.031586195 (539h22m19.90266924s) [noise=true , address=145.79.227.32:4836 , piece_id=JJUWCZKKHRKMPZRXIGJOOGFNDHTFSBWOV5AQW2U2HGO3XMCAWGHA, seg_pos={0 0}, upload=473011]: Uploading piece... Dec 28 02:51:20.031958624 (539h22m19.903041669s) [noise=true , address=77.90.31.14:4664 , piece_id=BNQG4DBWXATO64OCIOXTZY3LNR7AQ6QLYIYOVHGNN33E2OEC2DOQ, seg_pos={0 0}, upload=473011]: Uploading piece... Dec 28 02:51:20.032273864 (539h22m19.903356909s) [noise=true , address=145.79.227.5:4223 , piece_id=EFVDVM3NVKYJHTI4A72J43LCL3UEU7CFLUSYL7VMGJKI6RAPVAAQ, seg_pos={0 0}, upload=473011]: Uploading piece... Dec 28 02:51:20.032649860 (539h22m19.903732896s) [noise=true , address=77.90.31.13:4626 , piece_id=GHYKBLPUNNADMLDRAOUNNOH26VFQWNGSBRT5MO73F5C4ZOHRRTRA, seg_pos={0 0}, upload=473011]: Uploading piece... Dec 28 02:51:20.033067155 (539h22m19.90415021s) [noise=true , address=145.79.227.52:4446 , piece_id=5ROYQZ5NKJV23L4QH5OX4CWL6WPVJD263B6SJQB2PJMOHCP5LUAA, seg_pos={0 0}, upload=473011]: Uploading piece... Dec 28 02:51:20.033431799 (539h22m19.904514844s) [noise=true , address=145.79.227.6:4258 , piece_id=S4A2MDLEAKMNKXB454CLGJI6XMRZNVHSSUIFNSFTPBF5277MBMLA, seg_pos={0 0}, upload=473011]: Uploading piece... Dec 28 02:51:20.033828174 (539h22m19.904911209s) [noise=true , address=145.79.227.2:4115 , piece_id=4GLBBVM4WCTWUOG5ATEDQEHMTUM6KX6JZBEKDM6HMGPUS2COQEFA, seg_pos={0 0}, upload=473011]: Uploading piece... Dec 28 02:51:20.034157331 (539h22m19.905240376s) [noise=true , address=145.79.227.37:4566 , piece_id=VJY5UNZXQR7TU6YYDPJG76GH6A65CUBHU23FB3RW5ADIN5ISFQPQ, seg_pos={0 0}, upload=473011]: Uploading piece... Dec 28 02:51:20.034481268 (539h22m19.905564303s) [noise=true , address=145.79.227.47:4265 , piece_id=MPU5CUU3C4VE6TSTBITHGSQXLAVPDMIWV6CH7ZQHEWSDC42KB2EQ, seg_pos={0 0}, upload=473011]: Uploading piece... Dec 28 02:51:20.034794575 (539h22m19.90587762s) [noise=true , address=45.91.116.11:4120 , piece_id=CO5T5D5JTKH7JIDCKWYQ5LJX3AQZFA67Q3J77NLHK5MV4LNQYG2A, seg_pos={0 0}, upload=473011]: Uploading piece... Dec 28 02:51:20.037900078 (539h22m19.908983113s) [noise=true , address=45.91.116.7:4009 , piece_id=GOZURANBAT2WUTFKUVTCCVF64ID3XFHPRO5P6SEV7JTVWM7EYXAQ, seg_pos={0 0}, upload=473011]: Uploading piece... Dec 28 02:51:21.767252753 (539h22m21.638335798s) [upload=473011]: Next returned nil segment Dec 28 02:51:21.767270577 (539h22m21.638353622s) [upload=473011]: Waiting for error group managing segments... Dec 28 02:51:22.060299342 (539h22m21.931382377s) [noise=true , address=145.79.227.45:4230 , piece_id=BOIHA23RRLSZVWPJUXUHNN437ABMU3NPDHGWN7WULZS2ACUOENMA, seg_pos={0 0}, upload=473011]: Done uploading piece. err: <nil> Dec 28 02:51:22.072970130 (539h22m21.944053165s) [noise=true , address=145.79.227.52:4446 , piece_id=5ROYQZ5NKJV23L4QH5OX4CWL6WPVJD263B6SJQB2PJMOHCP5LUAA, seg_pos={0 0}, upload=473011]: Done uploading piece. err: <nil> Dec 28 02:51:22.075545764 (539h22m21.946628809s) [noise=true , address=145.79.227.6:4258 , piece_id=S4A2MDLEAKMNKXB454CLGJI6XMRZNVHSSUIFNSFTPBF5277MBMLA, seg_pos={0 0}, upload=473011]: Done uploading piece. err: <nil> Dec 28 02:51:22.080550324 (539h22m21.951633369s) [noise=true , address=145.79.227.32:4836 , piece_id=JJUWCZKKHRKMPZRXIGJOOGFNDHTFSBWOV5AQW2U2HGO3XMCAWGHA, seg_pos={0 0}, upload=473011]: Done uploading piece. err: <nil> Dec 28 02:51:22.083897838 (539h22m21.954980883s) [noise=true , address=145.79.227.47:4265 , piece_id=MPU5CUU3C4VE6TSTBITHGSQXLAVPDMIWV6CH7ZQHEWSDC42KB2EQ, seg_pos={0 0}, upload=473011]: Done uploading piece. err: <nil> Dec 28 02:51:22.234388096 (539h22m22.105471131s) [noise=true , address=77.90.31.19:4841 , piece_id=SRUMJXFCZYB2KUPFDMKNMUUL5L35TU5NCBGMKGR5X5BRFLOKX3QA, seg_pos={0 0}, upload=473011]: Done uploading piece. err: <nil> Dec 28 02:51:22.238925276 (539h22m22.110008311s) [noise=true , address=77.90.31.17:4773 , piece_id=YM5XIQ63QJJKBGVCQAUKCMJ7TYP5EQU4653OUBZGMOT2DWM6HL5Q, seg_pos={0 0}, upload=473011]: Done uploading piece. err: <nil> Dec 28 02:51:22.281393533 (539h22m22.152476568s) [noise=true , address=45.91.116.14:4192 , piece_id=3UVHDI3W6NKOODBLNJFVLBUN66LB62BUIKZ6FJ2ECEVW2ZGSICZQ, seg_pos={0 0}, upload=473011]: Done uploading piece. err: <nil> Dec 28 02:51:22.417432283 (539h22m22.288515318s) [noise=true , address=77.90.31.13:4626 , piece_id=GHYKBLPUNNADMLDRAOUNNOH26VFQWNGSBRT5MO73F5C4ZOHRRTRA, seg_pos={0 0}, upload=473011]: Done uploading piece. err: <nil> Dec 28 02:51:22.554854738 (539h22m22.425937783s) [noise=true , address=77.90.31.14:4664 , piece_id=BNQG4DBWXATO64OCIOXTZY3LNR7AQ6QLYIYOVHGNN33E2OEC2DOQ, seg_pos={0 0}, upload=473011]: Done uploading piece. err: <nil> Dec 28 02:51:22.609201824 (539h22m22.480284859s) [noise=true , address=45.91.116.11:4120 , piece_id=CO5T5D5JTKH7JIDCKWYQ5LJX3AQZFA67Q3J77NLHK5MV4LNQYG2A, seg_pos={0 0}, upload=473011]: Done uploading piece. err: <nil> Dec 28 02:51:23.067421211 (539h22m22.938504256s) [noise=true , address=45.91.116.7:4009 , piece_id=GOZURANBAT2WUTFKUVTCCVF64ID3XFHPRO5P6SEV7JTVWM7EYXAQ, seg_pos={0 0}, upload=473011]: Done uploading piece. err: <nil> Dec 28 02:51:23.240321793 (539h22m23.111404828s) [noise=true , address=145.79.227.18:4007 , piece_id=6EHNO76DHTTKFMOOOHH53LRVNBT7KUTSWVVUQ7LLMSAZJ4K3T5KQ, seg_pos={0 0}, upload=473011]: Done uploading piece. err: <nil> Dec 28 02:51:23.240340197 (539h22m23.111423232s) [seg_pos={0 0}, upload=473011]: Segment reached optimal threshold of 13 pieces. Dec 28 02:51:23.240549175 (539h22m23.11163222s) [noise=true , address=145.79.227.2:4115 , piece_id=4GLBBVM4WCTWUOG5ATEDQEHMTUM6KX6JZBEKDM6HMGPUS2COQEFA, seg_pos={0 0}, upload=473011]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: write tcp 188.166.225.170:46048->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:46048->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46048->145.79.227.2:4115: use of closed network connection Dec 28 02:51:23.240557321 (539h22m23.111640356s) [noise=true , address=145.79.227.2:4115 , piece_id=4GLBBVM4WCTWUOG5ATEDQEHMTUM6KX6JZBEKDM6HMGPUS2COQEFA, seg_pos={0 0}, upload=473011]: Long tail cancellation due to optimal threshold Dec 28 02:51:23.240723998 (539h22m23.111807053s) [noise=true , address=145.79.227.37:4566 , piece_id=VJY5UNZXQR7TU6YYDPJG76GH6A65CUBHU23FB3RW5ADIN5ISFQPQ, seg_pos={0 0}, upload=473011]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SaNJZaVix7BSbEWhu2Vmt3XjDDWCsQ8wTSiFR7MQ8oGwD9N2F): protocol: expected piece hash; context canceled; EOF Dec 28 02:51:23.240731994 (539h22m23.111815029s) [noise=true , address=145.79.227.37:4566 , piece_id=VJY5UNZXQR7TU6YYDPJG76GH6A65CUBHU23FB3RW5ADIN5ISFQPQ, seg_pos={0 0}, upload=473011]: Long tail cancellation due to optimal threshold Dec 28 02:51:23.240742373 (539h22m23.111825418s) [noise=true , address=145.79.227.5:4223 , piece_id=EFVDVM3NVKYJHTI4A72J43LCL3UEU7CFLUSYL7VMGJKI6RAPVAAQ, seg_pos={0 0}, upload=473011]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE): protocol: expected piece hash; context canceled; EOF Dec 28 02:51:23.240748345 (539h22m23.11183139s) [noise=true , address=145.79.227.5:4223 , piece_id=EFVDVM3NVKYJHTI4A72J43LCL3UEU7CFLUSYL7VMGJKI6RAPVAAQ, seg_pos={0 0}, upload=473011]: Long tail cancellation due to optimal threshold Dec 28 02:51:23.240767191 (539h22m23.111850236s) [seg_pos={0 0}, upload=473011]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 02:51:23.240844448 (539h22m23.111927493s) [upload=473011]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 02:51:23.240850379 (539h22m23.111933414s) [upload=473011]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 02:51:23.536376059 (539h22m23.407459104s) [upload=473011]: Done uploading segments... Dec 28 02:51:23.536398441 (539h22m23.407481486s) [upload=473011]: Upload finished. err: <nil> Duration: 30.003003055s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.........................X................X....X............] duration: 5.001319894s trace: https://jaeger.dev.storj.io/trace/5efee46a94502508 FAIL (2025-12-28 02:51:48 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001337588s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [...X...................X.....X..............................] duration: 38.073184494s trace: https://jaeger.dev.storj.io/trace/1822be52d200e5b5 FAIL (2025-12-28 02:51:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1ZAfYZ4wgpZVvatj6wtHona3hTaPcLxAmXC1DT2uKQyo8nLQQ7, Piece ID: 5MMQ44MXRT4XEL4X6WO54WRG2XYMX3AN3FNQ2H2XIUTKBXI3EEPA) context deadline exceeded; piecestore: (Node ID: 124guAM82oibYZsMho67Em6Ke6yEZXwT6LPMLLSZZBLqwoFGUkn, Piece ID: 22WJIPX4SM5NTIAW5ZRM2UKX32CTDBFCXPCCHRRWRCUCOF5CRE5Q) context deadline exceeded; piecestore: (Node ID: 1e6p1TD5CqUyWdZjkrxh2kRqMuyDiMUr9eimfeayDAKwhtBrZ8, Piece ID: XRI5EJFN7YYUG4QKLDGEFT6YAUHCUTTSXF735UDR6LZZBGMAZDJQ) context deadline exceeded ======LOGS====== 5.234848204 (538h55m35.105931239s) [noise=true , address=77.90.31.11:4572 , piece_id=5MMQ44MXRT4XEL4X6WO54WRG2XYMX3AN3FNQ2H2XIUTKBXI3EEPA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.235412057 (538h55m35.106495102s) [noise=true , address=145.79.227.22:4512 , piece_id=D46YLTIKGHG36DQ4YBKBSUDV4GGB4WVDWNOJS27IYYLN6AYZ5QFQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.235820946 (538h55m35.106903981s) [noise=true , address=77.90.31.19:4874 , piece_id=RHZUXUYMJU6RKQGSDZM5WDWIN5SQ7YIRSRATNNKT5GEMU2QH444A, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.236192192 (538h55m35.107275237s) [noise=true , address=77.90.31.16:4760 , piece_id=XRI5EJFN7YYUG4QKLDGEFT6YAUHCUTTSXF735UDR6LZZBGMAZDJQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.236564772 (538h55m35.107647817s) [noise=true , address=145.79.227.40:4062 , piece_id=PW6QWIPQWVUETLC6DJ624G6IB4P6EM4SYB72NAMQ4V6NUKNLUYLQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.236929766 (538h55m35.108012812s) [noise=true , address=64.49.13.139:4480 , piece_id=JAJY3F7IDFFMFKCK2QJV6W74RIESLRKBLXMN27Q3C45JWEPLBZLQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.237916846 (538h55m35.108999881s) [noise=true , address=145.79.227.34:4932 , piece_id=UPXSCQ45QDGPKNJXRFJ33EAA3QENVHRVQARU5TFJTEWICZNR253Q, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.238852948 (538h55m35.109935983s) [noise=true , address=45.91.116.12:4257 , piece_id=ZI67LIAHZW4G357GWKDPTB5SMXYGNXZTTTDMNAETDS2B6SWVDAEQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239152910 (538h55m35.110235945s) [noise=true , address=145.79.227.44:4222 , piece_id=CZ2CS62CTHQFAIE3S3JO2N3VJ4XIR5YJ2LMRVLLX7I4VM2VZVQVA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239409969 (538h55m35.110493004s) [noise=true , address=145.79.227.30:4773 , piece_id=KUGJXZCNMUE7WSK3VLY4AUBSOT4H6IY4XB4NCBGAGCNUG7F7IODQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239681136 (538h55m35.110764171s) [noise=true , address=77.90.31.13:4635 , piece_id=IIOY5X2W3XW4C2OFEBRCABMYFBI24RRTYNUSKN6KJU7CTUW4KONQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239954676 (538h55m35.111037711s) [noise=true , address=45.91.116.9:4281 , piece_id=JZLDW777CBX2BP7MLOQV4NFP3H4XJVCNQXTHNSUWPVDOJLPZK36A, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.240189914 (538h55m35.111272949s) [noise=true , address=145.79.227.35:4512 , piece_id=3E3VCCC45XVT5FHFDKXSTGGAGYBEML3BKU6JJTT33LCD2YMOMBZQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.240449147 (538h55m35.111532182s) [noise=true , address=45.91.116.8:4064 , piece_id=22WJIPX4SM5NTIAW5ZRM2UKX32CTDBFCXPCCHRRWRCUCOF5CRE5Q, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.240639640 (538h55m35.111722675s) [noise=true , address=145.79.227.32:4853 , piece_id=MBNCFDBQ4SZZXV4F7J5TRMOUYUQ5IPKQCGYGGM5SOIFJ6ZYCAUYA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.241137798 (538h55m35.112220833s) [noise=true , address=77.90.31.16:4751 , piece_id=N3Z2LLMCIA2F42VXZA7IDRU4XEKGOICH6TRVERODU76MFFE4QSHA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.241383797 (538h55m35.112466842s) [noise=true , address=77.90.31.18:4836 , piece_id=BSVEFI52EKN44CMINFUSZBNOH53IW5TXLVTF5IH6ZUTNII6RRLVA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:36.592880816 (538h55m36.463963871s) [upload=472631]: Next returned nil segment Dec 28 02:24:36.592901625 (538h55m36.46398467s) [upload=472631]: Waiting for error group managing segments... Dec 28 02:24:36.946933741 (538h55m36.818016786s) [noise=true , address=64.49.13.139:4480 , piece_id=JAJY3F7IDFFMFKCK2QJV6W74RIESLRKBLXMN27Q3C45JWEPLBZLQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.128189692 (538h55m36.999272737s) [noise=true , address=145.79.227.35:4512 , piece_id=3E3VCCC45XVT5FHFDKXSTGGAGYBEML3BKU6JJTT33LCD2YMOMBZQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.160760084 (538h55m37.031843119s) [noise=true , address=145.79.227.34:4932 , piece_id=UPXSCQ45QDGPKNJXRFJ33EAA3QENVHRVQARU5TFJTEWICZNR253Q, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.288838207 (538h55m37.159921252s) [noise=true , address=77.90.31.11:4572 , piece_id=5MMQ44MXRT4XEL4X6WO54WRG2XYMX3AN3FNQ2H2XIUTKBXI3EEPA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.319996310 (538h55m37.191079345s) [noise=true , address=145.79.227.22:4512 , piece_id=D46YLTIKGHG36DQ4YBKBSUDV4GGB4WVDWNOJS27IYYLN6AYZ5QFQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.320513273 (538h55m37.191596319s) [noise=true , address=77.90.31.18:4836 , piece_id=BSVEFI52EKN44CMINFUSZBNOH53IW5TXLVTF5IH6ZUTNII6RRLVA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.336522926 (538h55m37.207605971s) [noise=true , address=145.79.227.32:4853 , piece_id=MBNCFDBQ4SZZXV4F7J5TRMOUYUQ5IPKQCGYGGM5SOIFJ6ZYCAUYA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.384868031 (538h55m37.255951066s) [noise=true , address=77.90.31.16:4751 , piece_id=N3Z2LLMCIA2F42VXZA7IDRU4XEKGOICH6TRVERODU76MFFE4QSHA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.453563554 (538h55m37.324646599s) [noise=true , address=45.91.116.8:4064 , piece_id=22WJIPX4SM5NTIAW5ZRM2UKX32CTDBFCXPCCHRRWRCUCOF5CRE5Q, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.479454839 (538h55m37.350537884s) [noise=true , address=45.91.116.12:4257 , piece_id=ZI67LIAHZW4G357GWKDPTB5SMXYGNXZTTTDMNAETDS2B6SWVDAEQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.659710174 (538h55m37.530793219s) [noise=true , address=77.90.31.13:4635 , piece_id=IIOY5X2W3XW4C2OFEBRCABMYFBI24RRTYNUSKN6KJU7CTUW4KONQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.715039128 (538h55m37.586122163s) [noise=true , address=77.90.31.16:4760 , piece_id=XRI5EJFN7YYUG4QKLDGEFT6YAUHCUTTSXF735UDR6LZZBGMAZDJQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.977473406 (538h55m37.848556441s) [noise=true , address=77.90.31.19:4874 , piece_id=RHZUXUYMJU6RKQGSDZM5WDWIN5SQ7YIRSRATNNKT5GEMU2QH444A, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:38.016302745 (538h55m37.88738578s) [noise=true , address=145.79.227.30:4773 , piece_id=KUGJXZCNMUE7WSK3VLY4AUBSOT4H6IY4XB4NCBGAGCNUG7F7IODQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:38.016326982 (538h55m37.887410027s) [seg_pos={0 0}, upload=472631]: Segment reached optimal threshold of 14 pieces. Dec 28 02:24:38.016585353 (538h55m37.887668398s) [noise=true , address=145.79.227.30:4766 , piece_id=67M2DMPAVWIQKQSWUAG73D4E4NRRSG4XEVPOUVIJPEQOUV7A6VVA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1k6zdamuKPkBdzyNZNWx4FCmaTkv33PchcwvTRR8bgcETBs4K7): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016594450 (538h55m37.887677495s) [noise=true , address=145.79.227.30:4766 , piece_id=67M2DMPAVWIQKQSWUAG73D4E4NRRSG4XEVPOUVIJPEQOUV7A6VVA, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016627093 (538h55m37.887710148s) [noise=true , address=45.91.116.9:4281 , piece_id=JZLDW777CBX2BP7MLOQV4NFP3H4XJVCNQXTHNSUWPVDOJLPZK36A, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gxf7DpJZthyR6f1PExX56tJavjhPw9Gxb4h8mtmeYKW3t3tS1): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016635258 (538h55m37.887718293s) [noise=true , address=45.91.116.9:4281 , piece_id=JZLDW777CBX2BP7MLOQV4NFP3H4XJVCNQXTHNSUWPVDOJLPZK36A, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016871298 (538h55m37.887954363s) [noise=true , address=145.79.227.44:4222 , piece_id=CZ2CS62CTHQFAIE3S3JO2N3VJ4XIR5YJ2LMRVLLX7I4VM2VZVQVA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZKW35KvrsoYRefbYeUFZo8bh4XZYycNKGEvTFjzibezjXgj3e): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016880124 (538h55m37.887963169s) [noise=true , address=145.79.227.44:4222 , piece_id=CZ2CS62CTHQFAIE3S3JO2N3VJ4XIR5YJ2LMRVLLX7I4VM2VZVQVA, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016934839 (538h55m37.888017884s) [noise=true , address=145.79.227.40:4062 , piece_id=PW6QWIPQWVUETLC6DJ624G6IB4P6EM4SYB72NAMQ4V6NUKNLUYLQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016943425 (538h55m37.88802649s) [noise=true , address=145.79.227.40:4062 , piece_id=PW6QWIPQWVUETLC6DJ624G6IB4P6EM4SYB72NAMQ4V6NUKNLUYLQ, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016968252 (538h55m37.888051297s) [seg_pos={0 0}, upload=472631]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 02:24:38.017067822 (538h55m37.888150867s) [upload=472631]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 02:24:38.017074244 (538h55m37.888157289s) [upload=472631]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 02:24:38.304306667 (538h55m38.175389712s) [upload=472631]: Done uploading segments... Dec 28 02:24:38.304329641 (538h55m38.175412686s) [upload=472631]: Upload finished. err: <nil> Duration: 38.073223587s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 39 out of 60 is failed (threshold=3) [...XX..X.XXX.XXX.X.....XXX.XXXXXX.XXXX.XXXX..X..XXXXX.XXXXXX] duration: 30.002606949s trace: https://jaeger.dev.storj.io/trace/4e54152642eba617 FAIL (2025-12-28 02:50:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, Piece ID: MX6RY5XQUZWHWHLFLJVYYH7HQUZBGKBTY2FUX2U55AVXWAYEAAHQ) context deadline exceeded; piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: 3A3GDIGDSEB2SEBNUCQUTHLZ4PE55RBW2MDXIYVD27AG57HY4IUA) context deadline exceeded ======LOGS====== Dec 28 02:50:24.508108823 (539h21m24.379191858s) [upload=473001]: Starting upload Dec 28 02:50:24.508175039 (539h21m24.379258084s) [upload=473001]: Done starting upload Dec 28 02:50:24.508307942 (539h21m24.379390977s) [upload=473001]: Uploading segments... Dec 28 02:50:24.508386191 (539h21m24.379469226s) [upload=473001]: Got next segment. Inline: false Dec 28 02:50:24.508422660 (539h21m24.379505695s) [upload=473001]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 02:50:24.508427710 (539h21m24.379510745s) [upload=473001]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 02:50:24.748420066 (539h21m24.619503101s) [seg_pos={0 0}, upload=473001]: Begin upload segment... Dec 28 02:50:24.748480260 (539h21m24.619563305s) [seg_pos={0 0}, upload=473001]: Done begin upload segment. Dec 28 02:50:24.748509075 (539h21m24.619592111s) [noise=true , address=45.91.116.12:4226 , piece_id=NOT2PO3PGK54TRWDFX6FGJBXUEIJRVNTHBVBJW7ZPPGOXMPHSAXQ, seg_pos={0 0}, upload=473001]: Uploading piece... Dec 28 02:50:24.748525967 (539h21m24.619609002s) [noise=true , address=45.91.116.10:4083 , piece_id=3DWY33PEJEHR7HNOIENCYTLUKPK63OAHDXEQYSHRD2JN7F66Z3TA, seg_pos={0 0}, upload=473001]: Uploading piece... Dec 28 02:50:24.749092887 (539h21m24.620175932s) [noise=true , address=145.79.227.33:4871 , piece_id=YB4KMMFDBO7N2K7LI7LLVGHG7P4F37CECWEGQVYBFLU6TNMGGNCQ, seg_pos={0 0}, upload=473001]: Uploading piece... Dec 28 02:50:24.749784203 (539h21m24.620867248s) [noise=true , address=145.79.227.35:4482 , piece_id=QT2UF5ZPDOEAKKZWZQPBIYPGNVQURUQ5V7UWQLA5I443AQ7L45JQ, seg_pos={0 0}, upload=473001]: Uploading piece... Dec 28 02:50:24.750317198 (539h21m24.621400234s) [noise=true , address=45.91.116.14:4190 , piece_id=3A3GDIGDSEB2SEBNUCQUTHLZ4PE55RBW2MDXIYVD27AG57HY4IUA, seg_pos={0 0}, upload=473001]: Uploading piece... Dec 28 02:50:24.750709886 (539h21m24.621792921s) [noise=true , address=145.79.227.45:4229 , piece_id=XQASUF464MVZMR7YLCB5Z5BSPNGZGAYTR6FC5YWKEOULIL7OVYXA, seg_pos={0 0}, upload=473001]: Uploading piece... Dec 28 02:50:24.751017402 (539h21m24.622100447s) [noise=true , address=145.79.227.5:4221 , piece_id=LTWZFQXHNVFWLJT44MZQ2CEYEOWQLX7PKIDRVZSYXG7476X4OVLA, seg_pos={0 0}, upload=473001]: Uploading piece... Dec 28 02:50:24.751314397 (539h21m24.622397432s) [noise=true , address=145.79.227.46:4603 , piece_id=LLPAIWDPU3YFC64OMFYVUB7R6G7PLC6HWEXOBB3RSCN53DC3OTEA, seg_pos={0 0}, upload=473001]: Uploading piece... Dec 28 02:50:24.752741113 (539h21m24.623824158s) [noise=true , address=145.79.227.36:4518 , piece_id=3B5RMZX7APP22VRODYWU3RWU2YFHOZTEUDW2JEPCJ7PK6EL66DSQ, seg_pos={0 0}, upload=473001]: Uploading piece... Dec 28 02:50:24.752867805 (539h21m24.62395085s) [noise=true , address=77.90.31.11:4555 , piece_id=MX6RY5XQUZWHWHLFLJVYYH7HQUZBGKBTY2FUX2U55AVXWAYEAAHQ, seg_pos={0 0}, upload=473001]: Uploading piece... Dec 28 02:50:24.754154986 (539h21m24.625238031s) [noise=true , address=145.79.227.42:4121 , piece_id=LW6TUYOCSOKGIB7MUW4YBOHN5B5LFYGXXPTLGGUXMB4W7Q7MZPEA, seg_pos={0 0}, upload=473001]: Uploading piece... Dec 28 02:50:24.754260988 (539h21m24.625344033s) [noise=true , address=145.79.227.31:4797 , piece_id=FFTLKASIOOKRAWYXA2BDA3FIJUUNUH7JV6NZRPP4P4OZSGJQW3HA, seg_pos={0 0}, upload=473001]: Uploading piece... Dec 28 02:50:24.755862978 (539h21m24.626946023s) [noise=true , address=145.79.227.50:4374 , piece_id=7E4QTXTXAKMZ6PHUF3WJUPG35YTFEISDXIPVPPO65N4T3LNEOKXQ, seg_pos={0 0}, upload=473001]: Uploading piece... Dec 28 02:50:24.756413968 (539h21m24.627497003s) [noise=true , address=145.79.227.18:4006 , piece_id=W2OOUZHJMQRESJ5JTJZLTARCWZBK6URROFPNVGWAIBXGKSDHHGZQ, seg_pos={0 0}, upload=473001]: Uploading piece... Dec 28 02:50:24.757956344 (539h21m24.629039379s) [noise=true , address=77.90.31.12:4589 , piece_id=U2UC7ZDCOAOB46UUYMVQRELALQNZP7WLJMIHDLYK4URXSLUVGNPA, seg_pos={0 0}, upload=473001]: Uploading piece... Dec 28 02:50:24.759238606 (539h21m24.630321641s) [noise=true , address=145.79.227.39:4014 , piece_id=2IR5Y43NIMHTQEL4G5NALHADMCSYVVZOH5Q6QPODUXCD36RMH3WQ, seg_pos={0 0}, upload=473001]: Uploading piece... Dec 28 02:50:26.504120341 (539h21m26.375203386s) [upload=473001]: Next returned nil segment Dec 28 02:50:26.504160347 (539h21m26.375243392s) [upload=473001]: Waiting for error group managing segments... Dec 28 02:50:26.787989395 (539h21m26.65907243s) [noise=true , address=145.79.227.35:4482 , piece_id=QT2UF5ZPDOEAKKZWZQPBIYPGNVQURUQ5V7UWQLA5I443AQ7L45JQ, seg_pos={0 0}, upload=473001]: Done uploading piece. err: <nil> Dec 28 02:50:26.799354406 (539h21m26.670437451s) [noise=true , address=145.79.227.42:4121 , piece_id=LW6TUYOCSOKGIB7MUW4YBOHN5B5LFYGXXPTLGGUXMB4W7Q7MZPEA, seg_pos={0 0}, upload=473001]: Done uploading piece. err: <nil> Dec 28 02:50:26.817935798 (539h21m26.689018833s) [noise=true , address=145.79.227.36:4518 , piece_id=3B5RMZX7APP22VRODYWU3RWU2YFHOZTEUDW2JEPCJ7PK6EL66DSQ, seg_pos={0 0}, upload=473001]: Done uploading piece. err: <nil> Dec 28 02:50:26.904047018 (539h21m26.775130063s) [noise=true , address=145.79.227.31:4797 , piece_id=FFTLKASIOOKRAWYXA2BDA3FIJUUNUH7JV6NZRPP4P4OZSGJQW3HA, seg_pos={0 0}, upload=473001]: Done uploading piece. err: <nil> Dec 28 02:50:26.940858663 (539h21m26.811941709s) [noise=true , address=77.90.31.11:4555 , piece_id=MX6RY5XQUZWHWHLFLJVYYH7HQUZBGKBTY2FUX2U55AVXWAYEAAHQ, seg_pos={0 0}, upload=473001]: Done uploading piece. err: <nil> Dec 28 02:50:26.943334949 (539h21m26.814417994s) [noise=true , address=145.79.227.33:4871 , piece_id=YB4KMMFDBO7N2K7LI7LLVGHG7P4F37CECWEGQVYBFLU6TNMGGNCQ, seg_pos={0 0}, upload=473001]: Done uploading piece. err: <nil> Dec 28 02:50:27.002385100 (539h21m26.873468135s) [noise=true , address=45.91.116.14:4190 , piece_id=3A3GDIGDSEB2SEBNUCQUTHLZ4PE55RBW2MDXIYVD27AG57HY4IUA, seg_pos={0 0}, upload=473001]: Done uploading piece. err: <nil> Dec 28 02:50:27.444214958 (539h21m27.315297993s) [noise=true , address=45.91.116.12:4226 , piece_id=NOT2PO3PGK54TRWDFX6FGJBXUEIJRVNTHBVBJW7ZPPGOXMPHSAXQ, seg_pos={0 0}, upload=473001]: Done uploading piece. err: <nil> Dec 28 02:50:27.822254067 (539h21m27.693337112s) [noise=true , address=77.90.31.12:4589 , piece_id=U2UC7ZDCOAOB46UUYMVQRELALQNZP7WLJMIHDLYK4URXSLUVGNPA, seg_pos={0 0}, upload=473001]: Done uploading piece. err: <nil> Dec 28 02:50:27.931856258 (539h21m27.802939293s) [noise=true , address=145.79.227.46:4603 , piece_id=LLPAIWDPU3YFC64OMFYVUB7R6G7PLC6HWEXOBB3RSCN53DC3OTEA, seg_pos={0 0}, upload=473001]: Done uploading piece. err: <nil> Dec 28 02:50:27.939375937 (539h21m27.810458982s) [noise=true , address=145.79.227.45:4229 , piece_id=XQASUF464MVZMR7YLCB5Z5BSPNGZGAYTR6FC5YWKEOULIL7OVYXA, seg_pos={0 0}, upload=473001]: Done uploading piece. err: <nil> Dec 28 02:50:27.944953196 (539h21m27.816036231s) [noise=true , address=145.79.227.39:4014 , piece_id=2IR5Y43NIMHTQEL4G5NALHADMCSYVVZOH5Q6QPODUXCD36RMH3WQ, seg_pos={0 0}, upload=473001]: Done uploading piece. err: <nil> Dec 28 02:50:27.947603363 (539h21m27.818686408s) [noise=true , address=145.79.227.50:4374 , piece_id=7E4QTXTXAKMZ6PHUF3WJUPG35YTFEISDXIPVPPO65N4T3LNEOKXQ, seg_pos={0 0}, upload=473001]: Done uploading piece. err: <nil> Dec 28 02:50:27.947621788 (539h21m27.818704833s) [seg_pos={0 0}, upload=473001]: Segment reached optimal threshold of 13 pieces. Dec 28 02:50:27.947851966 (539h21m27.818935011s) [noise=true , address=145.79.227.5:4221 , piece_id=LTWZFQXHNVFWLJT44MZQ2CEYEOWQLX7PKIDRVZSYXG7476X4OVLA, seg_pos={0 0}, upload=473001]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR): protocol: expected piece hash; context canceled; EOF Dec 28 02:50:27.947860813 (539h21m27.818943859s) [noise=true , address=145.79.227.5:4221 , piece_id=LTWZFQXHNVFWLJT44MZQ2CEYEOWQLX7PKIDRVZSYXG7476X4OVLA, seg_pos={0 0}, upload=473001]: Long tail cancellation due to optimal threshold Dec 28 02:50:27.948131248 (539h21m27.819214293s) [noise=true , address=145.79.227.18:4006 , piece_id=W2OOUZHJMQRESJ5JTJZLTARCWZBK6URROFPNVGWAIBXGKSDHHGZQ, seg_pos={0 0}, upload=473001]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc): protocol: expected piece hash; context canceled; EOF Dec 28 02:50:27.948139574 (539h21m27.819222629s) [noise=true , address=145.79.227.18:4006 , piece_id=W2OOUZHJMQRESJ5JTJZLTARCWZBK6URROFPNVGWAIBXGKSDHHGZQ, seg_pos={0 0}, upload=473001]: Long tail cancellation due to optimal threshold Dec 28 02:50:27.948293828 (539h21m27.819376873s) [noise=true , address=45.91.116.10:4083 , piece_id=3DWY33PEJEHR7HNOIENCYTLUKPK63OAHDXEQYSHRD2JN7F66Z3TA, seg_pos={0 0}, upload=473001]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 28 02:50:27.948301632 (539h21m27.819384667s) [noise=true , address=45.91.116.10:4083 , piece_id=3DWY33PEJEHR7HNOIENCYTLUKPK63OAHDXEQYSHRD2JN7F66Z3TA, seg_pos={0 0}, upload=473001]: Long tail cancellation due to optimal threshold Dec 28 02:50:27.948320619 (539h21m27.819403664s) [seg_pos={0 0}, upload=473001]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 02:50:27.948396203 (539h21m27.819479238s) [upload=473001]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 02:50:27.948402144 (539h21m27.819485179s) [upload=473001]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 02:50:28.231528152 (539h21m28.102611197s) [upload=473001]: Done uploading segments... Dec 28 02:50:28.231551216 (539h21m28.102634262s) [upload=473001]: Upload finished. err: <nil> Duration: 30.002629623s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [X.........................X................X....X...........] duration: 3.554171235s trace: https://jaeger.dev.storj.io/trace/5efee46a94502508 FAIL (2025-12-28 02:50:46 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.554525871s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [....X...................X.....X.............................] duration: 29.169847065s trace: https://jaeger.dev.storj.io/trace/1822be52d200e5b5 FAIL (2025-12-28 02:50:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1ZAfYZ4wgpZVvatj6wtHona3hTaPcLxAmXC1DT2uKQyo8nLQQ7, Piece ID: 5MMQ44MXRT4XEL4X6WO54WRG2XYMX3AN3FNQ2H2XIUTKBXI3EEPA) context deadline exceeded; piecestore: (Node ID: 124guAM82oibYZsMho67Em6Ke6yEZXwT6LPMLLSZZBLqwoFGUkn, Piece ID: 22WJIPX4SM5NTIAW5ZRM2UKX32CTDBFCXPCCHRRWRCUCOF5CRE5Q) context deadline exceeded; piecestore: (Node ID: 1e6p1TD5CqUyWdZjkrxh2kRqMuyDiMUr9eimfeayDAKwhtBrZ8, Piece ID: XRI5EJFN7YYUG4QKLDGEFT6YAUHCUTTSXF735UDR6LZZBGMAZDJQ) context deadline exceeded ======LOGS====== 5.234848204 (538h55m35.105931239s) [noise=true , address=77.90.31.11:4572 , piece_id=5MMQ44MXRT4XEL4X6WO54WRG2XYMX3AN3FNQ2H2XIUTKBXI3EEPA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.235412057 (538h55m35.106495102s) [noise=true , address=145.79.227.22:4512 , piece_id=D46YLTIKGHG36DQ4YBKBSUDV4GGB4WVDWNOJS27IYYLN6AYZ5QFQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.235820946 (538h55m35.106903981s) [noise=true , address=77.90.31.19:4874 , piece_id=RHZUXUYMJU6RKQGSDZM5WDWIN5SQ7YIRSRATNNKT5GEMU2QH444A, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.236192192 (538h55m35.107275237s) [noise=true , address=77.90.31.16:4760 , piece_id=XRI5EJFN7YYUG4QKLDGEFT6YAUHCUTTSXF735UDR6LZZBGMAZDJQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.236564772 (538h55m35.107647817s) [noise=true , address=145.79.227.40:4062 , piece_id=PW6QWIPQWVUETLC6DJ624G6IB4P6EM4SYB72NAMQ4V6NUKNLUYLQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.236929766 (538h55m35.108012812s) [noise=true , address=64.49.13.139:4480 , piece_id=JAJY3F7IDFFMFKCK2QJV6W74RIESLRKBLXMN27Q3C45JWEPLBZLQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.237916846 (538h55m35.108999881s) [noise=true , address=145.79.227.34:4932 , piece_id=UPXSCQ45QDGPKNJXRFJ33EAA3QENVHRVQARU5TFJTEWICZNR253Q, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.238852948 (538h55m35.109935983s) [noise=true , address=45.91.116.12:4257 , piece_id=ZI67LIAHZW4G357GWKDPTB5SMXYGNXZTTTDMNAETDS2B6SWVDAEQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239152910 (538h55m35.110235945s) [noise=true , address=145.79.227.44:4222 , piece_id=CZ2CS62CTHQFAIE3S3JO2N3VJ4XIR5YJ2LMRVLLX7I4VM2VZVQVA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239409969 (538h55m35.110493004s) [noise=true , address=145.79.227.30:4773 , piece_id=KUGJXZCNMUE7WSK3VLY4AUBSOT4H6IY4XB4NCBGAGCNUG7F7IODQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239681136 (538h55m35.110764171s) [noise=true , address=77.90.31.13:4635 , piece_id=IIOY5X2W3XW4C2OFEBRCABMYFBI24RRTYNUSKN6KJU7CTUW4KONQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239954676 (538h55m35.111037711s) [noise=true , address=45.91.116.9:4281 , piece_id=JZLDW777CBX2BP7MLOQV4NFP3H4XJVCNQXTHNSUWPVDOJLPZK36A, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.240189914 (538h55m35.111272949s) [noise=true , address=145.79.227.35:4512 , piece_id=3E3VCCC45XVT5FHFDKXSTGGAGYBEML3BKU6JJTT33LCD2YMOMBZQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.240449147 (538h55m35.111532182s) [noise=true , address=45.91.116.8:4064 , piece_id=22WJIPX4SM5NTIAW5ZRM2UKX32CTDBFCXPCCHRRWRCUCOF5CRE5Q, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.240639640 (538h55m35.111722675s) [noise=true , address=145.79.227.32:4853 , piece_id=MBNCFDBQ4SZZXV4F7J5TRMOUYUQ5IPKQCGYGGM5SOIFJ6ZYCAUYA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.241137798 (538h55m35.112220833s) [noise=true , address=77.90.31.16:4751 , piece_id=N3Z2LLMCIA2F42VXZA7IDRU4XEKGOICH6TRVERODU76MFFE4QSHA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.241383797 (538h55m35.112466842s) [noise=true , address=77.90.31.18:4836 , piece_id=BSVEFI52EKN44CMINFUSZBNOH53IW5TXLVTF5IH6ZUTNII6RRLVA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:36.592880816 (538h55m36.463963871s) [upload=472631]: Next returned nil segment Dec 28 02:24:36.592901625 (538h55m36.46398467s) [upload=472631]: Waiting for error group managing segments... Dec 28 02:24:36.946933741 (538h55m36.818016786s) [noise=true , address=64.49.13.139:4480 , piece_id=JAJY3F7IDFFMFKCK2QJV6W74RIESLRKBLXMN27Q3C45JWEPLBZLQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.128189692 (538h55m36.999272737s) [noise=true , address=145.79.227.35:4512 , piece_id=3E3VCCC45XVT5FHFDKXSTGGAGYBEML3BKU6JJTT33LCD2YMOMBZQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.160760084 (538h55m37.031843119s) [noise=true , address=145.79.227.34:4932 , piece_id=UPXSCQ45QDGPKNJXRFJ33EAA3QENVHRVQARU5TFJTEWICZNR253Q, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.288838207 (538h55m37.159921252s) [noise=true , address=77.90.31.11:4572 , piece_id=5MMQ44MXRT4XEL4X6WO54WRG2XYMX3AN3FNQ2H2XIUTKBXI3EEPA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.319996310 (538h55m37.191079345s) [noise=true , address=145.79.227.22:4512 , piece_id=D46YLTIKGHG36DQ4YBKBSUDV4GGB4WVDWNOJS27IYYLN6AYZ5QFQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.320513273 (538h55m37.191596319s) [noise=true , address=77.90.31.18:4836 , piece_id=BSVEFI52EKN44CMINFUSZBNOH53IW5TXLVTF5IH6ZUTNII6RRLVA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.336522926 (538h55m37.207605971s) [noise=true , address=145.79.227.32:4853 , piece_id=MBNCFDBQ4SZZXV4F7J5TRMOUYUQ5IPKQCGYGGM5SOIFJ6ZYCAUYA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.384868031 (538h55m37.255951066s) [noise=true , address=77.90.31.16:4751 , piece_id=N3Z2LLMCIA2F42VXZA7IDRU4XEKGOICH6TRVERODU76MFFE4QSHA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.453563554 (538h55m37.324646599s) [noise=true , address=45.91.116.8:4064 , piece_id=22WJIPX4SM5NTIAW5ZRM2UKX32CTDBFCXPCCHRRWRCUCOF5CRE5Q, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.479454839 (538h55m37.350537884s) [noise=true , address=45.91.116.12:4257 , piece_id=ZI67LIAHZW4G357GWKDPTB5SMXYGNXZTTTDMNAETDS2B6SWVDAEQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.659710174 (538h55m37.530793219s) [noise=true , address=77.90.31.13:4635 , piece_id=IIOY5X2W3XW4C2OFEBRCABMYFBI24RRTYNUSKN6KJU7CTUW4KONQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.715039128 (538h55m37.586122163s) [noise=true , address=77.90.31.16:4760 , piece_id=XRI5EJFN7YYUG4QKLDGEFT6YAUHCUTTSXF735UDR6LZZBGMAZDJQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.977473406 (538h55m37.848556441s) [noise=true , address=77.90.31.19:4874 , piece_id=RHZUXUYMJU6RKQGSDZM5WDWIN5SQ7YIRSRATNNKT5GEMU2QH444A, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:38.016302745 (538h55m37.88738578s) [noise=true , address=145.79.227.30:4773 , piece_id=KUGJXZCNMUE7WSK3VLY4AUBSOT4H6IY4XB4NCBGAGCNUG7F7IODQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:38.016326982 (538h55m37.887410027s) [seg_pos={0 0}, upload=472631]: Segment reached optimal threshold of 14 pieces. Dec 28 02:24:38.016585353 (538h55m37.887668398s) [noise=true , address=145.79.227.30:4766 , piece_id=67M2DMPAVWIQKQSWUAG73D4E4NRRSG4XEVPOUVIJPEQOUV7A6VVA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1k6zdamuKPkBdzyNZNWx4FCmaTkv33PchcwvTRR8bgcETBs4K7): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016594450 (538h55m37.887677495s) [noise=true , address=145.79.227.30:4766 , piece_id=67M2DMPAVWIQKQSWUAG73D4E4NRRSG4XEVPOUVIJPEQOUV7A6VVA, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016627093 (538h55m37.887710148s) [noise=true , address=45.91.116.9:4281 , piece_id=JZLDW777CBX2BP7MLOQV4NFP3H4XJVCNQXTHNSUWPVDOJLPZK36A, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gxf7DpJZthyR6f1PExX56tJavjhPw9Gxb4h8mtmeYKW3t3tS1): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016635258 (538h55m37.887718293s) [noise=true , address=45.91.116.9:4281 , piece_id=JZLDW777CBX2BP7MLOQV4NFP3H4XJVCNQXTHNSUWPVDOJLPZK36A, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016871298 (538h55m37.887954363s) [noise=true , address=145.79.227.44:4222 , piece_id=CZ2CS62CTHQFAIE3S3JO2N3VJ4XIR5YJ2LMRVLLX7I4VM2VZVQVA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZKW35KvrsoYRefbYeUFZo8bh4XZYycNKGEvTFjzibezjXgj3e): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016880124 (538h55m37.887963169s) [noise=true , address=145.79.227.44:4222 , piece_id=CZ2CS62CTHQFAIE3S3JO2N3VJ4XIR5YJ2LMRVLLX7I4VM2VZVQVA, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016934839 (538h55m37.888017884s) [noise=true , address=145.79.227.40:4062 , piece_id=PW6QWIPQWVUETLC6DJ624G6IB4P6EM4SYB72NAMQ4V6NUKNLUYLQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016943425 (538h55m37.88802649s) [noise=true , address=145.79.227.40:4062 , piece_id=PW6QWIPQWVUETLC6DJ624G6IB4P6EM4SYB72NAMQ4V6NUKNLUYLQ, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016968252 (538h55m37.888051297s) [seg_pos={0 0}, upload=472631]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 02:24:38.017067822 (538h55m37.888150867s) [upload=472631]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 02:24:38.017074244 (538h55m37.888157289s) [upload=472631]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 02:24:38.304306667 (538h55m38.175389712s) [upload=472631]: Done uploading segments... Dec 28 02:24:38.304329641 (538h55m38.175412686s) [upload=472631]: Upload finished. err: <nil> Duration: 29.169879298s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 39 out of 60 is failed (threshold=3) [X...XX..X.XXX.XXX.X.....XXX.XXXXXX.XXXX.XXXX..X..XXXXX.XXXXX] duration: 30.002278063s trace: https://jaeger.dev.storj.io/trace/559443fb6d9a906e FAIL (2025-12-28 02:49:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: KODMFMERXQFX4574LCI5EOQBMFPRGZLN57S4SO6T4ABAA3YL2PKQ) context deadline exceeded; piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: KOKYWY2WYAIJK3J2Z2F5Z5DSK4M677FSRYJOOATZPV7TL5GCDCEQ) context deadline exceeded ======LOGS====== Dec 28 02:49:27.699363514 (539h20m27.570446549s) [upload=472986]: Starting upload Dec 28 02:49:27.699448206 (539h20m27.570531271s) [upload=472986]: Done starting upload Dec 28 02:49:27.699649088 (539h20m27.570732133s) [upload=472986]: Uploading segments... Dec 28 02:49:27.699891710 (539h20m27.570974745s) [upload=472986]: Got next segment. Inline: false Dec 28 02:49:27.699991781 (539h20m27.571074816s) [upload=472986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 02:49:27.699998784 (539h20m27.571081829s) [upload=472986]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 02:49:27.944586891 (539h20m27.815669926s) [seg_pos={0 0}, upload=472986]: Begin upload segment... Dec 28 02:49:27.944645122 (539h20m27.815728157s) [seg_pos={0 0}, upload=472986]: Done begin upload segment. Dec 28 02:49:27.944686090 (539h20m27.815769135s) [noise=true , address=145.79.227.29:4943 , piece_id=7JW6PC6HVCVDVNNY6HVWZU6L6TJPUT4TUGROZ3KK2DU5ETTAVIOA, seg_pos={0 0}, upload=472986]: Uploading piece... Dec 28 02:49:27.944717740 (539h20m27.815800785s) [noise=true , address=145.79.227.5:4221 , piece_id=6IFWZXMZDBSKN2AFNNCJO5X3EGN3ZYJP6LIHE77O67TS2NWKZ2UQ, seg_pos={0 0}, upload=472986]: Uploading piece... Dec 28 02:49:27.945603146 (539h20m27.816686181s) [noise=true , address=45.91.116.14:4192 , piece_id=2EWC2MIT7NODXBAYGS6QE2NAJXXXEEYCWJLLVBFKMRY24IPNZXYA, seg_pos={0 0}, upload=472986]: Uploading piece... Dec 28 02:49:27.945984221 (539h20m27.817067266s) [noise=true , address=45.91.116.12:4225 , piece_id=BMDQBLSY5ZP4OJK6MK5F5Q2NNIZZVOJRXQAFJUJNSE37GSZJIYXQ, seg_pos={0 0}, upload=472986]: Uploading piece... Dec 28 02:49:27.946169074 (539h20m27.817252119s) [noise=true , address=145.79.227.14:4840 , piece_id=6KZ44L2C63W3AH7MIO4H7BUT5Z6V2LRMRLBP7NMPUOAHQDHSA7BQ, seg_pos={0 0}, upload=472986]: Uploading piece... Dec 28 02:49:27.946486448 (539h20m27.817569483s) [noise=true , address=145.79.227.3:4150 , piece_id=XFDSUHM6ELHIKO5TME25INIUNSZ4JMU4GAL7A5MPAHN7X3QOZMMQ, seg_pos={0 0}, upload=472986]: Uploading piece... Dec 28 02:49:27.946815965 (539h20m27.81789901s) [noise=true , address=77.90.31.15:4700 , piece_id=KODMFMERXQFX4574LCI5EOQBMFPRGZLN57S4SO6T4ABAA3YL2PKQ, seg_pos={0 0}, upload=472986]: Uploading piece... Dec 28 02:49:27.947085168 (539h20m27.818168203s) [noise=true , address=145.79.227.39:4015 , piece_id=UWIZI55OOQNSIP3U3E3VA2XESC47ER5K3BEXXC7NEZDX3PCXXP7A, seg_pos={0 0}, upload=472986]: Uploading piece... Dec 28 02:49:27.950278448 (539h20m27.821361483s) [noise=true , address=145.79.227.13:4804 , piece_id=STOCZ4FLBHEZJGERXQWDHKMC7ZRGGBAKH7JLGPLEVZSV6DA6R7CQ, seg_pos={0 0}, upload=472986]: Uploading piece... Dec 28 02:49:27.953156508 (539h20m27.824239544s) [noise=true , address=145.79.227.4:4185 , piece_id=P6VIUKJXN7A7BNZT6LUNF6XBCO42O6QAYIPARRDDY54GJC7Z547A, seg_pos={0 0}, upload=472986]: Uploading piece... Dec 28 02:49:27.955215419 (539h20m27.826298454s) [noise=true , address=77.90.31.11:4553 , piece_id=2UDURMK26UUVBZ3W5GIETBX3CAMBNY2ZSLO6EOFGJTBZC23NRKOQ, seg_pos={0 0}, upload=472986]: Uploading piece... Dec 28 02:49:27.956674066 (539h20m27.827757112s) [noise=true , address=45.91.116.7:4010 , piece_id=KOKYWY2WYAIJK3J2Z2F5Z5DSK4M677FSRYJOOATZPV7TL5GCDCEQ, seg_pos={0 0}, upload=472986]: Uploading piece... Dec 28 02:49:27.958107766 (539h20m27.829190801s) [noise=true , address=145.79.227.15:4873 , piece_id=KAV3MRJYVFHTJOXXLPTS4OFBMHJVR3V7OBVT4EWSO6LEKLOZJSIA, seg_pos={0 0}, upload=472986]: Uploading piece... Dec 28 02:49:27.959492523 (539h20m27.830575558s) [noise=true , address=145.79.227.52:4447 , piece_id=4SIIYM6TJ2PSYGPRZVEZRF64BUCZNT7F5K7UU3KKN53R6VNN5AQQ, seg_pos={0 0}, upload=472986]: Uploading piece... Dec 28 02:49:27.960070514 (539h20m27.831153559s) [noise=true , address=145.79.227.7:4293 , piece_id=MLSR3NRKGI4EP7DQVMF6ILZVJEJUKVQVDAG37MEYXYTE4LL4Z3BQ, seg_pos={0 0}, upload=472986]: Uploading piece... Dec 28 02:49:27.961311457 (539h20m27.832394492s) [noise=true , address=145.79.227.46:4603 , piece_id=NOKILYQO4MWHNLQ6OKVVY6NEAKDYS7NFID2TPI6QR3C5XE2BOKOQ, seg_pos={0 0}, upload=472986]: Uploading piece... Dec 28 02:49:29.697640314 (539h20m29.568723349s) [upload=472986]: Next returned nil segment Dec 28 02:49:29.697650323 (539h20m29.568733368s) [upload=472986]: Waiting for error group managing segments... Dec 28 02:49:30.108165542 (539h20m29.979248577s) [noise=true , address=145.79.227.13:4804 , piece_id=STOCZ4FLBHEZJGERXQWDHKMC7ZRGGBAKH7JLGPLEVZSV6DA6R7CQ, seg_pos={0 0}, upload=472986]: Done uploading piece. err: <nil> Dec 28 02:49:30.119299313 (539h20m29.990382348s) [noise=true , address=145.79.227.5:4221 , piece_id=6IFWZXMZDBSKN2AFNNCJO5X3EGN3ZYJP6LIHE77O67TS2NWKZ2UQ, seg_pos={0 0}, upload=472986]: Done uploading piece. err: <nil> Dec 28 02:49:30.122878589 (539h20m29.993961624s) [noise=true , address=145.79.227.39:4015 , piece_id=UWIZI55OOQNSIP3U3E3VA2XESC47ER5K3BEXXC7NEZDX3PCXXP7A, seg_pos={0 0}, upload=472986]: Done uploading piece. err: <nil> Dec 28 02:49:30.130910633 (539h20m30.001993679s) [noise=true , address=145.79.227.52:4447 , piece_id=4SIIYM6TJ2PSYGPRZVEZRF64BUCZNT7F5K7UU3KKN53R6VNN5AQQ, seg_pos={0 0}, upload=472986]: Done uploading piece. err: <nil> Dec 28 02:49:30.155421526 (539h20m30.026504561s) [noise=true , address=145.79.227.15:4873 , piece_id=KAV3MRJYVFHTJOXXLPTS4OFBMHJVR3V7OBVT4EWSO6LEKLOZJSIA, seg_pos={0 0}, upload=472986]: Done uploading piece. err: <nil> Dec 28 02:49:30.165455983 (539h20m30.036539028s) [noise=true , address=145.79.227.7:4293 , piece_id=MLSR3NRKGI4EP7DQVMF6ILZVJEJUKVQVDAG37MEYXYTE4LL4Z3BQ, seg_pos={0 0}, upload=472986]: Done uploading piece. err: <nil> Dec 28 02:49:30.168960516 (539h20m30.040043561s) [noise=true , address=145.79.227.4:4185 , piece_id=P6VIUKJXN7A7BNZT6LUNF6XBCO42O6QAYIPARRDDY54GJC7Z547A, seg_pos={0 0}, upload=472986]: Done uploading piece. err: <nil> Dec 28 02:49:30.176477029 (539h20m30.047560074s) [noise=true , address=145.79.227.46:4603 , piece_id=NOKILYQO4MWHNLQ6OKVVY6NEAKDYS7NFID2TPI6QR3C5XE2BOKOQ, seg_pos={0 0}, upload=472986]: Done uploading piece. err: <nil> Dec 28 02:49:30.195268541 (539h20m30.066351576s) [noise=true , address=145.79.227.29:4943 , piece_id=7JW6PC6HVCVDVNNY6HVWZU6L6TJPUT4TUGROZ3KK2DU5ETTAVIOA, seg_pos={0 0}, upload=472986]: Done uploading piece. err: <nil> Dec 28 02:49:30.231728377 (539h20m30.102811422s) [noise=true , address=77.90.31.15:4700 , piece_id=KODMFMERXQFX4574LCI5EOQBMFPRGZLN57S4SO6T4ABAA3YL2PKQ, seg_pos={0 0}, upload=472986]: Done uploading piece. err: <nil> Dec 28 02:49:30.305897807 (539h20m30.176980853s) [noise=true , address=45.91.116.14:4192 , piece_id=2EWC2MIT7NODXBAYGS6QE2NAJXXXEEYCWJLLVBFKMRY24IPNZXYA, seg_pos={0 0}, upload=472986]: Done uploading piece. err: <nil> Dec 28 02:49:30.484309012 (539h20m30.355392057s) [noise=true , address=145.79.227.3:4150 , piece_id=XFDSUHM6ELHIKO5TME25INIUNSZ4JMU4GAL7A5MPAHN7X3QOZMMQ, seg_pos={0 0}, upload=472986]: Done uploading piece. err: <nil> Dec 28 02:49:30.589856012 (539h20m30.460939057s) [noise=true , address=45.91.116.7:4010 , piece_id=KOKYWY2WYAIJK3J2Z2F5Z5DSK4M677FSRYJOOATZPV7TL5GCDCEQ, seg_pos={0 0}, upload=472986]: Done uploading piece. err: <nil> Dec 28 02:49:30.589881359 (539h20m30.460964394s) [seg_pos={0 0}, upload=472986]: Segment reached optimal threshold of 13 pieces. Dec 28 02:49:30.590136375 (539h20m30.461219421s) [noise=true , address=77.90.31.11:4553 , piece_id=2UDURMK26UUVBZ3W5GIETBX3CAMBNY2ZSLO6EOFGJTBZC23NRKOQ, seg_pos={0 0}, upload=472986]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: expected piece hash; context canceled; EOF Dec 28 02:49:30.590145222 (539h20m30.461228257s) [noise=true , address=77.90.31.11:4553 , piece_id=2UDURMK26UUVBZ3W5GIETBX3CAMBNY2ZSLO6EOFGJTBZC23NRKOQ, seg_pos={0 0}, upload=472986]: Long tail cancellation due to optimal threshold Dec 28 02:49:30.590404505 (539h20m30.46148755s) [noise=true , address=145.79.227.14:4840 , piece_id=6KZ44L2C63W3AH7MIO4H7BUT5Z6V2LRMRLBP7NMPUOAHQDHSA7BQ, seg_pos={0 0}, upload=472986]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: expected piece hash; context canceled; EOF Dec 28 02:49:30.590425496 (539h20m30.461508531s) [noise=true , address=145.79.227.14:4840 , piece_id=6KZ44L2C63W3AH7MIO4H7BUT5Z6V2LRMRLBP7NMPUOAHQDHSA7BQ, seg_pos={0 0}, upload=472986]: Long tail cancellation due to optimal threshold Dec 28 02:49:30.591596154 (539h20m30.462679199s) [noise=true , address=45.91.116.12:4225 , piece_id=BMDQBLSY5ZP4OJK6MK5F5Q2NNIZZVOJRXQAFJUJNSE37GSZJIYXQ, seg_pos={0 0}, upload=472986]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Dec 28 02:49:30.591612586 (539h20m30.462695651s) [noise=true , address=45.91.116.12:4225 , piece_id=BMDQBLSY5ZP4OJK6MK5F5Q2NNIZZVOJRXQAFJUJNSE37GSZJIYXQ, seg_pos={0 0}, upload=472986]: Long tail cancellation due to optimal threshold Dec 28 02:49:30.591650668 (539h20m30.462733714s) [seg_pos={0 0}, upload=472986]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 02:49:30.591745279 (539h20m30.462828324s) [upload=472986]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 02:49:30.591751561 (539h20m30.462834606s) [upload=472986]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 02:49:30.890909437 (539h20m30.761992482s) [upload=472986]: Done uploading segments... Dec 28 02:49:30.890934584 (539h20m30.762017619s) [upload=472986]: Upload finished. err: <nil> Duration: 30.002291468s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [.X.........................X................X....X..........] duration: 3.423422081s trace: https://jaeger.dev.storj.io/trace/5efee46a94502508 FAIL (2025-12-28 02:49:36 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.423442881s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 38 out of 60 is failed (threshold=3) [.X...XX..X.XXX.XXX.X.....XXX.XXXXXX.XXXX.XXXX..X..XXXXX.XXXX] duration: 30.001906832s trace: https://jaeger.dev.storj.io/trace/4a2258b14501e83b FAIL (2025-12-28 02:49:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: NCVEPTBMV7SQJ4533GALYLFPWFBJR3D44YIJOTKY5BFCWJGIEHFQ) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: 5JIL2SG2BG236VLDFNL6QKH3JECUCS3NNPTW7E72OO3VVGHFQIUA) context deadline exceeded ======LOGS====== 48:57.436957510 (539h19m57.308040545s) [upload=472979]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 02:48:57.681203315 (539h19m57.55228636s) [seg_pos={0 0}, upload=472979]: Begin upload segment... Dec 28 02:48:57.681255002 (539h19m57.552338048s) [seg_pos={0 0}, upload=472979]: Done begin upload segment. Dec 28 02:48:57.681282305 (539h19m57.55236534s) [noise=true , address=145.79.227.18:4006 , piece_id=635KKHHHI6TVRMCQQCQRIV5OMS5H5F3ZIYUFJW5F2FCPFMN3FJKA, seg_pos={0 0}, upload=472979]: Uploading piece... Dec 28 02:48:57.681292123 (539h19m57.552375158s) [noise=true , address=145.79.227.6:4258 , piece_id=2RLEOTMHJKRBOC75UIWXQBM7YUR2FRYCB47XCPHFJQR6AFNMQNBA, seg_pos={0 0}, upload=472979]: Uploading piece... Dec 28 02:48:57.681858712 (539h19m57.552941747s) [noise=true , address=145.79.227.11:4732 , piece_id=ELUX5YYLUIAE4UEK4NGNWDXY56P2JTECJEVNPX5T7BI2UCBYPV4A, seg_pos={0 0}, upload=472979]: Uploading piece... Dec 28 02:48:57.682151650 (539h19m57.553234695s) [noise=true , address=45.91.116.6:4973 , piece_id=5JIL2SG2BG236VLDFNL6QKH3JECUCS3NNPTW7E72OO3VVGHFQIUA, seg_pos={0 0}, upload=472979]: Uploading piece... Dec 28 02:48:57.682476769 (539h19m57.553559814s) [noise=true , address=145.79.227.36:4518 , piece_id=LV2H36EUVXLIAFL6ECB5USXCNBJSWIWPRZEFALUKHXHDZLG6O67A, seg_pos={0 0}, upload=472979]: Uploading piece... Dec 28 02:48:57.682956192 (539h19m57.554039227s) [noise=true , address=45.91.116.8:4048 , piece_id=MTZIROSGVWOM63NGEL6SFLR7CU4N3Y7XGSGUPLVJLN3LU5CNS2MQ, seg_pos={0 0}, upload=472979]: Uploading piece... Dec 28 02:48:57.683328431 (539h19m57.554411476s) [noise=true , address=145.79.227.41:4086 , piece_id=CYKODHTD7ERGMOBTKXDNTREMMBTVMH6XRG2TJTC3QJMCTOCESCZA, seg_pos={0 0}, upload=472979]: Uploading piece... Dec 28 02:48:57.685191278 (539h19m57.556274323s) [noise=true , address=45.91.116.11:4120 , piece_id=E72VV6T5CPTC5D73BV2UAEM3NOBOTWB6KHXYFPRURIMD63Z5DZDQ, seg_pos={0 0}, upload=472979]: Uploading piece... Dec 28 02:48:57.685397631 (539h19m57.556480676s) [noise=true , address=45.91.116.13:4153 , piece_id=JOYRWIAUOYAB3BS63UQ74KE3JGC2P24CCZQ47NVQDO2JNYJ4R3CA, seg_pos={0 0}, upload=472979]: Uploading piece... Dec 28 02:48:57.687344829 (539h19m57.558427864s) [noise=true , address=145.79.227.22:4511 , piece_id=EHT5HLVQEFG2VDSRVRSGS4NRNRPQ2ZMLAXQIDRB77I6CQSATOACA, seg_pos={0 0}, upload=472979]: Uploading piece... Dec 28 02:48:57.688916802 (539h19m57.559999847s) [noise=true , address=145.79.227.51:4411 , piece_id=AHZXX4JAQCADSONTQLQGRZPBBE6G5ZPUMPCIWM2H7FINJQLAUI6Q, seg_pos={0 0}, upload=472979]: Uploading piece... Dec 28 02:48:57.690232347 (539h19m57.561315382s) [noise=true , address=45.91.116.7:4012 , piece_id=NCVEPTBMV7SQJ4533GALYLFPWFBJR3D44YIJOTKY5BFCWJGIEHFQ, seg_pos={0 0}, upload=472979]: Uploading piece... Dec 28 02:48:57.691601894 (539h19m57.56268494s) [noise=true , address=145.79.227.19:4078 , piece_id=3P7W7J7HXMPQB5ISWQLUXO4BLUDKURI6D7LQOLRKU23ZVAXJFJ3A, seg_pos={0 0}, upload=472979]: Uploading piece... Dec 28 02:48:57.692416446 (539h19m57.563499491s) [noise=true , address=45.91.116.4:4937 , piece_id=XYCXB5BHXMY4SKMYRONTP4MUBNSUORWTHF4WLGRWUGOHHAWHWFUA, seg_pos={0 0}, upload=472979]: Uploading piece... Dec 28 02:48:57.693462077 (539h19m57.564545112s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHKDQOAPBMSZPZT56ZLYITQPFQXMETWNWOF35BWHRYVMOMCO2AQ, seg_pos={0 0}, upload=472979]: Uploading piece... Dec 28 02:48:57.694631513 (539h19m57.565714548s) [noise=true , address=145.79.227.13:4803 , piece_id=TR6LNTYKBMWW3TCK47EJXMIEKQPSVQ77O6BXBZ667SVJQPIHUEXA, seg_pos={0 0}, upload=472979]: Uploading piece... Dec 28 02:48:59.405041057 (539h19m59.276124132s) [upload=472979]: Next returned nil segment Dec 28 02:48:59.405061286 (539h19m59.276144331s) [upload=472979]: Waiting for error group managing segments... Dec 28 02:48:59.690065591 (539h19m59.561148626s) [noise=true , address=145.79.227.41:4086 , piece_id=CYKODHTD7ERGMOBTKXDNTREMMBTVMH6XRG2TJTC3QJMCTOCESCZA, seg_pos={0 0}, upload=472979]: Done uploading piece. err: <nil> Dec 28 02:48:59.691698160 (539h19m59.562781195s) [noise=true , address=145.79.227.36:4518 , piece_id=LV2H36EUVXLIAFL6ECB5USXCNBJSWIWPRZEFALUKHXHDZLG6O67A, seg_pos={0 0}, upload=472979]: Done uploading piece. err: <nil> Dec 28 02:48:59.709932360 (539h19m59.581015395s) [noise=true , address=145.79.227.11:4732 , piece_id=ELUX5YYLUIAE4UEK4NGNWDXY56P2JTECJEVNPX5T7BI2UCBYPV4A, seg_pos={0 0}, upload=472979]: Done uploading piece. err: <nil> Dec 28 02:48:59.713570037 (539h19m59.584653072s) [noise=true , address=145.79.227.6:4258 , piece_id=2RLEOTMHJKRBOC75UIWXQBM7YUR2FRYCB47XCPHFJQR6AFNMQNBA, seg_pos={0 0}, upload=472979]: Done uploading piece. err: <nil> Dec 28 02:48:59.725022034 (539h19m59.596105069s) [noise=true , address=145.79.227.51:4411 , piece_id=AHZXX4JAQCADSONTQLQGRZPBBE6G5ZPUMPCIWM2H7FINJQLAUI6Q, seg_pos={0 0}, upload=472979]: Done uploading piece. err: <nil> Dec 28 02:48:59.732852294 (539h19m59.603935339s) [noise=true , address=145.79.227.37:4566 , piece_id=TPHKDQOAPBMSZPZT56ZLYITQPFQXMETWNWOF35BWHRYVMOMCO2AQ, seg_pos={0 0}, upload=472979]: Done uploading piece. err: <nil> Dec 28 02:48:59.899030916 (539h19m59.770113951s) [noise=true , address=45.91.116.11:4120 , piece_id=E72VV6T5CPTC5D73BV2UAEM3NOBOTWB6KHXYFPRURIMD63Z5DZDQ, seg_pos={0 0}, upload=472979]: Done uploading piece. err: <nil> Dec 28 02:48:59.926158011 (539h19m59.797241056s) [noise=true , address=45.91.116.7:4012 , piece_id=NCVEPTBMV7SQJ4533GALYLFPWFBJR3D44YIJOTKY5BFCWJGIEHFQ, seg_pos={0 0}, upload=472979]: Done uploading piece. err: <nil> Dec 28 02:48:59.964464703 (539h19m59.835547748s) [noise=true , address=45.91.116.6:4973 , piece_id=5JIL2SG2BG236VLDFNL6QKH3JECUCS3NNPTW7E72OO3VVGHFQIUA, seg_pos={0 0}, upload=472979]: Done uploading piece. err: <nil> Dec 28 02:49:00.360388136 (539h20m0.231471181s) [noise=true , address=145.79.227.22:4511 , piece_id=EHT5HLVQEFG2VDSRVRSGS4NRNRPQ2ZMLAXQIDRB77I6CQSATOACA, seg_pos={0 0}, upload=472979]: Done uploading piece. err: <nil> Dec 28 02:49:00.681906649 (539h20m0.552989684s) [noise=true , address=45.91.116.13:4153 , piece_id=JOYRWIAUOYAB3BS63UQ74KE3JGC2P24CCZQ47NVQDO2JNYJ4R3CA, seg_pos={0 0}, upload=472979]: Done uploading piece. err: <nil> Dec 28 02:49:00.692877469 (539h20m0.563960524s) [noise=true , address=45.91.116.8:4048 , piece_id=MTZIROSGVWOM63NGEL6SFLR7CU4N3Y7XGSGUPLVJLN3LU5CNS2MQ, seg_pos={0 0}, upload=472979]: Done uploading piece. err: <nil> Dec 28 02:49:00.945886330 (539h20m0.816969375s) [noise=true , address=145.79.227.13:4803 , piece_id=TR6LNTYKBMWW3TCK47EJXMIEKQPSVQ77O6BXBZ667SVJQPIHUEXA, seg_pos={0 0}, upload=472979]: Done uploading piece. err: <nil> Dec 28 02:49:00.945914153 (539h20m0.816997188s) [seg_pos={0 0}, upload=472979]: Segment reached optimal threshold of 13 pieces. Dec 28 02:49:00.946158478 (539h20m0.817241513s) [noise=true , address=45.91.116.4:4937 , piece_id=XYCXB5BHXMY4SKMYRONTP4MUBNSUORWTHF4WLGRWUGOHHAWHWFUA, seg_pos={0 0}, upload=472979]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:50346->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:50346->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50346->45.91.116.4:4937: use of closed network connection Dec 28 02:49:00.946165481 (539h20m0.817248516s) [noise=true , address=45.91.116.4:4937 , piece_id=XYCXB5BHXMY4SKMYRONTP4MUBNSUORWTHF4WLGRWUGOHHAWHWFUA, seg_pos={0 0}, upload=472979]: Long tail cancellation due to optimal threshold Dec 28 02:49:00.946305909 (539h20m0.817388954s) [noise=true , address=145.79.227.19:4078 , piece_id=3P7W7J7HXMPQB5ISWQLUXO4BLUDKURI6D7LQOLRKU23ZVAXJFJ3A, seg_pos={0 0}, upload=472979]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: write tcp 188.166.225.170:54894->145.79.227.19:4078: use of closed network connection; write tcp 188.166.225.170:54894->145.79.227.19:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54894->145.79.227.19:4078: use of closed network connection Dec 28 02:49:00.946329294 (539h20m0.817412329s) [noise=true , address=145.79.227.19:4078 , piece_id=3P7W7J7HXMPQB5ISWQLUXO4BLUDKURI6D7LQOLRKU23ZVAXJFJ3A, seg_pos={0 0}, upload=472979]: Long tail cancellation due to optimal threshold Dec 28 02:49:00.946668490 (539h20m0.817751535s) [noise=true , address=145.79.227.18:4006 , piece_id=635KKHHHI6TVRMCQQCQRIV5OMS5H5F3ZIYUFJW5F2FCPFMN3FJKA, seg_pos={0 0}, upload=472979]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc): protocol: write tcp 188.166.225.170:50414->145.79.227.18:4006: use of closed network connection; write tcp 188.166.225.170:50414->145.79.227.18:4006: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50414->145.79.227.18:4006: use of closed network connection Dec 28 02:49:00.946681374 (539h20m0.817764429s) [noise=true , address=145.79.227.18:4006 , piece_id=635KKHHHI6TVRMCQQCQRIV5OMS5H5F3ZIYUFJW5F2FCPFMN3FJKA, seg_pos={0 0}, upload=472979]: Long tail cancellation due to optimal threshold Dec 28 02:49:00.946710540 (539h20m0.817793585s) [seg_pos={0 0}, upload=472979]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 02:49:00.947427816 (539h20m0.818510861s) [upload=472979]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 02:49:00.947436642 (539h20m0.818519677s) [upload=472979]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 02:49:01.242840671 (539h20m1.113923706s) [upload=472979]: Done uploading segments... Dec 28 02:49:01.242860569 (539h20m1.113943604s) [upload=472979]: Upload finished. err: <nil> Duration:* 30.001927952s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....X...................X.....X............................] duration: 46.798305723s trace: https://jaeger.dev.storj.io/trace/1822be52d200e5b5 FAIL (2025-12-28 02:49:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1ZAfYZ4wgpZVvatj6wtHona3hTaPcLxAmXC1DT2uKQyo8nLQQ7, Piece ID: 5MMQ44MXRT4XEL4X6WO54WRG2XYMX3AN3FNQ2H2XIUTKBXI3EEPA) context deadline exceeded; piecestore: (Node ID: 124guAM82oibYZsMho67Em6Ke6yEZXwT6LPMLLSZZBLqwoFGUkn, Piece ID: 22WJIPX4SM5NTIAW5ZRM2UKX32CTDBFCXPCCHRRWRCUCOF5CRE5Q) context deadline exceeded; piecestore: (Node ID: 1e6p1TD5CqUyWdZjkrxh2kRqMuyDiMUr9eimfeayDAKwhtBrZ8, Piece ID: XRI5EJFN7YYUG4QKLDGEFT6YAUHCUTTSXF735UDR6LZZBGMAZDJQ) context deadline exceeded ======LOGS====== 5.234848204 (538h55m35.105931239s) [noise=true , address=77.90.31.11:4572 , piece_id=5MMQ44MXRT4XEL4X6WO54WRG2XYMX3AN3FNQ2H2XIUTKBXI3EEPA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.235412057 (538h55m35.106495102s) [noise=true , address=145.79.227.22:4512 , piece_id=D46YLTIKGHG36DQ4YBKBSUDV4GGB4WVDWNOJS27IYYLN6AYZ5QFQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.235820946 (538h55m35.106903981s) [noise=true , address=77.90.31.19:4874 , piece_id=RHZUXUYMJU6RKQGSDZM5WDWIN5SQ7YIRSRATNNKT5GEMU2QH444A, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.236192192 (538h55m35.107275237s) [noise=true , address=77.90.31.16:4760 , piece_id=XRI5EJFN7YYUG4QKLDGEFT6YAUHCUTTSXF735UDR6LZZBGMAZDJQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.236564772 (538h55m35.107647817s) [noise=true , address=145.79.227.40:4062 , piece_id=PW6QWIPQWVUETLC6DJ624G6IB4P6EM4SYB72NAMQ4V6NUKNLUYLQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.236929766 (538h55m35.108012812s) [noise=true , address=64.49.13.139:4480 , piece_id=JAJY3F7IDFFMFKCK2QJV6W74RIESLRKBLXMN27Q3C45JWEPLBZLQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.237916846 (538h55m35.108999881s) [noise=true , address=145.79.227.34:4932 , piece_id=UPXSCQ45QDGPKNJXRFJ33EAA3QENVHRVQARU5TFJTEWICZNR253Q, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.238852948 (538h55m35.109935983s) [noise=true , address=45.91.116.12:4257 , piece_id=ZI67LIAHZW4G357GWKDPTB5SMXYGNXZTTTDMNAETDS2B6SWVDAEQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239152910 (538h55m35.110235945s) [noise=true , address=145.79.227.44:4222 , piece_id=CZ2CS62CTHQFAIE3S3JO2N3VJ4XIR5YJ2LMRVLLX7I4VM2VZVQVA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239409969 (538h55m35.110493004s) [noise=true , address=145.79.227.30:4773 , piece_id=KUGJXZCNMUE7WSK3VLY4AUBSOT4H6IY4XB4NCBGAGCNUG7F7IODQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239681136 (538h55m35.110764171s) [noise=true , address=77.90.31.13:4635 , piece_id=IIOY5X2W3XW4C2OFEBRCABMYFBI24RRTYNUSKN6KJU7CTUW4KONQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239954676 (538h55m35.111037711s) [noise=true , address=45.91.116.9:4281 , piece_id=JZLDW777CBX2BP7MLOQV4NFP3H4XJVCNQXTHNSUWPVDOJLPZK36A, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.240189914 (538h55m35.111272949s) [noise=true , address=145.79.227.35:4512 , piece_id=3E3VCCC45XVT5FHFDKXSTGGAGYBEML3BKU6JJTT33LCD2YMOMBZQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.240449147 (538h55m35.111532182s) [noise=true , address=45.91.116.8:4064 , piece_id=22WJIPX4SM5NTIAW5ZRM2UKX32CTDBFCXPCCHRRWRCUCOF5CRE5Q, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.240639640 (538h55m35.111722675s) [noise=true , address=145.79.227.32:4853 , piece_id=MBNCFDBQ4SZZXV4F7J5TRMOUYUQ5IPKQCGYGGM5SOIFJ6ZYCAUYA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.241137798 (538h55m35.112220833s) [noise=true , address=77.90.31.16:4751 , piece_id=N3Z2LLMCIA2F42VXZA7IDRU4XEKGOICH6TRVERODU76MFFE4QSHA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.241383797 (538h55m35.112466842s) [noise=true , address=77.90.31.18:4836 , piece_id=BSVEFI52EKN44CMINFUSZBNOH53IW5TXLVTF5IH6ZUTNII6RRLVA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:36.592880816 (538h55m36.463963871s) [upload=472631]: Next returned nil segment Dec 28 02:24:36.592901625 (538h55m36.46398467s) [upload=472631]: Waiting for error group managing segments... Dec 28 02:24:36.946933741 (538h55m36.818016786s) [noise=true , address=64.49.13.139:4480 , piece_id=JAJY3F7IDFFMFKCK2QJV6W74RIESLRKBLXMN27Q3C45JWEPLBZLQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.128189692 (538h55m36.999272737s) [noise=true , address=145.79.227.35:4512 , piece_id=3E3VCCC45XVT5FHFDKXSTGGAGYBEML3BKU6JJTT33LCD2YMOMBZQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.160760084 (538h55m37.031843119s) [noise=true , address=145.79.227.34:4932 , piece_id=UPXSCQ45QDGPKNJXRFJ33EAA3QENVHRVQARU5TFJTEWICZNR253Q, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.288838207 (538h55m37.159921252s) [noise=true , address=77.90.31.11:4572 , piece_id=5MMQ44MXRT4XEL4X6WO54WRG2XYMX3AN3FNQ2H2XIUTKBXI3EEPA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.319996310 (538h55m37.191079345s) [noise=true , address=145.79.227.22:4512 , piece_id=D46YLTIKGHG36DQ4YBKBSUDV4GGB4WVDWNOJS27IYYLN6AYZ5QFQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.320513273 (538h55m37.191596319s) [noise=true , address=77.90.31.18:4836 , piece_id=BSVEFI52EKN44CMINFUSZBNOH53IW5TXLVTF5IH6ZUTNII6RRLVA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.336522926 (538h55m37.207605971s) [noise=true , address=145.79.227.32:4853 , piece_id=MBNCFDBQ4SZZXV4F7J5TRMOUYUQ5IPKQCGYGGM5SOIFJ6ZYCAUYA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.384868031 (538h55m37.255951066s) [noise=true , address=77.90.31.16:4751 , piece_id=N3Z2LLMCIA2F42VXZA7IDRU4XEKGOICH6TRVERODU76MFFE4QSHA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.453563554 (538h55m37.324646599s) [noise=true , address=45.91.116.8:4064 , piece_id=22WJIPX4SM5NTIAW5ZRM2UKX32CTDBFCXPCCHRRWRCUCOF5CRE5Q, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.479454839 (538h55m37.350537884s) [noise=true , address=45.91.116.12:4257 , piece_id=ZI67LIAHZW4G357GWKDPTB5SMXYGNXZTTTDMNAETDS2B6SWVDAEQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.659710174 (538h55m37.530793219s) [noise=true , address=77.90.31.13:4635 , piece_id=IIOY5X2W3XW4C2OFEBRCABMYFBI24RRTYNUSKN6KJU7CTUW4KONQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.715039128 (538h55m37.586122163s) [noise=true , address=77.90.31.16:4760 , piece_id=XRI5EJFN7YYUG4QKLDGEFT6YAUHCUTTSXF735UDR6LZZBGMAZDJQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.977473406 (538h55m37.848556441s) [noise=true , address=77.90.31.19:4874 , piece_id=RHZUXUYMJU6RKQGSDZM5WDWIN5SQ7YIRSRATNNKT5GEMU2QH444A, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:38.016302745 (538h55m37.88738578s) [noise=true , address=145.79.227.30:4773 , piece_id=KUGJXZCNMUE7WSK3VLY4AUBSOT4H6IY4XB4NCBGAGCNUG7F7IODQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:38.016326982 (538h55m37.887410027s) [seg_pos={0 0}, upload=472631]: Segment reached optimal threshold of 14 pieces. Dec 28 02:24:38.016585353 (538h55m37.887668398s) [noise=true , address=145.79.227.30:4766 , piece_id=67M2DMPAVWIQKQSWUAG73D4E4NRRSG4XEVPOUVIJPEQOUV7A6VVA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1k6zdamuKPkBdzyNZNWx4FCmaTkv33PchcwvTRR8bgcETBs4K7): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016594450 (538h55m37.887677495s) [noise=true , address=145.79.227.30:4766 , piece_id=67M2DMPAVWIQKQSWUAG73D4E4NRRSG4XEVPOUVIJPEQOUV7A6VVA, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016627093 (538h55m37.887710148s) [noise=true , address=45.91.116.9:4281 , piece_id=JZLDW777CBX2BP7MLOQV4NFP3H4XJVCNQXTHNSUWPVDOJLPZK36A, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gxf7DpJZthyR6f1PExX56tJavjhPw9Gxb4h8mtmeYKW3t3tS1): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016635258 (538h55m37.887718293s) [noise=true , address=45.91.116.9:4281 , piece_id=JZLDW777CBX2BP7MLOQV4NFP3H4XJVCNQXTHNSUWPVDOJLPZK36A, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016871298 (538h55m37.887954363s) [noise=true , address=145.79.227.44:4222 , piece_id=CZ2CS62CTHQFAIE3S3JO2N3VJ4XIR5YJ2LMRVLLX7I4VM2VZVQVA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZKW35KvrsoYRefbYeUFZo8bh4XZYycNKGEvTFjzibezjXgj3e): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016880124 (538h55m37.887963169s) [noise=true , address=145.79.227.44:4222 , piece_id=CZ2CS62CTHQFAIE3S3JO2N3VJ4XIR5YJ2LMRVLLX7I4VM2VZVQVA, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016934839 (538h55m37.888017884s) [noise=true , address=145.79.227.40:4062 , piece_id=PW6QWIPQWVUETLC6DJ624G6IB4P6EM4SYB72NAMQ4V6NUKNLUYLQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016943425 (538h55m37.88802649s) [noise=true , address=145.79.227.40:4062 , piece_id=PW6QWIPQWVUETLC6DJ624G6IB4P6EM4SYB72NAMQ4V6NUKNLUYLQ, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016968252 (538h55m37.888051297s) [seg_pos={0 0}, upload=472631]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 02:24:38.017067822 (538h55m37.888150867s) [upload=472631]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 02:24:38.017074244 (538h55m37.888157289s) [upload=472631]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 02:24:38.304306667 (538h55m38.175389712s) [upload=472631]: Done uploading segments... Dec 28 02:24:38.304329641 (538h55m38.175412686s) [upload=472631]: Upload finished. err: <nil> Duration: 46.798334707s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [..X.........................X................X....X.........] duration: 4.316155485s trace: https://jaeger.dev.storj.io/trace/5efee46a94502508 FAIL (2025-12-28 02:48:50 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.316181384s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 37 out of 60 is failed (threshold=3) [..X...XX..X.XXX.XXX.X.....XXX.XXXXXX.XXXX.XXXX..X..XXXXX.XXX] duration: 30.0029915s trace: https://jaeger.dev.storj.io/trace/1b590cf6c5c97fee FAIL (2025-12-28 02:48:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== egin Dec 28 02:48:08.525987126 (539h19m8.397070171s) [upload=472966]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 02:48:08.774819644 (539h19m8.645902679s) [seg_pos={0 0}, upload=472966]: Begin upload segment... Dec 28 02:48:08.774881452 (539h19m8.645964487s) [seg_pos={0 0}, upload=472966]: Done begin upload segment. Dec 28 02:48:08.774928341 (539h19m8.646011386s) [noise=true , address=145.79.227.18:4006 , piece_id=SMFZKQVKTGYTCXG333EME6R27MI3NWYWTPNGXPOYUP6GIEEYAUEA, seg_pos={0 0}, upload=472966]: Uploading piece... Dec 28 02:48:08.774951074 (539h19m8.646034109s) [noise=true , address=145.79.227.52:4447 , piece_id=K4PJNQOSUOKNI6AMNJJE4SUGB5T2U5NNZ5S6WEFLMPY4QF7ZJBFA, seg_pos={0 0}, upload=472966]: Uploading piece... Dec 28 02:48:08.775501853 (539h19m8.646584888s) [noise=true , address=145.79.227.51:4410 , piece_id=GXLX6NGMPLKTNWONWYZTQCBV3WQPJRP3NHA5PMG34KVFTNIFXX3A, seg_pos={0 0}, upload=472966]: Uploading piece... Dec 28 02:48:08.775881265 (539h19m8.6469643s) [noise=true , address=45.91.116.13:4155 , piece_id=KGBTH5PYUSKO4UZFXQVUNQBUJZKATHHHPDHWQFGQH5YFDXQXDI5Q, seg_pos={0 0}, upload=472966]: Uploading piece... Dec 28 02:48:08.776344447 (539h19m8.647427482s) [noise=true , address=77.90.31.11:4553 , piece_id=Y3LLQHC6WQG6JKQB56GRUT3P7E6YDOEZLDDBJPRU3E6VOWOIPGJQ, seg_pos={0 0}, upload=472966]: Uploading piece... Dec 28 02:48:08.776794354 (539h19m8.647877389s) [noise=true , address=77.90.31.21:4878 , piece_id=JAAHWDOLBLJRYNSIWUBPU3GY3XZ33VBBY673SLTWPNFCYCQI45LQ, seg_pos={0 0}, upload=472966]: Uploading piece... Dec 28 02:48:08.777255432 (539h19m8.648338467s) [noise=true , address=145.79.227.37:4567 , piece_id=55X5OICJGOZ5MIMYLGPV5RO4ZXQC7KNDWLXXOD55DSVIPFDVL3YA, seg_pos={0 0}, upload=472966]: Uploading piece... Dec 28 02:48:08.777734294 (539h19m8.648817329s) [noise=true , address=145.79.227.38:4977 , piece_id=UHQW5HGZ57RIRK7YWVUMSSEFU5P37R7MZG2AKHEDBKV6LSK3D6AQ, seg_pos={0 0}, upload=472966]: Uploading piece... Dec 28 02:48:08.778265235 (539h19m8.64934828s) [noise=true , address=145.79.227.3:4150 , piece_id=ZFGYT36TLZXG2L3TTSGYKVCZ544EFRTHUQAW4MNWIOXFOZTMIMPA, seg_pos={0 0}, upload=472966]: Uploading piece... Dec 28 02:48:08.779015413 (539h19m8.650098458s) [noise=true , address=145.79.227.50:4375 , piece_id=SCVWARBRGLK7R3QTHVUK4MVQ4H5QGSZNDANA5GXMZIPTM32N7M4Q, seg_pos={0 0}, upload=472966]: Uploading piece... Dec 28 02:48:08.780756177 (539h19m8.651839212s) [noise=true , address=45.91.116.4:4937 , piece_id=FGP77TID47WOFQ2BSQGYXLIQ4CHGD6VGEFKT34B3DMJY5LNTZAJQ, seg_pos={0 0}, upload=472966]: Uploading piece... Dec 28 02:48:08.782022670 (539h19m8.653105705s) [noise=true , address=145.79.227.42:4122 , piece_id=CYGIT6EP5MNHDK57C3IVYOLKEMOV23ISSHUNUCD2GEM4TQHQOZEQ, seg_pos={0 0}, upload=472966]: Uploading piece... Dec 28 02:48:08.782692405 (539h19m8.65377544s) [noise=true , address=145.79.227.36:4517 , piece_id=YCLHMPUBNGEU5HASWFFXOLTYTH2EJTLKLEWSJLLWHJZOUSBATX7A, seg_pos={0 0}, upload=472966]: Uploading piece... Dec 28 02:48:08.791396028 (539h19m8.662479063s) [noise=true , address=145.79.227.15:4873 , piece_id=O7FJ4SYP5U5MTXYUSO3VSQI46SYZPNTMNBJXG3L2L5CFEYDZFYAQ, seg_pos={0 0}, upload=472966]: Uploading piece... Dec 28 02:48:08.793923431 (539h19m8.665006476s) [noise=true , address=45.91.116.9:4263 , piece_id=BXPFQ6DQVFW7FOC2BZPGCDD2VXHFK6KUFIYRYUB7N5BM6IDRWLUA, seg_pos={0 0}, upload=472966]: Uploading piece... Dec 28 02:48:08.795567902 (539h19m8.666650947s) [noise=true , address=145.79.227.35:4482 , piece_id=CFEUOAI4AX7OWKKIS6WDMRQVHRAWT63TPTEOOIHABXKOTVF3PKKA, seg_pos={0 0}, upload=472966]: Uploading piece... Dec 28 02:48:10.512912351 (539h19m10.383995386s) [upload=472966]: Next returned nil segment Dec 28 02:48:10.512941134 (539h19m10.38402417s) [upload=472966]: Waiting for error group managing segments... Dec 28 02:48:10.798311908 (539h19m10.669394943s) [noise=true , address=145.79.227.52:4447 , piece_id=K4PJNQOSUOKNI6AMNJJE4SUGB5T2U5NNZ5S6WEFLMPY4QF7ZJBFA, seg_pos={0 0}, upload=472966]: Done uploading piece. err: <nil> Dec 28 02:48:10.806197664 (539h19m10.677280699s) [noise=true , address=145.79.227.36:4517 , piece_id=YCLHMPUBNGEU5HASWFFXOLTYTH2EJTLKLEWSJLLWHJZOUSBATX7A, seg_pos={0 0}, upload=472966]: Done uploading piece. err: <nil> Dec 28 02:48:10.818647962 (539h19m10.689730997s) [noise=true , address=145.79.227.18:4006 , piece_id=SMFZKQVKTGYTCXG333EME6R27MI3NWYWTPNGXPOYUP6GIEEYAUEA, seg_pos={0 0}, upload=472966]: Done uploading piece. err: <nil> Dec 28 02:48:10.818847171 (539h19m10.689930216s) [noise=true , address=145.79.227.51:4410 , piece_id=GXLX6NGMPLKTNWONWYZTQCBV3WQPJRP3NHA5PMG34KVFTNIFXX3A, seg_pos={0 0}, upload=472966]: Done uploading piece. err: <nil> Dec 28 02:48:10.820563099 (539h19m10.691646134s) [noise=true , address=145.79.227.42:4122 , piece_id=CYGIT6EP5MNHDK57C3IVYOLKEMOV23ISSHUNUCD2GEM4TQHQOZEQ, seg_pos={0 0}, upload=472966]: Done uploading piece. err: <nil> Dec 28 02:48:10.828772470 (539h19m10.699855515s) [noise=true , address=145.79.227.37:4567 , piece_id=55X5OICJGOZ5MIMYLGPV5RO4ZXQC7KNDWLXXOD55DSVIPFDVL3YA, seg_pos={0 0}, upload=472966]: Done uploading piece. err: <nil> Dec 28 02:48:10.835424847 (539h19m10.706507882s) [noise=true , address=145.79.227.35:4482 , piece_id=CFEUOAI4AX7OWKKIS6WDMRQVHRAWT63TPTEOOIHABXKOTVF3PKKA, seg_pos={0 0}, upload=472966]: Done uploading piece. err: <nil> Dec 28 02:48:11.037975849 (539h19m10.909058884s) [noise=true , address=45.91.116.9:4263 , piece_id=BXPFQ6DQVFW7FOC2BZPGCDD2VXHFK6KUFIYRYUB7N5BM6IDRWLUA, seg_pos={0 0}, upload=472966]: Done uploading piece. err: <nil> Dec 28 02:48:11.045835193 (539h19m10.916918249s) [noise=true , address=145.79.227.50:4375 , piece_id=SCVWARBRGLK7R3QTHVUK4MVQ4H5QGSZNDANA5GXMZIPTM32N7M4Q, seg_pos={0 0}, upload=472966]: Done uploading piece. err: <nil> Dec 28 02:48:11.061953125 (539h19m10.93303617s) [noise=true , address=145.79.227.38:4977 , piece_id=UHQW5HGZ57RIRK7YWVUMSSEFU5P37R7MZG2AKHEDBKV6LSK3D6AQ, seg_pos={0 0}, upload=472966]: Done uploading piece. err: <nil> Dec 28 02:48:11.062445051 (539h19m10.933528096s) [noise=true , address=145.79.227.15:4873 , piece_id=O7FJ4SYP5U5MTXYUSO3VSQI46SYZPNTMNBJXG3L2L5CFEYDZFYAQ, seg_pos={0 0}, upload=472966]: Done uploading piece. err: <nil> Dec 28 02:48:11.084465663 (539h19m10.955548708s) [noise=true , address=45.91.116.13:4155 , piece_id=KGBTH5PYUSKO4UZFXQVUNQBUJZKATHHHPDHWQFGQH5YFDXQXDI5Q, seg_pos={0 0}, upload=472966]: Done uploading piece. err: <nil> Dec 28 02:48:11.287058493 (539h19m11.158141528s) [noise=true , address=45.91.116.4:4937 , piece_id=FGP77TID47WOFQ2BSQGYXLIQ4CHGD6VGEFKT34B3DMJY5LNTZAJQ, seg_pos={0 0}, upload=472966]: Done uploading piece. err: <nil> Dec 28 02:48:11.287083070 (539h19m11.158166125s) [seg_pos={0 0}, upload=472966]: Segment reached optimal threshold of 13 pieces. Dec 28 02:48:11.287326774 (539h19m11.158409819s) [noise=true , address=145.79.227.3:4150 , piece_id=ZFGYT36TLZXG2L3TTSGYKVCZ544EFRTHUQAW4MNWIOXFOZTMIMPA, seg_pos={0 0}, upload=472966]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK): protocol: write tcp 188.166.225.170:55286->145.79.227.3:4150: use of closed network connection; write tcp 188.166.225.170:55286->145.79.227.3:4150: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55286->145.79.227.3:4150: use of closed network connection Dec 28 02:48:11.287337965 (539h19m11.15842101s) [noise=true , address=145.79.227.3:4150 , piece_id=ZFGYT36TLZXG2L3TTSGYKVCZ544EFRTHUQAW4MNWIOXFOZTMIMPA, seg_pos={0 0}, upload=472966]: Long tail cancellation due to optimal threshold Dec 28 02:48:11.287441923 (539h19m11.158524978s) [noise=true , address=77.90.31.21:4878 , piece_id=JAAHWDOLBLJRYNSIWUBPU3GY3XZ33VBBY673SLTWPNFCYCQI45LQ, seg_pos={0 0}, upload=472966]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: write tcp 188.166.225.170:56116->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:56116->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56116->77.90.31.21:4878: use of closed network connection Dec 28 02:48:11.287451331 (539h19m11.158534377s) [noise=true , address=77.90.31.21:4878 , piece_id=JAAHWDOLBLJRYNSIWUBPU3GY3XZ33VBBY673SLTWPNFCYCQI45LQ, seg_pos={0 0}, upload=472966]: Long tail cancellation due to optimal threshold Dec 28 02:48:11.287660249 (539h19m11.158743294s) [noise=true , address=77.90.31.11:4553 , piece_id=Y3LLQHC6WQG6JKQB56GRUT3P7E6YDOEZLDDBJPRU3E6VOWOIPGJQ, seg_pos={0 0}, upload=472966]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: write tcp 188.166.225.170:38788->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:38788->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38788->77.90.31.11:4553: use of closed network connection Dec 28 02:48:11.287670248 (539h19m11.158753283s) [noise=true , address=77.90.31.11:4553 , piece_id=Y3LLQHC6WQG6JKQB56GRUT3P7E6YDOEZLDDBJPRU3E6VOWOIPGJQ, seg_pos={0 0}, upload=472966]: Long tail cancellation due to optimal threshold Dec 28 02:48:11.287690756 (539h19m11.158773802s) [seg_pos={0 0}, upload=472966]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 02:48:11.287777362 (539h19m11.158860407s) [upload=472966]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 02:48:11.287785637 (539h19m11.158868683s) [upload=472966]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 02:48:11.581134853 (539h19m11.452217898s) [upload=472966]: Done uploading segments... Dec 28 02:48:11.581162696 (539h19m11.452245741s) [upload=472966]: Upload finished. err: <nil> Duration:* 30.003004326s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [......X...................X.....X...........................] duration: 41.94377203s trace: https://jaeger.dev.storj.io/trace/1822be52d200e5b5 FAIL (2025-12-28 02:48:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1ZAfYZ4wgpZVvatj6wtHona3hTaPcLxAmXC1DT2uKQyo8nLQQ7, Piece ID: 5MMQ44MXRT4XEL4X6WO54WRG2XYMX3AN3FNQ2H2XIUTKBXI3EEPA) context deadline exceeded; piecestore: (Node ID: 124guAM82oibYZsMho67Em6Ke6yEZXwT6LPMLLSZZBLqwoFGUkn, Piece ID: 22WJIPX4SM5NTIAW5ZRM2UKX32CTDBFCXPCCHRRWRCUCOF5CRE5Q) context deadline exceeded; piecestore: (Node ID: 1e6p1TD5CqUyWdZjkrxh2kRqMuyDiMUr9eimfeayDAKwhtBrZ8, Piece ID: XRI5EJFN7YYUG4QKLDGEFT6YAUHCUTTSXF735UDR6LZZBGMAZDJQ) context deadline exceeded ======LOGS====== 5.234848204 (538h55m35.105931239s) [noise=true , address=77.90.31.11:4572 , piece_id=5MMQ44MXRT4XEL4X6WO54WRG2XYMX3AN3FNQ2H2XIUTKBXI3EEPA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.235412057 (538h55m35.106495102s) [noise=true , address=145.79.227.22:4512 , piece_id=D46YLTIKGHG36DQ4YBKBSUDV4GGB4WVDWNOJS27IYYLN6AYZ5QFQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.235820946 (538h55m35.106903981s) [noise=true , address=77.90.31.19:4874 , piece_id=RHZUXUYMJU6RKQGSDZM5WDWIN5SQ7YIRSRATNNKT5GEMU2QH444A, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.236192192 (538h55m35.107275237s) [noise=true , address=77.90.31.16:4760 , piece_id=XRI5EJFN7YYUG4QKLDGEFT6YAUHCUTTSXF735UDR6LZZBGMAZDJQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.236564772 (538h55m35.107647817s) [noise=true , address=145.79.227.40:4062 , piece_id=PW6QWIPQWVUETLC6DJ624G6IB4P6EM4SYB72NAMQ4V6NUKNLUYLQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.236929766 (538h55m35.108012812s) [noise=true , address=64.49.13.139:4480 , piece_id=JAJY3F7IDFFMFKCK2QJV6W74RIESLRKBLXMN27Q3C45JWEPLBZLQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.237916846 (538h55m35.108999881s) [noise=true , address=145.79.227.34:4932 , piece_id=UPXSCQ45QDGPKNJXRFJ33EAA3QENVHRVQARU5TFJTEWICZNR253Q, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.238852948 (538h55m35.109935983s) [noise=true , address=45.91.116.12:4257 , piece_id=ZI67LIAHZW4G357GWKDPTB5SMXYGNXZTTTDMNAETDS2B6SWVDAEQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239152910 (538h55m35.110235945s) [noise=true , address=145.79.227.44:4222 , piece_id=CZ2CS62CTHQFAIE3S3JO2N3VJ4XIR5YJ2LMRVLLX7I4VM2VZVQVA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239409969 (538h55m35.110493004s) [noise=true , address=145.79.227.30:4773 , piece_id=KUGJXZCNMUE7WSK3VLY4AUBSOT4H6IY4XB4NCBGAGCNUG7F7IODQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239681136 (538h55m35.110764171s) [noise=true , address=77.90.31.13:4635 , piece_id=IIOY5X2W3XW4C2OFEBRCABMYFBI24RRTYNUSKN6KJU7CTUW4KONQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239954676 (538h55m35.111037711s) [noise=true , address=45.91.116.9:4281 , piece_id=JZLDW777CBX2BP7MLOQV4NFP3H4XJVCNQXTHNSUWPVDOJLPZK36A, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.240189914 (538h55m35.111272949s) [noise=true , address=145.79.227.35:4512 , piece_id=3E3VCCC45XVT5FHFDKXSTGGAGYBEML3BKU6JJTT33LCD2YMOMBZQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.240449147 (538h55m35.111532182s) [noise=true , address=45.91.116.8:4064 , piece_id=22WJIPX4SM5NTIAW5ZRM2UKX32CTDBFCXPCCHRRWRCUCOF5CRE5Q, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.240639640 (538h55m35.111722675s) [noise=true , address=145.79.227.32:4853 , piece_id=MBNCFDBQ4SZZXV4F7J5TRMOUYUQ5IPKQCGYGGM5SOIFJ6ZYCAUYA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.241137798 (538h55m35.112220833s) [noise=true , address=77.90.31.16:4751 , piece_id=N3Z2LLMCIA2F42VXZA7IDRU4XEKGOICH6TRVERODU76MFFE4QSHA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.241383797 (538h55m35.112466842s) [noise=true , address=77.90.31.18:4836 , piece_id=BSVEFI52EKN44CMINFUSZBNOH53IW5TXLVTF5IH6ZUTNII6RRLVA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:36.592880816 (538h55m36.463963871s) [upload=472631]: Next returned nil segment Dec 28 02:24:36.592901625 (538h55m36.46398467s) [upload=472631]: Waiting for error group managing segments... Dec 28 02:24:36.946933741 (538h55m36.818016786s) [noise=true , address=64.49.13.139:4480 , piece_id=JAJY3F7IDFFMFKCK2QJV6W74RIESLRKBLXMN27Q3C45JWEPLBZLQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.128189692 (538h55m36.999272737s) [noise=true , address=145.79.227.35:4512 , piece_id=3E3VCCC45XVT5FHFDKXSTGGAGYBEML3BKU6JJTT33LCD2YMOMBZQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.160760084 (538h55m37.031843119s) [noise=true , address=145.79.227.34:4932 , piece_id=UPXSCQ45QDGPKNJXRFJ33EAA3QENVHRVQARU5TFJTEWICZNR253Q, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.288838207 (538h55m37.159921252s) [noise=true , address=77.90.31.11:4572 , piece_id=5MMQ44MXRT4XEL4X6WO54WRG2XYMX3AN3FNQ2H2XIUTKBXI3EEPA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.319996310 (538h55m37.191079345s) [noise=true , address=145.79.227.22:4512 , piece_id=D46YLTIKGHG36DQ4YBKBSUDV4GGB4WVDWNOJS27IYYLN6AYZ5QFQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.320513273 (538h55m37.191596319s) [noise=true , address=77.90.31.18:4836 , piece_id=BSVEFI52EKN44CMINFUSZBNOH53IW5TXLVTF5IH6ZUTNII6RRLVA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.336522926 (538h55m37.207605971s) [noise=true , address=145.79.227.32:4853 , piece_id=MBNCFDBQ4SZZXV4F7J5TRMOUYUQ5IPKQCGYGGM5SOIFJ6ZYCAUYA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.384868031 (538h55m37.255951066s) [noise=true , address=77.90.31.16:4751 , piece_id=N3Z2LLMCIA2F42VXZA7IDRU4XEKGOICH6TRVERODU76MFFE4QSHA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.453563554 (538h55m37.324646599s) [noise=true , address=45.91.116.8:4064 , piece_id=22WJIPX4SM5NTIAW5ZRM2UKX32CTDBFCXPCCHRRWRCUCOF5CRE5Q, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.479454839 (538h55m37.350537884s) [noise=true , address=45.91.116.12:4257 , piece_id=ZI67LIAHZW4G357GWKDPTB5SMXYGNXZTTTDMNAETDS2B6SWVDAEQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.659710174 (538h55m37.530793219s) [noise=true , address=77.90.31.13:4635 , piece_id=IIOY5X2W3XW4C2OFEBRCABMYFBI24RRTYNUSKN6KJU7CTUW4KONQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.715039128 (538h55m37.586122163s) [noise=true , address=77.90.31.16:4760 , piece_id=XRI5EJFN7YYUG4QKLDGEFT6YAUHCUTTSXF735UDR6LZZBGMAZDJQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.977473406 (538h55m37.848556441s) [noise=true , address=77.90.31.19:4874 , piece_id=RHZUXUYMJU6RKQGSDZM5WDWIN5SQ7YIRSRATNNKT5GEMU2QH444A, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:38.016302745 (538h55m37.88738578s) [noise=true , address=145.79.227.30:4773 , piece_id=KUGJXZCNMUE7WSK3VLY4AUBSOT4H6IY4XB4NCBGAGCNUG7F7IODQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:38.016326982 (538h55m37.887410027s) [seg_pos={0 0}, upload=472631]: Segment reached optimal threshold of 14 pieces. Dec 28 02:24:38.016585353 (538h55m37.887668398s) [noise=true , address=145.79.227.30:4766 , piece_id=67M2DMPAVWIQKQSWUAG73D4E4NRRSG4XEVPOUVIJPEQOUV7A6VVA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1k6zdamuKPkBdzyNZNWx4FCmaTkv33PchcwvTRR8bgcETBs4K7): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016594450 (538h55m37.887677495s) [noise=true , address=145.79.227.30:4766 , piece_id=67M2DMPAVWIQKQSWUAG73D4E4NRRSG4XEVPOUVIJPEQOUV7A6VVA, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016627093 (538h55m37.887710148s) [noise=true , address=45.91.116.9:4281 , piece_id=JZLDW777CBX2BP7MLOQV4NFP3H4XJVCNQXTHNSUWPVDOJLPZK36A, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gxf7DpJZthyR6f1PExX56tJavjhPw9Gxb4h8mtmeYKW3t3tS1): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016635258 (538h55m37.887718293s) [noise=true , address=45.91.116.9:4281 , piece_id=JZLDW777CBX2BP7MLOQV4NFP3H4XJVCNQXTHNSUWPVDOJLPZK36A, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016871298 (538h55m37.887954363s) [noise=true , address=145.79.227.44:4222 , piece_id=CZ2CS62CTHQFAIE3S3JO2N3VJ4XIR5YJ2LMRVLLX7I4VM2VZVQVA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZKW35KvrsoYRefbYeUFZo8bh4XZYycNKGEvTFjzibezjXgj3e): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016880124 (538h55m37.887963169s) [noise=true , address=145.79.227.44:4222 , piece_id=CZ2CS62CTHQFAIE3S3JO2N3VJ4XIR5YJ2LMRVLLX7I4VM2VZVQVA, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016934839 (538h55m37.888017884s) [noise=true , address=145.79.227.40:4062 , piece_id=PW6QWIPQWVUETLC6DJ624G6IB4P6EM4SYB72NAMQ4V6NUKNLUYLQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016943425 (538h55m37.88802649s) [noise=true , address=145.79.227.40:4062 , piece_id=PW6QWIPQWVUETLC6DJ624G6IB4P6EM4SYB72NAMQ4V6NUKNLUYLQ, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016968252 (538h55m37.888051297s) [seg_pos={0 0}, upload=472631]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 02:24:38.017067822 (538h55m37.888150867s) [upload=472631]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 02:24:38.017074244 (538h55m37.888157289s) [upload=472631]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 02:24:38.304306667 (538h55m38.175389712s) [upload=472631]: Done uploading segments... Dec 28 02:24:38.304329641 (538h55m38.175412686s) [upload=472631]: Upload finished. err: <nil> Duration: 41.943803421s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 37 out of 60 is failed (threshold=3) [X..X...XX..X.XXX.XXX.X.....XXX.XXXXXX.XXXX.XXXX..X..XXXXX.XX] duration: 30.00225639s trace: https://jaeger.dev.storj.io/trace/27a826d62fe2fd12 FAIL (2025-12-28 02:48:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 02:47:38.295722756 (539h18m38.166805791s) [upload=472961]: Starting upload Dec 28 02:47:38.295773704 (539h18m38.166856739s) [upload=472961]: Done starting upload Dec 28 02:47:38.295941032 (539h18m38.167024067s) [upload=472961]: Uploading segments... Dec 28 02:47:38.295976019 (539h18m38.167059064s) [upload=472961]: Got next segment. Inline: false Dec 28 02:47:38.295998822 (539h18m38.167081857s) [upload=472961]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 02:47:38.296002569 (539h18m38.167085604s) [upload=472961]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 02:47:38.539850566 (539h18m38.410933602s) [seg_pos={0 0}, upload=472961]: Begin upload segment... Dec 28 02:47:38.539889811 (539h18m38.410972846s) [seg_pos={0 0}, upload=472961]: Done begin upload segment. Dec 28 02:47:38.539925118 (539h18m38.411008163s) [noise=true , address=145.79.227.11:4729 , piece_id=6KAOXNHOSLZB3ESA34KD2CIETGDNA7PAZCV6CH3XYZXDGHAFPKHA, seg_pos={0 0}, upload=472961]: Uploading piece... Dec 28 02:47:38.539931921 (539h18m38.411014956s) [noise=true , address=45.91.116.13:4153 , piece_id=YEXL2UXKLXX6BCQD3LOKMMNADD7MPG6IGERPVQLVDNMZYGXLIVTQ, seg_pos={0 0}, upload=472961]: Uploading piece... Dec 28 02:47:38.540347012 (539h18m38.411430047s) [noise=true , address=77.90.31.12:4590 , piece_id=GXRI4ZYSENJZBN2BFALM3FVEVP5CWWLD4BTJZL5U3IA6D3M2TCMQ, seg_pos={0 0}, upload=472961]: Uploading piece... Dec 28 02:47:38.540647404 (539h18m38.411730439s) [noise=true , address=145.79.227.52:4445 , piece_id=ARD2B4BF76XQOHPZI225Y6HC7QS4YMEMGNW33H3V4ZE3QWWRMOTQ, seg_pos={0 0}, upload=472961]: Uploading piece... Dec 28 02:47:38.540911095 (539h18m38.411994141s) [noise=true , address=45.91.116.4:4939 , piece_id=YZFEX32PDYRNWC4FBDL2QXUVYBH74JOIZ6VAXPK2J5TFIV77FMLA, seg_pos={0 0}, upload=472961]: Uploading piece... Dec 28 02:47:38.541226025 (539h18m38.41230906s) [noise=true , address=145.79.227.45:4229 , piece_id=MBATVFXFB7X3QOOLFMYJXNZJ4Y2GLUREZPGYGBSOXLZ6DOJKM4VQ, seg_pos={0 0}, upload=472961]: Uploading piece... Dec 28 02:47:38.541603835 (539h18m38.41268688s) [noise=true , address=45.91.116.9:4262 , piece_id=JVFKTC7NCULRFMUDEF54GJ4OHP6HC4GC6PLE5W3UI74RFQHKCUQA, seg_pos={0 0}, upload=472961]: Uploading piece... Dec 28 02:47:38.541915528 (539h18m38.412998563s) [noise=true , address=145.79.227.46:4603 , piece_id=GRUF4AF3HQ2MFOPK5FMGQBSBXTWEG2452TIKWMAHHKUB4C342DEA, seg_pos={0 0}, upload=472961]: Uploading piece... Dec 28 02:47:38.542223986 (539h18m38.413307021s) [noise=true , address=145.79.227.14:4838 , piece_id=4JIWQ272MTVJLESOJQRDXMGKQ5D67VC3IL6I6XG3YQPE37D3YKEA, seg_pos={0 0}, upload=472961]: Uploading piece... Dec 28 02:47:38.543640724 (539h18m38.414723769s) [noise=true , address=145.79.227.17:4972 , piece_id=HUNOX4PTKQHMKYZNG33EJYEB7QMVXYF7DKJHZZ2U52TUWNIFN54Q, seg_pos={0 0}, upload=472961]: Uploading piece... Dec 28 02:47:38.544905091 (539h18m38.415988126s) [noise=true , address=77.90.31.11:4554 , piece_id=FVDHLGZNBMO2ATNGTN4VKYD6X2NEYANOTYTIUX5MGXMLHFWMQQ6A, seg_pos={0 0}, upload=472961]: Uploading piece... Dec 28 02:47:38.545970620 (539h18m38.417053655s) [noise=true , address=145.79.227.5:4221 , piece_id=ODS4T4JDZ7GFSVHCMOWDX36P5Y7XFAU6ALCB3EUZMAOCOI2QQ6IQ, seg_pos={0 0}, upload=472961]: Uploading piece... Dec 28 02:47:38.547212514 (539h18m38.418295549s) [noise=true , address=145.79.227.38:4978 , piece_id=3DJEFLV4ZP5QV2JELOYV33UV5M4Y4PSBJ6DLST2XLQJAGOL6ONCA, seg_pos={0 0}, upload=472961]: Uploading piece... Dec 28 02:47:38.548318551 (539h18m38.419401586s) [noise=true , address=145.79.227.12:4765 , piece_id=K5KTDZNSIBXVSL4XV7KR4PNCHQ2RABSFIW43STNBSHBVKNKKX5SQ, seg_pos={0 0}, upload=472961]: Uploading piece... Dec 28 02:47:38.549496434 (539h18m38.420579469s) [noise=true , address=145.79.227.7:4295 , piece_id=KSKC47MBUHRLIBZADYFV7WDHKZRP3EZQBM57VKJUNXLBJ2JR4PLQ, seg_pos={0 0}, upload=472961]: Uploading piece... Dec 28 02:47:38.551193686 (539h18m38.422276721s) [noise=true , address=45.91.116.6:4976 , piece_id=BV4UFLB5KO7JW4SQL7QKF3SL5XRC2ITOJHBDRFX3PJ3YNM637RSA, seg_pos={0 0}, upload=472961]: Uploading piece... Dec 28 02:47:40.268934860 (539h18m40.140017905s) [upload=472961]: Next returned nil segment Dec 28 02:47:40.268948656 (539h18m40.140031691s) [upload=472961]: Waiting for error group managing segments... Dec 28 02:47:40.556283249 (539h18m40.427366294s) [noise=true , address=145.79.227.45:4229 , piece_id=MBATVFXFB7X3QOOLFMYJXNZJ4Y2GLUREZPGYGBSOXLZ6DOJKM4VQ, seg_pos={0 0}, upload=472961]: Done uploading piece. err: <nil> Dec 28 02:47:40.561722977 (539h18m40.432806012s) [noise=true , address=145.79.227.14:4838 , piece_id=4JIWQ272MTVJLESOJQRDXMGKQ5D67VC3IL6I6XG3YQPE37D3YKEA, seg_pos={0 0}, upload=472961]: Done uploading piece. err: <nil> Dec 28 02:47:40.573462541 (539h18m40.444545586s) [noise=true , address=145.79.227.7:4295 , piece_id=KSKC47MBUHRLIBZADYFV7WDHKZRP3EZQBM57VKJUNXLBJ2JR4PLQ, seg_pos={0 0}, upload=472961]: Done uploading piece. err: <nil> Dec 28 02:47:40.575169451 (539h18m40.446252496s) [noise=true , address=145.79.227.12:4765 , piece_id=K5KTDZNSIBXVSL4XV7KR4PNCHQ2RABSFIW43STNBSHBVKNKKX5SQ, seg_pos={0 0}, upload=472961]: Done uploading piece. err: <nil> Dec 28 02:47:40.584985212 (539h18m40.456068257s) [noise=true , address=145.79.227.17:4972 , piece_id=HUNOX4PTKQHMKYZNG33EJYEB7QMVXYF7DKJHZZ2U52TUWNIFN54Q, seg_pos={0 0}, upload=472961]: Done uploading piece. err: <nil> Dec 28 02:47:40.595015461 (539h18m40.466098506s) [noise=true , address=145.79.227.38:4978 , piece_id=3DJEFLV4ZP5QV2JELOYV33UV5M4Y4PSBJ6DLST2XLQJAGOL6ONCA, seg_pos={0 0}, upload=472961]: Done uploading piece. err: <nil> Dec 28 02:47:40.595382601 (539h18m40.466465636s) [noise=true , address=145.79.227.5:4221 , piece_id=ODS4T4JDZ7GFSVHCMOWDX36P5Y7XFAU6ALCB3EUZMAOCOI2QQ6IQ, seg_pos={0 0}, upload=472961]: Done uploading piece. err: <nil> Dec 28 02:47:40.737615586 (539h18m40.608698621s) [noise=true , address=77.90.31.12:4590 , piece_id=GXRI4ZYSENJZBN2BFALM3FVEVP5CWWLD4BTJZL5U3IA6D3M2TCMQ, seg_pos={0 0}, upload=472961]: Done uploading piece. err: <nil> Dec 28 02:47:40.804425703 (539h18m40.675508738s) [noise=true , address=45.91.116.6:4976 , piece_id=BV4UFLB5KO7JW4SQL7QKF3SL5XRC2ITOJHBDRFX3PJ3YNM637RSA, seg_pos={0 0}, upload=472961]: Done uploading piece. err: <nil> Dec 28 02:47:40.853035474 (539h18m40.724118519s) [noise=true , address=77.90.31.11:4554 , piece_id=FVDHLGZNBMO2ATNGTN4VKYD6X2NEYANOTYTIUX5MGXMLHFWMQQ6A, seg_pos={0 0}, upload=472961]: Done uploading piece. err: <nil> Dec 28 02:47:40.855012869 (539h18m40.726095904s) [noise=true , address=45.91.116.13:4153 , piece_id=YEXL2UXKLXX6BCQD3LOKMMNADD7MPG6IGERPVQLVDNMZYGXLIVTQ, seg_pos={0 0}, upload=472961]: Done uploading piece. err: <nil> Dec 28 02:47:40.910474948 (539h18m40.781557993s) [noise=true , address=45.91.116.9:4262 , piece_id=JVFKTC7NCULRFMUDEF54GJ4OHP6HC4GC6PLE5W3UI74RFQHKCUQA, seg_pos={0 0}, upload=472961]: Done uploading piece. err: <nil> Dec 28 02:47:41.085137715 (539h18m40.95622075s) [noise=true , address=145.79.227.46:4603 , piece_id=GRUF4AF3HQ2MFOPK5FMGQBSBXTWEG2452TIKWMAHHKUB4C342DEA, seg_pos={0 0}, upload=472961]: Done uploading piece. err: <nil> Dec 28 02:47:41.085161641 (539h18m40.956244676s) [seg_pos={0 0}, upload=472961]: Segment reached optimal threshold of 13 pieces. Dec 28 02:47:41.085374406 (539h18m40.956457461s) [noise=true , address=145.79.227.52:4445 , piece_id=ARD2B4BF76XQOHPZI225Y6HC7QS4YMEMGNW33H3V4ZE3QWWRMOTQ, seg_pos={0 0}, upload=472961]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12vXrkhLqCofhR4Fmfqb98gxnqWQF6vMV4q3ATbRvPfcqi82WSo): protocol: expected piece hash; context canceled; EOF Dec 28 02:47:41.085382541 (539h18m40.956465586s) [noise=true , address=145.79.227.52:4445 , piece_id=ARD2B4BF76XQOHPZI225Y6HC7QS4YMEMGNW33H3V4ZE3QWWRMOTQ, seg_pos={0 0}, upload=472961]: Long tail cancellation due to optimal threshold Dec 28 02:47:41.085521245 (539h18m40.95660428s) [noise=true , address=45.91.116.4:4939 , piece_id=YZFEX32PDYRNWC4FBDL2QXUVYBH74JOIZ6VAXPK2J5TFIV77FMLA, seg_pos={0 0}, upload=472961]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 28 02:47:41.085535603 (539h18m40.956618638s) [noise=true , address=45.91.116.4:4939 , piece_id=YZFEX32PDYRNWC4FBDL2QXUVYBH74JOIZ6VAXPK2J5TFIV77FMLA, seg_pos={0 0}, upload=472961]: Long tail cancellation due to optimal threshold Dec 28 02:47:41.085693122 (539h18m40.956776198s) [noise=true , address=145.79.227.11:4729 , piece_id=6KAOXNHOSLZB3ESA34KD2CIETGDNA7PAZCV6CH3XYZXDGHAFPKHA, seg_pos={0 0}, upload=472961]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR): protocol: expected piece hash; context canceled; EOF Dec 28 02:47:41.085702561 (539h18m40.956785596s) [noise=true , address=145.79.227.11:4729 , piece_id=6KAOXNHOSLZB3ESA34KD2CIETGDNA7PAZCV6CH3XYZXDGHAFPKHA, seg_pos={0 0}, upload=472961]: Long tail cancellation due to optimal threshold Dec 28 02:47:41.085727508 (539h18m40.956810553s) [seg_pos={0 0}, upload=472961]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 02:47:41.085995258 (539h18m40.957078293s) [upload=472961]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 02:47:41.086002752 (539h18m40.957085787s) [upload=472961]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 02:47:41.379350044 (539h18m41.250433079s) [upload=472961]: Done uploading segments... Dec 28 02:47:41.379370173 (539h18m41.250453208s) [upload=472961]: Upload finished. err: <nil> Duration: 30.002280906s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [...X.........................X................X....X........] duration: 5.000664053s trace: https://jaeger.dev.storj.io/trace/5efee46a94502508 FAIL (2025-12-28 02:47:53 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000687377s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......X...................X.....X..........................] duration: 42.752672493s trace: https://jaeger.dev.storj.io/trace/1822be52d200e5b5 FAIL (2025-12-28 02:47:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1ZAfYZ4wgpZVvatj6wtHona3hTaPcLxAmXC1DT2uKQyo8nLQQ7, Piece ID: 5MMQ44MXRT4XEL4X6WO54WRG2XYMX3AN3FNQ2H2XIUTKBXI3EEPA) context deadline exceeded; piecestore: (Node ID: 124guAM82oibYZsMho67Em6Ke6yEZXwT6LPMLLSZZBLqwoFGUkn, Piece ID: 22WJIPX4SM5NTIAW5ZRM2UKX32CTDBFCXPCCHRRWRCUCOF5CRE5Q) context deadline exceeded; piecestore: (Node ID: 1e6p1TD5CqUyWdZjkrxh2kRqMuyDiMUr9eimfeayDAKwhtBrZ8, Piece ID: XRI5EJFN7YYUG4QKLDGEFT6YAUHCUTTSXF735UDR6LZZBGMAZDJQ) context deadline exceeded ======LOGS====== 5.234848204 (538h55m35.105931239s) [noise=true , address=77.90.31.11:4572 , piece_id=5MMQ44MXRT4XEL4X6WO54WRG2XYMX3AN3FNQ2H2XIUTKBXI3EEPA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.235412057 (538h55m35.106495102s) [noise=true , address=145.79.227.22:4512 , piece_id=D46YLTIKGHG36DQ4YBKBSUDV4GGB4WVDWNOJS27IYYLN6AYZ5QFQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.235820946 (538h55m35.106903981s) [noise=true , address=77.90.31.19:4874 , piece_id=RHZUXUYMJU6RKQGSDZM5WDWIN5SQ7YIRSRATNNKT5GEMU2QH444A, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.236192192 (538h55m35.107275237s) [noise=true , address=77.90.31.16:4760 , piece_id=XRI5EJFN7YYUG4QKLDGEFT6YAUHCUTTSXF735UDR6LZZBGMAZDJQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.236564772 (538h55m35.107647817s) [noise=true , address=145.79.227.40:4062 , piece_id=PW6QWIPQWVUETLC6DJ624G6IB4P6EM4SYB72NAMQ4V6NUKNLUYLQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.236929766 (538h55m35.108012812s) [noise=true , address=64.49.13.139:4480 , piece_id=JAJY3F7IDFFMFKCK2QJV6W74RIESLRKBLXMN27Q3C45JWEPLBZLQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.237916846 (538h55m35.108999881s) [noise=true , address=145.79.227.34:4932 , piece_id=UPXSCQ45QDGPKNJXRFJ33EAA3QENVHRVQARU5TFJTEWICZNR253Q, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.238852948 (538h55m35.109935983s) [noise=true , address=45.91.116.12:4257 , piece_id=ZI67LIAHZW4G357GWKDPTB5SMXYGNXZTTTDMNAETDS2B6SWVDAEQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239152910 (538h55m35.110235945s) [noise=true , address=145.79.227.44:4222 , piece_id=CZ2CS62CTHQFAIE3S3JO2N3VJ4XIR5YJ2LMRVLLX7I4VM2VZVQVA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239409969 (538h55m35.110493004s) [noise=true , address=145.79.227.30:4773 , piece_id=KUGJXZCNMUE7WSK3VLY4AUBSOT4H6IY4XB4NCBGAGCNUG7F7IODQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239681136 (538h55m35.110764171s) [noise=true , address=77.90.31.13:4635 , piece_id=IIOY5X2W3XW4C2OFEBRCABMYFBI24RRTYNUSKN6KJU7CTUW4KONQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239954676 (538h55m35.111037711s) [noise=true , address=45.91.116.9:4281 , piece_id=JZLDW777CBX2BP7MLOQV4NFP3H4XJVCNQXTHNSUWPVDOJLPZK36A, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.240189914 (538h55m35.111272949s) [noise=true , address=145.79.227.35:4512 , piece_id=3E3VCCC45XVT5FHFDKXSTGGAGYBEML3BKU6JJTT33LCD2YMOMBZQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.240449147 (538h55m35.111532182s) [noise=true , address=45.91.116.8:4064 , piece_id=22WJIPX4SM5NTIAW5ZRM2UKX32CTDBFCXPCCHRRWRCUCOF5CRE5Q, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.240639640 (538h55m35.111722675s) [noise=true , address=145.79.227.32:4853 , piece_id=MBNCFDBQ4SZZXV4F7J5TRMOUYUQ5IPKQCGYGGM5SOIFJ6ZYCAUYA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.241137798 (538h55m35.112220833s) [noise=true , address=77.90.31.16:4751 , piece_id=N3Z2LLMCIA2F42VXZA7IDRU4XEKGOICH6TRVERODU76MFFE4QSHA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.241383797 (538h55m35.112466842s) [noise=true , address=77.90.31.18:4836 , piece_id=BSVEFI52EKN44CMINFUSZBNOH53IW5TXLVTF5IH6ZUTNII6RRLVA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:36.592880816 (538h55m36.463963871s) [upload=472631]: Next returned nil segment Dec 28 02:24:36.592901625 (538h55m36.46398467s) [upload=472631]: Waiting for error group managing segments... Dec 28 02:24:36.946933741 (538h55m36.818016786s) [noise=true , address=64.49.13.139:4480 , piece_id=JAJY3F7IDFFMFKCK2QJV6W74RIESLRKBLXMN27Q3C45JWEPLBZLQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.128189692 (538h55m36.999272737s) [noise=true , address=145.79.227.35:4512 , piece_id=3E3VCCC45XVT5FHFDKXSTGGAGYBEML3BKU6JJTT33LCD2YMOMBZQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.160760084 (538h55m37.031843119s) [noise=true , address=145.79.227.34:4932 , piece_id=UPXSCQ45QDGPKNJXRFJ33EAA3QENVHRVQARU5TFJTEWICZNR253Q, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.288838207 (538h55m37.159921252s) [noise=true , address=77.90.31.11:4572 , piece_id=5MMQ44MXRT4XEL4X6WO54WRG2XYMX3AN3FNQ2H2XIUTKBXI3EEPA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.319996310 (538h55m37.191079345s) [noise=true , address=145.79.227.22:4512 , piece_id=D46YLTIKGHG36DQ4YBKBSUDV4GGB4WVDWNOJS27IYYLN6AYZ5QFQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.320513273 (538h55m37.191596319s) [noise=true , address=77.90.31.18:4836 , piece_id=BSVEFI52EKN44CMINFUSZBNOH53IW5TXLVTF5IH6ZUTNII6RRLVA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.336522926 (538h55m37.207605971s) [noise=true , address=145.79.227.32:4853 , piece_id=MBNCFDBQ4SZZXV4F7J5TRMOUYUQ5IPKQCGYGGM5SOIFJ6ZYCAUYA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.384868031 (538h55m37.255951066s) [noise=true , address=77.90.31.16:4751 , piece_id=N3Z2LLMCIA2F42VXZA7IDRU4XEKGOICH6TRVERODU76MFFE4QSHA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.453563554 (538h55m37.324646599s) [noise=true , address=45.91.116.8:4064 , piece_id=22WJIPX4SM5NTIAW5ZRM2UKX32CTDBFCXPCCHRRWRCUCOF5CRE5Q, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.479454839 (538h55m37.350537884s) [noise=true , address=45.91.116.12:4257 , piece_id=ZI67LIAHZW4G357GWKDPTB5SMXYGNXZTTTDMNAETDS2B6SWVDAEQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.659710174 (538h55m37.530793219s) [noise=true , address=77.90.31.13:4635 , piece_id=IIOY5X2W3XW4C2OFEBRCABMYFBI24RRTYNUSKN6KJU7CTUW4KONQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.715039128 (538h55m37.586122163s) [noise=true , address=77.90.31.16:4760 , piece_id=XRI5EJFN7YYUG4QKLDGEFT6YAUHCUTTSXF735UDR6LZZBGMAZDJQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.977473406 (538h55m37.848556441s) [noise=true , address=77.90.31.19:4874 , piece_id=RHZUXUYMJU6RKQGSDZM5WDWIN5SQ7YIRSRATNNKT5GEMU2QH444A, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:38.016302745 (538h55m37.88738578s) [noise=true , address=145.79.227.30:4773 , piece_id=KUGJXZCNMUE7WSK3VLY4AUBSOT4H6IY4XB4NCBGAGCNUG7F7IODQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:38.016326982 (538h55m37.887410027s) [seg_pos={0 0}, upload=472631]: Segment reached optimal threshold of 14 pieces. Dec 28 02:24:38.016585353 (538h55m37.887668398s) [noise=true , address=145.79.227.30:4766 , piece_id=67M2DMPAVWIQKQSWUAG73D4E4NRRSG4XEVPOUVIJPEQOUV7A6VVA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1k6zdamuKPkBdzyNZNWx4FCmaTkv33PchcwvTRR8bgcETBs4K7): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016594450 (538h55m37.887677495s) [noise=true , address=145.79.227.30:4766 , piece_id=67M2DMPAVWIQKQSWUAG73D4E4NRRSG4XEVPOUVIJPEQOUV7A6VVA, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016627093 (538h55m37.887710148s) [noise=true , address=45.91.116.9:4281 , piece_id=JZLDW777CBX2BP7MLOQV4NFP3H4XJVCNQXTHNSUWPVDOJLPZK36A, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gxf7DpJZthyR6f1PExX56tJavjhPw9Gxb4h8mtmeYKW3t3tS1): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016635258 (538h55m37.887718293s) [noise=true , address=45.91.116.9:4281 , piece_id=JZLDW777CBX2BP7MLOQV4NFP3H4XJVCNQXTHNSUWPVDOJLPZK36A, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016871298 (538h55m37.887954363s) [noise=true , address=145.79.227.44:4222 , piece_id=CZ2CS62CTHQFAIE3S3JO2N3VJ4XIR5YJ2LMRVLLX7I4VM2VZVQVA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZKW35KvrsoYRefbYeUFZo8bh4XZYycNKGEvTFjzibezjXgj3e): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016880124 (538h55m37.887963169s) [noise=true , address=145.79.227.44:4222 , piece_id=CZ2CS62CTHQFAIE3S3JO2N3VJ4XIR5YJ2LMRVLLX7I4VM2VZVQVA, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016934839 (538h55m37.888017884s) [noise=true , address=145.79.227.40:4062 , piece_id=PW6QWIPQWVUETLC6DJ624G6IB4P6EM4SYB72NAMQ4V6NUKNLUYLQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016943425 (538h55m37.88802649s) [noise=true , address=145.79.227.40:4062 , piece_id=PW6QWIPQWVUETLC6DJ624G6IB4P6EM4SYB72NAMQ4V6NUKNLUYLQ, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016968252 (538h55m37.888051297s) [seg_pos={0 0}, upload=472631]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 02:24:38.017067822 (538h55m37.888150867s) [upload=472631]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 02:24:38.017074244 (538h55m37.888157289s) [upload=472631]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 02:24:38.304306667 (538h55m38.175389712s) [upload=472631]: Done uploading segments... Dec 28 02:24:38.304329641 (538h55m38.175412686s) [upload=472631]: Upload finished. err: <nil> Duration: 42.752688785s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [....X.........................X................X....X.......] duration: 5.000428363s trace: https://jaeger.dev.storj.io/trace/5efee46a94502508 FAIL (2025-12-28 02:47:29 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000453631s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 36 out of 60 is failed (threshold=3) [.X..X...XX..X.XXX.XXX.X.....XXX.XXXXXX.XXXX.XXXX..X..XXXXX.X] duration: 30.00378894s trace: https://jaeger.dev.storj.io/trace/74654f8f508593b6 FAIL (2025-12-28 02:46:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 02:46:29.789005880 (539h17m29.660088925s) [upload=472946]: Starting upload Dec 28 02:46:29.789059483 (539h17m29.660142518s) [upload=472946]: Done starting upload Dec 28 02:46:29.789257629 (539h17m29.660340674s) [upload=472946]: Uploading segments... Dec 28 02:46:29.789405711 (539h17m29.660488756s) [upload=472946]: Got next segment. Inline: false Dec 28 02:46:29.789445417 (539h17m29.660528462s) [upload=472946]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 02:46:29.789496123 (539h17m29.660579168s) [upload=472946]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 02:46:30.038832401 (539h17m29.909915456s) [seg_pos={0 0}, upload=472946]: Begin upload segment... Dec 28 02:46:30.038883788 (539h17m29.909966834s) [seg_pos={0 0}, upload=472946]: Done begin upload segment. Dec 28 02:46:30.038933734 (539h17m29.910016779s) [noise=true , address=145.79.227.13:4804 , piece_id=NHYMEBKSQPXVYXLQABFQH6NAKPPVP6KJ6UQRUN5A7PXV7IBPSD3Q, seg_pos={0 0}, upload=472946]: Uploading piece... Dec 28 02:46:30.038999029 (539h17m29.910082074s) [noise=true , address=145.79.227.46:4602 , piece_id=ROXBREZ6AVKZ5JXLV6MZGEJPGRODAMGEVCUSYUWSQHXOR6EOKKBQ, seg_pos={0 0}, upload=472946]: Uploading piece... Dec 28 02:46:30.039475265 (539h17m29.9105583s) [noise=true , address=145.79.227.16:4909 , piece_id=C32E7PQX6IJZF7XM5737T7AW3YFVQIQ347EPNJIB7FKEQJSZQCQQ, seg_pos={0 0}, upload=472946]: Uploading piece... Dec 28 02:46:30.039779274 (539h17m29.910862309s) [noise=true , address=145.79.227.12:4765 , piece_id=5HYMJBHEA7KEMLUP6PJ3H47QWARBEKS4KEH4WLRRC3URGF4RMXLA, seg_pos={0 0}, upload=472946]: Uploading piece... Dec 28 02:46:30.040071380 (539h17m29.911154465s) [noise=true , address=145.79.227.45:4229 , piece_id=JA4JTEMYQVD2EF3JN5VHQB32OVUQWXS6OTDAYZFNQCHB6CEYLLDQ, seg_pos={0 0}, upload=472946]: Uploading piece... Dec 28 02:46:30.040377974 (539h17m29.911461009s) [noise=true , address=77.90.31.14:4664 , piece_id=T63LC2SFIFZOOSUFKSVGUKK7WTXCTTD5FRRYRQXNEQHIFDWIKKNQ, seg_pos={0 0}, upload=472946]: Uploading piece... Dec 28 02:46:30.040662055 (539h17m29.91174509s) [noise=true , address=145.79.227.38:4979 , piece_id=GSBOOEZ2DVCAMVLOFETPXHYGQ232S6BGPOUWCNZMRCESBSI34IWA, seg_pos={0 0}, upload=472946]: Uploading piece... Dec 28 02:46:30.040919055 (539h17m29.91200209s) [noise=true , address=145.79.227.52:4446 , piece_id=6OOBFHCDDX54MAULKN2RBCXN2RJ7FJMS4VRPGZZLWVEHPU36S5HA, seg_pos={0 0}, upload=472946]: Uploading piece... Dec 28 02:46:30.041207123 (539h17m29.912290158s) [noise=true , address=145.79.227.22:4511 , piece_id=DLJ5RSC3HO3PZXY2NQ2JMEY6JPDA54FM2EIT7VEZ3W5APEZC5MTA, seg_pos={0 0}, upload=472946]: Uploading piece... Dec 28 02:46:30.042863326 (539h17m29.913946371s) [noise=true , address=145.79.227.11:4729 , piece_id=R7UKIJARGQA7KEXX6KMAQK2ZXACW747USY44Q324J5PUSWQ22YQQ, seg_pos={0 0}, upload=472946]: Uploading piece... Dec 28 02:46:30.044391977 (539h17m29.915475012s) [noise=true , address=45.91.116.14:4189 , piece_id=M6FRL5BLMKMQVDW7RSZFPZTZRK5NHHT2WQT6SHYEKVBMLPZSJZYQ, seg_pos={0 0}, upload=472946]: Uploading piece... Dec 28 02:46:30.045647928 (539h17m29.916730963s) [noise=true , address=77.90.31.21:4880 , piece_id=FZQ3VOONWCCMKCXFRTNNIRCMAJYP7L5RJV363FGJTGGKHRWKBMAA, seg_pos={0 0}, upload=472946]: Uploading piece... Dec 28 02:46:30.047192249 (539h17m29.918275284s) [noise=true , address=145.79.227.24:4583 , piece_id=BHSA27TOOWY63YQ2MS77KYPNSU4UYXVHOBZ4BI2NMENMHMCWOOQA, seg_pos={0 0}, upload=472946]: Uploading piece... Dec 28 02:46:30.048529375 (539h17m29.91961241s) [noise=true , address=145.79.227.18:4007 , piece_id=P6U3TLBOKUQDP3PABCW7OPCZEBVXEFOOQNEPX5HA2KSDP7YLQWJQ, seg_pos={0 0}, upload=472946]: Uploading piece... Dec 28 02:46:30.049986179 (539h17m29.921069214s) [noise=true , address=77.90.31.11:4553 , piece_id=2HVIMKYRB5ARLU724JR2XLTOH6QZI4KPM26O5YWA5TMWW3EN6AWQ, seg_pos={0 0}, upload=472946]: Uploading piece... Dec 28 02:46:30.051195521 (539h17m29.922278567s) [noise=true , address=145.79.227.3:4150 , piece_id=OWJ557UJBGTLMLSNBHHS7RTFVKWUBZE4UDPM6FQ4M5G4WSTKUGDQ, seg_pos={0 0}, upload=472946]: Uploading piece... Dec 28 02:46:31.674271789 (539h17m31.545354824s) [upload=472946]: Next returned nil segment Dec 28 02:46:31.674289744 (539h17m31.545372779s) [upload=472946]: Waiting for error group managing segments... Dec 28 02:46:32.061102268 (539h17m31.932185313s) [noise=true , address=145.79.227.45:4229 , piece_id=JA4JTEMYQVD2EF3JN5VHQB32OVUQWXS6OTDAYZFNQCHB6CEYLLDQ, seg_pos={0 0}, upload=472946]: Done uploading piece. err: <nil> Dec 28 02:46:32.062465483 (539h17m31.933548518s) [noise=true , address=145.79.227.52:4446 , piece_id=6OOBFHCDDX54MAULKN2RBCXN2RJ7FJMS4VRPGZZLWVEHPU36S5HA, seg_pos={0 0}, upload=472946]: Done uploading piece. err: <nil> Dec 28 02:46:32.070168262 (539h17m31.941251297s) [noise=true , address=145.79.227.24:4583 , piece_id=BHSA27TOOWY63YQ2MS77KYPNSU4UYXVHOBZ4BI2NMENMHMCWOOQA, seg_pos={0 0}, upload=472946]: Done uploading piece. err: <nil> Dec 28 02:46:32.076709527 (539h17m31.947792562s) [noise=true , address=145.79.227.13:4804 , piece_id=NHYMEBKSQPXVYXLQABFQH6NAKPPVP6KJ6UQRUN5A7PXV7IBPSD3Q, seg_pos={0 0}, upload=472946]: Done uploading piece. err: <nil> Dec 28 02:46:32.076726319 (539h17m31.947809364s) [noise=true , address=145.79.227.22:4511 , piece_id=DLJ5RSC3HO3PZXY2NQ2JMEY6JPDA54FM2EIT7VEZ3W5APEZC5MTA, seg_pos={0 0}, upload=472946]: Done uploading piece. err: <nil> Dec 28 02:46:32.079187045 (539h17m31.95027008s) [noise=true , address=145.79.227.46:4602 , piece_id=ROXBREZ6AVKZ5JXLV6MZGEJPGRODAMGEVCUSYUWSQHXOR6EOKKBQ, seg_pos={0 0}, upload=472946]: Done uploading piece. err: <nil> Dec 28 02:46:32.091216250 (539h17m31.962299295s) [noise=true , address=145.79.227.11:4729 , piece_id=R7UKIJARGQA7KEXX6KMAQK2ZXACW747USY44Q324J5PUSWQ22YQQ, seg_pos={0 0}, upload=472946]: Done uploading piece. err: <nil> Dec 28 02:46:32.357683136 (539h17m32.228766181s) [noise=true , address=145.79.227.12:4765 , piece_id=5HYMJBHEA7KEMLUP6PJ3H47QWARBEKS4KEH4WLRRC3URGF4RMXLA, seg_pos={0 0}, upload=472946]: Done uploading piece. err: <nil> Dec 28 02:46:32.357915459 (539h17m32.228998494s) [noise=true , address=77.90.31.11:4553 , piece_id=2HVIMKYRB5ARLU724JR2XLTOH6QZI4KPM26O5YWA5TMWW3EN6AWQ, seg_pos={0 0}, upload=472946]: Done uploading piece. err: <nil> Dec 28 02:46:32.544559059 (539h17m32.415642104s) [noise=true , address=77.90.31.14:4664 , piece_id=T63LC2SFIFZOOSUFKSVGUKK7WTXCTTD5FRRYRQXNEQHIFDWIKKNQ, seg_pos={0 0}, upload=472946]: Done uploading piece. err: <nil> Dec 28 02:46:32.909843170 (539h17m32.780926205s) [noise=true , address=77.90.31.21:4880 , piece_id=FZQ3VOONWCCMKCXFRTNNIRCMAJYP7L5RJV363FGJTGGKHRWKBMAA, seg_pos={0 0}, upload=472946]: Done uploading piece. err: <nil> Dec 28 02:46:33.085913538 (539h17m32.956996583s) [noise=true , address=145.79.227.16:4909 , piece_id=C32E7PQX6IJZF7XM5737T7AW3YFVQIQ347EPNJIB7FKEQJSZQCQQ, seg_pos={0 0}, upload=472946]: Done uploading piece. err: <nil> Dec 28 02:46:33.128197303 (539h17m32.999280338s) [noise=true , address=145.79.227.38:4979 , piece_id=GSBOOEZ2DVCAMVLOFETPXHYGQ232S6BGPOUWCNZMRCESBSI34IWA, seg_pos={0 0}, upload=472946]: Done uploading piece. err: <nil> Dec 28 02:46:33.128221960 (539h17m32.999305005s) [seg_pos={0 0}, upload=472946]: Segment reached optimal threshold of 13 pieces. Dec 28 02:46:33.128461657 (539h17m32.999544692s) [noise=true , address=145.79.227.3:4150 , piece_id=OWJ557UJBGTLMLSNBHHS7RTFVKWUBZE4UDPM6FQ4M5G4WSTKUGDQ, seg_pos={0 0}, upload=472946]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK): protocol: expected piece hash; context canceled; EOF Dec 28 02:46:33.128470844 (539h17m32.999553879s) [noise=true , address=145.79.227.3:4150 , piece_id=OWJ557UJBGTLMLSNBHHS7RTFVKWUBZE4UDPM6FQ4M5G4WSTKUGDQ, seg_pos={0 0}, upload=472946]: Long tail cancellation due to optimal threshold Dec 28 02:46:33.128546628 (539h17m32.999629663s) [noise=true , address=45.91.116.14:4189 , piece_id=M6FRL5BLMKMQVDW7RSZFPZTZRK5NHHT2WQT6SHYEKVBMLPZSJZYQ, seg_pos={0 0}, upload=472946]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 28 02:46:33.128561706 (539h17m32.999644741s) [noise=true , address=45.91.116.14:4189 , piece_id=M6FRL5BLMKMQVDW7RSZFPZTZRK5NHHT2WQT6SHYEKVBMLPZSJZYQ, seg_pos={0 0}, upload=472946]: Long tail cancellation due to optimal threshold Dec 28 02:46:33.128718595 (539h17m32.999801641s) [noise=true , address=145.79.227.18:4007 , piece_id=P6U3TLBOKUQDP3PABCW7OPCZEBVXEFOOQNEPX5HA2KSDP7YLQWJQ, seg_pos={0 0}, upload=472946]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp): protocol: write tcp 188.166.225.170:51822->145.79.227.18:4007: use of closed network connection; write tcp 188.166.225.170:51822->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51822->145.79.227.18:4007: use of closed network connection Dec 28 02:46:33.128725508 (539h17m32.999808553s) [noise=true , address=145.79.227.18:4007 , piece_id=P6U3TLBOKUQDP3PABCW7OPCZEBVXEFOOQNEPX5HA2KSDP7YLQWJQ, seg_pos={0 0}, upload=472946]: Long tail cancellation due to optimal threshold Dec 28 02:46:33.128742251 (539h17m32.999825286s) [seg_pos={0 0}, upload=472946]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 02:46:33.128822163 (539h17m32.999905198s) [upload=472946]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 02:46:33.128827232 (539h17m32.999910267s) [upload=472946]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 02:46:33.425963469 (539h17m33.297046504s) [upload=472946]: Done uploading segments... Dec 28 02:46:33.425990551 (539h17m33.297073586s) [upload=472946]: Upload finished. err: <nil> Duration: 30.003831843s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [........X...................X.....X.........................] duration: 47.899254252s trace: https://jaeger.dev.storj.io/trace/1822be52d200e5b5 FAIL (2025-12-28 02:46:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1ZAfYZ4wgpZVvatj6wtHona3hTaPcLxAmXC1DT2uKQyo8nLQQ7, Piece ID: 5MMQ44MXRT4XEL4X6WO54WRG2XYMX3AN3FNQ2H2XIUTKBXI3EEPA) context deadline exceeded; piecestore: (Node ID: 124guAM82oibYZsMho67Em6Ke6yEZXwT6LPMLLSZZBLqwoFGUkn, Piece ID: 22WJIPX4SM5NTIAW5ZRM2UKX32CTDBFCXPCCHRRWRCUCOF5CRE5Q) context deadline exceeded; piecestore: (Node ID: 1e6p1TD5CqUyWdZjkrxh2kRqMuyDiMUr9eimfeayDAKwhtBrZ8, Piece ID: XRI5EJFN7YYUG4QKLDGEFT6YAUHCUTTSXF735UDR6LZZBGMAZDJQ) context deadline exceeded ======LOGS====== 5.234848204 (538h55m35.105931239s) [noise=true , address=77.90.31.11:4572 , piece_id=5MMQ44MXRT4XEL4X6WO54WRG2XYMX3AN3FNQ2H2XIUTKBXI3EEPA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.235412057 (538h55m35.106495102s) [noise=true , address=145.79.227.22:4512 , piece_id=D46YLTIKGHG36DQ4YBKBSUDV4GGB4WVDWNOJS27IYYLN6AYZ5QFQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.235820946 (538h55m35.106903981s) [noise=true , address=77.90.31.19:4874 , piece_id=RHZUXUYMJU6RKQGSDZM5WDWIN5SQ7YIRSRATNNKT5GEMU2QH444A, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.236192192 (538h55m35.107275237s) [noise=true , address=77.90.31.16:4760 , piece_id=XRI5EJFN7YYUG4QKLDGEFT6YAUHCUTTSXF735UDR6LZZBGMAZDJQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.236564772 (538h55m35.107647817s) [noise=true , address=145.79.227.40:4062 , piece_id=PW6QWIPQWVUETLC6DJ624G6IB4P6EM4SYB72NAMQ4V6NUKNLUYLQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.236929766 (538h55m35.108012812s) [noise=true , address=64.49.13.139:4480 , piece_id=JAJY3F7IDFFMFKCK2QJV6W74RIESLRKBLXMN27Q3C45JWEPLBZLQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.237916846 (538h55m35.108999881s) [noise=true , address=145.79.227.34:4932 , piece_id=UPXSCQ45QDGPKNJXRFJ33EAA3QENVHRVQARU5TFJTEWICZNR253Q, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.238852948 (538h55m35.109935983s) [noise=true , address=45.91.116.12:4257 , piece_id=ZI67LIAHZW4G357GWKDPTB5SMXYGNXZTTTDMNAETDS2B6SWVDAEQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239152910 (538h55m35.110235945s) [noise=true , address=145.79.227.44:4222 , piece_id=CZ2CS62CTHQFAIE3S3JO2N3VJ4XIR5YJ2LMRVLLX7I4VM2VZVQVA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239409969 (538h55m35.110493004s) [noise=true , address=145.79.227.30:4773 , piece_id=KUGJXZCNMUE7WSK3VLY4AUBSOT4H6IY4XB4NCBGAGCNUG7F7IODQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239681136 (538h55m35.110764171s) [noise=true , address=77.90.31.13:4635 , piece_id=IIOY5X2W3XW4C2OFEBRCABMYFBI24RRTYNUSKN6KJU7CTUW4KONQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239954676 (538h55m35.111037711s) [noise=true , address=45.91.116.9:4281 , piece_id=JZLDW777CBX2BP7MLOQV4NFP3H4XJVCNQXTHNSUWPVDOJLPZK36A, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.240189914 (538h55m35.111272949s) [noise=true , address=145.79.227.35:4512 , piece_id=3E3VCCC45XVT5FHFDKXSTGGAGYBEML3BKU6JJTT33LCD2YMOMBZQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.240449147 (538h55m35.111532182s) [noise=true , address=45.91.116.8:4064 , piece_id=22WJIPX4SM5NTIAW5ZRM2UKX32CTDBFCXPCCHRRWRCUCOF5CRE5Q, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.240639640 (538h55m35.111722675s) [noise=true , address=145.79.227.32:4853 , piece_id=MBNCFDBQ4SZZXV4F7J5TRMOUYUQ5IPKQCGYGGM5SOIFJ6ZYCAUYA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.241137798 (538h55m35.112220833s) [noise=true , address=77.90.31.16:4751 , piece_id=N3Z2LLMCIA2F42VXZA7IDRU4XEKGOICH6TRVERODU76MFFE4QSHA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.241383797 (538h55m35.112466842s) [noise=true , address=77.90.31.18:4836 , piece_id=BSVEFI52EKN44CMINFUSZBNOH53IW5TXLVTF5IH6ZUTNII6RRLVA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:36.592880816 (538h55m36.463963871s) [upload=472631]: Next returned nil segment Dec 28 02:24:36.592901625 (538h55m36.46398467s) [upload=472631]: Waiting for error group managing segments... Dec 28 02:24:36.946933741 (538h55m36.818016786s) [noise=true , address=64.49.13.139:4480 , piece_id=JAJY3F7IDFFMFKCK2QJV6W74RIESLRKBLXMN27Q3C45JWEPLBZLQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.128189692 (538h55m36.999272737s) [noise=true , address=145.79.227.35:4512 , piece_id=3E3VCCC45XVT5FHFDKXSTGGAGYBEML3BKU6JJTT33LCD2YMOMBZQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.160760084 (538h55m37.031843119s) [noise=true , address=145.79.227.34:4932 , piece_id=UPXSCQ45QDGPKNJXRFJ33EAA3QENVHRVQARU5TFJTEWICZNR253Q, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.288838207 (538h55m37.159921252s) [noise=true , address=77.90.31.11:4572 , piece_id=5MMQ44MXRT4XEL4X6WO54WRG2XYMX3AN3FNQ2H2XIUTKBXI3EEPA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.319996310 (538h55m37.191079345s) [noise=true , address=145.79.227.22:4512 , piece_id=D46YLTIKGHG36DQ4YBKBSUDV4GGB4WVDWNOJS27IYYLN6AYZ5QFQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.320513273 (538h55m37.191596319s) [noise=true , address=77.90.31.18:4836 , piece_id=BSVEFI52EKN44CMINFUSZBNOH53IW5TXLVTF5IH6ZUTNII6RRLVA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.336522926 (538h55m37.207605971s) [noise=true , address=145.79.227.32:4853 , piece_id=MBNCFDBQ4SZZXV4F7J5TRMOUYUQ5IPKQCGYGGM5SOIFJ6ZYCAUYA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.384868031 (538h55m37.255951066s) [noise=true , address=77.90.31.16:4751 , piece_id=N3Z2LLMCIA2F42VXZA7IDRU4XEKGOICH6TRVERODU76MFFE4QSHA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.453563554 (538h55m37.324646599s) [noise=true , address=45.91.116.8:4064 , piece_id=22WJIPX4SM5NTIAW5ZRM2UKX32CTDBFCXPCCHRRWRCUCOF5CRE5Q, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.479454839 (538h55m37.350537884s) [noise=true , address=45.91.116.12:4257 , piece_id=ZI67LIAHZW4G357GWKDPTB5SMXYGNXZTTTDMNAETDS2B6SWVDAEQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.659710174 (538h55m37.530793219s) [noise=true , address=77.90.31.13:4635 , piece_id=IIOY5X2W3XW4C2OFEBRCABMYFBI24RRTYNUSKN6KJU7CTUW4KONQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.715039128 (538h55m37.586122163s) [noise=true , address=77.90.31.16:4760 , piece_id=XRI5EJFN7YYUG4QKLDGEFT6YAUHCUTTSXF735UDR6LZZBGMAZDJQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.977473406 (538h55m37.848556441s) [noise=true , address=77.90.31.19:4874 , piece_id=RHZUXUYMJU6RKQGSDZM5WDWIN5SQ7YIRSRATNNKT5GEMU2QH444A, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:38.016302745 (538h55m37.88738578s) [noise=true , address=145.79.227.30:4773 , piece_id=KUGJXZCNMUE7WSK3VLY4AUBSOT4H6IY4XB4NCBGAGCNUG7F7IODQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:38.016326982 (538h55m37.887410027s) [seg_pos={0 0}, upload=472631]: Segment reached optimal threshold of 14 pieces. Dec 28 02:24:38.016585353 (538h55m37.887668398s) [noise=true , address=145.79.227.30:4766 , piece_id=67M2DMPAVWIQKQSWUAG73D4E4NRRSG4XEVPOUVIJPEQOUV7A6VVA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1k6zdamuKPkBdzyNZNWx4FCmaTkv33PchcwvTRR8bgcETBs4K7): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016594450 (538h55m37.887677495s) [noise=true , address=145.79.227.30:4766 , piece_id=67M2DMPAVWIQKQSWUAG73D4E4NRRSG4XEVPOUVIJPEQOUV7A6VVA, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016627093 (538h55m37.887710148s) [noise=true , address=45.91.116.9:4281 , piece_id=JZLDW777CBX2BP7MLOQV4NFP3H4XJVCNQXTHNSUWPVDOJLPZK36A, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gxf7DpJZthyR6f1PExX56tJavjhPw9Gxb4h8mtmeYKW3t3tS1): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016635258 (538h55m37.887718293s) [noise=true , address=45.91.116.9:4281 , piece_id=JZLDW777CBX2BP7MLOQV4NFP3H4XJVCNQXTHNSUWPVDOJLPZK36A, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016871298 (538h55m37.887954363s) [noise=true , address=145.79.227.44:4222 , piece_id=CZ2CS62CTHQFAIE3S3JO2N3VJ4XIR5YJ2LMRVLLX7I4VM2VZVQVA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZKW35KvrsoYRefbYeUFZo8bh4XZYycNKGEvTFjzibezjXgj3e): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016880124 (538h55m37.887963169s) [noise=true , address=145.79.227.44:4222 , piece_id=CZ2CS62CTHQFAIE3S3JO2N3VJ4XIR5YJ2LMRVLLX7I4VM2VZVQVA, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016934839 (538h55m37.888017884s) [noise=true , address=145.79.227.40:4062 , piece_id=PW6QWIPQWVUETLC6DJ624G6IB4P6EM4SYB72NAMQ4V6NUKNLUYLQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016943425 (538h55m37.88802649s) [noise=true , address=145.79.227.40:4062 , piece_id=PW6QWIPQWVUETLC6DJ624G6IB4P6EM4SYB72NAMQ4V6NUKNLUYLQ, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016968252 (538h55m37.888051297s) [seg_pos={0 0}, upload=472631]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 02:24:38.017067822 (538h55m37.888150867s) [upload=472631]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 02:24:38.017074244 (538h55m37.888157289s) [upload=472631]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 02:24:38.304306667 (538h55m38.175389712s) [upload=472631]: Done uploading segments... Dec 28 02:24:38.304329641 (538h55m38.175412686s) [upload=472631]: Upload finished. err: <nil> Duration: 47.899282897s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [.....X.........................X................X....X......] duration: 5.000759595s trace: https://jaeger.dev.storj.io/trace/5efee46a94502508 FAIL (2025-12-28 02:46:22 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000786497s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)