Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 37.77361372s trace: https://jaeger.dev.storj.io/trace/665ea65f6e9ccc33 FAIL (2025-12-26 18:18:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== e=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789316979 (506h12m54.660400024s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789326507 (506h12m54.660409552s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789611339 (506h12m54.660694384s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789619275 (506h12m54.66070231s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789914006 (506h12m54.660997041s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789949443 (506h12m54.661032478s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790181566 (506h12m54.661264601s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790295632 (506h12m54.661378668s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790510872 (506h12m54.661593917s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790692980 (506h12m54.661776015s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790787379 (506h12m54.661870414s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790992970 (506h12m54.662076005s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791120053 (506h12m54.662203098s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791275218 (506h12m54.662358253s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791443709 (506h12m54.662526744s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:56.112313145 (506h12m55.98339619s) [upload=444191]: Next returned nil segment Dec 26 17:41:56.112330638 (506h12m55.983413683s) [upload=444191]: Waiting for error group managing segments... Dec 26 17:41:56.485322720 (506h12m56.356405765s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.485637079 (506h12m56.356720124s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.666807539 (506h12m56.537890584s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.678753987 (506h12m56.549837032s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.687788039 (506h12m56.558871074s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.694318023 (506h12m56.565401058s) [noise=true , address=145.79.227.7:4326 , piece_id=DMQ5RPI46X2U7TJTS42ZHQEF32BDJRKJJO5HWOYAPNS6VWVG5B5Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.716581187 (506h12m56.587664232s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.972183007 (506h12m56.843266042s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.011917999 (506h12m56.883001034s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.054168622 (506h12m56.925251667s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.114312777 (506h12m56.985395812s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.499846250 (506h12m57.370929285s) [noise=true , address=145.79.227.34:4919 , piece_id=YRWHW3BJVEGVJRSA6RJB3T7VJQSDTGKEZQKRICFIV2G3T7AN4MIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.612649066 (506h12m57.483732101s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649413733 (506h12m57.520496768s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649440282 (506h12m57.520523328s) [seg_pos={0 0}, upload=444191]: Segment reached optimal threshold of 14 pieces. Dec 26 17:41:57.649674729 (506h12m57.520757774s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G7ty71tPwe7cZSBNK9u74DFUCGomnxDsNvAif4C3SbP7HWwfo): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649684087 (506h12m57.520767142s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649747708 (506h12m57.520830763s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v6zXZuMmyL2MU8qb3q6gg1iwTYvndjzMjVpv8btEpeX9Lnqmu): protocol: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection Dec 26 17:41:57.649785219 (506h12m57.520868274s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649899828 (506h12m57.520982903s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649908775 (506h12m57.52099182s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650074641 (506h12m57.521157696s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.650083287 (506h12m57.521166342s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650132821 (506h12m57.521215866s) [seg_pos={0 0}, upload=444191]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 17:41:57.650220138 (506h12m57.521303183s) [upload=444191]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:41:57.650227171 (506h12m57.521310206s) [upload=444191]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:41:57.958890870 (506h12m57.829973915s) [upload=444191]: Done uploading segments... Dec 26 17:41:57.958911620 (506h12m57.829994665s) [upload=444191]: Upload finished. err: <nil> Duration: 37.773638628s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [XX......X.XXX.....XX.X.XXX.........X..X.....X...XX.........X] duration: 30.003423559s trace: https://jaeger.dev.storj.io/trace/3b66cb5bbee176a1 FAIL (2025-12-26 18:18:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: HTIDJEQFTYA5OEDTOSXUUV7AVYISWN5BOEP6GD7KD7J4ON3CBPFQ) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: 6U4LPZYNWILR72DVNPKP22VKKNOZPWMY73OQAIEGRDVY6PFNGV7A) context deadline exceeded ======LOGS====== gments... Dec 26 18:17:44.233899075 (506h48m44.104982321s) [upload=444720]: Got next segment. Inline: false Dec 26 18:17:44.234055674 (506h48m44.105138719s) [upload=444720]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 18:17:44.234062016 (506h48m44.105145051s) [upload=444720]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 18:17:44.482104382 (506h48m44.353187437s) [seg_pos={0 0}, upload=444720]: Begin upload segment... Dec 26 18:17:44.482168003 (506h48m44.353251038s) [seg_pos={0 0}, upload=444720]: Done begin upload segment. Dec 26 18:17:44.482244459 (506h48m44.353327494s) [noise=true , address=45.91.116.14:4190 , piece_id=HTIDJEQFTYA5OEDTOSXUUV7AVYISWN5BOEP6GD7KD7J4ON3CBPFQ, seg_pos={0 0}, upload=444720]: Uploading piece... Dec 26 18:17:44.482284294 (506h48m44.35336734s) [noise=true , address=45.91.116.6:4973 , piece_id=6U4LPZYNWILR72DVNPKP22VKKNOZPWMY73OQAIEGRDVY6PFNGV7A, seg_pos={0 0}, upload=444720]: Uploading piece... Dec 26 18:17:44.483125356 (506h48m44.354208401s) [noise=true , address=145.79.227.7:4294 , piece_id=HMLTVW3MUFCAR6BG74SVDTNSICW72VYZX4J2YNIH4OAZA6M2MYNA, seg_pos={0 0}, upload=444720]: Uploading piece... Dec 26 18:17:44.483466666 (506h48m44.354549701s) [noise=true , address=145.79.227.5:4222 , piece_id=X4ES5FPY3MJKP2PSV6PLHT6JPBTCPHSEIISJ3VWXBWOL6FSYIWEQ, seg_pos={0 0}, upload=444720]: Uploading piece... Dec 26 18:17:44.483697005 (506h48m44.35478005s) [noise=true , address=145.79.227.11:4731 , piece_id=MFVIRTTK4HMMEVYLEQLNJI4GHPHDSKNCKES5265OOKHP35V2BZ4Q, seg_pos={0 0}, upload=444720]: Uploading piece... Dec 26 18:17:44.483845718 (506h48m44.354928753s) [noise=true , address=45.91.116.5:4298 , piece_id=P7URE5INZE6MEYA7OB2EWBMNV264UFRFWHECXGTDCNQWNDEZLUIQ, seg_pos={0 0}, upload=444720]: Uploading piece... Dec 26 18:17:44.484134077 (506h48m44.355217112s) [noise=true , address=45.91.116.9:4264 , piece_id=LH2TMZXETF4QZ2PD7ZHAYZFHUKWXA4XOODPD4PINBKASGIIWEK4Q, seg_pos={0 0}, upload=444720]: Uploading piece... Dec 26 18:17:44.484298069 (506h48m44.355381104s) [noise=true , address=145.79.227.52:4446 , piece_id=JQV34TEV346A4PWGH3CDKD2GHAP6KNVUHJ3DZJD6C34G2I23EBFA, seg_pos={0 0}, upload=444720]: Uploading piece... Dec 26 18:17:44.484400655 (506h48m44.3554837s) [noise=true , address=145.79.227.49:4337 , piece_id=SXX4XTTPZFHVLPQ6MT2S6KDMQBTSUDR35Q4WZMVJSTGHR5S47VFQ, seg_pos={0 0}, upload=444720]: Uploading piece... Dec 26 18:17:44.484691678 (506h48m44.355774713s) [noise=true , address=145.79.227.37:4566 , piece_id=2UGNFWUMBFESEFINRGNYGP5BSZ2AGBP4V5KGMYJXIWISYGFSY7XQ, seg_pos={0 0}, upload=444720]: Uploading piece... Dec 26 18:17:44.484819723 (506h48m44.355902768s) [noise=true , address=145.79.227.29:4942 , piece_id=UVESXQQA4WWA6BLWFQYKKXZSB72OS5S7VOWLOTKSGC64CPMVRVCA, seg_pos={0 0}, upload=444720]: Uploading piece... Dec 26 18:17:44.484986671 (506h48m44.356069706s) [noise=true , address=145.79.227.36:4517 , piece_id=GNQIP445TCJH3DRP34QUKON66OCCGA54XZ67ILL6G5WXFHBV54UQ, seg_pos={0 0}, upload=444720]: Uploading piece... Dec 26 18:17:44.485248549 (506h48m44.356331614s) [noise=true , address=145.79.227.22:4511 , piece_id=JEVHL5N6JFIIZJGIMMKPGGW5O4FRYU3TLL6SGA2DXYUU7IEWFSXA, seg_pos={0 0}, upload=444720]: Uploading piece... Dec 26 18:17:44.485634844 (506h48m44.356717879s) [noise=true , address=145.79.227.4:4187 , piece_id=Z4HX7IC2HBZWCEC73QWCU5K57MP46MIFX2YJH4Q52AYOCBIGUT2A, seg_pos={0 0}, upload=444720]: Uploading piece... Dec 26 18:17:44.486143163 (506h48m44.357226208s) [noise=true , address=145.79.227.31:4799 , piece_id=7VDEU4P6RWTT2SL6GFZGAYJYVGMGNEUZBX5ELRGQQME2AVI4EGYA, seg_pos={0 0}, upload=444720]: Uploading piece... Dec 26 18:17:44.486483892 (506h48m44.357566937s) [noise=true , address=145.79.227.47:4266 , piece_id=LMYKFSRNB4UBLK2Z7FRWKCRUIP3PTXNMSZA4VIUUHOIIMVCZEHHA, seg_pos={0 0}, upload=444720]: Uploading piece... Dec 26 18:17:46.098958535 (506h48m45.970041571s) [upload=444720]: Next returned nil segment Dec 26 18:17:46.098973043 (506h48m45.970056078s) [upload=444720]: Waiting for error group managing segments... Dec 26 18:17:46.482233768 (506h48m46.353316813s) [noise=true , address=145.79.227.36:4517 , piece_id=GNQIP445TCJH3DRP34QUKON66OCCGA54XZ67ILL6G5WXFHBV54UQ, seg_pos={0 0}, upload=444720]: Done uploading piece. err: <nil> Dec 26 18:17:46.484253554 (506h48m46.35533662s) [noise=true , address=145.79.227.22:4511 , piece_id=JEVHL5N6JFIIZJGIMMKPGGW5O4FRYU3TLL6SGA2DXYUU7IEWFSXA, seg_pos={0 0}, upload=444720]: Done uploading piece. err: <nil> Dec 26 18:17:46.484520213 (506h48m46.355603258s) [noise=true , address=145.79.227.49:4337 , piece_id=SXX4XTTPZFHVLPQ6MT2S6KDMQBTSUDR35Q4WZMVJSTGHR5S47VFQ, seg_pos={0 0}, upload=444720]: Done uploading piece. err: <nil> Dec 26 18:17:46.485024783 (506h48m46.356107838s) [noise=true , address=145.79.227.29:4942 , piece_id=UVESXQQA4WWA6BLWFQYKKXZSB72OS5S7VOWLOTKSGC64CPMVRVCA, seg_pos={0 0}, upload=444720]: Done uploading piece. err: <nil> Dec 26 18:17:46.490941920 (506h48m46.362024955s) [noise=true , address=145.79.227.37:4566 , piece_id=2UGNFWUMBFESEFINRGNYGP5BSZ2AGBP4V5KGMYJXIWISYGFSY7XQ, seg_pos={0 0}, upload=444720]: Done uploading piece. err: <nil> Dec 26 18:17:46.491343145 (506h48m46.3624262s) [noise=true , address=145.79.227.52:4446 , piece_id=JQV34TEV346A4PWGH3CDKD2GHAP6KNVUHJ3DZJD6C34G2I23EBFA, seg_pos={0 0}, upload=444720]: Done uploading piece. err: <nil> Dec 26 18:17:46.491517597 (506h48m46.362600642s) [noise=true , address=145.79.227.11:4731 , piece_id=MFVIRTTK4HMMEVYLEQLNJI4GHPHDSKNCKES5265OOKHP35V2BZ4Q, seg_pos={0 0}, upload=444720]: Done uploading piece. err: <nil> Dec 26 18:17:46.505557661 (506h48m46.376640706s) [noise=true , address=145.79.227.47:4266 , piece_id=LMYKFSRNB4UBLK2Z7FRWKCRUIP3PTXNMSZA4VIUUHOIIMVCZEHHA, seg_pos={0 0}, upload=444720]: Done uploading piece. err: <nil> Dec 26 18:17:46.511497923 (506h48m46.382580968s) [noise=true , address=145.79.227.31:4799 , piece_id=7VDEU4P6RWTT2SL6GFZGAYJYVGMGNEUZBX5ELRGQQME2AVI4EGYA, seg_pos={0 0}, upload=444720]: Done uploading piece. err: <nil> Dec 26 18:17:46.748964861 (506h48m46.620047906s) [noise=true , address=145.79.227.7:4294 , piece_id=HMLTVW3MUFCAR6BG74SVDTNSICW72VYZX4J2YNIH4OAZA6M2MYNA, seg_pos={0 0}, upload=444720]: Done uploading piece. err: <nil> Dec 26 18:17:47.018364295 (506h48m46.88944734s) [noise=true , address=45.91.116.14:4190 , piece_id=HTIDJEQFTYA5OEDTOSXUUV7AVYISWN5BOEP6GD7KD7J4ON3CBPFQ, seg_pos={0 0}, upload=444720]: Done uploading piece. err: <nil> Dec 26 18:17:47.520456737 (506h48m47.391539782s) [noise=true , address=45.91.116.6:4973 , piece_id=6U4LPZYNWILR72DVNPKP22VKKNOZPWMY73OQAIEGRDVY6PFNGV7A, seg_pos={0 0}, upload=444720]: Done uploading piece. err: <nil> Dec 26 18:17:47.646618581 (506h48m47.517701626s) [noise=true , address=45.91.116.5:4298 , piece_id=P7URE5INZE6MEYA7OB2EWBMNV264UFRFWHECXGTDCNQWNDEZLUIQ, seg_pos={0 0}, upload=444720]: Done uploading piece. err: <nil> Dec 26 18:17:47.646719584 (506h48m47.517802619s) [seg_pos={0 0}, upload=444720]: Segment reached optimal threshold of 13 pieces. Dec 26 18:17:47.646919054 (506h48m47.518002089s) [noise=true , address=145.79.227.4:4187 , piece_id=Z4HX7IC2HBZWCEC73QWCU5K57MP46MIFX2YJH4Q52AYOCBIGUT2A, seg_pos={0 0}, upload=444720]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x): protocol: write tcp 188.166.225.170:43536->145.79.227.4:4187: use of closed network connection; write tcp 188.166.225.170:43536->145.79.227.4:4187: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43536->145.79.227.4:4187: use of closed network connection Dec 26 18:17:47.646927289 (506h48m47.518010334s) [noise=true , address=145.79.227.4:4187 , piece_id=Z4HX7IC2HBZWCEC73QWCU5K57MP46MIFX2YJH4Q52AYOCBIGUT2A, seg_pos={0 0}, upload=444720]: Long tail cancellation due to optimal threshold Dec 26 18:17:47.646940303 (506h48m47.518023339s) [noise=true , address=145.79.227.5:4222 , piece_id=X4ES5FPY3MJKP2PSV6PLHT6JPBTCPHSEIISJ3VWXBWOL6FSYIWEQ, seg_pos={0 0}, upload=444720]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1): protocol: write tcp 188.166.225.170:49674->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:49674->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49674->145.79.227.5:4222: use of closed network connection Dec 26 18:17:47.646950263 (506h48m47.518033298s) [noise=true , address=145.79.227.5:4222 , piece_id=X4ES5FPY3MJKP2PSV6PLHT6JPBTCPHSEIISJ3VWXBWOL6FSYIWEQ, seg_pos={0 0}, upload=444720]: Long tail cancellation due to optimal threshold Dec 26 18:17:47.647272486 (506h48m47.518355522s) [noise=true , address=45.91.116.9:4264 , piece_id=LH2TMZXETF4QZ2PD7ZHAYZFHUKWXA4XOODPD4PINBKASGIIWEK4Q, seg_pos={0 0}, upload=444720]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 26 18:17:47.647280201 (506h48m47.518363236s) [noise=true , address=45.91.116.9:4264 , piece_id=LH2TMZXETF4QZ2PD7ZHAYZFHUKWXA4XOODPD4PINBKASGIIWEK4Q, seg_pos={0 0}, upload=444720]: Long tail cancellation due to optimal threshold Dec 26 18:17:47.647299127 (506h48m47.518382162s) [seg_pos={0 0}, upload=444720]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 18:17:47.647384159 (506h48m47.518467194s) [upload=444720]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 18:17:47.647390200 (506h48m47.518473235s) [upload=444720]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 18:17:47.945659407 (506h48m47.816742442s) [upload=444720]: Done uploading segments... Dec 26 18:17:47.945688893 (506h48m47.816771928s) [upload=444720]: Upload finished. err: <nil> Duration: 30.003448316s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 27.967344048s trace: https://jaeger.dev.storj.io/trace/665ea65f6e9ccc33 FAIL (2025-12-26 18:17:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== e=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789316979 (506h12m54.660400024s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789326507 (506h12m54.660409552s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789611339 (506h12m54.660694384s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789619275 (506h12m54.66070231s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789914006 (506h12m54.660997041s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789949443 (506h12m54.661032478s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790181566 (506h12m54.661264601s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790295632 (506h12m54.661378668s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790510872 (506h12m54.661593917s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790692980 (506h12m54.661776015s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790787379 (506h12m54.661870414s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790992970 (506h12m54.662076005s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791120053 (506h12m54.662203098s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791275218 (506h12m54.662358253s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791443709 (506h12m54.662526744s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:56.112313145 (506h12m55.98339619s) [upload=444191]: Next returned nil segment Dec 26 17:41:56.112330638 (506h12m55.983413683s) [upload=444191]: Waiting for error group managing segments... Dec 26 17:41:56.485322720 (506h12m56.356405765s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.485637079 (506h12m56.356720124s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.666807539 (506h12m56.537890584s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.678753987 (506h12m56.549837032s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.687788039 (506h12m56.558871074s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.694318023 (506h12m56.565401058s) [noise=true , address=145.79.227.7:4326 , piece_id=DMQ5RPI46X2U7TJTS42ZHQEF32BDJRKJJO5HWOYAPNS6VWVG5B5Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.716581187 (506h12m56.587664232s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.972183007 (506h12m56.843266042s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.011917999 (506h12m56.883001034s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.054168622 (506h12m56.925251667s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.114312777 (506h12m56.985395812s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.499846250 (506h12m57.370929285s) [noise=true , address=145.79.227.34:4919 , piece_id=YRWHW3BJVEGVJRSA6RJB3T7VJQSDTGKEZQKRICFIV2G3T7AN4MIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.612649066 (506h12m57.483732101s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649413733 (506h12m57.520496768s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649440282 (506h12m57.520523328s) [seg_pos={0 0}, upload=444191]: Segment reached optimal threshold of 14 pieces. Dec 26 17:41:57.649674729 (506h12m57.520757774s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G7ty71tPwe7cZSBNK9u74DFUCGomnxDsNvAif4C3SbP7HWwfo): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649684087 (506h12m57.520767142s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649747708 (506h12m57.520830763s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v6zXZuMmyL2MU8qb3q6gg1iwTYvndjzMjVpv8btEpeX9Lnqmu): protocol: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection Dec 26 17:41:57.649785219 (506h12m57.520868274s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649899828 (506h12m57.520982903s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649908775 (506h12m57.52099182s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650074641 (506h12m57.521157696s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.650083287 (506h12m57.521166342s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650132821 (506h12m57.521215866s) [seg_pos={0 0}, upload=444191]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 17:41:57.650220138 (506h12m57.521303183s) [upload=444191]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:41:57.650227171 (506h12m57.521310206s) [upload=444191]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:41:57.958890870 (506h12m57.829973915s) [upload=444191]: Done uploading segments... Dec 26 17:41:57.958911620 (506h12m57.829994665s) [upload=444191]: Upload finished. err: <nil> Duration: 27.967361531s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [.XX......X.XXX.....XX.X.XXX.........X..X.....X...XX.........] duration: 10.029333968s trace: https://jaeger.dev.storj.io/trace/3992379534f794fb FAIL (2025-12-26 18:17:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: LLOE3CT5REHRQEWDHHSGDBGJPWC35OQMVR4MEZB4FDOZ4AWGFGHA) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: DLDNZ7WRHTHESIBXZBLTGAQNLJRAJBDKPZHSCQZW6GYUNLYIONDA) context deadline exceeded ======LOGS====== Starting upload Dec 26 18:11:00.104273846 (506h41m59.975356892s) [upload=444620]: Done starting upload Dec 26 18:11:00.104464159 (506h41m59.975547194s) [upload=444620]: Uploading segments... Dec 26 18:11:00.104513854 (506h41m59.975596889s) [upload=444620]: Got next segment. Inline: false Dec 26 18:11:00.104544612 (506h41m59.975627647s) [upload=444620]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 18:11:00.104549912 (506h41m59.975632957s) [upload=444620]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 18:11:00.350198911 (506h42m0.221281966s) [seg_pos={0 0}, upload=444620]: Begin upload segment... Dec 26 18:11:00.350254707 (506h42m0.221337752s) [seg_pos={0 0}, upload=444620]: Done begin upload segment. Dec 26 18:11:00.350321845 (506h42m0.22140489s) [noise=true , address=145.79.227.11:4731 , piece_id=NJN4CQGBUBFZWEUDP3KBVU2AVACTXCQ2HE6YJ7W7MKC4E3D4LXIQ, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.350328608 (506h42m0.221411653s) [noise=true , address=45.91.116.7:4009 , piece_id=LLOE3CT5REHRQEWDHHSGDBGJPWC35OQMVR4MEZB4FDOZ4AWGFGHA, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.350959219 (506h42m0.222042264s) [noise=true , address=145.79.227.8:4330 , piece_id=PDB6ALO5ZPYSMSEQHF3NPY3VPWG7XGCXPXA22HWBFRGA5YYA3G5A, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.351321138 (506h42m0.222404173s) [noise=true , address=45.91.116.5:4297 , piece_id=APSDM2AGSPHTXZ36DROJOVMPFVPCHUHY5UBTILU32LERBQQOWK2Q, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.351684470 (506h42m0.222767515s) [noise=true , address=145.79.227.42:4121 , piece_id=H56KQOPDWK557ZZJCWBH7YLWWL3DK24BO3654Q4EWQ6ALZ5AU2UQ, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.352040457 (506h42m0.223123492s) [noise=true , address=45.91.116.8:4047 , piece_id=K5Q2OWU2L6QI5Y77KA7OS5UYOGDYUZ6B6XWURJMSUQXCEUAFC5UA, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.352047270 (506h42m0.223130305s) [noise=true , address=145.79.227.49:4337 , piece_id=BUOWQXWRHHZCKHJPC3D6KNYPHLNNVNSYZCMLG4JICHTJZVFVB7VQ, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.352385835 (506h42m0.22346888s) [noise=true , address=145.79.227.47:4267 , piece_id=XRQQF5K3O3A4V5545MZA2YG2N6EVRG4KIM56A6W6IRRWISTU6WAQ, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.352443936 (506h42m0.223526971s) [noise=true , address=45.91.116.10:4083 , piece_id=OBCQWLK3PCTYKVBP6DPVWXVQPJZVDVHMXS7333U5V5ROKQ2KQEEQ, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.352694222 (506h42m0.223777257s) [noise=true , address=145.79.227.22:4511 , piece_id=S6ZS6SOLNLZXGX7H4CUFN565Y4AYUIZCC2CXBJS6L3CSVB4OC3XQ, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.352808039 (506h42m0.223891084s) [noise=true , address=145.79.227.36:4517 , piece_id=ZL6T3RQT5XNQURM6JLOZ2YGAXNGLYGN5FYHO47IKWPPXZFCUEVBA, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.353054909 (506h42m0.224137944s) [noise=true , address=145.79.227.5:4223 , piece_id=5SXLARFJKURKC7AIF765Z7MWPHOEDHOFMNLCHMQBZUYBOC6AVNOQ, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.353570130 (506h42m0.224653175s) [noise=true , address=145.79.227.38:4978 , piece_id=6UFEMERR2PHRITTQKOGZV3CZCYLHONY36POKRX4TJ3IG3LB4RLQQ, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.354135146 (506h42m0.225218181s) [noise=true , address=45.91.116.11:4119 , piece_id=GA4DRGZTVG4TKE6MPO2YWCWERELOSSF2QZYG5L5MG7TEG5C6TFVA, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.354603829 (506h42m0.225686874s) [noise=true , address=145.79.227.50:4374 , piece_id=TMHRSFYNG6WI6VZF6CER7JWWG4PAAMSJ22QP6RAI64CJO4GAHNGQ, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.355020522 (506h42m0.226103567s) [noise=true , address=45.91.116.6:4974 , piece_id=DLDNZ7WRHTHESIBXZBLTGAQNLJRAJBDKPZHSCQZW6GYUNLYIONDA, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:02.032410415 (506h42m1.90349346s) [upload=444620]: Next returned nil segment Dec 26 18:11:02.032430152 (506h42m1.903513197s) [upload=444620]: Waiting for error group managing segments... Dec 26 18:11:02.348324681 (506h42m2.219407716s) [noise=true , address=145.79.227.22:4511 , piece_id=S6ZS6SOLNLZXGX7H4CUFN565Y4AYUIZCC2CXBJS6L3CSVB4OC3XQ, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.348549048 (506h42m2.219632083s) [noise=true , address=145.79.227.38:4978 , piece_id=6UFEMERR2PHRITTQKOGZV3CZCYLHONY36POKRX4TJ3IG3LB4RLQQ, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.351071442 (506h42m2.222154487s) [noise=true , address=145.79.227.8:4330 , piece_id=PDB6ALO5ZPYSMSEQHF3NPY3VPWG7XGCXPXA22HWBFRGA5YYA3G5A, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.355344347 (506h42m2.226427392s) [noise=true , address=145.79.227.50:4374 , piece_id=TMHRSFYNG6WI6VZF6CER7JWWG4PAAMSJ22QP6RAI64CJO4GAHNGQ, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.359503999 (506h42m2.230587044s) [noise=true , address=145.79.227.49:4337 , piece_id=BUOWQXWRHHZCKHJPC3D6KNYPHLNNVNSYZCMLG4JICHTJZVFVB7VQ, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.361316750 (506h42m2.232399785s) [noise=true , address=145.79.227.5:4223 , piece_id=5SXLARFJKURKC7AIF765Z7MWPHOEDHOFMNLCHMQBZUYBOC6AVNOQ, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.373894200 (506h42m2.244977235s) [noise=true , address=145.79.227.47:4267 , piece_id=XRQQF5K3O3A4V5545MZA2YG2N6EVRG4KIM56A6W6IRRWISTU6WAQ, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.376201063 (506h42m2.247284108s) [noise=true , address=145.79.227.36:4517 , piece_id=ZL6T3RQT5XNQURM6JLOZ2YGAXNGLYGN5FYHO47IKWPPXZFCUEVBA, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.380841749 (506h42m2.251924804s) [noise=true , address=145.79.227.11:4731 , piece_id=NJN4CQGBUBFZWEUDP3KBVU2AVACTXCQ2HE6YJ7W7MKC4E3D4LXIQ, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.388670316 (506h42m2.259753351s) [noise=true , address=145.79.227.42:4121 , piece_id=H56KQOPDWK557ZZJCWBH7YLWWL3DK24BO3654Q4EWQ6ALZ5AU2UQ, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.586101119 (506h42m2.457184164s) [noise=true , address=45.91.116.7:4009 , piece_id=LLOE3CT5REHRQEWDHHSGDBGJPWC35OQMVR4MEZB4FDOZ4AWGFGHA, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.597900427 (506h42m2.468983462s) [noise=true , address=45.91.116.10:4083 , piece_id=OBCQWLK3PCTYKVBP6DPVWXVQPJZVDVHMXS7333U5V5ROKQ2KQEEQ, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.711289730 (506h42m2.582372765s) [noise=true , address=45.91.116.6:4974 , piece_id=DLDNZ7WRHTHESIBXZBLTGAQNLJRAJBDKPZHSCQZW6GYUNLYIONDA, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.711310149 (506h42m2.582393184s) [seg_pos={0 0}, upload=444620]: Segment reached optimal threshold of 13 pieces. Dec 26 18:11:02.711579893 (506h42m2.582662938s) [noise=true , address=45.91.116.11:4119 , piece_id=GA4DRGZTVG4TKE6MPO2YWCWERELOSSF2QZYG5L5MG7TEG5C6TFVA, seg_pos={0 0}, upload=444620]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: read tcp 188.166.225.170:54212->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 18:11:02.711589541 (506h42m2.582672576s) [noise=true , address=45.91.116.11:4119 , piece_id=GA4DRGZTVG4TKE6MPO2YWCWERELOSSF2QZYG5L5MG7TEG5C6TFVA, seg_pos={0 0}, upload=444620]: Long tail cancellation due to optimal threshold Dec 26 18:11:02.711667138 (506h42m2.582750184s) [noise=true , address=45.91.116.8:4047 , piece_id=K5Q2OWU2L6QI5Y77KA7OS5UYOGDYUZ6B6XWURJMSUQXCEUAFC5UA, seg_pos={0 0}, upload=444620]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 26 18:11:02.711691024 (506h42m2.582774069s) [noise=true , address=45.91.116.8:4047 , piece_id=K5Q2OWU2L6QI5Y77KA7OS5UYOGDYUZ6B6XWURJMSUQXCEUAFC5UA, seg_pos={0 0}, upload=444620]: Long tail cancellation due to optimal threshold Dec 26 18:11:02.711887848 (506h42m2.582970913s) [noise=true , address=45.91.116.5:4297 , piece_id=APSDM2AGSPHTXZ36DROJOVMPFVPCHUHY5UBTILU32LERBQQOWK2Q, seg_pos={0 0}, upload=444620]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: write tcp 188.166.225.170:38572->45.91.116.5:4297: use of closed network connection; write tcp 188.166.225.170:38572->45.91.116.5:4297: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38572->45.91.116.5:4297: use of closed network connection Dec 26 18:11:02.711896776 (506h42m2.582979821s) [noise=true , address=45.91.116.5:4297 , piece_id=APSDM2AGSPHTXZ36DROJOVMPFVPCHUHY5UBTILU32LERBQQOWK2Q, seg_pos={0 0}, upload=444620]: Long tail cancellation due to optimal threshold Dec 26 18:11:02.711919389 (506h42m2.583002434s) [seg_pos={0 0}, upload=444620]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 18:11:02.712023507 (506h42m2.583106542s) [upload=444620]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 18:11:02.712031572 (506h42m2.583114607s) [upload=444620]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 18:11:03.015981797 (506h42m2.887064842s) [upload=444620]: Done uploading segments... Dec 26 18:11:03.016006694 (506h42m2.887089739s) [upload=444620]: Upload finished. err: <nil> Duration: 10.029361169s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 40.733296134s trace: https://jaeger.dev.storj.io/trace/665ea65f6e9ccc33 FAIL (2025-12-26 18:16:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== e=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789316979 (506h12m54.660400024s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789326507 (506h12m54.660409552s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789611339 (506h12m54.660694384s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789619275 (506h12m54.66070231s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789914006 (506h12m54.660997041s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789949443 (506h12m54.661032478s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790181566 (506h12m54.661264601s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790295632 (506h12m54.661378668s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790510872 (506h12m54.661593917s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790692980 (506h12m54.661776015s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790787379 (506h12m54.661870414s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790992970 (506h12m54.662076005s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791120053 (506h12m54.662203098s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791275218 (506h12m54.662358253s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791443709 (506h12m54.662526744s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:56.112313145 (506h12m55.98339619s) [upload=444191]: Next returned nil segment Dec 26 17:41:56.112330638 (506h12m55.983413683s) [upload=444191]: Waiting for error group managing segments... Dec 26 17:41:56.485322720 (506h12m56.356405765s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.485637079 (506h12m56.356720124s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.666807539 (506h12m56.537890584s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.678753987 (506h12m56.549837032s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.687788039 (506h12m56.558871074s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.694318023 (506h12m56.565401058s) [noise=true , address=145.79.227.7:4326 , piece_id=DMQ5RPI46X2U7TJTS42ZHQEF32BDJRKJJO5HWOYAPNS6VWVG5B5Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.716581187 (506h12m56.587664232s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.972183007 (506h12m56.843266042s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.011917999 (506h12m56.883001034s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.054168622 (506h12m56.925251667s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.114312777 (506h12m56.985395812s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.499846250 (506h12m57.370929285s) [noise=true , address=145.79.227.34:4919 , piece_id=YRWHW3BJVEGVJRSA6RJB3T7VJQSDTGKEZQKRICFIV2G3T7AN4MIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.612649066 (506h12m57.483732101s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649413733 (506h12m57.520496768s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649440282 (506h12m57.520523328s) [seg_pos={0 0}, upload=444191]: Segment reached optimal threshold of 14 pieces. Dec 26 17:41:57.649674729 (506h12m57.520757774s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G7ty71tPwe7cZSBNK9u74DFUCGomnxDsNvAif4C3SbP7HWwfo): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649684087 (506h12m57.520767142s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649747708 (506h12m57.520830763s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v6zXZuMmyL2MU8qb3q6gg1iwTYvndjzMjVpv8btEpeX9Lnqmu): protocol: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection Dec 26 17:41:57.649785219 (506h12m57.520868274s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649899828 (506h12m57.520982903s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649908775 (506h12m57.52099182s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650074641 (506h12m57.521157696s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.650083287 (506h12m57.521166342s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650132821 (506h12m57.521215866s) [seg_pos={0 0}, upload=444191]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 17:41:57.650220138 (506h12m57.521303183s) [upload=444191]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:41:57.650227171 (506h12m57.521310206s) [upload=444191]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:41:57.958890870 (506h12m57.829973915s) [upload=444191]: Done uploading segments... Dec 26 17:41:57.958911620 (506h12m57.829994665s) [upload=444191]: Upload finished. err: <nil> Duration: 40.733322695s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [..XX......X.XXX.....XX.X.XXX.........X..X.....X...XX........] duration: 5.836468248s trace: https://jaeger.dev.storj.io/trace/3992379534f794fb FAIL (2025-12-26 18:16:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: LLOE3CT5REHRQEWDHHSGDBGJPWC35OQMVR4MEZB4FDOZ4AWGFGHA) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: DLDNZ7WRHTHESIBXZBLTGAQNLJRAJBDKPZHSCQZW6GYUNLYIONDA) context deadline exceeded ======LOGS====== Starting upload Dec 26 18:11:00.104273846 (506h41m59.975356892s) [upload=444620]: Done starting upload Dec 26 18:11:00.104464159 (506h41m59.975547194s) [upload=444620]: Uploading segments... Dec 26 18:11:00.104513854 (506h41m59.975596889s) [upload=444620]: Got next segment. Inline: false Dec 26 18:11:00.104544612 (506h41m59.975627647s) [upload=444620]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 18:11:00.104549912 (506h41m59.975632957s) [upload=444620]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 18:11:00.350198911 (506h42m0.221281966s) [seg_pos={0 0}, upload=444620]: Begin upload segment... Dec 26 18:11:00.350254707 (506h42m0.221337752s) [seg_pos={0 0}, upload=444620]: Done begin upload segment. Dec 26 18:11:00.350321845 (506h42m0.22140489s) [noise=true , address=145.79.227.11:4731 , piece_id=NJN4CQGBUBFZWEUDP3KBVU2AVACTXCQ2HE6YJ7W7MKC4E3D4LXIQ, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.350328608 (506h42m0.221411653s) [noise=true , address=45.91.116.7:4009 , piece_id=LLOE3CT5REHRQEWDHHSGDBGJPWC35OQMVR4MEZB4FDOZ4AWGFGHA, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.350959219 (506h42m0.222042264s) [noise=true , address=145.79.227.8:4330 , piece_id=PDB6ALO5ZPYSMSEQHF3NPY3VPWG7XGCXPXA22HWBFRGA5YYA3G5A, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.351321138 (506h42m0.222404173s) [noise=true , address=45.91.116.5:4297 , piece_id=APSDM2AGSPHTXZ36DROJOVMPFVPCHUHY5UBTILU32LERBQQOWK2Q, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.351684470 (506h42m0.222767515s) [noise=true , address=145.79.227.42:4121 , piece_id=H56KQOPDWK557ZZJCWBH7YLWWL3DK24BO3654Q4EWQ6ALZ5AU2UQ, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.352040457 (506h42m0.223123492s) [noise=true , address=45.91.116.8:4047 , piece_id=K5Q2OWU2L6QI5Y77KA7OS5UYOGDYUZ6B6XWURJMSUQXCEUAFC5UA, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.352047270 (506h42m0.223130305s) [noise=true , address=145.79.227.49:4337 , piece_id=BUOWQXWRHHZCKHJPC3D6KNYPHLNNVNSYZCMLG4JICHTJZVFVB7VQ, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.352385835 (506h42m0.22346888s) [noise=true , address=145.79.227.47:4267 , piece_id=XRQQF5K3O3A4V5545MZA2YG2N6EVRG4KIM56A6W6IRRWISTU6WAQ, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.352443936 (506h42m0.223526971s) [noise=true , address=45.91.116.10:4083 , piece_id=OBCQWLK3PCTYKVBP6DPVWXVQPJZVDVHMXS7333U5V5ROKQ2KQEEQ, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.352694222 (506h42m0.223777257s) [noise=true , address=145.79.227.22:4511 , piece_id=S6ZS6SOLNLZXGX7H4CUFN565Y4AYUIZCC2CXBJS6L3CSVB4OC3XQ, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.352808039 (506h42m0.223891084s) [noise=true , address=145.79.227.36:4517 , piece_id=ZL6T3RQT5XNQURM6JLOZ2YGAXNGLYGN5FYHO47IKWPPXZFCUEVBA, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.353054909 (506h42m0.224137944s) [noise=true , address=145.79.227.5:4223 , piece_id=5SXLARFJKURKC7AIF765Z7MWPHOEDHOFMNLCHMQBZUYBOC6AVNOQ, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.353570130 (506h42m0.224653175s) [noise=true , address=145.79.227.38:4978 , piece_id=6UFEMERR2PHRITTQKOGZV3CZCYLHONY36POKRX4TJ3IG3LB4RLQQ, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.354135146 (506h42m0.225218181s) [noise=true , address=45.91.116.11:4119 , piece_id=GA4DRGZTVG4TKE6MPO2YWCWERELOSSF2QZYG5L5MG7TEG5C6TFVA, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.354603829 (506h42m0.225686874s) [noise=true , address=145.79.227.50:4374 , piece_id=TMHRSFYNG6WI6VZF6CER7JWWG4PAAMSJ22QP6RAI64CJO4GAHNGQ, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.355020522 (506h42m0.226103567s) [noise=true , address=45.91.116.6:4974 , piece_id=DLDNZ7WRHTHESIBXZBLTGAQNLJRAJBDKPZHSCQZW6GYUNLYIONDA, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:02.032410415 (506h42m1.90349346s) [upload=444620]: Next returned nil segment Dec 26 18:11:02.032430152 (506h42m1.903513197s) [upload=444620]: Waiting for error group managing segments... Dec 26 18:11:02.348324681 (506h42m2.219407716s) [noise=true , address=145.79.227.22:4511 , piece_id=S6ZS6SOLNLZXGX7H4CUFN565Y4AYUIZCC2CXBJS6L3CSVB4OC3XQ, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.348549048 (506h42m2.219632083s) [noise=true , address=145.79.227.38:4978 , piece_id=6UFEMERR2PHRITTQKOGZV3CZCYLHONY36POKRX4TJ3IG3LB4RLQQ, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.351071442 (506h42m2.222154487s) [noise=true , address=145.79.227.8:4330 , piece_id=PDB6ALO5ZPYSMSEQHF3NPY3VPWG7XGCXPXA22HWBFRGA5YYA3G5A, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.355344347 (506h42m2.226427392s) [noise=true , address=145.79.227.50:4374 , piece_id=TMHRSFYNG6WI6VZF6CER7JWWG4PAAMSJ22QP6RAI64CJO4GAHNGQ, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.359503999 (506h42m2.230587044s) [noise=true , address=145.79.227.49:4337 , piece_id=BUOWQXWRHHZCKHJPC3D6KNYPHLNNVNSYZCMLG4JICHTJZVFVB7VQ, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.361316750 (506h42m2.232399785s) [noise=true , address=145.79.227.5:4223 , piece_id=5SXLARFJKURKC7AIF765Z7MWPHOEDHOFMNLCHMQBZUYBOC6AVNOQ, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.373894200 (506h42m2.244977235s) [noise=true , address=145.79.227.47:4267 , piece_id=XRQQF5K3O3A4V5545MZA2YG2N6EVRG4KIM56A6W6IRRWISTU6WAQ, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.376201063 (506h42m2.247284108s) [noise=true , address=145.79.227.36:4517 , piece_id=ZL6T3RQT5XNQURM6JLOZ2YGAXNGLYGN5FYHO47IKWPPXZFCUEVBA, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.380841749 (506h42m2.251924804s) [noise=true , address=145.79.227.11:4731 , piece_id=NJN4CQGBUBFZWEUDP3KBVU2AVACTXCQ2HE6YJ7W7MKC4E3D4LXIQ, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.388670316 (506h42m2.259753351s) [noise=true , address=145.79.227.42:4121 , piece_id=H56KQOPDWK557ZZJCWBH7YLWWL3DK24BO3654Q4EWQ6ALZ5AU2UQ, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.586101119 (506h42m2.457184164s) [noise=true , address=45.91.116.7:4009 , piece_id=LLOE3CT5REHRQEWDHHSGDBGJPWC35OQMVR4MEZB4FDOZ4AWGFGHA, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.597900427 (506h42m2.468983462s) [noise=true , address=45.91.116.10:4083 , piece_id=OBCQWLK3PCTYKVBP6DPVWXVQPJZVDVHMXS7333U5V5ROKQ2KQEEQ, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.711289730 (506h42m2.582372765s) [noise=true , address=45.91.116.6:4974 , piece_id=DLDNZ7WRHTHESIBXZBLTGAQNLJRAJBDKPZHSCQZW6GYUNLYIONDA, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.711310149 (506h42m2.582393184s) [seg_pos={0 0}, upload=444620]: Segment reached optimal threshold of 13 pieces. Dec 26 18:11:02.711579893 (506h42m2.582662938s) [noise=true , address=45.91.116.11:4119 , piece_id=GA4DRGZTVG4TKE6MPO2YWCWERELOSSF2QZYG5L5MG7TEG5C6TFVA, seg_pos={0 0}, upload=444620]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: read tcp 188.166.225.170:54212->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 18:11:02.711589541 (506h42m2.582672576s) [noise=true , address=45.91.116.11:4119 , piece_id=GA4DRGZTVG4TKE6MPO2YWCWERELOSSF2QZYG5L5MG7TEG5C6TFVA, seg_pos={0 0}, upload=444620]: Long tail cancellation due to optimal threshold Dec 26 18:11:02.711667138 (506h42m2.582750184s) [noise=true , address=45.91.116.8:4047 , piece_id=K5Q2OWU2L6QI5Y77KA7OS5UYOGDYUZ6B6XWURJMSUQXCEUAFC5UA, seg_pos={0 0}, upload=444620]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 26 18:11:02.711691024 (506h42m2.582774069s) [noise=true , address=45.91.116.8:4047 , piece_id=K5Q2OWU2L6QI5Y77KA7OS5UYOGDYUZ6B6XWURJMSUQXCEUAFC5UA, seg_pos={0 0}, upload=444620]: Long tail cancellation due to optimal threshold Dec 26 18:11:02.711887848 (506h42m2.582970913s) [noise=true , address=45.91.116.5:4297 , piece_id=APSDM2AGSPHTXZ36DROJOVMPFVPCHUHY5UBTILU32LERBQQOWK2Q, seg_pos={0 0}, upload=444620]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: write tcp 188.166.225.170:38572->45.91.116.5:4297: use of closed network connection; write tcp 188.166.225.170:38572->45.91.116.5:4297: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38572->45.91.116.5:4297: use of closed network connection Dec 26 18:11:02.711896776 (506h42m2.582979821s) [noise=true , address=45.91.116.5:4297 , piece_id=APSDM2AGSPHTXZ36DROJOVMPFVPCHUHY5UBTILU32LERBQQOWK2Q, seg_pos={0 0}, upload=444620]: Long tail cancellation due to optimal threshold Dec 26 18:11:02.711919389 (506h42m2.583002434s) [seg_pos={0 0}, upload=444620]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 18:11:02.712023507 (506h42m2.583106542s) [upload=444620]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 18:11:02.712031572 (506h42m2.583114607s) [upload=444620]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 18:11:03.015981797 (506h42m2.887064842s) [upload=444620]: Done uploading segments... Dec 26 18:11:03.016006694 (506h42m2.887089739s) [upload=444620]: Upload finished. err: <nil> Duration: 5.836499206s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [X..XX......X.XXX.....XX.X.XXX.........X..X.....X...XX.......] duration: 5.868920902s trace: https://jaeger.dev.storj.io/trace/3992379534f794fb FAIL (2025-12-26 18:16:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: LLOE3CT5REHRQEWDHHSGDBGJPWC35OQMVR4MEZB4FDOZ4AWGFGHA) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: DLDNZ7WRHTHESIBXZBLTGAQNLJRAJBDKPZHSCQZW6GYUNLYIONDA) context deadline exceeded ======LOGS====== Starting upload Dec 26 18:11:00.104273846 (506h41m59.975356892s) [upload=444620]: Done starting upload Dec 26 18:11:00.104464159 (506h41m59.975547194s) [upload=444620]: Uploading segments... Dec 26 18:11:00.104513854 (506h41m59.975596889s) [upload=444620]: Got next segment. Inline: false Dec 26 18:11:00.104544612 (506h41m59.975627647s) [upload=444620]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 18:11:00.104549912 (506h41m59.975632957s) [upload=444620]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 18:11:00.350198911 (506h42m0.221281966s) [seg_pos={0 0}, upload=444620]: Begin upload segment... Dec 26 18:11:00.350254707 (506h42m0.221337752s) [seg_pos={0 0}, upload=444620]: Done begin upload segment. Dec 26 18:11:00.350321845 (506h42m0.22140489s) [noise=true , address=145.79.227.11:4731 , piece_id=NJN4CQGBUBFZWEUDP3KBVU2AVACTXCQ2HE6YJ7W7MKC4E3D4LXIQ, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.350328608 (506h42m0.221411653s) [noise=true , address=45.91.116.7:4009 , piece_id=LLOE3CT5REHRQEWDHHSGDBGJPWC35OQMVR4MEZB4FDOZ4AWGFGHA, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.350959219 (506h42m0.222042264s) [noise=true , address=145.79.227.8:4330 , piece_id=PDB6ALO5ZPYSMSEQHF3NPY3VPWG7XGCXPXA22HWBFRGA5YYA3G5A, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.351321138 (506h42m0.222404173s) [noise=true , address=45.91.116.5:4297 , piece_id=APSDM2AGSPHTXZ36DROJOVMPFVPCHUHY5UBTILU32LERBQQOWK2Q, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.351684470 (506h42m0.222767515s) [noise=true , address=145.79.227.42:4121 , piece_id=H56KQOPDWK557ZZJCWBH7YLWWL3DK24BO3654Q4EWQ6ALZ5AU2UQ, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.352040457 (506h42m0.223123492s) [noise=true , address=45.91.116.8:4047 , piece_id=K5Q2OWU2L6QI5Y77KA7OS5UYOGDYUZ6B6XWURJMSUQXCEUAFC5UA, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.352047270 (506h42m0.223130305s) [noise=true , address=145.79.227.49:4337 , piece_id=BUOWQXWRHHZCKHJPC3D6KNYPHLNNVNSYZCMLG4JICHTJZVFVB7VQ, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.352385835 (506h42m0.22346888s) [noise=true , address=145.79.227.47:4267 , piece_id=XRQQF5K3O3A4V5545MZA2YG2N6EVRG4KIM56A6W6IRRWISTU6WAQ, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.352443936 (506h42m0.223526971s) [noise=true , address=45.91.116.10:4083 , piece_id=OBCQWLK3PCTYKVBP6DPVWXVQPJZVDVHMXS7333U5V5ROKQ2KQEEQ, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.352694222 (506h42m0.223777257s) [noise=true , address=145.79.227.22:4511 , piece_id=S6ZS6SOLNLZXGX7H4CUFN565Y4AYUIZCC2CXBJS6L3CSVB4OC3XQ, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.352808039 (506h42m0.223891084s) [noise=true , address=145.79.227.36:4517 , piece_id=ZL6T3RQT5XNQURM6JLOZ2YGAXNGLYGN5FYHO47IKWPPXZFCUEVBA, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.353054909 (506h42m0.224137944s) [noise=true , address=145.79.227.5:4223 , piece_id=5SXLARFJKURKC7AIF765Z7MWPHOEDHOFMNLCHMQBZUYBOC6AVNOQ, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.353570130 (506h42m0.224653175s) [noise=true , address=145.79.227.38:4978 , piece_id=6UFEMERR2PHRITTQKOGZV3CZCYLHONY36POKRX4TJ3IG3LB4RLQQ, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.354135146 (506h42m0.225218181s) [noise=true , address=45.91.116.11:4119 , piece_id=GA4DRGZTVG4TKE6MPO2YWCWERELOSSF2QZYG5L5MG7TEG5C6TFVA, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.354603829 (506h42m0.225686874s) [noise=true , address=145.79.227.50:4374 , piece_id=TMHRSFYNG6WI6VZF6CER7JWWG4PAAMSJ22QP6RAI64CJO4GAHNGQ, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.355020522 (506h42m0.226103567s) [noise=true , address=45.91.116.6:4974 , piece_id=DLDNZ7WRHTHESIBXZBLTGAQNLJRAJBDKPZHSCQZW6GYUNLYIONDA, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:02.032410415 (506h42m1.90349346s) [upload=444620]: Next returned nil segment Dec 26 18:11:02.032430152 (506h42m1.903513197s) [upload=444620]: Waiting for error group managing segments... Dec 26 18:11:02.348324681 (506h42m2.219407716s) [noise=true , address=145.79.227.22:4511 , piece_id=S6ZS6SOLNLZXGX7H4CUFN565Y4AYUIZCC2CXBJS6L3CSVB4OC3XQ, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.348549048 (506h42m2.219632083s) [noise=true , address=145.79.227.38:4978 , piece_id=6UFEMERR2PHRITTQKOGZV3CZCYLHONY36POKRX4TJ3IG3LB4RLQQ, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.351071442 (506h42m2.222154487s) [noise=true , address=145.79.227.8:4330 , piece_id=PDB6ALO5ZPYSMSEQHF3NPY3VPWG7XGCXPXA22HWBFRGA5YYA3G5A, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.355344347 (506h42m2.226427392s) [noise=true , address=145.79.227.50:4374 , piece_id=TMHRSFYNG6WI6VZF6CER7JWWG4PAAMSJ22QP6RAI64CJO4GAHNGQ, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.359503999 (506h42m2.230587044s) [noise=true , address=145.79.227.49:4337 , piece_id=BUOWQXWRHHZCKHJPC3D6KNYPHLNNVNSYZCMLG4JICHTJZVFVB7VQ, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.361316750 (506h42m2.232399785s) [noise=true , address=145.79.227.5:4223 , piece_id=5SXLARFJKURKC7AIF765Z7MWPHOEDHOFMNLCHMQBZUYBOC6AVNOQ, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.373894200 (506h42m2.244977235s) [noise=true , address=145.79.227.47:4267 , piece_id=XRQQF5K3O3A4V5545MZA2YG2N6EVRG4KIM56A6W6IRRWISTU6WAQ, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.376201063 (506h42m2.247284108s) [noise=true , address=145.79.227.36:4517 , piece_id=ZL6T3RQT5XNQURM6JLOZ2YGAXNGLYGN5FYHO47IKWPPXZFCUEVBA, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.380841749 (506h42m2.251924804s) [noise=true , address=145.79.227.11:4731 , piece_id=NJN4CQGBUBFZWEUDP3KBVU2AVACTXCQ2HE6YJ7W7MKC4E3D4LXIQ, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.388670316 (506h42m2.259753351s) [noise=true , address=145.79.227.42:4121 , piece_id=H56KQOPDWK557ZZJCWBH7YLWWL3DK24BO3654Q4EWQ6ALZ5AU2UQ, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.586101119 (506h42m2.457184164s) [noise=true , address=45.91.116.7:4009 , piece_id=LLOE3CT5REHRQEWDHHSGDBGJPWC35OQMVR4MEZB4FDOZ4AWGFGHA, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.597900427 (506h42m2.468983462s) [noise=true , address=45.91.116.10:4083 , piece_id=OBCQWLK3PCTYKVBP6DPVWXVQPJZVDVHMXS7333U5V5ROKQ2KQEEQ, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.711289730 (506h42m2.582372765s) [noise=true , address=45.91.116.6:4974 , piece_id=DLDNZ7WRHTHESIBXZBLTGAQNLJRAJBDKPZHSCQZW6GYUNLYIONDA, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.711310149 (506h42m2.582393184s) [seg_pos={0 0}, upload=444620]: Segment reached optimal threshold of 13 pieces. Dec 26 18:11:02.711579893 (506h42m2.582662938s) [noise=true , address=45.91.116.11:4119 , piece_id=GA4DRGZTVG4TKE6MPO2YWCWERELOSSF2QZYG5L5MG7TEG5C6TFVA, seg_pos={0 0}, upload=444620]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: read tcp 188.166.225.170:54212->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 18:11:02.711589541 (506h42m2.582672576s) [noise=true , address=45.91.116.11:4119 , piece_id=GA4DRGZTVG4TKE6MPO2YWCWERELOSSF2QZYG5L5MG7TEG5C6TFVA, seg_pos={0 0}, upload=444620]: Long tail cancellation due to optimal threshold Dec 26 18:11:02.711667138 (506h42m2.582750184s) [noise=true , address=45.91.116.8:4047 , piece_id=K5Q2OWU2L6QI5Y77KA7OS5UYOGDYUZ6B6XWURJMSUQXCEUAFC5UA, seg_pos={0 0}, upload=444620]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 26 18:11:02.711691024 (506h42m2.582774069s) [noise=true , address=45.91.116.8:4047 , piece_id=K5Q2OWU2L6QI5Y77KA7OS5UYOGDYUZ6B6XWURJMSUQXCEUAFC5UA, seg_pos={0 0}, upload=444620]: Long tail cancellation due to optimal threshold Dec 26 18:11:02.711887848 (506h42m2.582970913s) [noise=true , address=45.91.116.5:4297 , piece_id=APSDM2AGSPHTXZ36DROJOVMPFVPCHUHY5UBTILU32LERBQQOWK2Q, seg_pos={0 0}, upload=444620]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: write tcp 188.166.225.170:38572->45.91.116.5:4297: use of closed network connection; write tcp 188.166.225.170:38572->45.91.116.5:4297: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38572->45.91.116.5:4297: use of closed network connection Dec 26 18:11:02.711896776 (506h42m2.582979821s) [noise=true , address=45.91.116.5:4297 , piece_id=APSDM2AGSPHTXZ36DROJOVMPFVPCHUHY5UBTILU32LERBQQOWK2Q, seg_pos={0 0}, upload=444620]: Long tail cancellation due to optimal threshold Dec 26 18:11:02.711919389 (506h42m2.583002434s) [seg_pos={0 0}, upload=444620]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 18:11:02.712023507 (506h42m2.583106542s) [upload=444620]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 18:11:02.712031572 (506h42m2.583114607s) [upload=444620]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 18:11:03.015981797 (506h42m2.887064842s) [upload=444620]: Done uploading segments... Dec 26 18:11:03.016006694 (506h42m2.887089739s) [upload=444620]: Upload finished. err: <nil> Duration: 5.868954396s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [XX..XX......X.XXX.....XX.X.XXX.........X..X.....X...XX......] duration: 6.088202772s trace: https://jaeger.dev.storj.io/trace/3992379534f794fb FAIL (2025-12-26 18:15:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: LLOE3CT5REHRQEWDHHSGDBGJPWC35OQMVR4MEZB4FDOZ4AWGFGHA) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: DLDNZ7WRHTHESIBXZBLTGAQNLJRAJBDKPZHSCQZW6GYUNLYIONDA) context deadline exceeded ======LOGS====== Starting upload Dec 26 18:11:00.104273846 (506h41m59.975356892s) [upload=444620]: Done starting upload Dec 26 18:11:00.104464159 (506h41m59.975547194s) [upload=444620]: Uploading segments... Dec 26 18:11:00.104513854 (506h41m59.975596889s) [upload=444620]: Got next segment. Inline: false Dec 26 18:11:00.104544612 (506h41m59.975627647s) [upload=444620]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 18:11:00.104549912 (506h41m59.975632957s) [upload=444620]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 18:11:00.350198911 (506h42m0.221281966s) [seg_pos={0 0}, upload=444620]: Begin upload segment... Dec 26 18:11:00.350254707 (506h42m0.221337752s) [seg_pos={0 0}, upload=444620]: Done begin upload segment. Dec 26 18:11:00.350321845 (506h42m0.22140489s) [noise=true , address=145.79.227.11:4731 , piece_id=NJN4CQGBUBFZWEUDP3KBVU2AVACTXCQ2HE6YJ7W7MKC4E3D4LXIQ, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.350328608 (506h42m0.221411653s) [noise=true , address=45.91.116.7:4009 , piece_id=LLOE3CT5REHRQEWDHHSGDBGJPWC35OQMVR4MEZB4FDOZ4AWGFGHA, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.350959219 (506h42m0.222042264s) [noise=true , address=145.79.227.8:4330 , piece_id=PDB6ALO5ZPYSMSEQHF3NPY3VPWG7XGCXPXA22HWBFRGA5YYA3G5A, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.351321138 (506h42m0.222404173s) [noise=true , address=45.91.116.5:4297 , piece_id=APSDM2AGSPHTXZ36DROJOVMPFVPCHUHY5UBTILU32LERBQQOWK2Q, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.351684470 (506h42m0.222767515s) [noise=true , address=145.79.227.42:4121 , piece_id=H56KQOPDWK557ZZJCWBH7YLWWL3DK24BO3654Q4EWQ6ALZ5AU2UQ, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.352040457 (506h42m0.223123492s) [noise=true , address=45.91.116.8:4047 , piece_id=K5Q2OWU2L6QI5Y77KA7OS5UYOGDYUZ6B6XWURJMSUQXCEUAFC5UA, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.352047270 (506h42m0.223130305s) [noise=true , address=145.79.227.49:4337 , piece_id=BUOWQXWRHHZCKHJPC3D6KNYPHLNNVNSYZCMLG4JICHTJZVFVB7VQ, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.352385835 (506h42m0.22346888s) [noise=true , address=145.79.227.47:4267 , piece_id=XRQQF5K3O3A4V5545MZA2YG2N6EVRG4KIM56A6W6IRRWISTU6WAQ, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.352443936 (506h42m0.223526971s) [noise=true , address=45.91.116.10:4083 , piece_id=OBCQWLK3PCTYKVBP6DPVWXVQPJZVDVHMXS7333U5V5ROKQ2KQEEQ, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.352694222 (506h42m0.223777257s) [noise=true , address=145.79.227.22:4511 , piece_id=S6ZS6SOLNLZXGX7H4CUFN565Y4AYUIZCC2CXBJS6L3CSVB4OC3XQ, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.352808039 (506h42m0.223891084s) [noise=true , address=145.79.227.36:4517 , piece_id=ZL6T3RQT5XNQURM6JLOZ2YGAXNGLYGN5FYHO47IKWPPXZFCUEVBA, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.353054909 (506h42m0.224137944s) [noise=true , address=145.79.227.5:4223 , piece_id=5SXLARFJKURKC7AIF765Z7MWPHOEDHOFMNLCHMQBZUYBOC6AVNOQ, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.353570130 (506h42m0.224653175s) [noise=true , address=145.79.227.38:4978 , piece_id=6UFEMERR2PHRITTQKOGZV3CZCYLHONY36POKRX4TJ3IG3LB4RLQQ, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.354135146 (506h42m0.225218181s) [noise=true , address=45.91.116.11:4119 , piece_id=GA4DRGZTVG4TKE6MPO2YWCWERELOSSF2QZYG5L5MG7TEG5C6TFVA, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.354603829 (506h42m0.225686874s) [noise=true , address=145.79.227.50:4374 , piece_id=TMHRSFYNG6WI6VZF6CER7JWWG4PAAMSJ22QP6RAI64CJO4GAHNGQ, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.355020522 (506h42m0.226103567s) [noise=true , address=45.91.116.6:4974 , piece_id=DLDNZ7WRHTHESIBXZBLTGAQNLJRAJBDKPZHSCQZW6GYUNLYIONDA, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:02.032410415 (506h42m1.90349346s) [upload=444620]: Next returned nil segment Dec 26 18:11:02.032430152 (506h42m1.903513197s) [upload=444620]: Waiting for error group managing segments... Dec 26 18:11:02.348324681 (506h42m2.219407716s) [noise=true , address=145.79.227.22:4511 , piece_id=S6ZS6SOLNLZXGX7H4CUFN565Y4AYUIZCC2CXBJS6L3CSVB4OC3XQ, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.348549048 (506h42m2.219632083s) [noise=true , address=145.79.227.38:4978 , piece_id=6UFEMERR2PHRITTQKOGZV3CZCYLHONY36POKRX4TJ3IG3LB4RLQQ, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.351071442 (506h42m2.222154487s) [noise=true , address=145.79.227.8:4330 , piece_id=PDB6ALO5ZPYSMSEQHF3NPY3VPWG7XGCXPXA22HWBFRGA5YYA3G5A, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.355344347 (506h42m2.226427392s) [noise=true , address=145.79.227.50:4374 , piece_id=TMHRSFYNG6WI6VZF6CER7JWWG4PAAMSJ22QP6RAI64CJO4GAHNGQ, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.359503999 (506h42m2.230587044s) [noise=true , address=145.79.227.49:4337 , piece_id=BUOWQXWRHHZCKHJPC3D6KNYPHLNNVNSYZCMLG4JICHTJZVFVB7VQ, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.361316750 (506h42m2.232399785s) [noise=true , address=145.79.227.5:4223 , piece_id=5SXLARFJKURKC7AIF765Z7MWPHOEDHOFMNLCHMQBZUYBOC6AVNOQ, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.373894200 (506h42m2.244977235s) [noise=true , address=145.79.227.47:4267 , piece_id=XRQQF5K3O3A4V5545MZA2YG2N6EVRG4KIM56A6W6IRRWISTU6WAQ, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.376201063 (506h42m2.247284108s) [noise=true , address=145.79.227.36:4517 , piece_id=ZL6T3RQT5XNQURM6JLOZ2YGAXNGLYGN5FYHO47IKWPPXZFCUEVBA, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.380841749 (506h42m2.251924804s) [noise=true , address=145.79.227.11:4731 , piece_id=NJN4CQGBUBFZWEUDP3KBVU2AVACTXCQ2HE6YJ7W7MKC4E3D4LXIQ, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.388670316 (506h42m2.259753351s) [noise=true , address=145.79.227.42:4121 , piece_id=H56KQOPDWK557ZZJCWBH7YLWWL3DK24BO3654Q4EWQ6ALZ5AU2UQ, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.586101119 (506h42m2.457184164s) [noise=true , address=45.91.116.7:4009 , piece_id=LLOE3CT5REHRQEWDHHSGDBGJPWC35OQMVR4MEZB4FDOZ4AWGFGHA, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.597900427 (506h42m2.468983462s) [noise=true , address=45.91.116.10:4083 , piece_id=OBCQWLK3PCTYKVBP6DPVWXVQPJZVDVHMXS7333U5V5ROKQ2KQEEQ, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.711289730 (506h42m2.582372765s) [noise=true , address=45.91.116.6:4974 , piece_id=DLDNZ7WRHTHESIBXZBLTGAQNLJRAJBDKPZHSCQZW6GYUNLYIONDA, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.711310149 (506h42m2.582393184s) [seg_pos={0 0}, upload=444620]: Segment reached optimal threshold of 13 pieces. Dec 26 18:11:02.711579893 (506h42m2.582662938s) [noise=true , address=45.91.116.11:4119 , piece_id=GA4DRGZTVG4TKE6MPO2YWCWERELOSSF2QZYG5L5MG7TEG5C6TFVA, seg_pos={0 0}, upload=444620]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: read tcp 188.166.225.170:54212->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 18:11:02.711589541 (506h42m2.582672576s) [noise=true , address=45.91.116.11:4119 , piece_id=GA4DRGZTVG4TKE6MPO2YWCWERELOSSF2QZYG5L5MG7TEG5C6TFVA, seg_pos={0 0}, upload=444620]: Long tail cancellation due to optimal threshold Dec 26 18:11:02.711667138 (506h42m2.582750184s) [noise=true , address=45.91.116.8:4047 , piece_id=K5Q2OWU2L6QI5Y77KA7OS5UYOGDYUZ6B6XWURJMSUQXCEUAFC5UA, seg_pos={0 0}, upload=444620]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 26 18:11:02.711691024 (506h42m2.582774069s) [noise=true , address=45.91.116.8:4047 , piece_id=K5Q2OWU2L6QI5Y77KA7OS5UYOGDYUZ6B6XWURJMSUQXCEUAFC5UA, seg_pos={0 0}, upload=444620]: Long tail cancellation due to optimal threshold Dec 26 18:11:02.711887848 (506h42m2.582970913s) [noise=true , address=45.91.116.5:4297 , piece_id=APSDM2AGSPHTXZ36DROJOVMPFVPCHUHY5UBTILU32LERBQQOWK2Q, seg_pos={0 0}, upload=444620]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: write tcp 188.166.225.170:38572->45.91.116.5:4297: use of closed network connection; write tcp 188.166.225.170:38572->45.91.116.5:4297: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38572->45.91.116.5:4297: use of closed network connection Dec 26 18:11:02.711896776 (506h42m2.582979821s) [noise=true , address=45.91.116.5:4297 , piece_id=APSDM2AGSPHTXZ36DROJOVMPFVPCHUHY5UBTILU32LERBQQOWK2Q, seg_pos={0 0}, upload=444620]: Long tail cancellation due to optimal threshold Dec 26 18:11:02.711919389 (506h42m2.583002434s) [seg_pos={0 0}, upload=444620]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 18:11:02.712023507 (506h42m2.583106542s) [upload=444620]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 18:11:02.712031572 (506h42m2.583114607s) [upload=444620]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 18:11:03.015981797 (506h42m2.887064842s) [upload=444620]: Done uploading segments... Dec 26 18:11:03.016006694 (506h42m2.887089739s) [upload=444620]: Upload finished. err: <nil> Duration: 6.088227228s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 15.508716861s trace: https://jaeger.dev.storj.io/trace/665ea65f6e9ccc33 FAIL (2025-12-26 18:15:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== e=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789316979 (506h12m54.660400024s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789326507 (506h12m54.660409552s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789611339 (506h12m54.660694384s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789619275 (506h12m54.66070231s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789914006 (506h12m54.660997041s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789949443 (506h12m54.661032478s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790181566 (506h12m54.661264601s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790295632 (506h12m54.661378668s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790510872 (506h12m54.661593917s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790692980 (506h12m54.661776015s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790787379 (506h12m54.661870414s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790992970 (506h12m54.662076005s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791120053 (506h12m54.662203098s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791275218 (506h12m54.662358253s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791443709 (506h12m54.662526744s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:56.112313145 (506h12m55.98339619s) [upload=444191]: Next returned nil segment Dec 26 17:41:56.112330638 (506h12m55.983413683s) [upload=444191]: Waiting for error group managing segments... Dec 26 17:41:56.485322720 (506h12m56.356405765s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.485637079 (506h12m56.356720124s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.666807539 (506h12m56.537890584s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.678753987 (506h12m56.549837032s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.687788039 (506h12m56.558871074s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.694318023 (506h12m56.565401058s) [noise=true , address=145.79.227.7:4326 , piece_id=DMQ5RPI46X2U7TJTS42ZHQEF32BDJRKJJO5HWOYAPNS6VWVG5B5Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.716581187 (506h12m56.587664232s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.972183007 (506h12m56.843266042s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.011917999 (506h12m56.883001034s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.054168622 (506h12m56.925251667s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.114312777 (506h12m56.985395812s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.499846250 (506h12m57.370929285s) [noise=true , address=145.79.227.34:4919 , piece_id=YRWHW3BJVEGVJRSA6RJB3T7VJQSDTGKEZQKRICFIV2G3T7AN4MIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.612649066 (506h12m57.483732101s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649413733 (506h12m57.520496768s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649440282 (506h12m57.520523328s) [seg_pos={0 0}, upload=444191]: Segment reached optimal threshold of 14 pieces. Dec 26 17:41:57.649674729 (506h12m57.520757774s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G7ty71tPwe7cZSBNK9u74DFUCGomnxDsNvAif4C3SbP7HWwfo): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649684087 (506h12m57.520767142s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649747708 (506h12m57.520830763s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v6zXZuMmyL2MU8qb3q6gg1iwTYvndjzMjVpv8btEpeX9Lnqmu): protocol: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection Dec 26 17:41:57.649785219 (506h12m57.520868274s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649899828 (506h12m57.520982903s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649908775 (506h12m57.52099182s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650074641 (506h12m57.521157696s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.650083287 (506h12m57.521166342s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650132821 (506h12m57.521215866s) [seg_pos={0 0}, upload=444191]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 17:41:57.650220138 (506h12m57.521303183s) [upload=444191]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:41:57.650227171 (506h12m57.521310206s) [upload=444191]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:41:57.958890870 (506h12m57.829973915s) [upload=444191]: Done uploading segments... Dec 26 17:41:57.958911620 (506h12m57.829994665s) [upload=444191]: Upload finished. err: <nil> Duration: 15.508745705s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [.XX..XX......X.XXX.....XX.X.XXX.........X..X.....X...XX.....] duration: 19.040029409s trace: https://jaeger.dev.storj.io/trace/3992379534f794fb FAIL (2025-12-26 18:15:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: LLOE3CT5REHRQEWDHHSGDBGJPWC35OQMVR4MEZB4FDOZ4AWGFGHA) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: DLDNZ7WRHTHESIBXZBLTGAQNLJRAJBDKPZHSCQZW6GYUNLYIONDA) context deadline exceeded ======LOGS====== Starting upload Dec 26 18:11:00.104273846 (506h41m59.975356892s) [upload=444620]: Done starting upload Dec 26 18:11:00.104464159 (506h41m59.975547194s) [upload=444620]: Uploading segments... Dec 26 18:11:00.104513854 (506h41m59.975596889s) [upload=444620]: Got next segment. Inline: false Dec 26 18:11:00.104544612 (506h41m59.975627647s) [upload=444620]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 18:11:00.104549912 (506h41m59.975632957s) [upload=444620]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 18:11:00.350198911 (506h42m0.221281966s) [seg_pos={0 0}, upload=444620]: Begin upload segment... Dec 26 18:11:00.350254707 (506h42m0.221337752s) [seg_pos={0 0}, upload=444620]: Done begin upload segment. Dec 26 18:11:00.350321845 (506h42m0.22140489s) [noise=true , address=145.79.227.11:4731 , piece_id=NJN4CQGBUBFZWEUDP3KBVU2AVACTXCQ2HE6YJ7W7MKC4E3D4LXIQ, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.350328608 (506h42m0.221411653s) [noise=true , address=45.91.116.7:4009 , piece_id=LLOE3CT5REHRQEWDHHSGDBGJPWC35OQMVR4MEZB4FDOZ4AWGFGHA, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.350959219 (506h42m0.222042264s) [noise=true , address=145.79.227.8:4330 , piece_id=PDB6ALO5ZPYSMSEQHF3NPY3VPWG7XGCXPXA22HWBFRGA5YYA3G5A, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.351321138 (506h42m0.222404173s) [noise=true , address=45.91.116.5:4297 , piece_id=APSDM2AGSPHTXZ36DROJOVMPFVPCHUHY5UBTILU32LERBQQOWK2Q, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.351684470 (506h42m0.222767515s) [noise=true , address=145.79.227.42:4121 , piece_id=H56KQOPDWK557ZZJCWBH7YLWWL3DK24BO3654Q4EWQ6ALZ5AU2UQ, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.352040457 (506h42m0.223123492s) [noise=true , address=45.91.116.8:4047 , piece_id=K5Q2OWU2L6QI5Y77KA7OS5UYOGDYUZ6B6XWURJMSUQXCEUAFC5UA, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.352047270 (506h42m0.223130305s) [noise=true , address=145.79.227.49:4337 , piece_id=BUOWQXWRHHZCKHJPC3D6KNYPHLNNVNSYZCMLG4JICHTJZVFVB7VQ, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.352385835 (506h42m0.22346888s) [noise=true , address=145.79.227.47:4267 , piece_id=XRQQF5K3O3A4V5545MZA2YG2N6EVRG4KIM56A6W6IRRWISTU6WAQ, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.352443936 (506h42m0.223526971s) [noise=true , address=45.91.116.10:4083 , piece_id=OBCQWLK3PCTYKVBP6DPVWXVQPJZVDVHMXS7333U5V5ROKQ2KQEEQ, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.352694222 (506h42m0.223777257s) [noise=true , address=145.79.227.22:4511 , piece_id=S6ZS6SOLNLZXGX7H4CUFN565Y4AYUIZCC2CXBJS6L3CSVB4OC3XQ, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.352808039 (506h42m0.223891084s) [noise=true , address=145.79.227.36:4517 , piece_id=ZL6T3RQT5XNQURM6JLOZ2YGAXNGLYGN5FYHO47IKWPPXZFCUEVBA, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.353054909 (506h42m0.224137944s) [noise=true , address=145.79.227.5:4223 , piece_id=5SXLARFJKURKC7AIF765Z7MWPHOEDHOFMNLCHMQBZUYBOC6AVNOQ, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.353570130 (506h42m0.224653175s) [noise=true , address=145.79.227.38:4978 , piece_id=6UFEMERR2PHRITTQKOGZV3CZCYLHONY36POKRX4TJ3IG3LB4RLQQ, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.354135146 (506h42m0.225218181s) [noise=true , address=45.91.116.11:4119 , piece_id=GA4DRGZTVG4TKE6MPO2YWCWERELOSSF2QZYG5L5MG7TEG5C6TFVA, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.354603829 (506h42m0.225686874s) [noise=true , address=145.79.227.50:4374 , piece_id=TMHRSFYNG6WI6VZF6CER7JWWG4PAAMSJ22QP6RAI64CJO4GAHNGQ, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.355020522 (506h42m0.226103567s) [noise=true , address=45.91.116.6:4974 , piece_id=DLDNZ7WRHTHESIBXZBLTGAQNLJRAJBDKPZHSCQZW6GYUNLYIONDA, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:02.032410415 (506h42m1.90349346s) [upload=444620]: Next returned nil segment Dec 26 18:11:02.032430152 (506h42m1.903513197s) [upload=444620]: Waiting for error group managing segments... Dec 26 18:11:02.348324681 (506h42m2.219407716s) [noise=true , address=145.79.227.22:4511 , piece_id=S6ZS6SOLNLZXGX7H4CUFN565Y4AYUIZCC2CXBJS6L3CSVB4OC3XQ, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.348549048 (506h42m2.219632083s) [noise=true , address=145.79.227.38:4978 , piece_id=6UFEMERR2PHRITTQKOGZV3CZCYLHONY36POKRX4TJ3IG3LB4RLQQ, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.351071442 (506h42m2.222154487s) [noise=true , address=145.79.227.8:4330 , piece_id=PDB6ALO5ZPYSMSEQHF3NPY3VPWG7XGCXPXA22HWBFRGA5YYA3G5A, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.355344347 (506h42m2.226427392s) [noise=true , address=145.79.227.50:4374 , piece_id=TMHRSFYNG6WI6VZF6CER7JWWG4PAAMSJ22QP6RAI64CJO4GAHNGQ, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.359503999 (506h42m2.230587044s) [noise=true , address=145.79.227.49:4337 , piece_id=BUOWQXWRHHZCKHJPC3D6KNYPHLNNVNSYZCMLG4JICHTJZVFVB7VQ, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.361316750 (506h42m2.232399785s) [noise=true , address=145.79.227.5:4223 , piece_id=5SXLARFJKURKC7AIF765Z7MWPHOEDHOFMNLCHMQBZUYBOC6AVNOQ, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.373894200 (506h42m2.244977235s) [noise=true , address=145.79.227.47:4267 , piece_id=XRQQF5K3O3A4V5545MZA2YG2N6EVRG4KIM56A6W6IRRWISTU6WAQ, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.376201063 (506h42m2.247284108s) [noise=true , address=145.79.227.36:4517 , piece_id=ZL6T3RQT5XNQURM6JLOZ2YGAXNGLYGN5FYHO47IKWPPXZFCUEVBA, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.380841749 (506h42m2.251924804s) [noise=true , address=145.79.227.11:4731 , piece_id=NJN4CQGBUBFZWEUDP3KBVU2AVACTXCQ2HE6YJ7W7MKC4E3D4LXIQ, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.388670316 (506h42m2.259753351s) [noise=true , address=145.79.227.42:4121 , piece_id=H56KQOPDWK557ZZJCWBH7YLWWL3DK24BO3654Q4EWQ6ALZ5AU2UQ, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.586101119 (506h42m2.457184164s) [noise=true , address=45.91.116.7:4009 , piece_id=LLOE3CT5REHRQEWDHHSGDBGJPWC35OQMVR4MEZB4FDOZ4AWGFGHA, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.597900427 (506h42m2.468983462s) [noise=true , address=45.91.116.10:4083 , piece_id=OBCQWLK3PCTYKVBP6DPVWXVQPJZVDVHMXS7333U5V5ROKQ2KQEEQ, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.711289730 (506h42m2.582372765s) [noise=true , address=45.91.116.6:4974 , piece_id=DLDNZ7WRHTHESIBXZBLTGAQNLJRAJBDKPZHSCQZW6GYUNLYIONDA, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.711310149 (506h42m2.582393184s) [seg_pos={0 0}, upload=444620]: Segment reached optimal threshold of 13 pieces. Dec 26 18:11:02.711579893 (506h42m2.582662938s) [noise=true , address=45.91.116.11:4119 , piece_id=GA4DRGZTVG4TKE6MPO2YWCWERELOSSF2QZYG5L5MG7TEG5C6TFVA, seg_pos={0 0}, upload=444620]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: read tcp 188.166.225.170:54212->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 18:11:02.711589541 (506h42m2.582672576s) [noise=true , address=45.91.116.11:4119 , piece_id=GA4DRGZTVG4TKE6MPO2YWCWERELOSSF2QZYG5L5MG7TEG5C6TFVA, seg_pos={0 0}, upload=444620]: Long tail cancellation due to optimal threshold Dec 26 18:11:02.711667138 (506h42m2.582750184s) [noise=true , address=45.91.116.8:4047 , piece_id=K5Q2OWU2L6QI5Y77KA7OS5UYOGDYUZ6B6XWURJMSUQXCEUAFC5UA, seg_pos={0 0}, upload=444620]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 26 18:11:02.711691024 (506h42m2.582774069s) [noise=true , address=45.91.116.8:4047 , piece_id=K5Q2OWU2L6QI5Y77KA7OS5UYOGDYUZ6B6XWURJMSUQXCEUAFC5UA, seg_pos={0 0}, upload=444620]: Long tail cancellation due to optimal threshold Dec 26 18:11:02.711887848 (506h42m2.582970913s) [noise=true , address=45.91.116.5:4297 , piece_id=APSDM2AGSPHTXZ36DROJOVMPFVPCHUHY5UBTILU32LERBQQOWK2Q, seg_pos={0 0}, upload=444620]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: write tcp 188.166.225.170:38572->45.91.116.5:4297: use of closed network connection; write tcp 188.166.225.170:38572->45.91.116.5:4297: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38572->45.91.116.5:4297: use of closed network connection Dec 26 18:11:02.711896776 (506h42m2.582979821s) [noise=true , address=45.91.116.5:4297 , piece_id=APSDM2AGSPHTXZ36DROJOVMPFVPCHUHY5UBTILU32LERBQQOWK2Q, seg_pos={0 0}, upload=444620]: Long tail cancellation due to optimal threshold Dec 26 18:11:02.711919389 (506h42m2.583002434s) [seg_pos={0 0}, upload=444620]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 18:11:02.712023507 (506h42m2.583106542s) [upload=444620]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 18:11:02.712031572 (506h42m2.583114607s) [upload=444620]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 18:11:03.015981797 (506h42m2.887064842s) [upload=444620]: Done uploading segments... Dec 26 18:11:03.016006694 (506h42m2.887089739s) [upload=444620]: Upload finished. err: <nil> Duration: 19.040060937s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 27.779506861s trace: https://jaeger.dev.storj.io/trace/665ea65f6e9ccc33 FAIL (2025-12-26 18:15:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== e=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789316979 (506h12m54.660400024s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789326507 (506h12m54.660409552s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789611339 (506h12m54.660694384s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789619275 (506h12m54.66070231s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789914006 (506h12m54.660997041s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789949443 (506h12m54.661032478s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790181566 (506h12m54.661264601s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790295632 (506h12m54.661378668s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790510872 (506h12m54.661593917s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790692980 (506h12m54.661776015s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790787379 (506h12m54.661870414s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790992970 (506h12m54.662076005s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791120053 (506h12m54.662203098s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791275218 (506h12m54.662358253s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791443709 (506h12m54.662526744s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:56.112313145 (506h12m55.98339619s) [upload=444191]: Next returned nil segment Dec 26 17:41:56.112330638 (506h12m55.983413683s) [upload=444191]: Waiting for error group managing segments... Dec 26 17:41:56.485322720 (506h12m56.356405765s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.485637079 (506h12m56.356720124s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.666807539 (506h12m56.537890584s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.678753987 (506h12m56.549837032s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.687788039 (506h12m56.558871074s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.694318023 (506h12m56.565401058s) [noise=true , address=145.79.227.7:4326 , piece_id=DMQ5RPI46X2U7TJTS42ZHQEF32BDJRKJJO5HWOYAPNS6VWVG5B5Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.716581187 (506h12m56.587664232s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.972183007 (506h12m56.843266042s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.011917999 (506h12m56.883001034s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.054168622 (506h12m56.925251667s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.114312777 (506h12m56.985395812s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.499846250 (506h12m57.370929285s) [noise=true , address=145.79.227.34:4919 , piece_id=YRWHW3BJVEGVJRSA6RJB3T7VJQSDTGKEZQKRICFIV2G3T7AN4MIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.612649066 (506h12m57.483732101s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649413733 (506h12m57.520496768s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649440282 (506h12m57.520523328s) [seg_pos={0 0}, upload=444191]: Segment reached optimal threshold of 14 pieces. Dec 26 17:41:57.649674729 (506h12m57.520757774s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G7ty71tPwe7cZSBNK9u74DFUCGomnxDsNvAif4C3SbP7HWwfo): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649684087 (506h12m57.520767142s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649747708 (506h12m57.520830763s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v6zXZuMmyL2MU8qb3q6gg1iwTYvndjzMjVpv8btEpeX9Lnqmu): protocol: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection Dec 26 17:41:57.649785219 (506h12m57.520868274s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649899828 (506h12m57.520982903s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649908775 (506h12m57.52099182s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650074641 (506h12m57.521157696s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.650083287 (506h12m57.521166342s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650132821 (506h12m57.521215866s) [seg_pos={0 0}, upload=444191]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 17:41:57.650220138 (506h12m57.521303183s) [upload=444191]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:41:57.650227171 (506h12m57.521310206s) [upload=444191]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:41:57.958890870 (506h12m57.829973915s) [upload=444191]: Done uploading segments... Dec 26 17:41:57.958911620 (506h12m57.829994665s) [upload=444191]: Upload finished. err: <nil> Duration: 27.779524866s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [..XX..XX......X.XXX.....XX.X.XXX.........X..X.....X...XX....] duration: 6.40741499s trace: https://jaeger.dev.storj.io/trace/3992379534f794fb FAIL (2025-12-26 18:14:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: LLOE3CT5REHRQEWDHHSGDBGJPWC35OQMVR4MEZB4FDOZ4AWGFGHA) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: DLDNZ7WRHTHESIBXZBLTGAQNLJRAJBDKPZHSCQZW6GYUNLYIONDA) context deadline exceeded ======LOGS====== Starting upload Dec 26 18:11:00.104273846 (506h41m59.975356892s) [upload=444620]: Done starting upload Dec 26 18:11:00.104464159 (506h41m59.975547194s) [upload=444620]: Uploading segments... Dec 26 18:11:00.104513854 (506h41m59.975596889s) [upload=444620]: Got next segment. Inline: false Dec 26 18:11:00.104544612 (506h41m59.975627647s) [upload=444620]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 18:11:00.104549912 (506h41m59.975632957s) [upload=444620]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 18:11:00.350198911 (506h42m0.221281966s) [seg_pos={0 0}, upload=444620]: Begin upload segment... Dec 26 18:11:00.350254707 (506h42m0.221337752s) [seg_pos={0 0}, upload=444620]: Done begin upload segment. Dec 26 18:11:00.350321845 (506h42m0.22140489s) [noise=true , address=145.79.227.11:4731 , piece_id=NJN4CQGBUBFZWEUDP3KBVU2AVACTXCQ2HE6YJ7W7MKC4E3D4LXIQ, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.350328608 (506h42m0.221411653s) [noise=true , address=45.91.116.7:4009 , piece_id=LLOE3CT5REHRQEWDHHSGDBGJPWC35OQMVR4MEZB4FDOZ4AWGFGHA, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.350959219 (506h42m0.222042264s) [noise=true , address=145.79.227.8:4330 , piece_id=PDB6ALO5ZPYSMSEQHF3NPY3VPWG7XGCXPXA22HWBFRGA5YYA3G5A, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.351321138 (506h42m0.222404173s) [noise=true , address=45.91.116.5:4297 , piece_id=APSDM2AGSPHTXZ36DROJOVMPFVPCHUHY5UBTILU32LERBQQOWK2Q, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.351684470 (506h42m0.222767515s) [noise=true , address=145.79.227.42:4121 , piece_id=H56KQOPDWK557ZZJCWBH7YLWWL3DK24BO3654Q4EWQ6ALZ5AU2UQ, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.352040457 (506h42m0.223123492s) [noise=true , address=45.91.116.8:4047 , piece_id=K5Q2OWU2L6QI5Y77KA7OS5UYOGDYUZ6B6XWURJMSUQXCEUAFC5UA, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.352047270 (506h42m0.223130305s) [noise=true , address=145.79.227.49:4337 , piece_id=BUOWQXWRHHZCKHJPC3D6KNYPHLNNVNSYZCMLG4JICHTJZVFVB7VQ, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.352385835 (506h42m0.22346888s) [noise=true , address=145.79.227.47:4267 , piece_id=XRQQF5K3O3A4V5545MZA2YG2N6EVRG4KIM56A6W6IRRWISTU6WAQ, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.352443936 (506h42m0.223526971s) [noise=true , address=45.91.116.10:4083 , piece_id=OBCQWLK3PCTYKVBP6DPVWXVQPJZVDVHMXS7333U5V5ROKQ2KQEEQ, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.352694222 (506h42m0.223777257s) [noise=true , address=145.79.227.22:4511 , piece_id=S6ZS6SOLNLZXGX7H4CUFN565Y4AYUIZCC2CXBJS6L3CSVB4OC3XQ, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.352808039 (506h42m0.223891084s) [noise=true , address=145.79.227.36:4517 , piece_id=ZL6T3RQT5XNQURM6JLOZ2YGAXNGLYGN5FYHO47IKWPPXZFCUEVBA, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.353054909 (506h42m0.224137944s) [noise=true , address=145.79.227.5:4223 , piece_id=5SXLARFJKURKC7AIF765Z7MWPHOEDHOFMNLCHMQBZUYBOC6AVNOQ, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.353570130 (506h42m0.224653175s) [noise=true , address=145.79.227.38:4978 , piece_id=6UFEMERR2PHRITTQKOGZV3CZCYLHONY36POKRX4TJ3IG3LB4RLQQ, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.354135146 (506h42m0.225218181s) [noise=true , address=45.91.116.11:4119 , piece_id=GA4DRGZTVG4TKE6MPO2YWCWERELOSSF2QZYG5L5MG7TEG5C6TFVA, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.354603829 (506h42m0.225686874s) [noise=true , address=145.79.227.50:4374 , piece_id=TMHRSFYNG6WI6VZF6CER7JWWG4PAAMSJ22QP6RAI64CJO4GAHNGQ, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.355020522 (506h42m0.226103567s) [noise=true , address=45.91.116.6:4974 , piece_id=DLDNZ7WRHTHESIBXZBLTGAQNLJRAJBDKPZHSCQZW6GYUNLYIONDA, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:02.032410415 (506h42m1.90349346s) [upload=444620]: Next returned nil segment Dec 26 18:11:02.032430152 (506h42m1.903513197s) [upload=444620]: Waiting for error group managing segments... Dec 26 18:11:02.348324681 (506h42m2.219407716s) [noise=true , address=145.79.227.22:4511 , piece_id=S6ZS6SOLNLZXGX7H4CUFN565Y4AYUIZCC2CXBJS6L3CSVB4OC3XQ, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.348549048 (506h42m2.219632083s) [noise=true , address=145.79.227.38:4978 , piece_id=6UFEMERR2PHRITTQKOGZV3CZCYLHONY36POKRX4TJ3IG3LB4RLQQ, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.351071442 (506h42m2.222154487s) [noise=true , address=145.79.227.8:4330 , piece_id=PDB6ALO5ZPYSMSEQHF3NPY3VPWG7XGCXPXA22HWBFRGA5YYA3G5A, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.355344347 (506h42m2.226427392s) [noise=true , address=145.79.227.50:4374 , piece_id=TMHRSFYNG6WI6VZF6CER7JWWG4PAAMSJ22QP6RAI64CJO4GAHNGQ, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.359503999 (506h42m2.230587044s) [noise=true , address=145.79.227.49:4337 , piece_id=BUOWQXWRHHZCKHJPC3D6KNYPHLNNVNSYZCMLG4JICHTJZVFVB7VQ, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.361316750 (506h42m2.232399785s) [noise=true , address=145.79.227.5:4223 , piece_id=5SXLARFJKURKC7AIF765Z7MWPHOEDHOFMNLCHMQBZUYBOC6AVNOQ, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.373894200 (506h42m2.244977235s) [noise=true , address=145.79.227.47:4267 , piece_id=XRQQF5K3O3A4V5545MZA2YG2N6EVRG4KIM56A6W6IRRWISTU6WAQ, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.376201063 (506h42m2.247284108s) [noise=true , address=145.79.227.36:4517 , piece_id=ZL6T3RQT5XNQURM6JLOZ2YGAXNGLYGN5FYHO47IKWPPXZFCUEVBA, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.380841749 (506h42m2.251924804s) [noise=true , address=145.79.227.11:4731 , piece_id=NJN4CQGBUBFZWEUDP3KBVU2AVACTXCQ2HE6YJ7W7MKC4E3D4LXIQ, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.388670316 (506h42m2.259753351s) [noise=true , address=145.79.227.42:4121 , piece_id=H56KQOPDWK557ZZJCWBH7YLWWL3DK24BO3654Q4EWQ6ALZ5AU2UQ, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.586101119 (506h42m2.457184164s) [noise=true , address=45.91.116.7:4009 , piece_id=LLOE3CT5REHRQEWDHHSGDBGJPWC35OQMVR4MEZB4FDOZ4AWGFGHA, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.597900427 (506h42m2.468983462s) [noise=true , address=45.91.116.10:4083 , piece_id=OBCQWLK3PCTYKVBP6DPVWXVQPJZVDVHMXS7333U5V5ROKQ2KQEEQ, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.711289730 (506h42m2.582372765s) [noise=true , address=45.91.116.6:4974 , piece_id=DLDNZ7WRHTHESIBXZBLTGAQNLJRAJBDKPZHSCQZW6GYUNLYIONDA, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.711310149 (506h42m2.582393184s) [seg_pos={0 0}, upload=444620]: Segment reached optimal threshold of 13 pieces. Dec 26 18:11:02.711579893 (506h42m2.582662938s) [noise=true , address=45.91.116.11:4119 , piece_id=GA4DRGZTVG4TKE6MPO2YWCWERELOSSF2QZYG5L5MG7TEG5C6TFVA, seg_pos={0 0}, upload=444620]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: read tcp 188.166.225.170:54212->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 18:11:02.711589541 (506h42m2.582672576s) [noise=true , address=45.91.116.11:4119 , piece_id=GA4DRGZTVG4TKE6MPO2YWCWERELOSSF2QZYG5L5MG7TEG5C6TFVA, seg_pos={0 0}, upload=444620]: Long tail cancellation due to optimal threshold Dec 26 18:11:02.711667138 (506h42m2.582750184s) [noise=true , address=45.91.116.8:4047 , piece_id=K5Q2OWU2L6QI5Y77KA7OS5UYOGDYUZ6B6XWURJMSUQXCEUAFC5UA, seg_pos={0 0}, upload=444620]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 26 18:11:02.711691024 (506h42m2.582774069s) [noise=true , address=45.91.116.8:4047 , piece_id=K5Q2OWU2L6QI5Y77KA7OS5UYOGDYUZ6B6XWURJMSUQXCEUAFC5UA, seg_pos={0 0}, upload=444620]: Long tail cancellation due to optimal threshold Dec 26 18:11:02.711887848 (506h42m2.582970913s) [noise=true , address=45.91.116.5:4297 , piece_id=APSDM2AGSPHTXZ36DROJOVMPFVPCHUHY5UBTILU32LERBQQOWK2Q, seg_pos={0 0}, upload=444620]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: write tcp 188.166.225.170:38572->45.91.116.5:4297: use of closed network connection; write tcp 188.166.225.170:38572->45.91.116.5:4297: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38572->45.91.116.5:4297: use of closed network connection Dec 26 18:11:02.711896776 (506h42m2.582979821s) [noise=true , address=45.91.116.5:4297 , piece_id=APSDM2AGSPHTXZ36DROJOVMPFVPCHUHY5UBTILU32LERBQQOWK2Q, seg_pos={0 0}, upload=444620]: Long tail cancellation due to optimal threshold Dec 26 18:11:02.711919389 (506h42m2.583002434s) [seg_pos={0 0}, upload=444620]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 18:11:02.712023507 (506h42m2.583106542s) [upload=444620]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 18:11:02.712031572 (506h42m2.583114607s) [upload=444620]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 18:11:03.015981797 (506h42m2.887064842s) [upload=444620]: Done uploading segments... Dec 26 18:11:03.016006694 (506h42m2.887089739s) [upload=444620]: Upload finished. err: <nil> Duration: 6.407437131s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 44.005804133s trace: https://jaeger.dev.storj.io/trace/665ea65f6e9ccc33 FAIL (2025-12-26 18:14:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== e=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789316979 (506h12m54.660400024s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789326507 (506h12m54.660409552s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789611339 (506h12m54.660694384s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789619275 (506h12m54.66070231s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789914006 (506h12m54.660997041s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789949443 (506h12m54.661032478s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790181566 (506h12m54.661264601s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790295632 (506h12m54.661378668s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790510872 (506h12m54.661593917s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790692980 (506h12m54.661776015s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790787379 (506h12m54.661870414s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790992970 (506h12m54.662076005s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791120053 (506h12m54.662203098s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791275218 (506h12m54.662358253s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791443709 (506h12m54.662526744s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:56.112313145 (506h12m55.98339619s) [upload=444191]: Next returned nil segment Dec 26 17:41:56.112330638 (506h12m55.983413683s) [upload=444191]: Waiting for error group managing segments... Dec 26 17:41:56.485322720 (506h12m56.356405765s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.485637079 (506h12m56.356720124s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.666807539 (506h12m56.537890584s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.678753987 (506h12m56.549837032s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.687788039 (506h12m56.558871074s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.694318023 (506h12m56.565401058s) [noise=true , address=145.79.227.7:4326 , piece_id=DMQ5RPI46X2U7TJTS42ZHQEF32BDJRKJJO5HWOYAPNS6VWVG5B5Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.716581187 (506h12m56.587664232s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.972183007 (506h12m56.843266042s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.011917999 (506h12m56.883001034s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.054168622 (506h12m56.925251667s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.114312777 (506h12m56.985395812s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.499846250 (506h12m57.370929285s) [noise=true , address=145.79.227.34:4919 , piece_id=YRWHW3BJVEGVJRSA6RJB3T7VJQSDTGKEZQKRICFIV2G3T7AN4MIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.612649066 (506h12m57.483732101s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649413733 (506h12m57.520496768s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649440282 (506h12m57.520523328s) [seg_pos={0 0}, upload=444191]: Segment reached optimal threshold of 14 pieces. Dec 26 17:41:57.649674729 (506h12m57.520757774s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G7ty71tPwe7cZSBNK9u74DFUCGomnxDsNvAif4C3SbP7HWwfo): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649684087 (506h12m57.520767142s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649747708 (506h12m57.520830763s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v6zXZuMmyL2MU8qb3q6gg1iwTYvndjzMjVpv8btEpeX9Lnqmu): protocol: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection Dec 26 17:41:57.649785219 (506h12m57.520868274s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649899828 (506h12m57.520982903s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649908775 (506h12m57.52099182s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650074641 (506h12m57.521157696s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.650083287 (506h12m57.521166342s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650132821 (506h12m57.521215866s) [seg_pos={0 0}, upload=444191]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 17:41:57.650220138 (506h12m57.521303183s) [upload=444191]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:41:57.650227171 (506h12m57.521310206s) [upload=444191]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:41:57.958890870 (506h12m57.829973915s) [upload=444191]: Done uploading segments... Dec 26 17:41:57.958911620 (506h12m57.829994665s) [upload=444191]: Upload finished. err: <nil> Duration: 44.005831874s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [...XX..XX......X.XXX.....XX.X.XXX.........X..X.....X...XX...] duration: 6.106300123s trace: https://jaeger.dev.storj.io/trace/3992379534f794fb FAIL (2025-12-26 18:13:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: LLOE3CT5REHRQEWDHHSGDBGJPWC35OQMVR4MEZB4FDOZ4AWGFGHA) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: DLDNZ7WRHTHESIBXZBLTGAQNLJRAJBDKPZHSCQZW6GYUNLYIONDA) context deadline exceeded ======LOGS====== Starting upload Dec 26 18:11:00.104273846 (506h41m59.975356892s) [upload=444620]: Done starting upload Dec 26 18:11:00.104464159 (506h41m59.975547194s) [upload=444620]: Uploading segments... Dec 26 18:11:00.104513854 (506h41m59.975596889s) [upload=444620]: Got next segment. Inline: false Dec 26 18:11:00.104544612 (506h41m59.975627647s) [upload=444620]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 18:11:00.104549912 (506h41m59.975632957s) [upload=444620]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 18:11:00.350198911 (506h42m0.221281966s) [seg_pos={0 0}, upload=444620]: Begin upload segment... Dec 26 18:11:00.350254707 (506h42m0.221337752s) [seg_pos={0 0}, upload=444620]: Done begin upload segment. Dec 26 18:11:00.350321845 (506h42m0.22140489s) [noise=true , address=145.79.227.11:4731 , piece_id=NJN4CQGBUBFZWEUDP3KBVU2AVACTXCQ2HE6YJ7W7MKC4E3D4LXIQ, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.350328608 (506h42m0.221411653s) [noise=true , address=45.91.116.7:4009 , piece_id=LLOE3CT5REHRQEWDHHSGDBGJPWC35OQMVR4MEZB4FDOZ4AWGFGHA, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.350959219 (506h42m0.222042264s) [noise=true , address=145.79.227.8:4330 , piece_id=PDB6ALO5ZPYSMSEQHF3NPY3VPWG7XGCXPXA22HWBFRGA5YYA3G5A, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.351321138 (506h42m0.222404173s) [noise=true , address=45.91.116.5:4297 , piece_id=APSDM2AGSPHTXZ36DROJOVMPFVPCHUHY5UBTILU32LERBQQOWK2Q, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.351684470 (506h42m0.222767515s) [noise=true , address=145.79.227.42:4121 , piece_id=H56KQOPDWK557ZZJCWBH7YLWWL3DK24BO3654Q4EWQ6ALZ5AU2UQ, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.352040457 (506h42m0.223123492s) [noise=true , address=45.91.116.8:4047 , piece_id=K5Q2OWU2L6QI5Y77KA7OS5UYOGDYUZ6B6XWURJMSUQXCEUAFC5UA, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.352047270 (506h42m0.223130305s) [noise=true , address=145.79.227.49:4337 , piece_id=BUOWQXWRHHZCKHJPC3D6KNYPHLNNVNSYZCMLG4JICHTJZVFVB7VQ, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.352385835 (506h42m0.22346888s) [noise=true , address=145.79.227.47:4267 , piece_id=XRQQF5K3O3A4V5545MZA2YG2N6EVRG4KIM56A6W6IRRWISTU6WAQ, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.352443936 (506h42m0.223526971s) [noise=true , address=45.91.116.10:4083 , piece_id=OBCQWLK3PCTYKVBP6DPVWXVQPJZVDVHMXS7333U5V5ROKQ2KQEEQ, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.352694222 (506h42m0.223777257s) [noise=true , address=145.79.227.22:4511 , piece_id=S6ZS6SOLNLZXGX7H4CUFN565Y4AYUIZCC2CXBJS6L3CSVB4OC3XQ, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.352808039 (506h42m0.223891084s) [noise=true , address=145.79.227.36:4517 , piece_id=ZL6T3RQT5XNQURM6JLOZ2YGAXNGLYGN5FYHO47IKWPPXZFCUEVBA, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.353054909 (506h42m0.224137944s) [noise=true , address=145.79.227.5:4223 , piece_id=5SXLARFJKURKC7AIF765Z7MWPHOEDHOFMNLCHMQBZUYBOC6AVNOQ, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.353570130 (506h42m0.224653175s) [noise=true , address=145.79.227.38:4978 , piece_id=6UFEMERR2PHRITTQKOGZV3CZCYLHONY36POKRX4TJ3IG3LB4RLQQ, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.354135146 (506h42m0.225218181s) [noise=true , address=45.91.116.11:4119 , piece_id=GA4DRGZTVG4TKE6MPO2YWCWERELOSSF2QZYG5L5MG7TEG5C6TFVA, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.354603829 (506h42m0.225686874s) [noise=true , address=145.79.227.50:4374 , piece_id=TMHRSFYNG6WI6VZF6CER7JWWG4PAAMSJ22QP6RAI64CJO4GAHNGQ, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.355020522 (506h42m0.226103567s) [noise=true , address=45.91.116.6:4974 , piece_id=DLDNZ7WRHTHESIBXZBLTGAQNLJRAJBDKPZHSCQZW6GYUNLYIONDA, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:02.032410415 (506h42m1.90349346s) [upload=444620]: Next returned nil segment Dec 26 18:11:02.032430152 (506h42m1.903513197s) [upload=444620]: Waiting for error group managing segments... Dec 26 18:11:02.348324681 (506h42m2.219407716s) [noise=true , address=145.79.227.22:4511 , piece_id=S6ZS6SOLNLZXGX7H4CUFN565Y4AYUIZCC2CXBJS6L3CSVB4OC3XQ, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.348549048 (506h42m2.219632083s) [noise=true , address=145.79.227.38:4978 , piece_id=6UFEMERR2PHRITTQKOGZV3CZCYLHONY36POKRX4TJ3IG3LB4RLQQ, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.351071442 (506h42m2.222154487s) [noise=true , address=145.79.227.8:4330 , piece_id=PDB6ALO5ZPYSMSEQHF3NPY3VPWG7XGCXPXA22HWBFRGA5YYA3G5A, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.355344347 (506h42m2.226427392s) [noise=true , address=145.79.227.50:4374 , piece_id=TMHRSFYNG6WI6VZF6CER7JWWG4PAAMSJ22QP6RAI64CJO4GAHNGQ, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.359503999 (506h42m2.230587044s) [noise=true , address=145.79.227.49:4337 , piece_id=BUOWQXWRHHZCKHJPC3D6KNYPHLNNVNSYZCMLG4JICHTJZVFVB7VQ, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.361316750 (506h42m2.232399785s) [noise=true , address=145.79.227.5:4223 , piece_id=5SXLARFJKURKC7AIF765Z7MWPHOEDHOFMNLCHMQBZUYBOC6AVNOQ, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.373894200 (506h42m2.244977235s) [noise=true , address=145.79.227.47:4267 , piece_id=XRQQF5K3O3A4V5545MZA2YG2N6EVRG4KIM56A6W6IRRWISTU6WAQ, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.376201063 (506h42m2.247284108s) [noise=true , address=145.79.227.36:4517 , piece_id=ZL6T3RQT5XNQURM6JLOZ2YGAXNGLYGN5FYHO47IKWPPXZFCUEVBA, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.380841749 (506h42m2.251924804s) [noise=true , address=145.79.227.11:4731 , piece_id=NJN4CQGBUBFZWEUDP3KBVU2AVACTXCQ2HE6YJ7W7MKC4E3D4LXIQ, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.388670316 (506h42m2.259753351s) [noise=true , address=145.79.227.42:4121 , piece_id=H56KQOPDWK557ZZJCWBH7YLWWL3DK24BO3654Q4EWQ6ALZ5AU2UQ, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.586101119 (506h42m2.457184164s) [noise=true , address=45.91.116.7:4009 , piece_id=LLOE3CT5REHRQEWDHHSGDBGJPWC35OQMVR4MEZB4FDOZ4AWGFGHA, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.597900427 (506h42m2.468983462s) [noise=true , address=45.91.116.10:4083 , piece_id=OBCQWLK3PCTYKVBP6DPVWXVQPJZVDVHMXS7333U5V5ROKQ2KQEEQ, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.711289730 (506h42m2.582372765s) [noise=true , address=45.91.116.6:4974 , piece_id=DLDNZ7WRHTHESIBXZBLTGAQNLJRAJBDKPZHSCQZW6GYUNLYIONDA, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.711310149 (506h42m2.582393184s) [seg_pos={0 0}, upload=444620]: Segment reached optimal threshold of 13 pieces. Dec 26 18:11:02.711579893 (506h42m2.582662938s) [noise=true , address=45.91.116.11:4119 , piece_id=GA4DRGZTVG4TKE6MPO2YWCWERELOSSF2QZYG5L5MG7TEG5C6TFVA, seg_pos={0 0}, upload=444620]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: read tcp 188.166.225.170:54212->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 18:11:02.711589541 (506h42m2.582672576s) [noise=true , address=45.91.116.11:4119 , piece_id=GA4DRGZTVG4TKE6MPO2YWCWERELOSSF2QZYG5L5MG7TEG5C6TFVA, seg_pos={0 0}, upload=444620]: Long tail cancellation due to optimal threshold Dec 26 18:11:02.711667138 (506h42m2.582750184s) [noise=true , address=45.91.116.8:4047 , piece_id=K5Q2OWU2L6QI5Y77KA7OS5UYOGDYUZ6B6XWURJMSUQXCEUAFC5UA, seg_pos={0 0}, upload=444620]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 26 18:11:02.711691024 (506h42m2.582774069s) [noise=true , address=45.91.116.8:4047 , piece_id=K5Q2OWU2L6QI5Y77KA7OS5UYOGDYUZ6B6XWURJMSUQXCEUAFC5UA, seg_pos={0 0}, upload=444620]: Long tail cancellation due to optimal threshold Dec 26 18:11:02.711887848 (506h42m2.582970913s) [noise=true , address=45.91.116.5:4297 , piece_id=APSDM2AGSPHTXZ36DROJOVMPFVPCHUHY5UBTILU32LERBQQOWK2Q, seg_pos={0 0}, upload=444620]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: write tcp 188.166.225.170:38572->45.91.116.5:4297: use of closed network connection; write tcp 188.166.225.170:38572->45.91.116.5:4297: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38572->45.91.116.5:4297: use of closed network connection Dec 26 18:11:02.711896776 (506h42m2.582979821s) [noise=true , address=45.91.116.5:4297 , piece_id=APSDM2AGSPHTXZ36DROJOVMPFVPCHUHY5UBTILU32LERBQQOWK2Q, seg_pos={0 0}, upload=444620]: Long tail cancellation due to optimal threshold Dec 26 18:11:02.711919389 (506h42m2.583002434s) [seg_pos={0 0}, upload=444620]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 18:11:02.712023507 (506h42m2.583106542s) [upload=444620]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 18:11:02.712031572 (506h42m2.583114607s) [upload=444620]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 18:11:03.015981797 (506h42m2.887064842s) [upload=444620]: Done uploading segments... Dec 26 18:11:03.016006694 (506h42m2.887089739s) [upload=444620]: Upload finished. err: <nil> Duration: 6.106325991s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 51.179138595s trace: https://jaeger.dev.storj.io/trace/665ea65f6e9ccc33 FAIL (2025-12-26 18:13:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== e=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789316979 (506h12m54.660400024s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789326507 (506h12m54.660409552s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789611339 (506h12m54.660694384s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789619275 (506h12m54.66070231s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789914006 (506h12m54.660997041s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789949443 (506h12m54.661032478s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790181566 (506h12m54.661264601s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790295632 (506h12m54.661378668s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790510872 (506h12m54.661593917s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790692980 (506h12m54.661776015s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790787379 (506h12m54.661870414s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790992970 (506h12m54.662076005s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791120053 (506h12m54.662203098s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791275218 (506h12m54.662358253s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791443709 (506h12m54.662526744s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:56.112313145 (506h12m55.98339619s) [upload=444191]: Next returned nil segment Dec 26 17:41:56.112330638 (506h12m55.983413683s) [upload=444191]: Waiting for error group managing segments... Dec 26 17:41:56.485322720 (506h12m56.356405765s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.485637079 (506h12m56.356720124s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.666807539 (506h12m56.537890584s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.678753987 (506h12m56.549837032s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.687788039 (506h12m56.558871074s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.694318023 (506h12m56.565401058s) [noise=true , address=145.79.227.7:4326 , piece_id=DMQ5RPI46X2U7TJTS42ZHQEF32BDJRKJJO5HWOYAPNS6VWVG5B5Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.716581187 (506h12m56.587664232s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.972183007 (506h12m56.843266042s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.011917999 (506h12m56.883001034s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.054168622 (506h12m56.925251667s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.114312777 (506h12m56.985395812s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.499846250 (506h12m57.370929285s) [noise=true , address=145.79.227.34:4919 , piece_id=YRWHW3BJVEGVJRSA6RJB3T7VJQSDTGKEZQKRICFIV2G3T7AN4MIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.612649066 (506h12m57.483732101s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649413733 (506h12m57.520496768s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649440282 (506h12m57.520523328s) [seg_pos={0 0}, upload=444191]: Segment reached optimal threshold of 14 pieces. Dec 26 17:41:57.649674729 (506h12m57.520757774s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G7ty71tPwe7cZSBNK9u74DFUCGomnxDsNvAif4C3SbP7HWwfo): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649684087 (506h12m57.520767142s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649747708 (506h12m57.520830763s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v6zXZuMmyL2MU8qb3q6gg1iwTYvndjzMjVpv8btEpeX9Lnqmu): protocol: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection Dec 26 17:41:57.649785219 (506h12m57.520868274s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649899828 (506h12m57.520982903s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649908775 (506h12m57.52099182s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650074641 (506h12m57.521157696s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.650083287 (506h12m57.521166342s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650132821 (506h12m57.521215866s) [seg_pos={0 0}, upload=444191]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 17:41:57.650220138 (506h12m57.521303183s) [upload=444191]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:41:57.650227171 (506h12m57.521310206s) [upload=444191]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:41:57.958890870 (506h12m57.829973915s) [upload=444191]: Done uploading segments... Dec 26 17:41:57.958911620 (506h12m57.829994665s) [upload=444191]: Upload finished. err: <nil> Duration: 51.179165567s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [....XX..XX......X.XXX.....XX.X.XXX.........X..X.....X...XX..] duration: 15.923579191s trace: https://jaeger.dev.storj.io/trace/3992379534f794fb FAIL (2025-12-26 18:12:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: LLOE3CT5REHRQEWDHHSGDBGJPWC35OQMVR4MEZB4FDOZ4AWGFGHA) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: DLDNZ7WRHTHESIBXZBLTGAQNLJRAJBDKPZHSCQZW6GYUNLYIONDA) context deadline exceeded ======LOGS====== Starting upload Dec 26 18:11:00.104273846 (506h41m59.975356892s) [upload=444620]: Done starting upload Dec 26 18:11:00.104464159 (506h41m59.975547194s) [upload=444620]: Uploading segments... Dec 26 18:11:00.104513854 (506h41m59.975596889s) [upload=444620]: Got next segment. Inline: false Dec 26 18:11:00.104544612 (506h41m59.975627647s) [upload=444620]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 18:11:00.104549912 (506h41m59.975632957s) [upload=444620]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 18:11:00.350198911 (506h42m0.221281966s) [seg_pos={0 0}, upload=444620]: Begin upload segment... Dec 26 18:11:00.350254707 (506h42m0.221337752s) [seg_pos={0 0}, upload=444620]: Done begin upload segment. Dec 26 18:11:00.350321845 (506h42m0.22140489s) [noise=true , address=145.79.227.11:4731 , piece_id=NJN4CQGBUBFZWEUDP3KBVU2AVACTXCQ2HE6YJ7W7MKC4E3D4LXIQ, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.350328608 (506h42m0.221411653s) [noise=true , address=45.91.116.7:4009 , piece_id=LLOE3CT5REHRQEWDHHSGDBGJPWC35OQMVR4MEZB4FDOZ4AWGFGHA, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.350959219 (506h42m0.222042264s) [noise=true , address=145.79.227.8:4330 , piece_id=PDB6ALO5ZPYSMSEQHF3NPY3VPWG7XGCXPXA22HWBFRGA5YYA3G5A, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.351321138 (506h42m0.222404173s) [noise=true , address=45.91.116.5:4297 , piece_id=APSDM2AGSPHTXZ36DROJOVMPFVPCHUHY5UBTILU32LERBQQOWK2Q, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.351684470 (506h42m0.222767515s) [noise=true , address=145.79.227.42:4121 , piece_id=H56KQOPDWK557ZZJCWBH7YLWWL3DK24BO3654Q4EWQ6ALZ5AU2UQ, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.352040457 (506h42m0.223123492s) [noise=true , address=45.91.116.8:4047 , piece_id=K5Q2OWU2L6QI5Y77KA7OS5UYOGDYUZ6B6XWURJMSUQXCEUAFC5UA, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.352047270 (506h42m0.223130305s) [noise=true , address=145.79.227.49:4337 , piece_id=BUOWQXWRHHZCKHJPC3D6KNYPHLNNVNSYZCMLG4JICHTJZVFVB7VQ, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.352385835 (506h42m0.22346888s) [noise=true , address=145.79.227.47:4267 , piece_id=XRQQF5K3O3A4V5545MZA2YG2N6EVRG4KIM56A6W6IRRWISTU6WAQ, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.352443936 (506h42m0.223526971s) [noise=true , address=45.91.116.10:4083 , piece_id=OBCQWLK3PCTYKVBP6DPVWXVQPJZVDVHMXS7333U5V5ROKQ2KQEEQ, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.352694222 (506h42m0.223777257s) [noise=true , address=145.79.227.22:4511 , piece_id=S6ZS6SOLNLZXGX7H4CUFN565Y4AYUIZCC2CXBJS6L3CSVB4OC3XQ, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.352808039 (506h42m0.223891084s) [noise=true , address=145.79.227.36:4517 , piece_id=ZL6T3RQT5XNQURM6JLOZ2YGAXNGLYGN5FYHO47IKWPPXZFCUEVBA, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.353054909 (506h42m0.224137944s) [noise=true , address=145.79.227.5:4223 , piece_id=5SXLARFJKURKC7AIF765Z7MWPHOEDHOFMNLCHMQBZUYBOC6AVNOQ, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.353570130 (506h42m0.224653175s) [noise=true , address=145.79.227.38:4978 , piece_id=6UFEMERR2PHRITTQKOGZV3CZCYLHONY36POKRX4TJ3IG3LB4RLQQ, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.354135146 (506h42m0.225218181s) [noise=true , address=45.91.116.11:4119 , piece_id=GA4DRGZTVG4TKE6MPO2YWCWERELOSSF2QZYG5L5MG7TEG5C6TFVA, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.354603829 (506h42m0.225686874s) [noise=true , address=145.79.227.50:4374 , piece_id=TMHRSFYNG6WI6VZF6CER7JWWG4PAAMSJ22QP6RAI64CJO4GAHNGQ, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.355020522 (506h42m0.226103567s) [noise=true , address=45.91.116.6:4974 , piece_id=DLDNZ7WRHTHESIBXZBLTGAQNLJRAJBDKPZHSCQZW6GYUNLYIONDA, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:02.032410415 (506h42m1.90349346s) [upload=444620]: Next returned nil segment Dec 26 18:11:02.032430152 (506h42m1.903513197s) [upload=444620]: Waiting for error group managing segments... Dec 26 18:11:02.348324681 (506h42m2.219407716s) [noise=true , address=145.79.227.22:4511 , piece_id=S6ZS6SOLNLZXGX7H4CUFN565Y4AYUIZCC2CXBJS6L3CSVB4OC3XQ, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.348549048 (506h42m2.219632083s) [noise=true , address=145.79.227.38:4978 , piece_id=6UFEMERR2PHRITTQKOGZV3CZCYLHONY36POKRX4TJ3IG3LB4RLQQ, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.351071442 (506h42m2.222154487s) [noise=true , address=145.79.227.8:4330 , piece_id=PDB6ALO5ZPYSMSEQHF3NPY3VPWG7XGCXPXA22HWBFRGA5YYA3G5A, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.355344347 (506h42m2.226427392s) [noise=true , address=145.79.227.50:4374 , piece_id=TMHRSFYNG6WI6VZF6CER7JWWG4PAAMSJ22QP6RAI64CJO4GAHNGQ, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.359503999 (506h42m2.230587044s) [noise=true , address=145.79.227.49:4337 , piece_id=BUOWQXWRHHZCKHJPC3D6KNYPHLNNVNSYZCMLG4JICHTJZVFVB7VQ, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.361316750 (506h42m2.232399785s) [noise=true , address=145.79.227.5:4223 , piece_id=5SXLARFJKURKC7AIF765Z7MWPHOEDHOFMNLCHMQBZUYBOC6AVNOQ, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.373894200 (506h42m2.244977235s) [noise=true , address=145.79.227.47:4267 , piece_id=XRQQF5K3O3A4V5545MZA2YG2N6EVRG4KIM56A6W6IRRWISTU6WAQ, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.376201063 (506h42m2.247284108s) [noise=true , address=145.79.227.36:4517 , piece_id=ZL6T3RQT5XNQURM6JLOZ2YGAXNGLYGN5FYHO47IKWPPXZFCUEVBA, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.380841749 (506h42m2.251924804s) [noise=true , address=145.79.227.11:4731 , piece_id=NJN4CQGBUBFZWEUDP3KBVU2AVACTXCQ2HE6YJ7W7MKC4E3D4LXIQ, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.388670316 (506h42m2.259753351s) [noise=true , address=145.79.227.42:4121 , piece_id=H56KQOPDWK557ZZJCWBH7YLWWL3DK24BO3654Q4EWQ6ALZ5AU2UQ, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.586101119 (506h42m2.457184164s) [noise=true , address=45.91.116.7:4009 , piece_id=LLOE3CT5REHRQEWDHHSGDBGJPWC35OQMVR4MEZB4FDOZ4AWGFGHA, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.597900427 (506h42m2.468983462s) [noise=true , address=45.91.116.10:4083 , piece_id=OBCQWLK3PCTYKVBP6DPVWXVQPJZVDVHMXS7333U5V5ROKQ2KQEEQ, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.711289730 (506h42m2.582372765s) [noise=true , address=45.91.116.6:4974 , piece_id=DLDNZ7WRHTHESIBXZBLTGAQNLJRAJBDKPZHSCQZW6GYUNLYIONDA, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.711310149 (506h42m2.582393184s) [seg_pos={0 0}, upload=444620]: Segment reached optimal threshold of 13 pieces. Dec 26 18:11:02.711579893 (506h42m2.582662938s) [noise=true , address=45.91.116.11:4119 , piece_id=GA4DRGZTVG4TKE6MPO2YWCWERELOSSF2QZYG5L5MG7TEG5C6TFVA, seg_pos={0 0}, upload=444620]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: read tcp 188.166.225.170:54212->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 18:11:02.711589541 (506h42m2.582672576s) [noise=true , address=45.91.116.11:4119 , piece_id=GA4DRGZTVG4TKE6MPO2YWCWERELOSSF2QZYG5L5MG7TEG5C6TFVA, seg_pos={0 0}, upload=444620]: Long tail cancellation due to optimal threshold Dec 26 18:11:02.711667138 (506h42m2.582750184s) [noise=true , address=45.91.116.8:4047 , piece_id=K5Q2OWU2L6QI5Y77KA7OS5UYOGDYUZ6B6XWURJMSUQXCEUAFC5UA, seg_pos={0 0}, upload=444620]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 26 18:11:02.711691024 (506h42m2.582774069s) [noise=true , address=45.91.116.8:4047 , piece_id=K5Q2OWU2L6QI5Y77KA7OS5UYOGDYUZ6B6XWURJMSUQXCEUAFC5UA, seg_pos={0 0}, upload=444620]: Long tail cancellation due to optimal threshold Dec 26 18:11:02.711887848 (506h42m2.582970913s) [noise=true , address=45.91.116.5:4297 , piece_id=APSDM2AGSPHTXZ36DROJOVMPFVPCHUHY5UBTILU32LERBQQOWK2Q, seg_pos={0 0}, upload=444620]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: write tcp 188.166.225.170:38572->45.91.116.5:4297: use of closed network connection; write tcp 188.166.225.170:38572->45.91.116.5:4297: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38572->45.91.116.5:4297: use of closed network connection Dec 26 18:11:02.711896776 (506h42m2.582979821s) [noise=true , address=45.91.116.5:4297 , piece_id=APSDM2AGSPHTXZ36DROJOVMPFVPCHUHY5UBTILU32LERBQQOWK2Q, seg_pos={0 0}, upload=444620]: Long tail cancellation due to optimal threshold Dec 26 18:11:02.711919389 (506h42m2.583002434s) [seg_pos={0 0}, upload=444620]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 18:11:02.712023507 (506h42m2.583106542s) [upload=444620]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 18:11:02.712031572 (506h42m2.583114607s) [upload=444620]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 18:11:03.015981797 (506h42m2.887064842s) [upload=444620]: Done uploading segments... Dec 26 18:11:03.016006694 (506h42m2.887089739s) [upload=444620]: Upload finished. err: <nil> Duration: 15.923609841s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 50.176314934s trace: https://jaeger.dev.storj.io/trace/665ea65f6e9ccc33 FAIL (2025-12-26 18:12:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== e=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789316979 (506h12m54.660400024s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789326507 (506h12m54.660409552s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789611339 (506h12m54.660694384s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789619275 (506h12m54.66070231s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789914006 (506h12m54.660997041s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789949443 (506h12m54.661032478s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790181566 (506h12m54.661264601s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790295632 (506h12m54.661378668s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790510872 (506h12m54.661593917s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790692980 (506h12m54.661776015s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790787379 (506h12m54.661870414s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790992970 (506h12m54.662076005s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791120053 (506h12m54.662203098s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791275218 (506h12m54.662358253s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791443709 (506h12m54.662526744s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:56.112313145 (506h12m55.98339619s) [upload=444191]: Next returned nil segment Dec 26 17:41:56.112330638 (506h12m55.983413683s) [upload=444191]: Waiting for error group managing segments... Dec 26 17:41:56.485322720 (506h12m56.356405765s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.485637079 (506h12m56.356720124s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.666807539 (506h12m56.537890584s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.678753987 (506h12m56.549837032s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.687788039 (506h12m56.558871074s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.694318023 (506h12m56.565401058s) [noise=true , address=145.79.227.7:4326 , piece_id=DMQ5RPI46X2U7TJTS42ZHQEF32BDJRKJJO5HWOYAPNS6VWVG5B5Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.716581187 (506h12m56.587664232s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.972183007 (506h12m56.843266042s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.011917999 (506h12m56.883001034s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.054168622 (506h12m56.925251667s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.114312777 (506h12m56.985395812s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.499846250 (506h12m57.370929285s) [noise=true , address=145.79.227.34:4919 , piece_id=YRWHW3BJVEGVJRSA6RJB3T7VJQSDTGKEZQKRICFIV2G3T7AN4MIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.612649066 (506h12m57.483732101s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649413733 (506h12m57.520496768s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649440282 (506h12m57.520523328s) [seg_pos={0 0}, upload=444191]: Segment reached optimal threshold of 14 pieces. Dec 26 17:41:57.649674729 (506h12m57.520757774s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G7ty71tPwe7cZSBNK9u74DFUCGomnxDsNvAif4C3SbP7HWwfo): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649684087 (506h12m57.520767142s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649747708 (506h12m57.520830763s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v6zXZuMmyL2MU8qb3q6gg1iwTYvndjzMjVpv8btEpeX9Lnqmu): protocol: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection Dec 26 17:41:57.649785219 (506h12m57.520868274s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649899828 (506h12m57.520982903s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649908775 (506h12m57.52099182s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650074641 (506h12m57.521157696s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.650083287 (506h12m57.521166342s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650132821 (506h12m57.521215866s) [seg_pos={0 0}, upload=444191]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 17:41:57.650220138 (506h12m57.521303183s) [upload=444191]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:41:57.650227171 (506h12m57.521310206s) [upload=444191]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:41:57.958890870 (506h12m57.829973915s) [upload=444191]: Done uploading segments... Dec 26 17:41:57.958911620 (506h12m57.829994665s) [upload=444191]: Upload finished. err: <nil> Duration: 50.176344389s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [X....XX..XX......X.XXX.....XX.X.XXX.........X..X.....X...XX.] duration: 5.762285269s trace: https://jaeger.dev.storj.io/trace/3992379534f794fb FAIL (2025-12-26 18:12:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: LLOE3CT5REHRQEWDHHSGDBGJPWC35OQMVR4MEZB4FDOZ4AWGFGHA) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: DLDNZ7WRHTHESIBXZBLTGAQNLJRAJBDKPZHSCQZW6GYUNLYIONDA) context deadline exceeded ======LOGS====== Starting upload Dec 26 18:11:00.104273846 (506h41m59.975356892s) [upload=444620]: Done starting upload Dec 26 18:11:00.104464159 (506h41m59.975547194s) [upload=444620]: Uploading segments... Dec 26 18:11:00.104513854 (506h41m59.975596889s) [upload=444620]: Got next segment. Inline: false Dec 26 18:11:00.104544612 (506h41m59.975627647s) [upload=444620]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 18:11:00.104549912 (506h41m59.975632957s) [upload=444620]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 18:11:00.350198911 (506h42m0.221281966s) [seg_pos={0 0}, upload=444620]: Begin upload segment... Dec 26 18:11:00.350254707 (506h42m0.221337752s) [seg_pos={0 0}, upload=444620]: Done begin upload segment. Dec 26 18:11:00.350321845 (506h42m0.22140489s) [noise=true , address=145.79.227.11:4731 , piece_id=NJN4CQGBUBFZWEUDP3KBVU2AVACTXCQ2HE6YJ7W7MKC4E3D4LXIQ, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.350328608 (506h42m0.221411653s) [noise=true , address=45.91.116.7:4009 , piece_id=LLOE3CT5REHRQEWDHHSGDBGJPWC35OQMVR4MEZB4FDOZ4AWGFGHA, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.350959219 (506h42m0.222042264s) [noise=true , address=145.79.227.8:4330 , piece_id=PDB6ALO5ZPYSMSEQHF3NPY3VPWG7XGCXPXA22HWBFRGA5YYA3G5A, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.351321138 (506h42m0.222404173s) [noise=true , address=45.91.116.5:4297 , piece_id=APSDM2AGSPHTXZ36DROJOVMPFVPCHUHY5UBTILU32LERBQQOWK2Q, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.351684470 (506h42m0.222767515s) [noise=true , address=145.79.227.42:4121 , piece_id=H56KQOPDWK557ZZJCWBH7YLWWL3DK24BO3654Q4EWQ6ALZ5AU2UQ, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.352040457 (506h42m0.223123492s) [noise=true , address=45.91.116.8:4047 , piece_id=K5Q2OWU2L6QI5Y77KA7OS5UYOGDYUZ6B6XWURJMSUQXCEUAFC5UA, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.352047270 (506h42m0.223130305s) [noise=true , address=145.79.227.49:4337 , piece_id=BUOWQXWRHHZCKHJPC3D6KNYPHLNNVNSYZCMLG4JICHTJZVFVB7VQ, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.352385835 (506h42m0.22346888s) [noise=true , address=145.79.227.47:4267 , piece_id=XRQQF5K3O3A4V5545MZA2YG2N6EVRG4KIM56A6W6IRRWISTU6WAQ, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.352443936 (506h42m0.223526971s) [noise=true , address=45.91.116.10:4083 , piece_id=OBCQWLK3PCTYKVBP6DPVWXVQPJZVDVHMXS7333U5V5ROKQ2KQEEQ, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.352694222 (506h42m0.223777257s) [noise=true , address=145.79.227.22:4511 , piece_id=S6ZS6SOLNLZXGX7H4CUFN565Y4AYUIZCC2CXBJS6L3CSVB4OC3XQ, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.352808039 (506h42m0.223891084s) [noise=true , address=145.79.227.36:4517 , piece_id=ZL6T3RQT5XNQURM6JLOZ2YGAXNGLYGN5FYHO47IKWPPXZFCUEVBA, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.353054909 (506h42m0.224137944s) [noise=true , address=145.79.227.5:4223 , piece_id=5SXLARFJKURKC7AIF765Z7MWPHOEDHOFMNLCHMQBZUYBOC6AVNOQ, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.353570130 (506h42m0.224653175s) [noise=true , address=145.79.227.38:4978 , piece_id=6UFEMERR2PHRITTQKOGZV3CZCYLHONY36POKRX4TJ3IG3LB4RLQQ, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.354135146 (506h42m0.225218181s) [noise=true , address=45.91.116.11:4119 , piece_id=GA4DRGZTVG4TKE6MPO2YWCWERELOSSF2QZYG5L5MG7TEG5C6TFVA, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.354603829 (506h42m0.225686874s) [noise=true , address=145.79.227.50:4374 , piece_id=TMHRSFYNG6WI6VZF6CER7JWWG4PAAMSJ22QP6RAI64CJO4GAHNGQ, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.355020522 (506h42m0.226103567s) [noise=true , address=45.91.116.6:4974 , piece_id=DLDNZ7WRHTHESIBXZBLTGAQNLJRAJBDKPZHSCQZW6GYUNLYIONDA, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:02.032410415 (506h42m1.90349346s) [upload=444620]: Next returned nil segment Dec 26 18:11:02.032430152 (506h42m1.903513197s) [upload=444620]: Waiting for error group managing segments... Dec 26 18:11:02.348324681 (506h42m2.219407716s) [noise=true , address=145.79.227.22:4511 , piece_id=S6ZS6SOLNLZXGX7H4CUFN565Y4AYUIZCC2CXBJS6L3CSVB4OC3XQ, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.348549048 (506h42m2.219632083s) [noise=true , address=145.79.227.38:4978 , piece_id=6UFEMERR2PHRITTQKOGZV3CZCYLHONY36POKRX4TJ3IG3LB4RLQQ, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.351071442 (506h42m2.222154487s) [noise=true , address=145.79.227.8:4330 , piece_id=PDB6ALO5ZPYSMSEQHF3NPY3VPWG7XGCXPXA22HWBFRGA5YYA3G5A, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.355344347 (506h42m2.226427392s) [noise=true , address=145.79.227.50:4374 , piece_id=TMHRSFYNG6WI6VZF6CER7JWWG4PAAMSJ22QP6RAI64CJO4GAHNGQ, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.359503999 (506h42m2.230587044s) [noise=true , address=145.79.227.49:4337 , piece_id=BUOWQXWRHHZCKHJPC3D6KNYPHLNNVNSYZCMLG4JICHTJZVFVB7VQ, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.361316750 (506h42m2.232399785s) [noise=true , address=145.79.227.5:4223 , piece_id=5SXLARFJKURKC7AIF765Z7MWPHOEDHOFMNLCHMQBZUYBOC6AVNOQ, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.373894200 (506h42m2.244977235s) [noise=true , address=145.79.227.47:4267 , piece_id=XRQQF5K3O3A4V5545MZA2YG2N6EVRG4KIM56A6W6IRRWISTU6WAQ, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.376201063 (506h42m2.247284108s) [noise=true , address=145.79.227.36:4517 , piece_id=ZL6T3RQT5XNQURM6JLOZ2YGAXNGLYGN5FYHO47IKWPPXZFCUEVBA, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.380841749 (506h42m2.251924804s) [noise=true , address=145.79.227.11:4731 , piece_id=NJN4CQGBUBFZWEUDP3KBVU2AVACTXCQ2HE6YJ7W7MKC4E3D4LXIQ, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.388670316 (506h42m2.259753351s) [noise=true , address=145.79.227.42:4121 , piece_id=H56KQOPDWK557ZZJCWBH7YLWWL3DK24BO3654Q4EWQ6ALZ5AU2UQ, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.586101119 (506h42m2.457184164s) [noise=true , address=45.91.116.7:4009 , piece_id=LLOE3CT5REHRQEWDHHSGDBGJPWC35OQMVR4MEZB4FDOZ4AWGFGHA, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.597900427 (506h42m2.468983462s) [noise=true , address=45.91.116.10:4083 , piece_id=OBCQWLK3PCTYKVBP6DPVWXVQPJZVDVHMXS7333U5V5ROKQ2KQEEQ, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.711289730 (506h42m2.582372765s) [noise=true , address=45.91.116.6:4974 , piece_id=DLDNZ7WRHTHESIBXZBLTGAQNLJRAJBDKPZHSCQZW6GYUNLYIONDA, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.711310149 (506h42m2.582393184s) [seg_pos={0 0}, upload=444620]: Segment reached optimal threshold of 13 pieces. Dec 26 18:11:02.711579893 (506h42m2.582662938s) [noise=true , address=45.91.116.11:4119 , piece_id=GA4DRGZTVG4TKE6MPO2YWCWERELOSSF2QZYG5L5MG7TEG5C6TFVA, seg_pos={0 0}, upload=444620]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: read tcp 188.166.225.170:54212->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 18:11:02.711589541 (506h42m2.582672576s) [noise=true , address=45.91.116.11:4119 , piece_id=GA4DRGZTVG4TKE6MPO2YWCWERELOSSF2QZYG5L5MG7TEG5C6TFVA, seg_pos={0 0}, upload=444620]: Long tail cancellation due to optimal threshold Dec 26 18:11:02.711667138 (506h42m2.582750184s) [noise=true , address=45.91.116.8:4047 , piece_id=K5Q2OWU2L6QI5Y77KA7OS5UYOGDYUZ6B6XWURJMSUQXCEUAFC5UA, seg_pos={0 0}, upload=444620]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 26 18:11:02.711691024 (506h42m2.582774069s) [noise=true , address=45.91.116.8:4047 , piece_id=K5Q2OWU2L6QI5Y77KA7OS5UYOGDYUZ6B6XWURJMSUQXCEUAFC5UA, seg_pos={0 0}, upload=444620]: Long tail cancellation due to optimal threshold Dec 26 18:11:02.711887848 (506h42m2.582970913s) [noise=true , address=45.91.116.5:4297 , piece_id=APSDM2AGSPHTXZ36DROJOVMPFVPCHUHY5UBTILU32LERBQQOWK2Q, seg_pos={0 0}, upload=444620]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: write tcp 188.166.225.170:38572->45.91.116.5:4297: use of closed network connection; write tcp 188.166.225.170:38572->45.91.116.5:4297: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38572->45.91.116.5:4297: use of closed network connection Dec 26 18:11:02.711896776 (506h42m2.582979821s) [noise=true , address=45.91.116.5:4297 , piece_id=APSDM2AGSPHTXZ36DROJOVMPFVPCHUHY5UBTILU32LERBQQOWK2Q, seg_pos={0 0}, upload=444620]: Long tail cancellation due to optimal threshold Dec 26 18:11:02.711919389 (506h42m2.583002434s) [seg_pos={0 0}, upload=444620]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 18:11:02.712023507 (506h42m2.583106542s) [upload=444620]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 18:11:02.712031572 (506h42m2.583114607s) [upload=444620]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 18:11:03.015981797 (506h42m2.887064842s) [upload=444620]: Done uploading segments... Dec 26 18:11:03.016006694 (506h42m2.887089739s) [upload=444620]: Upload finished. err: <nil> Duration: 5.762312601s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 43.495578161s trace: https://jaeger.dev.storj.io/trace/665ea65f6e9ccc33 FAIL (2025-12-26 18:11:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== e=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789316979 (506h12m54.660400024s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789326507 (506h12m54.660409552s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789611339 (506h12m54.660694384s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789619275 (506h12m54.66070231s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789914006 (506h12m54.660997041s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789949443 (506h12m54.661032478s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790181566 (506h12m54.661264601s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790295632 (506h12m54.661378668s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790510872 (506h12m54.661593917s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790692980 (506h12m54.661776015s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790787379 (506h12m54.661870414s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790992970 (506h12m54.662076005s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791120053 (506h12m54.662203098s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791275218 (506h12m54.662358253s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791443709 (506h12m54.662526744s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:56.112313145 (506h12m55.98339619s) [upload=444191]: Next returned nil segment Dec 26 17:41:56.112330638 (506h12m55.983413683s) [upload=444191]: Waiting for error group managing segments... Dec 26 17:41:56.485322720 (506h12m56.356405765s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.485637079 (506h12m56.356720124s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.666807539 (506h12m56.537890584s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.678753987 (506h12m56.549837032s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.687788039 (506h12m56.558871074s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.694318023 (506h12m56.565401058s) [noise=true , address=145.79.227.7:4326 , piece_id=DMQ5RPI46X2U7TJTS42ZHQEF32BDJRKJJO5HWOYAPNS6VWVG5B5Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.716581187 (506h12m56.587664232s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.972183007 (506h12m56.843266042s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.011917999 (506h12m56.883001034s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.054168622 (506h12m56.925251667s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.114312777 (506h12m56.985395812s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.499846250 (506h12m57.370929285s) [noise=true , address=145.79.227.34:4919 , piece_id=YRWHW3BJVEGVJRSA6RJB3T7VJQSDTGKEZQKRICFIV2G3T7AN4MIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.612649066 (506h12m57.483732101s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649413733 (506h12m57.520496768s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649440282 (506h12m57.520523328s) [seg_pos={0 0}, upload=444191]: Segment reached optimal threshold of 14 pieces. Dec 26 17:41:57.649674729 (506h12m57.520757774s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G7ty71tPwe7cZSBNK9u74DFUCGomnxDsNvAif4C3SbP7HWwfo): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649684087 (506h12m57.520767142s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649747708 (506h12m57.520830763s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v6zXZuMmyL2MU8qb3q6gg1iwTYvndjzMjVpv8btEpeX9Lnqmu): protocol: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection Dec 26 17:41:57.649785219 (506h12m57.520868274s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649899828 (506h12m57.520982903s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649908775 (506h12m57.52099182s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650074641 (506h12m57.521157696s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.650083287 (506h12m57.521166342s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650132821 (506h12m57.521215866s) [seg_pos={0 0}, upload=444191]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 17:41:57.650220138 (506h12m57.521303183s) [upload=444191]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:41:57.650227171 (506h12m57.521310206s) [upload=444191]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:41:57.958890870 (506h12m57.829973915s) [upload=444191]: Done uploading segments... Dec 26 17:41:57.958911620 (506h12m57.829994665s) [upload=444191]: Upload finished. err: <nil> Duration: 43.495595193s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [XX....XX..XX......X.XXX.....XX.X.XXX.........X..X.....X...XX] duration: 30.003665037s trace: https://jaeger.dev.storj.io/trace/3992379534f794fb FAIL (2025-12-26 18:11:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: LLOE3CT5REHRQEWDHHSGDBGJPWC35OQMVR4MEZB4FDOZ4AWGFGHA) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: DLDNZ7WRHTHESIBXZBLTGAQNLJRAJBDKPZHSCQZW6GYUNLYIONDA) context deadline exceeded ======LOGS====== Starting upload Dec 26 18:11:00.104273846 (506h41m59.975356892s) [upload=444620]: Done starting upload Dec 26 18:11:00.104464159 (506h41m59.975547194s) [upload=444620]: Uploading segments... Dec 26 18:11:00.104513854 (506h41m59.975596889s) [upload=444620]: Got next segment. Inline: false Dec 26 18:11:00.104544612 (506h41m59.975627647s) [upload=444620]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 18:11:00.104549912 (506h41m59.975632957s) [upload=444620]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 18:11:00.350198911 (506h42m0.221281966s) [seg_pos={0 0}, upload=444620]: Begin upload segment... Dec 26 18:11:00.350254707 (506h42m0.221337752s) [seg_pos={0 0}, upload=444620]: Done begin upload segment. Dec 26 18:11:00.350321845 (506h42m0.22140489s) [noise=true , address=145.79.227.11:4731 , piece_id=NJN4CQGBUBFZWEUDP3KBVU2AVACTXCQ2HE6YJ7W7MKC4E3D4LXIQ, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.350328608 (506h42m0.221411653s) [noise=true , address=45.91.116.7:4009 , piece_id=LLOE3CT5REHRQEWDHHSGDBGJPWC35OQMVR4MEZB4FDOZ4AWGFGHA, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.350959219 (506h42m0.222042264s) [noise=true , address=145.79.227.8:4330 , piece_id=PDB6ALO5ZPYSMSEQHF3NPY3VPWG7XGCXPXA22HWBFRGA5YYA3G5A, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.351321138 (506h42m0.222404173s) [noise=true , address=45.91.116.5:4297 , piece_id=APSDM2AGSPHTXZ36DROJOVMPFVPCHUHY5UBTILU32LERBQQOWK2Q, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.351684470 (506h42m0.222767515s) [noise=true , address=145.79.227.42:4121 , piece_id=H56KQOPDWK557ZZJCWBH7YLWWL3DK24BO3654Q4EWQ6ALZ5AU2UQ, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.352040457 (506h42m0.223123492s) [noise=true , address=45.91.116.8:4047 , piece_id=K5Q2OWU2L6QI5Y77KA7OS5UYOGDYUZ6B6XWURJMSUQXCEUAFC5UA, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.352047270 (506h42m0.223130305s) [noise=true , address=145.79.227.49:4337 , piece_id=BUOWQXWRHHZCKHJPC3D6KNYPHLNNVNSYZCMLG4JICHTJZVFVB7VQ, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.352385835 (506h42m0.22346888s) [noise=true , address=145.79.227.47:4267 , piece_id=XRQQF5K3O3A4V5545MZA2YG2N6EVRG4KIM56A6W6IRRWISTU6WAQ, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.352443936 (506h42m0.223526971s) [noise=true , address=45.91.116.10:4083 , piece_id=OBCQWLK3PCTYKVBP6DPVWXVQPJZVDVHMXS7333U5V5ROKQ2KQEEQ, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.352694222 (506h42m0.223777257s) [noise=true , address=145.79.227.22:4511 , piece_id=S6ZS6SOLNLZXGX7H4CUFN565Y4AYUIZCC2CXBJS6L3CSVB4OC3XQ, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.352808039 (506h42m0.223891084s) [noise=true , address=145.79.227.36:4517 , piece_id=ZL6T3RQT5XNQURM6JLOZ2YGAXNGLYGN5FYHO47IKWPPXZFCUEVBA, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.353054909 (506h42m0.224137944s) [noise=true , address=145.79.227.5:4223 , piece_id=5SXLARFJKURKC7AIF765Z7MWPHOEDHOFMNLCHMQBZUYBOC6AVNOQ, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.353570130 (506h42m0.224653175s) [noise=true , address=145.79.227.38:4978 , piece_id=6UFEMERR2PHRITTQKOGZV3CZCYLHONY36POKRX4TJ3IG3LB4RLQQ, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.354135146 (506h42m0.225218181s) [noise=true , address=45.91.116.11:4119 , piece_id=GA4DRGZTVG4TKE6MPO2YWCWERELOSSF2QZYG5L5MG7TEG5C6TFVA, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.354603829 (506h42m0.225686874s) [noise=true , address=145.79.227.50:4374 , piece_id=TMHRSFYNG6WI6VZF6CER7JWWG4PAAMSJ22QP6RAI64CJO4GAHNGQ, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.355020522 (506h42m0.226103567s) [noise=true , address=45.91.116.6:4974 , piece_id=DLDNZ7WRHTHESIBXZBLTGAQNLJRAJBDKPZHSCQZW6GYUNLYIONDA, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:02.032410415 (506h42m1.90349346s) [upload=444620]: Next returned nil segment Dec 26 18:11:02.032430152 (506h42m1.903513197s) [upload=444620]: Waiting for error group managing segments... Dec 26 18:11:02.348324681 (506h42m2.219407716s) [noise=true , address=145.79.227.22:4511 , piece_id=S6ZS6SOLNLZXGX7H4CUFN565Y4AYUIZCC2CXBJS6L3CSVB4OC3XQ, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.348549048 (506h42m2.219632083s) [noise=true , address=145.79.227.38:4978 , piece_id=6UFEMERR2PHRITTQKOGZV3CZCYLHONY36POKRX4TJ3IG3LB4RLQQ, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.351071442 (506h42m2.222154487s) [noise=true , address=145.79.227.8:4330 , piece_id=PDB6ALO5ZPYSMSEQHF3NPY3VPWG7XGCXPXA22HWBFRGA5YYA3G5A, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.355344347 (506h42m2.226427392s) [noise=true , address=145.79.227.50:4374 , piece_id=TMHRSFYNG6WI6VZF6CER7JWWG4PAAMSJ22QP6RAI64CJO4GAHNGQ, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.359503999 (506h42m2.230587044s) [noise=true , address=145.79.227.49:4337 , piece_id=BUOWQXWRHHZCKHJPC3D6KNYPHLNNVNSYZCMLG4JICHTJZVFVB7VQ, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.361316750 (506h42m2.232399785s) [noise=true , address=145.79.227.5:4223 , piece_id=5SXLARFJKURKC7AIF765Z7MWPHOEDHOFMNLCHMQBZUYBOC6AVNOQ, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.373894200 (506h42m2.244977235s) [noise=true , address=145.79.227.47:4267 , piece_id=XRQQF5K3O3A4V5545MZA2YG2N6EVRG4KIM56A6W6IRRWISTU6WAQ, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.376201063 (506h42m2.247284108s) [noise=true , address=145.79.227.36:4517 , piece_id=ZL6T3RQT5XNQURM6JLOZ2YGAXNGLYGN5FYHO47IKWPPXZFCUEVBA, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.380841749 (506h42m2.251924804s) [noise=true , address=145.79.227.11:4731 , piece_id=NJN4CQGBUBFZWEUDP3KBVU2AVACTXCQ2HE6YJ7W7MKC4E3D4LXIQ, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.388670316 (506h42m2.259753351s) [noise=true , address=145.79.227.42:4121 , piece_id=H56KQOPDWK557ZZJCWBH7YLWWL3DK24BO3654Q4EWQ6ALZ5AU2UQ, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.586101119 (506h42m2.457184164s) [noise=true , address=45.91.116.7:4009 , piece_id=LLOE3CT5REHRQEWDHHSGDBGJPWC35OQMVR4MEZB4FDOZ4AWGFGHA, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.597900427 (506h42m2.468983462s) [noise=true , address=45.91.116.10:4083 , piece_id=OBCQWLK3PCTYKVBP6DPVWXVQPJZVDVHMXS7333U5V5ROKQ2KQEEQ, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.711289730 (506h42m2.582372765s) [noise=true , address=45.91.116.6:4974 , piece_id=DLDNZ7WRHTHESIBXZBLTGAQNLJRAJBDKPZHSCQZW6GYUNLYIONDA, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.711310149 (506h42m2.582393184s) [seg_pos={0 0}, upload=444620]: Segment reached optimal threshold of 13 pieces. Dec 26 18:11:02.711579893 (506h42m2.582662938s) [noise=true , address=45.91.116.11:4119 , piece_id=GA4DRGZTVG4TKE6MPO2YWCWERELOSSF2QZYG5L5MG7TEG5C6TFVA, seg_pos={0 0}, upload=444620]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: read tcp 188.166.225.170:54212->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 18:11:02.711589541 (506h42m2.582672576s) [noise=true , address=45.91.116.11:4119 , piece_id=GA4DRGZTVG4TKE6MPO2YWCWERELOSSF2QZYG5L5MG7TEG5C6TFVA, seg_pos={0 0}, upload=444620]: Long tail cancellation due to optimal threshold Dec 26 18:11:02.711667138 (506h42m2.582750184s) [noise=true , address=45.91.116.8:4047 , piece_id=K5Q2OWU2L6QI5Y77KA7OS5UYOGDYUZ6B6XWURJMSUQXCEUAFC5UA, seg_pos={0 0}, upload=444620]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 26 18:11:02.711691024 (506h42m2.582774069s) [noise=true , address=45.91.116.8:4047 , piece_id=K5Q2OWU2L6QI5Y77KA7OS5UYOGDYUZ6B6XWURJMSUQXCEUAFC5UA, seg_pos={0 0}, upload=444620]: Long tail cancellation due to optimal threshold Dec 26 18:11:02.711887848 (506h42m2.582970913s) [noise=true , address=45.91.116.5:4297 , piece_id=APSDM2AGSPHTXZ36DROJOVMPFVPCHUHY5UBTILU32LERBQQOWK2Q, seg_pos={0 0}, upload=444620]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: write tcp 188.166.225.170:38572->45.91.116.5:4297: use of closed network connection; write tcp 188.166.225.170:38572->45.91.116.5:4297: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38572->45.91.116.5:4297: use of closed network connection Dec 26 18:11:02.711896776 (506h42m2.582979821s) [noise=true , address=45.91.116.5:4297 , piece_id=APSDM2AGSPHTXZ36DROJOVMPFVPCHUHY5UBTILU32LERBQQOWK2Q, seg_pos={0 0}, upload=444620]: Long tail cancellation due to optimal threshold Dec 26 18:11:02.711919389 (506h42m2.583002434s) [seg_pos={0 0}, upload=444620]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 18:11:02.712023507 (506h42m2.583106542s) [upload=444620]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 18:11:02.712031572 (506h42m2.583114607s) [upload=444620]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 18:11:03.015981797 (506h42m2.887064842s) [upload=444620]: Done uploading segments... Dec 26 18:11:03.016006694 (506h42m2.887089739s) [upload=444620]: Upload finished. err: <nil> Duration: 30.003686869s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 46.809534705s trace: https://jaeger.dev.storj.io/trace/665ea65f6e9ccc33 FAIL (2025-12-26 18:10:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== e=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789316979 (506h12m54.660400024s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789326507 (506h12m54.660409552s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789611339 (506h12m54.660694384s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789619275 (506h12m54.66070231s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789914006 (506h12m54.660997041s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789949443 (506h12m54.661032478s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790181566 (506h12m54.661264601s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790295632 (506h12m54.661378668s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790510872 (506h12m54.661593917s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790692980 (506h12m54.661776015s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790787379 (506h12m54.661870414s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790992970 (506h12m54.662076005s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791120053 (506h12m54.662203098s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791275218 (506h12m54.662358253s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791443709 (506h12m54.662526744s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:56.112313145 (506h12m55.98339619s) [upload=444191]: Next returned nil segment Dec 26 17:41:56.112330638 (506h12m55.983413683s) [upload=444191]: Waiting for error group managing segments... Dec 26 17:41:56.485322720 (506h12m56.356405765s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.485637079 (506h12m56.356720124s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.666807539 (506h12m56.537890584s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.678753987 (506h12m56.549837032s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.687788039 (506h12m56.558871074s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.694318023 (506h12m56.565401058s) [noise=true , address=145.79.227.7:4326 , piece_id=DMQ5RPI46X2U7TJTS42ZHQEF32BDJRKJJO5HWOYAPNS6VWVG5B5Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.716581187 (506h12m56.587664232s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.972183007 (506h12m56.843266042s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.011917999 (506h12m56.883001034s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.054168622 (506h12m56.925251667s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.114312777 (506h12m56.985395812s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.499846250 (506h12m57.370929285s) [noise=true , address=145.79.227.34:4919 , piece_id=YRWHW3BJVEGVJRSA6RJB3T7VJQSDTGKEZQKRICFIV2G3T7AN4MIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.612649066 (506h12m57.483732101s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649413733 (506h12m57.520496768s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649440282 (506h12m57.520523328s) [seg_pos={0 0}, upload=444191]: Segment reached optimal threshold of 14 pieces. Dec 26 17:41:57.649674729 (506h12m57.520757774s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G7ty71tPwe7cZSBNK9u74DFUCGomnxDsNvAif4C3SbP7HWwfo): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649684087 (506h12m57.520767142s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649747708 (506h12m57.520830763s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v6zXZuMmyL2MU8qb3q6gg1iwTYvndjzMjVpv8btEpeX9Lnqmu): protocol: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection Dec 26 17:41:57.649785219 (506h12m57.520868274s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649899828 (506h12m57.520982903s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649908775 (506h12m57.52099182s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650074641 (506h12m57.521157696s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.650083287 (506h12m57.521166342s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650132821 (506h12m57.521215866s) [seg_pos={0 0}, upload=444191]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 17:41:57.650220138 (506h12m57.521303183s) [upload=444191]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:41:57.650227171 (506h12m57.521310206s) [upload=444191]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:41:57.958890870 (506h12m57.829973915s) [upload=444191]: Done uploading segments... Dec 26 17:41:57.958911620 (506h12m57.829994665s) [upload=444191]: Upload finished. err: <nil> Duration: 46.809559653s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [.XX....XX..XX......X.XXX.....XX.X.XXX.........X..X.....X...X] duration: 30.004015215s trace: https://jaeger.dev.storj.io/trace/3da2ec831e5b4935 FAIL (2025-12-26 18:10:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 18:10:13.585657599 (506h41m13.456740644s) [upload=444609]: Starting upload Dec 26 18:10:13.585760735 (506h41m13.456843791s) [upload=444609]: Done starting upload Dec 26 18:10:13.585987768 (506h41m13.457070813s) [upload=444609]: Uploading segments... Dec 26 18:10:13.586115180 (506h41m13.457198225s) [upload=444609]: Got next segment. Inline: false Dec 26 18:10:13.586193008 (506h41m13.457276063s) [upload=444609]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 18:10:13.586199481 (506h41m13.457282516s) [upload=444609]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 18:10:13.836577946 (506h41m13.707660991s) [seg_pos={0 0}, upload=444609]: Begin upload segment... Dec 26 18:10:13.836643210 (506h41m13.707726245s) [seg_pos={0 0}, upload=444609]: Done begin upload segment. Dec 26 18:10:13.836705679 (506h41m13.707788724s) [noise=true , address=77.90.31.21:4880 , piece_id=G3WGLPD6F5BBNM2WLN5ETTKWNSJEVRYCKL47GLAO6X2ZPGF4BASA, seg_pos={0 0}, upload=444609]: Uploading piece... Dec 26 18:10:13.836727370 (506h41m13.707810405s) [noise=true , address=145.79.227.46:4601 , piece_id=QI247KMII6DEHXV3LIJAYBLRVPCWG6HUKNMJ2ILLCUK2E7EI323Q, seg_pos={0 0}, upload=444609]: Uploading piece... Dec 26 18:10:13.837580174 (506h41m13.708663219s) [noise=true , address=45.91.116.11:4118 , piece_id=QVKNNE5WJZQOG4WUDGW7O5BDNIH3KFCOY2Q2TCVLU3MORGTC25IA, seg_pos={0 0}, upload=444609]: Uploading piece... Dec 26 18:10:13.838319672 (506h41m13.709402727s) [noise=true , address=145.79.227.19:4077 , piece_id=3Q6THBEJ27Y5JTUSXB3RI73DYBVN6F7MVDKXEUBHKJAYXSW2KUYQ, seg_pos={0 0}, upload=444609]: Uploading piece... Dec 26 18:10:13.839201020 (506h41m13.710284065s) [noise=true , address=45.91.116.10:4081 , piece_id=R7LJY5NXJWVTDSAQURURLWN7XMBQHG2IRCN4IV643WQSFQFKMVWA, seg_pos={0 0}, upload=444609]: Uploading piece... Dec 26 18:10:13.839482266 (506h41m13.710565302s) [noise=true , address=45.91.116.9:4263 , piece_id=WZS7UQHATABAB26PHMWLPHFXJ3IFK7KHPAGQYAWRQML2RXUCNCYQ, seg_pos={0 0}, upload=444609]: Uploading piece... Dec 26 18:10:13.840563105 (506h41m13.71164614s) [noise=true , address=145.79.227.48:4301 , piece_id=PQ4Q3QTYF35BXEDMIH52SQ7RKZYZLJWGVZH5DOOXEIAYYNMU73YA, seg_pos={0 0}, upload=444609]: Uploading piece... Dec 26 18:10:13.840905045 (506h41m13.71198808s) [noise=true , address=145.79.227.45:4231 , piece_id=COLLZGJO73MPRKGHRRO7727WV7N6FGWXCBOSNUETCTZLTLQIVUHQ, seg_pos={0 0}, upload=444609]: Uploading piece... Dec 26 18:10:13.842132924 (506h41m13.713215959s) [noise=true , address=145.79.227.36:4517 , piece_id=BV3JEGA47KFWHSH5IE2KBT2GKDBHJVMIN4PEKFLHE5TOAGMIJ7DQ, seg_pos={0 0}, upload=444609]: Uploading piece... Dec 26 18:10:13.842571168 (506h41m13.713654213s) [noise=true , address=145.79.227.16:4911 , piece_id=V3PPI4KSH64BHHHNJ7VTOH7DGP2JSKG3DRBOLLN76VXI2UHXL74Q, seg_pos={0 0}, upload=444609]: Uploading piece... Dec 26 18:10:13.843649090 (506h41m13.714732125s) [noise=true , address=77.90.31.16:4735 , piece_id=RHFGI636MXS3DXOYHEDH2NOLIPI7TMXCNCIUQBYAACSJM6S4MLDA, seg_pos={0 0}, upload=444609]: Uploading piece... Dec 26 18:10:13.844279140 (506h41m13.715362185s) [noise=true , address=45.91.116.13:4155 , piece_id=Z4C2Y6GKT65T3ZESRTRVO3CNIFHXEDDPVAF3BXYJ656DFGSTISDQ, seg_pos={0 0}, upload=444609]: Uploading piece... Dec 26 18:10:13.844979875 (506h41m13.71606292s) [noise=true , address=45.91.116.5:4300 , piece_id=FJSJ6SXUZU63CQW5HL4JL47SJ2KLWZW2VNSW3XOZNKNRWGXHIRBA, seg_pos={0 0}, upload=444609]: Uploading piece... Dec 26 18:10:13.845387370 (506h41m13.716470415s) [noise=true , address=145.79.227.11:4732 , piece_id=ATCGVHCPHSACQTWQHOMJDD6YRMORAQMVPZBZ37MF5L5ZPUJPDE4A, seg_pos={0 0}, upload=444609]: Uploading piece... Dec 26 18:10:13.846145564 (506h41m13.717228599s) [noise=true , address=145.79.227.49:4337 , piece_id=KZKFVF3ARUHMHRCCN6CRJ5TL4NF4CLTMMXRBIVKWUFQYI74HTSEQ, seg_pos={0 0}, upload=444609]: Uploading piece... Dec 26 18:10:13.850627729 (506h41m13.721710764s) [noise=true , address=45.91.116.8:4048 , piece_id=A5BS5HHPRXXPAUSATNJ3JVXYX5PN2IZ44LEL55VWN7HWLOJY3PDA, seg_pos={0 0}, upload=444609]: Uploading piece... Dec 26 18:10:15.566005228 (506h41m15.437088273s) [upload=444609]: Next returned nil segment Dec 26 18:10:15.566022080 (506h41m15.437105126s) [upload=444609]: Waiting for error group managing segments... Dec 26 18:10:15.842366599 (506h41m15.713449644s) [noise=true , address=145.79.227.36:4517 , piece_id=BV3JEGA47KFWHSH5IE2KBT2GKDBHJVMIN4PEKFLHE5TOAGMIJ7DQ, seg_pos={0 0}, upload=444609]: Done uploading piece. err: <nil> Dec 26 18:10:15.851278488 (506h41m15.722361533s) [noise=true , address=145.79.227.46:4601 , piece_id=QI247KMII6DEHXV3LIJAYBLRVPCWG6HUKNMJ2ILLCUK2E7EI323Q, seg_pos={0 0}, upload=444609]: Done uploading piece. err: <nil> Dec 26 18:10:15.856874984 (506h41m15.727958019s) [noise=true , address=145.79.227.45:4231 , piece_id=COLLZGJO73MPRKGHRRO7727WV7N6FGWXCBOSNUETCTZLTLQIVUHQ, seg_pos={0 0}, upload=444609]: Done uploading piece. err: <nil> Dec 26 18:10:15.865356685 (506h41m15.73643973s) [noise=true , address=145.79.227.16:4911 , piece_id=V3PPI4KSH64BHHHNJ7VTOH7DGP2JSKG3DRBOLLN76VXI2UHXL74Q, seg_pos={0 0}, upload=444609]: Done uploading piece. err: <nil> Dec 26 18:10:15.872461835 (506h41m15.74354491s) [noise=true , address=145.79.227.49:4337 , piece_id=KZKFVF3ARUHMHRCCN6CRJ5TL4NF4CLTMMXRBIVKWUFQYI74HTSEQ, seg_pos={0 0}, upload=444609]: Done uploading piece. err: <nil> Dec 26 18:10:15.873485264 (506h41m15.744568319s) [noise=true , address=145.79.227.11:4732 , piece_id=ATCGVHCPHSACQTWQHOMJDD6YRMORAQMVPZBZ37MF5L5ZPUJPDE4A, seg_pos={0 0}, upload=444609]: Done uploading piece. err: <nil> Dec 26 18:10:15.881168734 (506h41m15.752251779s) [noise=true , address=145.79.227.48:4301 , piece_id=PQ4Q3QTYF35BXEDMIH52SQ7RKZYZLJWGVZH5DOOXEIAYYNMU73YA, seg_pos={0 0}, upload=444609]: Done uploading piece. err: <nil> Dec 26 18:10:16.114505528 (506h41m15.985588564s) [noise=true , address=45.91.116.9:4263 , piece_id=WZS7UQHATABAB26PHMWLPHFXJ3IFK7KHPAGQYAWRQML2RXUCNCYQ, seg_pos={0 0}, upload=444609]: Done uploading piece. err: <nil> Dec 26 18:10:16.120315021 (506h41m15.991398066s) [noise=true , address=77.90.31.21:4880 , piece_id=G3WGLPD6F5BBNM2WLN5ETTKWNSJEVRYCKL47GLAO6X2ZPGF4BASA, seg_pos={0 0}, upload=444609]: Done uploading piece. err: <nil> Dec 26 18:10:16.160815611 (506h41m16.031898656s) [noise=true , address=77.90.31.16:4735 , piece_id=RHFGI636MXS3DXOYHEDH2NOLIPI7TMXCNCIUQBYAACSJM6S4MLDA, seg_pos={0 0}, upload=444609]: Done uploading piece. err: <nil> Dec 26 18:10:16.290468102 (506h41m16.161551157s) [noise=true , address=45.91.116.10:4081 , piece_id=R7LJY5NXJWVTDSAQURURLWN7XMBQHG2IRCN4IV643WQSFQFKMVWA, seg_pos={0 0}, upload=444609]: Done uploading piece. err: <nil> Dec 26 18:10:16.296874140 (506h41m16.167957175s) [noise=true , address=45.91.116.11:4118 , piece_id=QVKNNE5WJZQOG4WUDGW7O5BDNIH3KFCOY2Q2TCVLU3MORGTC25IA, seg_pos={0 0}, upload=444609]: Done uploading piece. err: <nil> Dec 26 18:10:17.169652402 (506h41m17.040735437s) [noise=true , address=45.91.116.5:4300 , piece_id=FJSJ6SXUZU63CQW5HL4JL47SJ2KLWZW2VNSW3XOZNKNRWGXHIRBA, seg_pos={0 0}, upload=444609]: Done uploading piece. err: <nil> Dec 26 18:10:17.169673924 (506h41m17.040756959s) [seg_pos={0 0}, upload=444609]: Segment reached optimal threshold of 13 pieces. Dec 26 18:10:17.169897629 (506h41m17.040980664s) [noise=true , address=145.79.227.19:4077 , piece_id=3Q6THBEJ27Y5JTUSXB3RI73DYBVN6F7MVDKXEUBHKJAYXSW2KUYQ, seg_pos={0 0}, upload=444609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk): protocol: write tcp 188.166.225.170:52636->145.79.227.19:4077: use of closed network connection; write tcp 188.166.225.170:52636->145.79.227.19:4077: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52636->145.79.227.19:4077: use of closed network connection Dec 26 18:10:17.169906055 (506h41m17.0409891s) [noise=true , address=145.79.227.19:4077 , piece_id=3Q6THBEJ27Y5JTUSXB3RI73DYBVN6F7MVDKXEUBHKJAYXSW2KUYQ, seg_pos={0 0}, upload=444609]: Long tail cancellation due to optimal threshold Dec 26 18:10:17.170164428 (506h41m17.041247463s) [noise=true , address=45.91.116.13:4155 , piece_id=Z4C2Y6GKT65T3ZESRTRVO3CNIFHXEDDPVAF3BXYJ656DFGSTISDQ, seg_pos={0 0}, upload=444609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 26 18:10:17.170170529 (506h41m17.041253584s) [noise=true , address=45.91.116.13:4155 , piece_id=Z4C2Y6GKT65T3ZESRTRVO3CNIFHXEDDPVAF3BXYJ656DFGSTISDQ, seg_pos={0 0}, upload=444609]: Long tail cancellation due to optimal threshold Dec 26 18:10:17.170358507 (506h41m17.041441542s) [noise=true , address=45.91.116.8:4048 , piece_id=A5BS5HHPRXXPAUSATNJ3JVXYX5PN2IZ44LEL55VWN7HWLOJY3PDA, seg_pos={0 0}, upload=444609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 26 18:10:17.170365290 (506h41m17.041448335s) [noise=true , address=45.91.116.8:4048 , piece_id=A5BS5HHPRXXPAUSATNJ3JVXYX5PN2IZ44LEL55VWN7HWLOJY3PDA, seg_pos={0 0}, upload=444609]: Long tail cancellation due to optimal threshold Dec 26 18:10:17.170383545 (506h41m17.04146659s) [seg_pos={0 0}, upload=444609]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 18:10:17.170486531 (506h41m17.041569576s) [upload=444609]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 18:10:17.170493874 (506h41m17.04157692s) [upload=444609]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 18:10:17.467239551 (506h41m17.338322596s) [upload=444609]: Done uploading segments... Dec 26 18:10:17.467263777 (506h41m17.338346822s) [upload=444609]: Upload finished. err: <nil> Duration: 30.004041285s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 51.746293448s trace: https://jaeger.dev.storj.io/trace/665ea65f6e9ccc33 FAIL (2025-12-26 18:09:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== e=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789316979 (506h12m54.660400024s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789326507 (506h12m54.660409552s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789611339 (506h12m54.660694384s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789619275 (506h12m54.66070231s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789914006 (506h12m54.660997041s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789949443 (506h12m54.661032478s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790181566 (506h12m54.661264601s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790295632 (506h12m54.661378668s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790510872 (506h12m54.661593917s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790692980 (506h12m54.661776015s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790787379 (506h12m54.661870414s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790992970 (506h12m54.662076005s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791120053 (506h12m54.662203098s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791275218 (506h12m54.662358253s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791443709 (506h12m54.662526744s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:56.112313145 (506h12m55.98339619s) [upload=444191]: Next returned nil segment Dec 26 17:41:56.112330638 (506h12m55.983413683s) [upload=444191]: Waiting for error group managing segments... Dec 26 17:41:56.485322720 (506h12m56.356405765s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.485637079 (506h12m56.356720124s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.666807539 (506h12m56.537890584s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.678753987 (506h12m56.549837032s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.687788039 (506h12m56.558871074s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.694318023 (506h12m56.565401058s) [noise=true , address=145.79.227.7:4326 , piece_id=DMQ5RPI46X2U7TJTS42ZHQEF32BDJRKJJO5HWOYAPNS6VWVG5B5Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.716581187 (506h12m56.587664232s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.972183007 (506h12m56.843266042s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.011917999 (506h12m56.883001034s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.054168622 (506h12m56.925251667s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.114312777 (506h12m56.985395812s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.499846250 (506h12m57.370929285s) [noise=true , address=145.79.227.34:4919 , piece_id=YRWHW3BJVEGVJRSA6RJB3T7VJQSDTGKEZQKRICFIV2G3T7AN4MIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.612649066 (506h12m57.483732101s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649413733 (506h12m57.520496768s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649440282 (506h12m57.520523328s) [seg_pos={0 0}, upload=444191]: Segment reached optimal threshold of 14 pieces. Dec 26 17:41:57.649674729 (506h12m57.520757774s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G7ty71tPwe7cZSBNK9u74DFUCGomnxDsNvAif4C3SbP7HWwfo): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649684087 (506h12m57.520767142s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649747708 (506h12m57.520830763s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v6zXZuMmyL2MU8qb3q6gg1iwTYvndjzMjVpv8btEpeX9Lnqmu): protocol: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection Dec 26 17:41:57.649785219 (506h12m57.520868274s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649899828 (506h12m57.520982903s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649908775 (506h12m57.52099182s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650074641 (506h12m57.521157696s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.650083287 (506h12m57.521166342s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650132821 (506h12m57.521215866s) [seg_pos={0 0}, upload=444191]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 17:41:57.650220138 (506h12m57.521303183s) [upload=444191]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:41:57.650227171 (506h12m57.521310206s) [upload=444191]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:41:57.958890870 (506h12m57.829973915s) [upload=444191]: Done uploading segments... Dec 26 17:41:57.958911620 (506h12m57.829994665s) [upload=444191]: Upload finished. err: <nil> Duration: 51.746326121s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [..XX....XX..XX......X.XXX.....XX.X.XXX.........X..X.....X...] duration: 5.937663621s trace: https://jaeger.dev.storj.io/trace/6f5ab3f88a41764a FAIL (2025-12-26 18:09:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m, Piece ID: TCHSBMRXCDNQYYLQFP3OIVLSCZS5GCWOHCUK5K46EU3GJ6GFAUUQ) context deadline exceeded; piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: ZSI7J4C5BSGB7ROCCKOU6KWZT3PRYCCVITRWEQ3C5AKALPWZVHMQ) context deadline exceeded ======LOGS====== 399 (506h38m5.518466444s) [upload=444561]: Uploading segments... Dec 26 18:07:05.647552551 (506h38m5.518635596s) [upload=444561]: Got next segment. Inline: false Dec 26 18:07:05.647613107 (506h38m5.518696142s) [upload=444561]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 18:07:05.647618066 (506h38m5.518701101s) [upload=444561]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 18:07:05.893317271 (506h38m5.764400316s) [seg_pos={0 0}, upload=444561]: Begin upload segment... Dec 26 18:07:05.893413573 (506h38m5.764496618s) [seg_pos={0 0}, upload=444561]: Done begin upload segment. Dec 26 18:07:05.893454061 (506h38m5.764537106s) [noise=true , address=145.79.227.49:4337 , piece_id=P77OCEPHI3OAC464EWXBAK3LWROAGULPQDSDC5CNFFANHHMO33MQ, seg_pos={0 0}, upload=444561]: Uploading piece... Dec 26 18:07:05.893541878 (506h38m5.764624913s) [noise=true , address=77.90.31.21:4878 , piece_id=OIM2BL3NK47PE6RGZGTBEIJQOSV5QJTPV6EWZASDAI6ICAOFP35A, seg_pos={0 0}, upload=444561]: Uploading piece... Dec 26 18:07:05.894186135 (506h38m5.76526918s) [noise=true , address=145.79.227.46:4603 , piece_id=2Z6UZAVQPKXBOI3XC3R6L4Z6QQE2Q2FBD7UGMZVKKRPTOL6OSQKQ, seg_pos={0 0}, upload=444561]: Uploading piece... Dec 26 18:07:05.894703070 (506h38m5.765786115s) [noise=true , address=145.79.227.21:4474 , piece_id=OQIDGFYSDEEEOOOZBVLZJD6KSJ2XZP6DWYHTFJ3KTTE67VPRPSJQ, seg_pos={0 0}, upload=444561]: Uploading piece... Dec 26 18:07:05.895022979 (506h38m5.766106014s) [noise=true , address=45.91.116.9:4264 , piece_id=TCHSBMRXCDNQYYLQFP3OIVLSCZS5GCWOHCUK5K46EU3GJ6GFAUUQ, seg_pos={0 0}, upload=444561]: Uploading piece... Dec 26 18:07:05.895604606 (506h38m5.766687651s) [noise=true , address=45.91.116.13:4155 , piece_id=GQSX7CVSG3ZHEPYGCKFMRN43BFDZ344DEG2NO2LXLRYGTRFUOJ5Q, seg_pos={0 0}, upload=444561]: Uploading piece... Dec 26 18:07:05.895877936 (506h38m5.766960971s) [noise=true , address=145.79.227.39:4014 , piece_id=ZS7QXCMU46IXAW6433NWCTVJYM3OMGBTJIQ2KSPJWKFCVNUW7IMQ, seg_pos={0 0}, upload=444561]: Uploading piece... Dec 26 18:07:05.896574142 (506h38m5.767657177s) [noise=true , address=77.90.31.18:4808 , piece_id=ZSI7J4C5BSGB7ROCCKOU6KWZT3PRYCCVITRWEQ3C5AKALPWZVHMQ, seg_pos={0 0}, upload=444561]: Uploading piece... Dec 26 18:07:05.896865577 (506h38m5.767948623s) [noise=true , address=145.79.227.41:4085 , piece_id=SZQWIUPASZ2CRYQVPSQXW2XE7EUV6R3VQPQRIALB3KVHWJWLAWAQ, seg_pos={0 0}, upload=444561]: Uploading piece... Dec 26 18:07:05.897453567 (506h38m5.768536602s) [noise=true , address=145.79.227.8:4330 , piece_id=4MMSYKVJMW23XYJONUFULK7GICSZGQAC6E7W65AJ4MGASDSFP3WA, seg_pos={0 0}, upload=444561]: Uploading piece... Dec 26 18:07:05.897679557 (506h38m5.768762592s) [noise=true , address=145.79.227.22:4511 , piece_id=2PEP6MXPHOL3DMM2LJUXYNMPFZB2MW4Q664SU5P5LL6ECNO34C2Q, seg_pos={0 0}, upload=444561]: Uploading piece... Dec 26 18:07:05.898400360 (506h38m5.769483395s) [noise=true , address=77.90.31.17:4773 , piece_id=MGL2FRFSSUTDQLTMI3ICWMNLRD4ZOHHVDI6RVKCXRQDZANCT4WXQ, seg_pos={0 0}, upload=444561]: Uploading piece... Dec 26 18:07:05.898672889 (506h38m5.769755934s) [noise=true , address=145.79.227.48:4303 , piece_id=LYRAHY4DMH3AJPQCBIN6URWP35W7P6CMF2W2DEUCFYD7V7F6IAWA, seg_pos={0 0}, upload=444561]: Uploading piece... Dec 26 18:07:05.899152481 (506h38m5.770235516s) [noise=true , address=145.79.227.30:4761 , piece_id=MZWR4L4PPROWB4IUBXIY4BZ56RXF22FL2YHHACOQE2DFSLEPFKUQ, seg_pos={0 0}, upload=444561]: Uploading piece... Dec 26 18:07:05.899395544 (506h38m5.770478579s) [noise=true , address=145.79.227.47:4266 , piece_id=CGYGCEZ2334TCEOVXF5YHBPF4VUXHC4FJTF3A5ZZQM6Q5JYDLUBA, seg_pos={0 0}, upload=444561]: Uploading piece... Dec 26 18:07:05.904889376 (506h38m5.775972421s) [noise=true , address=145.79.227.37:4567 , piece_id=2OIDEOOUHBE7BZCWGBC7HDAP2KCGYW644MYEPPSE6RAZWWFNQUAA, seg_pos={0 0}, upload=444561]: Uploading piece... Dec 26 18:07:07.582540777 (506h38m7.453623822s) [upload=444561]: Next returned nil segment Dec 26 18:07:07.582553913 (506h38m7.453636958s) [upload=444561]: Waiting for error group managing segments... Dec 26 18:07:07.903698448 (506h38m7.774781493s) [noise=true , address=145.79.227.47:4266 , piece_id=CGYGCEZ2334TCEOVXF5YHBPF4VUXHC4FJTF3A5ZZQM6Q5JYDLUBA, seg_pos={0 0}, upload=444561]: Done uploading piece. err: <nil> Dec 26 18:07:07.904133025 (506h38m7.77521606s) [noise=true , address=145.79.227.48:4303 , piece_id=LYRAHY4DMH3AJPQCBIN6URWP35W7P6CMF2W2DEUCFYD7V7F6IAWA, seg_pos={0 0}, upload=444561]: Done uploading piece. err: <nil> Dec 26 18:07:07.904510023 (506h38m7.775593058s) [noise=true , address=145.79.227.49:4337 , piece_id=P77OCEPHI3OAC464EWXBAK3LWROAGULPQDSDC5CNFFANHHMO33MQ, seg_pos={0 0}, upload=444561]: Done uploading piece. err: <nil> Dec 26 18:07:07.904611216 (506h38m7.775694251s) [noise=true , address=145.79.227.37:4567 , piece_id=2OIDEOOUHBE7BZCWGBC7HDAP2KCGYW644MYEPPSE6RAZWWFNQUAA, seg_pos={0 0}, upload=444561]: Done uploading piece. err: <nil> Dec 26 18:07:07.905953782 (506h38m7.777036817s) [noise=true , address=145.79.227.41:4085 , piece_id=SZQWIUPASZ2CRYQVPSQXW2XE7EUV6R3VQPQRIALB3KVHWJWLAWAQ, seg_pos={0 0}, upload=444561]: Done uploading piece. err: <nil> Dec 26 18:07:07.906362129 (506h38m7.777445164s) [noise=true , address=145.79.227.46:4603 , piece_id=2Z6UZAVQPKXBOI3XC3R6L4Z6QQE2Q2FBD7UGMZVKKRPTOL6OSQKQ, seg_pos={0 0}, upload=444561]: Done uploading piece. err: <nil> Dec 26 18:07:07.912393825 (506h38m7.78347687s) [noise=true , address=145.79.227.21:4474 , piece_id=OQIDGFYSDEEEOOOZBVLZJD6KSJ2XZP6DWYHTFJ3KTTE67VPRPSJQ, seg_pos={0 0}, upload=444561]: Done uploading piece. err: <nil> Dec 26 18:07:07.926216957 (506h38m7.797300002s) [noise=true , address=145.79.227.8:4330 , piece_id=4MMSYKVJMW23XYJONUFULK7GICSZGQAC6E7W65AJ4MGASDSFP3WA, seg_pos={0 0}, upload=444561]: Done uploading piece. err: <nil> Dec 26 18:07:07.931470309 (506h38m7.802553345s) [noise=true , address=145.79.227.39:4014 , piece_id=ZS7QXCMU46IXAW6433NWCTVJYM3OMGBTJIQ2KSPJWKFCVNUW7IMQ, seg_pos={0 0}, upload=444561]: Done uploading piece. err: <nil> Dec 26 18:07:07.931844102 (506h38m7.802927147s) [noise=true , address=145.79.227.22:4511 , piece_id=2PEP6MXPHOL3DMM2LJUXYNMPFZB2MW4Q664SU5P5LL6ECNO34C2Q, seg_pos={0 0}, upload=444561]: Done uploading piece. err: <nil> Dec 26 18:07:08.157225668 (506h38m8.028308714s) [noise=true , address=45.91.116.13:4155 , piece_id=GQSX7CVSG3ZHEPYGCKFMRN43BFDZ344DEG2NO2LXLRYGTRFUOJ5Q, seg_pos={0 0}, upload=444561]: Done uploading piece. err: <nil> Dec 26 18:07:08.162911395 (506h38m8.03399444s) [noise=true , address=45.91.116.9:4264 , piece_id=TCHSBMRXCDNQYYLQFP3OIVLSCZS5GCWOHCUK5K46EU3GJ6GFAUUQ, seg_pos={0 0}, upload=444561]: Done uploading piece. err: <nil> Dec 26 18:07:08.508925465 (506h38m8.3800085s) [noise=true , address=77.90.31.18:4808 , piece_id=ZSI7J4C5BSGB7ROCCKOU6KWZT3PRYCCVITRWEQ3C5AKALPWZVHMQ, seg_pos={0 0}, upload=444561]: Done uploading piece. err: <nil> Dec 26 18:07:08.508951195 (506h38m8.38003423s) [seg_pos={0 0}, upload=444561]: Segment reached optimal threshold of 13 pieces. Dec 26 18:07:08.509141928 (506h38m8.380224963s) [noise=true , address=77.90.31.21:4878 , piece_id=OIM2BL3NK47PE6RGZGTBEIJQOSV5QJTPV6EWZASDAI6ICAOFP35A, seg_pos={0 0}, upload=444561]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: write tcp 188.166.225.170:33740->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:33740->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33740->77.90.31.21:4878: use of closed network connection Dec 26 18:07:08.509148721 (506h38m8.380231756s) [noise=true , address=77.90.31.21:4878 , piece_id=OIM2BL3NK47PE6RGZGTBEIJQOSV5QJTPV6EWZASDAI6ICAOFP35A, seg_pos={0 0}, upload=444561]: Long tail cancellation due to optimal threshold Dec 26 18:07:08.509293136 (506h38m8.380376171s) [noise=true , address=145.79.227.30:4761 , piece_id=MZWR4L4PPROWB4IUBXIY4BZ56RXF22FL2YHHACOQE2DFSLEPFKUQ, seg_pos={0 0}, upload=444561]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 26 18:07:08.509323624 (506h38m8.380406669s) [noise=true , address=145.79.227.30:4761 , piece_id=MZWR4L4PPROWB4IUBXIY4BZ56RXF22FL2YHHACOQE2DFSLEPFKUQ, seg_pos={0 0}, upload=444561]: Long tail cancellation due to optimal threshold Dec 26 18:07:08.509430027 (506h38m8.380513062s) [noise=true , address=77.90.31.17:4773 , piece_id=MGL2FRFSSUTDQLTMI3ICWMNLRD4ZOHHVDI6RVKCXRQDZANCT4WXQ, seg_pos={0 0}, upload=444561]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: write tcp 188.166.225.170:43318->77.90.31.17:4773: use of closed network connection; write tcp 188.166.225.170:43318->77.90.31.17:4773: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43318->77.90.31.17:4773: use of closed network connection Dec 26 18:07:08.509438292 (506h38m8.380521327s) [noise=true , address=77.90.31.17:4773 , piece_id=MGL2FRFSSUTDQLTMI3ICWMNLRD4ZOHHVDI6RVKCXRQDZANCT4WXQ, seg_pos={0 0}, upload=444561]: Long tail cancellation due to optimal threshold Dec 26 18:07:08.509456648 (506h38m8.380539693s) [seg_pos={0 0}, upload=444561]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 18:07:08.509544855 (506h38m8.3806279s) [upload=444561]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 18:07:08.509553511 (506h38m8.380636556s) [upload=444561]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 18:07:08.802296606 (506h38m8.673379641s) [upload=444561]: Done uploading segments... Dec 26 18:07:08.802315842 (506h38m8.673398887s) [upload=444561]: Upload finished. err: <nil> Duration: 5.937686654s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [...XX....XX..XX......X.XXX.....XX.X.XXX.........X..X.....X..] duration: 6.18160034s trace: https://jaeger.dev.storj.io/trace/6f5ab3f88a41764a FAIL (2025-12-26 18:08:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m, Piece ID: TCHSBMRXCDNQYYLQFP3OIVLSCZS5GCWOHCUK5K46EU3GJ6GFAUUQ) context deadline exceeded; piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: ZSI7J4C5BSGB7ROCCKOU6KWZT3PRYCCVITRWEQ3C5AKALPWZVHMQ) context deadline exceeded ======LOGS====== 399 (506h38m5.518466444s) [upload=444561]: Uploading segments... Dec 26 18:07:05.647552551 (506h38m5.518635596s) [upload=444561]: Got next segment. Inline: false Dec 26 18:07:05.647613107 (506h38m5.518696142s) [upload=444561]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 18:07:05.647618066 (506h38m5.518701101s) [upload=444561]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 18:07:05.893317271 (506h38m5.764400316s) [seg_pos={0 0}, upload=444561]: Begin upload segment... Dec 26 18:07:05.893413573 (506h38m5.764496618s) [seg_pos={0 0}, upload=444561]: Done begin upload segment. Dec 26 18:07:05.893454061 (506h38m5.764537106s) [noise=true , address=145.79.227.49:4337 , piece_id=P77OCEPHI3OAC464EWXBAK3LWROAGULPQDSDC5CNFFANHHMO33MQ, seg_pos={0 0}, upload=444561]: Uploading piece... Dec 26 18:07:05.893541878 (506h38m5.764624913s) [noise=true , address=77.90.31.21:4878 , piece_id=OIM2BL3NK47PE6RGZGTBEIJQOSV5QJTPV6EWZASDAI6ICAOFP35A, seg_pos={0 0}, upload=444561]: Uploading piece... Dec 26 18:07:05.894186135 (506h38m5.76526918s) [noise=true , address=145.79.227.46:4603 , piece_id=2Z6UZAVQPKXBOI3XC3R6L4Z6QQE2Q2FBD7UGMZVKKRPTOL6OSQKQ, seg_pos={0 0}, upload=444561]: Uploading piece... Dec 26 18:07:05.894703070 (506h38m5.765786115s) [noise=true , address=145.79.227.21:4474 , piece_id=OQIDGFYSDEEEOOOZBVLZJD6KSJ2XZP6DWYHTFJ3KTTE67VPRPSJQ, seg_pos={0 0}, upload=444561]: Uploading piece... Dec 26 18:07:05.895022979 (506h38m5.766106014s) [noise=true , address=45.91.116.9:4264 , piece_id=TCHSBMRXCDNQYYLQFP3OIVLSCZS5GCWOHCUK5K46EU3GJ6GFAUUQ, seg_pos={0 0}, upload=444561]: Uploading piece... Dec 26 18:07:05.895604606 (506h38m5.766687651s) [noise=true , address=45.91.116.13:4155 , piece_id=GQSX7CVSG3ZHEPYGCKFMRN43BFDZ344DEG2NO2LXLRYGTRFUOJ5Q, seg_pos={0 0}, upload=444561]: Uploading piece... Dec 26 18:07:05.895877936 (506h38m5.766960971s) [noise=true , address=145.79.227.39:4014 , piece_id=ZS7QXCMU46IXAW6433NWCTVJYM3OMGBTJIQ2KSPJWKFCVNUW7IMQ, seg_pos={0 0}, upload=444561]: Uploading piece... Dec 26 18:07:05.896574142 (506h38m5.767657177s) [noise=true , address=77.90.31.18:4808 , piece_id=ZSI7J4C5BSGB7ROCCKOU6KWZT3PRYCCVITRWEQ3C5AKALPWZVHMQ, seg_pos={0 0}, upload=444561]: Uploading piece... Dec 26 18:07:05.896865577 (506h38m5.767948623s) [noise=true , address=145.79.227.41:4085 , piece_id=SZQWIUPASZ2CRYQVPSQXW2XE7EUV6R3VQPQRIALB3KVHWJWLAWAQ, seg_pos={0 0}, upload=444561]: Uploading piece... Dec 26 18:07:05.897453567 (506h38m5.768536602s) [noise=true , address=145.79.227.8:4330 , piece_id=4MMSYKVJMW23XYJONUFULK7GICSZGQAC6E7W65AJ4MGASDSFP3WA, seg_pos={0 0}, upload=444561]: Uploading piece... Dec 26 18:07:05.897679557 (506h38m5.768762592s) [noise=true , address=145.79.227.22:4511 , piece_id=2PEP6MXPHOL3DMM2LJUXYNMPFZB2MW4Q664SU5P5LL6ECNO34C2Q, seg_pos={0 0}, upload=444561]: Uploading piece... Dec 26 18:07:05.898400360 (506h38m5.769483395s) [noise=true , address=77.90.31.17:4773 , piece_id=MGL2FRFSSUTDQLTMI3ICWMNLRD4ZOHHVDI6RVKCXRQDZANCT4WXQ, seg_pos={0 0}, upload=444561]: Uploading piece... Dec 26 18:07:05.898672889 (506h38m5.769755934s) [noise=true , address=145.79.227.48:4303 , piece_id=LYRAHY4DMH3AJPQCBIN6URWP35W7P6CMF2W2DEUCFYD7V7F6IAWA, seg_pos={0 0}, upload=444561]: Uploading piece... Dec 26 18:07:05.899152481 (506h38m5.770235516s) [noise=true , address=145.79.227.30:4761 , piece_id=MZWR4L4PPROWB4IUBXIY4BZ56RXF22FL2YHHACOQE2DFSLEPFKUQ, seg_pos={0 0}, upload=444561]: Uploading piece... Dec 26 18:07:05.899395544 (506h38m5.770478579s) [noise=true , address=145.79.227.47:4266 , piece_id=CGYGCEZ2334TCEOVXF5YHBPF4VUXHC4FJTF3A5ZZQM6Q5JYDLUBA, seg_pos={0 0}, upload=444561]: Uploading piece... Dec 26 18:07:05.904889376 (506h38m5.775972421s) [noise=true , address=145.79.227.37:4567 , piece_id=2OIDEOOUHBE7BZCWGBC7HDAP2KCGYW644MYEPPSE6RAZWWFNQUAA, seg_pos={0 0}, upload=444561]: Uploading piece... Dec 26 18:07:07.582540777 (506h38m7.453623822s) [upload=444561]: Next returned nil segment Dec 26 18:07:07.582553913 (506h38m7.453636958s) [upload=444561]: Waiting for error group managing segments... Dec 26 18:07:07.903698448 (506h38m7.774781493s) [noise=true , address=145.79.227.47:4266 , piece_id=CGYGCEZ2334TCEOVXF5YHBPF4VUXHC4FJTF3A5ZZQM6Q5JYDLUBA, seg_pos={0 0}, upload=444561]: Done uploading piece. err: <nil> Dec 26 18:07:07.904133025 (506h38m7.77521606s) [noise=true , address=145.79.227.48:4303 , piece_id=LYRAHY4DMH3AJPQCBIN6URWP35W7P6CMF2W2DEUCFYD7V7F6IAWA, seg_pos={0 0}, upload=444561]: Done uploading piece. err: <nil> Dec 26 18:07:07.904510023 (506h38m7.775593058s) [noise=true , address=145.79.227.49:4337 , piece_id=P77OCEPHI3OAC464EWXBAK3LWROAGULPQDSDC5CNFFANHHMO33MQ, seg_pos={0 0}, upload=444561]: Done uploading piece. err: <nil> Dec 26 18:07:07.904611216 (506h38m7.775694251s) [noise=true , address=145.79.227.37:4567 , piece_id=2OIDEOOUHBE7BZCWGBC7HDAP2KCGYW644MYEPPSE6RAZWWFNQUAA, seg_pos={0 0}, upload=444561]: Done uploading piece. err: <nil> Dec 26 18:07:07.905953782 (506h38m7.777036817s) [noise=true , address=145.79.227.41:4085 , piece_id=SZQWIUPASZ2CRYQVPSQXW2XE7EUV6R3VQPQRIALB3KVHWJWLAWAQ, seg_pos={0 0}, upload=444561]: Done uploading piece. err: <nil> Dec 26 18:07:07.906362129 (506h38m7.777445164s) [noise=true , address=145.79.227.46:4603 , piece_id=2Z6UZAVQPKXBOI3XC3R6L4Z6QQE2Q2FBD7UGMZVKKRPTOL6OSQKQ, seg_pos={0 0}, upload=444561]: Done uploading piece. err: <nil> Dec 26 18:07:07.912393825 (506h38m7.78347687s) [noise=true , address=145.79.227.21:4474 , piece_id=OQIDGFYSDEEEOOOZBVLZJD6KSJ2XZP6DWYHTFJ3KTTE67VPRPSJQ, seg_pos={0 0}, upload=444561]: Done uploading piece. err: <nil> Dec 26 18:07:07.926216957 (506h38m7.797300002s) [noise=true , address=145.79.227.8:4330 , piece_id=4MMSYKVJMW23XYJONUFULK7GICSZGQAC6E7W65AJ4MGASDSFP3WA, seg_pos={0 0}, upload=444561]: Done uploading piece. err: <nil> Dec 26 18:07:07.931470309 (506h38m7.802553345s) [noise=true , address=145.79.227.39:4014 , piece_id=ZS7QXCMU46IXAW6433NWCTVJYM3OMGBTJIQ2KSPJWKFCVNUW7IMQ, seg_pos={0 0}, upload=444561]: Done uploading piece. err: <nil> Dec 26 18:07:07.931844102 (506h38m7.802927147s) [noise=true , address=145.79.227.22:4511 , piece_id=2PEP6MXPHOL3DMM2LJUXYNMPFZB2MW4Q664SU5P5LL6ECNO34C2Q, seg_pos={0 0}, upload=444561]: Done uploading piece. err: <nil> Dec 26 18:07:08.157225668 (506h38m8.028308714s) [noise=true , address=45.91.116.13:4155 , piece_id=GQSX7CVSG3ZHEPYGCKFMRN43BFDZ344DEG2NO2LXLRYGTRFUOJ5Q, seg_pos={0 0}, upload=444561]: Done uploading piece. err: <nil> Dec 26 18:07:08.162911395 (506h38m8.03399444s) [noise=true , address=45.91.116.9:4264 , piece_id=TCHSBMRXCDNQYYLQFP3OIVLSCZS5GCWOHCUK5K46EU3GJ6GFAUUQ, seg_pos={0 0}, upload=444561]: Done uploading piece. err: <nil> Dec 26 18:07:08.508925465 (506h38m8.3800085s) [noise=true , address=77.90.31.18:4808 , piece_id=ZSI7J4C5BSGB7ROCCKOU6KWZT3PRYCCVITRWEQ3C5AKALPWZVHMQ, seg_pos={0 0}, upload=444561]: Done uploading piece. err: <nil> Dec 26 18:07:08.508951195 (506h38m8.38003423s) [seg_pos={0 0}, upload=444561]: Segment reached optimal threshold of 13 pieces. Dec 26 18:07:08.509141928 (506h38m8.380224963s) [noise=true , address=77.90.31.21:4878 , piece_id=OIM2BL3NK47PE6RGZGTBEIJQOSV5QJTPV6EWZASDAI6ICAOFP35A, seg_pos={0 0}, upload=444561]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: write tcp 188.166.225.170:33740->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:33740->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33740->77.90.31.21:4878: use of closed network connection Dec 26 18:07:08.509148721 (506h38m8.380231756s) [noise=true , address=77.90.31.21:4878 , piece_id=OIM2BL3NK47PE6RGZGTBEIJQOSV5QJTPV6EWZASDAI6ICAOFP35A, seg_pos={0 0}, upload=444561]: Long tail cancellation due to optimal threshold Dec 26 18:07:08.509293136 (506h38m8.380376171s) [noise=true , address=145.79.227.30:4761 , piece_id=MZWR4L4PPROWB4IUBXIY4BZ56RXF22FL2YHHACOQE2DFSLEPFKUQ, seg_pos={0 0}, upload=444561]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 26 18:07:08.509323624 (506h38m8.380406669s) [noise=true , address=145.79.227.30:4761 , piece_id=MZWR4L4PPROWB4IUBXIY4BZ56RXF22FL2YHHACOQE2DFSLEPFKUQ, seg_pos={0 0}, upload=444561]: Long tail cancellation due to optimal threshold Dec 26 18:07:08.509430027 (506h38m8.380513062s) [noise=true , address=77.90.31.17:4773 , piece_id=MGL2FRFSSUTDQLTMI3ICWMNLRD4ZOHHVDI6RVKCXRQDZANCT4WXQ, seg_pos={0 0}, upload=444561]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: write tcp 188.166.225.170:43318->77.90.31.17:4773: use of closed network connection; write tcp 188.166.225.170:43318->77.90.31.17:4773: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43318->77.90.31.17:4773: use of closed network connection Dec 26 18:07:08.509438292 (506h38m8.380521327s) [noise=true , address=77.90.31.17:4773 , piece_id=MGL2FRFSSUTDQLTMI3ICWMNLRD4ZOHHVDI6RVKCXRQDZANCT4WXQ, seg_pos={0 0}, upload=444561]: Long tail cancellation due to optimal threshold Dec 26 18:07:08.509456648 (506h38m8.380539693s) [seg_pos={0 0}, upload=444561]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 18:07:08.509544855 (506h38m8.3806279s) [upload=444561]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 18:07:08.509553511 (506h38m8.380636556s) [upload=444561]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 18:07:08.802296606 (506h38m8.673379641s) [upload=444561]: Done uploading segments... Dec 26 18:07:08.802315842 (506h38m8.673398887s) [upload=444561]: Upload finished. err: <nil> Duration: 6.18162712s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 30.573616013s trace: https://jaeger.dev.storj.io/trace/665ea65f6e9ccc33 FAIL (2025-12-26 18:08:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== e=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789316979 (506h12m54.660400024s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789326507 (506h12m54.660409552s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789611339 (506h12m54.660694384s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789619275 (506h12m54.66070231s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789914006 (506h12m54.660997041s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789949443 (506h12m54.661032478s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790181566 (506h12m54.661264601s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790295632 (506h12m54.661378668s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790510872 (506h12m54.661593917s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790692980 (506h12m54.661776015s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790787379 (506h12m54.661870414s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790992970 (506h12m54.662076005s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791120053 (506h12m54.662203098s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791275218 (506h12m54.662358253s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791443709 (506h12m54.662526744s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:56.112313145 (506h12m55.98339619s) [upload=444191]: Next returned nil segment Dec 26 17:41:56.112330638 (506h12m55.983413683s) [upload=444191]: Waiting for error group managing segments... Dec 26 17:41:56.485322720 (506h12m56.356405765s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.485637079 (506h12m56.356720124s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.666807539 (506h12m56.537890584s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.678753987 (506h12m56.549837032s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.687788039 (506h12m56.558871074s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.694318023 (506h12m56.565401058s) [noise=true , address=145.79.227.7:4326 , piece_id=DMQ5RPI46X2U7TJTS42ZHQEF32BDJRKJJO5HWOYAPNS6VWVG5B5Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.716581187 (506h12m56.587664232s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.972183007 (506h12m56.843266042s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.011917999 (506h12m56.883001034s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.054168622 (506h12m56.925251667s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.114312777 (506h12m56.985395812s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.499846250 (506h12m57.370929285s) [noise=true , address=145.79.227.34:4919 , piece_id=YRWHW3BJVEGVJRSA6RJB3T7VJQSDTGKEZQKRICFIV2G3T7AN4MIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.612649066 (506h12m57.483732101s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649413733 (506h12m57.520496768s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649440282 (506h12m57.520523328s) [seg_pos={0 0}, upload=444191]: Segment reached optimal threshold of 14 pieces. Dec 26 17:41:57.649674729 (506h12m57.520757774s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G7ty71tPwe7cZSBNK9u74DFUCGomnxDsNvAif4C3SbP7HWwfo): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649684087 (506h12m57.520767142s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649747708 (506h12m57.520830763s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v6zXZuMmyL2MU8qb3q6gg1iwTYvndjzMjVpv8btEpeX9Lnqmu): protocol: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection Dec 26 17:41:57.649785219 (506h12m57.520868274s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649899828 (506h12m57.520982903s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649908775 (506h12m57.52099182s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650074641 (506h12m57.521157696s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.650083287 (506h12m57.521166342s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650132821 (506h12m57.521215866s) [seg_pos={0 0}, upload=444191]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 17:41:57.650220138 (506h12m57.521303183s) [upload=444191]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:41:57.650227171 (506h12m57.521310206s) [upload=444191]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:41:57.958890870 (506h12m57.829973915s) [upload=444191]: Done uploading segments... Dec 26 17:41:57.958911620 (506h12m57.829994665s) [upload=444191]: Upload finished. err: <nil> Duration: 30.57364559s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [X...XX....XX..XX......X.XXX.....XX.X.XXX.........X..X.....X.] duration: 5.934448959s trace: https://jaeger.dev.storj.io/trace/6f5ab3f88a41764a FAIL (2025-12-26 18:08:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m, Piece ID: TCHSBMRXCDNQYYLQFP3OIVLSCZS5GCWOHCUK5K46EU3GJ6GFAUUQ) context deadline exceeded; piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: ZSI7J4C5BSGB7ROCCKOU6KWZT3PRYCCVITRWEQ3C5AKALPWZVHMQ) context deadline exceeded ======LOGS====== 399 (506h38m5.518466444s) [upload=444561]: Uploading segments... Dec 26 18:07:05.647552551 (506h38m5.518635596s) [upload=444561]: Got next segment. Inline: false Dec 26 18:07:05.647613107 (506h38m5.518696142s) [upload=444561]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 18:07:05.647618066 (506h38m5.518701101s) [upload=444561]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 18:07:05.893317271 (506h38m5.764400316s) [seg_pos={0 0}, upload=444561]: Begin upload segment... Dec 26 18:07:05.893413573 (506h38m5.764496618s) [seg_pos={0 0}, upload=444561]: Done begin upload segment. Dec 26 18:07:05.893454061 (506h38m5.764537106s) [noise=true , address=145.79.227.49:4337 , piece_id=P77OCEPHI3OAC464EWXBAK3LWROAGULPQDSDC5CNFFANHHMO33MQ, seg_pos={0 0}, upload=444561]: Uploading piece... Dec 26 18:07:05.893541878 (506h38m5.764624913s) [noise=true , address=77.90.31.21:4878 , piece_id=OIM2BL3NK47PE6RGZGTBEIJQOSV5QJTPV6EWZASDAI6ICAOFP35A, seg_pos={0 0}, upload=444561]: Uploading piece... Dec 26 18:07:05.894186135 (506h38m5.76526918s) [noise=true , address=145.79.227.46:4603 , piece_id=2Z6UZAVQPKXBOI3XC3R6L4Z6QQE2Q2FBD7UGMZVKKRPTOL6OSQKQ, seg_pos={0 0}, upload=444561]: Uploading piece... Dec 26 18:07:05.894703070 (506h38m5.765786115s) [noise=true , address=145.79.227.21:4474 , piece_id=OQIDGFYSDEEEOOOZBVLZJD6KSJ2XZP6DWYHTFJ3KTTE67VPRPSJQ, seg_pos={0 0}, upload=444561]: Uploading piece... Dec 26 18:07:05.895022979 (506h38m5.766106014s) [noise=true , address=45.91.116.9:4264 , piece_id=TCHSBMRXCDNQYYLQFP3OIVLSCZS5GCWOHCUK5K46EU3GJ6GFAUUQ, seg_pos={0 0}, upload=444561]: Uploading piece... Dec 26 18:07:05.895604606 (506h38m5.766687651s) [noise=true , address=45.91.116.13:4155 , piece_id=GQSX7CVSG3ZHEPYGCKFMRN43BFDZ344DEG2NO2LXLRYGTRFUOJ5Q, seg_pos={0 0}, upload=444561]: Uploading piece... Dec 26 18:07:05.895877936 (506h38m5.766960971s) [noise=true , address=145.79.227.39:4014 , piece_id=ZS7QXCMU46IXAW6433NWCTVJYM3OMGBTJIQ2KSPJWKFCVNUW7IMQ, seg_pos={0 0}, upload=444561]: Uploading piece... Dec 26 18:07:05.896574142 (506h38m5.767657177s) [noise=true , address=77.90.31.18:4808 , piece_id=ZSI7J4C5BSGB7ROCCKOU6KWZT3PRYCCVITRWEQ3C5AKALPWZVHMQ, seg_pos={0 0}, upload=444561]: Uploading piece... Dec 26 18:07:05.896865577 (506h38m5.767948623s) [noise=true , address=145.79.227.41:4085 , piece_id=SZQWIUPASZ2CRYQVPSQXW2XE7EUV6R3VQPQRIALB3KVHWJWLAWAQ, seg_pos={0 0}, upload=444561]: Uploading piece... Dec 26 18:07:05.897453567 (506h38m5.768536602s) [noise=true , address=145.79.227.8:4330 , piece_id=4MMSYKVJMW23XYJONUFULK7GICSZGQAC6E7W65AJ4MGASDSFP3WA, seg_pos={0 0}, upload=444561]: Uploading piece... Dec 26 18:07:05.897679557 (506h38m5.768762592s) [noise=true , address=145.79.227.22:4511 , piece_id=2PEP6MXPHOL3DMM2LJUXYNMPFZB2MW4Q664SU5P5LL6ECNO34C2Q, seg_pos={0 0}, upload=444561]: Uploading piece... Dec 26 18:07:05.898400360 (506h38m5.769483395s) [noise=true , address=77.90.31.17:4773 , piece_id=MGL2FRFSSUTDQLTMI3ICWMNLRD4ZOHHVDI6RVKCXRQDZANCT4WXQ, seg_pos={0 0}, upload=444561]: Uploading piece... Dec 26 18:07:05.898672889 (506h38m5.769755934s) [noise=true , address=145.79.227.48:4303 , piece_id=LYRAHY4DMH3AJPQCBIN6URWP35W7P6CMF2W2DEUCFYD7V7F6IAWA, seg_pos={0 0}, upload=444561]: Uploading piece... Dec 26 18:07:05.899152481 (506h38m5.770235516s) [noise=true , address=145.79.227.30:4761 , piece_id=MZWR4L4PPROWB4IUBXIY4BZ56RXF22FL2YHHACOQE2DFSLEPFKUQ, seg_pos={0 0}, upload=444561]: Uploading piece... Dec 26 18:07:05.899395544 (506h38m5.770478579s) [noise=true , address=145.79.227.47:4266 , piece_id=CGYGCEZ2334TCEOVXF5YHBPF4VUXHC4FJTF3A5ZZQM6Q5JYDLUBA, seg_pos={0 0}, upload=444561]: Uploading piece... Dec 26 18:07:05.904889376 (506h38m5.775972421s) [noise=true , address=145.79.227.37:4567 , piece_id=2OIDEOOUHBE7BZCWGBC7HDAP2KCGYW644MYEPPSE6RAZWWFNQUAA, seg_pos={0 0}, upload=444561]: Uploading piece... Dec 26 18:07:07.582540777 (506h38m7.453623822s) [upload=444561]: Next returned nil segment Dec 26 18:07:07.582553913 (506h38m7.453636958s) [upload=444561]: Waiting for error group managing segments... Dec 26 18:07:07.903698448 (506h38m7.774781493s) [noise=true , address=145.79.227.47:4266 , piece_id=CGYGCEZ2334TCEOVXF5YHBPF4VUXHC4FJTF3A5ZZQM6Q5JYDLUBA, seg_pos={0 0}, upload=444561]: Done uploading piece. err: <nil> Dec 26 18:07:07.904133025 (506h38m7.77521606s) [noise=true , address=145.79.227.48:4303 , piece_id=LYRAHY4DMH3AJPQCBIN6URWP35W7P6CMF2W2DEUCFYD7V7F6IAWA, seg_pos={0 0}, upload=444561]: Done uploading piece. err: <nil> Dec 26 18:07:07.904510023 (506h38m7.775593058s) [noise=true , address=145.79.227.49:4337 , piece_id=P77OCEPHI3OAC464EWXBAK3LWROAGULPQDSDC5CNFFANHHMO33MQ, seg_pos={0 0}, upload=444561]: Done uploading piece. err: <nil> Dec 26 18:07:07.904611216 (506h38m7.775694251s) [noise=true , address=145.79.227.37:4567 , piece_id=2OIDEOOUHBE7BZCWGBC7HDAP2KCGYW644MYEPPSE6RAZWWFNQUAA, seg_pos={0 0}, upload=444561]: Done uploading piece. err: <nil> Dec 26 18:07:07.905953782 (506h38m7.777036817s) [noise=true , address=145.79.227.41:4085 , piece_id=SZQWIUPASZ2CRYQVPSQXW2XE7EUV6R3VQPQRIALB3KVHWJWLAWAQ, seg_pos={0 0}, upload=444561]: Done uploading piece. err: <nil> Dec 26 18:07:07.906362129 (506h38m7.777445164s) [noise=true , address=145.79.227.46:4603 , piece_id=2Z6UZAVQPKXBOI3XC3R6L4Z6QQE2Q2FBD7UGMZVKKRPTOL6OSQKQ, seg_pos={0 0}, upload=444561]: Done uploading piece. err: <nil> Dec 26 18:07:07.912393825 (506h38m7.78347687s) [noise=true , address=145.79.227.21:4474 , piece_id=OQIDGFYSDEEEOOOZBVLZJD6KSJ2XZP6DWYHTFJ3KTTE67VPRPSJQ, seg_pos={0 0}, upload=444561]: Done uploading piece. err: <nil> Dec 26 18:07:07.926216957 (506h38m7.797300002s) [noise=true , address=145.79.227.8:4330 , piece_id=4MMSYKVJMW23XYJONUFULK7GICSZGQAC6E7W65AJ4MGASDSFP3WA, seg_pos={0 0}, upload=444561]: Done uploading piece. err: <nil> Dec 26 18:07:07.931470309 (506h38m7.802553345s) [noise=true , address=145.79.227.39:4014 , piece_id=ZS7QXCMU46IXAW6433NWCTVJYM3OMGBTJIQ2KSPJWKFCVNUW7IMQ, seg_pos={0 0}, upload=444561]: Done uploading piece. err: <nil> Dec 26 18:07:07.931844102 (506h38m7.802927147s) [noise=true , address=145.79.227.22:4511 , piece_id=2PEP6MXPHOL3DMM2LJUXYNMPFZB2MW4Q664SU5P5LL6ECNO34C2Q, seg_pos={0 0}, upload=444561]: Done uploading piece. err: <nil> Dec 26 18:07:08.157225668 (506h38m8.028308714s) [noise=true , address=45.91.116.13:4155 , piece_id=GQSX7CVSG3ZHEPYGCKFMRN43BFDZ344DEG2NO2LXLRYGTRFUOJ5Q, seg_pos={0 0}, upload=444561]: Done uploading piece. err: <nil> Dec 26 18:07:08.162911395 (506h38m8.03399444s) [noise=true , address=45.91.116.9:4264 , piece_id=TCHSBMRXCDNQYYLQFP3OIVLSCZS5GCWOHCUK5K46EU3GJ6GFAUUQ, seg_pos={0 0}, upload=444561]: Done uploading piece. err: <nil> Dec 26 18:07:08.508925465 (506h38m8.3800085s) [noise=true , address=77.90.31.18:4808 , piece_id=ZSI7J4C5BSGB7ROCCKOU6KWZT3PRYCCVITRWEQ3C5AKALPWZVHMQ, seg_pos={0 0}, upload=444561]: Done uploading piece. err: <nil> Dec 26 18:07:08.508951195 (506h38m8.38003423s) [seg_pos={0 0}, upload=444561]: Segment reached optimal threshold of 13 pieces. Dec 26 18:07:08.509141928 (506h38m8.380224963s) [noise=true , address=77.90.31.21:4878 , piece_id=OIM2BL3NK47PE6RGZGTBEIJQOSV5QJTPV6EWZASDAI6ICAOFP35A, seg_pos={0 0}, upload=444561]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: write tcp 188.166.225.170:33740->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:33740->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33740->77.90.31.21:4878: use of closed network connection Dec 26 18:07:08.509148721 (506h38m8.380231756s) [noise=true , address=77.90.31.21:4878 , piece_id=OIM2BL3NK47PE6RGZGTBEIJQOSV5QJTPV6EWZASDAI6ICAOFP35A, seg_pos={0 0}, upload=444561]: Long tail cancellation due to optimal threshold Dec 26 18:07:08.509293136 (506h38m8.380376171s) [noise=true , address=145.79.227.30:4761 , piece_id=MZWR4L4PPROWB4IUBXIY4BZ56RXF22FL2YHHACOQE2DFSLEPFKUQ, seg_pos={0 0}, upload=444561]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 26 18:07:08.509323624 (506h38m8.380406669s) [noise=true , address=145.79.227.30:4761 , piece_id=MZWR4L4PPROWB4IUBXIY4BZ56RXF22FL2YHHACOQE2DFSLEPFKUQ, seg_pos={0 0}, upload=444561]: Long tail cancellation due to optimal threshold Dec 26 18:07:08.509430027 (506h38m8.380513062s) [noise=true , address=77.90.31.17:4773 , piece_id=MGL2FRFSSUTDQLTMI3ICWMNLRD4ZOHHVDI6RVKCXRQDZANCT4WXQ, seg_pos={0 0}, upload=444561]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: write tcp 188.166.225.170:43318->77.90.31.17:4773: use of closed network connection; write tcp 188.166.225.170:43318->77.90.31.17:4773: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43318->77.90.31.17:4773: use of closed network connection Dec 26 18:07:08.509438292 (506h38m8.380521327s) [noise=true , address=77.90.31.17:4773 , piece_id=MGL2FRFSSUTDQLTMI3ICWMNLRD4ZOHHVDI6RVKCXRQDZANCT4WXQ, seg_pos={0 0}, upload=444561]: Long tail cancellation due to optimal threshold Dec 26 18:07:08.509456648 (506h38m8.380539693s) [seg_pos={0 0}, upload=444561]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 18:07:08.509544855 (506h38m8.3806279s) [upload=444561]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 18:07:08.509553511 (506h38m8.380636556s) [upload=444561]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 18:07:08.802296606 (506h38m8.673379641s) [upload=444561]: Done uploading segments... Dec 26 18:07:08.802315842 (506h38m8.673398887s) [upload=444561]: Upload finished. err: <nil> Duration: 5.934474959s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 30.006760192s trace: https://jaeger.dev.storj.io/trace/665ea65f6e9ccc33 FAIL (2025-12-26 18:08:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== e=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789316979 (506h12m54.660400024s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789326507 (506h12m54.660409552s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789611339 (506h12m54.660694384s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789619275 (506h12m54.66070231s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789914006 (506h12m54.660997041s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789949443 (506h12m54.661032478s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790181566 (506h12m54.661264601s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790295632 (506h12m54.661378668s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790510872 (506h12m54.661593917s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790692980 (506h12m54.661776015s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790787379 (506h12m54.661870414s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790992970 (506h12m54.662076005s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791120053 (506h12m54.662203098s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791275218 (506h12m54.662358253s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791443709 (506h12m54.662526744s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:56.112313145 (506h12m55.98339619s) [upload=444191]: Next returned nil segment Dec 26 17:41:56.112330638 (506h12m55.983413683s) [upload=444191]: Waiting for error group managing segments... Dec 26 17:41:56.485322720 (506h12m56.356405765s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.485637079 (506h12m56.356720124s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.666807539 (506h12m56.537890584s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.678753987 (506h12m56.549837032s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.687788039 (506h12m56.558871074s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.694318023 (506h12m56.565401058s) [noise=true , address=145.79.227.7:4326 , piece_id=DMQ5RPI46X2U7TJTS42ZHQEF32BDJRKJJO5HWOYAPNS6VWVG5B5Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.716581187 (506h12m56.587664232s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.972183007 (506h12m56.843266042s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.011917999 (506h12m56.883001034s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.054168622 (506h12m56.925251667s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.114312777 (506h12m56.985395812s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.499846250 (506h12m57.370929285s) [noise=true , address=145.79.227.34:4919 , piece_id=YRWHW3BJVEGVJRSA6RJB3T7VJQSDTGKEZQKRICFIV2G3T7AN4MIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.612649066 (506h12m57.483732101s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649413733 (506h12m57.520496768s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649440282 (506h12m57.520523328s) [seg_pos={0 0}, upload=444191]: Segment reached optimal threshold of 14 pieces. Dec 26 17:41:57.649674729 (506h12m57.520757774s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G7ty71tPwe7cZSBNK9u74DFUCGomnxDsNvAif4C3SbP7HWwfo): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649684087 (506h12m57.520767142s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649747708 (506h12m57.520830763s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v6zXZuMmyL2MU8qb3q6gg1iwTYvndjzMjVpv8btEpeX9Lnqmu): protocol: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection Dec 26 17:41:57.649785219 (506h12m57.520868274s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649899828 (506h12m57.520982903s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649908775 (506h12m57.52099182s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650074641 (506h12m57.521157696s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.650083287 (506h12m57.521166342s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650132821 (506h12m57.521215866s) [seg_pos={0 0}, upload=444191]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 17:41:57.650220138 (506h12m57.521303183s) [upload=444191]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:41:57.650227171 (506h12m57.521310206s) [upload=444191]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:41:57.958890870 (506h12m57.829973915s) [upload=444191]: Done uploading segments... Dec 26 17:41:57.958911620 (506h12m57.829994665s) [upload=444191]: Upload finished. err: <nil> Duration: 30.006791341s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [XX...XX....XX..XX......X.XXX.....XX.X.XXX.........X..X.....X] duration: 30.003499433s trace: https://jaeger.dev.storj.io/trace/6f5ab3f88a41764a FAIL (2025-12-26 18:07:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m, Piece ID: TCHSBMRXCDNQYYLQFP3OIVLSCZS5GCWOHCUK5K46EU3GJ6GFAUUQ) context deadline exceeded; piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: ZSI7J4C5BSGB7ROCCKOU6KWZT3PRYCCVITRWEQ3C5AKALPWZVHMQ) context deadline exceeded ======LOGS====== 399 (506h38m5.518466444s) [upload=444561]: Uploading segments... Dec 26 18:07:05.647552551 (506h38m5.518635596s) [upload=444561]: Got next segment. Inline: false Dec 26 18:07:05.647613107 (506h38m5.518696142s) [upload=444561]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 18:07:05.647618066 (506h38m5.518701101s) [upload=444561]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 18:07:05.893317271 (506h38m5.764400316s) [seg_pos={0 0}, upload=444561]: Begin upload segment... Dec 26 18:07:05.893413573 (506h38m5.764496618s) [seg_pos={0 0}, upload=444561]: Done begin upload segment. Dec 26 18:07:05.893454061 (506h38m5.764537106s) [noise=true , address=145.79.227.49:4337 , piece_id=P77OCEPHI3OAC464EWXBAK3LWROAGULPQDSDC5CNFFANHHMO33MQ, seg_pos={0 0}, upload=444561]: Uploading piece... Dec 26 18:07:05.893541878 (506h38m5.764624913s) [noise=true , address=77.90.31.21:4878 , piece_id=OIM2BL3NK47PE6RGZGTBEIJQOSV5QJTPV6EWZASDAI6ICAOFP35A, seg_pos={0 0}, upload=444561]: Uploading piece... Dec 26 18:07:05.894186135 (506h38m5.76526918s) [noise=true , address=145.79.227.46:4603 , piece_id=2Z6UZAVQPKXBOI3XC3R6L4Z6QQE2Q2FBD7UGMZVKKRPTOL6OSQKQ, seg_pos={0 0}, upload=444561]: Uploading piece... Dec 26 18:07:05.894703070 (506h38m5.765786115s) [noise=true , address=145.79.227.21:4474 , piece_id=OQIDGFYSDEEEOOOZBVLZJD6KSJ2XZP6DWYHTFJ3KTTE67VPRPSJQ, seg_pos={0 0}, upload=444561]: Uploading piece... Dec 26 18:07:05.895022979 (506h38m5.766106014s) [noise=true , address=45.91.116.9:4264 , piece_id=TCHSBMRXCDNQYYLQFP3OIVLSCZS5GCWOHCUK5K46EU3GJ6GFAUUQ, seg_pos={0 0}, upload=444561]: Uploading piece... Dec 26 18:07:05.895604606 (506h38m5.766687651s) [noise=true , address=45.91.116.13:4155 , piece_id=GQSX7CVSG3ZHEPYGCKFMRN43BFDZ344DEG2NO2LXLRYGTRFUOJ5Q, seg_pos={0 0}, upload=444561]: Uploading piece... Dec 26 18:07:05.895877936 (506h38m5.766960971s) [noise=true , address=145.79.227.39:4014 , piece_id=ZS7QXCMU46IXAW6433NWCTVJYM3OMGBTJIQ2KSPJWKFCVNUW7IMQ, seg_pos={0 0}, upload=444561]: Uploading piece... Dec 26 18:07:05.896574142 (506h38m5.767657177s) [noise=true , address=77.90.31.18:4808 , piece_id=ZSI7J4C5BSGB7ROCCKOU6KWZT3PRYCCVITRWEQ3C5AKALPWZVHMQ, seg_pos={0 0}, upload=444561]: Uploading piece... Dec 26 18:07:05.896865577 (506h38m5.767948623s) [noise=true , address=145.79.227.41:4085 , piece_id=SZQWIUPASZ2CRYQVPSQXW2XE7EUV6R3VQPQRIALB3KVHWJWLAWAQ, seg_pos={0 0}, upload=444561]: Uploading piece... Dec 26 18:07:05.897453567 (506h38m5.768536602s) [noise=true , address=145.79.227.8:4330 , piece_id=4MMSYKVJMW23XYJONUFULK7GICSZGQAC6E7W65AJ4MGASDSFP3WA, seg_pos={0 0}, upload=444561]: Uploading piece... Dec 26 18:07:05.897679557 (506h38m5.768762592s) [noise=true , address=145.79.227.22:4511 , piece_id=2PEP6MXPHOL3DMM2LJUXYNMPFZB2MW4Q664SU5P5LL6ECNO34C2Q, seg_pos={0 0}, upload=444561]: Uploading piece... Dec 26 18:07:05.898400360 (506h38m5.769483395s) [noise=true , address=77.90.31.17:4773 , piece_id=MGL2FRFSSUTDQLTMI3ICWMNLRD4ZOHHVDI6RVKCXRQDZANCT4WXQ, seg_pos={0 0}, upload=444561]: Uploading piece... Dec 26 18:07:05.898672889 (506h38m5.769755934s) [noise=true , address=145.79.227.48:4303 , piece_id=LYRAHY4DMH3AJPQCBIN6URWP35W7P6CMF2W2DEUCFYD7V7F6IAWA, seg_pos={0 0}, upload=444561]: Uploading piece... Dec 26 18:07:05.899152481 (506h38m5.770235516s) [noise=true , address=145.79.227.30:4761 , piece_id=MZWR4L4PPROWB4IUBXIY4BZ56RXF22FL2YHHACOQE2DFSLEPFKUQ, seg_pos={0 0}, upload=444561]: Uploading piece... Dec 26 18:07:05.899395544 (506h38m5.770478579s) [noise=true , address=145.79.227.47:4266 , piece_id=CGYGCEZ2334TCEOVXF5YHBPF4VUXHC4FJTF3A5ZZQM6Q5JYDLUBA, seg_pos={0 0}, upload=444561]: Uploading piece... Dec 26 18:07:05.904889376 (506h38m5.775972421s) [noise=true , address=145.79.227.37:4567 , piece_id=2OIDEOOUHBE7BZCWGBC7HDAP2KCGYW644MYEPPSE6RAZWWFNQUAA, seg_pos={0 0}, upload=444561]: Uploading piece... Dec 26 18:07:07.582540777 (506h38m7.453623822s) [upload=444561]: Next returned nil segment Dec 26 18:07:07.582553913 (506h38m7.453636958s) [upload=444561]: Waiting for error group managing segments... Dec 26 18:07:07.903698448 (506h38m7.774781493s) [noise=true , address=145.79.227.47:4266 , piece_id=CGYGCEZ2334TCEOVXF5YHBPF4VUXHC4FJTF3A5ZZQM6Q5JYDLUBA, seg_pos={0 0}, upload=444561]: Done uploading piece. err: <nil> Dec 26 18:07:07.904133025 (506h38m7.77521606s) [noise=true , address=145.79.227.48:4303 , piece_id=LYRAHY4DMH3AJPQCBIN6URWP35W7P6CMF2W2DEUCFYD7V7F6IAWA, seg_pos={0 0}, upload=444561]: Done uploading piece. err: <nil> Dec 26 18:07:07.904510023 (506h38m7.775593058s) [noise=true , address=145.79.227.49:4337 , piece_id=P77OCEPHI3OAC464EWXBAK3LWROAGULPQDSDC5CNFFANHHMO33MQ, seg_pos={0 0}, upload=444561]: Done uploading piece. err: <nil> Dec 26 18:07:07.904611216 (506h38m7.775694251s) [noise=true , address=145.79.227.37:4567 , piece_id=2OIDEOOUHBE7BZCWGBC7HDAP2KCGYW644MYEPPSE6RAZWWFNQUAA, seg_pos={0 0}, upload=444561]: Done uploading piece. err: <nil> Dec 26 18:07:07.905953782 (506h38m7.777036817s) [noise=true , address=145.79.227.41:4085 , piece_id=SZQWIUPASZ2CRYQVPSQXW2XE7EUV6R3VQPQRIALB3KVHWJWLAWAQ, seg_pos={0 0}, upload=444561]: Done uploading piece. err: <nil> Dec 26 18:07:07.906362129 (506h38m7.777445164s) [noise=true , address=145.79.227.46:4603 , piece_id=2Z6UZAVQPKXBOI3XC3R6L4Z6QQE2Q2FBD7UGMZVKKRPTOL6OSQKQ, seg_pos={0 0}, upload=444561]: Done uploading piece. err: <nil> Dec 26 18:07:07.912393825 (506h38m7.78347687s) [noise=true , address=145.79.227.21:4474 , piece_id=OQIDGFYSDEEEOOOZBVLZJD6KSJ2XZP6DWYHTFJ3KTTE67VPRPSJQ, seg_pos={0 0}, upload=444561]: Done uploading piece. err: <nil> Dec 26 18:07:07.926216957 (506h38m7.797300002s) [noise=true , address=145.79.227.8:4330 , piece_id=4MMSYKVJMW23XYJONUFULK7GICSZGQAC6E7W65AJ4MGASDSFP3WA, seg_pos={0 0}, upload=444561]: Done uploading piece. err: <nil> Dec 26 18:07:07.931470309 (506h38m7.802553345s) [noise=true , address=145.79.227.39:4014 , piece_id=ZS7QXCMU46IXAW6433NWCTVJYM3OMGBTJIQ2KSPJWKFCVNUW7IMQ, seg_pos={0 0}, upload=444561]: Done uploading piece. err: <nil> Dec 26 18:07:07.931844102 (506h38m7.802927147s) [noise=true , address=145.79.227.22:4511 , piece_id=2PEP6MXPHOL3DMM2LJUXYNMPFZB2MW4Q664SU5P5LL6ECNO34C2Q, seg_pos={0 0}, upload=444561]: Done uploading piece. err: <nil> Dec 26 18:07:08.157225668 (506h38m8.028308714s) [noise=true , address=45.91.116.13:4155 , piece_id=GQSX7CVSG3ZHEPYGCKFMRN43BFDZ344DEG2NO2LXLRYGTRFUOJ5Q, seg_pos={0 0}, upload=444561]: Done uploading piece. err: <nil> Dec 26 18:07:08.162911395 (506h38m8.03399444s) [noise=true , address=45.91.116.9:4264 , piece_id=TCHSBMRXCDNQYYLQFP3OIVLSCZS5GCWOHCUK5K46EU3GJ6GFAUUQ, seg_pos={0 0}, upload=444561]: Done uploading piece. err: <nil> Dec 26 18:07:08.508925465 (506h38m8.3800085s) [noise=true , address=77.90.31.18:4808 , piece_id=ZSI7J4C5BSGB7ROCCKOU6KWZT3PRYCCVITRWEQ3C5AKALPWZVHMQ, seg_pos={0 0}, upload=444561]: Done uploading piece. err: <nil> Dec 26 18:07:08.508951195 (506h38m8.38003423s) [seg_pos={0 0}, upload=444561]: Segment reached optimal threshold of 13 pieces. Dec 26 18:07:08.509141928 (506h38m8.380224963s) [noise=true , address=77.90.31.21:4878 , piece_id=OIM2BL3NK47PE6RGZGTBEIJQOSV5QJTPV6EWZASDAI6ICAOFP35A, seg_pos={0 0}, upload=444561]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: write tcp 188.166.225.170:33740->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:33740->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33740->77.90.31.21:4878: use of closed network connection Dec 26 18:07:08.509148721 (506h38m8.380231756s) [noise=true , address=77.90.31.21:4878 , piece_id=OIM2BL3NK47PE6RGZGTBEIJQOSV5QJTPV6EWZASDAI6ICAOFP35A, seg_pos={0 0}, upload=444561]: Long tail cancellation due to optimal threshold Dec 26 18:07:08.509293136 (506h38m8.380376171s) [noise=true , address=145.79.227.30:4761 , piece_id=MZWR4L4PPROWB4IUBXIY4BZ56RXF22FL2YHHACOQE2DFSLEPFKUQ, seg_pos={0 0}, upload=444561]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 26 18:07:08.509323624 (506h38m8.380406669s) [noise=true , address=145.79.227.30:4761 , piece_id=MZWR4L4PPROWB4IUBXIY4BZ56RXF22FL2YHHACOQE2DFSLEPFKUQ, seg_pos={0 0}, upload=444561]: Long tail cancellation due to optimal threshold Dec 26 18:07:08.509430027 (506h38m8.380513062s) [noise=true , address=77.90.31.17:4773 , piece_id=MGL2FRFSSUTDQLTMI3ICWMNLRD4ZOHHVDI6RVKCXRQDZANCT4WXQ, seg_pos={0 0}, upload=444561]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: write tcp 188.166.225.170:43318->77.90.31.17:4773: use of closed network connection; write tcp 188.166.225.170:43318->77.90.31.17:4773: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43318->77.90.31.17:4773: use of closed network connection Dec 26 18:07:08.509438292 (506h38m8.380521327s) [noise=true , address=77.90.31.17:4773 , piece_id=MGL2FRFSSUTDQLTMI3ICWMNLRD4ZOHHVDI6RVKCXRQDZANCT4WXQ, seg_pos={0 0}, upload=444561]: Long tail cancellation due to optimal threshold Dec 26 18:07:08.509456648 (506h38m8.380539693s) [seg_pos={0 0}, upload=444561]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 18:07:08.509544855 (506h38m8.3806279s) [upload=444561]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 18:07:08.509553511 (506h38m8.380636556s) [upload=444561]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 18:07:08.802296606 (506h38m8.673379641s) [upload=444561]: Done uploading segments... Dec 26 18:07:08.802315842 (506h38m8.673398887s) [upload=444561]: Upload finished. err: <nil> Duration: 30.003526785s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 33.572999476s trace: https://jaeger.dev.storj.io/trace/665ea65f6e9ccc33 FAIL (2025-12-26 18:06:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== e=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789316979 (506h12m54.660400024s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789326507 (506h12m54.660409552s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789611339 (506h12m54.660694384s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789619275 (506h12m54.66070231s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789914006 (506h12m54.660997041s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789949443 (506h12m54.661032478s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790181566 (506h12m54.661264601s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790295632 (506h12m54.661378668s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790510872 (506h12m54.661593917s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790692980 (506h12m54.661776015s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790787379 (506h12m54.661870414s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790992970 (506h12m54.662076005s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791120053 (506h12m54.662203098s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791275218 (506h12m54.662358253s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791443709 (506h12m54.662526744s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:56.112313145 (506h12m55.98339619s) [upload=444191]: Next returned nil segment Dec 26 17:41:56.112330638 (506h12m55.983413683s) [upload=444191]: Waiting for error group managing segments... Dec 26 17:41:56.485322720 (506h12m56.356405765s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.485637079 (506h12m56.356720124s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.666807539 (506h12m56.537890584s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.678753987 (506h12m56.549837032s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.687788039 (506h12m56.558871074s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.694318023 (506h12m56.565401058s) [noise=true , address=145.79.227.7:4326 , piece_id=DMQ5RPI46X2U7TJTS42ZHQEF32BDJRKJJO5HWOYAPNS6VWVG5B5Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.716581187 (506h12m56.587664232s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.972183007 (506h12m56.843266042s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.011917999 (506h12m56.883001034s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.054168622 (506h12m56.925251667s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.114312777 (506h12m56.985395812s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.499846250 (506h12m57.370929285s) [noise=true , address=145.79.227.34:4919 , piece_id=YRWHW3BJVEGVJRSA6RJB3T7VJQSDTGKEZQKRICFIV2G3T7AN4MIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.612649066 (506h12m57.483732101s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649413733 (506h12m57.520496768s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649440282 (506h12m57.520523328s) [seg_pos={0 0}, upload=444191]: Segment reached optimal threshold of 14 pieces. Dec 26 17:41:57.649674729 (506h12m57.520757774s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G7ty71tPwe7cZSBNK9u74DFUCGomnxDsNvAif4C3SbP7HWwfo): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649684087 (506h12m57.520767142s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649747708 (506h12m57.520830763s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v6zXZuMmyL2MU8qb3q6gg1iwTYvndjzMjVpv8btEpeX9Lnqmu): protocol: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection Dec 26 17:41:57.649785219 (506h12m57.520868274s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649899828 (506h12m57.520982903s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649908775 (506h12m57.52099182s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650074641 (506h12m57.521157696s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.650083287 (506h12m57.521166342s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650132821 (506h12m57.521215866s) [seg_pos={0 0}, upload=444191]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 17:41:57.650220138 (506h12m57.521303183s) [upload=444191]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:41:57.650227171 (506h12m57.521310206s) [upload=444191]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:41:57.958890870 (506h12m57.829973915s) [upload=444191]: Done uploading segments... Dec 26 17:41:57.958911620 (506h12m57.829994665s) [upload=444191]: Upload finished. err: <nil> Duration: 33.57302845s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [.XX...XX....XX..XX......X.XXX.....XX.X.XXX.........X..X.....] duration: 6.85460627s trace: https://jaeger.dev.storj.io/trace/243f66ab99efb3ee FAIL (2025-12-26 18:06:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: UVYN4DXLISSVHNJFRXG6IZYN2LW7V6ZKFQRHNTB24T2UEAG2KW4Q) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: 6NSZ2L7LU5TU3OXOM6G7FVDUBDONHLESTCHJCILFY7WS7DU7YEKA) context deadline exceeded ======LOGS====== Dec 26 18:03:10.947413330 (506h34m10.818496385s) [upload=444504]: Starting upload Dec 26 18:03:10.947509102 (506h34m10.818592137s) [upload=444504]: Done starting upload Dec 26 18:03:10.947593964 (506h34m10.818676999s) [upload=444504]: Uploading segments... Dec 26 18:03:10.947770049 (506h34m10.818853084s) [upload=444504]: Got next segment. Inline: false Dec 26 18:03:10.947835494 (506h34m10.818918539s) [upload=444504]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 18:03:10.947841495 (506h34m10.81892453s) [upload=444504]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 18:03:11.196337715 (506h34m11.06742076s) [seg_pos={0 0}, upload=444504]: Begin upload segment... Dec 26 18:03:11.196399363 (506h34m11.067482398s) [seg_pos={0 0}, upload=444504]: Done begin upload segment. Dec 26 18:03:11.196462302 (506h34m11.067545347s) [noise=true , address=145.79.227.39:4015 , piece_id=3EIJJ2FW3KFAE7OHXOH3M2BARFKUPV6AER472K5F7KHJFLXTG63Q, seg_pos={0 0}, upload=444504]: Uploading piece... Dec 26 18:03:11.196638999 (506h34m11.067722044s) [noise=true , address=145.79.227.4:4186 , piece_id=HHX2ZHYKTMM55GULLNOMQJ2WPZRP444XFAO65HU2RVAYWUCDZZ7A, seg_pos={0 0}, upload=444504]: Uploading piece... Dec 26 18:03:11.197357437 (506h34m11.068440482s) [noise=true , address=45.91.116.9:4264 , piece_id=V6G45QGAWGT3MWIYQGPGHTF5XBCKD4G4G3FBXXFYGOKQBHE33LRQ, seg_pos={0 0}, upload=444504]: Uploading piece... Dec 26 18:03:11.197503895 (506h34m11.06858694s) [noise=true , address=77.90.31.11:4553 , piece_id=W3JRDU7L66LTYHTGHMK3PE554MUYSOIQMSSRBL34K4IPOSAE474A, seg_pos={0 0}, upload=444504]: Uploading piece... Dec 26 18:03:11.198008657 (506h34m11.069091702s) [noise=true , address=145.79.227.49:4339 , piece_id=2M7VCO7OY3MU3WWZNM7FCYQHJMTQO35BGT57SSYDMHJCPX2LZT2A, seg_pos={0 0}, upload=444504]: Uploading piece... Dec 26 18:03:11.198082277 (506h34m11.069165322s) [noise=true , address=145.79.227.41:4086 , piece_id=RAUVGKJR7TNL5YJHPNAUIPI72CX5GCHNBT4HOR3IUB4AKKCLMO5Q, seg_pos={0 0}, upload=444504]: Uploading piece... Dec 26 18:03:11.198472169 (506h34m11.069555214s) [noise=true , address=77.90.31.13:4627 , piece_id=FYB4X6MKKBN3L4RW3UKMR3YAX4WOQN63LQCN2MWHZJG3TVESVRQQ, seg_pos={0 0}, upload=444504]: Uploading piece... Dec 26 18:03:11.198538466 (506h34m11.069621501s) [noise=true , address=145.79.227.14:4839 , piece_id=EF5AB6ZDBM5O5DKMUY3JAG2QC7YT2DW5ODTK2Z7IUSKTBCCTXFSA, seg_pos={0 0}, upload=444504]: Uploading piece... Dec 26 18:03:11.199136184 (506h34m11.070219229s) [noise=true , address=145.79.227.47:4267 , piece_id=KWHHCDIZ6KRBJK335BTJ2XN7DTZVIRF3LSNLRTLZSFKKRRHKPHUA, seg_pos={0 0}, upload=444504]: Uploading piece... Dec 26 18:03:11.199257004 (506h34m11.070340049s) [noise=true , address=145.79.227.15:4874 , piece_id=6JDOK6U5OIFFQTEDA5IVGHFFZVHUXMRF4GTAM5ETPZNYGDIV7LPQ, seg_pos={0 0}, upload=444504]: Uploading piece... Dec 26 18:03:11.199660883 (506h34m11.070743928s) [noise=true , address=45.91.116.13:4156 , piece_id=UVYN4DXLISSVHNJFRXG6IZYN2LW7V6ZKFQRHNTB24T2UEAG2KW4Q, seg_pos={0 0}, upload=444504]: Uploading piece... Dec 26 18:03:11.199676052 (506h34m11.070759097s) [noise=true , address=45.91.116.10:4081 , piece_id=QC77DKAG4LZRC3UFN5UBTFBY7EC7P6Q5RUVITKZNUDVXL4LDTI6A, seg_pos={0 0}, upload=444504]: Uploading piece... Dec 26 18:03:11.200151407 (506h34m11.071234452s) [noise=true , address=145.79.227.45:4229 , piece_id=4QVKJHC6IXIKWJIW6XHTLWJGIHUW5HZ3ORB4B65527NCRNQURUNA, seg_pos={0 0}, upload=444504]: Uploading piece... Dec 26 18:03:11.200184359 (506h34m11.071267404s) [noise=true , address=45.91.116.14:4191 , piece_id=LG4MVMFR5622ZJOOMYG3IPLMMOPSWSSMHWOFCU76WXIAJIW4XUBQ, seg_pos={0 0}, upload=444504]: Uploading piece... Dec 26 18:03:11.200606214 (506h34m11.071689259s) [noise=true , address=45.91.116.8:4045 , piece_id=6NSZ2L7LU5TU3OXOM6G7FVDUBDONHLESTCHJCILFY7WS7DU7YEKA, seg_pos={0 0}, upload=444504]: Uploading piece... Dec 26 18:03:11.201053194 (506h34m11.072136239s) [noise=true , address=145.79.227.46:4601 , piece_id=JFFS2GUNTINTQBO7EEB324PK37ZXJ2ARH6XJ4L53GQV25WYQRAUA, seg_pos={0 0}, upload=444504]: Uploading piece... Dec 26 18:03:12.350290563 (506h34m12.221373608s) [noise=true , address=77.90.31.11:4553 , piece_id=W3JRDU7L66LTYHTGHMK3PE554MUYSOIQMSSRBL34K4IPOSAE474A, seg_pos={0 0}, upload=444504]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: storage node overloaded, request limit: 2000 Dec 26 18:03:12.915003815 (506h34m12.78608685s) [upload=444504]: Next returned nil segment Dec 26 18:03:12.915016951 (506h34m12.786099996s) [upload=444504]: Waiting for error group managing segments... Dec 26 18:03:13.194034659 (506h34m13.065117694s) [noise=true , address=145.79.227.15:4874 , piece_id=6JDOK6U5OIFFQTEDA5IVGHFFZVHUXMRF4GTAM5ETPZNYGDIV7LPQ, seg_pos={0 0}, upload=444504]: Done uploading piece. err: <nil> Dec 26 18:03:13.199551184 (506h34m13.070634219s) [noise=true , address=145.79.227.46:4601 , piece_id=JFFS2GUNTINTQBO7EEB324PK37ZXJ2ARH6XJ4L53GQV25WYQRAUA, seg_pos={0 0}, upload=444504]: Done uploading piece. err: <nil> Dec 26 18:03:13.204584307 (506h34m13.075667352s) [noise=true , address=145.79.227.49:4339 , piece_id=2M7VCO7OY3MU3WWZNM7FCYQHJMTQO35BGT57SSYDMHJCPX2LZT2A, seg_pos={0 0}, upload=444504]: Done uploading piece. err: <nil> Dec 26 18:03:13.206994768 (506h34m13.078077813s) [noise=true , address=145.79.227.39:4015 , piece_id=3EIJJ2FW3KFAE7OHXOH3M2BARFKUPV6AER472K5F7KHJFLXTG63Q, seg_pos={0 0}, upload=444504]: Done uploading piece. err: <nil> Dec 26 18:03:13.219091742 (506h34m13.090174777s) [noise=true , address=145.79.227.47:4267 , piece_id=KWHHCDIZ6KRBJK335BTJ2XN7DTZVIRF3LSNLRTLZSFKKRRHKPHUA, seg_pos={0 0}, upload=444504]: Done uploading piece. err: <nil> Dec 26 18:03:13.220706828 (506h34m13.091789863s) [noise=true , address=145.79.227.45:4229 , piece_id=4QVKJHC6IXIKWJIW6XHTLWJGIHUW5HZ3ORB4B65527NCRNQURUNA, seg_pos={0 0}, upload=444504]: Done uploading piece. err: <nil> Dec 26 18:03:13.230304272 (506h34m13.101387307s) [noise=true , address=145.79.227.14:4839 , piece_id=EF5AB6ZDBM5O5DKMUY3JAG2QC7YT2DW5ODTK2Z7IUSKTBCCTXFSA, seg_pos={0 0}, upload=444504]: Done uploading piece. err: <nil> Dec 26 18:03:13.232301426 (506h34m13.103384461s) [noise=true , address=145.79.227.4:4186 , piece_id=HHX2ZHYKTMM55GULLNOMQJ2WPZRP444XFAO65HU2RVAYWUCDZZ7A, seg_pos={0 0}, upload=444504]: Done uploading piece. err: <nil> Dec 26 18:03:13.439749002 (506h34m13.310832047s) [noise=true , address=45.91.116.9:4264 , piece_id=V6G45QGAWGT3MWIYQGPGHTF5XBCKD4G4G3FBXXFYGOKQBHE33LRQ, seg_pos={0 0}, upload=444504]: Done uploading piece. err: <nil> Dec 26 18:03:13.491708832 (506h34m13.362791878s) [noise=true , address=45.91.116.13:4156 , piece_id=UVYN4DXLISSVHNJFRXG6IZYN2LW7V6ZKFQRHNTB24T2UEAG2KW4Q, seg_pos={0 0}, upload=444504]: Done uploading piece. err: <nil> Dec 26 18:03:13.514759485 (506h34m13.38584253s) [noise=true , address=77.90.31.13:4627 , piece_id=FYB4X6MKKBN3L4RW3UKMR3YAX4WOQN63LQCN2MWHZJG3TVESVRQQ, seg_pos={0 0}, upload=444504]: Done uploading piece. err: <nil> Dec 26 18:03:13.585088304 (506h34m13.456171349s) [noise=true , address=45.91.116.14:4191 , piece_id=LG4MVMFR5622ZJOOMYG3IPLMMOPSWSSMHWOFCU76WXIAJIW4XUBQ, seg_pos={0 0}, upload=444504]: Done uploading piece. err: <nil> Dec 26 18:03:14.293444978 (506h34m14.164528033s) [noise=true , address=45.91.116.8:4045 , piece_id=6NSZ2L7LU5TU3OXOM6G7FVDUBDONHLESTCHJCILFY7WS7DU7YEKA, seg_pos={0 0}, upload=444504]: Done uploading piece. err: <nil> Dec 26 18:03:14.293466870 (506h34m14.164549925s) [seg_pos={0 0}, upload=444504]: Segment reached optimal threshold of 13 pieces. Dec 26 18:03:14.293642154 (506h34m14.164725219s) [noise=true , address=145.79.227.41:4086 , piece_id=RAUVGKJR7TNL5YJHPNAUIPI72CX5GCHNBT4HOR3IUB4AKKCLMO5Q, seg_pos={0 0}, upload=444504]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm): protocol: expected piece hash; context canceled; EOF Dec 26 18:03:14.293652554 (506h34m14.164735589s) [noise=true , address=145.79.227.41:4086 , piece_id=RAUVGKJR7TNL5YJHPNAUIPI72CX5GCHNBT4HOR3IUB4AKKCLMO5Q, seg_pos={0 0}, upload=444504]: Long tail cancellation due to optimal threshold Dec 26 18:03:14.294191048 (506h34m14.165274094s) [noise=true , address=45.91.116.10:4081 , piece_id=QC77DKAG4LZRC3UFN5UBTFBY7EC7P6Q5RUVITKZNUDVXL4LDTI6A, seg_pos={0 0}, upload=444504]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: context canceled; context canceled Dec 26 18:03:14.294210877 (506h34m14.165293922s) [noise=true , address=45.91.116.10:4081 , piece_id=QC77DKAG4LZRC3UFN5UBTFBY7EC7P6Q5RUVITKZNUDVXL4LDTI6A, seg_pos={0 0}, upload=444504]: Long tail cancellation due to optimal threshold Dec 26 18:03:14.294238509 (506h34m14.165321555s) [seg_pos={0 0}, upload=444504]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 18:03:14.294369770 (506h34m14.165452815s) [upload=444504]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 18:03:14.294377755 (506h34m14.1654608s) [upload=444504]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 18:03:14.588297970 (506h34m14.459381005s) [upload=444504]: Done uploading segments... Dec 26 18:03:14.588319040 (506h34m14.459402075s) [upload=444504]: Upload finished. err: <nil> Duration: 6.854637549s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [..XX...XX....XX..XX......X.XXX.....XX.X.XXX.........X..X....] duration: 5.77116667s trace: https://jaeger.dev.storj.io/trace/243f66ab99efb3ee FAIL (2025-12-26 18:05:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: UVYN4DXLISSVHNJFRXG6IZYN2LW7V6ZKFQRHNTB24T2UEAG2KW4Q) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: 6NSZ2L7LU5TU3OXOM6G7FVDUBDONHLESTCHJCILFY7WS7DU7YEKA) context deadline exceeded ======LOGS====== Dec 26 18:03:10.947413330 (506h34m10.818496385s) [upload=444504]: Starting upload Dec 26 18:03:10.947509102 (506h34m10.818592137s) [upload=444504]: Done starting upload Dec 26 18:03:10.947593964 (506h34m10.818676999s) [upload=444504]: Uploading segments... Dec 26 18:03:10.947770049 (506h34m10.818853084s) [upload=444504]: Got next segment. Inline: false Dec 26 18:03:10.947835494 (506h34m10.818918539s) [upload=444504]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 18:03:10.947841495 (506h34m10.81892453s) [upload=444504]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 18:03:11.196337715 (506h34m11.06742076s) [seg_pos={0 0}, upload=444504]: Begin upload segment... Dec 26 18:03:11.196399363 (506h34m11.067482398s) [seg_pos={0 0}, upload=444504]: Done begin upload segment. Dec 26 18:03:11.196462302 (506h34m11.067545347s) [noise=true , address=145.79.227.39:4015 , piece_id=3EIJJ2FW3KFAE7OHXOH3M2BARFKUPV6AER472K5F7KHJFLXTG63Q, seg_pos={0 0}, upload=444504]: Uploading piece... Dec 26 18:03:11.196638999 (506h34m11.067722044s) [noise=true , address=145.79.227.4:4186 , piece_id=HHX2ZHYKTMM55GULLNOMQJ2WPZRP444XFAO65HU2RVAYWUCDZZ7A, seg_pos={0 0}, upload=444504]: Uploading piece... Dec 26 18:03:11.197357437 (506h34m11.068440482s) [noise=true , address=45.91.116.9:4264 , piece_id=V6G45QGAWGT3MWIYQGPGHTF5XBCKD4G4G3FBXXFYGOKQBHE33LRQ, seg_pos={0 0}, upload=444504]: Uploading piece... Dec 26 18:03:11.197503895 (506h34m11.06858694s) [noise=true , address=77.90.31.11:4553 , piece_id=W3JRDU7L66LTYHTGHMK3PE554MUYSOIQMSSRBL34K4IPOSAE474A, seg_pos={0 0}, upload=444504]: Uploading piece... Dec 26 18:03:11.198008657 (506h34m11.069091702s) [noise=true , address=145.79.227.49:4339 , piece_id=2M7VCO7OY3MU3WWZNM7FCYQHJMTQO35BGT57SSYDMHJCPX2LZT2A, seg_pos={0 0}, upload=444504]: Uploading piece... Dec 26 18:03:11.198082277 (506h34m11.069165322s) [noise=true , address=145.79.227.41:4086 , piece_id=RAUVGKJR7TNL5YJHPNAUIPI72CX5GCHNBT4HOR3IUB4AKKCLMO5Q, seg_pos={0 0}, upload=444504]: Uploading piece... Dec 26 18:03:11.198472169 (506h34m11.069555214s) [noise=true , address=77.90.31.13:4627 , piece_id=FYB4X6MKKBN3L4RW3UKMR3YAX4WOQN63LQCN2MWHZJG3TVESVRQQ, seg_pos={0 0}, upload=444504]: Uploading piece... Dec 26 18:03:11.198538466 (506h34m11.069621501s) [noise=true , address=145.79.227.14:4839 , piece_id=EF5AB6ZDBM5O5DKMUY3JAG2QC7YT2DW5ODTK2Z7IUSKTBCCTXFSA, seg_pos={0 0}, upload=444504]: Uploading piece... Dec 26 18:03:11.199136184 (506h34m11.070219229s) [noise=true , address=145.79.227.47:4267 , piece_id=KWHHCDIZ6KRBJK335BTJ2XN7DTZVIRF3LSNLRTLZSFKKRRHKPHUA, seg_pos={0 0}, upload=444504]: Uploading piece... Dec 26 18:03:11.199257004 (506h34m11.070340049s) [noise=true , address=145.79.227.15:4874 , piece_id=6JDOK6U5OIFFQTEDA5IVGHFFZVHUXMRF4GTAM5ETPZNYGDIV7LPQ, seg_pos={0 0}, upload=444504]: Uploading piece... Dec 26 18:03:11.199660883 (506h34m11.070743928s) [noise=true , address=45.91.116.13:4156 , piece_id=UVYN4DXLISSVHNJFRXG6IZYN2LW7V6ZKFQRHNTB24T2UEAG2KW4Q, seg_pos={0 0}, upload=444504]: Uploading piece... Dec 26 18:03:11.199676052 (506h34m11.070759097s) [noise=true , address=45.91.116.10:4081 , piece_id=QC77DKAG4LZRC3UFN5UBTFBY7EC7P6Q5RUVITKZNUDVXL4LDTI6A, seg_pos={0 0}, upload=444504]: Uploading piece... Dec 26 18:03:11.200151407 (506h34m11.071234452s) [noise=true , address=145.79.227.45:4229 , piece_id=4QVKJHC6IXIKWJIW6XHTLWJGIHUW5HZ3ORB4B65527NCRNQURUNA, seg_pos={0 0}, upload=444504]: Uploading piece... Dec 26 18:03:11.200184359 (506h34m11.071267404s) [noise=true , address=45.91.116.14:4191 , piece_id=LG4MVMFR5622ZJOOMYG3IPLMMOPSWSSMHWOFCU76WXIAJIW4XUBQ, seg_pos={0 0}, upload=444504]: Uploading piece... Dec 26 18:03:11.200606214 (506h34m11.071689259s) [noise=true , address=45.91.116.8:4045 , piece_id=6NSZ2L7LU5TU3OXOM6G7FVDUBDONHLESTCHJCILFY7WS7DU7YEKA, seg_pos={0 0}, upload=444504]: Uploading piece... Dec 26 18:03:11.201053194 (506h34m11.072136239s) [noise=true , address=145.79.227.46:4601 , piece_id=JFFS2GUNTINTQBO7EEB324PK37ZXJ2ARH6XJ4L53GQV25WYQRAUA, seg_pos={0 0}, upload=444504]: Uploading piece... Dec 26 18:03:12.350290563 (506h34m12.221373608s) [noise=true , address=77.90.31.11:4553 , piece_id=W3JRDU7L66LTYHTGHMK3PE554MUYSOIQMSSRBL34K4IPOSAE474A, seg_pos={0 0}, upload=444504]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: storage node overloaded, request limit: 2000 Dec 26 18:03:12.915003815 (506h34m12.78608685s) [upload=444504]: Next returned nil segment Dec 26 18:03:12.915016951 (506h34m12.786099996s) [upload=444504]: Waiting for error group managing segments... Dec 26 18:03:13.194034659 (506h34m13.065117694s) [noise=true , address=145.79.227.15:4874 , piece_id=6JDOK6U5OIFFQTEDA5IVGHFFZVHUXMRF4GTAM5ETPZNYGDIV7LPQ, seg_pos={0 0}, upload=444504]: Done uploading piece. err: <nil> Dec 26 18:03:13.199551184 (506h34m13.070634219s) [noise=true , address=145.79.227.46:4601 , piece_id=JFFS2GUNTINTQBO7EEB324PK37ZXJ2ARH6XJ4L53GQV25WYQRAUA, seg_pos={0 0}, upload=444504]: Done uploading piece. err: <nil> Dec 26 18:03:13.204584307 (506h34m13.075667352s) [noise=true , address=145.79.227.49:4339 , piece_id=2M7VCO7OY3MU3WWZNM7FCYQHJMTQO35BGT57SSYDMHJCPX2LZT2A, seg_pos={0 0}, upload=444504]: Done uploading piece. err: <nil> Dec 26 18:03:13.206994768 (506h34m13.078077813s) [noise=true , address=145.79.227.39:4015 , piece_id=3EIJJ2FW3KFAE7OHXOH3M2BARFKUPV6AER472K5F7KHJFLXTG63Q, seg_pos={0 0}, upload=444504]: Done uploading piece. err: <nil> Dec 26 18:03:13.219091742 (506h34m13.090174777s) [noise=true , address=145.79.227.47:4267 , piece_id=KWHHCDIZ6KRBJK335BTJ2XN7DTZVIRF3LSNLRTLZSFKKRRHKPHUA, seg_pos={0 0}, upload=444504]: Done uploading piece. err: <nil> Dec 26 18:03:13.220706828 (506h34m13.091789863s) [noise=true , address=145.79.227.45:4229 , piece_id=4QVKJHC6IXIKWJIW6XHTLWJGIHUW5HZ3ORB4B65527NCRNQURUNA, seg_pos={0 0}, upload=444504]: Done uploading piece. err: <nil> Dec 26 18:03:13.230304272 (506h34m13.101387307s) [noise=true , address=145.79.227.14:4839 , piece_id=EF5AB6ZDBM5O5DKMUY3JAG2QC7YT2DW5ODTK2Z7IUSKTBCCTXFSA, seg_pos={0 0}, upload=444504]: Done uploading piece. err: <nil> Dec 26 18:03:13.232301426 (506h34m13.103384461s) [noise=true , address=145.79.227.4:4186 , piece_id=HHX2ZHYKTMM55GULLNOMQJ2WPZRP444XFAO65HU2RVAYWUCDZZ7A, seg_pos={0 0}, upload=444504]: Done uploading piece. err: <nil> Dec 26 18:03:13.439749002 (506h34m13.310832047s) [noise=true , address=45.91.116.9:4264 , piece_id=V6G45QGAWGT3MWIYQGPGHTF5XBCKD4G4G3FBXXFYGOKQBHE33LRQ, seg_pos={0 0}, upload=444504]: Done uploading piece. err: <nil> Dec 26 18:03:13.491708832 (506h34m13.362791878s) [noise=true , address=45.91.116.13:4156 , piece_id=UVYN4DXLISSVHNJFRXG6IZYN2LW7V6ZKFQRHNTB24T2UEAG2KW4Q, seg_pos={0 0}, upload=444504]: Done uploading piece. err: <nil> Dec 26 18:03:13.514759485 (506h34m13.38584253s) [noise=true , address=77.90.31.13:4627 , piece_id=FYB4X6MKKBN3L4RW3UKMR3YAX4WOQN63LQCN2MWHZJG3TVESVRQQ, seg_pos={0 0}, upload=444504]: Done uploading piece. err: <nil> Dec 26 18:03:13.585088304 (506h34m13.456171349s) [noise=true , address=45.91.116.14:4191 , piece_id=LG4MVMFR5622ZJOOMYG3IPLMMOPSWSSMHWOFCU76WXIAJIW4XUBQ, seg_pos={0 0}, upload=444504]: Done uploading piece. err: <nil> Dec 26 18:03:14.293444978 (506h34m14.164528033s) [noise=true , address=45.91.116.8:4045 , piece_id=6NSZ2L7LU5TU3OXOM6G7FVDUBDONHLESTCHJCILFY7WS7DU7YEKA, seg_pos={0 0}, upload=444504]: Done uploading piece. err: <nil> Dec 26 18:03:14.293466870 (506h34m14.164549925s) [seg_pos={0 0}, upload=444504]: Segment reached optimal threshold of 13 pieces. Dec 26 18:03:14.293642154 (506h34m14.164725219s) [noise=true , address=145.79.227.41:4086 , piece_id=RAUVGKJR7TNL5YJHPNAUIPI72CX5GCHNBT4HOR3IUB4AKKCLMO5Q, seg_pos={0 0}, upload=444504]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm): protocol: expected piece hash; context canceled; EOF Dec 26 18:03:14.293652554 (506h34m14.164735589s) [noise=true , address=145.79.227.41:4086 , piece_id=RAUVGKJR7TNL5YJHPNAUIPI72CX5GCHNBT4HOR3IUB4AKKCLMO5Q, seg_pos={0 0}, upload=444504]: Long tail cancellation due to optimal threshold Dec 26 18:03:14.294191048 (506h34m14.165274094s) [noise=true , address=45.91.116.10:4081 , piece_id=QC77DKAG4LZRC3UFN5UBTFBY7EC7P6Q5RUVITKZNUDVXL4LDTI6A, seg_pos={0 0}, upload=444504]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: context canceled; context canceled Dec 26 18:03:14.294210877 (506h34m14.165293922s) [noise=true , address=45.91.116.10:4081 , piece_id=QC77DKAG4LZRC3UFN5UBTFBY7EC7P6Q5RUVITKZNUDVXL4LDTI6A, seg_pos={0 0}, upload=444504]: Long tail cancellation due to optimal threshold Dec 26 18:03:14.294238509 (506h34m14.165321555s) [seg_pos={0 0}, upload=444504]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 18:03:14.294369770 (506h34m14.165452815s) [upload=444504]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 18:03:14.294377755 (506h34m14.1654608s) [upload=444504]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 18:03:14.588297970 (506h34m14.459381005s) [upload=444504]: Done uploading segments... Dec 26 18:03:14.588319040 (506h34m14.459402075s) [upload=444504]: Upload finished. err: <nil> Duration: 5.771269656s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 43.237520451s trace: https://jaeger.dev.storj.io/trace/665ea65f6e9ccc33 FAIL (2025-12-26 18:05:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== e=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789316979 (506h12m54.660400024s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789326507 (506h12m54.660409552s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789611339 (506h12m54.660694384s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789619275 (506h12m54.66070231s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789914006 (506h12m54.660997041s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789949443 (506h12m54.661032478s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790181566 (506h12m54.661264601s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790295632 (506h12m54.661378668s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790510872 (506h12m54.661593917s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790692980 (506h12m54.661776015s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790787379 (506h12m54.661870414s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790992970 (506h12m54.662076005s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791120053 (506h12m54.662203098s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791275218 (506h12m54.662358253s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791443709 (506h12m54.662526744s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:56.112313145 (506h12m55.98339619s) [upload=444191]: Next returned nil segment Dec 26 17:41:56.112330638 (506h12m55.983413683s) [upload=444191]: Waiting for error group managing segments... Dec 26 17:41:56.485322720 (506h12m56.356405765s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.485637079 (506h12m56.356720124s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.666807539 (506h12m56.537890584s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.678753987 (506h12m56.549837032s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.687788039 (506h12m56.558871074s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.694318023 (506h12m56.565401058s) [noise=true , address=145.79.227.7:4326 , piece_id=DMQ5RPI46X2U7TJTS42ZHQEF32BDJRKJJO5HWOYAPNS6VWVG5B5Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.716581187 (506h12m56.587664232s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.972183007 (506h12m56.843266042s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.011917999 (506h12m56.883001034s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.054168622 (506h12m56.925251667s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.114312777 (506h12m56.985395812s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.499846250 (506h12m57.370929285s) [noise=true , address=145.79.227.34:4919 , piece_id=YRWHW3BJVEGVJRSA6RJB3T7VJQSDTGKEZQKRICFIV2G3T7AN4MIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.612649066 (506h12m57.483732101s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649413733 (506h12m57.520496768s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649440282 (506h12m57.520523328s) [seg_pos={0 0}, upload=444191]: Segment reached optimal threshold of 14 pieces. Dec 26 17:41:57.649674729 (506h12m57.520757774s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G7ty71tPwe7cZSBNK9u74DFUCGomnxDsNvAif4C3SbP7HWwfo): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649684087 (506h12m57.520767142s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649747708 (506h12m57.520830763s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v6zXZuMmyL2MU8qb3q6gg1iwTYvndjzMjVpv8btEpeX9Lnqmu): protocol: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection Dec 26 17:41:57.649785219 (506h12m57.520868274s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649899828 (506h12m57.520982903s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649908775 (506h12m57.52099182s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650074641 (506h12m57.521157696s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.650083287 (506h12m57.521166342s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650132821 (506h12m57.521215866s) [seg_pos={0 0}, upload=444191]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 17:41:57.650220138 (506h12m57.521303183s) [upload=444191]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:41:57.650227171 (506h12m57.521310206s) [upload=444191]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:41:57.958890870 (506h12m57.829973915s) [upload=444191]: Done uploading segments... Dec 26 17:41:57.958911620 (506h12m57.829994665s) [upload=444191]: Upload finished. err: <nil> Duration: 43.237894625s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [...XX...XX....XX..XX......X.XXX.....XX.X.XXX.........X..X...] duration: 5.829312839s trace: https://jaeger.dev.storj.io/trace/243f66ab99efb3ee FAIL (2025-12-26 18:05:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: UVYN4DXLISSVHNJFRXG6IZYN2LW7V6ZKFQRHNTB24T2UEAG2KW4Q) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: 6NSZ2L7LU5TU3OXOM6G7FVDUBDONHLESTCHJCILFY7WS7DU7YEKA) context deadline exceeded ======LOGS====== Dec 26 18:03:10.947413330 (506h34m10.818496385s) [upload=444504]: Starting upload Dec 26 18:03:10.947509102 (506h34m10.818592137s) [upload=444504]: Done starting upload Dec 26 18:03:10.947593964 (506h34m10.818676999s) [upload=444504]: Uploading segments... Dec 26 18:03:10.947770049 (506h34m10.818853084s) [upload=444504]: Got next segment. Inline: false Dec 26 18:03:10.947835494 (506h34m10.818918539s) [upload=444504]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 18:03:10.947841495 (506h34m10.81892453s) [upload=444504]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 18:03:11.196337715 (506h34m11.06742076s) [seg_pos={0 0}, upload=444504]: Begin upload segment... Dec 26 18:03:11.196399363 (506h34m11.067482398s) [seg_pos={0 0}, upload=444504]: Done begin upload segment. Dec 26 18:03:11.196462302 (506h34m11.067545347s) [noise=true , address=145.79.227.39:4015 , piece_id=3EIJJ2FW3KFAE7OHXOH3M2BARFKUPV6AER472K5F7KHJFLXTG63Q, seg_pos={0 0}, upload=444504]: Uploading piece... Dec 26 18:03:11.196638999 (506h34m11.067722044s) [noise=true , address=145.79.227.4:4186 , piece_id=HHX2ZHYKTMM55GULLNOMQJ2WPZRP444XFAO65HU2RVAYWUCDZZ7A, seg_pos={0 0}, upload=444504]: Uploading piece... Dec 26 18:03:11.197357437 (506h34m11.068440482s) [noise=true , address=45.91.116.9:4264 , piece_id=V6G45QGAWGT3MWIYQGPGHTF5XBCKD4G4G3FBXXFYGOKQBHE33LRQ, seg_pos={0 0}, upload=444504]: Uploading piece... Dec 26 18:03:11.197503895 (506h34m11.06858694s) [noise=true , address=77.90.31.11:4553 , piece_id=W3JRDU7L66LTYHTGHMK3PE554MUYSOIQMSSRBL34K4IPOSAE474A, seg_pos={0 0}, upload=444504]: Uploading piece... Dec 26 18:03:11.198008657 (506h34m11.069091702s) [noise=true , address=145.79.227.49:4339 , piece_id=2M7VCO7OY3MU3WWZNM7FCYQHJMTQO35BGT57SSYDMHJCPX2LZT2A, seg_pos={0 0}, upload=444504]: Uploading piece... Dec 26 18:03:11.198082277 (506h34m11.069165322s) [noise=true , address=145.79.227.41:4086 , piece_id=RAUVGKJR7TNL5YJHPNAUIPI72CX5GCHNBT4HOR3IUB4AKKCLMO5Q, seg_pos={0 0}, upload=444504]: Uploading piece... Dec 26 18:03:11.198472169 (506h34m11.069555214s) [noise=true , address=77.90.31.13:4627 , piece_id=FYB4X6MKKBN3L4RW3UKMR3YAX4WOQN63LQCN2MWHZJG3TVESVRQQ, seg_pos={0 0}, upload=444504]: Uploading piece... Dec 26 18:03:11.198538466 (506h34m11.069621501s) [noise=true , address=145.79.227.14:4839 , piece_id=EF5AB6ZDBM5O5DKMUY3JAG2QC7YT2DW5ODTK2Z7IUSKTBCCTXFSA, seg_pos={0 0}, upload=444504]: Uploading piece... Dec 26 18:03:11.199136184 (506h34m11.070219229s) [noise=true , address=145.79.227.47:4267 , piece_id=KWHHCDIZ6KRBJK335BTJ2XN7DTZVIRF3LSNLRTLZSFKKRRHKPHUA, seg_pos={0 0}, upload=444504]: Uploading piece... Dec 26 18:03:11.199257004 (506h34m11.070340049s) [noise=true , address=145.79.227.15:4874 , piece_id=6JDOK6U5OIFFQTEDA5IVGHFFZVHUXMRF4GTAM5ETPZNYGDIV7LPQ, seg_pos={0 0}, upload=444504]: Uploading piece... Dec 26 18:03:11.199660883 (506h34m11.070743928s) [noise=true , address=45.91.116.13:4156 , piece_id=UVYN4DXLISSVHNJFRXG6IZYN2LW7V6ZKFQRHNTB24T2UEAG2KW4Q, seg_pos={0 0}, upload=444504]: Uploading piece... Dec 26 18:03:11.199676052 (506h34m11.070759097s) [noise=true , address=45.91.116.10:4081 , piece_id=QC77DKAG4LZRC3UFN5UBTFBY7EC7P6Q5RUVITKZNUDVXL4LDTI6A, seg_pos={0 0}, upload=444504]: Uploading piece... Dec 26 18:03:11.200151407 (506h34m11.071234452s) [noise=true , address=145.79.227.45:4229 , piece_id=4QVKJHC6IXIKWJIW6XHTLWJGIHUW5HZ3ORB4B65527NCRNQURUNA, seg_pos={0 0}, upload=444504]: Uploading piece... Dec 26 18:03:11.200184359 (506h34m11.071267404s) [noise=true , address=45.91.116.14:4191 , piece_id=LG4MVMFR5622ZJOOMYG3IPLMMOPSWSSMHWOFCU76WXIAJIW4XUBQ, seg_pos={0 0}, upload=444504]: Uploading piece... Dec 26 18:03:11.200606214 (506h34m11.071689259s) [noise=true , address=45.91.116.8:4045 , piece_id=6NSZ2L7LU5TU3OXOM6G7FVDUBDONHLESTCHJCILFY7WS7DU7YEKA, seg_pos={0 0}, upload=444504]: Uploading piece... Dec 26 18:03:11.201053194 (506h34m11.072136239s) [noise=true , address=145.79.227.46:4601 , piece_id=JFFS2GUNTINTQBO7EEB324PK37ZXJ2ARH6XJ4L53GQV25WYQRAUA, seg_pos={0 0}, upload=444504]: Uploading piece... Dec 26 18:03:12.350290563 (506h34m12.221373608s) [noise=true , address=77.90.31.11:4553 , piece_id=W3JRDU7L66LTYHTGHMK3PE554MUYSOIQMSSRBL34K4IPOSAE474A, seg_pos={0 0}, upload=444504]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: storage node overloaded, request limit: 2000 Dec 26 18:03:12.915003815 (506h34m12.78608685s) [upload=444504]: Next returned nil segment Dec 26 18:03:12.915016951 (506h34m12.786099996s) [upload=444504]: Waiting for error group managing segments... Dec 26 18:03:13.194034659 (506h34m13.065117694s) [noise=true , address=145.79.227.15:4874 , piece_id=6JDOK6U5OIFFQTEDA5IVGHFFZVHUXMRF4GTAM5ETPZNYGDIV7LPQ, seg_pos={0 0}, upload=444504]: Done uploading piece. err: <nil> Dec 26 18:03:13.199551184 (506h34m13.070634219s) [noise=true , address=145.79.227.46:4601 , piece_id=JFFS2GUNTINTQBO7EEB324PK37ZXJ2ARH6XJ4L53GQV25WYQRAUA, seg_pos={0 0}, upload=444504]: Done uploading piece. err: <nil> Dec 26 18:03:13.204584307 (506h34m13.075667352s) [noise=true , address=145.79.227.49:4339 , piece_id=2M7VCO7OY3MU3WWZNM7FCYQHJMTQO35BGT57SSYDMHJCPX2LZT2A, seg_pos={0 0}, upload=444504]: Done uploading piece. err: <nil> Dec 26 18:03:13.206994768 (506h34m13.078077813s) [noise=true , address=145.79.227.39:4015 , piece_id=3EIJJ2FW3KFAE7OHXOH3M2BARFKUPV6AER472K5F7KHJFLXTG63Q, seg_pos={0 0}, upload=444504]: Done uploading piece. err: <nil> Dec 26 18:03:13.219091742 (506h34m13.090174777s) [noise=true , address=145.79.227.47:4267 , piece_id=KWHHCDIZ6KRBJK335BTJ2XN7DTZVIRF3LSNLRTLZSFKKRRHKPHUA, seg_pos={0 0}, upload=444504]: Done uploading piece. err: <nil> Dec 26 18:03:13.220706828 (506h34m13.091789863s) [noise=true , address=145.79.227.45:4229 , piece_id=4QVKJHC6IXIKWJIW6XHTLWJGIHUW5HZ3ORB4B65527NCRNQURUNA, seg_pos={0 0}, upload=444504]: Done uploading piece. err: <nil> Dec 26 18:03:13.230304272 (506h34m13.101387307s) [noise=true , address=145.79.227.14:4839 , piece_id=EF5AB6ZDBM5O5DKMUY3JAG2QC7YT2DW5ODTK2Z7IUSKTBCCTXFSA, seg_pos={0 0}, upload=444504]: Done uploading piece. err: <nil> Dec 26 18:03:13.232301426 (506h34m13.103384461s) [noise=true , address=145.79.227.4:4186 , piece_id=HHX2ZHYKTMM55GULLNOMQJ2WPZRP444XFAO65HU2RVAYWUCDZZ7A, seg_pos={0 0}, upload=444504]: Done uploading piece. err: <nil> Dec 26 18:03:13.439749002 (506h34m13.310832047s) [noise=true , address=45.91.116.9:4264 , piece_id=V6G45QGAWGT3MWIYQGPGHTF5XBCKD4G4G3FBXXFYGOKQBHE33LRQ, seg_pos={0 0}, upload=444504]: Done uploading piece. err: <nil> Dec 26 18:03:13.491708832 (506h34m13.362791878s) [noise=true , address=45.91.116.13:4156 , piece_id=UVYN4DXLISSVHNJFRXG6IZYN2LW7V6ZKFQRHNTB24T2UEAG2KW4Q, seg_pos={0 0}, upload=444504]: Done uploading piece. err: <nil> Dec 26 18:03:13.514759485 (506h34m13.38584253s) [noise=true , address=77.90.31.13:4627 , piece_id=FYB4X6MKKBN3L4RW3UKMR3YAX4WOQN63LQCN2MWHZJG3TVESVRQQ, seg_pos={0 0}, upload=444504]: Done uploading piece. err: <nil> Dec 26 18:03:13.585088304 (506h34m13.456171349s) [noise=true , address=45.91.116.14:4191 , piece_id=LG4MVMFR5622ZJOOMYG3IPLMMOPSWSSMHWOFCU76WXIAJIW4XUBQ, seg_pos={0 0}, upload=444504]: Done uploading piece. err: <nil> Dec 26 18:03:14.293444978 (506h34m14.164528033s) [noise=true , address=45.91.116.8:4045 , piece_id=6NSZ2L7LU5TU3OXOM6G7FVDUBDONHLESTCHJCILFY7WS7DU7YEKA, seg_pos={0 0}, upload=444504]: Done uploading piece. err: <nil> Dec 26 18:03:14.293466870 (506h34m14.164549925s) [seg_pos={0 0}, upload=444504]: Segment reached optimal threshold of 13 pieces. Dec 26 18:03:14.293642154 (506h34m14.164725219s) [noise=true , address=145.79.227.41:4086 , piece_id=RAUVGKJR7TNL5YJHPNAUIPI72CX5GCHNBT4HOR3IUB4AKKCLMO5Q, seg_pos={0 0}, upload=444504]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm): protocol: expected piece hash; context canceled; EOF Dec 26 18:03:14.293652554 (506h34m14.164735589s) [noise=true , address=145.79.227.41:4086 , piece_id=RAUVGKJR7TNL5YJHPNAUIPI72CX5GCHNBT4HOR3IUB4AKKCLMO5Q, seg_pos={0 0}, upload=444504]: Long tail cancellation due to optimal threshold Dec 26 18:03:14.294191048 (506h34m14.165274094s) [noise=true , address=45.91.116.10:4081 , piece_id=QC77DKAG4LZRC3UFN5UBTFBY7EC7P6Q5RUVITKZNUDVXL4LDTI6A, seg_pos={0 0}, upload=444504]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: context canceled; context canceled Dec 26 18:03:14.294210877 (506h34m14.165293922s) [noise=true , address=45.91.116.10:4081 , piece_id=QC77DKAG4LZRC3UFN5UBTFBY7EC7P6Q5RUVITKZNUDVXL4LDTI6A, seg_pos={0 0}, upload=444504]: Long tail cancellation due to optimal threshold Dec 26 18:03:14.294238509 (506h34m14.165321555s) [seg_pos={0 0}, upload=444504]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 18:03:14.294369770 (506h34m14.165452815s) [upload=444504]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 18:03:14.294377755 (506h34m14.1654608s) [upload=444504]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 18:03:14.588297970 (506h34m14.459381005s) [upload=444504]: Done uploading segments... Dec 26 18:03:14.588319040 (506h34m14.459402075s) [upload=444504]: Upload finished. err: <nil> Duration: 5.829345591s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 29.661488151s trace: https://jaeger.dev.storj.io/trace/665ea65f6e9ccc33 FAIL (2025-12-26 18:04:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== e=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789316979 (506h12m54.660400024s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789326507 (506h12m54.660409552s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789611339 (506h12m54.660694384s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789619275 (506h12m54.66070231s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789914006 (506h12m54.660997041s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789949443 (506h12m54.661032478s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790181566 (506h12m54.661264601s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790295632 (506h12m54.661378668s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790510872 (506h12m54.661593917s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790692980 (506h12m54.661776015s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790787379 (506h12m54.661870414s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790992970 (506h12m54.662076005s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791120053 (506h12m54.662203098s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791275218 (506h12m54.662358253s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791443709 (506h12m54.662526744s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:56.112313145 (506h12m55.98339619s) [upload=444191]: Next returned nil segment Dec 26 17:41:56.112330638 (506h12m55.983413683s) [upload=444191]: Waiting for error group managing segments... Dec 26 17:41:56.485322720 (506h12m56.356405765s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.485637079 (506h12m56.356720124s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.666807539 (506h12m56.537890584s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.678753987 (506h12m56.549837032s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.687788039 (506h12m56.558871074s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.694318023 (506h12m56.565401058s) [noise=true , address=145.79.227.7:4326 , piece_id=DMQ5RPI46X2U7TJTS42ZHQEF32BDJRKJJO5HWOYAPNS6VWVG5B5Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.716581187 (506h12m56.587664232s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.972183007 (506h12m56.843266042s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.011917999 (506h12m56.883001034s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.054168622 (506h12m56.925251667s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.114312777 (506h12m56.985395812s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.499846250 (506h12m57.370929285s) [noise=true , address=145.79.227.34:4919 , piece_id=YRWHW3BJVEGVJRSA6RJB3T7VJQSDTGKEZQKRICFIV2G3T7AN4MIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.612649066 (506h12m57.483732101s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649413733 (506h12m57.520496768s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649440282 (506h12m57.520523328s) [seg_pos={0 0}, upload=444191]: Segment reached optimal threshold of 14 pieces. Dec 26 17:41:57.649674729 (506h12m57.520757774s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G7ty71tPwe7cZSBNK9u74DFUCGomnxDsNvAif4C3SbP7HWwfo): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649684087 (506h12m57.520767142s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649747708 (506h12m57.520830763s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v6zXZuMmyL2MU8qb3q6gg1iwTYvndjzMjVpv8btEpeX9Lnqmu): protocol: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection Dec 26 17:41:57.649785219 (506h12m57.520868274s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649899828 (506h12m57.520982903s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649908775 (506h12m57.52099182s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650074641 (506h12m57.521157696s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.650083287 (506h12m57.521166342s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650132821 (506h12m57.521215866s) [seg_pos={0 0}, upload=444191]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 17:41:57.650220138 (506h12m57.521303183s) [upload=444191]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:41:57.650227171 (506h12m57.521310206s) [upload=444191]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:41:57.958890870 (506h12m57.829973915s) [upload=444191]: Done uploading segments... Dec 26 17:41:57.958911620 (506h12m57.829994665s) [upload=444191]: Upload finished. err: <nil> Duration: 29.661505243s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [X...XX...XX....XX..XX......X.XXX.....XX.X.XXX.........X..X..] duration: 6.418615584s trace: https://jaeger.dev.storj.io/trace/243f66ab99efb3ee FAIL (2025-12-26 18:04:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: UVYN4DXLISSVHNJFRXG6IZYN2LW7V6ZKFQRHNTB24T2UEAG2KW4Q) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: 6NSZ2L7LU5TU3OXOM6G7FVDUBDONHLESTCHJCILFY7WS7DU7YEKA) context deadline exceeded ======LOGS====== Dec 26 18:03:10.947413330 (506h34m10.818496385s) [upload=444504]: Starting upload Dec 26 18:03:10.947509102 (506h34m10.818592137s) [upload=444504]: Done starting upload Dec 26 18:03:10.947593964 (506h34m10.818676999s) [upload=444504]: Uploading segments... Dec 26 18:03:10.947770049 (506h34m10.818853084s) [upload=444504]: Got next segment. Inline: false Dec 26 18:03:10.947835494 (506h34m10.818918539s) [upload=444504]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 18:03:10.947841495 (506h34m10.81892453s) [upload=444504]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 18:03:11.196337715 (506h34m11.06742076s) [seg_pos={0 0}, upload=444504]: Begin upload segment... Dec 26 18:03:11.196399363 (506h34m11.067482398s) [seg_pos={0 0}, upload=444504]: Done begin upload segment. Dec 26 18:03:11.196462302 (506h34m11.067545347s) [noise=true , address=145.79.227.39:4015 , piece_id=3EIJJ2FW3KFAE7OHXOH3M2BARFKUPV6AER472K5F7KHJFLXTG63Q, seg_pos={0 0}, upload=444504]: Uploading piece... Dec 26 18:03:11.196638999 (506h34m11.067722044s) [noise=true , address=145.79.227.4:4186 , piece_id=HHX2ZHYKTMM55GULLNOMQJ2WPZRP444XFAO65HU2RVAYWUCDZZ7A, seg_pos={0 0}, upload=444504]: Uploading piece... Dec 26 18:03:11.197357437 (506h34m11.068440482s) [noise=true , address=45.91.116.9:4264 , piece_id=V6G45QGAWGT3MWIYQGPGHTF5XBCKD4G4G3FBXXFYGOKQBHE33LRQ, seg_pos={0 0}, upload=444504]: Uploading piece... Dec 26 18:03:11.197503895 (506h34m11.06858694s) [noise=true , address=77.90.31.11:4553 , piece_id=W3JRDU7L66LTYHTGHMK3PE554MUYSOIQMSSRBL34K4IPOSAE474A, seg_pos={0 0}, upload=444504]: Uploading piece... Dec 26 18:03:11.198008657 (506h34m11.069091702s) [noise=true , address=145.79.227.49:4339 , piece_id=2M7VCO7OY3MU3WWZNM7FCYQHJMTQO35BGT57SSYDMHJCPX2LZT2A, seg_pos={0 0}, upload=444504]: Uploading piece... Dec 26 18:03:11.198082277 (506h34m11.069165322s) [noise=true , address=145.79.227.41:4086 , piece_id=RAUVGKJR7TNL5YJHPNAUIPI72CX5GCHNBT4HOR3IUB4AKKCLMO5Q, seg_pos={0 0}, upload=444504]: Uploading piece... Dec 26 18:03:11.198472169 (506h34m11.069555214s) [noise=true , address=77.90.31.13:4627 , piece_id=FYB4X6MKKBN3L4RW3UKMR3YAX4WOQN63LQCN2MWHZJG3TVESVRQQ, seg_pos={0 0}, upload=444504]: Uploading piece... Dec 26 18:03:11.198538466 (506h34m11.069621501s) [noise=true , address=145.79.227.14:4839 , piece_id=EF5AB6ZDBM5O5DKMUY3JAG2QC7YT2DW5ODTK2Z7IUSKTBCCTXFSA, seg_pos={0 0}, upload=444504]: Uploading piece... Dec 26 18:03:11.199136184 (506h34m11.070219229s) [noise=true , address=145.79.227.47:4267 , piece_id=KWHHCDIZ6KRBJK335BTJ2XN7DTZVIRF3LSNLRTLZSFKKRRHKPHUA, seg_pos={0 0}, upload=444504]: Uploading piece... Dec 26 18:03:11.199257004 (506h34m11.070340049s) [noise=true , address=145.79.227.15:4874 , piece_id=6JDOK6U5OIFFQTEDA5IVGHFFZVHUXMRF4GTAM5ETPZNYGDIV7LPQ, seg_pos={0 0}, upload=444504]: Uploading piece... Dec 26 18:03:11.199660883 (506h34m11.070743928s) [noise=true , address=45.91.116.13:4156 , piece_id=UVYN4DXLISSVHNJFRXG6IZYN2LW7V6ZKFQRHNTB24T2UEAG2KW4Q, seg_pos={0 0}, upload=444504]: Uploading piece... Dec 26 18:03:11.199676052 (506h34m11.070759097s) [noise=true , address=45.91.116.10:4081 , piece_id=QC77DKAG4LZRC3UFN5UBTFBY7EC7P6Q5RUVITKZNUDVXL4LDTI6A, seg_pos={0 0}, upload=444504]: Uploading piece... Dec 26 18:03:11.200151407 (506h34m11.071234452s) [noise=true , address=145.79.227.45:4229 , piece_id=4QVKJHC6IXIKWJIW6XHTLWJGIHUW5HZ3ORB4B65527NCRNQURUNA, seg_pos={0 0}, upload=444504]: Uploading piece... Dec 26 18:03:11.200184359 (506h34m11.071267404s) [noise=true , address=45.91.116.14:4191 , piece_id=LG4MVMFR5622ZJOOMYG3IPLMMOPSWSSMHWOFCU76WXIAJIW4XUBQ, seg_pos={0 0}, upload=444504]: Uploading piece... Dec 26 18:03:11.200606214 (506h34m11.071689259s) [noise=true , address=45.91.116.8:4045 , piece_id=6NSZ2L7LU5TU3OXOM6G7FVDUBDONHLESTCHJCILFY7WS7DU7YEKA, seg_pos={0 0}, upload=444504]: Uploading piece... Dec 26 18:03:11.201053194 (506h34m11.072136239s) [noise=true , address=145.79.227.46:4601 , piece_id=JFFS2GUNTINTQBO7EEB324PK37ZXJ2ARH6XJ4L53GQV25WYQRAUA, seg_pos={0 0}, upload=444504]: Uploading piece... Dec 26 18:03:12.350290563 (506h34m12.221373608s) [noise=true , address=77.90.31.11:4553 , piece_id=W3JRDU7L66LTYHTGHMK3PE554MUYSOIQMSSRBL34K4IPOSAE474A, seg_pos={0 0}, upload=444504]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: storage node overloaded, request limit: 2000 Dec 26 18:03:12.915003815 (506h34m12.78608685s) [upload=444504]: Next returned nil segment Dec 26 18:03:12.915016951 (506h34m12.786099996s) [upload=444504]: Waiting for error group managing segments... Dec 26 18:03:13.194034659 (506h34m13.065117694s) [noise=true , address=145.79.227.15:4874 , piece_id=6JDOK6U5OIFFQTEDA5IVGHFFZVHUXMRF4GTAM5ETPZNYGDIV7LPQ, seg_pos={0 0}, upload=444504]: Done uploading piece. err: <nil> Dec 26 18:03:13.199551184 (506h34m13.070634219s) [noise=true , address=145.79.227.46:4601 , piece_id=JFFS2GUNTINTQBO7EEB324PK37ZXJ2ARH6XJ4L53GQV25WYQRAUA, seg_pos={0 0}, upload=444504]: Done uploading piece. err: <nil> Dec 26 18:03:13.204584307 (506h34m13.075667352s) [noise=true , address=145.79.227.49:4339 , piece_id=2M7VCO7OY3MU3WWZNM7FCYQHJMTQO35BGT57SSYDMHJCPX2LZT2A, seg_pos={0 0}, upload=444504]: Done uploading piece. err: <nil> Dec 26 18:03:13.206994768 (506h34m13.078077813s) [noise=true , address=145.79.227.39:4015 , piece_id=3EIJJ2FW3KFAE7OHXOH3M2BARFKUPV6AER472K5F7KHJFLXTG63Q, seg_pos={0 0}, upload=444504]: Done uploading piece. err: <nil> Dec 26 18:03:13.219091742 (506h34m13.090174777s) [noise=true , address=145.79.227.47:4267 , piece_id=KWHHCDIZ6KRBJK335BTJ2XN7DTZVIRF3LSNLRTLZSFKKRRHKPHUA, seg_pos={0 0}, upload=444504]: Done uploading piece. err: <nil> Dec 26 18:03:13.220706828 (506h34m13.091789863s) [noise=true , address=145.79.227.45:4229 , piece_id=4QVKJHC6IXIKWJIW6XHTLWJGIHUW5HZ3ORB4B65527NCRNQURUNA, seg_pos={0 0}, upload=444504]: Done uploading piece. err: <nil> Dec 26 18:03:13.230304272 (506h34m13.101387307s) [noise=true , address=145.79.227.14:4839 , piece_id=EF5AB6ZDBM5O5DKMUY3JAG2QC7YT2DW5ODTK2Z7IUSKTBCCTXFSA, seg_pos={0 0}, upload=444504]: Done uploading piece. err: <nil> Dec 26 18:03:13.232301426 (506h34m13.103384461s) [noise=true , address=145.79.227.4:4186 , piece_id=HHX2ZHYKTMM55GULLNOMQJ2WPZRP444XFAO65HU2RVAYWUCDZZ7A, seg_pos={0 0}, upload=444504]: Done uploading piece. err: <nil> Dec 26 18:03:13.439749002 (506h34m13.310832047s) [noise=true , address=45.91.116.9:4264 , piece_id=V6G45QGAWGT3MWIYQGPGHTF5XBCKD4G4G3FBXXFYGOKQBHE33LRQ, seg_pos={0 0}, upload=444504]: Done uploading piece. err: <nil> Dec 26 18:03:13.491708832 (506h34m13.362791878s) [noise=true , address=45.91.116.13:4156 , piece_id=UVYN4DXLISSVHNJFRXG6IZYN2LW7V6ZKFQRHNTB24T2UEAG2KW4Q, seg_pos={0 0}, upload=444504]: Done uploading piece. err: <nil> Dec 26 18:03:13.514759485 (506h34m13.38584253s) [noise=true , address=77.90.31.13:4627 , piece_id=FYB4X6MKKBN3L4RW3UKMR3YAX4WOQN63LQCN2MWHZJG3TVESVRQQ, seg_pos={0 0}, upload=444504]: Done uploading piece. err: <nil> Dec 26 18:03:13.585088304 (506h34m13.456171349s) [noise=true , address=45.91.116.14:4191 , piece_id=LG4MVMFR5622ZJOOMYG3IPLMMOPSWSSMHWOFCU76WXIAJIW4XUBQ, seg_pos={0 0}, upload=444504]: Done uploading piece. err: <nil> Dec 26 18:03:14.293444978 (506h34m14.164528033s) [noise=true , address=45.91.116.8:4045 , piece_id=6NSZ2L7LU5TU3OXOM6G7FVDUBDONHLESTCHJCILFY7WS7DU7YEKA, seg_pos={0 0}, upload=444504]: Done uploading piece. err: <nil> Dec 26 18:03:14.293466870 (506h34m14.164549925s) [seg_pos={0 0}, upload=444504]: Segment reached optimal threshold of 13 pieces. Dec 26 18:03:14.293642154 (506h34m14.164725219s) [noise=true , address=145.79.227.41:4086 , piece_id=RAUVGKJR7TNL5YJHPNAUIPI72CX5GCHNBT4HOR3IUB4AKKCLMO5Q, seg_pos={0 0}, upload=444504]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm): protocol: expected piece hash; context canceled; EOF Dec 26 18:03:14.293652554 (506h34m14.164735589s) [noise=true , address=145.79.227.41:4086 , piece_id=RAUVGKJR7TNL5YJHPNAUIPI72CX5GCHNBT4HOR3IUB4AKKCLMO5Q, seg_pos={0 0}, upload=444504]: Long tail cancellation due to optimal threshold Dec 26 18:03:14.294191048 (506h34m14.165274094s) [noise=true , address=45.91.116.10:4081 , piece_id=QC77DKAG4LZRC3UFN5UBTFBY7EC7P6Q5RUVITKZNUDVXL4LDTI6A, seg_pos={0 0}, upload=444504]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: context canceled; context canceled Dec 26 18:03:14.294210877 (506h34m14.165293922s) [noise=true , address=45.91.116.10:4081 , piece_id=QC77DKAG4LZRC3UFN5UBTFBY7EC7P6Q5RUVITKZNUDVXL4LDTI6A, seg_pos={0 0}, upload=444504]: Long tail cancellation due to optimal threshold Dec 26 18:03:14.294238509 (506h34m14.165321555s) [seg_pos={0 0}, upload=444504]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 18:03:14.294369770 (506h34m14.165452815s) [upload=444504]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 18:03:14.294377755 (506h34m14.1654608s) [upload=444504]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 18:03:14.588297970 (506h34m14.459381005s) [upload=444504]: Done uploading segments... Dec 26 18:03:14.588319040 (506h34m14.459402075s) [upload=444504]: Upload finished. err: <nil> Duration: 6.418643558s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [XX...XX...XX....XX..XX......X.XXX.....XX.X.XXX.........X..X.] duration: 6.151544976s trace: https://jaeger.dev.storj.io/trace/243f66ab99efb3ee FAIL (2025-12-26 18:04:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: UVYN4DXLISSVHNJFRXG6IZYN2LW7V6ZKFQRHNTB24T2UEAG2KW4Q) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: 6NSZ2L7LU5TU3OXOM6G7FVDUBDONHLESTCHJCILFY7WS7DU7YEKA) context deadline exceeded ======LOGS====== Dec 26 18:03:10.947413330 (506h34m10.818496385s) [upload=444504]: Starting upload Dec 26 18:03:10.947509102 (506h34m10.818592137s) [upload=444504]: Done starting upload Dec 26 18:03:10.947593964 (506h34m10.818676999s) [upload=444504]: Uploading segments... Dec 26 18:03:10.947770049 (506h34m10.818853084s) [upload=444504]: Got next segment. Inline: false Dec 26 18:03:10.947835494 (506h34m10.818918539s) [upload=444504]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 18:03:10.947841495 (506h34m10.81892453s) [upload=444504]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 18:03:11.196337715 (506h34m11.06742076s) [seg_pos={0 0}, upload=444504]: Begin upload segment... Dec 26 18:03:11.196399363 (506h34m11.067482398s) [seg_pos={0 0}, upload=444504]: Done begin upload segment. Dec 26 18:03:11.196462302 (506h34m11.067545347s) [noise=true , address=145.79.227.39:4015 , piece_id=3EIJJ2FW3KFAE7OHXOH3M2BARFKUPV6AER472K5F7KHJFLXTG63Q, seg_pos={0 0}, upload=444504]: Uploading piece... Dec 26 18:03:11.196638999 (506h34m11.067722044s) [noise=true , address=145.79.227.4:4186 , piece_id=HHX2ZHYKTMM55GULLNOMQJ2WPZRP444XFAO65HU2RVAYWUCDZZ7A, seg_pos={0 0}, upload=444504]: Uploading piece... Dec 26 18:03:11.197357437 (506h34m11.068440482s) [noise=true , address=45.91.116.9:4264 , piece_id=V6G45QGAWGT3MWIYQGPGHTF5XBCKD4G4G3FBXXFYGOKQBHE33LRQ, seg_pos={0 0}, upload=444504]: Uploading piece... Dec 26 18:03:11.197503895 (506h34m11.06858694s) [noise=true , address=77.90.31.11:4553 , piece_id=W3JRDU7L66LTYHTGHMK3PE554MUYSOIQMSSRBL34K4IPOSAE474A, seg_pos={0 0}, upload=444504]: Uploading piece... Dec 26 18:03:11.198008657 (506h34m11.069091702s) [noise=true , address=145.79.227.49:4339 , piece_id=2M7VCO7OY3MU3WWZNM7FCYQHJMTQO35BGT57SSYDMHJCPX2LZT2A, seg_pos={0 0}, upload=444504]: Uploading piece... Dec 26 18:03:11.198082277 (506h34m11.069165322s) [noise=true , address=145.79.227.41:4086 , piece_id=RAUVGKJR7TNL5YJHPNAUIPI72CX5GCHNBT4HOR3IUB4AKKCLMO5Q, seg_pos={0 0}, upload=444504]: Uploading piece... Dec 26 18:03:11.198472169 (506h34m11.069555214s) [noise=true , address=77.90.31.13:4627 , piece_id=FYB4X6MKKBN3L4RW3UKMR3YAX4WOQN63LQCN2MWHZJG3TVESVRQQ, seg_pos={0 0}, upload=444504]: Uploading piece... Dec 26 18:03:11.198538466 (506h34m11.069621501s) [noise=true , address=145.79.227.14:4839 , piece_id=EF5AB6ZDBM5O5DKMUY3JAG2QC7YT2DW5ODTK2Z7IUSKTBCCTXFSA, seg_pos={0 0}, upload=444504]: Uploading piece... Dec 26 18:03:11.199136184 (506h34m11.070219229s) [noise=true , address=145.79.227.47:4267 , piece_id=KWHHCDIZ6KRBJK335BTJ2XN7DTZVIRF3LSNLRTLZSFKKRRHKPHUA, seg_pos={0 0}, upload=444504]: Uploading piece... Dec 26 18:03:11.199257004 (506h34m11.070340049s) [noise=true , address=145.79.227.15:4874 , piece_id=6JDOK6U5OIFFQTEDA5IVGHFFZVHUXMRF4GTAM5ETPZNYGDIV7LPQ, seg_pos={0 0}, upload=444504]: Uploading piece... Dec 26 18:03:11.199660883 (506h34m11.070743928s) [noise=true , address=45.91.116.13:4156 , piece_id=UVYN4DXLISSVHNJFRXG6IZYN2LW7V6ZKFQRHNTB24T2UEAG2KW4Q, seg_pos={0 0}, upload=444504]: Uploading piece... Dec 26 18:03:11.199676052 (506h34m11.070759097s) [noise=true , address=45.91.116.10:4081 , piece_id=QC77DKAG4LZRC3UFN5UBTFBY7EC7P6Q5RUVITKZNUDVXL4LDTI6A, seg_pos={0 0}, upload=444504]: Uploading piece... Dec 26 18:03:11.200151407 (506h34m11.071234452s) [noise=true , address=145.79.227.45:4229 , piece_id=4QVKJHC6IXIKWJIW6XHTLWJGIHUW5HZ3ORB4B65527NCRNQURUNA, seg_pos={0 0}, upload=444504]: Uploading piece... Dec 26 18:03:11.200184359 (506h34m11.071267404s) [noise=true , address=45.91.116.14:4191 , piece_id=LG4MVMFR5622ZJOOMYG3IPLMMOPSWSSMHWOFCU76WXIAJIW4XUBQ, seg_pos={0 0}, upload=444504]: Uploading piece... Dec 26 18:03:11.200606214 (506h34m11.071689259s) [noise=true , address=45.91.116.8:4045 , piece_id=6NSZ2L7LU5TU3OXOM6G7FVDUBDONHLESTCHJCILFY7WS7DU7YEKA, seg_pos={0 0}, upload=444504]: Uploading piece... Dec 26 18:03:11.201053194 (506h34m11.072136239s) [noise=true , address=145.79.227.46:4601 , piece_id=JFFS2GUNTINTQBO7EEB324PK37ZXJ2ARH6XJ4L53GQV25WYQRAUA, seg_pos={0 0}, upload=444504]: Uploading piece... Dec 26 18:03:12.350290563 (506h34m12.221373608s) [noise=true , address=77.90.31.11:4553 , piece_id=W3JRDU7L66LTYHTGHMK3PE554MUYSOIQMSSRBL34K4IPOSAE474A, seg_pos={0 0}, upload=444504]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: storage node overloaded, request limit: 2000 Dec 26 18:03:12.915003815 (506h34m12.78608685s) [upload=444504]: Next returned nil segment Dec 26 18:03:12.915016951 (506h34m12.786099996s) [upload=444504]: Waiting for error group managing segments... Dec 26 18:03:13.194034659 (506h34m13.065117694s) [noise=true , address=145.79.227.15:4874 , piece_id=6JDOK6U5OIFFQTEDA5IVGHFFZVHUXMRF4GTAM5ETPZNYGDIV7LPQ, seg_pos={0 0}, upload=444504]: Done uploading piece. err: <nil> Dec 26 18:03:13.199551184 (506h34m13.070634219s) [noise=true , address=145.79.227.46:4601 , piece_id=JFFS2GUNTINTQBO7EEB324PK37ZXJ2ARH6XJ4L53GQV25WYQRAUA, seg_pos={0 0}, upload=444504]: Done uploading piece. err: <nil> Dec 26 18:03:13.204584307 (506h34m13.075667352s) [noise=true , address=145.79.227.49:4339 , piece_id=2M7VCO7OY3MU3WWZNM7FCYQHJMTQO35BGT57SSYDMHJCPX2LZT2A, seg_pos={0 0}, upload=444504]: Done uploading piece. err: <nil> Dec 26 18:03:13.206994768 (506h34m13.078077813s) [noise=true , address=145.79.227.39:4015 , piece_id=3EIJJ2FW3KFAE7OHXOH3M2BARFKUPV6AER472K5F7KHJFLXTG63Q, seg_pos={0 0}, upload=444504]: Done uploading piece. err: <nil> Dec 26 18:03:13.219091742 (506h34m13.090174777s) [noise=true , address=145.79.227.47:4267 , piece_id=KWHHCDIZ6KRBJK335BTJ2XN7DTZVIRF3LSNLRTLZSFKKRRHKPHUA, seg_pos={0 0}, upload=444504]: Done uploading piece. err: <nil> Dec 26 18:03:13.220706828 (506h34m13.091789863s) [noise=true , address=145.79.227.45:4229 , piece_id=4QVKJHC6IXIKWJIW6XHTLWJGIHUW5HZ3ORB4B65527NCRNQURUNA, seg_pos={0 0}, upload=444504]: Done uploading piece. err: <nil> Dec 26 18:03:13.230304272 (506h34m13.101387307s) [noise=true , address=145.79.227.14:4839 , piece_id=EF5AB6ZDBM5O5DKMUY3JAG2QC7YT2DW5ODTK2Z7IUSKTBCCTXFSA, seg_pos={0 0}, upload=444504]: Done uploading piece. err: <nil> Dec 26 18:03:13.232301426 (506h34m13.103384461s) [noise=true , address=145.79.227.4:4186 , piece_id=HHX2ZHYKTMM55GULLNOMQJ2WPZRP444XFAO65HU2RVAYWUCDZZ7A, seg_pos={0 0}, upload=444504]: Done uploading piece. err: <nil> Dec 26 18:03:13.439749002 (506h34m13.310832047s) [noise=true , address=45.91.116.9:4264 , piece_id=V6G45QGAWGT3MWIYQGPGHTF5XBCKD4G4G3FBXXFYGOKQBHE33LRQ, seg_pos={0 0}, upload=444504]: Done uploading piece. err: <nil> Dec 26 18:03:13.491708832 (506h34m13.362791878s) [noise=true , address=45.91.116.13:4156 , piece_id=UVYN4DXLISSVHNJFRXG6IZYN2LW7V6ZKFQRHNTB24T2UEAG2KW4Q, seg_pos={0 0}, upload=444504]: Done uploading piece. err: <nil> Dec 26 18:03:13.514759485 (506h34m13.38584253s) [noise=true , address=77.90.31.13:4627 , piece_id=FYB4X6MKKBN3L4RW3UKMR3YAX4WOQN63LQCN2MWHZJG3TVESVRQQ, seg_pos={0 0}, upload=444504]: Done uploading piece. err: <nil> Dec 26 18:03:13.585088304 (506h34m13.456171349s) [noise=true , address=45.91.116.14:4191 , piece_id=LG4MVMFR5622ZJOOMYG3IPLMMOPSWSSMHWOFCU76WXIAJIW4XUBQ, seg_pos={0 0}, upload=444504]: Done uploading piece. err: <nil> Dec 26 18:03:14.293444978 (506h34m14.164528033s) [noise=true , address=45.91.116.8:4045 , piece_id=6NSZ2L7LU5TU3OXOM6G7FVDUBDONHLESTCHJCILFY7WS7DU7YEKA, seg_pos={0 0}, upload=444504]: Done uploading piece. err: <nil> Dec 26 18:03:14.293466870 (506h34m14.164549925s) [seg_pos={0 0}, upload=444504]: Segment reached optimal threshold of 13 pieces. Dec 26 18:03:14.293642154 (506h34m14.164725219s) [noise=true , address=145.79.227.41:4086 , piece_id=RAUVGKJR7TNL5YJHPNAUIPI72CX5GCHNBT4HOR3IUB4AKKCLMO5Q, seg_pos={0 0}, upload=444504]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm): protocol: expected piece hash; context canceled; EOF Dec 26 18:03:14.293652554 (506h34m14.164735589s) [noise=true , address=145.79.227.41:4086 , piece_id=RAUVGKJR7TNL5YJHPNAUIPI72CX5GCHNBT4HOR3IUB4AKKCLMO5Q, seg_pos={0 0}, upload=444504]: Long tail cancellation due to optimal threshold Dec 26 18:03:14.294191048 (506h34m14.165274094s) [noise=true , address=45.91.116.10:4081 , piece_id=QC77DKAG4LZRC3UFN5UBTFBY7EC7P6Q5RUVITKZNUDVXL4LDTI6A, seg_pos={0 0}, upload=444504]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: context canceled; context canceled Dec 26 18:03:14.294210877 (506h34m14.165293922s) [noise=true , address=45.91.116.10:4081 , piece_id=QC77DKAG4LZRC3UFN5UBTFBY7EC7P6Q5RUVITKZNUDVXL4LDTI6A, seg_pos={0 0}, upload=444504]: Long tail cancellation due to optimal threshold Dec 26 18:03:14.294238509 (506h34m14.165321555s) [seg_pos={0 0}, upload=444504]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 18:03:14.294369770 (506h34m14.165452815s) [upload=444504]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 18:03:14.294377755 (506h34m14.1654608s) [upload=444504]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 18:03:14.588297970 (506h34m14.459381005s) [upload=444504]: Done uploading segments... Dec 26 18:03:14.588319040 (506h34m14.459402075s) [upload=444504]: Upload finished. err: <nil> Duration: 6.15157402s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [.XX...XX...XX....XX..XX......X.XXX.....XX.X.XXX.........X..X] duration: 30.003618969s trace: https://jaeger.dev.storj.io/trace/243f66ab99efb3ee FAIL (2025-12-26 18:03:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: UVYN4DXLISSVHNJFRXG6IZYN2LW7V6ZKFQRHNTB24T2UEAG2KW4Q) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: 6NSZ2L7LU5TU3OXOM6G7FVDUBDONHLESTCHJCILFY7WS7DU7YEKA) context deadline exceeded ======LOGS====== Dec 26 18:03:10.947413330 (506h34m10.818496385s) [upload=444504]: Starting upload Dec 26 18:03:10.947509102 (506h34m10.818592137s) [upload=444504]: Done starting upload Dec 26 18:03:10.947593964 (506h34m10.818676999s) [upload=444504]: Uploading segments... Dec 26 18:03:10.947770049 (506h34m10.818853084s) [upload=444504]: Got next segment. Inline: false Dec 26 18:03:10.947835494 (506h34m10.818918539s) [upload=444504]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 18:03:10.947841495 (506h34m10.81892453s) [upload=444504]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 18:03:11.196337715 (506h34m11.06742076s) [seg_pos={0 0}, upload=444504]: Begin upload segment... Dec 26 18:03:11.196399363 (506h34m11.067482398s) [seg_pos={0 0}, upload=444504]: Done begin upload segment. Dec 26 18:03:11.196462302 (506h34m11.067545347s) [noise=true , address=145.79.227.39:4015 , piece_id=3EIJJ2FW3KFAE7OHXOH3M2BARFKUPV6AER472K5F7KHJFLXTG63Q, seg_pos={0 0}, upload=444504]: Uploading piece... Dec 26 18:03:11.196638999 (506h34m11.067722044s) [noise=true , address=145.79.227.4:4186 , piece_id=HHX2ZHYKTMM55GULLNOMQJ2WPZRP444XFAO65HU2RVAYWUCDZZ7A, seg_pos={0 0}, upload=444504]: Uploading piece... Dec 26 18:03:11.197357437 (506h34m11.068440482s) [noise=true , address=45.91.116.9:4264 , piece_id=V6G45QGAWGT3MWIYQGPGHTF5XBCKD4G4G3FBXXFYGOKQBHE33LRQ, seg_pos={0 0}, upload=444504]: Uploading piece... Dec 26 18:03:11.197503895 (506h34m11.06858694s) [noise=true , address=77.90.31.11:4553 , piece_id=W3JRDU7L66LTYHTGHMK3PE554MUYSOIQMSSRBL34K4IPOSAE474A, seg_pos={0 0}, upload=444504]: Uploading piece... Dec 26 18:03:11.198008657 (506h34m11.069091702s) [noise=true , address=145.79.227.49:4339 , piece_id=2M7VCO7OY3MU3WWZNM7FCYQHJMTQO35BGT57SSYDMHJCPX2LZT2A, seg_pos={0 0}, upload=444504]: Uploading piece... Dec 26 18:03:11.198082277 (506h34m11.069165322s) [noise=true , address=145.79.227.41:4086 , piece_id=RAUVGKJR7TNL5YJHPNAUIPI72CX5GCHNBT4HOR3IUB4AKKCLMO5Q, seg_pos={0 0}, upload=444504]: Uploading piece... Dec 26 18:03:11.198472169 (506h34m11.069555214s) [noise=true , address=77.90.31.13:4627 , piece_id=FYB4X6MKKBN3L4RW3UKMR3YAX4WOQN63LQCN2MWHZJG3TVESVRQQ, seg_pos={0 0}, upload=444504]: Uploading piece... Dec 26 18:03:11.198538466 (506h34m11.069621501s) [noise=true , address=145.79.227.14:4839 , piece_id=EF5AB6ZDBM5O5DKMUY3JAG2QC7YT2DW5ODTK2Z7IUSKTBCCTXFSA, seg_pos={0 0}, upload=444504]: Uploading piece... Dec 26 18:03:11.199136184 (506h34m11.070219229s) [noise=true , address=145.79.227.47:4267 , piece_id=KWHHCDIZ6KRBJK335BTJ2XN7DTZVIRF3LSNLRTLZSFKKRRHKPHUA, seg_pos={0 0}, upload=444504]: Uploading piece... Dec 26 18:03:11.199257004 (506h34m11.070340049s) [noise=true , address=145.79.227.15:4874 , piece_id=6JDOK6U5OIFFQTEDA5IVGHFFZVHUXMRF4GTAM5ETPZNYGDIV7LPQ, seg_pos={0 0}, upload=444504]: Uploading piece... Dec 26 18:03:11.199660883 (506h34m11.070743928s) [noise=true , address=45.91.116.13:4156 , piece_id=UVYN4DXLISSVHNJFRXG6IZYN2LW7V6ZKFQRHNTB24T2UEAG2KW4Q, seg_pos={0 0}, upload=444504]: Uploading piece... Dec 26 18:03:11.199676052 (506h34m11.070759097s) [noise=true , address=45.91.116.10:4081 , piece_id=QC77DKAG4LZRC3UFN5UBTFBY7EC7P6Q5RUVITKZNUDVXL4LDTI6A, seg_pos={0 0}, upload=444504]: Uploading piece... Dec 26 18:03:11.200151407 (506h34m11.071234452s) [noise=true , address=145.79.227.45:4229 , piece_id=4QVKJHC6IXIKWJIW6XHTLWJGIHUW5HZ3ORB4B65527NCRNQURUNA, seg_pos={0 0}, upload=444504]: Uploading piece... Dec 26 18:03:11.200184359 (506h34m11.071267404s) [noise=true , address=45.91.116.14:4191 , piece_id=LG4MVMFR5622ZJOOMYG3IPLMMOPSWSSMHWOFCU76WXIAJIW4XUBQ, seg_pos={0 0}, upload=444504]: Uploading piece... Dec 26 18:03:11.200606214 (506h34m11.071689259s) [noise=true , address=45.91.116.8:4045 , piece_id=6NSZ2L7LU5TU3OXOM6G7FVDUBDONHLESTCHJCILFY7WS7DU7YEKA, seg_pos={0 0}, upload=444504]: Uploading piece... Dec 26 18:03:11.201053194 (506h34m11.072136239s) [noise=true , address=145.79.227.46:4601 , piece_id=JFFS2GUNTINTQBO7EEB324PK37ZXJ2ARH6XJ4L53GQV25WYQRAUA, seg_pos={0 0}, upload=444504]: Uploading piece... Dec 26 18:03:12.350290563 (506h34m12.221373608s) [noise=true , address=77.90.31.11:4553 , piece_id=W3JRDU7L66LTYHTGHMK3PE554MUYSOIQMSSRBL34K4IPOSAE474A, seg_pos={0 0}, upload=444504]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: storage node overloaded, request limit: 2000 Dec 26 18:03:12.915003815 (506h34m12.78608685s) [upload=444504]: Next returned nil segment Dec 26 18:03:12.915016951 (506h34m12.786099996s) [upload=444504]: Waiting for error group managing segments... Dec 26 18:03:13.194034659 (506h34m13.065117694s) [noise=true , address=145.79.227.15:4874 , piece_id=6JDOK6U5OIFFQTEDA5IVGHFFZVHUXMRF4GTAM5ETPZNYGDIV7LPQ, seg_pos={0 0}, upload=444504]: Done uploading piece. err: <nil> Dec 26 18:03:13.199551184 (506h34m13.070634219s) [noise=true , address=145.79.227.46:4601 , piece_id=JFFS2GUNTINTQBO7EEB324PK37ZXJ2ARH6XJ4L53GQV25WYQRAUA, seg_pos={0 0}, upload=444504]: Done uploading piece. err: <nil> Dec 26 18:03:13.204584307 (506h34m13.075667352s) [noise=true , address=145.79.227.49:4339 , piece_id=2M7VCO7OY3MU3WWZNM7FCYQHJMTQO35BGT57SSYDMHJCPX2LZT2A, seg_pos={0 0}, upload=444504]: Done uploading piece. err: <nil> Dec 26 18:03:13.206994768 (506h34m13.078077813s) [noise=true , address=145.79.227.39:4015 , piece_id=3EIJJ2FW3KFAE7OHXOH3M2BARFKUPV6AER472K5F7KHJFLXTG63Q, seg_pos={0 0}, upload=444504]: Done uploading piece. err: <nil> Dec 26 18:03:13.219091742 (506h34m13.090174777s) [noise=true , address=145.79.227.47:4267 , piece_id=KWHHCDIZ6KRBJK335BTJ2XN7DTZVIRF3LSNLRTLZSFKKRRHKPHUA, seg_pos={0 0}, upload=444504]: Done uploading piece. err: <nil> Dec 26 18:03:13.220706828 (506h34m13.091789863s) [noise=true , address=145.79.227.45:4229 , piece_id=4QVKJHC6IXIKWJIW6XHTLWJGIHUW5HZ3ORB4B65527NCRNQURUNA, seg_pos={0 0}, upload=444504]: Done uploading piece. err: <nil> Dec 26 18:03:13.230304272 (506h34m13.101387307s) [noise=true , address=145.79.227.14:4839 , piece_id=EF5AB6ZDBM5O5DKMUY3JAG2QC7YT2DW5ODTK2Z7IUSKTBCCTXFSA, seg_pos={0 0}, upload=444504]: Done uploading piece. err: <nil> Dec 26 18:03:13.232301426 (506h34m13.103384461s) [noise=true , address=145.79.227.4:4186 , piece_id=HHX2ZHYKTMM55GULLNOMQJ2WPZRP444XFAO65HU2RVAYWUCDZZ7A, seg_pos={0 0}, upload=444504]: Done uploading piece. err: <nil> Dec 26 18:03:13.439749002 (506h34m13.310832047s) [noise=true , address=45.91.116.9:4264 , piece_id=V6G45QGAWGT3MWIYQGPGHTF5XBCKD4G4G3FBXXFYGOKQBHE33LRQ, seg_pos={0 0}, upload=444504]: Done uploading piece. err: <nil> Dec 26 18:03:13.491708832 (506h34m13.362791878s) [noise=true , address=45.91.116.13:4156 , piece_id=UVYN4DXLISSVHNJFRXG6IZYN2LW7V6ZKFQRHNTB24T2UEAG2KW4Q, seg_pos={0 0}, upload=444504]: Done uploading piece. err: <nil> Dec 26 18:03:13.514759485 (506h34m13.38584253s) [noise=true , address=77.90.31.13:4627 , piece_id=FYB4X6MKKBN3L4RW3UKMR3YAX4WOQN63LQCN2MWHZJG3TVESVRQQ, seg_pos={0 0}, upload=444504]: Done uploading piece. err: <nil> Dec 26 18:03:13.585088304 (506h34m13.456171349s) [noise=true , address=45.91.116.14:4191 , piece_id=LG4MVMFR5622ZJOOMYG3IPLMMOPSWSSMHWOFCU76WXIAJIW4XUBQ, seg_pos={0 0}, upload=444504]: Done uploading piece. err: <nil> Dec 26 18:03:14.293444978 (506h34m14.164528033s) [noise=true , address=45.91.116.8:4045 , piece_id=6NSZ2L7LU5TU3OXOM6G7FVDUBDONHLESTCHJCILFY7WS7DU7YEKA, seg_pos={0 0}, upload=444504]: Done uploading piece. err: <nil> Dec 26 18:03:14.293466870 (506h34m14.164549925s) [seg_pos={0 0}, upload=444504]: Segment reached optimal threshold of 13 pieces. Dec 26 18:03:14.293642154 (506h34m14.164725219s) [noise=true , address=145.79.227.41:4086 , piece_id=RAUVGKJR7TNL5YJHPNAUIPI72CX5GCHNBT4HOR3IUB4AKKCLMO5Q, seg_pos={0 0}, upload=444504]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm): protocol: expected piece hash; context canceled; EOF Dec 26 18:03:14.293652554 (506h34m14.164735589s) [noise=true , address=145.79.227.41:4086 , piece_id=RAUVGKJR7TNL5YJHPNAUIPI72CX5GCHNBT4HOR3IUB4AKKCLMO5Q, seg_pos={0 0}, upload=444504]: Long tail cancellation due to optimal threshold Dec 26 18:03:14.294191048 (506h34m14.165274094s) [noise=true , address=45.91.116.10:4081 , piece_id=QC77DKAG4LZRC3UFN5UBTFBY7EC7P6Q5RUVITKZNUDVXL4LDTI6A, seg_pos={0 0}, upload=444504]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: context canceled; context canceled Dec 26 18:03:14.294210877 (506h34m14.165293922s) [noise=true , address=45.91.116.10:4081 , piece_id=QC77DKAG4LZRC3UFN5UBTFBY7EC7P6Q5RUVITKZNUDVXL4LDTI6A, seg_pos={0 0}, upload=444504]: Long tail cancellation due to optimal threshold Dec 26 18:03:14.294238509 (506h34m14.165321555s) [seg_pos={0 0}, upload=444504]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 18:03:14.294369770 (506h34m14.165452815s) [upload=444504]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 18:03:14.294377755 (506h34m14.1654608s) [upload=444504]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 18:03:14.588297970 (506h34m14.459381005s) [upload=444504]: Done uploading segments... Dec 26 18:03:14.588319040 (506h34m14.459402075s) [upload=444504]: Upload finished. err: <nil> Duration: 30.003642884s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 15.419372427s trace: https://jaeger.dev.storj.io/trace/665ea65f6e9ccc33 FAIL (2025-12-26 18:03:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== e=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789316979 (506h12m54.660400024s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789326507 (506h12m54.660409552s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789611339 (506h12m54.660694384s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789619275 (506h12m54.66070231s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789914006 (506h12m54.660997041s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789949443 (506h12m54.661032478s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790181566 (506h12m54.661264601s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790295632 (506h12m54.661378668s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790510872 (506h12m54.661593917s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790692980 (506h12m54.661776015s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790787379 (506h12m54.661870414s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790992970 (506h12m54.662076005s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791120053 (506h12m54.662203098s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791275218 (506h12m54.662358253s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791443709 (506h12m54.662526744s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:56.112313145 (506h12m55.98339619s) [upload=444191]: Next returned nil segment Dec 26 17:41:56.112330638 (506h12m55.983413683s) [upload=444191]: Waiting for error group managing segments... Dec 26 17:41:56.485322720 (506h12m56.356405765s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.485637079 (506h12m56.356720124s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.666807539 (506h12m56.537890584s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.678753987 (506h12m56.549837032s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.687788039 (506h12m56.558871074s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.694318023 (506h12m56.565401058s) [noise=true , address=145.79.227.7:4326 , piece_id=DMQ5RPI46X2U7TJTS42ZHQEF32BDJRKJJO5HWOYAPNS6VWVG5B5Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.716581187 (506h12m56.587664232s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.972183007 (506h12m56.843266042s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.011917999 (506h12m56.883001034s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.054168622 (506h12m56.925251667s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.114312777 (506h12m56.985395812s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.499846250 (506h12m57.370929285s) [noise=true , address=145.79.227.34:4919 , piece_id=YRWHW3BJVEGVJRSA6RJB3T7VJQSDTGKEZQKRICFIV2G3T7AN4MIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.612649066 (506h12m57.483732101s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649413733 (506h12m57.520496768s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649440282 (506h12m57.520523328s) [seg_pos={0 0}, upload=444191]: Segment reached optimal threshold of 14 pieces. Dec 26 17:41:57.649674729 (506h12m57.520757774s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G7ty71tPwe7cZSBNK9u74DFUCGomnxDsNvAif4C3SbP7HWwfo): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649684087 (506h12m57.520767142s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649747708 (506h12m57.520830763s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v6zXZuMmyL2MU8qb3q6gg1iwTYvndjzMjVpv8btEpeX9Lnqmu): protocol: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection Dec 26 17:41:57.649785219 (506h12m57.520868274s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649899828 (506h12m57.520982903s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649908775 (506h12m57.52099182s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650074641 (506h12m57.521157696s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.650083287 (506h12m57.521166342s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650132821 (506h12m57.521215866s) [seg_pos={0 0}, upload=444191]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 17:41:57.650220138 (506h12m57.521303183s) [upload=444191]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:41:57.650227171 (506h12m57.521310206s) [upload=444191]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:41:57.958890870 (506h12m57.829973915s) [upload=444191]: Done uploading segments... Dec 26 17:41:57.958911620 (506h12m57.829994665s) [upload=444191]: Upload finished. err: <nil> Duration: 15.419397193s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 15.534535388s trace: https://jaeger.dev.storj.io/trace/665ea65f6e9ccc33 FAIL (2025-12-26 18:03:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== e=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789316979 (506h12m54.660400024s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789326507 (506h12m54.660409552s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789611339 (506h12m54.660694384s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789619275 (506h12m54.66070231s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789914006 (506h12m54.660997041s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789949443 (506h12m54.661032478s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790181566 (506h12m54.661264601s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790295632 (506h12m54.661378668s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790510872 (506h12m54.661593917s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790692980 (506h12m54.661776015s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790787379 (506h12m54.661870414s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790992970 (506h12m54.662076005s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791120053 (506h12m54.662203098s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791275218 (506h12m54.662358253s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791443709 (506h12m54.662526744s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:56.112313145 (506h12m55.98339619s) [upload=444191]: Next returned nil segment Dec 26 17:41:56.112330638 (506h12m55.983413683s) [upload=444191]: Waiting for error group managing segments... Dec 26 17:41:56.485322720 (506h12m56.356405765s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.485637079 (506h12m56.356720124s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.666807539 (506h12m56.537890584s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.678753987 (506h12m56.549837032s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.687788039 (506h12m56.558871074s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.694318023 (506h12m56.565401058s) [noise=true , address=145.79.227.7:4326 , piece_id=DMQ5RPI46X2U7TJTS42ZHQEF32BDJRKJJO5HWOYAPNS6VWVG5B5Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.716581187 (506h12m56.587664232s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.972183007 (506h12m56.843266042s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.011917999 (506h12m56.883001034s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.054168622 (506h12m56.925251667s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.114312777 (506h12m56.985395812s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.499846250 (506h12m57.370929285s) [noise=true , address=145.79.227.34:4919 , piece_id=YRWHW3BJVEGVJRSA6RJB3T7VJQSDTGKEZQKRICFIV2G3T7AN4MIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.612649066 (506h12m57.483732101s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649413733 (506h12m57.520496768s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649440282 (506h12m57.520523328s) [seg_pos={0 0}, upload=444191]: Segment reached optimal threshold of 14 pieces. Dec 26 17:41:57.649674729 (506h12m57.520757774s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G7ty71tPwe7cZSBNK9u74DFUCGomnxDsNvAif4C3SbP7HWwfo): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649684087 (506h12m57.520767142s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649747708 (506h12m57.520830763s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v6zXZuMmyL2MU8qb3q6gg1iwTYvndjzMjVpv8btEpeX9Lnqmu): protocol: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection Dec 26 17:41:57.649785219 (506h12m57.520868274s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649899828 (506h12m57.520982903s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649908775 (506h12m57.52099182s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650074641 (506h12m57.521157696s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.650083287 (506h12m57.521166342s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650132821 (506h12m57.521215866s) [seg_pos={0 0}, upload=444191]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 17:41:57.650220138 (506h12m57.521303183s) [upload=444191]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:41:57.650227171 (506h12m57.521310206s) [upload=444191]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:41:57.958890870 (506h12m57.829973915s) [upload=444191]: Done uploading segments... Dec 26 17:41:57.958911620 (506h12m57.829994665s) [upload=444191]: Upload finished. err: <nil> Duration: 15.534587367s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [X.XX...XX...XX....XX..XX......X.XXX.....XX.X.XXX.........X..] duration: 5.72947971s trace: https://jaeger.dev.storj.io/trace/6c2ceda91b8dc6ef FAIL (2025-12-26 18:02:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: MGFBY7Y657TW2OEYGUEMDUVN6DYYWK5GTHQCGVACSYY33HMMCHJQ) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: JV6GLRMSHCB345DOFAMII6KWS74WJDZT3IGPLGMRZM2NUWYOB5PQ) context deadline exceeded ======LOGS====== 6 18:01:44.349873787 (506h32m44.220956832s) [upload=444482]: Starting upload Dec 26 18:01:44.349943950 (506h32m44.221026995s) [upload=444482]: Done starting upload Dec 26 18:01:44.350214976 (506h32m44.221298012s) [upload=444482]: Uploading segments... Dec 26 18:01:44.350283717 (506h32m44.221366773s) [upload=444482]: Got next segment. Inline: false Dec 26 18:01:44.350333432 (506h32m44.221416477s) [upload=444482]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 18:01:44.350339253 (506h32m44.221422298s) [upload=444482]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 18:01:44.601277424 (506h32m44.472360479s) [seg_pos={0 0}, upload=444482]: Begin upload segment... Dec 26 18:01:44.601356514 (506h32m44.472439559s) [seg_pos={0 0}, upload=444482]: Done begin upload segment. Dec 26 18:01:44.601440984 (506h32m44.472524019s) [noise=true , address=145.79.227.39:4013 , piece_id=EZHHDHZD3USFUSURBFORTHKGONZ6YJ2BCRUCUUKT4HUVSCL2LPTQ, seg_pos={0 0}, upload=444482]: Uploading piece... Dec 26 18:01:44.601479578 (506h32m44.472562613s) [noise=true , address=145.79.227.45:4231 , piece_id=XFMDANT7ZCIGASWV56DKO3OPAFAQNEJTRGEI563O37EO4YZSBELA, seg_pos={0 0}, upload=444482]: Uploading piece... Dec 26 18:01:44.602076505 (506h32m44.47315954s) [noise=true , address=45.91.116.14:4189 , piece_id=KUVACSJCPXTLHDNOL26KBRJA7KZMYXOFRU6S7ABZLCRYOOOMZD3Q, seg_pos={0 0}, upload=444482]: Uploading piece... Dec 26 18:01:44.602477328 (506h32m44.473560383s) [noise=true , address=145.79.227.51:4411 , piece_id=YKV7HVSLN4L7TPCJ3TPAKGMZ2ZRZXKC3B7NDJNBOFAQGMF4DDTRQ, seg_pos={0 0}, upload=444482]: Uploading piece... Dec 26 18:01:44.602736631 (506h32m44.473819666s) [noise=true , address=145.79.227.10:4401 , piece_id=XCYA3E6OVOJETWBYL34FP2NGVN47K75ZZTJRXPPH6NOKQ4XMJB7Q, seg_pos={0 0}, upload=444482]: Uploading piece... Dec 26 18:01:44.602955559 (506h32m44.474038604s) [noise=true , address=45.91.116.9:4263 , piece_id=MGFBY7Y657TW2OEYGUEMDUVN6DYYWK5GTHQCGVACSYY33HMMCHJQ, seg_pos={0 0}, upload=444482]: Uploading piece... Dec 26 18:01:44.603278083 (506h32m44.474361118s) [noise=true , address=77.90.31.19:4844 , piece_id=QC3UCTWSJDI2MDBDO3XWWFYRXL2APO7OIHY4BR7PAJSXPOZZYIDA, seg_pos={0 0}, upload=444482]: Uploading piece... Dec 26 18:01:44.603350230 (506h32m44.474433275s) [noise=true , address=145.79.227.8:4330 , piece_id=3YMCHIGG6LOX47OQ265IL4X6FJSK6RMZAWKMDF5C2LX7RDUBEINA, seg_pos={0 0}, upload=444482]: Uploading piece... Dec 26 18:01:44.603707049 (506h32m44.474790094s) [noise=true , address=77.90.31.14:4663 , piece_id=SESXAVIAYHPZ5KIQFD6QWNTI6NXKKCMSSYN6ZXDM2DUDN3CY7SIA, seg_pos={0 0}, upload=444482]: Uploading piece... Dec 26 18:01:44.604197894 (506h32m44.475280929s) [noise=true , address=77.90.31.10:4519 , piece_id=3WLJHO37GY3JLLHFJ2UDYXMOVR6VQ6T3U52YYPMJDAHHCE5LOPAA, seg_pos={0 0}, upload=444482]: Uploading piece... Dec 26 18:01:44.604605240 (506h32m44.475688275s) [noise=true , address=45.91.116.4:4937 , piece_id=GSM7OS24YKFP3VIINOAEGGX4EOBL2VQRDD6YKGU5T5GKXVNZ74HA, seg_pos={0 0}, upload=444482]: Uploading piece... Dec 26 18:01:44.604981677 (506h32m44.476064722s) [noise=true , address=145.79.227.11:4732 , piece_id=JGKV3PVFUSTOEK6VX3IX7J46TIJA22ZPDZNEBCKD6RI2FP45ZEGA, seg_pos={0 0}, upload=444482]: Uploading piece... Dec 26 18:01:44.605299181 (506h32m44.476382226s) [noise=true , address=45.91.116.6:4975 , piece_id=JV6GLRMSHCB345DOFAMII6KWS74WJDZT3IGPLGMRZM2NUWYOB5PQ, seg_pos={0 0}, upload=444482]: Uploading piece... Dec 26 18:01:44.605492490 (506h32m44.476575535s) [noise=true , address=145.79.227.31:4799 , piece_id=WEVOSTZDZUXV2WQ54RBGQHHG4GWVWNVS6JNPWPCCBO5RSLWX72OQ, seg_pos={0 0}, upload=444482]: Uploading piece... Dec 26 18:01:44.606080369 (506h32m44.477163414s) [noise=true , address=145.79.227.21:4474 , piece_id=HELCVB43WKB3ZDONRBS6BR3DMTYO76AWYORTIQ2NCV7SHI4KAMRQ, seg_pos={0 0}, upload=444482]: Uploading piece... Dec 26 18:01:44.606580110 (506h32m44.477663156s) [noise=true , address=145.79.227.4:4186 , piece_id=RWWTBRHMNLLAXYQBNHBGSCAPVNQVW4GBGNO3CKREZZU5WLOR3T7A, seg_pos={0 0}, upload=444482]: Uploading piece... Dec 26 18:01:46.319626871 (506h32m46.190709906s) [upload=444482]: Next returned nil segment Dec 26 18:01:46.319642711 (506h32m46.190725746s) [upload=444482]: Waiting for error group managing segments... Dec 26 18:01:46.607291835 (506h32m46.47837488s) [noise=true , address=145.79.227.10:4401 , piece_id=XCYA3E6OVOJETWBYL34FP2NGVN47K75ZZTJRXPPH6NOKQ4XMJB7Q, seg_pos={0 0}, upload=444482]: Done uploading piece. err: <nil> Dec 26 18:01:46.610693453 (506h32m46.481776488s) [noise=true , address=145.79.227.39:4013 , piece_id=EZHHDHZD3USFUSURBFORTHKGONZ6YJ2BCRUCUUKT4HUVSCL2LPTQ, seg_pos={0 0}, upload=444482]: Done uploading piece. err: <nil> Dec 26 18:01:46.611012420 (506h32m46.482095465s) [noise=true , address=145.79.227.8:4330 , piece_id=3YMCHIGG6LOX47OQ265IL4X6FJSK6RMZAWKMDF5C2LX7RDUBEINA, seg_pos={0 0}, upload=444482]: Done uploading piece. err: <nil> Dec 26 18:01:46.621508626 (506h32m46.492591671s) [noise=true , address=145.79.227.31:4799 , piece_id=WEVOSTZDZUXV2WQ54RBGQHHG4GWVWNVS6JNPWPCCBO5RSLWX72OQ, seg_pos={0 0}, upload=444482]: Done uploading piece. err: <nil> Dec 26 18:01:46.625365981 (506h32m46.496449016s) [noise=true , address=145.79.227.45:4231 , piece_id=XFMDANT7ZCIGASWV56DKO3OPAFAQNEJTRGEI563O37EO4YZSBELA, seg_pos={0 0}, upload=444482]: Done uploading piece. err: <nil> Dec 26 18:01:46.629445088 (506h32m46.500528133s) [noise=true , address=145.79.227.51:4411 , piece_id=YKV7HVSLN4L7TPCJ3TPAKGMZ2ZRZXKC3B7NDJNBOFAQGMF4DDTRQ, seg_pos={0 0}, upload=444482]: Done uploading piece. err: <nil> Dec 26 18:01:46.640495721 (506h32m46.511578756s) [noise=true , address=145.79.227.4:4186 , piece_id=RWWTBRHMNLLAXYQBNHBGSCAPVNQVW4GBGNO3CKREZZU5WLOR3T7A, seg_pos={0 0}, upload=444482]: Done uploading piece. err: <nil> Dec 26 18:01:46.645928436 (506h32m46.517011481s) [noise=true , address=145.79.227.21:4474 , piece_id=HELCVB43WKB3ZDONRBS6BR3DMTYO76AWYORTIQ2NCV7SHI4KAMRQ, seg_pos={0 0}, upload=444482]: Done uploading piece. err: <nil> Dec 26 18:01:46.654201760 (506h32m46.525284795s) [noise=true , address=145.79.227.11:4732 , piece_id=JGKV3PVFUSTOEK6VX3IX7J46TIJA22ZPDZNEBCKD6RI2FP45ZEGA, seg_pos={0 0}, upload=444482]: Done uploading piece. err: <nil> Dec 26 18:01:46.850051632 (506h32m46.721134677s) [noise=true , address=45.91.116.9:4263 , piece_id=MGFBY7Y657TW2OEYGUEMDUVN6DYYWK5GTHQCGVACSYY33HMMCHJQ, seg_pos={0 0}, upload=444482]: Done uploading piece. err: <nil> Dec 26 18:01:46.892001602 (506h32m46.763084637s) [noise=true , address=45.91.116.6:4975 , piece_id=JV6GLRMSHCB345DOFAMII6KWS74WJDZT3IGPLGMRZM2NUWYOB5PQ, seg_pos={0 0}, upload=444482]: Done uploading piece. err: <nil> Dec 26 18:01:46.896117650 (506h32m46.767200695s) [noise=true , address=45.91.116.14:4189 , piece_id=KUVACSJCPXTLHDNOL26KBRJA7KZMYXOFRU6S7ABZLCRYOOOMZD3Q, seg_pos={0 0}, upload=444482]: Done uploading piece. err: <nil> Dec 26 18:01:46.908261072 (506h32m46.779344118s) [noise=true , address=77.90.31.14:4663 , piece_id=SESXAVIAYHPZ5KIQFD6QWNTI6NXKKCMSSYN6ZXDM2DUDN3CY7SIA, seg_pos={0 0}, upload=444482]: Done uploading piece. err: <nil> Dec 26 18:01:46.908285720 (506h32m46.779368755s) [seg_pos={0 0}, upload=444482]: Segment reached optimal threshold of 13 pieces. Dec 26 18:01:46.908564050 (506h32m46.779647085s) [noise=true , address=77.90.31.19:4844 , piece_id=QC3UCTWSJDI2MDBDO3XWWFYRXL2APO7OIHY4BR7PAJSXPOZZYIDA, seg_pos={0 0}, upload=444482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: expected piece hash; context canceled; EOF Dec 26 18:01:46.908575322 (506h32m46.779658367s) [noise=true , address=77.90.31.19:4844 , piece_id=QC3UCTWSJDI2MDBDO3XWWFYRXL2APO7OIHY4BR7PAJSXPOZZYIDA, seg_pos={0 0}, upload=444482]: Long tail cancellation due to optimal threshold Dec 26 18:01:46.911452860 (506h32m46.782535926s) [noise=true , address=77.90.31.10:4519 , piece_id=3WLJHO37GY3JLLHFJ2UDYXMOVR6VQ6T3U52YYPMJDAHHCE5LOPAA, seg_pos={0 0}, upload=444482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: write tcp 188.166.225.170:50298->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:50298->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50298->77.90.31.10:4519: use of closed network connection Dec 26 18:01:46.911470074 (506h32m46.782553129s) [noise=true , address=77.90.31.10:4519 , piece_id=3WLJHO37GY3JLLHFJ2UDYXMOVR6VQ6T3U52YYPMJDAHHCE5LOPAA, seg_pos={0 0}, upload=444482]: Long tail cancellation due to optimal threshold Dec 26 18:01:46.911673732 (506h32m46.782756777s) [noise=true , address=45.91.116.4:4937 , piece_id=GSM7OS24YKFP3VIINOAEGGX4EOBL2VQRDD6YKGU5T5GKXVNZ74HA, seg_pos={0 0}, upload=444482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 26 18:01:46.911683430 (506h32m46.782766475s) [noise=true , address=45.91.116.4:4937 , piece_id=GSM7OS24YKFP3VIINOAEGGX4EOBL2VQRDD6YKGU5T5GKXVNZ74HA, seg_pos={0 0}, upload=444482]: Long tail cancellation due to optimal threshold Dec 26 18:01:46.911707225 (506h32m46.78279026s) [seg_pos={0 0}, upload=444482]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 18:01:46.911789291 (506h32m46.782872337s) [upload=444482]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 18:01:46.911795543 (506h32m46.782878588s) [upload=444482]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 18:01:47.214392370 (506h32m47.085475415s) [upload=444482]: Done uploading segments... Dec 26 18:01:47.214413361 (506h32m47.085496406s) [upload=444482]: Upload finished. err: <nil> Duration: 5.729504196s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [.X.XX...XX...XX....XX..XX......X.XXX.....XX.X.XXX.........X.] duration: 6.043270952s trace: https://jaeger.dev.storj.io/trace/6c2ceda91b8dc6ef FAIL (2025-12-26 18:02:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: MGFBY7Y657TW2OEYGUEMDUVN6DYYWK5GTHQCGVACSYY33HMMCHJQ) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: JV6GLRMSHCB345DOFAMII6KWS74WJDZT3IGPLGMRZM2NUWYOB5PQ) context deadline exceeded ======LOGS====== 6 18:01:44.349873787 (506h32m44.220956832s) [upload=444482]: Starting upload Dec 26 18:01:44.349943950 (506h32m44.221026995s) [upload=444482]: Done starting upload Dec 26 18:01:44.350214976 (506h32m44.221298012s) [upload=444482]: Uploading segments... Dec 26 18:01:44.350283717 (506h32m44.221366773s) [upload=444482]: Got next segment. Inline: false Dec 26 18:01:44.350333432 (506h32m44.221416477s) [upload=444482]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 18:01:44.350339253 (506h32m44.221422298s) [upload=444482]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 18:01:44.601277424 (506h32m44.472360479s) [seg_pos={0 0}, upload=444482]: Begin upload segment... Dec 26 18:01:44.601356514 (506h32m44.472439559s) [seg_pos={0 0}, upload=444482]: Done begin upload segment. Dec 26 18:01:44.601440984 (506h32m44.472524019s) [noise=true , address=145.79.227.39:4013 , piece_id=EZHHDHZD3USFUSURBFORTHKGONZ6YJ2BCRUCUUKT4HUVSCL2LPTQ, seg_pos={0 0}, upload=444482]: Uploading piece... Dec 26 18:01:44.601479578 (506h32m44.472562613s) [noise=true , address=145.79.227.45:4231 , piece_id=XFMDANT7ZCIGASWV56DKO3OPAFAQNEJTRGEI563O37EO4YZSBELA, seg_pos={0 0}, upload=444482]: Uploading piece... Dec 26 18:01:44.602076505 (506h32m44.47315954s) [noise=true , address=45.91.116.14:4189 , piece_id=KUVACSJCPXTLHDNOL26KBRJA7KZMYXOFRU6S7ABZLCRYOOOMZD3Q, seg_pos={0 0}, upload=444482]: Uploading piece... Dec 26 18:01:44.602477328 (506h32m44.473560383s) [noise=true , address=145.79.227.51:4411 , piece_id=YKV7HVSLN4L7TPCJ3TPAKGMZ2ZRZXKC3B7NDJNBOFAQGMF4DDTRQ, seg_pos={0 0}, upload=444482]: Uploading piece... Dec 26 18:01:44.602736631 (506h32m44.473819666s) [noise=true , address=145.79.227.10:4401 , piece_id=XCYA3E6OVOJETWBYL34FP2NGVN47K75ZZTJRXPPH6NOKQ4XMJB7Q, seg_pos={0 0}, upload=444482]: Uploading piece... Dec 26 18:01:44.602955559 (506h32m44.474038604s) [noise=true , address=45.91.116.9:4263 , piece_id=MGFBY7Y657TW2OEYGUEMDUVN6DYYWK5GTHQCGVACSYY33HMMCHJQ, seg_pos={0 0}, upload=444482]: Uploading piece... Dec 26 18:01:44.603278083 (506h32m44.474361118s) [noise=true , address=77.90.31.19:4844 , piece_id=QC3UCTWSJDI2MDBDO3XWWFYRXL2APO7OIHY4BR7PAJSXPOZZYIDA, seg_pos={0 0}, upload=444482]: Uploading piece... Dec 26 18:01:44.603350230 (506h32m44.474433275s) [noise=true , address=145.79.227.8:4330 , piece_id=3YMCHIGG6LOX47OQ265IL4X6FJSK6RMZAWKMDF5C2LX7RDUBEINA, seg_pos={0 0}, upload=444482]: Uploading piece... Dec 26 18:01:44.603707049 (506h32m44.474790094s) [noise=true , address=77.90.31.14:4663 , piece_id=SESXAVIAYHPZ5KIQFD6QWNTI6NXKKCMSSYN6ZXDM2DUDN3CY7SIA, seg_pos={0 0}, upload=444482]: Uploading piece... Dec 26 18:01:44.604197894 (506h32m44.475280929s) [noise=true , address=77.90.31.10:4519 , piece_id=3WLJHO37GY3JLLHFJ2UDYXMOVR6VQ6T3U52YYPMJDAHHCE5LOPAA, seg_pos={0 0}, upload=444482]: Uploading piece... Dec 26 18:01:44.604605240 (506h32m44.475688275s) [noise=true , address=45.91.116.4:4937 , piece_id=GSM7OS24YKFP3VIINOAEGGX4EOBL2VQRDD6YKGU5T5GKXVNZ74HA, seg_pos={0 0}, upload=444482]: Uploading piece... Dec 26 18:01:44.604981677 (506h32m44.476064722s) [noise=true , address=145.79.227.11:4732 , piece_id=JGKV3PVFUSTOEK6VX3IX7J46TIJA22ZPDZNEBCKD6RI2FP45ZEGA, seg_pos={0 0}, upload=444482]: Uploading piece... Dec 26 18:01:44.605299181 (506h32m44.476382226s) [noise=true , address=45.91.116.6:4975 , piece_id=JV6GLRMSHCB345DOFAMII6KWS74WJDZT3IGPLGMRZM2NUWYOB5PQ, seg_pos={0 0}, upload=444482]: Uploading piece... Dec 26 18:01:44.605492490 (506h32m44.476575535s) [noise=true , address=145.79.227.31:4799 , piece_id=WEVOSTZDZUXV2WQ54RBGQHHG4GWVWNVS6JNPWPCCBO5RSLWX72OQ, seg_pos={0 0}, upload=444482]: Uploading piece... Dec 26 18:01:44.606080369 (506h32m44.477163414s) [noise=true , address=145.79.227.21:4474 , piece_id=HELCVB43WKB3ZDONRBS6BR3DMTYO76AWYORTIQ2NCV7SHI4KAMRQ, seg_pos={0 0}, upload=444482]: Uploading piece... Dec 26 18:01:44.606580110 (506h32m44.477663156s) [noise=true , address=145.79.227.4:4186 , piece_id=RWWTBRHMNLLAXYQBNHBGSCAPVNQVW4GBGNO3CKREZZU5WLOR3T7A, seg_pos={0 0}, upload=444482]: Uploading piece... Dec 26 18:01:46.319626871 (506h32m46.190709906s) [upload=444482]: Next returned nil segment Dec 26 18:01:46.319642711 (506h32m46.190725746s) [upload=444482]: Waiting for error group managing segments... Dec 26 18:01:46.607291835 (506h32m46.47837488s) [noise=true , address=145.79.227.10:4401 , piece_id=XCYA3E6OVOJETWBYL34FP2NGVN47K75ZZTJRXPPH6NOKQ4XMJB7Q, seg_pos={0 0}, upload=444482]: Done uploading piece. err: <nil> Dec 26 18:01:46.610693453 (506h32m46.481776488s) [noise=true , address=145.79.227.39:4013 , piece_id=EZHHDHZD3USFUSURBFORTHKGONZ6YJ2BCRUCUUKT4HUVSCL2LPTQ, seg_pos={0 0}, upload=444482]: Done uploading piece. err: <nil> Dec 26 18:01:46.611012420 (506h32m46.482095465s) [noise=true , address=145.79.227.8:4330 , piece_id=3YMCHIGG6LOX47OQ265IL4X6FJSK6RMZAWKMDF5C2LX7RDUBEINA, seg_pos={0 0}, upload=444482]: Done uploading piece. err: <nil> Dec 26 18:01:46.621508626 (506h32m46.492591671s) [noise=true , address=145.79.227.31:4799 , piece_id=WEVOSTZDZUXV2WQ54RBGQHHG4GWVWNVS6JNPWPCCBO5RSLWX72OQ, seg_pos={0 0}, upload=444482]: Done uploading piece. err: <nil> Dec 26 18:01:46.625365981 (506h32m46.496449016s) [noise=true , address=145.79.227.45:4231 , piece_id=XFMDANT7ZCIGASWV56DKO3OPAFAQNEJTRGEI563O37EO4YZSBELA, seg_pos={0 0}, upload=444482]: Done uploading piece. err: <nil> Dec 26 18:01:46.629445088 (506h32m46.500528133s) [noise=true , address=145.79.227.51:4411 , piece_id=YKV7HVSLN4L7TPCJ3TPAKGMZ2ZRZXKC3B7NDJNBOFAQGMF4DDTRQ, seg_pos={0 0}, upload=444482]: Done uploading piece. err: <nil> Dec 26 18:01:46.640495721 (506h32m46.511578756s) [noise=true , address=145.79.227.4:4186 , piece_id=RWWTBRHMNLLAXYQBNHBGSCAPVNQVW4GBGNO3CKREZZU5WLOR3T7A, seg_pos={0 0}, upload=444482]: Done uploading piece. err: <nil> Dec 26 18:01:46.645928436 (506h32m46.517011481s) [noise=true , address=145.79.227.21:4474 , piece_id=HELCVB43WKB3ZDONRBS6BR3DMTYO76AWYORTIQ2NCV7SHI4KAMRQ, seg_pos={0 0}, upload=444482]: Done uploading piece. err: <nil> Dec 26 18:01:46.654201760 (506h32m46.525284795s) [noise=true , address=145.79.227.11:4732 , piece_id=JGKV3PVFUSTOEK6VX3IX7J46TIJA22ZPDZNEBCKD6RI2FP45ZEGA, seg_pos={0 0}, upload=444482]: Done uploading piece. err: <nil> Dec 26 18:01:46.850051632 (506h32m46.721134677s) [noise=true , address=45.91.116.9:4263 , piece_id=MGFBY7Y657TW2OEYGUEMDUVN6DYYWK5GTHQCGVACSYY33HMMCHJQ, seg_pos={0 0}, upload=444482]: Done uploading piece. err: <nil> Dec 26 18:01:46.892001602 (506h32m46.763084637s) [noise=true , address=45.91.116.6:4975 , piece_id=JV6GLRMSHCB345DOFAMII6KWS74WJDZT3IGPLGMRZM2NUWYOB5PQ, seg_pos={0 0}, upload=444482]: Done uploading piece. err: <nil> Dec 26 18:01:46.896117650 (506h32m46.767200695s) [noise=true , address=45.91.116.14:4189 , piece_id=KUVACSJCPXTLHDNOL26KBRJA7KZMYXOFRU6S7ABZLCRYOOOMZD3Q, seg_pos={0 0}, upload=444482]: Done uploading piece. err: <nil> Dec 26 18:01:46.908261072 (506h32m46.779344118s) [noise=true , address=77.90.31.14:4663 , piece_id=SESXAVIAYHPZ5KIQFD6QWNTI6NXKKCMSSYN6ZXDM2DUDN3CY7SIA, seg_pos={0 0}, upload=444482]: Done uploading piece. err: <nil> Dec 26 18:01:46.908285720 (506h32m46.779368755s) [seg_pos={0 0}, upload=444482]: Segment reached optimal threshold of 13 pieces. Dec 26 18:01:46.908564050 (506h32m46.779647085s) [noise=true , address=77.90.31.19:4844 , piece_id=QC3UCTWSJDI2MDBDO3XWWFYRXL2APO7OIHY4BR7PAJSXPOZZYIDA, seg_pos={0 0}, upload=444482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: expected piece hash; context canceled; EOF Dec 26 18:01:46.908575322 (506h32m46.779658367s) [noise=true , address=77.90.31.19:4844 , piece_id=QC3UCTWSJDI2MDBDO3XWWFYRXL2APO7OIHY4BR7PAJSXPOZZYIDA, seg_pos={0 0}, upload=444482]: Long tail cancellation due to optimal threshold Dec 26 18:01:46.911452860 (506h32m46.782535926s) [noise=true , address=77.90.31.10:4519 , piece_id=3WLJHO37GY3JLLHFJ2UDYXMOVR6VQ6T3U52YYPMJDAHHCE5LOPAA, seg_pos={0 0}, upload=444482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: write tcp 188.166.225.170:50298->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:50298->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50298->77.90.31.10:4519: use of closed network connection Dec 26 18:01:46.911470074 (506h32m46.782553129s) [noise=true , address=77.90.31.10:4519 , piece_id=3WLJHO37GY3JLLHFJ2UDYXMOVR6VQ6T3U52YYPMJDAHHCE5LOPAA, seg_pos={0 0}, upload=444482]: Long tail cancellation due to optimal threshold Dec 26 18:01:46.911673732 (506h32m46.782756777s) [noise=true , address=45.91.116.4:4937 , piece_id=GSM7OS24YKFP3VIINOAEGGX4EOBL2VQRDD6YKGU5T5GKXVNZ74HA, seg_pos={0 0}, upload=444482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 26 18:01:46.911683430 (506h32m46.782766475s) [noise=true , address=45.91.116.4:4937 , piece_id=GSM7OS24YKFP3VIINOAEGGX4EOBL2VQRDD6YKGU5T5GKXVNZ74HA, seg_pos={0 0}, upload=444482]: Long tail cancellation due to optimal threshold Dec 26 18:01:46.911707225 (506h32m46.78279026s) [seg_pos={0 0}, upload=444482]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 18:01:46.911789291 (506h32m46.782872337s) [upload=444482]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 18:01:46.911795543 (506h32m46.782878588s) [upload=444482]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 18:01:47.214392370 (506h32m47.085475415s) [upload=444482]: Done uploading segments... Dec 26 18:01:47.214413361 (506h32m47.085496406s) [upload=444482]: Upload finished. err: <nil> Duration: 6.043302452s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 20.419914382s trace: https://jaeger.dev.storj.io/trace/665ea65f6e9ccc33 FAIL (2025-12-26 18:02:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== e=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789316979 (506h12m54.660400024s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789326507 (506h12m54.660409552s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789611339 (506h12m54.660694384s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789619275 (506h12m54.66070231s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789914006 (506h12m54.660997041s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789949443 (506h12m54.661032478s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790181566 (506h12m54.661264601s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790295632 (506h12m54.661378668s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790510872 (506h12m54.661593917s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790692980 (506h12m54.661776015s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790787379 (506h12m54.661870414s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790992970 (506h12m54.662076005s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791120053 (506h12m54.662203098s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791275218 (506h12m54.662358253s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791443709 (506h12m54.662526744s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:56.112313145 (506h12m55.98339619s) [upload=444191]: Next returned nil segment Dec 26 17:41:56.112330638 (506h12m55.983413683s) [upload=444191]: Waiting for error group managing segments... Dec 26 17:41:56.485322720 (506h12m56.356405765s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.485637079 (506h12m56.356720124s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.666807539 (506h12m56.537890584s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.678753987 (506h12m56.549837032s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.687788039 (506h12m56.558871074s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.694318023 (506h12m56.565401058s) [noise=true , address=145.79.227.7:4326 , piece_id=DMQ5RPI46X2U7TJTS42ZHQEF32BDJRKJJO5HWOYAPNS6VWVG5B5Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.716581187 (506h12m56.587664232s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.972183007 (506h12m56.843266042s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.011917999 (506h12m56.883001034s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.054168622 (506h12m56.925251667s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.114312777 (506h12m56.985395812s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.499846250 (506h12m57.370929285s) [noise=true , address=145.79.227.34:4919 , piece_id=YRWHW3BJVEGVJRSA6RJB3T7VJQSDTGKEZQKRICFIV2G3T7AN4MIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.612649066 (506h12m57.483732101s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649413733 (506h12m57.520496768s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649440282 (506h12m57.520523328s) [seg_pos={0 0}, upload=444191]: Segment reached optimal threshold of 14 pieces. Dec 26 17:41:57.649674729 (506h12m57.520757774s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G7ty71tPwe7cZSBNK9u74DFUCGomnxDsNvAif4C3SbP7HWwfo): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649684087 (506h12m57.520767142s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649747708 (506h12m57.520830763s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v6zXZuMmyL2MU8qb3q6gg1iwTYvndjzMjVpv8btEpeX9Lnqmu): protocol: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection Dec 26 17:41:57.649785219 (506h12m57.520868274s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649899828 (506h12m57.520982903s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649908775 (506h12m57.52099182s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650074641 (506h12m57.521157696s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.650083287 (506h12m57.521166342s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650132821 (506h12m57.521215866s) [seg_pos={0 0}, upload=444191]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 17:41:57.650220138 (506h12m57.521303183s) [upload=444191]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:41:57.650227171 (506h12m57.521310206s) [upload=444191]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:41:57.958890870 (506h12m57.829973915s) [upload=444191]: Done uploading segments... Dec 26 17:41:57.958911620 (506h12m57.829994665s) [upload=444191]: Upload finished. err: <nil> Duration: 20.419936203s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [X.X.XX...XX...XX....XX..XX......X.XXX.....XX.X.XXX.........X] duration: 30.003482929s trace: https://jaeger.dev.storj.io/trace/6c2ceda91b8dc6ef FAIL (2025-12-26 18:02:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: MGFBY7Y657TW2OEYGUEMDUVN6DYYWK5GTHQCGVACSYY33HMMCHJQ) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: JV6GLRMSHCB345DOFAMII6KWS74WJDZT3IGPLGMRZM2NUWYOB5PQ) context deadline exceeded ======LOGS====== 6 18:01:44.349873787 (506h32m44.220956832s) [upload=444482]: Starting upload Dec 26 18:01:44.349943950 (506h32m44.221026995s) [upload=444482]: Done starting upload Dec 26 18:01:44.350214976 (506h32m44.221298012s) [upload=444482]: Uploading segments... Dec 26 18:01:44.350283717 (506h32m44.221366773s) [upload=444482]: Got next segment. Inline: false Dec 26 18:01:44.350333432 (506h32m44.221416477s) [upload=444482]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 18:01:44.350339253 (506h32m44.221422298s) [upload=444482]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 18:01:44.601277424 (506h32m44.472360479s) [seg_pos={0 0}, upload=444482]: Begin upload segment... Dec 26 18:01:44.601356514 (506h32m44.472439559s) [seg_pos={0 0}, upload=444482]: Done begin upload segment. Dec 26 18:01:44.601440984 (506h32m44.472524019s) [noise=true , address=145.79.227.39:4013 , piece_id=EZHHDHZD3USFUSURBFORTHKGONZ6YJ2BCRUCUUKT4HUVSCL2LPTQ, seg_pos={0 0}, upload=444482]: Uploading piece... Dec 26 18:01:44.601479578 (506h32m44.472562613s) [noise=true , address=145.79.227.45:4231 , piece_id=XFMDANT7ZCIGASWV56DKO3OPAFAQNEJTRGEI563O37EO4YZSBELA, seg_pos={0 0}, upload=444482]: Uploading piece... Dec 26 18:01:44.602076505 (506h32m44.47315954s) [noise=true , address=45.91.116.14:4189 , piece_id=KUVACSJCPXTLHDNOL26KBRJA7KZMYXOFRU6S7ABZLCRYOOOMZD3Q, seg_pos={0 0}, upload=444482]: Uploading piece... Dec 26 18:01:44.602477328 (506h32m44.473560383s) [noise=true , address=145.79.227.51:4411 , piece_id=YKV7HVSLN4L7TPCJ3TPAKGMZ2ZRZXKC3B7NDJNBOFAQGMF4DDTRQ, seg_pos={0 0}, upload=444482]: Uploading piece... Dec 26 18:01:44.602736631 (506h32m44.473819666s) [noise=true , address=145.79.227.10:4401 , piece_id=XCYA3E6OVOJETWBYL34FP2NGVN47K75ZZTJRXPPH6NOKQ4XMJB7Q, seg_pos={0 0}, upload=444482]: Uploading piece... Dec 26 18:01:44.602955559 (506h32m44.474038604s) [noise=true , address=45.91.116.9:4263 , piece_id=MGFBY7Y657TW2OEYGUEMDUVN6DYYWK5GTHQCGVACSYY33HMMCHJQ, seg_pos={0 0}, upload=444482]: Uploading piece... Dec 26 18:01:44.603278083 (506h32m44.474361118s) [noise=true , address=77.90.31.19:4844 , piece_id=QC3UCTWSJDI2MDBDO3XWWFYRXL2APO7OIHY4BR7PAJSXPOZZYIDA, seg_pos={0 0}, upload=444482]: Uploading piece... Dec 26 18:01:44.603350230 (506h32m44.474433275s) [noise=true , address=145.79.227.8:4330 , piece_id=3YMCHIGG6LOX47OQ265IL4X6FJSK6RMZAWKMDF5C2LX7RDUBEINA, seg_pos={0 0}, upload=444482]: Uploading piece... Dec 26 18:01:44.603707049 (506h32m44.474790094s) [noise=true , address=77.90.31.14:4663 , piece_id=SESXAVIAYHPZ5KIQFD6QWNTI6NXKKCMSSYN6ZXDM2DUDN3CY7SIA, seg_pos={0 0}, upload=444482]: Uploading piece... Dec 26 18:01:44.604197894 (506h32m44.475280929s) [noise=true , address=77.90.31.10:4519 , piece_id=3WLJHO37GY3JLLHFJ2UDYXMOVR6VQ6T3U52YYPMJDAHHCE5LOPAA, seg_pos={0 0}, upload=444482]: Uploading piece... Dec 26 18:01:44.604605240 (506h32m44.475688275s) [noise=true , address=45.91.116.4:4937 , piece_id=GSM7OS24YKFP3VIINOAEGGX4EOBL2VQRDD6YKGU5T5GKXVNZ74HA, seg_pos={0 0}, upload=444482]: Uploading piece... Dec 26 18:01:44.604981677 (506h32m44.476064722s) [noise=true , address=145.79.227.11:4732 , piece_id=JGKV3PVFUSTOEK6VX3IX7J46TIJA22ZPDZNEBCKD6RI2FP45ZEGA, seg_pos={0 0}, upload=444482]: Uploading piece... Dec 26 18:01:44.605299181 (506h32m44.476382226s) [noise=true , address=45.91.116.6:4975 , piece_id=JV6GLRMSHCB345DOFAMII6KWS74WJDZT3IGPLGMRZM2NUWYOB5PQ, seg_pos={0 0}, upload=444482]: Uploading piece... Dec 26 18:01:44.605492490 (506h32m44.476575535s) [noise=true , address=145.79.227.31:4799 , piece_id=WEVOSTZDZUXV2WQ54RBGQHHG4GWVWNVS6JNPWPCCBO5RSLWX72OQ, seg_pos={0 0}, upload=444482]: Uploading piece... Dec 26 18:01:44.606080369 (506h32m44.477163414s) [noise=true , address=145.79.227.21:4474 , piece_id=HELCVB43WKB3ZDONRBS6BR3DMTYO76AWYORTIQ2NCV7SHI4KAMRQ, seg_pos={0 0}, upload=444482]: Uploading piece... Dec 26 18:01:44.606580110 (506h32m44.477663156s) [noise=true , address=145.79.227.4:4186 , piece_id=RWWTBRHMNLLAXYQBNHBGSCAPVNQVW4GBGNO3CKREZZU5WLOR3T7A, seg_pos={0 0}, upload=444482]: Uploading piece... Dec 26 18:01:46.319626871 (506h32m46.190709906s) [upload=444482]: Next returned nil segment Dec 26 18:01:46.319642711 (506h32m46.190725746s) [upload=444482]: Waiting for error group managing segments... Dec 26 18:01:46.607291835 (506h32m46.47837488s) [noise=true , address=145.79.227.10:4401 , piece_id=XCYA3E6OVOJETWBYL34FP2NGVN47K75ZZTJRXPPH6NOKQ4XMJB7Q, seg_pos={0 0}, upload=444482]: Done uploading piece. err: <nil> Dec 26 18:01:46.610693453 (506h32m46.481776488s) [noise=true , address=145.79.227.39:4013 , piece_id=EZHHDHZD3USFUSURBFORTHKGONZ6YJ2BCRUCUUKT4HUVSCL2LPTQ, seg_pos={0 0}, upload=444482]: Done uploading piece. err: <nil> Dec 26 18:01:46.611012420 (506h32m46.482095465s) [noise=true , address=145.79.227.8:4330 , piece_id=3YMCHIGG6LOX47OQ265IL4X6FJSK6RMZAWKMDF5C2LX7RDUBEINA, seg_pos={0 0}, upload=444482]: Done uploading piece. err: <nil> Dec 26 18:01:46.621508626 (506h32m46.492591671s) [noise=true , address=145.79.227.31:4799 , piece_id=WEVOSTZDZUXV2WQ54RBGQHHG4GWVWNVS6JNPWPCCBO5RSLWX72OQ, seg_pos={0 0}, upload=444482]: Done uploading piece. err: <nil> Dec 26 18:01:46.625365981 (506h32m46.496449016s) [noise=true , address=145.79.227.45:4231 , piece_id=XFMDANT7ZCIGASWV56DKO3OPAFAQNEJTRGEI563O37EO4YZSBELA, seg_pos={0 0}, upload=444482]: Done uploading piece. err: <nil> Dec 26 18:01:46.629445088 (506h32m46.500528133s) [noise=true , address=145.79.227.51:4411 , piece_id=YKV7HVSLN4L7TPCJ3TPAKGMZ2ZRZXKC3B7NDJNBOFAQGMF4DDTRQ, seg_pos={0 0}, upload=444482]: Done uploading piece. err: <nil> Dec 26 18:01:46.640495721 (506h32m46.511578756s) [noise=true , address=145.79.227.4:4186 , piece_id=RWWTBRHMNLLAXYQBNHBGSCAPVNQVW4GBGNO3CKREZZU5WLOR3T7A, seg_pos={0 0}, upload=444482]: Done uploading piece. err: <nil> Dec 26 18:01:46.645928436 (506h32m46.517011481s) [noise=true , address=145.79.227.21:4474 , piece_id=HELCVB43WKB3ZDONRBS6BR3DMTYO76AWYORTIQ2NCV7SHI4KAMRQ, seg_pos={0 0}, upload=444482]: Done uploading piece. err: <nil> Dec 26 18:01:46.654201760 (506h32m46.525284795s) [noise=true , address=145.79.227.11:4732 , piece_id=JGKV3PVFUSTOEK6VX3IX7J46TIJA22ZPDZNEBCKD6RI2FP45ZEGA, seg_pos={0 0}, upload=444482]: Done uploading piece. err: <nil> Dec 26 18:01:46.850051632 (506h32m46.721134677s) [noise=true , address=45.91.116.9:4263 , piece_id=MGFBY7Y657TW2OEYGUEMDUVN6DYYWK5GTHQCGVACSYY33HMMCHJQ, seg_pos={0 0}, upload=444482]: Done uploading piece. err: <nil> Dec 26 18:01:46.892001602 (506h32m46.763084637s) [noise=true , address=45.91.116.6:4975 , piece_id=JV6GLRMSHCB345DOFAMII6KWS74WJDZT3IGPLGMRZM2NUWYOB5PQ, seg_pos={0 0}, upload=444482]: Done uploading piece. err: <nil> Dec 26 18:01:46.896117650 (506h32m46.767200695s) [noise=true , address=45.91.116.14:4189 , piece_id=KUVACSJCPXTLHDNOL26KBRJA7KZMYXOFRU6S7ABZLCRYOOOMZD3Q, seg_pos={0 0}, upload=444482]: Done uploading piece. err: <nil> Dec 26 18:01:46.908261072 (506h32m46.779344118s) [noise=true , address=77.90.31.14:4663 , piece_id=SESXAVIAYHPZ5KIQFD6QWNTI6NXKKCMSSYN6ZXDM2DUDN3CY7SIA, seg_pos={0 0}, upload=444482]: Done uploading piece. err: <nil> Dec 26 18:01:46.908285720 (506h32m46.779368755s) [seg_pos={0 0}, upload=444482]: Segment reached optimal threshold of 13 pieces. Dec 26 18:01:46.908564050 (506h32m46.779647085s) [noise=true , address=77.90.31.19:4844 , piece_id=QC3UCTWSJDI2MDBDO3XWWFYRXL2APO7OIHY4BR7PAJSXPOZZYIDA, seg_pos={0 0}, upload=444482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: expected piece hash; context canceled; EOF Dec 26 18:01:46.908575322 (506h32m46.779658367s) [noise=true , address=77.90.31.19:4844 , piece_id=QC3UCTWSJDI2MDBDO3XWWFYRXL2APO7OIHY4BR7PAJSXPOZZYIDA, seg_pos={0 0}, upload=444482]: Long tail cancellation due to optimal threshold Dec 26 18:01:46.911452860 (506h32m46.782535926s) [noise=true , address=77.90.31.10:4519 , piece_id=3WLJHO37GY3JLLHFJ2UDYXMOVR6VQ6T3U52YYPMJDAHHCE5LOPAA, seg_pos={0 0}, upload=444482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: write tcp 188.166.225.170:50298->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:50298->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50298->77.90.31.10:4519: use of closed network connection Dec 26 18:01:46.911470074 (506h32m46.782553129s) [noise=true , address=77.90.31.10:4519 , piece_id=3WLJHO37GY3JLLHFJ2UDYXMOVR6VQ6T3U52YYPMJDAHHCE5LOPAA, seg_pos={0 0}, upload=444482]: Long tail cancellation due to optimal threshold Dec 26 18:01:46.911673732 (506h32m46.782756777s) [noise=true , address=45.91.116.4:4937 , piece_id=GSM7OS24YKFP3VIINOAEGGX4EOBL2VQRDD6YKGU5T5GKXVNZ74HA, seg_pos={0 0}, upload=444482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 26 18:01:46.911683430 (506h32m46.782766475s) [noise=true , address=45.91.116.4:4937 , piece_id=GSM7OS24YKFP3VIINOAEGGX4EOBL2VQRDD6YKGU5T5GKXVNZ74HA, seg_pos={0 0}, upload=444482]: Long tail cancellation due to optimal threshold Dec 26 18:01:46.911707225 (506h32m46.78279026s) [seg_pos={0 0}, upload=444482]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 18:01:46.911789291 (506h32m46.782872337s) [upload=444482]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 18:01:46.911795543 (506h32m46.782878588s) [upload=444482]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 18:01:47.214392370 (506h32m47.085475415s) [upload=444482]: Done uploading segments... Dec 26 18:01:47.214413361 (506h32m47.085496406s) [upload=444482]: Upload finished. err: <nil> Duration: 30.003506404s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 19.274835321s trace: https://jaeger.dev.storj.io/trace/665ea65f6e9ccc33 FAIL (2025-12-26 18:01:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== e=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789316979 (506h12m54.660400024s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789326507 (506h12m54.660409552s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789611339 (506h12m54.660694384s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789619275 (506h12m54.66070231s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789914006 (506h12m54.660997041s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789949443 (506h12m54.661032478s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790181566 (506h12m54.661264601s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790295632 (506h12m54.661378668s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790510872 (506h12m54.661593917s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790692980 (506h12m54.661776015s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790787379 (506h12m54.661870414s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790992970 (506h12m54.662076005s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791120053 (506h12m54.662203098s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791275218 (506h12m54.662358253s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791443709 (506h12m54.662526744s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:56.112313145 (506h12m55.98339619s) [upload=444191]: Next returned nil segment Dec 26 17:41:56.112330638 (506h12m55.983413683s) [upload=444191]: Waiting for error group managing segments... Dec 26 17:41:56.485322720 (506h12m56.356405765s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.485637079 (506h12m56.356720124s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.666807539 (506h12m56.537890584s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.678753987 (506h12m56.549837032s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.687788039 (506h12m56.558871074s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.694318023 (506h12m56.565401058s) [noise=true , address=145.79.227.7:4326 , piece_id=DMQ5RPI46X2U7TJTS42ZHQEF32BDJRKJJO5HWOYAPNS6VWVG5B5Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.716581187 (506h12m56.587664232s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.972183007 (506h12m56.843266042s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.011917999 (506h12m56.883001034s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.054168622 (506h12m56.925251667s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.114312777 (506h12m56.985395812s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.499846250 (506h12m57.370929285s) [noise=true , address=145.79.227.34:4919 , piece_id=YRWHW3BJVEGVJRSA6RJB3T7VJQSDTGKEZQKRICFIV2G3T7AN4MIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.612649066 (506h12m57.483732101s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649413733 (506h12m57.520496768s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649440282 (506h12m57.520523328s) [seg_pos={0 0}, upload=444191]: Segment reached optimal threshold of 14 pieces. Dec 26 17:41:57.649674729 (506h12m57.520757774s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G7ty71tPwe7cZSBNK9u74DFUCGomnxDsNvAif4C3SbP7HWwfo): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649684087 (506h12m57.520767142s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649747708 (506h12m57.520830763s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v6zXZuMmyL2MU8qb3q6gg1iwTYvndjzMjVpv8btEpeX9Lnqmu): protocol: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection Dec 26 17:41:57.649785219 (506h12m57.520868274s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649899828 (506h12m57.520982903s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649908775 (506h12m57.52099182s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650074641 (506h12m57.521157696s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.650083287 (506h12m57.521166342s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650132821 (506h12m57.521215866s) [seg_pos={0 0}, upload=444191]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 17:41:57.650220138 (506h12m57.521303183s) [upload=444191]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:41:57.650227171 (506h12m57.521310206s) [upload=444191]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:41:57.958890870 (506h12m57.829973915s) [upload=444191]: Done uploading segments... Dec 26 17:41:57.958911620 (506h12m57.829994665s) [upload=444191]: Upload finished. err: <nil> Duration: 19.274863425s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [.X.X.XX...XX...XX....XX..XX......X.XXX.....XX.X.XXX.........] duration: 5.84582447s trace: https://jaeger.dev.storj.io/trace/66e1eaafd7692f5a FAIL (2025-12-26 18:00:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: VVU5B5KNVMOIGGZABQAAI22IQ2CZTWJURAJWQQ2JGY3ABTBPFK3A) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: AC62U7GFNGWMPNEFW5H2C4IMSFT6TSFNH5EHUCX24SWZXCPRKOQQ) context deadline exceeded ======LOGS====== Dec 26 17:54:05.860888315 (506h25m5.73197135s) [upload=444374]: Starting upload Dec 26 17:54:05.860960913 (506h25m5.732043958s) [upload=444374]: Done starting upload Dec 26 17:54:05.861243932 (506h25m5.732326967s) [upload=444374]: Uploading segments... Dec 26 17:54:05.861348571 (506h25m5.732431616s) [upload=444374]: Got next segment. Inline: false Dec 26 17:54:05.861399419 (506h25m5.732482464s) [upload=444374]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 17:54:05.861404999 (506h25m5.732488034s) [upload=444374]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 17:54:06.104522897 (506h25m5.975605933s) [seg_pos={0 0}, upload=444374]: Begin upload segment... Dec 26 17:54:06.104572262 (506h25m5.975655297s) [seg_pos={0 0}, upload=444374]: Done begin upload segment. Dec 26 17:54:06.104602710 (506h25m5.975685755s) [noise=true , address=77.90.31.12:4589 , piece_id=CJJUEAIEWP66VHOKZTYLIZUHN3OET6SAFJHXK73EBDFNB5QFE5RA, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.104713491 (506h25m5.975796536s) [noise=true , address=145.79.227.21:4475 , piece_id=UP7LFRPWG3GFBUFT6Y3TOMII7C2UE5BJEY7VCDKHWPYN7NHOL25A, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.105352147 (506h25m5.976435192s) [noise=true , address=145.79.227.51:4410 , piece_id=NJSE6VNZ2ICW5KYFAVC7TXUKOHPAQSHATMU2WW2XMXJF77VTZLVQ, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.105704788 (506h25m5.976787823s) [noise=true , address=77.90.31.15:4700 , piece_id=AC62U7GFNGWMPNEFW5H2C4IMSFT6TSFNH5EHUCX24SWZXCPRKOQQ, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.106020589 (506h25m5.977103624s) [noise=true , address=77.90.31.14:4663 , piece_id=WOQS65FTAS6NTIR7QVGUIFPIJY6POSNUFZTBGYCDZW67TKWZ2GZQ, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.106328646 (506h25m5.977411681s) [noise=true , address=145.79.227.42:4123 , piece_id=T5UOINXTMKBAACABHEVV5NGEMB52SGKWNSJPJ7Y6UKCQXNOUOJMA, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.106787600 (506h25m5.977870635s) [noise=true , address=45.91.116.6:4974 , piece_id=23YEXTSWEM6GOICKLZM5QVYLA7OK2HE55TIA52YG4YO26WBFK4LA, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.107122217 (506h25m5.978205252s) [noise=true , address=145.79.227.49:4339 , piece_id=NHMAMURTEDXM7SZDKB6OP4NU6SCIIKVLIHTVAOT2DVU7UAIT6V4Q, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.108570165 (506h25m5.97965324s) [noise=true , address=145.79.227.15:4874 , piece_id=7PRWX53PMTA2U7GAUIKHBWJXO6RANYJHXUIUFMG3UNASTUTNFIXQ, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.108781597 (506h25m5.979864633s) [noise=true , address=45.91.116.5:4297 , piece_id=I3IPV24ECC3YV4V3QRQDP2ZZT3JOO2CN6M6BISNPENMQZNSDKVAQ, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.110063228 (506h25m5.981146263s) [noise=true , address=145.79.227.11:4732 , piece_id=5FVE6U6ISMCIMDESG3OQ6SHFBGTWT2PPZZC7XRZAPSNK47HZIBWQ, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.111752484 (506h25m5.982835529s) [noise=true , address=145.79.227.36:4519 , piece_id=FJEF6GROIQF7AWQYSJHUEINR5FTBIO7USLJGFWEEEDRJSGW2JG4Q, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.113340689 (506h25m5.984423734s) [noise=true , address=145.79.227.10:4401 , piece_id=ZI4WZERG34DPXHUOLN6QZ55O6SJPZT5LPXPPUNMGGTYAZCAOTORA, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.114643860 (506h25m5.985726905s) [noise=true , address=45.91.116.13:4153 , piece_id=VVU5B5KNVMOIGGZABQAAI22IQ2CZTWJURAJWQQ2JGY3ABTBPFK3A, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.115852932 (506h25m5.986935967s) [noise=true , address=77.90.31.19:4844 , piece_id=AVEWXCOCGI7U437EYT37M4VY3TH73JESDRWAXRLHQKONPFPLAHTQ, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.116921628 (506h25m5.988004673s) [noise=true , address=145.79.227.39:4015 , piece_id=XD5CWP335WRWKXUP3WKMIO7VOFE5G4RN4EGONIRXB6HI43GOCJFQ, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:07.294144135 (506h25m7.16522718s) [noise=true , address=77.90.31.12:4589 , piece_id=CJJUEAIEWP66VHOKZTYLIZUHN3OET6SAFJHXK73EBDFNB5QFE5RA, seg_pos={0 0}, upload=444374]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 26 17:54:07.828164492 (506h25m7.699247537s) [upload=444374]: Next returned nil segment Dec 26 17:54:07.828181555 (506h25m7.6992646s) [upload=444374]: Waiting for error group managing segments... Dec 26 17:54:08.107862847 (506h25m7.978945892s) [noise=true , address=145.79.227.21:4475 , piece_id=UP7LFRPWG3GFBUFT6Y3TOMII7C2UE5BJEY7VCDKHWPYN7NHOL25A, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.112828612 (506h25m7.983911657s) [noise=true , address=145.79.227.51:4410 , piece_id=NJSE6VNZ2ICW5KYFAVC7TXUKOHPAQSHATMU2WW2XMXJF77VTZLVQ, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.119737338 (506h25m7.990820383s) [noise=true , address=145.79.227.10:4401 , piece_id=ZI4WZERG34DPXHUOLN6QZ55O6SJPZT5LPXPPUNMGGTYAZCAOTORA, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.121777863 (506h25m7.992860908s) [noise=true , address=145.79.227.11:4732 , piece_id=5FVE6U6ISMCIMDESG3OQ6SHFBGTWT2PPZZC7XRZAPSNK47HZIBWQ, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.130879434 (506h25m8.00196248s) [noise=true , address=145.79.227.39:4015 , piece_id=XD5CWP335WRWKXUP3WKMIO7VOFE5G4RN4EGONIRXB6HI43GOCJFQ, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.132848083 (506h25m8.003931118s) [noise=true , address=145.79.227.42:4123 , piece_id=T5UOINXTMKBAACABHEVV5NGEMB52SGKWNSJPJ7Y6UKCQXNOUOJMA, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.135816906 (506h25m8.006899952s) [noise=true , address=145.79.227.49:4339 , piece_id=NHMAMURTEDXM7SZDKB6OP4NU6SCIIKVLIHTVAOT2DVU7UAIT6V4Q, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.139078326 (506h25m8.010161361s) [noise=true , address=145.79.227.15:4874 , piece_id=7PRWX53PMTA2U7GAUIKHBWJXO6RANYJHXUIUFMG3UNASTUTNFIXQ, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.139833234 (506h25m8.010916279s) [noise=true , address=145.79.227.36:4519 , piece_id=FJEF6GROIQF7AWQYSJHUEINR5FTBIO7USLJGFWEEEDRJSGW2JG4Q, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.299232928 (506h25m8.170315963s) [noise=true , address=77.90.31.15:4700 , piece_id=AC62U7GFNGWMPNEFW5H2C4IMSFT6TSFNH5EHUCX24SWZXCPRKOQQ, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.362334064 (506h25m8.233417109s) [noise=true , address=45.91.116.13:4153 , piece_id=VVU5B5KNVMOIGGZABQAAI22IQ2CZTWJURAJWQQ2JGY3ABTBPFK3A, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.462770483 (506h25m8.333853538s) [noise=true , address=77.90.31.19:4844 , piece_id=AVEWXCOCGI7U437EYT37M4VY3TH73JESDRWAXRLHQKONPFPLAHTQ, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.474990001 (506h25m8.346073046s) [noise=true , address=45.91.116.6:4974 , piece_id=23YEXTSWEM6GOICKLZM5QVYLA7OK2HE55TIA52YG4YO26WBFK4LA, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.475014728 (506h25m8.346097793s) [seg_pos={0 0}, upload=444374]: Segment reached optimal threshold of 13 pieces. Dec 26 17:54:08.475260065 (506h25m8.34634311s) [noise=true , address=45.91.116.5:4297 , piece_id=I3IPV24ECC3YV4V3QRQDP2ZZT3JOO2CN6M6BISNPENMQZNSDKVAQ, seg_pos={0 0}, upload=444374]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 26 17:54:08.475266929 (506h25m8.346349964s) [noise=true , address=45.91.116.5:4297 , piece_id=I3IPV24ECC3YV4V3QRQDP2ZZT3JOO2CN6M6BISNPENMQZNSDKVAQ, seg_pos={0 0}, upload=444374]: Long tail cancellation due to optimal threshold Dec 26 17:54:08.475601164 (506h25m8.346684209s) [noise=true , address=77.90.31.14:4663 , piece_id=WOQS65FTAS6NTIR7QVGUIFPIJY6POSNUFZTBGYCDZW67TKWZ2GZQ, seg_pos={0 0}, upload=444374]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: expected piece hash; context canceled; EOF Dec 26 17:54:08.475628697 (506h25m8.346711732s) [noise=true , address=77.90.31.14:4663 , piece_id=WOQS65FTAS6NTIR7QVGUIFPIJY6POSNUFZTBGYCDZW67TKWZ2GZQ, seg_pos={0 0}, upload=444374]: Long tail cancellation due to optimal threshold Dec 26 17:54:08.475658875 (506h25m8.34674192s) [seg_pos={0 0}, upload=444374]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 17:54:08.475744387 (506h25m8.346827422s) [upload=444374]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:54:08.475750088 (506h25m8.346833133s) [upload=444374]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:54:08.767674804 (506h25m8.638757859s) [upload=444374]: Done uploading segments... Dec 26 17:54:08.767701635 (506h25m8.63878468s) [upload=444374]: Upload finished. err: <nil> Duration: 5.845842494s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [..X.X.XX...XX...XX....XX..XX......X.XXX.....XX.X.XXX........] duration: 6.64904854s trace: https://jaeger.dev.storj.io/trace/66e1eaafd7692f5a FAIL (2025-12-26 18:00:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: VVU5B5KNVMOIGGZABQAAI22IQ2CZTWJURAJWQQ2JGY3ABTBPFK3A) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: AC62U7GFNGWMPNEFW5H2C4IMSFT6TSFNH5EHUCX24SWZXCPRKOQQ) context deadline exceeded ======LOGS====== Dec 26 17:54:05.860888315 (506h25m5.73197135s) [upload=444374]: Starting upload Dec 26 17:54:05.860960913 (506h25m5.732043958s) [upload=444374]: Done starting upload Dec 26 17:54:05.861243932 (506h25m5.732326967s) [upload=444374]: Uploading segments... Dec 26 17:54:05.861348571 (506h25m5.732431616s) [upload=444374]: Got next segment. Inline: false Dec 26 17:54:05.861399419 (506h25m5.732482464s) [upload=444374]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 17:54:05.861404999 (506h25m5.732488034s) [upload=444374]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 17:54:06.104522897 (506h25m5.975605933s) [seg_pos={0 0}, upload=444374]: Begin upload segment... Dec 26 17:54:06.104572262 (506h25m5.975655297s) [seg_pos={0 0}, upload=444374]: Done begin upload segment. Dec 26 17:54:06.104602710 (506h25m5.975685755s) [noise=true , address=77.90.31.12:4589 , piece_id=CJJUEAIEWP66VHOKZTYLIZUHN3OET6SAFJHXK73EBDFNB5QFE5RA, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.104713491 (506h25m5.975796536s) [noise=true , address=145.79.227.21:4475 , piece_id=UP7LFRPWG3GFBUFT6Y3TOMII7C2UE5BJEY7VCDKHWPYN7NHOL25A, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.105352147 (506h25m5.976435192s) [noise=true , address=145.79.227.51:4410 , piece_id=NJSE6VNZ2ICW5KYFAVC7TXUKOHPAQSHATMU2WW2XMXJF77VTZLVQ, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.105704788 (506h25m5.976787823s) [noise=true , address=77.90.31.15:4700 , piece_id=AC62U7GFNGWMPNEFW5H2C4IMSFT6TSFNH5EHUCX24SWZXCPRKOQQ, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.106020589 (506h25m5.977103624s) [noise=true , address=77.90.31.14:4663 , piece_id=WOQS65FTAS6NTIR7QVGUIFPIJY6POSNUFZTBGYCDZW67TKWZ2GZQ, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.106328646 (506h25m5.977411681s) [noise=true , address=145.79.227.42:4123 , piece_id=T5UOINXTMKBAACABHEVV5NGEMB52SGKWNSJPJ7Y6UKCQXNOUOJMA, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.106787600 (506h25m5.977870635s) [noise=true , address=45.91.116.6:4974 , piece_id=23YEXTSWEM6GOICKLZM5QVYLA7OK2HE55TIA52YG4YO26WBFK4LA, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.107122217 (506h25m5.978205252s) [noise=true , address=145.79.227.49:4339 , piece_id=NHMAMURTEDXM7SZDKB6OP4NU6SCIIKVLIHTVAOT2DVU7UAIT6V4Q, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.108570165 (506h25m5.97965324s) [noise=true , address=145.79.227.15:4874 , piece_id=7PRWX53PMTA2U7GAUIKHBWJXO6RANYJHXUIUFMG3UNASTUTNFIXQ, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.108781597 (506h25m5.979864633s) [noise=true , address=45.91.116.5:4297 , piece_id=I3IPV24ECC3YV4V3QRQDP2ZZT3JOO2CN6M6BISNPENMQZNSDKVAQ, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.110063228 (506h25m5.981146263s) [noise=true , address=145.79.227.11:4732 , piece_id=5FVE6U6ISMCIMDESG3OQ6SHFBGTWT2PPZZC7XRZAPSNK47HZIBWQ, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.111752484 (506h25m5.982835529s) [noise=true , address=145.79.227.36:4519 , piece_id=FJEF6GROIQF7AWQYSJHUEINR5FTBIO7USLJGFWEEEDRJSGW2JG4Q, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.113340689 (506h25m5.984423734s) [noise=true , address=145.79.227.10:4401 , piece_id=ZI4WZERG34DPXHUOLN6QZ55O6SJPZT5LPXPPUNMGGTYAZCAOTORA, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.114643860 (506h25m5.985726905s) [noise=true , address=45.91.116.13:4153 , piece_id=VVU5B5KNVMOIGGZABQAAI22IQ2CZTWJURAJWQQ2JGY3ABTBPFK3A, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.115852932 (506h25m5.986935967s) [noise=true , address=77.90.31.19:4844 , piece_id=AVEWXCOCGI7U437EYT37M4VY3TH73JESDRWAXRLHQKONPFPLAHTQ, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.116921628 (506h25m5.988004673s) [noise=true , address=145.79.227.39:4015 , piece_id=XD5CWP335WRWKXUP3WKMIO7VOFE5G4RN4EGONIRXB6HI43GOCJFQ, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:07.294144135 (506h25m7.16522718s) [noise=true , address=77.90.31.12:4589 , piece_id=CJJUEAIEWP66VHOKZTYLIZUHN3OET6SAFJHXK73EBDFNB5QFE5RA, seg_pos={0 0}, upload=444374]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 26 17:54:07.828164492 (506h25m7.699247537s) [upload=444374]: Next returned nil segment Dec 26 17:54:07.828181555 (506h25m7.6992646s) [upload=444374]: Waiting for error group managing segments... Dec 26 17:54:08.107862847 (506h25m7.978945892s) [noise=true , address=145.79.227.21:4475 , piece_id=UP7LFRPWG3GFBUFT6Y3TOMII7C2UE5BJEY7VCDKHWPYN7NHOL25A, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.112828612 (506h25m7.983911657s) [noise=true , address=145.79.227.51:4410 , piece_id=NJSE6VNZ2ICW5KYFAVC7TXUKOHPAQSHATMU2WW2XMXJF77VTZLVQ, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.119737338 (506h25m7.990820383s) [noise=true , address=145.79.227.10:4401 , piece_id=ZI4WZERG34DPXHUOLN6QZ55O6SJPZT5LPXPPUNMGGTYAZCAOTORA, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.121777863 (506h25m7.992860908s) [noise=true , address=145.79.227.11:4732 , piece_id=5FVE6U6ISMCIMDESG3OQ6SHFBGTWT2PPZZC7XRZAPSNK47HZIBWQ, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.130879434 (506h25m8.00196248s) [noise=true , address=145.79.227.39:4015 , piece_id=XD5CWP335WRWKXUP3WKMIO7VOFE5G4RN4EGONIRXB6HI43GOCJFQ, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.132848083 (506h25m8.003931118s) [noise=true , address=145.79.227.42:4123 , piece_id=T5UOINXTMKBAACABHEVV5NGEMB52SGKWNSJPJ7Y6UKCQXNOUOJMA, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.135816906 (506h25m8.006899952s) [noise=true , address=145.79.227.49:4339 , piece_id=NHMAMURTEDXM7SZDKB6OP4NU6SCIIKVLIHTVAOT2DVU7UAIT6V4Q, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.139078326 (506h25m8.010161361s) [noise=true , address=145.79.227.15:4874 , piece_id=7PRWX53PMTA2U7GAUIKHBWJXO6RANYJHXUIUFMG3UNASTUTNFIXQ, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.139833234 (506h25m8.010916279s) [noise=true , address=145.79.227.36:4519 , piece_id=FJEF6GROIQF7AWQYSJHUEINR5FTBIO7USLJGFWEEEDRJSGW2JG4Q, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.299232928 (506h25m8.170315963s) [noise=true , address=77.90.31.15:4700 , piece_id=AC62U7GFNGWMPNEFW5H2C4IMSFT6TSFNH5EHUCX24SWZXCPRKOQQ, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.362334064 (506h25m8.233417109s) [noise=true , address=45.91.116.13:4153 , piece_id=VVU5B5KNVMOIGGZABQAAI22IQ2CZTWJURAJWQQ2JGY3ABTBPFK3A, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.462770483 (506h25m8.333853538s) [noise=true , address=77.90.31.19:4844 , piece_id=AVEWXCOCGI7U437EYT37M4VY3TH73JESDRWAXRLHQKONPFPLAHTQ, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.474990001 (506h25m8.346073046s) [noise=true , address=45.91.116.6:4974 , piece_id=23YEXTSWEM6GOICKLZM5QVYLA7OK2HE55TIA52YG4YO26WBFK4LA, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.475014728 (506h25m8.346097793s) [seg_pos={0 0}, upload=444374]: Segment reached optimal threshold of 13 pieces. Dec 26 17:54:08.475260065 (506h25m8.34634311s) [noise=true , address=45.91.116.5:4297 , piece_id=I3IPV24ECC3YV4V3QRQDP2ZZT3JOO2CN6M6BISNPENMQZNSDKVAQ, seg_pos={0 0}, upload=444374]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 26 17:54:08.475266929 (506h25m8.346349964s) [noise=true , address=45.91.116.5:4297 , piece_id=I3IPV24ECC3YV4V3QRQDP2ZZT3JOO2CN6M6BISNPENMQZNSDKVAQ, seg_pos={0 0}, upload=444374]: Long tail cancellation due to optimal threshold Dec 26 17:54:08.475601164 (506h25m8.346684209s) [noise=true , address=77.90.31.14:4663 , piece_id=WOQS65FTAS6NTIR7QVGUIFPIJY6POSNUFZTBGYCDZW67TKWZ2GZQ, seg_pos={0 0}, upload=444374]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: expected piece hash; context canceled; EOF Dec 26 17:54:08.475628697 (506h25m8.346711732s) [noise=true , address=77.90.31.14:4663 , piece_id=WOQS65FTAS6NTIR7QVGUIFPIJY6POSNUFZTBGYCDZW67TKWZ2GZQ, seg_pos={0 0}, upload=444374]: Long tail cancellation due to optimal threshold Dec 26 17:54:08.475658875 (506h25m8.34674192s) [seg_pos={0 0}, upload=444374]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 17:54:08.475744387 (506h25m8.346827422s) [upload=444374]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:54:08.475750088 (506h25m8.346833133s) [upload=444374]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:54:08.767674804 (506h25m8.638757859s) [upload=444374]: Done uploading segments... Dec 26 17:54:08.767701635 (506h25m8.63878468s) [upload=444374]: Upload finished. err: <nil> Duration: 6.649079819s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 26.017312004s trace: https://jaeger.dev.storj.io/trace/665ea65f6e9ccc33 FAIL (2025-12-26 18:00:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== e=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789316979 (506h12m54.660400024s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789326507 (506h12m54.660409552s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789611339 (506h12m54.660694384s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789619275 (506h12m54.66070231s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789914006 (506h12m54.660997041s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789949443 (506h12m54.661032478s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790181566 (506h12m54.661264601s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790295632 (506h12m54.661378668s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790510872 (506h12m54.661593917s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790692980 (506h12m54.661776015s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790787379 (506h12m54.661870414s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790992970 (506h12m54.662076005s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791120053 (506h12m54.662203098s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791275218 (506h12m54.662358253s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791443709 (506h12m54.662526744s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:56.112313145 (506h12m55.98339619s) [upload=444191]: Next returned nil segment Dec 26 17:41:56.112330638 (506h12m55.983413683s) [upload=444191]: Waiting for error group managing segments... Dec 26 17:41:56.485322720 (506h12m56.356405765s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.485637079 (506h12m56.356720124s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.666807539 (506h12m56.537890584s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.678753987 (506h12m56.549837032s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.687788039 (506h12m56.558871074s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.694318023 (506h12m56.565401058s) [noise=true , address=145.79.227.7:4326 , piece_id=DMQ5RPI46X2U7TJTS42ZHQEF32BDJRKJJO5HWOYAPNS6VWVG5B5Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.716581187 (506h12m56.587664232s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.972183007 (506h12m56.843266042s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.011917999 (506h12m56.883001034s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.054168622 (506h12m56.925251667s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.114312777 (506h12m56.985395812s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.499846250 (506h12m57.370929285s) [noise=true , address=145.79.227.34:4919 , piece_id=YRWHW3BJVEGVJRSA6RJB3T7VJQSDTGKEZQKRICFIV2G3T7AN4MIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.612649066 (506h12m57.483732101s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649413733 (506h12m57.520496768s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649440282 (506h12m57.520523328s) [seg_pos={0 0}, upload=444191]: Segment reached optimal threshold of 14 pieces. Dec 26 17:41:57.649674729 (506h12m57.520757774s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G7ty71tPwe7cZSBNK9u74DFUCGomnxDsNvAif4C3SbP7HWwfo): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649684087 (506h12m57.520767142s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649747708 (506h12m57.520830763s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v6zXZuMmyL2MU8qb3q6gg1iwTYvndjzMjVpv8btEpeX9Lnqmu): protocol: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection Dec 26 17:41:57.649785219 (506h12m57.520868274s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649899828 (506h12m57.520982903s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649908775 (506h12m57.52099182s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650074641 (506h12m57.521157696s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.650083287 (506h12m57.521166342s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650132821 (506h12m57.521215866s) [seg_pos={0 0}, upload=444191]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 17:41:57.650220138 (506h12m57.521303183s) [upload=444191]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:41:57.650227171 (506h12m57.521310206s) [upload=444191]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:41:57.958890870 (506h12m57.829973915s) [upload=444191]: Done uploading segments... Dec 26 17:41:57.958911620 (506h12m57.829994665s) [upload=444191]: Upload finished. err: <nil> Duration: 26.01734699s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 40.433925294s trace: https://jaeger.dev.storj.io/trace/665ea65f6e9ccc33 FAIL (2025-12-26 18:00:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== e=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789316979 (506h12m54.660400024s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789326507 (506h12m54.660409552s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789611339 (506h12m54.660694384s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789619275 (506h12m54.66070231s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789914006 (506h12m54.660997041s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789949443 (506h12m54.661032478s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790181566 (506h12m54.661264601s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790295632 (506h12m54.661378668s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790510872 (506h12m54.661593917s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790692980 (506h12m54.661776015s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790787379 (506h12m54.661870414s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790992970 (506h12m54.662076005s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791120053 (506h12m54.662203098s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791275218 (506h12m54.662358253s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791443709 (506h12m54.662526744s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:56.112313145 (506h12m55.98339619s) [upload=444191]: Next returned nil segment Dec 26 17:41:56.112330638 (506h12m55.983413683s) [upload=444191]: Waiting for error group managing segments... Dec 26 17:41:56.485322720 (506h12m56.356405765s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.485637079 (506h12m56.356720124s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.666807539 (506h12m56.537890584s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.678753987 (506h12m56.549837032s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.687788039 (506h12m56.558871074s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.694318023 (506h12m56.565401058s) [noise=true , address=145.79.227.7:4326 , piece_id=DMQ5RPI46X2U7TJTS42ZHQEF32BDJRKJJO5HWOYAPNS6VWVG5B5Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.716581187 (506h12m56.587664232s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.972183007 (506h12m56.843266042s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.011917999 (506h12m56.883001034s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.054168622 (506h12m56.925251667s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.114312777 (506h12m56.985395812s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.499846250 (506h12m57.370929285s) [noise=true , address=145.79.227.34:4919 , piece_id=YRWHW3BJVEGVJRSA6RJB3T7VJQSDTGKEZQKRICFIV2G3T7AN4MIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.612649066 (506h12m57.483732101s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649413733 (506h12m57.520496768s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649440282 (506h12m57.520523328s) [seg_pos={0 0}, upload=444191]: Segment reached optimal threshold of 14 pieces. Dec 26 17:41:57.649674729 (506h12m57.520757774s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G7ty71tPwe7cZSBNK9u74DFUCGomnxDsNvAif4C3SbP7HWwfo): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649684087 (506h12m57.520767142s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649747708 (506h12m57.520830763s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v6zXZuMmyL2MU8qb3q6gg1iwTYvndjzMjVpv8btEpeX9Lnqmu): protocol: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection Dec 26 17:41:57.649785219 (506h12m57.520868274s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649899828 (506h12m57.520982903s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649908775 (506h12m57.52099182s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650074641 (506h12m57.521157696s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.650083287 (506h12m57.521166342s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650132821 (506h12m57.521215866s) [seg_pos={0 0}, upload=444191]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 17:41:57.650220138 (506h12m57.521303183s) [upload=444191]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:41:57.650227171 (506h12m57.521310206s) [upload=444191]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:41:57.958890870 (506h12m57.829973915s) [upload=444191]: Done uploading segments... Dec 26 17:41:57.958911620 (506h12m57.829994665s) [upload=444191]: Upload finished. err: <nil> Duration: 40.433944379s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [...X.X.XX...XX...XX....XX..XX......X.XXX.....XX.X.XXX.......] duration: 6.229446153s trace: https://jaeger.dev.storj.io/trace/66e1eaafd7692f5a FAIL (2025-12-26 17:59:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: VVU5B5KNVMOIGGZABQAAI22IQ2CZTWJURAJWQQ2JGY3ABTBPFK3A) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: AC62U7GFNGWMPNEFW5H2C4IMSFT6TSFNH5EHUCX24SWZXCPRKOQQ) context deadline exceeded ======LOGS====== Dec 26 17:54:05.860888315 (506h25m5.73197135s) [upload=444374]: Starting upload Dec 26 17:54:05.860960913 (506h25m5.732043958s) [upload=444374]: Done starting upload Dec 26 17:54:05.861243932 (506h25m5.732326967s) [upload=444374]: Uploading segments... Dec 26 17:54:05.861348571 (506h25m5.732431616s) [upload=444374]: Got next segment. Inline: false Dec 26 17:54:05.861399419 (506h25m5.732482464s) [upload=444374]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 17:54:05.861404999 (506h25m5.732488034s) [upload=444374]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 17:54:06.104522897 (506h25m5.975605933s) [seg_pos={0 0}, upload=444374]: Begin upload segment... Dec 26 17:54:06.104572262 (506h25m5.975655297s) [seg_pos={0 0}, upload=444374]: Done begin upload segment. Dec 26 17:54:06.104602710 (506h25m5.975685755s) [noise=true , address=77.90.31.12:4589 , piece_id=CJJUEAIEWP66VHOKZTYLIZUHN3OET6SAFJHXK73EBDFNB5QFE5RA, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.104713491 (506h25m5.975796536s) [noise=true , address=145.79.227.21:4475 , piece_id=UP7LFRPWG3GFBUFT6Y3TOMII7C2UE5BJEY7VCDKHWPYN7NHOL25A, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.105352147 (506h25m5.976435192s) [noise=true , address=145.79.227.51:4410 , piece_id=NJSE6VNZ2ICW5KYFAVC7TXUKOHPAQSHATMU2WW2XMXJF77VTZLVQ, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.105704788 (506h25m5.976787823s) [noise=true , address=77.90.31.15:4700 , piece_id=AC62U7GFNGWMPNEFW5H2C4IMSFT6TSFNH5EHUCX24SWZXCPRKOQQ, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.106020589 (506h25m5.977103624s) [noise=true , address=77.90.31.14:4663 , piece_id=WOQS65FTAS6NTIR7QVGUIFPIJY6POSNUFZTBGYCDZW67TKWZ2GZQ, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.106328646 (506h25m5.977411681s) [noise=true , address=145.79.227.42:4123 , piece_id=T5UOINXTMKBAACABHEVV5NGEMB52SGKWNSJPJ7Y6UKCQXNOUOJMA, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.106787600 (506h25m5.977870635s) [noise=true , address=45.91.116.6:4974 , piece_id=23YEXTSWEM6GOICKLZM5QVYLA7OK2HE55TIA52YG4YO26WBFK4LA, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.107122217 (506h25m5.978205252s) [noise=true , address=145.79.227.49:4339 , piece_id=NHMAMURTEDXM7SZDKB6OP4NU6SCIIKVLIHTVAOT2DVU7UAIT6V4Q, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.108570165 (506h25m5.97965324s) [noise=true , address=145.79.227.15:4874 , piece_id=7PRWX53PMTA2U7GAUIKHBWJXO6RANYJHXUIUFMG3UNASTUTNFIXQ, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.108781597 (506h25m5.979864633s) [noise=true , address=45.91.116.5:4297 , piece_id=I3IPV24ECC3YV4V3QRQDP2ZZT3JOO2CN6M6BISNPENMQZNSDKVAQ, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.110063228 (506h25m5.981146263s) [noise=true , address=145.79.227.11:4732 , piece_id=5FVE6U6ISMCIMDESG3OQ6SHFBGTWT2PPZZC7XRZAPSNK47HZIBWQ, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.111752484 (506h25m5.982835529s) [noise=true , address=145.79.227.36:4519 , piece_id=FJEF6GROIQF7AWQYSJHUEINR5FTBIO7USLJGFWEEEDRJSGW2JG4Q, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.113340689 (506h25m5.984423734s) [noise=true , address=145.79.227.10:4401 , piece_id=ZI4WZERG34DPXHUOLN6QZ55O6SJPZT5LPXPPUNMGGTYAZCAOTORA, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.114643860 (506h25m5.985726905s) [noise=true , address=45.91.116.13:4153 , piece_id=VVU5B5KNVMOIGGZABQAAI22IQ2CZTWJURAJWQQ2JGY3ABTBPFK3A, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.115852932 (506h25m5.986935967s) [noise=true , address=77.90.31.19:4844 , piece_id=AVEWXCOCGI7U437EYT37M4VY3TH73JESDRWAXRLHQKONPFPLAHTQ, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.116921628 (506h25m5.988004673s) [noise=true , address=145.79.227.39:4015 , piece_id=XD5CWP335WRWKXUP3WKMIO7VOFE5G4RN4EGONIRXB6HI43GOCJFQ, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:07.294144135 (506h25m7.16522718s) [noise=true , address=77.90.31.12:4589 , piece_id=CJJUEAIEWP66VHOKZTYLIZUHN3OET6SAFJHXK73EBDFNB5QFE5RA, seg_pos={0 0}, upload=444374]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 26 17:54:07.828164492 (506h25m7.699247537s) [upload=444374]: Next returned nil segment Dec 26 17:54:07.828181555 (506h25m7.6992646s) [upload=444374]: Waiting for error group managing segments... Dec 26 17:54:08.107862847 (506h25m7.978945892s) [noise=true , address=145.79.227.21:4475 , piece_id=UP7LFRPWG3GFBUFT6Y3TOMII7C2UE5BJEY7VCDKHWPYN7NHOL25A, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.112828612 (506h25m7.983911657s) [noise=true , address=145.79.227.51:4410 , piece_id=NJSE6VNZ2ICW5KYFAVC7TXUKOHPAQSHATMU2WW2XMXJF77VTZLVQ, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.119737338 (506h25m7.990820383s) [noise=true , address=145.79.227.10:4401 , piece_id=ZI4WZERG34DPXHUOLN6QZ55O6SJPZT5LPXPPUNMGGTYAZCAOTORA, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.121777863 (506h25m7.992860908s) [noise=true , address=145.79.227.11:4732 , piece_id=5FVE6U6ISMCIMDESG3OQ6SHFBGTWT2PPZZC7XRZAPSNK47HZIBWQ, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.130879434 (506h25m8.00196248s) [noise=true , address=145.79.227.39:4015 , piece_id=XD5CWP335WRWKXUP3WKMIO7VOFE5G4RN4EGONIRXB6HI43GOCJFQ, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.132848083 (506h25m8.003931118s) [noise=true , address=145.79.227.42:4123 , piece_id=T5UOINXTMKBAACABHEVV5NGEMB52SGKWNSJPJ7Y6UKCQXNOUOJMA, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.135816906 (506h25m8.006899952s) [noise=true , address=145.79.227.49:4339 , piece_id=NHMAMURTEDXM7SZDKB6OP4NU6SCIIKVLIHTVAOT2DVU7UAIT6V4Q, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.139078326 (506h25m8.010161361s) [noise=true , address=145.79.227.15:4874 , piece_id=7PRWX53PMTA2U7GAUIKHBWJXO6RANYJHXUIUFMG3UNASTUTNFIXQ, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.139833234 (506h25m8.010916279s) [noise=true , address=145.79.227.36:4519 , piece_id=FJEF6GROIQF7AWQYSJHUEINR5FTBIO7USLJGFWEEEDRJSGW2JG4Q, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.299232928 (506h25m8.170315963s) [noise=true , address=77.90.31.15:4700 , piece_id=AC62U7GFNGWMPNEFW5H2C4IMSFT6TSFNH5EHUCX24SWZXCPRKOQQ, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.362334064 (506h25m8.233417109s) [noise=true , address=45.91.116.13:4153 , piece_id=VVU5B5KNVMOIGGZABQAAI22IQ2CZTWJURAJWQQ2JGY3ABTBPFK3A, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.462770483 (506h25m8.333853538s) [noise=true , address=77.90.31.19:4844 , piece_id=AVEWXCOCGI7U437EYT37M4VY3TH73JESDRWAXRLHQKONPFPLAHTQ, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.474990001 (506h25m8.346073046s) [noise=true , address=45.91.116.6:4974 , piece_id=23YEXTSWEM6GOICKLZM5QVYLA7OK2HE55TIA52YG4YO26WBFK4LA, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.475014728 (506h25m8.346097793s) [seg_pos={0 0}, upload=444374]: Segment reached optimal threshold of 13 pieces. Dec 26 17:54:08.475260065 (506h25m8.34634311s) [noise=true , address=45.91.116.5:4297 , piece_id=I3IPV24ECC3YV4V3QRQDP2ZZT3JOO2CN6M6BISNPENMQZNSDKVAQ, seg_pos={0 0}, upload=444374]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 26 17:54:08.475266929 (506h25m8.346349964s) [noise=true , address=45.91.116.5:4297 , piece_id=I3IPV24ECC3YV4V3QRQDP2ZZT3JOO2CN6M6BISNPENMQZNSDKVAQ, seg_pos={0 0}, upload=444374]: Long tail cancellation due to optimal threshold Dec 26 17:54:08.475601164 (506h25m8.346684209s) [noise=true , address=77.90.31.14:4663 , piece_id=WOQS65FTAS6NTIR7QVGUIFPIJY6POSNUFZTBGYCDZW67TKWZ2GZQ, seg_pos={0 0}, upload=444374]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: expected piece hash; context canceled; EOF Dec 26 17:54:08.475628697 (506h25m8.346711732s) [noise=true , address=77.90.31.14:4663 , piece_id=WOQS65FTAS6NTIR7QVGUIFPIJY6POSNUFZTBGYCDZW67TKWZ2GZQ, seg_pos={0 0}, upload=444374]: Long tail cancellation due to optimal threshold Dec 26 17:54:08.475658875 (506h25m8.34674192s) [seg_pos={0 0}, upload=444374]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 17:54:08.475744387 (506h25m8.346827422s) [upload=444374]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:54:08.475750088 (506h25m8.346833133s) [upload=444374]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:54:08.767674804 (506h25m8.638757859s) [upload=444374]: Done uploading segments... Dec 26 17:54:08.767701635 (506h25m8.63878468s) [upload=444374]: Upload finished. err: <nil> Duration: 6.229477613s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)