Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [X..X....X.X.XX...XX...XX....XX..XX......X.XXX.....XX.X.XXX..] duration: 5.77130468s trace: https://jaeger.dev.storj.io/trace/66e1eaafd7692f5a FAIL (2025-12-26 17:55:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: VVU5B5KNVMOIGGZABQAAI22IQ2CZTWJURAJWQQ2JGY3ABTBPFK3A) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: AC62U7GFNGWMPNEFW5H2C4IMSFT6TSFNH5EHUCX24SWZXCPRKOQQ) context deadline exceeded ======LOGS====== Dec 26 17:54:05.860888315 (506h25m5.73197135s) [upload=444374]: Starting upload Dec 26 17:54:05.860960913 (506h25m5.732043958s) [upload=444374]: Done starting upload Dec 26 17:54:05.861243932 (506h25m5.732326967s) [upload=444374]: Uploading segments... Dec 26 17:54:05.861348571 (506h25m5.732431616s) [upload=444374]: Got next segment. Inline: false Dec 26 17:54:05.861399419 (506h25m5.732482464s) [upload=444374]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 17:54:05.861404999 (506h25m5.732488034s) [upload=444374]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 17:54:06.104522897 (506h25m5.975605933s) [seg_pos={0 0}, upload=444374]: Begin upload segment... Dec 26 17:54:06.104572262 (506h25m5.975655297s) [seg_pos={0 0}, upload=444374]: Done begin upload segment. Dec 26 17:54:06.104602710 (506h25m5.975685755s) [noise=true , address=77.90.31.12:4589 , piece_id=CJJUEAIEWP66VHOKZTYLIZUHN3OET6SAFJHXK73EBDFNB5QFE5RA, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.104713491 (506h25m5.975796536s) [noise=true , address=145.79.227.21:4475 , piece_id=UP7LFRPWG3GFBUFT6Y3TOMII7C2UE5BJEY7VCDKHWPYN7NHOL25A, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.105352147 (506h25m5.976435192s) [noise=true , address=145.79.227.51:4410 , piece_id=NJSE6VNZ2ICW5KYFAVC7TXUKOHPAQSHATMU2WW2XMXJF77VTZLVQ, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.105704788 (506h25m5.976787823s) [noise=true , address=77.90.31.15:4700 , piece_id=AC62U7GFNGWMPNEFW5H2C4IMSFT6TSFNH5EHUCX24SWZXCPRKOQQ, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.106020589 (506h25m5.977103624s) [noise=true , address=77.90.31.14:4663 , piece_id=WOQS65FTAS6NTIR7QVGUIFPIJY6POSNUFZTBGYCDZW67TKWZ2GZQ, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.106328646 (506h25m5.977411681s) [noise=true , address=145.79.227.42:4123 , piece_id=T5UOINXTMKBAACABHEVV5NGEMB52SGKWNSJPJ7Y6UKCQXNOUOJMA, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.106787600 (506h25m5.977870635s) [noise=true , address=45.91.116.6:4974 , piece_id=23YEXTSWEM6GOICKLZM5QVYLA7OK2HE55TIA52YG4YO26WBFK4LA, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.107122217 (506h25m5.978205252s) [noise=true , address=145.79.227.49:4339 , piece_id=NHMAMURTEDXM7SZDKB6OP4NU6SCIIKVLIHTVAOT2DVU7UAIT6V4Q, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.108570165 (506h25m5.97965324s) [noise=true , address=145.79.227.15:4874 , piece_id=7PRWX53PMTA2U7GAUIKHBWJXO6RANYJHXUIUFMG3UNASTUTNFIXQ, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.108781597 (506h25m5.979864633s) [noise=true , address=45.91.116.5:4297 , piece_id=I3IPV24ECC3YV4V3QRQDP2ZZT3JOO2CN6M6BISNPENMQZNSDKVAQ, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.110063228 (506h25m5.981146263s) [noise=true , address=145.79.227.11:4732 , piece_id=5FVE6U6ISMCIMDESG3OQ6SHFBGTWT2PPZZC7XRZAPSNK47HZIBWQ, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.111752484 (506h25m5.982835529s) [noise=true , address=145.79.227.36:4519 , piece_id=FJEF6GROIQF7AWQYSJHUEINR5FTBIO7USLJGFWEEEDRJSGW2JG4Q, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.113340689 (506h25m5.984423734s) [noise=true , address=145.79.227.10:4401 , piece_id=ZI4WZERG34DPXHUOLN6QZ55O6SJPZT5LPXPPUNMGGTYAZCAOTORA, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.114643860 (506h25m5.985726905s) [noise=true , address=45.91.116.13:4153 , piece_id=VVU5B5KNVMOIGGZABQAAI22IQ2CZTWJURAJWQQ2JGY3ABTBPFK3A, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.115852932 (506h25m5.986935967s) [noise=true , address=77.90.31.19:4844 , piece_id=AVEWXCOCGI7U437EYT37M4VY3TH73JESDRWAXRLHQKONPFPLAHTQ, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.116921628 (506h25m5.988004673s) [noise=true , address=145.79.227.39:4015 , piece_id=XD5CWP335WRWKXUP3WKMIO7VOFE5G4RN4EGONIRXB6HI43GOCJFQ, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:07.294144135 (506h25m7.16522718s) [noise=true , address=77.90.31.12:4589 , piece_id=CJJUEAIEWP66VHOKZTYLIZUHN3OET6SAFJHXK73EBDFNB5QFE5RA, seg_pos={0 0}, upload=444374]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 26 17:54:07.828164492 (506h25m7.699247537s) [upload=444374]: Next returned nil segment Dec 26 17:54:07.828181555 (506h25m7.6992646s) [upload=444374]: Waiting for error group managing segments... Dec 26 17:54:08.107862847 (506h25m7.978945892s) [noise=true , address=145.79.227.21:4475 , piece_id=UP7LFRPWG3GFBUFT6Y3TOMII7C2UE5BJEY7VCDKHWPYN7NHOL25A, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.112828612 (506h25m7.983911657s) [noise=true , address=145.79.227.51:4410 , piece_id=NJSE6VNZ2ICW5KYFAVC7TXUKOHPAQSHATMU2WW2XMXJF77VTZLVQ, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.119737338 (506h25m7.990820383s) [noise=true , address=145.79.227.10:4401 , piece_id=ZI4WZERG34DPXHUOLN6QZ55O6SJPZT5LPXPPUNMGGTYAZCAOTORA, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.121777863 (506h25m7.992860908s) [noise=true , address=145.79.227.11:4732 , piece_id=5FVE6U6ISMCIMDESG3OQ6SHFBGTWT2PPZZC7XRZAPSNK47HZIBWQ, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.130879434 (506h25m8.00196248s) [noise=true , address=145.79.227.39:4015 , piece_id=XD5CWP335WRWKXUP3WKMIO7VOFE5G4RN4EGONIRXB6HI43GOCJFQ, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.132848083 (506h25m8.003931118s) [noise=true , address=145.79.227.42:4123 , piece_id=T5UOINXTMKBAACABHEVV5NGEMB52SGKWNSJPJ7Y6UKCQXNOUOJMA, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.135816906 (506h25m8.006899952s) [noise=true , address=145.79.227.49:4339 , piece_id=NHMAMURTEDXM7SZDKB6OP4NU6SCIIKVLIHTVAOT2DVU7UAIT6V4Q, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.139078326 (506h25m8.010161361s) [noise=true , address=145.79.227.15:4874 , piece_id=7PRWX53PMTA2U7GAUIKHBWJXO6RANYJHXUIUFMG3UNASTUTNFIXQ, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.139833234 (506h25m8.010916279s) [noise=true , address=145.79.227.36:4519 , piece_id=FJEF6GROIQF7AWQYSJHUEINR5FTBIO7USLJGFWEEEDRJSGW2JG4Q, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.299232928 (506h25m8.170315963s) [noise=true , address=77.90.31.15:4700 , piece_id=AC62U7GFNGWMPNEFW5H2C4IMSFT6TSFNH5EHUCX24SWZXCPRKOQQ, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.362334064 (506h25m8.233417109s) [noise=true , address=45.91.116.13:4153 , piece_id=VVU5B5KNVMOIGGZABQAAI22IQ2CZTWJURAJWQQ2JGY3ABTBPFK3A, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.462770483 (506h25m8.333853538s) [noise=true , address=77.90.31.19:4844 , piece_id=AVEWXCOCGI7U437EYT37M4VY3TH73JESDRWAXRLHQKONPFPLAHTQ, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.474990001 (506h25m8.346073046s) [noise=true , address=45.91.116.6:4974 , piece_id=23YEXTSWEM6GOICKLZM5QVYLA7OK2HE55TIA52YG4YO26WBFK4LA, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.475014728 (506h25m8.346097793s) [seg_pos={0 0}, upload=444374]: Segment reached optimal threshold of 13 pieces. Dec 26 17:54:08.475260065 (506h25m8.34634311s) [noise=true , address=45.91.116.5:4297 , piece_id=I3IPV24ECC3YV4V3QRQDP2ZZT3JOO2CN6M6BISNPENMQZNSDKVAQ, seg_pos={0 0}, upload=444374]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 26 17:54:08.475266929 (506h25m8.346349964s) [noise=true , address=45.91.116.5:4297 , piece_id=I3IPV24ECC3YV4V3QRQDP2ZZT3JOO2CN6M6BISNPENMQZNSDKVAQ, seg_pos={0 0}, upload=444374]: Long tail cancellation due to optimal threshold Dec 26 17:54:08.475601164 (506h25m8.346684209s) [noise=true , address=77.90.31.14:4663 , piece_id=WOQS65FTAS6NTIR7QVGUIFPIJY6POSNUFZTBGYCDZW67TKWZ2GZQ, seg_pos={0 0}, upload=444374]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: expected piece hash; context canceled; EOF Dec 26 17:54:08.475628697 (506h25m8.346711732s) [noise=true , address=77.90.31.14:4663 , piece_id=WOQS65FTAS6NTIR7QVGUIFPIJY6POSNUFZTBGYCDZW67TKWZ2GZQ, seg_pos={0 0}, upload=444374]: Long tail cancellation due to optimal threshold Dec 26 17:54:08.475658875 (506h25m8.34674192s) [seg_pos={0 0}, upload=444374]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 17:54:08.475744387 (506h25m8.346827422s) [upload=444374]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:54:08.475750088 (506h25m8.346833133s) [upload=444374]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:54:08.767674804 (506h25m8.638757859s) [upload=444374]: Done uploading segments... Dec 26 17:54:08.767701635 (506h25m8.63878468s) [upload=444374]: Upload finished. err: <nil> Duration: 5.771331303s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 27.366989456s trace: https://jaeger.dev.storj.io/trace/665ea65f6e9ccc33 FAIL (2025-12-26 17:55:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== e=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789316979 (506h12m54.660400024s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789326507 (506h12m54.660409552s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789611339 (506h12m54.660694384s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789619275 (506h12m54.66070231s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789914006 (506h12m54.660997041s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789949443 (506h12m54.661032478s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790181566 (506h12m54.661264601s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790295632 (506h12m54.661378668s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790510872 (506h12m54.661593917s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790692980 (506h12m54.661776015s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790787379 (506h12m54.661870414s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790992970 (506h12m54.662076005s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791120053 (506h12m54.662203098s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791275218 (506h12m54.662358253s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791443709 (506h12m54.662526744s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:56.112313145 (506h12m55.98339619s) [upload=444191]: Next returned nil segment Dec 26 17:41:56.112330638 (506h12m55.983413683s) [upload=444191]: Waiting for error group managing segments... Dec 26 17:41:56.485322720 (506h12m56.356405765s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.485637079 (506h12m56.356720124s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.666807539 (506h12m56.537890584s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.678753987 (506h12m56.549837032s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.687788039 (506h12m56.558871074s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.694318023 (506h12m56.565401058s) [noise=true , address=145.79.227.7:4326 , piece_id=DMQ5RPI46X2U7TJTS42ZHQEF32BDJRKJJO5HWOYAPNS6VWVG5B5Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.716581187 (506h12m56.587664232s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.972183007 (506h12m56.843266042s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.011917999 (506h12m56.883001034s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.054168622 (506h12m56.925251667s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.114312777 (506h12m56.985395812s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.499846250 (506h12m57.370929285s) [noise=true , address=145.79.227.34:4919 , piece_id=YRWHW3BJVEGVJRSA6RJB3T7VJQSDTGKEZQKRICFIV2G3T7AN4MIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.612649066 (506h12m57.483732101s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649413733 (506h12m57.520496768s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649440282 (506h12m57.520523328s) [seg_pos={0 0}, upload=444191]: Segment reached optimal threshold of 14 pieces. Dec 26 17:41:57.649674729 (506h12m57.520757774s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G7ty71tPwe7cZSBNK9u74DFUCGomnxDsNvAif4C3SbP7HWwfo): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649684087 (506h12m57.520767142s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649747708 (506h12m57.520830763s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v6zXZuMmyL2MU8qb3q6gg1iwTYvndjzMjVpv8btEpeX9Lnqmu): protocol: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection Dec 26 17:41:57.649785219 (506h12m57.520868274s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649899828 (506h12m57.520982903s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649908775 (506h12m57.52099182s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650074641 (506h12m57.521157696s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.650083287 (506h12m57.521166342s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650132821 (506h12m57.521215866s) [seg_pos={0 0}, upload=444191]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 17:41:57.650220138 (506h12m57.521303183s) [upload=444191]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:41:57.650227171 (506h12m57.521310206s) [upload=444191]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:41:57.958890870 (506h12m57.829973915s) [upload=444191]: Done uploading segments... Dec 26 17:41:57.958911620 (506h12m57.829994665s) [upload=444191]: Upload finished. err: <nil> Duration: 27.367016037s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [XX..X....X.X.XX...XX...XX....XX..XX......X.XXX.....XX.X.XXX.] duration: 6.32380054s trace: https://jaeger.dev.storj.io/trace/66e1eaafd7692f5a FAIL (2025-12-26 17:54:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: VVU5B5KNVMOIGGZABQAAI22IQ2CZTWJURAJWQQ2JGY3ABTBPFK3A) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: AC62U7GFNGWMPNEFW5H2C4IMSFT6TSFNH5EHUCX24SWZXCPRKOQQ) context deadline exceeded ======LOGS====== Dec 26 17:54:05.860888315 (506h25m5.73197135s) [upload=444374]: Starting upload Dec 26 17:54:05.860960913 (506h25m5.732043958s) [upload=444374]: Done starting upload Dec 26 17:54:05.861243932 (506h25m5.732326967s) [upload=444374]: Uploading segments... Dec 26 17:54:05.861348571 (506h25m5.732431616s) [upload=444374]: Got next segment. Inline: false Dec 26 17:54:05.861399419 (506h25m5.732482464s) [upload=444374]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 17:54:05.861404999 (506h25m5.732488034s) [upload=444374]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 17:54:06.104522897 (506h25m5.975605933s) [seg_pos={0 0}, upload=444374]: Begin upload segment... Dec 26 17:54:06.104572262 (506h25m5.975655297s) [seg_pos={0 0}, upload=444374]: Done begin upload segment. Dec 26 17:54:06.104602710 (506h25m5.975685755s) [noise=true , address=77.90.31.12:4589 , piece_id=CJJUEAIEWP66VHOKZTYLIZUHN3OET6SAFJHXK73EBDFNB5QFE5RA, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.104713491 (506h25m5.975796536s) [noise=true , address=145.79.227.21:4475 , piece_id=UP7LFRPWG3GFBUFT6Y3TOMII7C2UE5BJEY7VCDKHWPYN7NHOL25A, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.105352147 (506h25m5.976435192s) [noise=true , address=145.79.227.51:4410 , piece_id=NJSE6VNZ2ICW5KYFAVC7TXUKOHPAQSHATMU2WW2XMXJF77VTZLVQ, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.105704788 (506h25m5.976787823s) [noise=true , address=77.90.31.15:4700 , piece_id=AC62U7GFNGWMPNEFW5H2C4IMSFT6TSFNH5EHUCX24SWZXCPRKOQQ, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.106020589 (506h25m5.977103624s) [noise=true , address=77.90.31.14:4663 , piece_id=WOQS65FTAS6NTIR7QVGUIFPIJY6POSNUFZTBGYCDZW67TKWZ2GZQ, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.106328646 (506h25m5.977411681s) [noise=true , address=145.79.227.42:4123 , piece_id=T5UOINXTMKBAACABHEVV5NGEMB52SGKWNSJPJ7Y6UKCQXNOUOJMA, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.106787600 (506h25m5.977870635s) [noise=true , address=45.91.116.6:4974 , piece_id=23YEXTSWEM6GOICKLZM5QVYLA7OK2HE55TIA52YG4YO26WBFK4LA, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.107122217 (506h25m5.978205252s) [noise=true , address=145.79.227.49:4339 , piece_id=NHMAMURTEDXM7SZDKB6OP4NU6SCIIKVLIHTVAOT2DVU7UAIT6V4Q, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.108570165 (506h25m5.97965324s) [noise=true , address=145.79.227.15:4874 , piece_id=7PRWX53PMTA2U7GAUIKHBWJXO6RANYJHXUIUFMG3UNASTUTNFIXQ, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.108781597 (506h25m5.979864633s) [noise=true , address=45.91.116.5:4297 , piece_id=I3IPV24ECC3YV4V3QRQDP2ZZT3JOO2CN6M6BISNPENMQZNSDKVAQ, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.110063228 (506h25m5.981146263s) [noise=true , address=145.79.227.11:4732 , piece_id=5FVE6U6ISMCIMDESG3OQ6SHFBGTWT2PPZZC7XRZAPSNK47HZIBWQ, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.111752484 (506h25m5.982835529s) [noise=true , address=145.79.227.36:4519 , piece_id=FJEF6GROIQF7AWQYSJHUEINR5FTBIO7USLJGFWEEEDRJSGW2JG4Q, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.113340689 (506h25m5.984423734s) [noise=true , address=145.79.227.10:4401 , piece_id=ZI4WZERG34DPXHUOLN6QZ55O6SJPZT5LPXPPUNMGGTYAZCAOTORA, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.114643860 (506h25m5.985726905s) [noise=true , address=45.91.116.13:4153 , piece_id=VVU5B5KNVMOIGGZABQAAI22IQ2CZTWJURAJWQQ2JGY3ABTBPFK3A, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.115852932 (506h25m5.986935967s) [noise=true , address=77.90.31.19:4844 , piece_id=AVEWXCOCGI7U437EYT37M4VY3TH73JESDRWAXRLHQKONPFPLAHTQ, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.116921628 (506h25m5.988004673s) [noise=true , address=145.79.227.39:4015 , piece_id=XD5CWP335WRWKXUP3WKMIO7VOFE5G4RN4EGONIRXB6HI43GOCJFQ, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:07.294144135 (506h25m7.16522718s) [noise=true , address=77.90.31.12:4589 , piece_id=CJJUEAIEWP66VHOKZTYLIZUHN3OET6SAFJHXK73EBDFNB5QFE5RA, seg_pos={0 0}, upload=444374]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 26 17:54:07.828164492 (506h25m7.699247537s) [upload=444374]: Next returned nil segment Dec 26 17:54:07.828181555 (506h25m7.6992646s) [upload=444374]: Waiting for error group managing segments... Dec 26 17:54:08.107862847 (506h25m7.978945892s) [noise=true , address=145.79.227.21:4475 , piece_id=UP7LFRPWG3GFBUFT6Y3TOMII7C2UE5BJEY7VCDKHWPYN7NHOL25A, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.112828612 (506h25m7.983911657s) [noise=true , address=145.79.227.51:4410 , piece_id=NJSE6VNZ2ICW5KYFAVC7TXUKOHPAQSHATMU2WW2XMXJF77VTZLVQ, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.119737338 (506h25m7.990820383s) [noise=true , address=145.79.227.10:4401 , piece_id=ZI4WZERG34DPXHUOLN6QZ55O6SJPZT5LPXPPUNMGGTYAZCAOTORA, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.121777863 (506h25m7.992860908s) [noise=true , address=145.79.227.11:4732 , piece_id=5FVE6U6ISMCIMDESG3OQ6SHFBGTWT2PPZZC7XRZAPSNK47HZIBWQ, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.130879434 (506h25m8.00196248s) [noise=true , address=145.79.227.39:4015 , piece_id=XD5CWP335WRWKXUP3WKMIO7VOFE5G4RN4EGONIRXB6HI43GOCJFQ, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.132848083 (506h25m8.003931118s) [noise=true , address=145.79.227.42:4123 , piece_id=T5UOINXTMKBAACABHEVV5NGEMB52SGKWNSJPJ7Y6UKCQXNOUOJMA, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.135816906 (506h25m8.006899952s) [noise=true , address=145.79.227.49:4339 , piece_id=NHMAMURTEDXM7SZDKB6OP4NU6SCIIKVLIHTVAOT2DVU7UAIT6V4Q, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.139078326 (506h25m8.010161361s) [noise=true , address=145.79.227.15:4874 , piece_id=7PRWX53PMTA2U7GAUIKHBWJXO6RANYJHXUIUFMG3UNASTUTNFIXQ, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.139833234 (506h25m8.010916279s) [noise=true , address=145.79.227.36:4519 , piece_id=FJEF6GROIQF7AWQYSJHUEINR5FTBIO7USLJGFWEEEDRJSGW2JG4Q, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.299232928 (506h25m8.170315963s) [noise=true , address=77.90.31.15:4700 , piece_id=AC62U7GFNGWMPNEFW5H2C4IMSFT6TSFNH5EHUCX24SWZXCPRKOQQ, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.362334064 (506h25m8.233417109s) [noise=true , address=45.91.116.13:4153 , piece_id=VVU5B5KNVMOIGGZABQAAI22IQ2CZTWJURAJWQQ2JGY3ABTBPFK3A, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.462770483 (506h25m8.333853538s) [noise=true , address=77.90.31.19:4844 , piece_id=AVEWXCOCGI7U437EYT37M4VY3TH73JESDRWAXRLHQKONPFPLAHTQ, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.474990001 (506h25m8.346073046s) [noise=true , address=45.91.116.6:4974 , piece_id=23YEXTSWEM6GOICKLZM5QVYLA7OK2HE55TIA52YG4YO26WBFK4LA, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.475014728 (506h25m8.346097793s) [seg_pos={0 0}, upload=444374]: Segment reached optimal threshold of 13 pieces. Dec 26 17:54:08.475260065 (506h25m8.34634311s) [noise=true , address=45.91.116.5:4297 , piece_id=I3IPV24ECC3YV4V3QRQDP2ZZT3JOO2CN6M6BISNPENMQZNSDKVAQ, seg_pos={0 0}, upload=444374]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 26 17:54:08.475266929 (506h25m8.346349964s) [noise=true , address=45.91.116.5:4297 , piece_id=I3IPV24ECC3YV4V3QRQDP2ZZT3JOO2CN6M6BISNPENMQZNSDKVAQ, seg_pos={0 0}, upload=444374]: Long tail cancellation due to optimal threshold Dec 26 17:54:08.475601164 (506h25m8.346684209s) [noise=true , address=77.90.31.14:4663 , piece_id=WOQS65FTAS6NTIR7QVGUIFPIJY6POSNUFZTBGYCDZW67TKWZ2GZQ, seg_pos={0 0}, upload=444374]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: expected piece hash; context canceled; EOF Dec 26 17:54:08.475628697 (506h25m8.346711732s) [noise=true , address=77.90.31.14:4663 , piece_id=WOQS65FTAS6NTIR7QVGUIFPIJY6POSNUFZTBGYCDZW67TKWZ2GZQ, seg_pos={0 0}, upload=444374]: Long tail cancellation due to optimal threshold Dec 26 17:54:08.475658875 (506h25m8.34674192s) [seg_pos={0 0}, upload=444374]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 17:54:08.475744387 (506h25m8.346827422s) [upload=444374]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:54:08.475750088 (506h25m8.346833133s) [upload=444374]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:54:08.767674804 (506h25m8.638757859s) [upload=444374]: Done uploading segments... Dec 26 17:54:08.767701635 (506h25m8.63878468s) [upload=444374]: Upload finished. err: <nil> Duration: 6.323831138s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [.XX..X....X.X.XX...XX...XX....XX..XX......X.XXX.....XX.X.XXX] duration: 30.003706616s trace: https://jaeger.dev.storj.io/trace/66e1eaafd7692f5a FAIL (2025-12-26 17:54:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: VVU5B5KNVMOIGGZABQAAI22IQ2CZTWJURAJWQQ2JGY3ABTBPFK3A) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: AC62U7GFNGWMPNEFW5H2C4IMSFT6TSFNH5EHUCX24SWZXCPRKOQQ) context deadline exceeded ======LOGS====== Dec 26 17:54:05.860888315 (506h25m5.73197135s) [upload=444374]: Starting upload Dec 26 17:54:05.860960913 (506h25m5.732043958s) [upload=444374]: Done starting upload Dec 26 17:54:05.861243932 (506h25m5.732326967s) [upload=444374]: Uploading segments... Dec 26 17:54:05.861348571 (506h25m5.732431616s) [upload=444374]: Got next segment. Inline: false Dec 26 17:54:05.861399419 (506h25m5.732482464s) [upload=444374]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 17:54:05.861404999 (506h25m5.732488034s) [upload=444374]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 17:54:06.104522897 (506h25m5.975605933s) [seg_pos={0 0}, upload=444374]: Begin upload segment... Dec 26 17:54:06.104572262 (506h25m5.975655297s) [seg_pos={0 0}, upload=444374]: Done begin upload segment. Dec 26 17:54:06.104602710 (506h25m5.975685755s) [noise=true , address=77.90.31.12:4589 , piece_id=CJJUEAIEWP66VHOKZTYLIZUHN3OET6SAFJHXK73EBDFNB5QFE5RA, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.104713491 (506h25m5.975796536s) [noise=true , address=145.79.227.21:4475 , piece_id=UP7LFRPWG3GFBUFT6Y3TOMII7C2UE5BJEY7VCDKHWPYN7NHOL25A, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.105352147 (506h25m5.976435192s) [noise=true , address=145.79.227.51:4410 , piece_id=NJSE6VNZ2ICW5KYFAVC7TXUKOHPAQSHATMU2WW2XMXJF77VTZLVQ, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.105704788 (506h25m5.976787823s) [noise=true , address=77.90.31.15:4700 , piece_id=AC62U7GFNGWMPNEFW5H2C4IMSFT6TSFNH5EHUCX24SWZXCPRKOQQ, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.106020589 (506h25m5.977103624s) [noise=true , address=77.90.31.14:4663 , piece_id=WOQS65FTAS6NTIR7QVGUIFPIJY6POSNUFZTBGYCDZW67TKWZ2GZQ, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.106328646 (506h25m5.977411681s) [noise=true , address=145.79.227.42:4123 , piece_id=T5UOINXTMKBAACABHEVV5NGEMB52SGKWNSJPJ7Y6UKCQXNOUOJMA, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.106787600 (506h25m5.977870635s) [noise=true , address=45.91.116.6:4974 , piece_id=23YEXTSWEM6GOICKLZM5QVYLA7OK2HE55TIA52YG4YO26WBFK4LA, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.107122217 (506h25m5.978205252s) [noise=true , address=145.79.227.49:4339 , piece_id=NHMAMURTEDXM7SZDKB6OP4NU6SCIIKVLIHTVAOT2DVU7UAIT6V4Q, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.108570165 (506h25m5.97965324s) [noise=true , address=145.79.227.15:4874 , piece_id=7PRWX53PMTA2U7GAUIKHBWJXO6RANYJHXUIUFMG3UNASTUTNFIXQ, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.108781597 (506h25m5.979864633s) [noise=true , address=45.91.116.5:4297 , piece_id=I3IPV24ECC3YV4V3QRQDP2ZZT3JOO2CN6M6BISNPENMQZNSDKVAQ, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.110063228 (506h25m5.981146263s) [noise=true , address=145.79.227.11:4732 , piece_id=5FVE6U6ISMCIMDESG3OQ6SHFBGTWT2PPZZC7XRZAPSNK47HZIBWQ, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.111752484 (506h25m5.982835529s) [noise=true , address=145.79.227.36:4519 , piece_id=FJEF6GROIQF7AWQYSJHUEINR5FTBIO7USLJGFWEEEDRJSGW2JG4Q, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.113340689 (506h25m5.984423734s) [noise=true , address=145.79.227.10:4401 , piece_id=ZI4WZERG34DPXHUOLN6QZ55O6SJPZT5LPXPPUNMGGTYAZCAOTORA, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.114643860 (506h25m5.985726905s) [noise=true , address=45.91.116.13:4153 , piece_id=VVU5B5KNVMOIGGZABQAAI22IQ2CZTWJURAJWQQ2JGY3ABTBPFK3A, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.115852932 (506h25m5.986935967s) [noise=true , address=77.90.31.19:4844 , piece_id=AVEWXCOCGI7U437EYT37M4VY3TH73JESDRWAXRLHQKONPFPLAHTQ, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.116921628 (506h25m5.988004673s) [noise=true , address=145.79.227.39:4015 , piece_id=XD5CWP335WRWKXUP3WKMIO7VOFE5G4RN4EGONIRXB6HI43GOCJFQ, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:07.294144135 (506h25m7.16522718s) [noise=true , address=77.90.31.12:4589 , piece_id=CJJUEAIEWP66VHOKZTYLIZUHN3OET6SAFJHXK73EBDFNB5QFE5RA, seg_pos={0 0}, upload=444374]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 26 17:54:07.828164492 (506h25m7.699247537s) [upload=444374]: Next returned nil segment Dec 26 17:54:07.828181555 (506h25m7.6992646s) [upload=444374]: Waiting for error group managing segments... Dec 26 17:54:08.107862847 (506h25m7.978945892s) [noise=true , address=145.79.227.21:4475 , piece_id=UP7LFRPWG3GFBUFT6Y3TOMII7C2UE5BJEY7VCDKHWPYN7NHOL25A, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.112828612 (506h25m7.983911657s) [noise=true , address=145.79.227.51:4410 , piece_id=NJSE6VNZ2ICW5KYFAVC7TXUKOHPAQSHATMU2WW2XMXJF77VTZLVQ, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.119737338 (506h25m7.990820383s) [noise=true , address=145.79.227.10:4401 , piece_id=ZI4WZERG34DPXHUOLN6QZ55O6SJPZT5LPXPPUNMGGTYAZCAOTORA, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.121777863 (506h25m7.992860908s) [noise=true , address=145.79.227.11:4732 , piece_id=5FVE6U6ISMCIMDESG3OQ6SHFBGTWT2PPZZC7XRZAPSNK47HZIBWQ, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.130879434 (506h25m8.00196248s) [noise=true , address=145.79.227.39:4015 , piece_id=XD5CWP335WRWKXUP3WKMIO7VOFE5G4RN4EGONIRXB6HI43GOCJFQ, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.132848083 (506h25m8.003931118s) [noise=true , address=145.79.227.42:4123 , piece_id=T5UOINXTMKBAACABHEVV5NGEMB52SGKWNSJPJ7Y6UKCQXNOUOJMA, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.135816906 (506h25m8.006899952s) [noise=true , address=145.79.227.49:4339 , piece_id=NHMAMURTEDXM7SZDKB6OP4NU6SCIIKVLIHTVAOT2DVU7UAIT6V4Q, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.139078326 (506h25m8.010161361s) [noise=true , address=145.79.227.15:4874 , piece_id=7PRWX53PMTA2U7GAUIKHBWJXO6RANYJHXUIUFMG3UNASTUTNFIXQ, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.139833234 (506h25m8.010916279s) [noise=true , address=145.79.227.36:4519 , piece_id=FJEF6GROIQF7AWQYSJHUEINR5FTBIO7USLJGFWEEEDRJSGW2JG4Q, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.299232928 (506h25m8.170315963s) [noise=true , address=77.90.31.15:4700 , piece_id=AC62U7GFNGWMPNEFW5H2C4IMSFT6TSFNH5EHUCX24SWZXCPRKOQQ, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.362334064 (506h25m8.233417109s) [noise=true , address=45.91.116.13:4153 , piece_id=VVU5B5KNVMOIGGZABQAAI22IQ2CZTWJURAJWQQ2JGY3ABTBPFK3A, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.462770483 (506h25m8.333853538s) [noise=true , address=77.90.31.19:4844 , piece_id=AVEWXCOCGI7U437EYT37M4VY3TH73JESDRWAXRLHQKONPFPLAHTQ, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.474990001 (506h25m8.346073046s) [noise=true , address=45.91.116.6:4974 , piece_id=23YEXTSWEM6GOICKLZM5QVYLA7OK2HE55TIA52YG4YO26WBFK4LA, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.475014728 (506h25m8.346097793s) [seg_pos={0 0}, upload=444374]: Segment reached optimal threshold of 13 pieces. Dec 26 17:54:08.475260065 (506h25m8.34634311s) [noise=true , address=45.91.116.5:4297 , piece_id=I3IPV24ECC3YV4V3QRQDP2ZZT3JOO2CN6M6BISNPENMQZNSDKVAQ, seg_pos={0 0}, upload=444374]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 26 17:54:08.475266929 (506h25m8.346349964s) [noise=true , address=45.91.116.5:4297 , piece_id=I3IPV24ECC3YV4V3QRQDP2ZZT3JOO2CN6M6BISNPENMQZNSDKVAQ, seg_pos={0 0}, upload=444374]: Long tail cancellation due to optimal threshold Dec 26 17:54:08.475601164 (506h25m8.346684209s) [noise=true , address=77.90.31.14:4663 , piece_id=WOQS65FTAS6NTIR7QVGUIFPIJY6POSNUFZTBGYCDZW67TKWZ2GZQ, seg_pos={0 0}, upload=444374]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: expected piece hash; context canceled; EOF Dec 26 17:54:08.475628697 (506h25m8.346711732s) [noise=true , address=77.90.31.14:4663 , piece_id=WOQS65FTAS6NTIR7QVGUIFPIJY6POSNUFZTBGYCDZW67TKWZ2GZQ, seg_pos={0 0}, upload=444374]: Long tail cancellation due to optimal threshold Dec 26 17:54:08.475658875 (506h25m8.34674192s) [seg_pos={0 0}, upload=444374]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 17:54:08.475744387 (506h25m8.346827422s) [upload=444374]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:54:08.475750088 (506h25m8.346833133s) [upload=444374]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:54:08.767674804 (506h25m8.638757859s) [upload=444374]: Done uploading segments... Dec 26 17:54:08.767701635 (506h25m8.63878468s) [upload=444374]: Upload finished. err: <nil> Duration: 30.003862322s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 36.592387464s trace: https://jaeger.dev.storj.io/trace/665ea65f6e9ccc33 FAIL (2025-12-26 17:54:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== e=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789316979 (506h12m54.660400024s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789326507 (506h12m54.660409552s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789611339 (506h12m54.660694384s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789619275 (506h12m54.66070231s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789914006 (506h12m54.660997041s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789949443 (506h12m54.661032478s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790181566 (506h12m54.661264601s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790295632 (506h12m54.661378668s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790510872 (506h12m54.661593917s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790692980 (506h12m54.661776015s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790787379 (506h12m54.661870414s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790992970 (506h12m54.662076005s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791120053 (506h12m54.662203098s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791275218 (506h12m54.662358253s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791443709 (506h12m54.662526744s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:56.112313145 (506h12m55.98339619s) [upload=444191]: Next returned nil segment Dec 26 17:41:56.112330638 (506h12m55.983413683s) [upload=444191]: Waiting for error group managing segments... Dec 26 17:41:56.485322720 (506h12m56.356405765s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.485637079 (506h12m56.356720124s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.666807539 (506h12m56.537890584s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.678753987 (506h12m56.549837032s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.687788039 (506h12m56.558871074s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.694318023 (506h12m56.565401058s) [noise=true , address=145.79.227.7:4326 , piece_id=DMQ5RPI46X2U7TJTS42ZHQEF32BDJRKJJO5HWOYAPNS6VWVG5B5Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.716581187 (506h12m56.587664232s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.972183007 (506h12m56.843266042s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.011917999 (506h12m56.883001034s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.054168622 (506h12m56.925251667s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.114312777 (506h12m56.985395812s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.499846250 (506h12m57.370929285s) [noise=true , address=145.79.227.34:4919 , piece_id=YRWHW3BJVEGVJRSA6RJB3T7VJQSDTGKEZQKRICFIV2G3T7AN4MIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.612649066 (506h12m57.483732101s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649413733 (506h12m57.520496768s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649440282 (506h12m57.520523328s) [seg_pos={0 0}, upload=444191]: Segment reached optimal threshold of 14 pieces. Dec 26 17:41:57.649674729 (506h12m57.520757774s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G7ty71tPwe7cZSBNK9u74DFUCGomnxDsNvAif4C3SbP7HWwfo): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649684087 (506h12m57.520767142s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649747708 (506h12m57.520830763s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v6zXZuMmyL2MU8qb3q6gg1iwTYvndjzMjVpv8btEpeX9Lnqmu): protocol: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection Dec 26 17:41:57.649785219 (506h12m57.520868274s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649899828 (506h12m57.520982903s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649908775 (506h12m57.52099182s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650074641 (506h12m57.521157696s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.650083287 (506h12m57.521166342s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650132821 (506h12m57.521215866s) [seg_pos={0 0}, upload=444191]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 17:41:57.650220138 (506h12m57.521303183s) [upload=444191]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:41:57.650227171 (506h12m57.521310206s) [upload=444191]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:41:57.958890870 (506h12m57.829973915s) [upload=444191]: Done uploading segments... Dec 26 17:41:57.958911620 (506h12m57.829994665s) [upload=444191]: Upload finished. err: <nil> Duration: 36.592416298s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [..XX..X....X.X.XX...XX...XX....XX..XX......X.XXX.....XX.X.XX] duration: 30.003100922s trace: https://jaeger.dev.storj.io/trace/2abebcbf6fa38e21 FAIL (2025-12-26 17:53:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== segments... Dec 26 17:53:24.656416197 (506h24m24.527499242s) [upload=444359]: Got next segment. Inline: false Dec 26 17:53:24.656456674 (506h24m24.527539729s) [upload=444359]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 17:53:24.656463027 (506h24m24.527546072s) [upload=444359]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 17:53:24.901285551 (506h24m24.772368596s) [seg_pos={0 0}, upload=444359]: Begin upload segment... Dec 26 17:53:24.901343923 (506h24m24.772426968s) [seg_pos={0 0}, upload=444359]: Done begin upload segment. Dec 26 17:53:24.901379551 (506h24m24.772462586s) [noise=true , address=45.91.116.7:4009 , piece_id=R2HC5FKC4KK2B246ZENUMQUGXMVFARRQ7644XRZOWYL27LAOS5JA, seg_pos={0 0}, upload=444359]: Uploading piece... Dec 26 17:53:24.901441237 (506h24m24.772524272s) [noise=true , address=145.79.227.32:4836 , piece_id=D2RII4MFOPZAEKQEJ5GFCH3MX6REPVWH36CKUUWVOI4PLLPQHNBA, seg_pos={0 0}, upload=444359]: Uploading piece... Dec 26 17:53:24.901986626 (506h24m24.773069661s) [noise=true , address=145.79.227.4:4185 , piece_id=2RCCJHTRWUZQEK2VWMSK6PM3KA2TG7WRI54I2WAIKHEI7TWK3C7A, seg_pos={0 0}, upload=444359]: Uploading piece... Dec 26 17:53:24.902285636 (506h24m24.773368671s) [noise=true , address=145.79.227.16:4911 , piece_id=ZAYREVMEEG4JCGG2TVYLIDM2DGJXUM2EK2M52RUOF233ELXEQHTQ, seg_pos={0 0}, upload=444359]: Uploading piece... Dec 26 17:53:24.902669958 (506h24m24.773752993s) [noise=true , address=145.79.227.45:4230 , piece_id=H27S2V5TF3GLQKPSO3MPE77MAQV7VD5QW5OIAEAZVTY3YJJEY6PQ, seg_pos={0 0}, upload=444359]: Uploading piece... Dec 26 17:53:24.903002761 (506h24m24.774085806s) [noise=true , address=45.91.116.6:4973 , piece_id=J6Q6VACQLGWWZV7KUUT3C3NZPQGQQUQCB3LQ5J2GZYXM7HCILOIA, seg_pos={0 0}, upload=444359]: Uploading piece... Dec 26 17:53:24.903313272 (506h24m24.774396307s) [noise=true , address=145.79.227.51:4409 , piece_id=VGK7WKNBJZMTGUFKEJJWQUMPAPCHWAQU6KQKSHVYO7YLYYXG7GTA, seg_pos={0 0}, upload=444359]: Uploading piece... Dec 26 17:53:24.903687525 (506h24m24.77477056s) [noise=true , address=145.79.227.39:4015 , piece_id=F5GEWAIHX5YXBHHM4IBYQURKVVKCMWH7UNKRP3FDWJNRGR3TAKLA, seg_pos={0 0}, upload=444359]: Uploading piece... Dec 26 17:53:24.904125209 (506h24m24.775208244s) [noise=true , address=77.90.31.10:4518 , piece_id=AJK75E2YYNTMDOVO35YDGZAFYKDFYU6XJ72APGPFIX2FVOU5ZUXQ, seg_pos={0 0}, upload=444359]: Uploading piece... Dec 26 17:53:24.905924605 (506h24m24.77700764s) [noise=true , address=145.79.227.29:4943 , piece_id=EUP2HCWNINL6OASYTIDCFBKQ6WIWOWGYK2VVJWDYXUNUG75CNAWQ, seg_pos={0 0}, upload=444359]: Uploading piece... Dec 26 17:53:24.907712139 (506h24m24.778795174s) [noise=true , address=45.91.116.5:4299 , piece_id=K6XBSLACN43NEW5DVXXZFUCJ7DCKF22ZX2DZCHLVGR3WCM4URMOA, seg_pos={0 0}, upload=444359]: Uploading piece... Dec 26 17:53:24.910007821 (506h24m24.781090856s) [noise=true , address=145.79.227.3:4151 , piece_id=SABCK6BFDPCMOIJ3PUR2VHWR4AFY3ZKSQDKQHGDLSR5WBEMYPFKA, seg_pos={0 0}, upload=444359]: Uploading piece... Dec 26 17:53:24.914044075 (506h24m24.78512711s) [noise=true , address=77.90.31.13:4629 , piece_id=YVJPN7JYN3I5PX3T7ATU3FB7TM5PO3PKVSXQUITB7VTFFEYCK3EQ, seg_pos={0 0}, upload=444359]: Uploading piece... Dec 26 17:53:24.916211383 (506h24m24.787294428s) [noise=true , address=145.79.227.50:4375 , piece_id=KXDIJRYERUBJ6AFBF7QQHORGRQGJUV74DEQEXWC3W5KBCOF4MRPQ, seg_pos={0 0}, upload=444359]: Uploading piece... Dec 26 17:53:24.918543624 (506h24m24.789626669s) [noise=true , address=145.79.227.37:4566 , piece_id=OFCCD3RUSOMNNHYSWUEPHK2WJFTBYOAZ4AZE2PESISRNMYCYB6GA, seg_pos={0 0}, upload=444359]: Uploading piece... Dec 26 17:53:24.920895542 (506h24m24.791978587s) [noise=true , address=145.79.227.36:4518 , piece_id=ELG4HYZQXS3FBVZH7WF4EPMHB3RMOVL623N6HHCKC56VITSCLDEQ, seg_pos={0 0}, upload=444359]: Uploading piece... Dec 26 17:53:26.613838321 (506h24m26.484921366s) [upload=444359]: Next returned nil segment Dec 26 17:53:26.613855213 (506h24m26.484938258s) [upload=444359]: Waiting for error group managing segments... Dec 26 17:53:26.902624851 (506h24m26.773707896s) [noise=true , address=145.79.227.45:4230 , piece_id=H27S2V5TF3GLQKPSO3MPE77MAQV7VD5QW5OIAEAZVTY3YJJEY6PQ, seg_pos={0 0}, upload=444359]: Done uploading piece. err: <nil> Dec 26 17:53:26.908198324 (506h24m26.779281369s) [noise=true , address=145.79.227.39:4015 , piece_id=F5GEWAIHX5YXBHHM4IBYQURKVVKCMWH7UNKRP3FDWJNRGR3TAKLA, seg_pos={0 0}, upload=444359]: Done uploading piece. err: <nil> Dec 26 17:53:26.913355935 (506h24m26.784438971s) [noise=true , address=145.79.227.16:4911 , piece_id=ZAYREVMEEG4JCGG2TVYLIDM2DGJXUM2EK2M52RUOF233ELXEQHTQ, seg_pos={0 0}, upload=444359]: Done uploading piece. err: <nil> Dec 26 17:53:26.916699462 (506h24m26.787782507s) [noise=true , address=145.79.227.37:4566 , piece_id=OFCCD3RUSOMNNHYSWUEPHK2WJFTBYOAZ4AZE2PESISRNMYCYB6GA, seg_pos={0 0}, upload=444359]: Done uploading piece. err: <nil> Dec 26 17:53:26.920272716 (506h24m26.791355751s) [noise=true , address=145.79.227.4:4185 , piece_id=2RCCJHTRWUZQEK2VWMSK6PM3KA2TG7WRI54I2WAIKHEI7TWK3C7A, seg_pos={0 0}, upload=444359]: Done uploading piece. err: <nil> Dec 26 17:53:26.924613201 (506h24m26.795696246s) [noise=true , address=145.79.227.36:4518 , piece_id=ELG4HYZQXS3FBVZH7WF4EPMHB3RMOVL623N6HHCKC56VITSCLDEQ, seg_pos={0 0}, upload=444359]: Done uploading piece. err: <nil> Dec 26 17:53:26.925364292 (506h24m26.796447337s) [noise=true , address=145.79.227.32:4836 , piece_id=D2RII4MFOPZAEKQEJ5GFCH3MX6REPVWH36CKUUWVOI4PLLPQHNBA, seg_pos={0 0}, upload=444359]: Done uploading piece. err: <nil> Dec 26 17:53:26.929735014 (506h24m26.800818049s) [noise=true , address=145.79.227.51:4409 , piece_id=VGK7WKNBJZMTGUFKEJJWQUMPAPCHWAQU6KQKSHVYO7YLYYXG7GTA, seg_pos={0 0}, upload=444359]: Done uploading piece. err: <nil> Dec 26 17:53:26.933299241 (506h24m26.804382296s) [noise=true , address=145.79.227.50:4375 , piece_id=KXDIJRYERUBJ6AFBF7QQHORGRQGJUV74DEQEXWC3W5KBCOF4MRPQ, seg_pos={0 0}, upload=444359]: Done uploading piece. err: <nil> Dec 26 17:53:26.938819322 (506h24m26.809902357s) [noise=true , address=145.79.227.3:4151 , piece_id=SABCK6BFDPCMOIJ3PUR2VHWR4AFY3ZKSQDKQHGDLSR5WBEMYPFKA, seg_pos={0 0}, upload=444359]: Done uploading piece. err: <nil> Dec 26 17:53:27.118792242 (506h24m26.989875287s) [noise=true , address=77.90.31.13:4629 , piece_id=YVJPN7JYN3I5PX3T7ATU3FB7TM5PO3PKVSXQUITB7VTFFEYCK3EQ, seg_pos={0 0}, upload=444359]: Done uploading piece. err: <nil> Dec 26 17:53:27.149000683 (506h24m27.020083719s) [noise=true , address=45.91.116.6:4973 , piece_id=J6Q6VACQLGWWZV7KUUT3C3NZPQGQQUQCB3LQ5J2GZYXM7HCILOIA, seg_pos={0 0}, upload=444359]: Done uploading piece. err: <nil> Dec 26 17:53:27.174838293 (506h24m27.045921339s) [noise=true , address=45.91.116.7:4009 , piece_id=R2HC5FKC4KK2B246ZENUMQUGXMVFARRQ7644XRZOWYL27LAOS5JA, seg_pos={0 0}, upload=444359]: Done uploading piece. err: <nil> Dec 26 17:53:27.174893578 (506h24m27.045976623s) [seg_pos={0 0}, upload=444359]: Segment reached optimal threshold of 13 pieces. Dec 26 17:53:27.175136361 (506h24m27.046219406s) [noise=true , address=77.90.31.10:4518 , piece_id=AJK75E2YYNTMDOVO35YDGZAFYKDFYU6XJ72APGPFIX2FVOU5ZUXQ, seg_pos={0 0}, upload=444359]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: write tcp 188.166.225.170:49692->77.90.31.10:4518: use of closed network connection; write tcp 188.166.225.170:49692->77.90.31.10:4518: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49692->77.90.31.10:4518: use of closed network connection Dec 26 17:53:27.175145960 (506h24m27.046229005s) [noise=true , address=77.90.31.10:4518 , piece_id=AJK75E2YYNTMDOVO35YDGZAFYKDFYU6XJ72APGPFIX2FVOU5ZUXQ, seg_pos={0 0}, upload=444359]: Long tail cancellation due to optimal threshold Dec 26 17:53:27.175241230 (506h24m27.046324275s) [noise=true , address=145.79.227.29:4943 , piece_id=EUP2HCWNINL6OASYTIDCFBKQ6WIWOWGYK2VVJWDYXUNUG75CNAWQ, seg_pos={0 0}, upload=444359]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz): protocol: expected piece hash; context canceled; EOF Dec 26 17:53:27.175253544 (506h24m27.046336589s) [noise=true , address=145.79.227.29:4943 , piece_id=EUP2HCWNINL6OASYTIDCFBKQ6WIWOWGYK2VVJWDYXUNUG75CNAWQ, seg_pos={0 0}, upload=444359]: Long tail cancellation due to optimal threshold Dec 26 17:53:27.175297768 (506h24m27.046380813s) [noise=true , address=45.91.116.5:4299 , piece_id=K6XBSLACN43NEW5DVXXZFUCJ7DCKF22ZX2DZCHLVGR3WCM4URMOA, seg_pos={0 0}, upload=444359]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: write tcp 188.166.225.170:45058->45.91.116.5:4299: use of closed network connection; write tcp 188.166.225.170:45058->45.91.116.5:4299: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45058->45.91.116.5:4299: use of closed network connection Dec 26 17:53:27.175306825 (506h24m27.04638987s) [noise=true , address=45.91.116.5:4299 , piece_id=K6XBSLACN43NEW5DVXXZFUCJ7DCKF22ZX2DZCHLVGR3WCM4URMOA, seg_pos={0 0}, upload=444359]: Long tail cancellation due to optimal threshold Dec 26 17:53:27.175407186 (506h24m27.046490221s) [seg_pos={0 0}, upload=444359]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 17:53:27.175499202 (506h24m27.046582247s) [upload=444359]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:53:27.175505534 (506h24m27.046588579s) [upload=444359]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:53:27.472394672 (506h24m27.343477717s) [upload=444359]: Done uploading segments... Dec 26 17:53:27.472418067 (506h24m27.343501112s) [upload=444359]: Upload finished. err: <nil> Duration: 30.00312607s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 29.874032573s trace: https://jaeger.dev.storj.io/trace/665ea65f6e9ccc33 FAIL (2025-12-26 17:53:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== e=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789316979 (506h12m54.660400024s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789326507 (506h12m54.660409552s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789611339 (506h12m54.660694384s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789619275 (506h12m54.66070231s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789914006 (506h12m54.660997041s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789949443 (506h12m54.661032478s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790181566 (506h12m54.661264601s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790295632 (506h12m54.661378668s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790510872 (506h12m54.661593917s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790692980 (506h12m54.661776015s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790787379 (506h12m54.661870414s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790992970 (506h12m54.662076005s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791120053 (506h12m54.662203098s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791275218 (506h12m54.662358253s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791443709 (506h12m54.662526744s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:56.112313145 (506h12m55.98339619s) [upload=444191]: Next returned nil segment Dec 26 17:41:56.112330638 (506h12m55.983413683s) [upload=444191]: Waiting for error group managing segments... Dec 26 17:41:56.485322720 (506h12m56.356405765s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.485637079 (506h12m56.356720124s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.666807539 (506h12m56.537890584s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.678753987 (506h12m56.549837032s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.687788039 (506h12m56.558871074s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.694318023 (506h12m56.565401058s) [noise=true , address=145.79.227.7:4326 , piece_id=DMQ5RPI46X2U7TJTS42ZHQEF32BDJRKJJO5HWOYAPNS6VWVG5B5Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.716581187 (506h12m56.587664232s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.972183007 (506h12m56.843266042s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.011917999 (506h12m56.883001034s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.054168622 (506h12m56.925251667s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.114312777 (506h12m56.985395812s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.499846250 (506h12m57.370929285s) [noise=true , address=145.79.227.34:4919 , piece_id=YRWHW3BJVEGVJRSA6RJB3T7VJQSDTGKEZQKRICFIV2G3T7AN4MIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.612649066 (506h12m57.483732101s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649413733 (506h12m57.520496768s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649440282 (506h12m57.520523328s) [seg_pos={0 0}, upload=444191]: Segment reached optimal threshold of 14 pieces. Dec 26 17:41:57.649674729 (506h12m57.520757774s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G7ty71tPwe7cZSBNK9u74DFUCGomnxDsNvAif4C3SbP7HWwfo): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649684087 (506h12m57.520767142s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649747708 (506h12m57.520830763s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v6zXZuMmyL2MU8qb3q6gg1iwTYvndjzMjVpv8btEpeX9Lnqmu): protocol: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection Dec 26 17:41:57.649785219 (506h12m57.520868274s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649899828 (506h12m57.520982903s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649908775 (506h12m57.52099182s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650074641 (506h12m57.521157696s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.650083287 (506h12m57.521166342s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650132821 (506h12m57.521215866s) [seg_pos={0 0}, upload=444191]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 17:41:57.650220138 (506h12m57.521303183s) [upload=444191]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:41:57.650227171 (506h12m57.521310206s) [upload=444191]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:41:57.958890870 (506h12m57.829973915s) [upload=444191]: Done uploading segments... Dec 26 17:41:57.958911620 (506h12m57.829994665s) [upload=444191]: Upload finished. err: <nil> Duration: 29.874049165s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [...XX..X....X.X.XX...XX...XX....XX..XX......X.XXX.....XX.X.X] duration: 30.002837059s trace: https://jaeger.dev.storj.io/trace/4c3ec4ef955343e5 FAIL (2025-12-26 17:53:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ... Dec 26 17:52:36.958132980 (506h23m36.829216045s) [upload=444347]: Got next segment. Inline: false Dec 26 17:52:36.958200809 (506h23m36.829283854s) [upload=444347]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 17:52:36.958207101 (506h23m36.829290136s) [upload=444347]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 17:52:37.205467347 (506h23m37.076550392s) [seg_pos={0 0}, upload=444347]: Begin upload segment... Dec 26 17:52:37.205541009 (506h23m37.076624044s) [seg_pos={0 0}, upload=444347]: Done begin upload segment. Dec 26 17:52:37.205578229 (506h23m37.076661274s) [noise=true , address=145.79.227.22:4511 , piece_id=CZWOZGJW2MBO637ILPITYTJISIGZSF5GAPA5SWPJSTFJ4I65SDZQ, seg_pos={0 0}, upload=444347]: Uploading piece... Dec 26 17:52:37.205666727 (506h23m37.076749762s) [noise=true , address=145.79.227.14:4839 , piece_id=NEU4CR4NHWJOX6CCIKVDRLYQGTVXIMLX7SAE4FESU4P7UPYEFL5Q, seg_pos={0 0}, upload=444347]: Uploading piece... Dec 26 17:52:37.206173813 (506h23m37.077256848s) [noise=true , address=45.91.116.6:4976 , piece_id=CDAFBV45VWVVV7WGZMIGYVKKTXENYXVODWBKTTOLL7SWHNUNRW3A, seg_pos={0 0}, upload=444347]: Uploading piece... Dec 26 17:52:37.206579515 (506h23m37.07766256s) [noise=true , address=77.90.31.21:4877 , piece_id=V54ODCUPFXSZUMFVXQ42NK4W6J3UWAWKPVGZBP2ATFJGZBTYXQZQ, seg_pos={0 0}, upload=444347]: Uploading piece... Dec 26 17:52:37.206588353 (506h23m37.077671398s) [noise=true , address=145.79.227.37:4566 , piece_id=G3NISFVXPUV42L5QIJVUVLJCRS7T4NKRIZRWRCT4QH6BCI6IW24Q, seg_pos={0 0}, upload=444347]: Uploading piece... Dec 26 17:52:37.206898984 (506h23m37.077982019s) [noise=true , address=145.79.227.11:4732 , piece_id=IFSGOZFLB64ULDCT7J6QVLPIRHE27URV22DQIZDR4FQBXMYSARFQ, seg_pos={0 0}, upload=444347]: Uploading piece... Dec 26 17:52:37.207009805 (506h23m37.07809284s) [noise=true , address=145.79.227.39:4013 , piece_id=SNSGNLGZSZQD6O5OTO2P5JGMGOKIXXUIFDEY7TH6JXBNSUVSW6XA, seg_pos={0 0}, upload=444347]: Uploading piece... Dec 26 17:52:37.207290690 (506h23m37.078373725s) [noise=true , address=145.79.227.10:4401 , piece_id=F6FIC5TD5NN4STBRXRK2UAWTJNHINXUSJ66ANOV5GUZ5SBXPHL7Q, seg_pos={0 0}, upload=444347]: Uploading piece... Dec 26 17:52:37.207350514 (506h23m37.078433549s) [noise=true , address=45.91.116.9:4262 , piece_id=NMN4LMG2QDXPP5LVMS26XVIKDQ6GFYFXJVFKCRL2W5CBO3NECYQA, seg_pos={0 0}, upload=444347]: Uploading piece... Dec 26 17:52:37.207681514 (506h23m37.078764559s) [noise=true , address=45.91.116.13:4155 , piece_id=UNORBCTRBMUSUNF5AO6YXBN7RY3URSONP2DNLPLFV76K5447NUNA, seg_pos={0 0}, upload=444347]: Uploading piece... Dec 26 17:52:37.207824446 (506h23m37.078907491s) [noise=true , address=45.91.116.5:4299 , piece_id=SSAE35TRUWLFCWUO4Y7VR6DOLNWXW4ZEYOEURFATDV6TXLV5KQ5A, seg_pos={0 0}, upload=444347]: Uploading piece... Dec 26 17:52:37.208125339 (506h23m37.079208374s) [noise=true , address=77.90.31.14:4661 , piece_id=S2O7XFHU26A7E5FS77VS7SVKL24NQXX7NKTAEOZVSYQC6JXDZOYQ, seg_pos={0 0}, upload=444347]: Uploading piece... Dec 26 17:52:37.208298830 (506h23m37.079381875s) [noise=true , address=145.79.227.36:4518 , piece_id=JXEVAWLBWPAOWC2BY6PRDUKOF2I3BY3X36DEFV6LTKS6GLVFVRAQ, seg_pos={0 0}, upload=444347]: Uploading piece... Dec 26 17:52:37.208472110 (506h23m37.079555145s) [noise=true , address=145.79.227.50:4374 , piece_id=WTIUMPRW6PDT3O3C5LIZLVN2UCKH574A62FQK56OJKX2DNULGNCQ, seg_pos={0 0}, upload=444347]: Uploading piece... Dec 26 17:52:37.208661981 (506h23m37.079745016s) [noise=true , address=45.91.116.11:4117 , piece_id=V7AAPZPBTFZ627FY7YOGDBGQONAMAXCKQJAFITBTRTLJFXJIOX2Q, seg_pos={0 0}, upload=444347]: Uploading piece... Dec 26 17:52:37.208869466 (506h23m37.079952501s) [noise=true , address=45.91.116.12:4226 , piece_id=5676LLY5CGKKJEYGQN7PW326A5YYK225YT3FZ5TBUQXCKNS3HHOQ, seg_pos={0 0}, upload=444347]: Uploading piece... Dec 26 17:52:38.918139605 (506h23m38.78922264s) [upload=444347]: Next returned nil segment Dec 26 17:52:38.918156828 (506h23m38.789239893s) [upload=444347]: Waiting for error group managing segments... Dec 26 17:52:39.208498188 (506h23m39.079581223s) [noise=true , address=145.79.227.37:4566 , piece_id=G3NISFVXPUV42L5QIJVUVLJCRS7T4NKRIZRWRCT4QH6BCI6IW24Q, seg_pos={0 0}, upload=444347]: Done uploading piece. err: <nil> Dec 26 17:52:39.213152029 (506h23m39.084235064s) [noise=true , address=145.79.227.11:4732 , piece_id=IFSGOZFLB64ULDCT7J6QVLPIRHE27URV22DQIZDR4FQBXMYSARFQ, seg_pos={0 0}, upload=444347]: Done uploading piece. err: <nil> Dec 26 17:52:39.219232758 (506h23m39.090315793s) [noise=true , address=145.79.227.22:4511 , piece_id=CZWOZGJW2MBO637ILPITYTJISIGZSF5GAPA5SWPJSTFJ4I65SDZQ, seg_pos={0 0}, upload=444347]: Done uploading piece. err: <nil> Dec 26 17:52:39.220396134 (506h23m39.091479169s) [noise=true , address=145.79.227.39:4013 , piece_id=SNSGNLGZSZQD6O5OTO2P5JGMGOKIXXUIFDEY7TH6JXBNSUVSW6XA, seg_pos={0 0}, upload=444347]: Done uploading piece. err: <nil> Dec 26 17:52:39.226717931 (506h23m39.097800987s) [noise=true , address=145.79.227.36:4518 , piece_id=JXEVAWLBWPAOWC2BY6PRDUKOF2I3BY3X36DEFV6LTKS6GLVFVRAQ, seg_pos={0 0}, upload=444347]: Done uploading piece. err: <nil> Dec 26 17:52:39.237771881 (506h23m39.108854916s) [noise=true , address=145.79.227.50:4374 , piece_id=WTIUMPRW6PDT3O3C5LIZLVN2UCKH574A62FQK56OJKX2DNULGNCQ, seg_pos={0 0}, upload=444347]: Done uploading piece. err: <nil> Dec 26 17:52:39.240742547 (506h23m39.111825582s) [noise=true , address=145.79.227.10:4401 , piece_id=F6FIC5TD5NN4STBRXRK2UAWTJNHINXUSJ66ANOV5GUZ5SBXPHL7Q, seg_pos={0 0}, upload=444347]: Done uploading piece. err: <nil> Dec 26 17:52:39.403754260 (506h23m39.274837295s) [noise=true , address=77.90.31.21:4877 , piece_id=V54ODCUPFXSZUMFVXQ42NK4W6J3UWAWKPVGZBP2ATFJGZBTYXQZQ, seg_pos={0 0}, upload=444347]: Done uploading piece. err: <nil> Dec 26 17:52:39.415742668 (506h23m39.286825713s) [noise=true , address=77.90.31.14:4661 , piece_id=S2O7XFHU26A7E5FS77VS7SVKL24NQXX7NKTAEOZVSYQC6JXDZOYQ, seg_pos={0 0}, upload=444347]: Done uploading piece. err: <nil> Dec 26 17:52:39.510946703 (506h23m39.382029738s) [noise=true , address=45.91.116.5:4299 , piece_id=SSAE35TRUWLFCWUO4Y7VR6DOLNWXW4ZEYOEURFATDV6TXLV5KQ5A, seg_pos={0 0}, upload=444347]: Done uploading piece. err: <nil> Dec 26 17:52:39.516619004 (506h23m39.387702039s) [noise=true , address=45.91.116.13:4155 , piece_id=UNORBCTRBMUSUNF5AO6YXBN7RY3URSONP2DNLPLFV76K5447NUNA, seg_pos={0 0}, upload=444347]: Done uploading piece. err: <nil> Dec 26 17:52:39.527490505 (506h23m39.39857354s) [noise=true , address=45.91.116.6:4976 , piece_id=CDAFBV45VWVVV7WGZMIGYVKKTXENYXVODWBKTTOLL7SWHNUNRW3A, seg_pos={0 0}, upload=444347]: Done uploading piece. err: <nil> Dec 26 17:52:39.563650190 (506h23m39.434733235s) [noise=true , address=45.91.116.11:4117 , piece_id=V7AAPZPBTFZ627FY7YOGDBGQONAMAXCKQJAFITBTRTLJFXJIOX2Q, seg_pos={0 0}, upload=444347]: Done uploading piece. err: <nil> Dec 26 17:52:39.563675488 (506h23m39.434758523s) [seg_pos={0 0}, upload=444347]: Segment reached optimal threshold of 13 pieces. Dec 26 17:52:39.563875107 (506h23m39.434958153s) [noise=true , address=45.91.116.12:4226 , piece_id=5676LLY5CGKKJEYGQN7PW326A5YYK225YT3FZ5TBUQXCKNS3HHOQ, seg_pos={0 0}, upload=444347]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: write tcp 188.166.225.170:45890->45.91.116.12:4226: use of closed network connection; write tcp 188.166.225.170:45890->45.91.116.12:4226: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45890->45.91.116.12:4226: use of closed network connection Dec 26 17:52:39.563883012 (506h23m39.434966047s) [noise=true , address=45.91.116.12:4226 , piece_id=5676LLY5CGKKJEYGQN7PW326A5YYK225YT3FZ5TBUQXCKNS3HHOQ, seg_pos={0 0}, upload=444347]: Long tail cancellation due to optimal threshold Dec 26 17:52:39.563939470 (506h23m39.435022505s) [noise=true , address=45.91.116.9:4262 , piece_id=NMN4LMG2QDXPP5LVMS26XVIKDQ6GFYFXJVFKCRL2W5CBO3NECYQA, seg_pos={0 0}, upload=444347]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 26 17:52:39.563948437 (506h23m39.435031472s) [noise=true , address=45.91.116.9:4262 , piece_id=NMN4LMG2QDXPP5LVMS26XVIKDQ6GFYFXJVFKCRL2W5CBO3NECYQA, seg_pos={0 0}, upload=444347]: Long tail cancellation due to optimal threshold Dec 26 17:52:39.564134823 (506h23m39.435217868s) [noise=true , address=145.79.227.14:4839 , piece_id=NEU4CR4NHWJOX6CCIKVDRLYQGTVXIMLX7SAE4FESU4P7UPYEFL5Q, seg_pos={0 0}, upload=444347]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv): protocol: write tcp 188.166.225.170:45666->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:45666->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45666->145.79.227.14:4839: use of closed network connection Dec 26 17:52:39.564144300 (506h23m39.435227335s) [noise=true , address=145.79.227.14:4839 , piece_id=NEU4CR4NHWJOX6CCIKVDRLYQGTVXIMLX7SAE4FESU4P7UPYEFL5Q, seg_pos={0 0}, upload=444347]: Long tail cancellation due to optimal threshold Dec 26 17:52:39.564167124 (506h23m39.435250159s) [seg_pos={0 0}, upload=444347]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 17:52:39.564251625 (506h23m39.43533467s) [upload=444347]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:52:39.564259249 (506h23m39.435342294s) [upload=444347]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:52:39.863068274 (506h23m39.734151329s) [upload=444347]: Done uploading segments... Dec 26 17:52:39.863091257 (506h23m39.734174302s) [upload=444347]: Upload finished. err: <nil> Duration: 30.002848742s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 38.360230105s trace: https://jaeger.dev.storj.io/trace/665ea65f6e9ccc33 FAIL (2025-12-26 17:52:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== e=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789316979 (506h12m54.660400024s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789326507 (506h12m54.660409552s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789611339 (506h12m54.660694384s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789619275 (506h12m54.66070231s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789914006 (506h12m54.660997041s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789949443 (506h12m54.661032478s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790181566 (506h12m54.661264601s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790295632 (506h12m54.661378668s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790510872 (506h12m54.661593917s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790692980 (506h12m54.661776015s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790787379 (506h12m54.661870414s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790992970 (506h12m54.662076005s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791120053 (506h12m54.662203098s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791275218 (506h12m54.662358253s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791443709 (506h12m54.662526744s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:56.112313145 (506h12m55.98339619s) [upload=444191]: Next returned nil segment Dec 26 17:41:56.112330638 (506h12m55.983413683s) [upload=444191]: Waiting for error group managing segments... Dec 26 17:41:56.485322720 (506h12m56.356405765s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.485637079 (506h12m56.356720124s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.666807539 (506h12m56.537890584s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.678753987 (506h12m56.549837032s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.687788039 (506h12m56.558871074s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.694318023 (506h12m56.565401058s) [noise=true , address=145.79.227.7:4326 , piece_id=DMQ5RPI46X2U7TJTS42ZHQEF32BDJRKJJO5HWOYAPNS6VWVG5B5Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.716581187 (506h12m56.587664232s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.972183007 (506h12m56.843266042s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.011917999 (506h12m56.883001034s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.054168622 (506h12m56.925251667s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.114312777 (506h12m56.985395812s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.499846250 (506h12m57.370929285s) [noise=true , address=145.79.227.34:4919 , piece_id=YRWHW3BJVEGVJRSA6RJB3T7VJQSDTGKEZQKRICFIV2G3T7AN4MIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.612649066 (506h12m57.483732101s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649413733 (506h12m57.520496768s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649440282 (506h12m57.520523328s) [seg_pos={0 0}, upload=444191]: Segment reached optimal threshold of 14 pieces. Dec 26 17:41:57.649674729 (506h12m57.520757774s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G7ty71tPwe7cZSBNK9u74DFUCGomnxDsNvAif4C3SbP7HWwfo): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649684087 (506h12m57.520767142s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649747708 (506h12m57.520830763s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v6zXZuMmyL2MU8qb3q6gg1iwTYvndjzMjVpv8btEpeX9Lnqmu): protocol: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection Dec 26 17:41:57.649785219 (506h12m57.520868274s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649899828 (506h12m57.520982903s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649908775 (506h12m57.52099182s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650074641 (506h12m57.521157696s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.650083287 (506h12m57.521166342s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650132821 (506h12m57.521215866s) [seg_pos={0 0}, upload=444191]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 17:41:57.650220138 (506h12m57.521303183s) [upload=444191]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:41:57.650227171 (506h12m57.521310206s) [upload=444191]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:41:57.958890870 (506h12m57.829973915s) [upload=444191]: Done uploading segments... Dec 26 17:41:57.958911620 (506h12m57.829994665s) [upload=444191]: Upload finished. err: <nil> Duration: 38.360249262s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [....XX..X....X.X.XX...XX...XX....XX..XX......X.XXX.....XX.X.] duration: 22.460526183s trace: https://jaeger.dev.storj.io/trace/6779a1c1690f0818 FAIL (2025-12-26 17:52:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 17:51:44.005473953 (506h22m43.876556998s) [upload=444333]: Starting upload Dec 26 17:51:44.005550680 (506h22m43.876633725s) [upload=444333]: Done starting upload Dec 26 17:51:44.005629710 (506h22m43.876712745s) [upload=444333]: Uploading segments... Dec 26 17:51:44.005815313 (506h22m43.876898358s) [upload=444333]: Got next segment. Inline: false Dec 26 17:51:44.006334462 (506h22m43.877417497s) [upload=444333]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 17:51:44.006342637 (506h22m43.877425682s) [upload=444333]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 17:51:44.253388747 (506h22m44.124471782s) [seg_pos={0 0}, upload=444333]: Begin upload segment... Dec 26 17:51:44.253449862 (506h22m44.124532897s) [seg_pos={0 0}, upload=444333]: Done begin upload segment. Dec 26 17:51:44.253524976 (506h22m44.124608031s) [noise=true , address=145.79.227.45:4229 , piece_id=AZJFYG26H6TS33VDSJLOZAMLB4IXDHNNCGNEYPVM5GAPI2RX3P6Q, seg_pos={0 0}, upload=444333]: Uploading piece... Dec 26 17:51:44.253945116 (506h22m44.125028151s) [noise=true , address=145.79.227.46:4602 , piece_id=ECGPTTVPWENOLT2KGHKS3LVH5UBIRSOJ7MOQOIVNPFANSUYQCMSA, seg_pos={0 0}, upload=444333]: Uploading piece... Dec 26 17:51:44.254587920 (506h22m44.125670955s) [noise=true , address=45.91.116.8:4045 , piece_id=LEBFSGXN42BETMF7Y2A7BGF4IRTZS5E4M3UQGPXBAKUPYC2EJZZQ, seg_pos={0 0}, upload=444333]: Uploading piece... Dec 26 17:51:44.255123360 (506h22m44.126206405s) [noise=true , address=145.79.227.42:4121 , piece_id=SVSSFSUNTQW7RAQWWTRL5I5GIQVHZZJC72JE67T4MOERTUQIU6VA, seg_pos={0 0}, upload=444333]: Uploading piece... Dec 26 17:51:44.255301148 (506h22m44.126384203s) [noise=true , address=145.79.227.8:4330 , piece_id=VFRBYMLXXLKNWXJ55AS3LY5OJ4NDYX63RO75IIGFNUJSVRUYG4BQ, seg_pos={0 0}, upload=444333]: Uploading piece... Dec 26 17:51:44.255791632 (506h22m44.126874677s) [noise=true , address=77.90.31.17:4770 , piece_id=MV7MNB7IEULOCTO2HP2SXMM3NEYA2OZGPITNVCKNNBAAWBDLGRQA, seg_pos={0 0}, upload=444333]: Uploading piece... Dec 26 17:51:44.256058289 (506h22m44.127141334s) [noise=true , address=77.90.31.14:4663 , piece_id=UPPD3AXAZGG4P5TTY4TEU2O6JHC6LUBJAVHKC74LCWJHUW3MAOBA, seg_pos={0 0}, upload=444333]: Uploading piece... Dec 26 17:51:44.256323264 (506h22m44.127406309s) [noise=true , address=145.79.227.41:4087 , piece_id=TBTU3YCEOW5MBSY3EEN5FOMMAXSVYYT77ZETHVOJIXKVLCNGDQ7Q, seg_pos={0 0}, upload=444333]: Uploading piece... Dec 26 17:51:44.256526151 (506h22m44.127609196s) [noise=true , address=145.79.227.38:4979 , piece_id=3VOAEVN4NVEESMVPIISWOBSCTJU76REM3REKJWSUGLQCYW5D7ZHA, seg_pos={0 0}, upload=444333]: Uploading piece... Dec 26 17:51:44.256901556 (506h22m44.127984601s) [noise=true , address=145.79.227.11:4730 , piece_id=WVALFZ65W45RU7FLO4IOE377B6CKJPZCZULUHQFFBSOMSFTV5VWQ, seg_pos={0 0}, upload=444333]: Uploading piece... Dec 26 17:51:44.256990264 (506h22m44.128073319s) [noise=true , address=145.79.227.15:4874 , piece_id=BW27X34FKIWARBEHQFIYWCKM5MIYG3BY4OTX37SF7RNOKNFVHY2Q, seg_pos={0 0}, upload=444333]: Uploading piece... Dec 26 17:51:44.257506828 (506h22m44.128589873s) [noise=true , address=145.79.227.47:4267 , piece_id=MY3VGFEQNDCFFCGX4EKY5SRDAEP52HDS5EPR3XXAJCAXP34TKQBQ, seg_pos={0 0}, upload=444333]: Uploading piece... Dec 26 17:51:44.257872845 (506h22m44.12895588s) [noise=true , address=45.91.116.14:4190 , piece_id=IZDDTQ6ZDIQINXQGNS3GD6RYVLI5TFDB4TCQT7RPPOHCSGIECNRA, seg_pos={0 0}, upload=444333]: Uploading piece... Dec 26 17:51:44.258130345 (506h22m44.12921339s) [noise=true , address=145.79.227.32:4836 , piece_id=BIDZSYZH7JZ4ARTHIPWDZQFQUPY2QDRQY2FKV7J4DEEJW2S2K3NQ, seg_pos={0 0}, upload=444333]: Uploading piece... Dec 26 17:51:44.258270612 (506h22m44.129353657s) [noise=true , address=77.90.31.18:4805 , piece_id=4BHLLSGBF6B74I6OOICNNFRJMM2WQXDVDSJVPXZGVZBB5IJ2XRQQ, seg_pos={0 0}, upload=444333]: Uploading piece... Dec 26 17:51:44.258961107 (506h22m44.130044152s) [noise=true , address=145.79.227.12:4765 , piece_id=ZT5AWKLZU2V7JS7BXIPAOJQQ3WFJDNIR7TRSD4QMH66KA6AVS44A, seg_pos={0 0}, upload=444333]: Uploading piece... Dec 26 17:51:45.429528151 (506h22m45.300611196s) [noise=true , address=77.90.31.18:4805 , piece_id=4BHLLSGBF6B74I6OOICNNFRJMM2WQXDVDSJVPXZGVZBB5IJ2XRQQ, seg_pos={0 0}, upload=444333]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: storage node overloaded, request limit: 2000 Dec 26 17:51:45.959357869 (506h22m45.830440914s) [upload=444333]: Next returned nil segment Dec 26 17:51:45.959375462 (506h22m45.830458507s) [upload=444333]: Waiting for error group managing segments... Dec 26 17:51:46.247253764 (506h22m46.118336799s) [noise=true , address=145.79.227.46:4602 , piece_id=ECGPTTVPWENOLT2KGHKS3LVH5UBIRSOJ7MOQOIVNPFANSUYQCMSA, seg_pos={0 0}, upload=444333]: Done uploading piece. err: <nil> Dec 26 17:51:46.250896720 (506h22m46.121979755s) [noise=true , address=145.79.227.45:4229 , piece_id=AZJFYG26H6TS33VDSJLOZAMLB4IXDHNNCGNEYPVM5GAPI2RX3P6Q, seg_pos={0 0}, upload=444333]: Done uploading piece. err: <nil> Dec 26 17:51:46.253591341 (506h22m46.124674386s) [noise=true , address=145.79.227.8:4330 , piece_id=VFRBYMLXXLKNWXJ55AS3LY5OJ4NDYX63RO75IIGFNUJSVRUYG4BQ, seg_pos={0 0}, upload=444333]: Done uploading piece. err: <nil> Dec 26 17:51:46.256258740 (506h22m46.127341775s) [noise=true , address=145.79.227.15:4874 , piece_id=BW27X34FKIWARBEHQFIYWCKM5MIYG3BY4OTX37SF7RNOKNFVHY2Q, seg_pos={0 0}, upload=444333]: Done uploading piece. err: <nil> Dec 26 17:51:46.279230472 (506h22m46.150313517s) [noise=true , address=145.79.227.32:4836 , piece_id=BIDZSYZH7JZ4ARTHIPWDZQFQUPY2QDRQY2FKV7J4DEEJW2S2K3NQ, seg_pos={0 0}, upload=444333]: Done uploading piece. err: <nil> Dec 26 17:51:46.280002032 (506h22m46.151085067s) [noise=true , address=145.79.227.38:4979 , piece_id=3VOAEVN4NVEESMVPIISWOBSCTJU76REM3REKJWSUGLQCYW5D7ZHA, seg_pos={0 0}, upload=444333]: Done uploading piece. err: <nil> Dec 26 17:51:46.281328107 (506h22m46.152411142s) [noise=true , address=145.79.227.47:4267 , piece_id=MY3VGFEQNDCFFCGX4EKY5SRDAEP52HDS5EPR3XXAJCAXP34TKQBQ, seg_pos={0 0}, upload=444333]: Done uploading piece. err: <nil> Dec 26 17:51:46.286020221 (506h22m46.157103256s) [noise=true , address=145.79.227.11:4730 , piece_id=WVALFZ65W45RU7FLO4IOE377B6CKJPZCZULUHQFFBSOMSFTV5VWQ, seg_pos={0 0}, upload=444333]: Done uploading piece. err: <nil> Dec 26 17:51:46.288375025 (506h22m46.15945806s) [noise=true , address=145.79.227.42:4121 , piece_id=SVSSFSUNTQW7RAQWWTRL5I5GIQVHZZJC72JE67T4MOERTUQIU6VA, seg_pos={0 0}, upload=444333]: Done uploading piece. err: <nil> Dec 26 17:51:46.588588915 (506h22m46.45967195s) [noise=true , address=77.90.31.14:4663 , piece_id=UPPD3AXAZGG4P5TTY4TEU2O6JHC6LUBJAVHKC74LCWJHUW3MAOBA, seg_pos={0 0}, upload=444333]: Done uploading piece. err: <nil> Dec 26 17:51:46.590231473 (506h22m46.461314518s) [noise=true , address=45.91.116.14:4190 , piece_id=IZDDTQ6ZDIQINXQGNS3GD6RYVLI5TFDB4TCQT7RPPOHCSGIECNRA, seg_pos={0 0}, upload=444333]: Done uploading piece. err: <nil> Dec 26 17:51:46.603359420 (506h22m46.474442465s) [noise=true , address=77.90.31.17:4770 , piece_id=MV7MNB7IEULOCTO2HP2SXMM3NEYA2OZGPITNVCKNNBAAWBDLGRQA, seg_pos={0 0}, upload=444333]: Done uploading piece. err: <nil> Dec 26 17:51:46.606138262 (506h22m46.477221307s) [noise=true , address=45.91.116.8:4045 , piece_id=LEBFSGXN42BETMF7Y2A7BGF4IRTZS5E4M3UQGPXBAKUPYC2EJZZQ, seg_pos={0 0}, upload=444333]: Done uploading piece. err: <nil> Dec 26 17:51:46.606159363 (506h22m46.477242408s) [seg_pos={0 0}, upload=444333]: Segment reached optimal threshold of 13 pieces. Dec 26 17:51:46.606376275 (506h22m46.47745932s) [noise=true , address=145.79.227.41:4087 , piece_id=TBTU3YCEOW5MBSY3EEN5FOMMAXSVYYT77ZETHVOJIXKVLCNGDQ7Q, seg_pos={0 0}, upload=444333]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR): protocol: write tcp 188.166.225.170:59534->145.79.227.41:4087: use of closed network connection; write tcp 188.166.225.170:59534->145.79.227.41:4087: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59534->145.79.227.41:4087: use of closed network connection Dec 26 17:51:46.606383268 (506h22m46.477466304s) [noise=true , address=145.79.227.41:4087 , piece_id=TBTU3YCEOW5MBSY3EEN5FOMMAXSVYYT77ZETHVOJIXKVLCNGDQ7Q, seg_pos={0 0}, upload=444333]: Long tail cancellation due to optimal threshold Dec 26 17:51:46.606386765 (506h22m46.47746981s) [noise=true , address=145.79.227.12:4765 , piece_id=ZT5AWKLZU2V7JS7BXIPAOJQQ3WFJDNIR7TRSD4QMH66KA6AVS44A, seg_pos={0 0}, upload=444333]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV): protocol: write tcp 188.166.225.170:40256->145.79.227.12:4765: use of closed network connection; write tcp 188.166.225.170:40256->145.79.227.12:4765: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40256->145.79.227.12:4765: use of closed network connection Dec 26 17:51:46.606397215 (506h22m46.47748027s) [noise=true , address=145.79.227.12:4765 , piece_id=ZT5AWKLZU2V7JS7BXIPAOJQQ3WFJDNIR7TRSD4QMH66KA6AVS44A, seg_pos={0 0}, upload=444333]: Long tail cancellation due to optimal threshold Dec 26 17:51:46.606452631 (506h22m46.477535676s) [seg_pos={0 0}, upload=444333]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 17:51:46.606720050 (506h22m46.477803095s) [upload=444333]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:51:46.606726051 (506h22m46.477809096s) [upload=444333]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:51:46.895225284 (506h22m46.766308329s) [upload=444333]: Done uploading segments... Dec 26 17:51:46.895243990 (506h22m46.766327025s) [upload=444333]: Upload finished. err: <nil> Duration: 22.460553225s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [.....XX..X....X.X.XX...XX...XX....XX..XX......X.XXX.....XX.X] duration: 30.003077226s trace: https://jaeger.dev.storj.io/trace/6779a1c1690f0818 FAIL (2025-12-26 17:52:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 17:51:44.005473953 (506h22m43.876556998s) [upload=444333]: Starting upload Dec 26 17:51:44.005550680 (506h22m43.876633725s) [upload=444333]: Done starting upload Dec 26 17:51:44.005629710 (506h22m43.876712745s) [upload=444333]: Uploading segments... Dec 26 17:51:44.005815313 (506h22m43.876898358s) [upload=444333]: Got next segment. Inline: false Dec 26 17:51:44.006334462 (506h22m43.877417497s) [upload=444333]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 17:51:44.006342637 (506h22m43.877425682s) [upload=444333]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 17:51:44.253388747 (506h22m44.124471782s) [seg_pos={0 0}, upload=444333]: Begin upload segment... Dec 26 17:51:44.253449862 (506h22m44.124532897s) [seg_pos={0 0}, upload=444333]: Done begin upload segment. Dec 26 17:51:44.253524976 (506h22m44.124608031s) [noise=true , address=145.79.227.45:4229 , piece_id=AZJFYG26H6TS33VDSJLOZAMLB4IXDHNNCGNEYPVM5GAPI2RX3P6Q, seg_pos={0 0}, upload=444333]: Uploading piece... Dec 26 17:51:44.253945116 (506h22m44.125028151s) [noise=true , address=145.79.227.46:4602 , piece_id=ECGPTTVPWENOLT2KGHKS3LVH5UBIRSOJ7MOQOIVNPFANSUYQCMSA, seg_pos={0 0}, upload=444333]: Uploading piece... Dec 26 17:51:44.254587920 (506h22m44.125670955s) [noise=true , address=45.91.116.8:4045 , piece_id=LEBFSGXN42BETMF7Y2A7BGF4IRTZS5E4M3UQGPXBAKUPYC2EJZZQ, seg_pos={0 0}, upload=444333]: Uploading piece... Dec 26 17:51:44.255123360 (506h22m44.126206405s) [noise=true , address=145.79.227.42:4121 , piece_id=SVSSFSUNTQW7RAQWWTRL5I5GIQVHZZJC72JE67T4MOERTUQIU6VA, seg_pos={0 0}, upload=444333]: Uploading piece... Dec 26 17:51:44.255301148 (506h22m44.126384203s) [noise=true , address=145.79.227.8:4330 , piece_id=VFRBYMLXXLKNWXJ55AS3LY5OJ4NDYX63RO75IIGFNUJSVRUYG4BQ, seg_pos={0 0}, upload=444333]: Uploading piece... Dec 26 17:51:44.255791632 (506h22m44.126874677s) [noise=true , address=77.90.31.17:4770 , piece_id=MV7MNB7IEULOCTO2HP2SXMM3NEYA2OZGPITNVCKNNBAAWBDLGRQA, seg_pos={0 0}, upload=444333]: Uploading piece... Dec 26 17:51:44.256058289 (506h22m44.127141334s) [noise=true , address=77.90.31.14:4663 , piece_id=UPPD3AXAZGG4P5TTY4TEU2O6JHC6LUBJAVHKC74LCWJHUW3MAOBA, seg_pos={0 0}, upload=444333]: Uploading piece... Dec 26 17:51:44.256323264 (506h22m44.127406309s) [noise=true , address=145.79.227.41:4087 , piece_id=TBTU3YCEOW5MBSY3EEN5FOMMAXSVYYT77ZETHVOJIXKVLCNGDQ7Q, seg_pos={0 0}, upload=444333]: Uploading piece... Dec 26 17:51:44.256526151 (506h22m44.127609196s) [noise=true , address=145.79.227.38:4979 , piece_id=3VOAEVN4NVEESMVPIISWOBSCTJU76REM3REKJWSUGLQCYW5D7ZHA, seg_pos={0 0}, upload=444333]: Uploading piece... Dec 26 17:51:44.256901556 (506h22m44.127984601s) [noise=true , address=145.79.227.11:4730 , piece_id=WVALFZ65W45RU7FLO4IOE377B6CKJPZCZULUHQFFBSOMSFTV5VWQ, seg_pos={0 0}, upload=444333]: Uploading piece... Dec 26 17:51:44.256990264 (506h22m44.128073319s) [noise=true , address=145.79.227.15:4874 , piece_id=BW27X34FKIWARBEHQFIYWCKM5MIYG3BY4OTX37SF7RNOKNFVHY2Q, seg_pos={0 0}, upload=444333]: Uploading piece... Dec 26 17:51:44.257506828 (506h22m44.128589873s) [noise=true , address=145.79.227.47:4267 , piece_id=MY3VGFEQNDCFFCGX4EKY5SRDAEP52HDS5EPR3XXAJCAXP34TKQBQ, seg_pos={0 0}, upload=444333]: Uploading piece... Dec 26 17:51:44.257872845 (506h22m44.12895588s) [noise=true , address=45.91.116.14:4190 , piece_id=IZDDTQ6ZDIQINXQGNS3GD6RYVLI5TFDB4TCQT7RPPOHCSGIECNRA, seg_pos={0 0}, upload=444333]: Uploading piece... Dec 26 17:51:44.258130345 (506h22m44.12921339s) [noise=true , address=145.79.227.32:4836 , piece_id=BIDZSYZH7JZ4ARTHIPWDZQFQUPY2QDRQY2FKV7J4DEEJW2S2K3NQ, seg_pos={0 0}, upload=444333]: Uploading piece... Dec 26 17:51:44.258270612 (506h22m44.129353657s) [noise=true , address=77.90.31.18:4805 , piece_id=4BHLLSGBF6B74I6OOICNNFRJMM2WQXDVDSJVPXZGVZBB5IJ2XRQQ, seg_pos={0 0}, upload=444333]: Uploading piece... Dec 26 17:51:44.258961107 (506h22m44.130044152s) [noise=true , address=145.79.227.12:4765 , piece_id=ZT5AWKLZU2V7JS7BXIPAOJQQ3WFJDNIR7TRSD4QMH66KA6AVS44A, seg_pos={0 0}, upload=444333]: Uploading piece... Dec 26 17:51:45.429528151 (506h22m45.300611196s) [noise=true , address=77.90.31.18:4805 , piece_id=4BHLLSGBF6B74I6OOICNNFRJMM2WQXDVDSJVPXZGVZBB5IJ2XRQQ, seg_pos={0 0}, upload=444333]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: storage node overloaded, request limit: 2000 Dec 26 17:51:45.959357869 (506h22m45.830440914s) [upload=444333]: Next returned nil segment Dec 26 17:51:45.959375462 (506h22m45.830458507s) [upload=444333]: Waiting for error group managing segments... Dec 26 17:51:46.247253764 (506h22m46.118336799s) [noise=true , address=145.79.227.46:4602 , piece_id=ECGPTTVPWENOLT2KGHKS3LVH5UBIRSOJ7MOQOIVNPFANSUYQCMSA, seg_pos={0 0}, upload=444333]: Done uploading piece. err: <nil> Dec 26 17:51:46.250896720 (506h22m46.121979755s) [noise=true , address=145.79.227.45:4229 , piece_id=AZJFYG26H6TS33VDSJLOZAMLB4IXDHNNCGNEYPVM5GAPI2RX3P6Q, seg_pos={0 0}, upload=444333]: Done uploading piece. err: <nil> Dec 26 17:51:46.253591341 (506h22m46.124674386s) [noise=true , address=145.79.227.8:4330 , piece_id=VFRBYMLXXLKNWXJ55AS3LY5OJ4NDYX63RO75IIGFNUJSVRUYG4BQ, seg_pos={0 0}, upload=444333]: Done uploading piece. err: <nil> Dec 26 17:51:46.256258740 (506h22m46.127341775s) [noise=true , address=145.79.227.15:4874 , piece_id=BW27X34FKIWARBEHQFIYWCKM5MIYG3BY4OTX37SF7RNOKNFVHY2Q, seg_pos={0 0}, upload=444333]: Done uploading piece. err: <nil> Dec 26 17:51:46.279230472 (506h22m46.150313517s) [noise=true , address=145.79.227.32:4836 , piece_id=BIDZSYZH7JZ4ARTHIPWDZQFQUPY2QDRQY2FKV7J4DEEJW2S2K3NQ, seg_pos={0 0}, upload=444333]: Done uploading piece. err: <nil> Dec 26 17:51:46.280002032 (506h22m46.151085067s) [noise=true , address=145.79.227.38:4979 , piece_id=3VOAEVN4NVEESMVPIISWOBSCTJU76REM3REKJWSUGLQCYW5D7ZHA, seg_pos={0 0}, upload=444333]: Done uploading piece. err: <nil> Dec 26 17:51:46.281328107 (506h22m46.152411142s) [noise=true , address=145.79.227.47:4267 , piece_id=MY3VGFEQNDCFFCGX4EKY5SRDAEP52HDS5EPR3XXAJCAXP34TKQBQ, seg_pos={0 0}, upload=444333]: Done uploading piece. err: <nil> Dec 26 17:51:46.286020221 (506h22m46.157103256s) [noise=true , address=145.79.227.11:4730 , piece_id=WVALFZ65W45RU7FLO4IOE377B6CKJPZCZULUHQFFBSOMSFTV5VWQ, seg_pos={0 0}, upload=444333]: Done uploading piece. err: <nil> Dec 26 17:51:46.288375025 (506h22m46.15945806s) [noise=true , address=145.79.227.42:4121 , piece_id=SVSSFSUNTQW7RAQWWTRL5I5GIQVHZZJC72JE67T4MOERTUQIU6VA, seg_pos={0 0}, upload=444333]: Done uploading piece. err: <nil> Dec 26 17:51:46.588588915 (506h22m46.45967195s) [noise=true , address=77.90.31.14:4663 , piece_id=UPPD3AXAZGG4P5TTY4TEU2O6JHC6LUBJAVHKC74LCWJHUW3MAOBA, seg_pos={0 0}, upload=444333]: Done uploading piece. err: <nil> Dec 26 17:51:46.590231473 (506h22m46.461314518s) [noise=true , address=45.91.116.14:4190 , piece_id=IZDDTQ6ZDIQINXQGNS3GD6RYVLI5TFDB4TCQT7RPPOHCSGIECNRA, seg_pos={0 0}, upload=444333]: Done uploading piece. err: <nil> Dec 26 17:51:46.603359420 (506h22m46.474442465s) [noise=true , address=77.90.31.17:4770 , piece_id=MV7MNB7IEULOCTO2HP2SXMM3NEYA2OZGPITNVCKNNBAAWBDLGRQA, seg_pos={0 0}, upload=444333]: Done uploading piece. err: <nil> Dec 26 17:51:46.606138262 (506h22m46.477221307s) [noise=true , address=45.91.116.8:4045 , piece_id=LEBFSGXN42BETMF7Y2A7BGF4IRTZS5E4M3UQGPXBAKUPYC2EJZZQ, seg_pos={0 0}, upload=444333]: Done uploading piece. err: <nil> Dec 26 17:51:46.606159363 (506h22m46.477242408s) [seg_pos={0 0}, upload=444333]: Segment reached optimal threshold of 13 pieces. Dec 26 17:51:46.606376275 (506h22m46.47745932s) [noise=true , address=145.79.227.41:4087 , piece_id=TBTU3YCEOW5MBSY3EEN5FOMMAXSVYYT77ZETHVOJIXKVLCNGDQ7Q, seg_pos={0 0}, upload=444333]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR): protocol: write tcp 188.166.225.170:59534->145.79.227.41:4087: use of closed network connection; write tcp 188.166.225.170:59534->145.79.227.41:4087: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59534->145.79.227.41:4087: use of closed network connection Dec 26 17:51:46.606383268 (506h22m46.477466304s) [noise=true , address=145.79.227.41:4087 , piece_id=TBTU3YCEOW5MBSY3EEN5FOMMAXSVYYT77ZETHVOJIXKVLCNGDQ7Q, seg_pos={0 0}, upload=444333]: Long tail cancellation due to optimal threshold Dec 26 17:51:46.606386765 (506h22m46.47746981s) [noise=true , address=145.79.227.12:4765 , piece_id=ZT5AWKLZU2V7JS7BXIPAOJQQ3WFJDNIR7TRSD4QMH66KA6AVS44A, seg_pos={0 0}, upload=444333]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV): protocol: write tcp 188.166.225.170:40256->145.79.227.12:4765: use of closed network connection; write tcp 188.166.225.170:40256->145.79.227.12:4765: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40256->145.79.227.12:4765: use of closed network connection Dec 26 17:51:46.606397215 (506h22m46.47748027s) [noise=true , address=145.79.227.12:4765 , piece_id=ZT5AWKLZU2V7JS7BXIPAOJQQ3WFJDNIR7TRSD4QMH66KA6AVS44A, seg_pos={0 0}, upload=444333]: Long tail cancellation due to optimal threshold Dec 26 17:51:46.606452631 (506h22m46.477535676s) [seg_pos={0 0}, upload=444333]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 17:51:46.606720050 (506h22m46.477803095s) [upload=444333]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:51:46.606726051 (506h22m46.477809096s) [upload=444333]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:51:46.895225284 (506h22m46.766308329s) [upload=444333]: Done uploading segments... Dec 26 17:51:46.895243990 (506h22m46.766327025s) [upload=444333]: Upload finished. err: <nil> Duration: 30.003101001s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 40.889787223s trace: https://jaeger.dev.storj.io/trace/665ea65f6e9ccc33 FAIL (2025-12-26 17:52:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== e=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789316979 (506h12m54.660400024s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789326507 (506h12m54.660409552s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789611339 (506h12m54.660694384s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789619275 (506h12m54.66070231s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789914006 (506h12m54.660997041s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789949443 (506h12m54.661032478s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790181566 (506h12m54.661264601s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790295632 (506h12m54.661378668s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790510872 (506h12m54.661593917s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790692980 (506h12m54.661776015s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790787379 (506h12m54.661870414s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790992970 (506h12m54.662076005s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791120053 (506h12m54.662203098s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791275218 (506h12m54.662358253s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791443709 (506h12m54.662526744s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:56.112313145 (506h12m55.98339619s) [upload=444191]: Next returned nil segment Dec 26 17:41:56.112330638 (506h12m55.983413683s) [upload=444191]: Waiting for error group managing segments... Dec 26 17:41:56.485322720 (506h12m56.356405765s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.485637079 (506h12m56.356720124s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.666807539 (506h12m56.537890584s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.678753987 (506h12m56.549837032s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.687788039 (506h12m56.558871074s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.694318023 (506h12m56.565401058s) [noise=true , address=145.79.227.7:4326 , piece_id=DMQ5RPI46X2U7TJTS42ZHQEF32BDJRKJJO5HWOYAPNS6VWVG5B5Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.716581187 (506h12m56.587664232s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.972183007 (506h12m56.843266042s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.011917999 (506h12m56.883001034s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.054168622 (506h12m56.925251667s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.114312777 (506h12m56.985395812s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.499846250 (506h12m57.370929285s) [noise=true , address=145.79.227.34:4919 , piece_id=YRWHW3BJVEGVJRSA6RJB3T7VJQSDTGKEZQKRICFIV2G3T7AN4MIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.612649066 (506h12m57.483732101s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649413733 (506h12m57.520496768s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649440282 (506h12m57.520523328s) [seg_pos={0 0}, upload=444191]: Segment reached optimal threshold of 14 pieces. Dec 26 17:41:57.649674729 (506h12m57.520757774s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G7ty71tPwe7cZSBNK9u74DFUCGomnxDsNvAif4C3SbP7HWwfo): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649684087 (506h12m57.520767142s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649747708 (506h12m57.520830763s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v6zXZuMmyL2MU8qb3q6gg1iwTYvndjzMjVpv8btEpeX9Lnqmu): protocol: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection Dec 26 17:41:57.649785219 (506h12m57.520868274s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649899828 (506h12m57.520982903s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649908775 (506h12m57.52099182s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650074641 (506h12m57.521157696s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.650083287 (506h12m57.521166342s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650132821 (506h12m57.521215866s) [seg_pos={0 0}, upload=444191]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 17:41:57.650220138 (506h12m57.521303183s) [upload=444191]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:41:57.650227171 (506h12m57.521310206s) [upload=444191]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:41:57.958890870 (506h12m57.829973915s) [upload=444191]: Done uploading segments... Dec 26 17:41:57.958911620 (506h12m57.829994665s) [upload=444191]: Upload finished. err: <nil> Duration: 40.889818943s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 22.912489619s trace: https://jaeger.dev.storj.io/trace/665ea65f6e9ccc33 FAIL (2025-12-26 17:51:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== e=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789316979 (506h12m54.660400024s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789326507 (506h12m54.660409552s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789611339 (506h12m54.660694384s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789619275 (506h12m54.66070231s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789914006 (506h12m54.660997041s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789949443 (506h12m54.661032478s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790181566 (506h12m54.661264601s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790295632 (506h12m54.661378668s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790510872 (506h12m54.661593917s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790692980 (506h12m54.661776015s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790787379 (506h12m54.661870414s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790992970 (506h12m54.662076005s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791120053 (506h12m54.662203098s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791275218 (506h12m54.662358253s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791443709 (506h12m54.662526744s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:56.112313145 (506h12m55.98339619s) [upload=444191]: Next returned nil segment Dec 26 17:41:56.112330638 (506h12m55.983413683s) [upload=444191]: Waiting for error group managing segments... Dec 26 17:41:56.485322720 (506h12m56.356405765s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.485637079 (506h12m56.356720124s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.666807539 (506h12m56.537890584s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.678753987 (506h12m56.549837032s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.687788039 (506h12m56.558871074s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.694318023 (506h12m56.565401058s) [noise=true , address=145.79.227.7:4326 , piece_id=DMQ5RPI46X2U7TJTS42ZHQEF32BDJRKJJO5HWOYAPNS6VWVG5B5Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.716581187 (506h12m56.587664232s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.972183007 (506h12m56.843266042s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.011917999 (506h12m56.883001034s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.054168622 (506h12m56.925251667s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.114312777 (506h12m56.985395812s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.499846250 (506h12m57.370929285s) [noise=true , address=145.79.227.34:4919 , piece_id=YRWHW3BJVEGVJRSA6RJB3T7VJQSDTGKEZQKRICFIV2G3T7AN4MIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.612649066 (506h12m57.483732101s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649413733 (506h12m57.520496768s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649440282 (506h12m57.520523328s) [seg_pos={0 0}, upload=444191]: Segment reached optimal threshold of 14 pieces. Dec 26 17:41:57.649674729 (506h12m57.520757774s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G7ty71tPwe7cZSBNK9u74DFUCGomnxDsNvAif4C3SbP7HWwfo): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649684087 (506h12m57.520767142s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649747708 (506h12m57.520830763s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v6zXZuMmyL2MU8qb3q6gg1iwTYvndjzMjVpv8btEpeX9Lnqmu): protocol: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection Dec 26 17:41:57.649785219 (506h12m57.520868274s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649899828 (506h12m57.520982903s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649908775 (506h12m57.52099182s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650074641 (506h12m57.521157696s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.650083287 (506h12m57.521166342s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650132821 (506h12m57.521215866s) [seg_pos={0 0}, upload=444191]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 17:41:57.650220138 (506h12m57.521303183s) [upload=444191]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:41:57.650227171 (506h12m57.521310206s) [upload=444191]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:41:57.958890870 (506h12m57.829973915s) [upload=444191]: Done uploading segments... Dec 26 17:41:57.958911620 (506h12m57.829994665s) [upload=444191]: Upload finished. err: <nil> Duration: 22.912520167s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 38.303469472s trace: https://jaeger.dev.storj.io/trace/665ea65f6e9ccc33 FAIL (2025-12-26 17:50:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== e=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789316979 (506h12m54.660400024s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789326507 (506h12m54.660409552s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789611339 (506h12m54.660694384s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789619275 (506h12m54.66070231s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789914006 (506h12m54.660997041s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789949443 (506h12m54.661032478s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790181566 (506h12m54.661264601s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790295632 (506h12m54.661378668s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790510872 (506h12m54.661593917s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790692980 (506h12m54.661776015s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790787379 (506h12m54.661870414s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790992970 (506h12m54.662076005s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791120053 (506h12m54.662203098s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791275218 (506h12m54.662358253s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791443709 (506h12m54.662526744s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:56.112313145 (506h12m55.98339619s) [upload=444191]: Next returned nil segment Dec 26 17:41:56.112330638 (506h12m55.983413683s) [upload=444191]: Waiting for error group managing segments... Dec 26 17:41:56.485322720 (506h12m56.356405765s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.485637079 (506h12m56.356720124s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.666807539 (506h12m56.537890584s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.678753987 (506h12m56.549837032s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.687788039 (506h12m56.558871074s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.694318023 (506h12m56.565401058s) [noise=true , address=145.79.227.7:4326 , piece_id=DMQ5RPI46X2U7TJTS42ZHQEF32BDJRKJJO5HWOYAPNS6VWVG5B5Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.716581187 (506h12m56.587664232s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.972183007 (506h12m56.843266042s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.011917999 (506h12m56.883001034s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.054168622 (506h12m56.925251667s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.114312777 (506h12m56.985395812s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.499846250 (506h12m57.370929285s) [noise=true , address=145.79.227.34:4919 , piece_id=YRWHW3BJVEGVJRSA6RJB3T7VJQSDTGKEZQKRICFIV2G3T7AN4MIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.612649066 (506h12m57.483732101s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649413733 (506h12m57.520496768s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649440282 (506h12m57.520523328s) [seg_pos={0 0}, upload=444191]: Segment reached optimal threshold of 14 pieces. Dec 26 17:41:57.649674729 (506h12m57.520757774s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G7ty71tPwe7cZSBNK9u74DFUCGomnxDsNvAif4C3SbP7HWwfo): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649684087 (506h12m57.520767142s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649747708 (506h12m57.520830763s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v6zXZuMmyL2MU8qb3q6gg1iwTYvndjzMjVpv8btEpeX9Lnqmu): protocol: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection Dec 26 17:41:57.649785219 (506h12m57.520868274s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649899828 (506h12m57.520982903s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649908775 (506h12m57.52099182s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650074641 (506h12m57.521157696s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.650083287 (506h12m57.521166342s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650132821 (506h12m57.521215866s) [seg_pos={0 0}, upload=444191]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 17:41:57.650220138 (506h12m57.521303183s) [upload=444191]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:41:57.650227171 (506h12m57.521310206s) [upload=444191]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:41:57.958890870 (506h12m57.829973915s) [upload=444191]: Done uploading segments... Dec 26 17:41:57.958911620 (506h12m57.829994665s) [upload=444191]: Upload finished. err: <nil> Duration: 38.303485894s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [......XX..X....X.X.XX...XX...XX....XX..XX......X.XXX.....XX.] duration: 5.817956153s trace: https://jaeger.dev.storj.io/trace/35e50250d8ec4d18 FAIL (2025-12-26 17:50:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 49:45.546933233 (506h20m45.418016268s) [upload=444303]: Starting upload Dec 26 17:49:45.547023725 (506h20m45.41810678s) [upload=444303]: Done starting upload Dec 26 17:49:45.547161267 (506h20m45.418244302s) [upload=444303]: Uploading segments... Dec 26 17:49:45.547605252 (506h20m45.418688297s) [upload=444303]: Got next segment. Inline: false Dec 26 17:49:45.547667982 (506h20m45.418751027s) [upload=444303]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 17:49:45.547672871 (506h20m45.418755906s) [upload=444303]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 17:49:45.798759755 (506h20m45.66984279s) [seg_pos={0 0}, upload=444303]: Begin upload segment... Dec 26 17:49:45.798803067 (506h20m45.669886102s) [seg_pos={0 0}, upload=444303]: Done begin upload segment. Dec 26 17:49:45.798827774 (506h20m45.669910809s) [noise=true , address=145.79.227.42:4123 , piece_id=PCF57B3KN5GCF66PIC4W25NWKPAREHUATMLNICZW2QORLZF5JXFA, seg_pos={0 0}, upload=444303]: Uploading piece... Dec 26 17:49:45.798988950 (506h20m45.670071985s) [noise=true , address=145.79.227.17:4970 , piece_id=5PBTPJ7CASBOGJHB6RNOKKB4QORQTNOCPRJHLSWHDKNFCJNRAVRQ, seg_pos={0 0}, upload=444303]: Uploading piece... Dec 26 17:49:45.799347293 (506h20m45.670430339s) [noise=true , address=145.79.227.41:4085 , piece_id=O2QUW63WBL5QCPWJMPKXLR7JFYKXOHDTE7FONSIO3C4KSWSG7FIA, seg_pos={0 0}, upload=444303]: Uploading piece... Dec 26 17:49:45.799818831 (506h20m45.670901876s) [noise=true , address=145.79.227.45:4231 , piece_id=YJNSD2UPVC44FPSZOSF2JJXUUFEKDT7ILHPHTNMRBJ6YTFNAUSCA, seg_pos={0 0}, upload=444303]: Uploading piece... Dec 26 17:49:45.800132238 (506h20m45.671215283s) [noise=true , address=45.91.116.11:4119 , piece_id=BQB6HVHKCV5MIX3J3N4IEG4K6PTQXHFVBFXHUC2RCHLSDYP6G4BA, seg_pos={0 0}, upload=444303]: Uploading piece... Dec 26 17:49:45.800396832 (506h20m45.671479867s) [noise=true , address=145.79.227.37:4567 , piece_id=HZLKTO2RYIIWJUYDKLBCBXM4OODKRWYVN7TX6S3NC4ALBDDSOK5A, seg_pos={0 0}, upload=444303]: Uploading piece... Dec 26 17:49:45.800665002 (506h20m45.671748037s) [noise=true , address=145.79.227.22:4511 , piece_id=HIFEE7NRC6URJD73GSDPWWQKQZ53ATEX3ILPGI3UWX4HAINHQZFQ, seg_pos={0 0}, upload=444303]: Uploading piece... Dec 26 17:49:45.800938272 (506h20m45.672021307s) [noise=true , address=145.79.227.35:4482 , piece_id=BDEH4XYBMLMEU7Y6C3R3C22NC5Z5SNIUALARXRZ3Y5LL4JX45K2Q, seg_pos={0 0}, upload=444303]: Uploading piece... Dec 26 17:49:45.801177298 (506h20m45.672260333s) [noise=true , address=145.79.227.38:4979 , piece_id=6OXP7ITZMGRMQPZGRKYMSWJPRFLAGV32TUK5V2PWW6SYVXMFY4JA, seg_pos={0 0}, upload=444303]: Uploading piece... Dec 26 17:49:45.801436361 (506h20m45.672519396s) [noise=true , address=45.91.116.12:4225 , piece_id=2JBAXZKMZPPGUACKEBK7M2QOU77X5G3A7NUNXDGRMWIS62COFKPA, seg_pos={0 0}, upload=444303]: Uploading piece... Dec 26 17:49:45.802369117 (506h20m45.673452152s) [noise=true , address=145.79.227.44:4194 , piece_id=DN36VWFP3N2CMTWMOXAMX7ZZPLBS5ZCGPDLMIAYIG3QIWXBGO4QA, seg_pos={0 0}, upload=444303]: Uploading piece... Dec 26 17:49:45.802775130 (506h20m45.673858175s) [noise=true , address=77.90.31.16:4736 , piece_id=IARA5RUI2HGVCDMQBC2E66OHLSA2NAPOS2GLZ3SMUPX536PDYOCA, seg_pos={0 0}, upload=444303]: Uploading piece... Dec 26 17:49:45.803110639 (506h20m45.674193674s) [noise=true , address=145.79.227.51:4411 , piece_id=U3MYVOJ5NCRZZ277CVQMSQYABZYHOLEHQOZI4NJACKRXLFOX2TNA, seg_pos={0 0}, upload=444303]: Uploading piece... Dec 26 17:49:45.803459403 (506h20m45.674542439s) [noise=true , address=45.91.116.5:4297 , piece_id=U7IN34XCBKJ6UUOCW3YPYLJ7YNYPVAYACUO5H254KZFWSSTMZ63Q, seg_pos={0 0}, upload=444303]: Uploading piece... Dec 26 17:49:45.803820701 (506h20m45.674903736s) [noise=true , address=145.79.227.15:4874 , piece_id=DL7H3UQQZZ6DWHL23YN7VN5RFLJ7CY6PLSOAJ2UMCBV2TPW3NKPA, seg_pos={0 0}, upload=444303]: Uploading piece... Dec 26 17:49:45.804169205 (506h20m45.67525224s) [noise=true , address=145.79.227.48:4303 , piece_id=GPUCXYNHJKXZFNTQAPMPMLF65X3W5MWLHTGQMDQL3K6UCHJDAY2A, seg_pos={0 0}, upload=444303]: Uploading piece... Dec 26 17:49:47.502662332 (506h20m47.373745377s) [upload=444303]: Next returned nil segment Dec 26 17:49:47.502676088 (506h20m47.373759133s) [upload=444303]: Waiting for error group managing segments... Dec 26 17:49:47.790650903 (506h20m47.661733938s) [noise=true , address=145.79.227.42:4123 , piece_id=PCF57B3KN5GCF66PIC4W25NWKPAREHUATMLNICZW2QORLZF5JXFA, seg_pos={0 0}, upload=444303]: Done uploading piece. err: <nil> Dec 26 17:49:47.805161804 (506h20m47.676244839s) [noise=true , address=145.79.227.35:4482 , piece_id=BDEH4XYBMLMEU7Y6C3R3C22NC5Z5SNIUALARXRZ3Y5LL4JX45K2Q, seg_pos={0 0}, upload=444303]: Done uploading piece. err: <nil> Dec 26 17:49:47.805680091 (506h20m47.676763126s) [noise=true , address=145.79.227.37:4567 , piece_id=HZLKTO2RYIIWJUYDKLBCBXM4OODKRWYVN7TX6S3NC4ALBDDSOK5A, seg_pos={0 0}, upload=444303]: Done uploading piece. err: <nil> Dec 26 17:49:47.808020628 (506h20m47.679103663s) [noise=true , address=145.79.227.15:4874 , piece_id=DL7H3UQQZZ6DWHL23YN7VN5RFLJ7CY6PLSOAJ2UMCBV2TPW3NKPA, seg_pos={0 0}, upload=444303]: Done uploading piece. err: <nil> Dec 26 17:49:47.821567703 (506h20m47.692650738s) [noise=true , address=145.79.227.45:4231 , piece_id=YJNSD2UPVC44FPSZOSF2JJXUUFEKDT7ILHPHTNMRBJ6YTFNAUSCA, seg_pos={0 0}, upload=444303]: Done uploading piece. err: <nil> Dec 26 17:49:47.822683879 (506h20m47.693766914s) [noise=true , address=145.79.227.51:4411 , piece_id=U3MYVOJ5NCRZZ277CVQMSQYABZYHOLEHQOZI4NJACKRXLFOX2TNA, seg_pos={0 0}, upload=444303]: Done uploading piece. err: <nil> Dec 26 17:49:47.823197246 (506h20m47.694280282s) [noise=true , address=145.79.227.22:4511 , piece_id=HIFEE7NRC6URJD73GSDPWWQKQZ53ATEX3ILPGI3UWX4HAINHQZFQ, seg_pos={0 0}, upload=444303]: Done uploading piece. err: <nil> Dec 26 17:49:47.836677435 (506h20m47.70776048s) [noise=true , address=145.79.227.48:4303 , piece_id=GPUCXYNHJKXZFNTQAPMPMLF65X3W5MWLHTGQMDQL3K6UCHJDAY2A, seg_pos={0 0}, upload=444303]: Done uploading piece. err: <nil> Dec 26 17:49:47.843337307 (506h20m47.714420342s) [noise=true , address=145.79.227.38:4979 , piece_id=6OXP7ITZMGRMQPZGRKYMSWJPRFLAGV32TUK5V2PWW6SYVXMFY4JA, seg_pos={0 0}, upload=444303]: Done uploading piece. err: <nil> Dec 26 17:49:48.083197271 (506h20m47.954280307s) [noise=true , address=77.90.31.16:4736 , piece_id=IARA5RUI2HGVCDMQBC2E66OHLSA2NAPOS2GLZ3SMUPX536PDYOCA, seg_pos={0 0}, upload=444303]: Done uploading piece. err: <nil> Dec 26 17:49:48.141985204 (506h20m48.013068249s) [noise=true , address=45.91.116.12:4225 , piece_id=2JBAXZKMZPPGUACKEBK7M2QOU77X5G3A7NUNXDGRMWIS62COFKPA, seg_pos={0 0}, upload=444303]: Done uploading piece. err: <nil> Dec 26 17:49:48.144369094 (506h20m48.015452139s) [noise=true , address=145.79.227.41:4085 , piece_id=O2QUW63WBL5QCPWJMPKXLR7JFYKXOHDTE7FONSIO3C4KSWSG7FIA, seg_pos={0 0}, upload=444303]: Done uploading piece. err: <nil> Dec 26 17:49:48.171456162 (506h20m48.042539207s) [noise=true , address=45.91.116.11:4119 , piece_id=BQB6HVHKCV5MIX3J3N4IEG4K6PTQXHFVBFXHUC2RCHLSDYP6G4BA, seg_pos={0 0}, upload=444303]: Done uploading piece. err: <nil> Dec 26 17:49:48.171484226 (506h20m48.042567261s) [seg_pos={0 0}, upload=444303]: Segment reached optimal threshold of 13 pieces. Dec 26 17:49:48.171729252 (506h20m48.042812297s) [noise=true , address=145.79.227.17:4970 , piece_id=5PBTPJ7CASBOGJHB6RNOKKB4QORQTNOCPRJHLSWHDKNFCJNRAVRQ, seg_pos={0 0}, upload=444303]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe): protocol: write tcp 188.166.225.170:34658->145.79.227.17:4970: use of closed network connection; write tcp 188.166.225.170:34658->145.79.227.17:4970: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34658->145.79.227.17:4970: use of closed network connection Dec 26 17:49:48.171740664 (506h20m48.042823709s) [noise=true , address=145.79.227.17:4970 , piece_id=5PBTPJ7CASBOGJHB6RNOKKB4QORQTNOCPRJHLSWHDKNFCJNRAVRQ, seg_pos={0 0}, upload=444303]: Long tail cancellation due to optimal threshold Dec 26 17:49:48.171789948 (506h20m48.042872983s) [noise=true , address=145.79.227.44:4194 , piece_id=DN36VWFP3N2CMTWMOXAMX7ZZPLBS5ZCGPDLMIAYIG3QIWXBGO4QA, seg_pos={0 0}, upload=444303]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 26 17:49:48.171797983 (506h20m48.042881018s) [noise=true , address=145.79.227.44:4194 , piece_id=DN36VWFP3N2CMTWMOXAMX7ZZPLBS5ZCGPDLMIAYIG3QIWXBGO4QA, seg_pos={0 0}, upload=444303]: Long tail cancellation due to optimal threshold Dec 26 17:49:48.172073217 (506h20m48.043156253s) [noise=true , address=45.91.116.5:4297 , piece_id=U7IN34XCBKJ6UUOCW3YPYLJ7YNYPVAYACUO5H254KZFWSSTMZ63Q, seg_pos={0 0}, upload=444303]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 26 17:49:48.172080601 (506h20m48.043163636s) [noise=true , address=45.91.116.5:4297 , piece_id=U7IN34XCBKJ6UUOCW3YPYLJ7YNYPVAYACUO5H254KZFWSSTMZ63Q, seg_pos={0 0}, upload=444303]: Long tail cancellation due to optimal threshold Dec 26 17:49:48.172097663 (506h20m48.043180708s) [seg_pos={0 0}, upload=444303]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 17:49:48.172200780 (506h20m48.043283815s) [upload=444303]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:49:48.172210990 (506h20m48.043294035s) [upload=444303]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:49:48.474685714 (506h20m48.345768769s) [upload=444303]: Done uploading segments... Dec 26 17:49:48.474708668 (506h20m48.345791713s) [upload=444303]: Upload finished. err: <nil> Duration: 5.817986792s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 47.075496393s trace: https://jaeger.dev.storj.io/trace/665ea65f6e9ccc33 FAIL (2025-12-26 17:50:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== e=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789316979 (506h12m54.660400024s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789326507 (506h12m54.660409552s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789611339 (506h12m54.660694384s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789619275 (506h12m54.66070231s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789914006 (506h12m54.660997041s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789949443 (506h12m54.661032478s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790181566 (506h12m54.661264601s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790295632 (506h12m54.661378668s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790510872 (506h12m54.661593917s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790692980 (506h12m54.661776015s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790787379 (506h12m54.661870414s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790992970 (506h12m54.662076005s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791120053 (506h12m54.662203098s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791275218 (506h12m54.662358253s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791443709 (506h12m54.662526744s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:56.112313145 (506h12m55.98339619s) [upload=444191]: Next returned nil segment Dec 26 17:41:56.112330638 (506h12m55.983413683s) [upload=444191]: Waiting for error group managing segments... Dec 26 17:41:56.485322720 (506h12m56.356405765s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.485637079 (506h12m56.356720124s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.666807539 (506h12m56.537890584s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.678753987 (506h12m56.549837032s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.687788039 (506h12m56.558871074s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.694318023 (506h12m56.565401058s) [noise=true , address=145.79.227.7:4326 , piece_id=DMQ5RPI46X2U7TJTS42ZHQEF32BDJRKJJO5HWOYAPNS6VWVG5B5Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.716581187 (506h12m56.587664232s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.972183007 (506h12m56.843266042s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.011917999 (506h12m56.883001034s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.054168622 (506h12m56.925251667s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.114312777 (506h12m56.985395812s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.499846250 (506h12m57.370929285s) [noise=true , address=145.79.227.34:4919 , piece_id=YRWHW3BJVEGVJRSA6RJB3T7VJQSDTGKEZQKRICFIV2G3T7AN4MIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.612649066 (506h12m57.483732101s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649413733 (506h12m57.520496768s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649440282 (506h12m57.520523328s) [seg_pos={0 0}, upload=444191]: Segment reached optimal threshold of 14 pieces. Dec 26 17:41:57.649674729 (506h12m57.520757774s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G7ty71tPwe7cZSBNK9u74DFUCGomnxDsNvAif4C3SbP7HWwfo): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649684087 (506h12m57.520767142s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649747708 (506h12m57.520830763s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v6zXZuMmyL2MU8qb3q6gg1iwTYvndjzMjVpv8btEpeX9Lnqmu): protocol: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection Dec 26 17:41:57.649785219 (506h12m57.520868274s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649899828 (506h12m57.520982903s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649908775 (506h12m57.52099182s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650074641 (506h12m57.521157696s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.650083287 (506h12m57.521166342s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650132821 (506h12m57.521215866s) [seg_pos={0 0}, upload=444191]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 17:41:57.650220138 (506h12m57.521303183s) [upload=444191]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:41:57.650227171 (506h12m57.521310206s) [upload=444191]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:41:57.958890870 (506h12m57.829973915s) [upload=444191]: Done uploading segments... Dec 26 17:41:57.958911620 (506h12m57.829994665s) [upload=444191]: Upload finished. err: <nil> Duration: 47.075520208s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [X......XX..X....X.X.XX...XX...XX....XX..XX......X.XXX.....XX] duration: 30.002812252s trace: https://jaeger.dev.storj.io/trace/35e50250d8ec4d18 FAIL (2025-12-26 17:50:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 49:45.546933233 (506h20m45.418016268s) [upload=444303]: Starting upload Dec 26 17:49:45.547023725 (506h20m45.41810678s) [upload=444303]: Done starting upload Dec 26 17:49:45.547161267 (506h20m45.418244302s) [upload=444303]: Uploading segments... Dec 26 17:49:45.547605252 (506h20m45.418688297s) [upload=444303]: Got next segment. Inline: false Dec 26 17:49:45.547667982 (506h20m45.418751027s) [upload=444303]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 17:49:45.547672871 (506h20m45.418755906s) [upload=444303]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 17:49:45.798759755 (506h20m45.66984279s) [seg_pos={0 0}, upload=444303]: Begin upload segment... Dec 26 17:49:45.798803067 (506h20m45.669886102s) [seg_pos={0 0}, upload=444303]: Done begin upload segment. Dec 26 17:49:45.798827774 (506h20m45.669910809s) [noise=true , address=145.79.227.42:4123 , piece_id=PCF57B3KN5GCF66PIC4W25NWKPAREHUATMLNICZW2QORLZF5JXFA, seg_pos={0 0}, upload=444303]: Uploading piece... Dec 26 17:49:45.798988950 (506h20m45.670071985s) [noise=true , address=145.79.227.17:4970 , piece_id=5PBTPJ7CASBOGJHB6RNOKKB4QORQTNOCPRJHLSWHDKNFCJNRAVRQ, seg_pos={0 0}, upload=444303]: Uploading piece... Dec 26 17:49:45.799347293 (506h20m45.670430339s) [noise=true , address=145.79.227.41:4085 , piece_id=O2QUW63WBL5QCPWJMPKXLR7JFYKXOHDTE7FONSIO3C4KSWSG7FIA, seg_pos={0 0}, upload=444303]: Uploading piece... Dec 26 17:49:45.799818831 (506h20m45.670901876s) [noise=true , address=145.79.227.45:4231 , piece_id=YJNSD2UPVC44FPSZOSF2JJXUUFEKDT7ILHPHTNMRBJ6YTFNAUSCA, seg_pos={0 0}, upload=444303]: Uploading piece... Dec 26 17:49:45.800132238 (506h20m45.671215283s) [noise=true , address=45.91.116.11:4119 , piece_id=BQB6HVHKCV5MIX3J3N4IEG4K6PTQXHFVBFXHUC2RCHLSDYP6G4BA, seg_pos={0 0}, upload=444303]: Uploading piece... Dec 26 17:49:45.800396832 (506h20m45.671479867s) [noise=true , address=145.79.227.37:4567 , piece_id=HZLKTO2RYIIWJUYDKLBCBXM4OODKRWYVN7TX6S3NC4ALBDDSOK5A, seg_pos={0 0}, upload=444303]: Uploading piece... Dec 26 17:49:45.800665002 (506h20m45.671748037s) [noise=true , address=145.79.227.22:4511 , piece_id=HIFEE7NRC6URJD73GSDPWWQKQZ53ATEX3ILPGI3UWX4HAINHQZFQ, seg_pos={0 0}, upload=444303]: Uploading piece... Dec 26 17:49:45.800938272 (506h20m45.672021307s) [noise=true , address=145.79.227.35:4482 , piece_id=BDEH4XYBMLMEU7Y6C3R3C22NC5Z5SNIUALARXRZ3Y5LL4JX45K2Q, seg_pos={0 0}, upload=444303]: Uploading piece... Dec 26 17:49:45.801177298 (506h20m45.672260333s) [noise=true , address=145.79.227.38:4979 , piece_id=6OXP7ITZMGRMQPZGRKYMSWJPRFLAGV32TUK5V2PWW6SYVXMFY4JA, seg_pos={0 0}, upload=444303]: Uploading piece... Dec 26 17:49:45.801436361 (506h20m45.672519396s) [noise=true , address=45.91.116.12:4225 , piece_id=2JBAXZKMZPPGUACKEBK7M2QOU77X5G3A7NUNXDGRMWIS62COFKPA, seg_pos={0 0}, upload=444303]: Uploading piece... Dec 26 17:49:45.802369117 (506h20m45.673452152s) [noise=true , address=145.79.227.44:4194 , piece_id=DN36VWFP3N2CMTWMOXAMX7ZZPLBS5ZCGPDLMIAYIG3QIWXBGO4QA, seg_pos={0 0}, upload=444303]: Uploading piece... Dec 26 17:49:45.802775130 (506h20m45.673858175s) [noise=true , address=77.90.31.16:4736 , piece_id=IARA5RUI2HGVCDMQBC2E66OHLSA2NAPOS2GLZ3SMUPX536PDYOCA, seg_pos={0 0}, upload=444303]: Uploading piece... Dec 26 17:49:45.803110639 (506h20m45.674193674s) [noise=true , address=145.79.227.51:4411 , piece_id=U3MYVOJ5NCRZZ277CVQMSQYABZYHOLEHQOZI4NJACKRXLFOX2TNA, seg_pos={0 0}, upload=444303]: Uploading piece... Dec 26 17:49:45.803459403 (506h20m45.674542439s) [noise=true , address=45.91.116.5:4297 , piece_id=U7IN34XCBKJ6UUOCW3YPYLJ7YNYPVAYACUO5H254KZFWSSTMZ63Q, seg_pos={0 0}, upload=444303]: Uploading piece... Dec 26 17:49:45.803820701 (506h20m45.674903736s) [noise=true , address=145.79.227.15:4874 , piece_id=DL7H3UQQZZ6DWHL23YN7VN5RFLJ7CY6PLSOAJ2UMCBV2TPW3NKPA, seg_pos={0 0}, upload=444303]: Uploading piece... Dec 26 17:49:45.804169205 (506h20m45.67525224s) [noise=true , address=145.79.227.48:4303 , piece_id=GPUCXYNHJKXZFNTQAPMPMLF65X3W5MWLHTGQMDQL3K6UCHJDAY2A, seg_pos={0 0}, upload=444303]: Uploading piece... Dec 26 17:49:47.502662332 (506h20m47.373745377s) [upload=444303]: Next returned nil segment Dec 26 17:49:47.502676088 (506h20m47.373759133s) [upload=444303]: Waiting for error group managing segments... Dec 26 17:49:47.790650903 (506h20m47.661733938s) [noise=true , address=145.79.227.42:4123 , piece_id=PCF57B3KN5GCF66PIC4W25NWKPAREHUATMLNICZW2QORLZF5JXFA, seg_pos={0 0}, upload=444303]: Done uploading piece. err: <nil> Dec 26 17:49:47.805161804 (506h20m47.676244839s) [noise=true , address=145.79.227.35:4482 , piece_id=BDEH4XYBMLMEU7Y6C3R3C22NC5Z5SNIUALARXRZ3Y5LL4JX45K2Q, seg_pos={0 0}, upload=444303]: Done uploading piece. err: <nil> Dec 26 17:49:47.805680091 (506h20m47.676763126s) [noise=true , address=145.79.227.37:4567 , piece_id=HZLKTO2RYIIWJUYDKLBCBXM4OODKRWYVN7TX6S3NC4ALBDDSOK5A, seg_pos={0 0}, upload=444303]: Done uploading piece. err: <nil> Dec 26 17:49:47.808020628 (506h20m47.679103663s) [noise=true , address=145.79.227.15:4874 , piece_id=DL7H3UQQZZ6DWHL23YN7VN5RFLJ7CY6PLSOAJ2UMCBV2TPW3NKPA, seg_pos={0 0}, upload=444303]: Done uploading piece. err: <nil> Dec 26 17:49:47.821567703 (506h20m47.692650738s) [noise=true , address=145.79.227.45:4231 , piece_id=YJNSD2UPVC44FPSZOSF2JJXUUFEKDT7ILHPHTNMRBJ6YTFNAUSCA, seg_pos={0 0}, upload=444303]: Done uploading piece. err: <nil> Dec 26 17:49:47.822683879 (506h20m47.693766914s) [noise=true , address=145.79.227.51:4411 , piece_id=U3MYVOJ5NCRZZ277CVQMSQYABZYHOLEHQOZI4NJACKRXLFOX2TNA, seg_pos={0 0}, upload=444303]: Done uploading piece. err: <nil> Dec 26 17:49:47.823197246 (506h20m47.694280282s) [noise=true , address=145.79.227.22:4511 , piece_id=HIFEE7NRC6URJD73GSDPWWQKQZ53ATEX3ILPGI3UWX4HAINHQZFQ, seg_pos={0 0}, upload=444303]: Done uploading piece. err: <nil> Dec 26 17:49:47.836677435 (506h20m47.70776048s) [noise=true , address=145.79.227.48:4303 , piece_id=GPUCXYNHJKXZFNTQAPMPMLF65X3W5MWLHTGQMDQL3K6UCHJDAY2A, seg_pos={0 0}, upload=444303]: Done uploading piece. err: <nil> Dec 26 17:49:47.843337307 (506h20m47.714420342s) [noise=true , address=145.79.227.38:4979 , piece_id=6OXP7ITZMGRMQPZGRKYMSWJPRFLAGV32TUK5V2PWW6SYVXMFY4JA, seg_pos={0 0}, upload=444303]: Done uploading piece. err: <nil> Dec 26 17:49:48.083197271 (506h20m47.954280307s) [noise=true , address=77.90.31.16:4736 , piece_id=IARA5RUI2HGVCDMQBC2E66OHLSA2NAPOS2GLZ3SMUPX536PDYOCA, seg_pos={0 0}, upload=444303]: Done uploading piece. err: <nil> Dec 26 17:49:48.141985204 (506h20m48.013068249s) [noise=true , address=45.91.116.12:4225 , piece_id=2JBAXZKMZPPGUACKEBK7M2QOU77X5G3A7NUNXDGRMWIS62COFKPA, seg_pos={0 0}, upload=444303]: Done uploading piece. err: <nil> Dec 26 17:49:48.144369094 (506h20m48.015452139s) [noise=true , address=145.79.227.41:4085 , piece_id=O2QUW63WBL5QCPWJMPKXLR7JFYKXOHDTE7FONSIO3C4KSWSG7FIA, seg_pos={0 0}, upload=444303]: Done uploading piece. err: <nil> Dec 26 17:49:48.171456162 (506h20m48.042539207s) [noise=true , address=45.91.116.11:4119 , piece_id=BQB6HVHKCV5MIX3J3N4IEG4K6PTQXHFVBFXHUC2RCHLSDYP6G4BA, seg_pos={0 0}, upload=444303]: Done uploading piece. err: <nil> Dec 26 17:49:48.171484226 (506h20m48.042567261s) [seg_pos={0 0}, upload=444303]: Segment reached optimal threshold of 13 pieces. Dec 26 17:49:48.171729252 (506h20m48.042812297s) [noise=true , address=145.79.227.17:4970 , piece_id=5PBTPJ7CASBOGJHB6RNOKKB4QORQTNOCPRJHLSWHDKNFCJNRAVRQ, seg_pos={0 0}, upload=444303]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe): protocol: write tcp 188.166.225.170:34658->145.79.227.17:4970: use of closed network connection; write tcp 188.166.225.170:34658->145.79.227.17:4970: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34658->145.79.227.17:4970: use of closed network connection Dec 26 17:49:48.171740664 (506h20m48.042823709s) [noise=true , address=145.79.227.17:4970 , piece_id=5PBTPJ7CASBOGJHB6RNOKKB4QORQTNOCPRJHLSWHDKNFCJNRAVRQ, seg_pos={0 0}, upload=444303]: Long tail cancellation due to optimal threshold Dec 26 17:49:48.171789948 (506h20m48.042872983s) [noise=true , address=145.79.227.44:4194 , piece_id=DN36VWFP3N2CMTWMOXAMX7ZZPLBS5ZCGPDLMIAYIG3QIWXBGO4QA, seg_pos={0 0}, upload=444303]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 26 17:49:48.171797983 (506h20m48.042881018s) [noise=true , address=145.79.227.44:4194 , piece_id=DN36VWFP3N2CMTWMOXAMX7ZZPLBS5ZCGPDLMIAYIG3QIWXBGO4QA, seg_pos={0 0}, upload=444303]: Long tail cancellation due to optimal threshold Dec 26 17:49:48.172073217 (506h20m48.043156253s) [noise=true , address=45.91.116.5:4297 , piece_id=U7IN34XCBKJ6UUOCW3YPYLJ7YNYPVAYACUO5H254KZFWSSTMZ63Q, seg_pos={0 0}, upload=444303]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 26 17:49:48.172080601 (506h20m48.043163636s) [noise=true , address=45.91.116.5:4297 , piece_id=U7IN34XCBKJ6UUOCW3YPYLJ7YNYPVAYACUO5H254KZFWSSTMZ63Q, seg_pos={0 0}, upload=444303]: Long tail cancellation due to optimal threshold Dec 26 17:49:48.172097663 (506h20m48.043180708s) [seg_pos={0 0}, upload=444303]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 17:49:48.172200780 (506h20m48.043283815s) [upload=444303]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:49:48.172210990 (506h20m48.043294035s) [upload=444303]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:49:48.474685714 (506h20m48.345768769s) [upload=444303]: Done uploading segments... Dec 26 17:49:48.474708668 (506h20m48.345791713s) [upload=444303]: Upload finished. err: <nil> Duration: 30.002823563s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [.X......XX..X....X.X.XX...XX...XX....XX..XX......X.XXX.....X] duration: 30.003380744s trace: https://jaeger.dev.storj.io/trace/3924adfdb49627e7 FAIL (2025-12-26 17:49:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: MTL6PGPEIXILZ7QNHK7CJQL7VEV7MPZD4ZPDF5OWLGLJPONHXVKA) context deadline exceeded; piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: 5GUNH3O2NJBNOUQIH3YJNLA7YI2RKWA2MEFK24KJLJLBFPRV7GLQ) context deadline exceeded ======LOGS====== Dec 26 17:49:15.289574008 (506h20m15.160657053s) [upload=444298]: Starting upload Dec 26 17:49:15.289667135 (506h20m15.16075017s) [upload=444298]: Done starting upload Dec 26 17:49:15.289718463 (506h20m15.160801498s) [upload=444298]: Uploading segments... Dec 26 17:49:15.289954463 (506h20m15.161037508s) [upload=444298]: Got next segment. Inline: false Dec 26 17:49:15.290000941 (506h20m15.161083996s) [upload=444298]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 17:49:15.290006361 (506h20m15.161089396s) [upload=444298]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 17:49:15.536008492 (506h20m15.407091547s) [seg_pos={0 0}, upload=444298]: Begin upload segment... Dec 26 17:49:15.536070210 (506h20m15.407153245s) [seg_pos={0 0}, upload=444298]: Done begin upload segment. Dec 26 17:49:15.536088765 (506h20m15.40717181s) [noise=true , address=45.91.116.9:4263 , piece_id=5GUNH3O2NJBNOUQIH3YJNLA7YI2RKWA2MEFK24KJLJLBFPRV7GLQ, seg_pos={0 0}, upload=444298]: Uploading piece... Dec 26 17:49:15.536114233 (506h20m15.407197268s) [noise=true , address=145.79.227.50:4375 , piece_id=HY5DOH636DYPS2UPKUPTA5L7O6BXUJVU2SVCPPYUQVAIQXXD7QJA, seg_pos={0 0}, upload=444298]: Uploading piece... Dec 26 17:49:15.536173667 (506h20m15.407256702s) [noise=true , address=145.79.227.49:4337 , piece_id=3FDPV4WTBLXQLAVKM5AZG6K57PZI3H7UNB5EYJE6SQSCL5DZLQWQ, seg_pos={0 0}, upload=444298]: Uploading piece... Dec 26 17:49:15.536609557 (506h20m15.407692602s) [noise=true , address=145.79.227.22:4511 , piece_id=IN72QA7IUWXLFIIPRA7ZDRY5OSVXCEZPBGB6DXWRBDXHFZB2GFHA, seg_pos={0 0}, upload=444298]: Uploading piece... Dec 26 17:49:15.537449126 (506h20m15.408532161s) [noise=true , address=45.91.116.6:4976 , piece_id=YZILLOJJQHM22YI63N6RQGLCKDFUBUMRM3H475YSDQJG3WBG3FJA, seg_pos={0 0}, upload=444298]: Uploading piece... Dec 26 17:49:15.537898542 (506h20m15.408981587s) [noise=true , address=77.90.31.14:4662 , piece_id=QN4JZEDFD6O73SOHQXGBV4HSJ5GXWUFN5KMA4NG64ICN5CMDC5LA, seg_pos={0 0}, upload=444298]: Uploading piece... Dec 26 17:49:15.538312891 (506h20m15.409395926s) [noise=true , address=145.79.227.46:4601 , piece_id=JOBBOXGBU2NMNYMBYK7A7ZSOWO375VKUY2RXQS7F5JATSOJBLIKA, seg_pos={0 0}, upload=444298]: Uploading piece... Dec 26 17:49:15.538786131 (506h20m15.409869176s) [noise=true , address=45.91.116.5:4300 , piece_id=MTL6PGPEIXILZ7QNHK7CJQL7VEV7MPZD4ZPDF5OWLGLJPONHXVKA, seg_pos={0 0}, upload=444298]: Uploading piece... Dec 26 17:49:15.539287657 (506h20m15.410370702s) [noise=true , address=145.79.227.4:4185 , piece_id=VTUPQDZ4ZE33VRBTERIJLVED7UI2Q5SWOQU7HSSAB63L4444DHOA, seg_pos={0 0}, upload=444298]: Uploading piece... Dec 26 17:49:15.539808367 (506h20m15.410891412s) [noise=true , address=145.79.227.51:4410 , piece_id=OBOHTWXWELFMY6RELPQEC46MQOSZWNTJXGKXKC43VM7LACHSZHDA, seg_pos={0 0}, upload=444298]: Uploading piece... Dec 26 17:49:15.540943649 (506h20m15.412026695s) [noise=true , address=45.91.116.10:4082 , piece_id=I7SNOQZP2ABOQCTJ4SJRM3UZKD32AZ2LENWSRTPEESX4IVNXVITQ, seg_pos={0 0}, upload=444298]: Uploading piece... Dec 26 17:49:15.541215267 (506h20m15.412298302s) [noise=true , address=145.79.227.14:4839 , piece_id=GHIQWMQPBJOY6UZP6S2BK5LIGAKBMOMBJGVTAVH47DWJPB5E53IA, seg_pos={0 0}, upload=444298]: Uploading piece... Dec 26 17:49:15.542706527 (506h20m15.413789572s) [noise=true , address=145.79.227.21:4475 , piece_id=TRQTFZ4MLBMU5VLWPEUVBR7IRTMFSEE6JQBQYMVLWHTNIVLDRWMQ, seg_pos={0 0}, upload=444298]: Uploading piece... Dec 26 17:49:15.542949750 (506h20m15.414032785s) [noise=true , address=145.79.227.37:4565 , piece_id=TBKSYNWDGU7LRU2P5NEJCZZDWUAJXJS7WK7TD66E6W5MZHXZ2FOA, seg_pos={0 0}, upload=444298]: Uploading piece... Dec 26 17:49:15.544793941 (506h20m15.415876987s) [noise=true , address=145.79.227.39:4013 , piece_id=F6QHPEDTSFUWOUI6CJNVH5WX76WM5FYJEHDT2OVMRLLESLNEONOQ, seg_pos={0 0}, upload=444298]: Uploading piece... Dec 26 17:49:15.546619567 (506h20m15.417702612s) [noise=true , address=145.79.227.16:4909 , piece_id=EYTTQTIDKRRJOCKR6G423QKGNT6XNL67RPZ3NO23HYL4HTGUX75Q, seg_pos={0 0}, upload=444298]: Uploading piece... Dec 26 17:49:17.246937189 (506h20m17.118020234s) [upload=444298]: Next returned nil segment Dec 26 17:49:17.246957578 (506h20m17.118040623s) [upload=444298]: Waiting for error group managing segments... Dec 26 17:49:17.529375160 (506h20m17.400458205s) [noise=true , address=145.79.227.50:4375 , piece_id=HY5DOH636DYPS2UPKUPTA5L7O6BXUJVU2SVCPPYUQVAIQXXD7QJA, seg_pos={0 0}, upload=444298]: Done uploading piece. err: <nil> Dec 26 17:49:17.542613168 (506h20m17.413696203s) [noise=true , address=145.79.227.49:4337 , piece_id=3FDPV4WTBLXQLAVKM5AZG6K57PZI3H7UNB5EYJE6SQSCL5DZLQWQ, seg_pos={0 0}, upload=444298]: Done uploading piece. err: <nil> Dec 26 17:49:17.545456011 (506h20m17.416539056s) [noise=true , address=145.79.227.39:4013 , piece_id=F6QHPEDTSFUWOUI6CJNVH5WX76WM5FYJEHDT2OVMRLLESLNEONOQ, seg_pos={0 0}, upload=444298]: Done uploading piece. err: <nil> Dec 26 17:49:17.546061995 (506h20m17.41714504s) [noise=true , address=145.79.227.21:4475 , piece_id=TRQTFZ4MLBMU5VLWPEUVBR7IRTMFSEE6JQBQYMVLWHTNIVLDRWMQ, seg_pos={0 0}, upload=444298]: Done uploading piece. err: <nil> Dec 26 17:49:17.546381954 (506h20m17.41746499s) [noise=true , address=145.79.227.51:4410 , piece_id=OBOHTWXWELFMY6RELPQEC46MQOSZWNTJXGKXKC43VM7LACHSZHDA, seg_pos={0 0}, upload=444298]: Done uploading piece. err: <nil> Dec 26 17:49:17.550722269 (506h20m17.421805305s) [noise=true , address=145.79.227.4:4185 , piece_id=VTUPQDZ4ZE33VRBTERIJLVED7UI2Q5SWOQU7HSSAB63L4444DHOA, seg_pos={0 0}, upload=444298]: Done uploading piece. err: <nil> Dec 26 17:49:17.554602247 (506h20m17.425685282s) [noise=true , address=145.79.227.22:4511 , piece_id=IN72QA7IUWXLFIIPRA7ZDRY5OSVXCEZPBGB6DXWRBDXHFZB2GFHA, seg_pos={0 0}, upload=444298]: Done uploading piece. err: <nil> Dec 26 17:49:17.568298828 (506h20m17.439381873s) [noise=true , address=145.79.227.37:4565 , piece_id=TBKSYNWDGU7LRU2P5NEJCZZDWUAJXJS7WK7TD66E6W5MZHXZ2FOA, seg_pos={0 0}, upload=444298]: Done uploading piece. err: <nil> Dec 26 17:49:17.570158719 (506h20m17.441241755s) [noise=true , address=145.79.227.46:4601 , piece_id=JOBBOXGBU2NMNYMBYK7A7ZSOWO375VKUY2RXQS7F5JATSOJBLIKA, seg_pos={0 0}, upload=444298]: Done uploading piece. err: <nil> Dec 26 17:49:17.585858535 (506h20m17.45694158s) [noise=true , address=145.79.227.16:4909 , piece_id=EYTTQTIDKRRJOCKR6G423QKGNT6XNL67RPZ3NO23HYL4HTGUX75Q, seg_pos={0 0}, upload=444298]: Done uploading piece. err: <nil> Dec 26 17:49:17.636854440 (506h20m17.507937475s) [noise=true , address=145.79.227.14:4839 , piece_id=GHIQWMQPBJOY6UZP6S2BK5LIGAKBMOMBJGVTAVH47DWJPB5E53IA, seg_pos={0 0}, upload=444298]: Done uploading piece. err: <nil> Dec 26 17:49:17.768189515 (506h20m17.63927255s) [noise=true , address=45.91.116.5:4300 , piece_id=MTL6PGPEIXILZ7QNHK7CJQL7VEV7MPZD4ZPDF5OWLGLJPONHXVKA, seg_pos={0 0}, upload=444298]: Done uploading piece. err: <nil> Dec 26 17:49:17.772556360 (506h20m17.643639395s) [noise=true , address=45.91.116.9:4263 , piece_id=5GUNH3O2NJBNOUQIH3YJNLA7YI2RKWA2MEFK24KJLJLBFPRV7GLQ, seg_pos={0 0}, upload=444298]: Done uploading piece. err: <nil> Dec 26 17:49:17.772575025 (506h20m17.64365807s) [seg_pos={0 0}, upload=444298]: Segment reached optimal threshold of 13 pieces. Dec 26 17:49:17.772782631 (506h20m17.643865676s) [noise=true , address=45.91.116.10:4082 , piece_id=I7SNOQZP2ABOQCTJ4SJRM3UZKD32AZ2LENWSRTPEESX4IVNXVITQ, seg_pos={0 0}, upload=444298]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 26 17:49:17.772790626 (506h20m17.643873661s) [noise=true , address=45.91.116.10:4082 , piece_id=I7SNOQZP2ABOQCTJ4SJRM3UZKD32AZ2LENWSRTPEESX4IVNXVITQ, seg_pos={0 0}, upload=444298]: Long tail cancellation due to optimal threshold Dec 26 17:49:17.772852263 (506h20m17.643935308s) [noise=true , address=45.91.116.6:4976 , piece_id=YZILLOJJQHM22YI63N6RQGLCKDFUBUMRM3H475YSDQJG3WBG3FJA, seg_pos={0 0}, upload=444298]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Dec 26 17:49:17.772863325 (506h20m17.64394637s) [noise=true , address=45.91.116.6:4976 , piece_id=YZILLOJJQHM22YI63N6RQGLCKDFUBUMRM3H475YSDQJG3WBG3FJA, seg_pos={0 0}, upload=444298]: Long tail cancellation due to optimal threshold Dec 26 17:49:17.773038038 (506h20m17.644121073s) [noise=true , address=77.90.31.14:4662 , piece_id=QN4JZEDFD6O73SOHQXGBV4HSJ5GXWUFN5KMA4NG64ICN5CMDC5LA, seg_pos={0 0}, upload=444298]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Dec 26 17:49:17.773045361 (506h20m17.644128396s) [noise=true , address=77.90.31.14:4662 , piece_id=QN4JZEDFD6O73SOHQXGBV4HSJ5GXWUFN5KMA4NG64ICN5CMDC5LA, seg_pos={0 0}, upload=444298]: Long tail cancellation due to optimal threshold Dec 26 17:49:17.773096729 (506h20m17.644179774s) [seg_pos={0 0}, upload=444298]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 17:49:17.773192491 (506h20m17.644275576s) [upload=444298]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:49:17.773206097 (506h20m17.644289182s) [upload=444298]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:49:18.065456723 (506h20m17.936539758s) [upload=444298]: Done uploading segments... Dec 26 17:49:18.065475199 (506h20m17.936558244s) [upload=444298]: Upload finished. err: <nil> Duration: 30.00341454s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 25.784887869s trace: https://jaeger.dev.storj.io/trace/665ea65f6e9ccc33 FAIL (2025-12-26 17:48:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== e=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789316979 (506h12m54.660400024s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789326507 (506h12m54.660409552s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789611339 (506h12m54.660694384s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789619275 (506h12m54.66070231s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789914006 (506h12m54.660997041s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789949443 (506h12m54.661032478s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790181566 (506h12m54.661264601s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790295632 (506h12m54.661378668s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790510872 (506h12m54.661593917s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790692980 (506h12m54.661776015s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790787379 (506h12m54.661870414s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790992970 (506h12m54.662076005s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791120053 (506h12m54.662203098s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791275218 (506h12m54.662358253s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791443709 (506h12m54.662526744s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:56.112313145 (506h12m55.98339619s) [upload=444191]: Next returned nil segment Dec 26 17:41:56.112330638 (506h12m55.983413683s) [upload=444191]: Waiting for error group managing segments... Dec 26 17:41:56.485322720 (506h12m56.356405765s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.485637079 (506h12m56.356720124s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.666807539 (506h12m56.537890584s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.678753987 (506h12m56.549837032s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.687788039 (506h12m56.558871074s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.694318023 (506h12m56.565401058s) [noise=true , address=145.79.227.7:4326 , piece_id=DMQ5RPI46X2U7TJTS42ZHQEF32BDJRKJJO5HWOYAPNS6VWVG5B5Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.716581187 (506h12m56.587664232s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.972183007 (506h12m56.843266042s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.011917999 (506h12m56.883001034s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.054168622 (506h12m56.925251667s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.114312777 (506h12m56.985395812s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.499846250 (506h12m57.370929285s) [noise=true , address=145.79.227.34:4919 , piece_id=YRWHW3BJVEGVJRSA6RJB3T7VJQSDTGKEZQKRICFIV2G3T7AN4MIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.612649066 (506h12m57.483732101s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649413733 (506h12m57.520496768s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649440282 (506h12m57.520523328s) [seg_pos={0 0}, upload=444191]: Segment reached optimal threshold of 14 pieces. Dec 26 17:41:57.649674729 (506h12m57.520757774s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G7ty71tPwe7cZSBNK9u74DFUCGomnxDsNvAif4C3SbP7HWwfo): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649684087 (506h12m57.520767142s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649747708 (506h12m57.520830763s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v6zXZuMmyL2MU8qb3q6gg1iwTYvndjzMjVpv8btEpeX9Lnqmu): protocol: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection Dec 26 17:41:57.649785219 (506h12m57.520868274s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649899828 (506h12m57.520982903s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649908775 (506h12m57.52099182s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650074641 (506h12m57.521157696s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.650083287 (506h12m57.521166342s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650132821 (506h12m57.521215866s) [seg_pos={0 0}, upload=444191]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 17:41:57.650220138 (506h12m57.521303183s) [upload=444191]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:41:57.650227171 (506h12m57.521310206s) [upload=444191]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:41:57.958890870 (506h12m57.829973915s) [upload=444191]: Done uploading segments... Dec 26 17:41:57.958911620 (506h12m57.829994665s) [upload=444191]: Upload finished. err: <nil> Duration: 25.784911104s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 33.5971771s trace: https://jaeger.dev.storj.io/trace/665ea65f6e9ccc33 FAIL (2025-12-26 17:48:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== e=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789316979 (506h12m54.660400024s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789326507 (506h12m54.660409552s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789611339 (506h12m54.660694384s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789619275 (506h12m54.66070231s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789914006 (506h12m54.660997041s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789949443 (506h12m54.661032478s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790181566 (506h12m54.661264601s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790295632 (506h12m54.661378668s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790510872 (506h12m54.661593917s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790692980 (506h12m54.661776015s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790787379 (506h12m54.661870414s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790992970 (506h12m54.662076005s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791120053 (506h12m54.662203098s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791275218 (506h12m54.662358253s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791443709 (506h12m54.662526744s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:56.112313145 (506h12m55.98339619s) [upload=444191]: Next returned nil segment Dec 26 17:41:56.112330638 (506h12m55.983413683s) [upload=444191]: Waiting for error group managing segments... Dec 26 17:41:56.485322720 (506h12m56.356405765s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.485637079 (506h12m56.356720124s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.666807539 (506h12m56.537890584s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.678753987 (506h12m56.549837032s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.687788039 (506h12m56.558871074s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.694318023 (506h12m56.565401058s) [noise=true , address=145.79.227.7:4326 , piece_id=DMQ5RPI46X2U7TJTS42ZHQEF32BDJRKJJO5HWOYAPNS6VWVG5B5Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.716581187 (506h12m56.587664232s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.972183007 (506h12m56.843266042s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.011917999 (506h12m56.883001034s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.054168622 (506h12m56.925251667s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.114312777 (506h12m56.985395812s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.499846250 (506h12m57.370929285s) [noise=true , address=145.79.227.34:4919 , piece_id=YRWHW3BJVEGVJRSA6RJB3T7VJQSDTGKEZQKRICFIV2G3T7AN4MIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.612649066 (506h12m57.483732101s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649413733 (506h12m57.520496768s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649440282 (506h12m57.520523328s) [seg_pos={0 0}, upload=444191]: Segment reached optimal threshold of 14 pieces. Dec 26 17:41:57.649674729 (506h12m57.520757774s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G7ty71tPwe7cZSBNK9u74DFUCGomnxDsNvAif4C3SbP7HWwfo): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649684087 (506h12m57.520767142s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649747708 (506h12m57.520830763s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v6zXZuMmyL2MU8qb3q6gg1iwTYvndjzMjVpv8btEpeX9Lnqmu): protocol: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection Dec 26 17:41:57.649785219 (506h12m57.520868274s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649899828 (506h12m57.520982903s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649908775 (506h12m57.52099182s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650074641 (506h12m57.521157696s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.650083287 (506h12m57.521166342s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650132821 (506h12m57.521215866s) [seg_pos={0 0}, upload=444191]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 17:41:57.650220138 (506h12m57.521303183s) [upload=444191]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:41:57.650227171 (506h12m57.521310206s) [upload=444191]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:41:57.958890870 (506h12m57.829973915s) [upload=444191]: Done uploading segments... Dec 26 17:41:57.958911620 (506h12m57.829994665s) [upload=444191]: Upload finished. err: <nil> Duration: 33.597191157s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [X.X......XX..X....X.X.XX...XX...XX....XX..XX......X.XXX.....] duration: 6.075010131s trace: https://jaeger.dev.storj.io/trace/3b761e3de974ea48 FAIL (2025-12-26 17:48:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA, Piece ID: LCYNTQGR2ZIUX6HBLYVSCUTCLAU6PFGHSA2ICB6DWE2ZFOI22NSA) context deadline exceeded; piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: DZPD74AMFXUMJYQEFLME4ZX7NRSLDIYE3ZBR5CIKSJE3XPXANUPA) context deadline exceeded ======LOGS====== Dec 26 17:43:10.974317425 (506h14m10.84540046s) [upload=444208]: Starting upload Dec 26 17:43:10.974446491 (506h14m10.845529526s) [upload=444208]: Done starting upload Dec 26 17:43:10.974581899 (506h14m10.845664944s) [upload=444208]: Uploading segments... Dec 26 17:43:10.974665328 (506h14m10.845748383s) [upload=444208]: Got next segment. Inline: false Dec 26 17:43:10.974721515 (506h14m10.84580456s) [upload=444208]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 17:43:10.974752394 (506h14m10.845835439s) [upload=444208]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 17:43:11.225224176 (506h14m11.096307221s) [seg_pos={0 0}, upload=444208]: Begin upload segment... Dec 26 17:43:11.225287356 (506h14m11.096370411s) [seg_pos={0 0}, upload=444208]: Done begin upload segment. Dec 26 17:43:11.225329857 (506h14m11.096412902s) [noise=true , address=145.79.227.46:4601 , piece_id=IM6NZ7Z4XQSEP4XG4WBWT4Y6P5DUX62KAMLQJANCBKSAU6ZVYRLA, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.225917767 (506h14m11.097000802s) [noise=true , address=145.79.227.8:4330 , piece_id=HEEUM6HSWGUYQFB2NP5T4DZST5GUJEDFNTLWDS6GEZC75P54UJQQ, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.231054878 (506h14m11.102137913s) [noise=true , address=145.79.227.49:4338 , piece_id=LNEIJJAJA7Q6J6ZGVQRPQFZELDCNKU2VKXX64K4ET6XVJ52LGVDQ, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.232622754 (506h14m11.103705799s) [noise=true , address=45.91.116.12:4228 , piece_id=LCYNTQGR2ZIUX6HBLYVSCUTCLAU6PFGHSA2ICB6DWE2ZFOI22NSA, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.234863150 (506h14m11.105946186s) [noise=true , address=145.79.227.45:4230 , piece_id=RKFJEJXGFXRGQZ2B3B3U3XZATBUAPFSV3IVYZQVIS63Q34RNCOSQ, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.236499346 (506h14m11.107582411s) [noise=true , address=45.91.116.6:4974 , piece_id=VKPME5624PBSFP3HLHKANNY7SBEVFL4Z636HEN2KXUQ55XPORJ2A, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.238064495 (506h14m11.10914753s) [noise=true , address=145.79.227.35:4481 , piece_id=AWRLZ2QPXDUTJWC2J6TCOM7QROIZJENTYBVWB6T2KPR2B6YYXS4A, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.243205044 (506h14m11.114288089s) [noise=true , address=145.79.227.37:4567 , piece_id=SLFOZCOSWTYSU5W3VXSJVHO6ZJSGWBUUMZKTWEV5YMDQONFHN53A, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.245389073 (506h14m11.116472108s) [noise=true , address=145.79.227.15:4874 , piece_id=TIWMO4OASWXEVFEG4COHHNDNGKHZUWZGTP6D6NNCR5PEZRSBYVIA, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.248134351 (506h14m11.119217426s) [noise=true , address=45.91.116.14:4191 , piece_id=DZPD74AMFXUMJYQEFLME4ZX7NRSLDIYE3ZBR5CIKSJE3XPXANUPA, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.251152537 (506h14m11.122235572s) [noise=true , address=145.79.227.3:4151 , piece_id=4QA7ARRZII6ZRW3GSYBHQVFBGDY4F676MVVJMQHMKWBZ2WS7P5VA, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.253552006 (506h14m11.124635051s) [noise=true , address=77.90.31.21:4880 , piece_id=KYMCS7O4EGMIVDSALBAUBNP7Q3TO5ZX4PLQ4KWBMKSWRUC7TM6EQ, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.255859751 (506h14m11.126942797s) [noise=true , address=145.79.227.22:4511 , piece_id=G7HZFZ7DF6NEWMGRUQQH4UOPUZIYVOVDH22TZNCDWTSQWB3ZEFGQ, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.256186012 (506h14m11.127269057s) [noise=true , address=145.79.227.48:4302 , piece_id=EGYOF4SZ3PKE3H6QYM3QHHRMITTGH55TPNY5D4NS5FSSTKLTML3Q, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.257647495 (506h14m11.12873056s) [noise=true , address=145.79.227.50:4373 , piece_id=DXRYJICGTQJBT7NDE6UUYPDPJZ2HD6MVGRTVZX5GAXXKJFSJ47FA, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.260156352 (506h14m11.131239398s) [noise=true , address=45.91.116.8:4048 , piece_id=JPMATYG4OTAACBWLQKX64FUDV7ONG4P4YZLF4SI4EVGBXOYSICLA, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:12.947257377 (506h14m12.818340412s) [upload=444208]: Next returned nil segment Dec 26 17:43:12.947333081 (506h14m12.818416126s) [upload=444208]: Waiting for error group managing segments... Dec 26 17:43:13.273287306 (506h14m13.144370351s) [noise=true , address=145.79.227.46:4601 , piece_id=IM6NZ7Z4XQSEP4XG4WBWT4Y6P5DUX62KAMLQJANCBKSAU6ZVYRLA, seg_pos={0 0}, upload=444208]: Done uploading piece. err: <nil> Dec 26 17:43:13.273494530 (506h14m13.144577575s) [noise=true , address=145.79.227.3:4151 , piece_id=4QA7ARRZII6ZRW3GSYBHQVFBGDY4F676MVVJMQHMKWBZ2WS7P5VA, seg_pos={0 0}, upload=444208]: Done uploading piece. err: <nil> Dec 26 17:43:13.275357928 (506h14m13.146440963s) [noise=true , address=145.79.227.15:4874 , piece_id=TIWMO4OASWXEVFEG4COHHNDNGKHZUWZGTP6D6NNCR5PEZRSBYVIA, seg_pos={0 0}, upload=444208]: Done uploading piece. err: <nil> Dec 26 17:43:13.276417696 (506h14m13.147500742s) [noise=true , address=145.79.227.45:4230 , piece_id=RKFJEJXGFXRGQZ2B3B3U3XZATBUAPFSV3IVYZQVIS63Q34RNCOSQ, seg_pos={0 0}, upload=444208]: Done uploading piece. err: <nil> Dec 26 17:43:13.277361283 (506h14m13.148444318s) [noise=true , address=145.79.227.50:4373 , piece_id=DXRYJICGTQJBT7NDE6UUYPDPJZ2HD6MVGRTVZX5GAXXKJFSJ47FA, seg_pos={0 0}, upload=444208]: Done uploading piece. err: <nil> Dec 26 17:43:13.277397332 (506h14m13.148480367s) [noise=true , address=145.79.227.37:4567 , piece_id=SLFOZCOSWTYSU5W3VXSJVHO6ZJSGWBUUMZKTWEV5YMDQONFHN53A, seg_pos={0 0}, upload=444208]: Done uploading piece. err: <nil> Dec 26 17:43:13.277910097 (506h14m13.148993142s) [noise=true , address=145.79.227.22:4511 , piece_id=G7HZFZ7DF6NEWMGRUQQH4UOPUZIYVOVDH22TZNCDWTSQWB3ZEFGQ, seg_pos={0 0}, upload=444208]: Done uploading piece. err: <nil> Dec 26 17:43:13.289773407 (506h14m13.160856452s) [noise=true , address=145.79.227.49:4338 , piece_id=LNEIJJAJA7Q6J6ZGVQRPQFZELDCNKU2VKXX64K4ET6XVJ52LGVDQ, seg_pos={0 0}, upload=444208]: Done uploading piece. err: <nil> Dec 26 17:43:13.290752031 (506h14m13.161835086s) [noise=true , address=145.79.227.48:4302 , piece_id=EGYOF4SZ3PKE3H6QYM3QHHRMITTGH55TPNY5D4NS5FSSTKLTML3Q, seg_pos={0 0}, upload=444208]: Done uploading piece. err: <nil> Dec 26 17:43:13.294786072 (506h14m13.165869117s) [noise=true , address=145.79.227.8:4330 , piece_id=HEEUM6HSWGUYQFB2NP5T4DZST5GUJEDFNTLWDS6GEZC75P54UJQQ, seg_pos={0 0}, upload=444208]: Done uploading piece. err: <nil> Dec 26 17:43:13.489465767 (506h14m13.360548802s) [noise=true , address=45.91.116.12:4228 , piece_id=LCYNTQGR2ZIUX6HBLYVSCUTCLAU6PFGHSA2ICB6DWE2ZFOI22NSA, seg_pos={0 0}, upload=444208]: Done uploading piece. err: <nil> Dec 26 17:43:13.566172019 (506h14m13.437255064s) [noise=true , address=45.91.116.14:4191 , piece_id=DZPD74AMFXUMJYQEFLME4ZX7NRSLDIYE3ZBR5CIKSJE3XPXANUPA, seg_pos={0 0}, upload=444208]: Done uploading piece. err: <nil> Dec 26 17:43:13.725036525 (506h14m13.59611956s) [noise=true , address=45.91.116.6:4974 , piece_id=VKPME5624PBSFP3HLHKANNY7SBEVFL4Z636HEN2KXUQ55XPORJ2A, seg_pos={0 0}, upload=444208]: Done uploading piece. err: <nil> Dec 26 17:43:13.725057735 (506h14m13.59614077s) [seg_pos={0 0}, upload=444208]: Segment reached optimal threshold of 13 pieces. Dec 26 17:43:13.725271673 (506h14m13.596354708s) [noise=true , address=145.79.227.35:4481 , piece_id=AWRLZ2QPXDUTJWC2J6TCOM7QROIZJENTYBVWB6T2KPR2B6YYXS4A, seg_pos={0 0}, upload=444208]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 26 17:43:13.725279558 (506h14m13.596362593s) [noise=true , address=145.79.227.35:4481 , piece_id=AWRLZ2QPXDUTJWC2J6TCOM7QROIZJENTYBVWB6T2KPR2B6YYXS4A, seg_pos={0 0}, upload=444208]: Long tail cancellation due to optimal threshold Dec 26 17:43:13.725338128 (506h14m13.596421174s) [noise=true , address=77.90.31.21:4880 , piece_id=KYMCS7O4EGMIVDSALBAUBNP7Q3TO5ZX4PLQ4KWBMKSWRUC7TM6EQ, seg_pos={0 0}, upload=444208]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: write tcp 188.166.225.170:37372->77.90.31.21:4880: use of closed network connection; write tcp 188.166.225.170:37372->77.90.31.21:4880: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37372->77.90.31.21:4880: use of closed network connection Dec 26 17:43:13.725346515 (506h14m13.59642955s) [noise=true , address=77.90.31.21:4880 , piece_id=KYMCS7O4EGMIVDSALBAUBNP7Q3TO5ZX4PLQ4KWBMKSWRUC7TM6EQ, seg_pos={0 0}, upload=444208]: Long tail cancellation due to optimal threshold Dec 26 17:43:13.725564680 (506h14m13.596647715s) [noise=true , address=45.91.116.8:4048 , piece_id=JPMATYG4OTAACBWLQKX64FUDV7ONG4P4YZLF4SI4EVGBXOYSICLA, seg_pos={0 0}, upload=444208]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: context canceled; context canceled Dec 26 17:43:13.725572675 (506h14m13.59665572s) [noise=true , address=45.91.116.8:4048 , piece_id=JPMATYG4OTAACBWLQKX64FUDV7ONG4P4YZLF4SI4EVGBXOYSICLA, seg_pos={0 0}, upload=444208]: Long tail cancellation due to optimal threshold Dec 26 17:43:13.725590309 (506h14m13.596673354s) [seg_pos={0 0}, upload=444208]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 17:43:13.725667807 (506h14m13.596750842s) [upload=444208]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:43:13.725675932 (506h14m13.596758977s) [upload=444208]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:43:14.020265812 (506h14m13.891348857s) [upload=444208]: Done uploading segments... Dec 26 17:43:14.020287173 (506h14m13.891370219s) [upload=444208]: Upload finished. err: <nil> Duration: 6.075079571s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 32.202732924s trace: https://jaeger.dev.storj.io/trace/665ea65f6e9ccc33 FAIL (2025-12-26 17:47:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== e=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789316979 (506h12m54.660400024s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789326507 (506h12m54.660409552s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789611339 (506h12m54.660694384s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789619275 (506h12m54.66070231s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789914006 (506h12m54.660997041s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789949443 (506h12m54.661032478s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790181566 (506h12m54.661264601s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790295632 (506h12m54.661378668s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790510872 (506h12m54.661593917s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790692980 (506h12m54.661776015s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790787379 (506h12m54.661870414s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790992970 (506h12m54.662076005s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791120053 (506h12m54.662203098s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791275218 (506h12m54.662358253s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791443709 (506h12m54.662526744s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:56.112313145 (506h12m55.98339619s) [upload=444191]: Next returned nil segment Dec 26 17:41:56.112330638 (506h12m55.983413683s) [upload=444191]: Waiting for error group managing segments... Dec 26 17:41:56.485322720 (506h12m56.356405765s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.485637079 (506h12m56.356720124s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.666807539 (506h12m56.537890584s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.678753987 (506h12m56.549837032s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.687788039 (506h12m56.558871074s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.694318023 (506h12m56.565401058s) [noise=true , address=145.79.227.7:4326 , piece_id=DMQ5RPI46X2U7TJTS42ZHQEF32BDJRKJJO5HWOYAPNS6VWVG5B5Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.716581187 (506h12m56.587664232s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.972183007 (506h12m56.843266042s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.011917999 (506h12m56.883001034s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.054168622 (506h12m56.925251667s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.114312777 (506h12m56.985395812s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.499846250 (506h12m57.370929285s) [noise=true , address=145.79.227.34:4919 , piece_id=YRWHW3BJVEGVJRSA6RJB3T7VJQSDTGKEZQKRICFIV2G3T7AN4MIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.612649066 (506h12m57.483732101s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649413733 (506h12m57.520496768s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649440282 (506h12m57.520523328s) [seg_pos={0 0}, upload=444191]: Segment reached optimal threshold of 14 pieces. Dec 26 17:41:57.649674729 (506h12m57.520757774s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G7ty71tPwe7cZSBNK9u74DFUCGomnxDsNvAif4C3SbP7HWwfo): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649684087 (506h12m57.520767142s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649747708 (506h12m57.520830763s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v6zXZuMmyL2MU8qb3q6gg1iwTYvndjzMjVpv8btEpeX9Lnqmu): protocol: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection Dec 26 17:41:57.649785219 (506h12m57.520868274s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649899828 (506h12m57.520982903s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649908775 (506h12m57.52099182s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650074641 (506h12m57.521157696s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.650083287 (506h12m57.521166342s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650132821 (506h12m57.521215866s) [seg_pos={0 0}, upload=444191]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 17:41:57.650220138 (506h12m57.521303183s) [upload=444191]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:41:57.650227171 (506h12m57.521310206s) [upload=444191]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:41:57.958890870 (506h12m57.829973915s) [upload=444191]: Done uploading segments... Dec 26 17:41:57.958911620 (506h12m57.829994665s) [upload=444191]: Upload finished. err: <nil> Duration: 32.202764103s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [.X.X......XX..X....X.X.XX...XX...XX....XX..XX......X.XXX....] duration: 6.854931927s trace: https://jaeger.dev.storj.io/trace/3b761e3de974ea48 FAIL (2025-12-26 17:47:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA, Piece ID: LCYNTQGR2ZIUX6HBLYVSCUTCLAU6PFGHSA2ICB6DWE2ZFOI22NSA) context deadline exceeded; piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: DZPD74AMFXUMJYQEFLME4ZX7NRSLDIYE3ZBR5CIKSJE3XPXANUPA) context deadline exceeded ======LOGS====== Dec 26 17:43:10.974317425 (506h14m10.84540046s) [upload=444208]: Starting upload Dec 26 17:43:10.974446491 (506h14m10.845529526s) [upload=444208]: Done starting upload Dec 26 17:43:10.974581899 (506h14m10.845664944s) [upload=444208]: Uploading segments... Dec 26 17:43:10.974665328 (506h14m10.845748383s) [upload=444208]: Got next segment. Inline: false Dec 26 17:43:10.974721515 (506h14m10.84580456s) [upload=444208]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 17:43:10.974752394 (506h14m10.845835439s) [upload=444208]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 17:43:11.225224176 (506h14m11.096307221s) [seg_pos={0 0}, upload=444208]: Begin upload segment... Dec 26 17:43:11.225287356 (506h14m11.096370411s) [seg_pos={0 0}, upload=444208]: Done begin upload segment. Dec 26 17:43:11.225329857 (506h14m11.096412902s) [noise=true , address=145.79.227.46:4601 , piece_id=IM6NZ7Z4XQSEP4XG4WBWT4Y6P5DUX62KAMLQJANCBKSAU6ZVYRLA, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.225917767 (506h14m11.097000802s) [noise=true , address=145.79.227.8:4330 , piece_id=HEEUM6HSWGUYQFB2NP5T4DZST5GUJEDFNTLWDS6GEZC75P54UJQQ, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.231054878 (506h14m11.102137913s) [noise=true , address=145.79.227.49:4338 , piece_id=LNEIJJAJA7Q6J6ZGVQRPQFZELDCNKU2VKXX64K4ET6XVJ52LGVDQ, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.232622754 (506h14m11.103705799s) [noise=true , address=45.91.116.12:4228 , piece_id=LCYNTQGR2ZIUX6HBLYVSCUTCLAU6PFGHSA2ICB6DWE2ZFOI22NSA, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.234863150 (506h14m11.105946186s) [noise=true , address=145.79.227.45:4230 , piece_id=RKFJEJXGFXRGQZ2B3B3U3XZATBUAPFSV3IVYZQVIS63Q34RNCOSQ, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.236499346 (506h14m11.107582411s) [noise=true , address=45.91.116.6:4974 , piece_id=VKPME5624PBSFP3HLHKANNY7SBEVFL4Z636HEN2KXUQ55XPORJ2A, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.238064495 (506h14m11.10914753s) [noise=true , address=145.79.227.35:4481 , piece_id=AWRLZ2QPXDUTJWC2J6TCOM7QROIZJENTYBVWB6T2KPR2B6YYXS4A, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.243205044 (506h14m11.114288089s) [noise=true , address=145.79.227.37:4567 , piece_id=SLFOZCOSWTYSU5W3VXSJVHO6ZJSGWBUUMZKTWEV5YMDQONFHN53A, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.245389073 (506h14m11.116472108s) [noise=true , address=145.79.227.15:4874 , piece_id=TIWMO4OASWXEVFEG4COHHNDNGKHZUWZGTP6D6NNCR5PEZRSBYVIA, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.248134351 (506h14m11.119217426s) [noise=true , address=45.91.116.14:4191 , piece_id=DZPD74AMFXUMJYQEFLME4ZX7NRSLDIYE3ZBR5CIKSJE3XPXANUPA, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.251152537 (506h14m11.122235572s) [noise=true , address=145.79.227.3:4151 , piece_id=4QA7ARRZII6ZRW3GSYBHQVFBGDY4F676MVVJMQHMKWBZ2WS7P5VA, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.253552006 (506h14m11.124635051s) [noise=true , address=77.90.31.21:4880 , piece_id=KYMCS7O4EGMIVDSALBAUBNP7Q3TO5ZX4PLQ4KWBMKSWRUC7TM6EQ, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.255859751 (506h14m11.126942797s) [noise=true , address=145.79.227.22:4511 , piece_id=G7HZFZ7DF6NEWMGRUQQH4UOPUZIYVOVDH22TZNCDWTSQWB3ZEFGQ, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.256186012 (506h14m11.127269057s) [noise=true , address=145.79.227.48:4302 , piece_id=EGYOF4SZ3PKE3H6QYM3QHHRMITTGH55TPNY5D4NS5FSSTKLTML3Q, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.257647495 (506h14m11.12873056s) [noise=true , address=145.79.227.50:4373 , piece_id=DXRYJICGTQJBT7NDE6UUYPDPJZ2HD6MVGRTVZX5GAXXKJFSJ47FA, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.260156352 (506h14m11.131239398s) [noise=true , address=45.91.116.8:4048 , piece_id=JPMATYG4OTAACBWLQKX64FUDV7ONG4P4YZLF4SI4EVGBXOYSICLA, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:12.947257377 (506h14m12.818340412s) [upload=444208]: Next returned nil segment Dec 26 17:43:12.947333081 (506h14m12.818416126s) [upload=444208]: Waiting for error group managing segments... Dec 26 17:43:13.273287306 (506h14m13.144370351s) [noise=true , address=145.79.227.46:4601 , piece_id=IM6NZ7Z4XQSEP4XG4WBWT4Y6P5DUX62KAMLQJANCBKSAU6ZVYRLA, seg_pos={0 0}, upload=444208]: Done uploading piece. err: <nil> Dec 26 17:43:13.273494530 (506h14m13.144577575s) [noise=true , address=145.79.227.3:4151 , piece_id=4QA7ARRZII6ZRW3GSYBHQVFBGDY4F676MVVJMQHMKWBZ2WS7P5VA, seg_pos={0 0}, upload=444208]: Done uploading piece. err: <nil> Dec 26 17:43:13.275357928 (506h14m13.146440963s) [noise=true , address=145.79.227.15:4874 , piece_id=TIWMO4OASWXEVFEG4COHHNDNGKHZUWZGTP6D6NNCR5PEZRSBYVIA, seg_pos={0 0}, upload=444208]: Done uploading piece. err: <nil> Dec 26 17:43:13.276417696 (506h14m13.147500742s) [noise=true , address=145.79.227.45:4230 , piece_id=RKFJEJXGFXRGQZ2B3B3U3XZATBUAPFSV3IVYZQVIS63Q34RNCOSQ, seg_pos={0 0}, upload=444208]: Done uploading piece. err: <nil> Dec 26 17:43:13.277361283 (506h14m13.148444318s) [noise=true , address=145.79.227.50:4373 , piece_id=DXRYJICGTQJBT7NDE6UUYPDPJZ2HD6MVGRTVZX5GAXXKJFSJ47FA, seg_pos={0 0}, upload=444208]: Done uploading piece. err: <nil> Dec 26 17:43:13.277397332 (506h14m13.148480367s) [noise=true , address=145.79.227.37:4567 , piece_id=SLFOZCOSWTYSU5W3VXSJVHO6ZJSGWBUUMZKTWEV5YMDQONFHN53A, seg_pos={0 0}, upload=444208]: Done uploading piece. err: <nil> Dec 26 17:43:13.277910097 (506h14m13.148993142s) [noise=true , address=145.79.227.22:4511 , piece_id=G7HZFZ7DF6NEWMGRUQQH4UOPUZIYVOVDH22TZNCDWTSQWB3ZEFGQ, seg_pos={0 0}, upload=444208]: Done uploading piece. err: <nil> Dec 26 17:43:13.289773407 (506h14m13.160856452s) [noise=true , address=145.79.227.49:4338 , piece_id=LNEIJJAJA7Q6J6ZGVQRPQFZELDCNKU2VKXX64K4ET6XVJ52LGVDQ, seg_pos={0 0}, upload=444208]: Done uploading piece. err: <nil> Dec 26 17:43:13.290752031 (506h14m13.161835086s) [noise=true , address=145.79.227.48:4302 , piece_id=EGYOF4SZ3PKE3H6QYM3QHHRMITTGH55TPNY5D4NS5FSSTKLTML3Q, seg_pos={0 0}, upload=444208]: Done uploading piece. err: <nil> Dec 26 17:43:13.294786072 (506h14m13.165869117s) [noise=true , address=145.79.227.8:4330 , piece_id=HEEUM6HSWGUYQFB2NP5T4DZST5GUJEDFNTLWDS6GEZC75P54UJQQ, seg_pos={0 0}, upload=444208]: Done uploading piece. err: <nil> Dec 26 17:43:13.489465767 (506h14m13.360548802s) [noise=true , address=45.91.116.12:4228 , piece_id=LCYNTQGR2ZIUX6HBLYVSCUTCLAU6PFGHSA2ICB6DWE2ZFOI22NSA, seg_pos={0 0}, upload=444208]: Done uploading piece. err: <nil> Dec 26 17:43:13.566172019 (506h14m13.437255064s) [noise=true , address=45.91.116.14:4191 , piece_id=DZPD74AMFXUMJYQEFLME4ZX7NRSLDIYE3ZBR5CIKSJE3XPXANUPA, seg_pos={0 0}, upload=444208]: Done uploading piece. err: <nil> Dec 26 17:43:13.725036525 (506h14m13.59611956s) [noise=true , address=45.91.116.6:4974 , piece_id=VKPME5624PBSFP3HLHKANNY7SBEVFL4Z636HEN2KXUQ55XPORJ2A, seg_pos={0 0}, upload=444208]: Done uploading piece. err: <nil> Dec 26 17:43:13.725057735 (506h14m13.59614077s) [seg_pos={0 0}, upload=444208]: Segment reached optimal threshold of 13 pieces. Dec 26 17:43:13.725271673 (506h14m13.596354708s) [noise=true , address=145.79.227.35:4481 , piece_id=AWRLZ2QPXDUTJWC2J6TCOM7QROIZJENTYBVWB6T2KPR2B6YYXS4A, seg_pos={0 0}, upload=444208]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 26 17:43:13.725279558 (506h14m13.596362593s) [noise=true , address=145.79.227.35:4481 , piece_id=AWRLZ2QPXDUTJWC2J6TCOM7QROIZJENTYBVWB6T2KPR2B6YYXS4A, seg_pos={0 0}, upload=444208]: Long tail cancellation due to optimal threshold Dec 26 17:43:13.725338128 (506h14m13.596421174s) [noise=true , address=77.90.31.21:4880 , piece_id=KYMCS7O4EGMIVDSALBAUBNP7Q3TO5ZX4PLQ4KWBMKSWRUC7TM6EQ, seg_pos={0 0}, upload=444208]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: write tcp 188.166.225.170:37372->77.90.31.21:4880: use of closed network connection; write tcp 188.166.225.170:37372->77.90.31.21:4880: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37372->77.90.31.21:4880: use of closed network connection Dec 26 17:43:13.725346515 (506h14m13.59642955s) [noise=true , address=77.90.31.21:4880 , piece_id=KYMCS7O4EGMIVDSALBAUBNP7Q3TO5ZX4PLQ4KWBMKSWRUC7TM6EQ, seg_pos={0 0}, upload=444208]: Long tail cancellation due to optimal threshold Dec 26 17:43:13.725564680 (506h14m13.596647715s) [noise=true , address=45.91.116.8:4048 , piece_id=JPMATYG4OTAACBWLQKX64FUDV7ONG4P4YZLF4SI4EVGBXOYSICLA, seg_pos={0 0}, upload=444208]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: context canceled; context canceled Dec 26 17:43:13.725572675 (506h14m13.59665572s) [noise=true , address=45.91.116.8:4048 , piece_id=JPMATYG4OTAACBWLQKX64FUDV7ONG4P4YZLF4SI4EVGBXOYSICLA, seg_pos={0 0}, upload=444208]: Long tail cancellation due to optimal threshold Dec 26 17:43:13.725590309 (506h14m13.596673354s) [seg_pos={0 0}, upload=444208]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 17:43:13.725667807 (506h14m13.596750842s) [upload=444208]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:43:13.725675932 (506h14m13.596758977s) [upload=444208]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:43:14.020265812 (506h14m13.891348857s) [upload=444208]: Done uploading segments... Dec 26 17:43:14.020287173 (506h14m13.891370219s) [upload=444208]: Upload finished. err: <nil> Duration: 6.8550234s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 45.875667435s trace: https://jaeger.dev.storj.io/trace/665ea65f6e9ccc33 FAIL (2025-12-26 17:46:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== e=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789316979 (506h12m54.660400024s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789326507 (506h12m54.660409552s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789611339 (506h12m54.660694384s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789619275 (506h12m54.66070231s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789914006 (506h12m54.660997041s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789949443 (506h12m54.661032478s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790181566 (506h12m54.661264601s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790295632 (506h12m54.661378668s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790510872 (506h12m54.661593917s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790692980 (506h12m54.661776015s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790787379 (506h12m54.661870414s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790992970 (506h12m54.662076005s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791120053 (506h12m54.662203098s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791275218 (506h12m54.662358253s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791443709 (506h12m54.662526744s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:56.112313145 (506h12m55.98339619s) [upload=444191]: Next returned nil segment Dec 26 17:41:56.112330638 (506h12m55.983413683s) [upload=444191]: Waiting for error group managing segments... Dec 26 17:41:56.485322720 (506h12m56.356405765s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.485637079 (506h12m56.356720124s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.666807539 (506h12m56.537890584s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.678753987 (506h12m56.549837032s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.687788039 (506h12m56.558871074s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.694318023 (506h12m56.565401058s) [noise=true , address=145.79.227.7:4326 , piece_id=DMQ5RPI46X2U7TJTS42ZHQEF32BDJRKJJO5HWOYAPNS6VWVG5B5Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.716581187 (506h12m56.587664232s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.972183007 (506h12m56.843266042s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.011917999 (506h12m56.883001034s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.054168622 (506h12m56.925251667s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.114312777 (506h12m56.985395812s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.499846250 (506h12m57.370929285s) [noise=true , address=145.79.227.34:4919 , piece_id=YRWHW3BJVEGVJRSA6RJB3T7VJQSDTGKEZQKRICFIV2G3T7AN4MIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.612649066 (506h12m57.483732101s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649413733 (506h12m57.520496768s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649440282 (506h12m57.520523328s) [seg_pos={0 0}, upload=444191]: Segment reached optimal threshold of 14 pieces. Dec 26 17:41:57.649674729 (506h12m57.520757774s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G7ty71tPwe7cZSBNK9u74DFUCGomnxDsNvAif4C3SbP7HWwfo): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649684087 (506h12m57.520767142s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649747708 (506h12m57.520830763s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v6zXZuMmyL2MU8qb3q6gg1iwTYvndjzMjVpv8btEpeX9Lnqmu): protocol: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection Dec 26 17:41:57.649785219 (506h12m57.520868274s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649899828 (506h12m57.520982903s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649908775 (506h12m57.52099182s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650074641 (506h12m57.521157696s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.650083287 (506h12m57.521166342s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650132821 (506h12m57.521215866s) [seg_pos={0 0}, upload=444191]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 17:41:57.650220138 (506h12m57.521303183s) [upload=444191]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:41:57.650227171 (506h12m57.521310206s) [upload=444191]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:41:57.958890870 (506h12m57.829973915s) [upload=444191]: Done uploading segments... Dec 26 17:41:57.958911620 (506h12m57.829994665s) [upload=444191]: Upload finished. err: <nil> Duration: 45.875693586s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [..X.X......XX..X....X.X.XX...XX...XX....XX..XX......X.XXX...] duration: 5.98046039s trace: https://jaeger.dev.storj.io/trace/3b761e3de974ea48 FAIL (2025-12-26 17:46:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA, Piece ID: LCYNTQGR2ZIUX6HBLYVSCUTCLAU6PFGHSA2ICB6DWE2ZFOI22NSA) context deadline exceeded; piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: DZPD74AMFXUMJYQEFLME4ZX7NRSLDIYE3ZBR5CIKSJE3XPXANUPA) context deadline exceeded ======LOGS====== Dec 26 17:43:10.974317425 (506h14m10.84540046s) [upload=444208]: Starting upload Dec 26 17:43:10.974446491 (506h14m10.845529526s) [upload=444208]: Done starting upload Dec 26 17:43:10.974581899 (506h14m10.845664944s) [upload=444208]: Uploading segments... Dec 26 17:43:10.974665328 (506h14m10.845748383s) [upload=444208]: Got next segment. Inline: false Dec 26 17:43:10.974721515 (506h14m10.84580456s) [upload=444208]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 17:43:10.974752394 (506h14m10.845835439s) [upload=444208]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 17:43:11.225224176 (506h14m11.096307221s) [seg_pos={0 0}, upload=444208]: Begin upload segment... Dec 26 17:43:11.225287356 (506h14m11.096370411s) [seg_pos={0 0}, upload=444208]: Done begin upload segment. Dec 26 17:43:11.225329857 (506h14m11.096412902s) [noise=true , address=145.79.227.46:4601 , piece_id=IM6NZ7Z4XQSEP4XG4WBWT4Y6P5DUX62KAMLQJANCBKSAU6ZVYRLA, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.225917767 (506h14m11.097000802s) [noise=true , address=145.79.227.8:4330 , piece_id=HEEUM6HSWGUYQFB2NP5T4DZST5GUJEDFNTLWDS6GEZC75P54UJQQ, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.231054878 (506h14m11.102137913s) [noise=true , address=145.79.227.49:4338 , piece_id=LNEIJJAJA7Q6J6ZGVQRPQFZELDCNKU2VKXX64K4ET6XVJ52LGVDQ, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.232622754 (506h14m11.103705799s) [noise=true , address=45.91.116.12:4228 , piece_id=LCYNTQGR2ZIUX6HBLYVSCUTCLAU6PFGHSA2ICB6DWE2ZFOI22NSA, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.234863150 (506h14m11.105946186s) [noise=true , address=145.79.227.45:4230 , piece_id=RKFJEJXGFXRGQZ2B3B3U3XZATBUAPFSV3IVYZQVIS63Q34RNCOSQ, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.236499346 (506h14m11.107582411s) [noise=true , address=45.91.116.6:4974 , piece_id=VKPME5624PBSFP3HLHKANNY7SBEVFL4Z636HEN2KXUQ55XPORJ2A, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.238064495 (506h14m11.10914753s) [noise=true , address=145.79.227.35:4481 , piece_id=AWRLZ2QPXDUTJWC2J6TCOM7QROIZJENTYBVWB6T2KPR2B6YYXS4A, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.243205044 (506h14m11.114288089s) [noise=true , address=145.79.227.37:4567 , piece_id=SLFOZCOSWTYSU5W3VXSJVHO6ZJSGWBUUMZKTWEV5YMDQONFHN53A, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.245389073 (506h14m11.116472108s) [noise=true , address=145.79.227.15:4874 , piece_id=TIWMO4OASWXEVFEG4COHHNDNGKHZUWZGTP6D6NNCR5PEZRSBYVIA, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.248134351 (506h14m11.119217426s) [noise=true , address=45.91.116.14:4191 , piece_id=DZPD74AMFXUMJYQEFLME4ZX7NRSLDIYE3ZBR5CIKSJE3XPXANUPA, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.251152537 (506h14m11.122235572s) [noise=true , address=145.79.227.3:4151 , piece_id=4QA7ARRZII6ZRW3GSYBHQVFBGDY4F676MVVJMQHMKWBZ2WS7P5VA, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.253552006 (506h14m11.124635051s) [noise=true , address=77.90.31.21:4880 , piece_id=KYMCS7O4EGMIVDSALBAUBNP7Q3TO5ZX4PLQ4KWBMKSWRUC7TM6EQ, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.255859751 (506h14m11.126942797s) [noise=true , address=145.79.227.22:4511 , piece_id=G7HZFZ7DF6NEWMGRUQQH4UOPUZIYVOVDH22TZNCDWTSQWB3ZEFGQ, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.256186012 (506h14m11.127269057s) [noise=true , address=145.79.227.48:4302 , piece_id=EGYOF4SZ3PKE3H6QYM3QHHRMITTGH55TPNY5D4NS5FSSTKLTML3Q, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.257647495 (506h14m11.12873056s) [noise=true , address=145.79.227.50:4373 , piece_id=DXRYJICGTQJBT7NDE6UUYPDPJZ2HD6MVGRTVZX5GAXXKJFSJ47FA, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.260156352 (506h14m11.131239398s) [noise=true , address=45.91.116.8:4048 , piece_id=JPMATYG4OTAACBWLQKX64FUDV7ONG4P4YZLF4SI4EVGBXOYSICLA, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:12.947257377 (506h14m12.818340412s) [upload=444208]: Next returned nil segment Dec 26 17:43:12.947333081 (506h14m12.818416126s) [upload=444208]: Waiting for error group managing segments... Dec 26 17:43:13.273287306 (506h14m13.144370351s) [noise=true , address=145.79.227.46:4601 , piece_id=IM6NZ7Z4XQSEP4XG4WBWT4Y6P5DUX62KAMLQJANCBKSAU6ZVYRLA, seg_pos={0 0}, upload=444208]: Done uploading piece. err: <nil> Dec 26 17:43:13.273494530 (506h14m13.144577575s) [noise=true , address=145.79.227.3:4151 , piece_id=4QA7ARRZII6ZRW3GSYBHQVFBGDY4F676MVVJMQHMKWBZ2WS7P5VA, seg_pos={0 0}, upload=444208]: Done uploading piece. err: <nil> Dec 26 17:43:13.275357928 (506h14m13.146440963s) [noise=true , address=145.79.227.15:4874 , piece_id=TIWMO4OASWXEVFEG4COHHNDNGKHZUWZGTP6D6NNCR5PEZRSBYVIA, seg_pos={0 0}, upload=444208]: Done uploading piece. err: <nil> Dec 26 17:43:13.276417696 (506h14m13.147500742s) [noise=true , address=145.79.227.45:4230 , piece_id=RKFJEJXGFXRGQZ2B3B3U3XZATBUAPFSV3IVYZQVIS63Q34RNCOSQ, seg_pos={0 0}, upload=444208]: Done uploading piece. err: <nil> Dec 26 17:43:13.277361283 (506h14m13.148444318s) [noise=true , address=145.79.227.50:4373 , piece_id=DXRYJICGTQJBT7NDE6UUYPDPJZ2HD6MVGRTVZX5GAXXKJFSJ47FA, seg_pos={0 0}, upload=444208]: Done uploading piece. err: <nil> Dec 26 17:43:13.277397332 (506h14m13.148480367s) [noise=true , address=145.79.227.37:4567 , piece_id=SLFOZCOSWTYSU5W3VXSJVHO6ZJSGWBUUMZKTWEV5YMDQONFHN53A, seg_pos={0 0}, upload=444208]: Done uploading piece. err: <nil> Dec 26 17:43:13.277910097 (506h14m13.148993142s) [noise=true , address=145.79.227.22:4511 , piece_id=G7HZFZ7DF6NEWMGRUQQH4UOPUZIYVOVDH22TZNCDWTSQWB3ZEFGQ, seg_pos={0 0}, upload=444208]: Done uploading piece. err: <nil> Dec 26 17:43:13.289773407 (506h14m13.160856452s) [noise=true , address=145.79.227.49:4338 , piece_id=LNEIJJAJA7Q6J6ZGVQRPQFZELDCNKU2VKXX64K4ET6XVJ52LGVDQ, seg_pos={0 0}, upload=444208]: Done uploading piece. err: <nil> Dec 26 17:43:13.290752031 (506h14m13.161835086s) [noise=true , address=145.79.227.48:4302 , piece_id=EGYOF4SZ3PKE3H6QYM3QHHRMITTGH55TPNY5D4NS5FSSTKLTML3Q, seg_pos={0 0}, upload=444208]: Done uploading piece. err: <nil> Dec 26 17:43:13.294786072 (506h14m13.165869117s) [noise=true , address=145.79.227.8:4330 , piece_id=HEEUM6HSWGUYQFB2NP5T4DZST5GUJEDFNTLWDS6GEZC75P54UJQQ, seg_pos={0 0}, upload=444208]: Done uploading piece. err: <nil> Dec 26 17:43:13.489465767 (506h14m13.360548802s) [noise=true , address=45.91.116.12:4228 , piece_id=LCYNTQGR2ZIUX6HBLYVSCUTCLAU6PFGHSA2ICB6DWE2ZFOI22NSA, seg_pos={0 0}, upload=444208]: Done uploading piece. err: <nil> Dec 26 17:43:13.566172019 (506h14m13.437255064s) [noise=true , address=45.91.116.14:4191 , piece_id=DZPD74AMFXUMJYQEFLME4ZX7NRSLDIYE3ZBR5CIKSJE3XPXANUPA, seg_pos={0 0}, upload=444208]: Done uploading piece. err: <nil> Dec 26 17:43:13.725036525 (506h14m13.59611956s) [noise=true , address=45.91.116.6:4974 , piece_id=VKPME5624PBSFP3HLHKANNY7SBEVFL4Z636HEN2KXUQ55XPORJ2A, seg_pos={0 0}, upload=444208]: Done uploading piece. err: <nil> Dec 26 17:43:13.725057735 (506h14m13.59614077s) [seg_pos={0 0}, upload=444208]: Segment reached optimal threshold of 13 pieces. Dec 26 17:43:13.725271673 (506h14m13.596354708s) [noise=true , address=145.79.227.35:4481 , piece_id=AWRLZ2QPXDUTJWC2J6TCOM7QROIZJENTYBVWB6T2KPR2B6YYXS4A, seg_pos={0 0}, upload=444208]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 26 17:43:13.725279558 (506h14m13.596362593s) [noise=true , address=145.79.227.35:4481 , piece_id=AWRLZ2QPXDUTJWC2J6TCOM7QROIZJENTYBVWB6T2KPR2B6YYXS4A, seg_pos={0 0}, upload=444208]: Long tail cancellation due to optimal threshold Dec 26 17:43:13.725338128 (506h14m13.596421174s) [noise=true , address=77.90.31.21:4880 , piece_id=KYMCS7O4EGMIVDSALBAUBNP7Q3TO5ZX4PLQ4KWBMKSWRUC7TM6EQ, seg_pos={0 0}, upload=444208]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: write tcp 188.166.225.170:37372->77.90.31.21:4880: use of closed network connection; write tcp 188.166.225.170:37372->77.90.31.21:4880: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37372->77.90.31.21:4880: use of closed network connection Dec 26 17:43:13.725346515 (506h14m13.59642955s) [noise=true , address=77.90.31.21:4880 , piece_id=KYMCS7O4EGMIVDSALBAUBNP7Q3TO5ZX4PLQ4KWBMKSWRUC7TM6EQ, seg_pos={0 0}, upload=444208]: Long tail cancellation due to optimal threshold Dec 26 17:43:13.725564680 (506h14m13.596647715s) [noise=true , address=45.91.116.8:4048 , piece_id=JPMATYG4OTAACBWLQKX64FUDV7ONG4P4YZLF4SI4EVGBXOYSICLA, seg_pos={0 0}, upload=444208]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: context canceled; context canceled Dec 26 17:43:13.725572675 (506h14m13.59665572s) [noise=true , address=45.91.116.8:4048 , piece_id=JPMATYG4OTAACBWLQKX64FUDV7ONG4P4YZLF4SI4EVGBXOYSICLA, seg_pos={0 0}, upload=444208]: Long tail cancellation due to optimal threshold Dec 26 17:43:13.725590309 (506h14m13.596673354s) [seg_pos={0 0}, upload=444208]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 17:43:13.725667807 (506h14m13.596750842s) [upload=444208]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:43:13.725675932 (506h14m13.596758977s) [upload=444208]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:43:14.020265812 (506h14m13.891348857s) [upload=444208]: Done uploading segments... Dec 26 17:43:14.020287173 (506h14m13.891370219s) [upload=444208]: Upload finished. err: <nil> Duration: 5.980493884s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 29.486971962s trace: https://jaeger.dev.storj.io/trace/665ea65f6e9ccc33 FAIL (2025-12-26 17:46:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== e=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789316979 (506h12m54.660400024s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789326507 (506h12m54.660409552s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789611339 (506h12m54.660694384s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789619275 (506h12m54.66070231s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789914006 (506h12m54.660997041s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789949443 (506h12m54.661032478s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790181566 (506h12m54.661264601s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790295632 (506h12m54.661378668s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790510872 (506h12m54.661593917s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790692980 (506h12m54.661776015s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790787379 (506h12m54.661870414s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790992970 (506h12m54.662076005s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791120053 (506h12m54.662203098s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791275218 (506h12m54.662358253s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791443709 (506h12m54.662526744s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:56.112313145 (506h12m55.98339619s) [upload=444191]: Next returned nil segment Dec 26 17:41:56.112330638 (506h12m55.983413683s) [upload=444191]: Waiting for error group managing segments... Dec 26 17:41:56.485322720 (506h12m56.356405765s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.485637079 (506h12m56.356720124s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.666807539 (506h12m56.537890584s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.678753987 (506h12m56.549837032s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.687788039 (506h12m56.558871074s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.694318023 (506h12m56.565401058s) [noise=true , address=145.79.227.7:4326 , piece_id=DMQ5RPI46X2U7TJTS42ZHQEF32BDJRKJJO5HWOYAPNS6VWVG5B5Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.716581187 (506h12m56.587664232s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.972183007 (506h12m56.843266042s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.011917999 (506h12m56.883001034s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.054168622 (506h12m56.925251667s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.114312777 (506h12m56.985395812s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.499846250 (506h12m57.370929285s) [noise=true , address=145.79.227.34:4919 , piece_id=YRWHW3BJVEGVJRSA6RJB3T7VJQSDTGKEZQKRICFIV2G3T7AN4MIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.612649066 (506h12m57.483732101s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649413733 (506h12m57.520496768s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649440282 (506h12m57.520523328s) [seg_pos={0 0}, upload=444191]: Segment reached optimal threshold of 14 pieces. Dec 26 17:41:57.649674729 (506h12m57.520757774s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G7ty71tPwe7cZSBNK9u74DFUCGomnxDsNvAif4C3SbP7HWwfo): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649684087 (506h12m57.520767142s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649747708 (506h12m57.520830763s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v6zXZuMmyL2MU8qb3q6gg1iwTYvndjzMjVpv8btEpeX9Lnqmu): protocol: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection Dec 26 17:41:57.649785219 (506h12m57.520868274s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649899828 (506h12m57.520982903s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649908775 (506h12m57.52099182s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650074641 (506h12m57.521157696s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.650083287 (506h12m57.521166342s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650132821 (506h12m57.521215866s) [seg_pos={0 0}, upload=444191]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 17:41:57.650220138 (506h12m57.521303183s) [upload=444191]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:41:57.650227171 (506h12m57.521310206s) [upload=444191]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:41:57.958890870 (506h12m57.829973915s) [upload=444191]: Done uploading segments... Dec 26 17:41:57.958911620 (506h12m57.829994665s) [upload=444191]: Upload finished. err: <nil> Duration: 29.486991309s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [X..X.X......XX..X....X.X.XX...XX...XX....XX..XX......X.XXX..] duration: 6.038367596s trace: https://jaeger.dev.storj.io/trace/3b761e3de974ea48 FAIL (2025-12-26 17:45:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA, Piece ID: LCYNTQGR2ZIUX6HBLYVSCUTCLAU6PFGHSA2ICB6DWE2ZFOI22NSA) context deadline exceeded; piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: DZPD74AMFXUMJYQEFLME4ZX7NRSLDIYE3ZBR5CIKSJE3XPXANUPA) context deadline exceeded ======LOGS====== Dec 26 17:43:10.974317425 (506h14m10.84540046s) [upload=444208]: Starting upload Dec 26 17:43:10.974446491 (506h14m10.845529526s) [upload=444208]: Done starting upload Dec 26 17:43:10.974581899 (506h14m10.845664944s) [upload=444208]: Uploading segments... Dec 26 17:43:10.974665328 (506h14m10.845748383s) [upload=444208]: Got next segment. Inline: false Dec 26 17:43:10.974721515 (506h14m10.84580456s) [upload=444208]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 17:43:10.974752394 (506h14m10.845835439s) [upload=444208]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 17:43:11.225224176 (506h14m11.096307221s) [seg_pos={0 0}, upload=444208]: Begin upload segment... Dec 26 17:43:11.225287356 (506h14m11.096370411s) [seg_pos={0 0}, upload=444208]: Done begin upload segment. Dec 26 17:43:11.225329857 (506h14m11.096412902s) [noise=true , address=145.79.227.46:4601 , piece_id=IM6NZ7Z4XQSEP4XG4WBWT4Y6P5DUX62KAMLQJANCBKSAU6ZVYRLA, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.225917767 (506h14m11.097000802s) [noise=true , address=145.79.227.8:4330 , piece_id=HEEUM6HSWGUYQFB2NP5T4DZST5GUJEDFNTLWDS6GEZC75P54UJQQ, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.231054878 (506h14m11.102137913s) [noise=true , address=145.79.227.49:4338 , piece_id=LNEIJJAJA7Q6J6ZGVQRPQFZELDCNKU2VKXX64K4ET6XVJ52LGVDQ, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.232622754 (506h14m11.103705799s) [noise=true , address=45.91.116.12:4228 , piece_id=LCYNTQGR2ZIUX6HBLYVSCUTCLAU6PFGHSA2ICB6DWE2ZFOI22NSA, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.234863150 (506h14m11.105946186s) [noise=true , address=145.79.227.45:4230 , piece_id=RKFJEJXGFXRGQZ2B3B3U3XZATBUAPFSV3IVYZQVIS63Q34RNCOSQ, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.236499346 (506h14m11.107582411s) [noise=true , address=45.91.116.6:4974 , piece_id=VKPME5624PBSFP3HLHKANNY7SBEVFL4Z636HEN2KXUQ55XPORJ2A, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.238064495 (506h14m11.10914753s) [noise=true , address=145.79.227.35:4481 , piece_id=AWRLZ2QPXDUTJWC2J6TCOM7QROIZJENTYBVWB6T2KPR2B6YYXS4A, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.243205044 (506h14m11.114288089s) [noise=true , address=145.79.227.37:4567 , piece_id=SLFOZCOSWTYSU5W3VXSJVHO6ZJSGWBUUMZKTWEV5YMDQONFHN53A, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.245389073 (506h14m11.116472108s) [noise=true , address=145.79.227.15:4874 , piece_id=TIWMO4OASWXEVFEG4COHHNDNGKHZUWZGTP6D6NNCR5PEZRSBYVIA, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.248134351 (506h14m11.119217426s) [noise=true , address=45.91.116.14:4191 , piece_id=DZPD74AMFXUMJYQEFLME4ZX7NRSLDIYE3ZBR5CIKSJE3XPXANUPA, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.251152537 (506h14m11.122235572s) [noise=true , address=145.79.227.3:4151 , piece_id=4QA7ARRZII6ZRW3GSYBHQVFBGDY4F676MVVJMQHMKWBZ2WS7P5VA, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.253552006 (506h14m11.124635051s) [noise=true , address=77.90.31.21:4880 , piece_id=KYMCS7O4EGMIVDSALBAUBNP7Q3TO5ZX4PLQ4KWBMKSWRUC7TM6EQ, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.255859751 (506h14m11.126942797s) [noise=true , address=145.79.227.22:4511 , piece_id=G7HZFZ7DF6NEWMGRUQQH4UOPUZIYVOVDH22TZNCDWTSQWB3ZEFGQ, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.256186012 (506h14m11.127269057s) [noise=true , address=145.79.227.48:4302 , piece_id=EGYOF4SZ3PKE3H6QYM3QHHRMITTGH55TPNY5D4NS5FSSTKLTML3Q, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.257647495 (506h14m11.12873056s) [noise=true , address=145.79.227.50:4373 , piece_id=DXRYJICGTQJBT7NDE6UUYPDPJZ2HD6MVGRTVZX5GAXXKJFSJ47FA, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.260156352 (506h14m11.131239398s) [noise=true , address=45.91.116.8:4048 , piece_id=JPMATYG4OTAACBWLQKX64FUDV7ONG4P4YZLF4SI4EVGBXOYSICLA, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:12.947257377 (506h14m12.818340412s) [upload=444208]: Next returned nil segment Dec 26 17:43:12.947333081 (506h14m12.818416126s) [upload=444208]: Waiting for error group managing segments... Dec 26 17:43:13.273287306 (506h14m13.144370351s) [noise=true , address=145.79.227.46:4601 , piece_id=IM6NZ7Z4XQSEP4XG4WBWT4Y6P5DUX62KAMLQJANCBKSAU6ZVYRLA, seg_pos={0 0}, upload=444208]: Done uploading piece. err: <nil> Dec 26 17:43:13.273494530 (506h14m13.144577575s) [noise=true , address=145.79.227.3:4151 , piece_id=4QA7ARRZII6ZRW3GSYBHQVFBGDY4F676MVVJMQHMKWBZ2WS7P5VA, seg_pos={0 0}, upload=444208]: Done uploading piece. err: <nil> Dec 26 17:43:13.275357928 (506h14m13.146440963s) [noise=true , address=145.79.227.15:4874 , piece_id=TIWMO4OASWXEVFEG4COHHNDNGKHZUWZGTP6D6NNCR5PEZRSBYVIA, seg_pos={0 0}, upload=444208]: Done uploading piece. err: <nil> Dec 26 17:43:13.276417696 (506h14m13.147500742s) [noise=true , address=145.79.227.45:4230 , piece_id=RKFJEJXGFXRGQZ2B3B3U3XZATBUAPFSV3IVYZQVIS63Q34RNCOSQ, seg_pos={0 0}, upload=444208]: Done uploading piece. err: <nil> Dec 26 17:43:13.277361283 (506h14m13.148444318s) [noise=true , address=145.79.227.50:4373 , piece_id=DXRYJICGTQJBT7NDE6UUYPDPJZ2HD6MVGRTVZX5GAXXKJFSJ47FA, seg_pos={0 0}, upload=444208]: Done uploading piece. err: <nil> Dec 26 17:43:13.277397332 (506h14m13.148480367s) [noise=true , address=145.79.227.37:4567 , piece_id=SLFOZCOSWTYSU5W3VXSJVHO6ZJSGWBUUMZKTWEV5YMDQONFHN53A, seg_pos={0 0}, upload=444208]: Done uploading piece. err: <nil> Dec 26 17:43:13.277910097 (506h14m13.148993142s) [noise=true , address=145.79.227.22:4511 , piece_id=G7HZFZ7DF6NEWMGRUQQH4UOPUZIYVOVDH22TZNCDWTSQWB3ZEFGQ, seg_pos={0 0}, upload=444208]: Done uploading piece. err: <nil> Dec 26 17:43:13.289773407 (506h14m13.160856452s) [noise=true , address=145.79.227.49:4338 , piece_id=LNEIJJAJA7Q6J6ZGVQRPQFZELDCNKU2VKXX64K4ET6XVJ52LGVDQ, seg_pos={0 0}, upload=444208]: Done uploading piece. err: <nil> Dec 26 17:43:13.290752031 (506h14m13.161835086s) [noise=true , address=145.79.227.48:4302 , piece_id=EGYOF4SZ3PKE3H6QYM3QHHRMITTGH55TPNY5D4NS5FSSTKLTML3Q, seg_pos={0 0}, upload=444208]: Done uploading piece. err: <nil> Dec 26 17:43:13.294786072 (506h14m13.165869117s) [noise=true , address=145.79.227.8:4330 , piece_id=HEEUM6HSWGUYQFB2NP5T4DZST5GUJEDFNTLWDS6GEZC75P54UJQQ, seg_pos={0 0}, upload=444208]: Done uploading piece. err: <nil> Dec 26 17:43:13.489465767 (506h14m13.360548802s) [noise=true , address=45.91.116.12:4228 , piece_id=LCYNTQGR2ZIUX6HBLYVSCUTCLAU6PFGHSA2ICB6DWE2ZFOI22NSA, seg_pos={0 0}, upload=444208]: Done uploading piece. err: <nil> Dec 26 17:43:13.566172019 (506h14m13.437255064s) [noise=true , address=45.91.116.14:4191 , piece_id=DZPD74AMFXUMJYQEFLME4ZX7NRSLDIYE3ZBR5CIKSJE3XPXANUPA, seg_pos={0 0}, upload=444208]: Done uploading piece. err: <nil> Dec 26 17:43:13.725036525 (506h14m13.59611956s) [noise=true , address=45.91.116.6:4974 , piece_id=VKPME5624PBSFP3HLHKANNY7SBEVFL4Z636HEN2KXUQ55XPORJ2A, seg_pos={0 0}, upload=444208]: Done uploading piece. err: <nil> Dec 26 17:43:13.725057735 (506h14m13.59614077s) [seg_pos={0 0}, upload=444208]: Segment reached optimal threshold of 13 pieces. Dec 26 17:43:13.725271673 (506h14m13.596354708s) [noise=true , address=145.79.227.35:4481 , piece_id=AWRLZ2QPXDUTJWC2J6TCOM7QROIZJENTYBVWB6T2KPR2B6YYXS4A, seg_pos={0 0}, upload=444208]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 26 17:43:13.725279558 (506h14m13.596362593s) [noise=true , address=145.79.227.35:4481 , piece_id=AWRLZ2QPXDUTJWC2J6TCOM7QROIZJENTYBVWB6T2KPR2B6YYXS4A, seg_pos={0 0}, upload=444208]: Long tail cancellation due to optimal threshold Dec 26 17:43:13.725338128 (506h14m13.596421174s) [noise=true , address=77.90.31.21:4880 , piece_id=KYMCS7O4EGMIVDSALBAUBNP7Q3TO5ZX4PLQ4KWBMKSWRUC7TM6EQ, seg_pos={0 0}, upload=444208]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: write tcp 188.166.225.170:37372->77.90.31.21:4880: use of closed network connection; write tcp 188.166.225.170:37372->77.90.31.21:4880: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37372->77.90.31.21:4880: use of closed network connection Dec 26 17:43:13.725346515 (506h14m13.59642955s) [noise=true , address=77.90.31.21:4880 , piece_id=KYMCS7O4EGMIVDSALBAUBNP7Q3TO5ZX4PLQ4KWBMKSWRUC7TM6EQ, seg_pos={0 0}, upload=444208]: Long tail cancellation due to optimal threshold Dec 26 17:43:13.725564680 (506h14m13.596647715s) [noise=true , address=45.91.116.8:4048 , piece_id=JPMATYG4OTAACBWLQKX64FUDV7ONG4P4YZLF4SI4EVGBXOYSICLA, seg_pos={0 0}, upload=444208]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: context canceled; context canceled Dec 26 17:43:13.725572675 (506h14m13.59665572s) [noise=true , address=45.91.116.8:4048 , piece_id=JPMATYG4OTAACBWLQKX64FUDV7ONG4P4YZLF4SI4EVGBXOYSICLA, seg_pos={0 0}, upload=444208]: Long tail cancellation due to optimal threshold Dec 26 17:43:13.725590309 (506h14m13.596673354s) [seg_pos={0 0}, upload=444208]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 17:43:13.725667807 (506h14m13.596750842s) [upload=444208]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:43:13.725675932 (506h14m13.596758977s) [upload=444208]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:43:14.020265812 (506h14m13.891348857s) [upload=444208]: Done uploading segments... Dec 26 17:43:14.020287173 (506h14m13.891370219s) [upload=444208]: Upload finished. err: <nil> Duration: 6.03840155s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 39.346101655s trace: https://jaeger.dev.storj.io/trace/665ea65f6e9ccc33 FAIL (2025-12-26 17:45:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== e=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789316979 (506h12m54.660400024s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789326507 (506h12m54.660409552s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789611339 (506h12m54.660694384s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789619275 (506h12m54.66070231s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789914006 (506h12m54.660997041s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789949443 (506h12m54.661032478s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790181566 (506h12m54.661264601s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790295632 (506h12m54.661378668s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790510872 (506h12m54.661593917s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790692980 (506h12m54.661776015s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790787379 (506h12m54.661870414s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790992970 (506h12m54.662076005s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791120053 (506h12m54.662203098s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791275218 (506h12m54.662358253s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791443709 (506h12m54.662526744s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:56.112313145 (506h12m55.98339619s) [upload=444191]: Next returned nil segment Dec 26 17:41:56.112330638 (506h12m55.983413683s) [upload=444191]: Waiting for error group managing segments... Dec 26 17:41:56.485322720 (506h12m56.356405765s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.485637079 (506h12m56.356720124s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.666807539 (506h12m56.537890584s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.678753987 (506h12m56.549837032s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.687788039 (506h12m56.558871074s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.694318023 (506h12m56.565401058s) [noise=true , address=145.79.227.7:4326 , piece_id=DMQ5RPI46X2U7TJTS42ZHQEF32BDJRKJJO5HWOYAPNS6VWVG5B5Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.716581187 (506h12m56.587664232s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.972183007 (506h12m56.843266042s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.011917999 (506h12m56.883001034s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.054168622 (506h12m56.925251667s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.114312777 (506h12m56.985395812s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.499846250 (506h12m57.370929285s) [noise=true , address=145.79.227.34:4919 , piece_id=YRWHW3BJVEGVJRSA6RJB3T7VJQSDTGKEZQKRICFIV2G3T7AN4MIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.612649066 (506h12m57.483732101s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649413733 (506h12m57.520496768s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649440282 (506h12m57.520523328s) [seg_pos={0 0}, upload=444191]: Segment reached optimal threshold of 14 pieces. Dec 26 17:41:57.649674729 (506h12m57.520757774s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G7ty71tPwe7cZSBNK9u74DFUCGomnxDsNvAif4C3SbP7HWwfo): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649684087 (506h12m57.520767142s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649747708 (506h12m57.520830763s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v6zXZuMmyL2MU8qb3q6gg1iwTYvndjzMjVpv8btEpeX9Lnqmu): protocol: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection Dec 26 17:41:57.649785219 (506h12m57.520868274s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649899828 (506h12m57.520982903s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649908775 (506h12m57.52099182s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650074641 (506h12m57.521157696s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.650083287 (506h12m57.521166342s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650132821 (506h12m57.521215866s) [seg_pos={0 0}, upload=444191]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 17:41:57.650220138 (506h12m57.521303183s) [upload=444191]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:41:57.650227171 (506h12m57.521310206s) [upload=444191]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:41:57.958890870 (506h12m57.829973915s) [upload=444191]: Done uploading segments... Dec 26 17:41:57.958911620 (506h12m57.829994665s) [upload=444191]: Upload finished. err: <nil> Duration: 39.34613097s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [XX..X.X......XX..X....X.X.XX...XX...XX....XX..XX......X.XXX.] duration: 20.537201736s trace: https://jaeger.dev.storj.io/trace/3b761e3de974ea48 FAIL (2025-12-26 17:44:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA, Piece ID: LCYNTQGR2ZIUX6HBLYVSCUTCLAU6PFGHSA2ICB6DWE2ZFOI22NSA) context deadline exceeded; piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: DZPD74AMFXUMJYQEFLME4ZX7NRSLDIYE3ZBR5CIKSJE3XPXANUPA) context deadline exceeded ======LOGS====== Dec 26 17:43:10.974317425 (506h14m10.84540046s) [upload=444208]: Starting upload Dec 26 17:43:10.974446491 (506h14m10.845529526s) [upload=444208]: Done starting upload Dec 26 17:43:10.974581899 (506h14m10.845664944s) [upload=444208]: Uploading segments... Dec 26 17:43:10.974665328 (506h14m10.845748383s) [upload=444208]: Got next segment. Inline: false Dec 26 17:43:10.974721515 (506h14m10.84580456s) [upload=444208]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 17:43:10.974752394 (506h14m10.845835439s) [upload=444208]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 17:43:11.225224176 (506h14m11.096307221s) [seg_pos={0 0}, upload=444208]: Begin upload segment... Dec 26 17:43:11.225287356 (506h14m11.096370411s) [seg_pos={0 0}, upload=444208]: Done begin upload segment. Dec 26 17:43:11.225329857 (506h14m11.096412902s) [noise=true , address=145.79.227.46:4601 , piece_id=IM6NZ7Z4XQSEP4XG4WBWT4Y6P5DUX62KAMLQJANCBKSAU6ZVYRLA, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.225917767 (506h14m11.097000802s) [noise=true , address=145.79.227.8:4330 , piece_id=HEEUM6HSWGUYQFB2NP5T4DZST5GUJEDFNTLWDS6GEZC75P54UJQQ, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.231054878 (506h14m11.102137913s) [noise=true , address=145.79.227.49:4338 , piece_id=LNEIJJAJA7Q6J6ZGVQRPQFZELDCNKU2VKXX64K4ET6XVJ52LGVDQ, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.232622754 (506h14m11.103705799s) [noise=true , address=45.91.116.12:4228 , piece_id=LCYNTQGR2ZIUX6HBLYVSCUTCLAU6PFGHSA2ICB6DWE2ZFOI22NSA, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.234863150 (506h14m11.105946186s) [noise=true , address=145.79.227.45:4230 , piece_id=RKFJEJXGFXRGQZ2B3B3U3XZATBUAPFSV3IVYZQVIS63Q34RNCOSQ, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.236499346 (506h14m11.107582411s) [noise=true , address=45.91.116.6:4974 , piece_id=VKPME5624PBSFP3HLHKANNY7SBEVFL4Z636HEN2KXUQ55XPORJ2A, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.238064495 (506h14m11.10914753s) [noise=true , address=145.79.227.35:4481 , piece_id=AWRLZ2QPXDUTJWC2J6TCOM7QROIZJENTYBVWB6T2KPR2B6YYXS4A, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.243205044 (506h14m11.114288089s) [noise=true , address=145.79.227.37:4567 , piece_id=SLFOZCOSWTYSU5W3VXSJVHO6ZJSGWBUUMZKTWEV5YMDQONFHN53A, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.245389073 (506h14m11.116472108s) [noise=true , address=145.79.227.15:4874 , piece_id=TIWMO4OASWXEVFEG4COHHNDNGKHZUWZGTP6D6NNCR5PEZRSBYVIA, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.248134351 (506h14m11.119217426s) [noise=true , address=45.91.116.14:4191 , piece_id=DZPD74AMFXUMJYQEFLME4ZX7NRSLDIYE3ZBR5CIKSJE3XPXANUPA, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.251152537 (506h14m11.122235572s) [noise=true , address=145.79.227.3:4151 , piece_id=4QA7ARRZII6ZRW3GSYBHQVFBGDY4F676MVVJMQHMKWBZ2WS7P5VA, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.253552006 (506h14m11.124635051s) [noise=true , address=77.90.31.21:4880 , piece_id=KYMCS7O4EGMIVDSALBAUBNP7Q3TO5ZX4PLQ4KWBMKSWRUC7TM6EQ, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.255859751 (506h14m11.126942797s) [noise=true , address=145.79.227.22:4511 , piece_id=G7HZFZ7DF6NEWMGRUQQH4UOPUZIYVOVDH22TZNCDWTSQWB3ZEFGQ, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.256186012 (506h14m11.127269057s) [noise=true , address=145.79.227.48:4302 , piece_id=EGYOF4SZ3PKE3H6QYM3QHHRMITTGH55TPNY5D4NS5FSSTKLTML3Q, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.257647495 (506h14m11.12873056s) [noise=true , address=145.79.227.50:4373 , piece_id=DXRYJICGTQJBT7NDE6UUYPDPJZ2HD6MVGRTVZX5GAXXKJFSJ47FA, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.260156352 (506h14m11.131239398s) [noise=true , address=45.91.116.8:4048 , piece_id=JPMATYG4OTAACBWLQKX64FUDV7ONG4P4YZLF4SI4EVGBXOYSICLA, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:12.947257377 (506h14m12.818340412s) [upload=444208]: Next returned nil segment Dec 26 17:43:12.947333081 (506h14m12.818416126s) [upload=444208]: Waiting for error group managing segments... Dec 26 17:43:13.273287306 (506h14m13.144370351s) [noise=true , address=145.79.227.46:4601 , piece_id=IM6NZ7Z4XQSEP4XG4WBWT4Y6P5DUX62KAMLQJANCBKSAU6ZVYRLA, seg_pos={0 0}, upload=444208]: Done uploading piece. err: <nil> Dec 26 17:43:13.273494530 (506h14m13.144577575s) [noise=true , address=145.79.227.3:4151 , piece_id=4QA7ARRZII6ZRW3GSYBHQVFBGDY4F676MVVJMQHMKWBZ2WS7P5VA, seg_pos={0 0}, upload=444208]: Done uploading piece. err: <nil> Dec 26 17:43:13.275357928 (506h14m13.146440963s) [noise=true , address=145.79.227.15:4874 , piece_id=TIWMO4OASWXEVFEG4COHHNDNGKHZUWZGTP6D6NNCR5PEZRSBYVIA, seg_pos={0 0}, upload=444208]: Done uploading piece. err: <nil> Dec 26 17:43:13.276417696 (506h14m13.147500742s) [noise=true , address=145.79.227.45:4230 , piece_id=RKFJEJXGFXRGQZ2B3B3U3XZATBUAPFSV3IVYZQVIS63Q34RNCOSQ, seg_pos={0 0}, upload=444208]: Done uploading piece. err: <nil> Dec 26 17:43:13.277361283 (506h14m13.148444318s) [noise=true , address=145.79.227.50:4373 , piece_id=DXRYJICGTQJBT7NDE6UUYPDPJZ2HD6MVGRTVZX5GAXXKJFSJ47FA, seg_pos={0 0}, upload=444208]: Done uploading piece. err: <nil> Dec 26 17:43:13.277397332 (506h14m13.148480367s) [noise=true , address=145.79.227.37:4567 , piece_id=SLFOZCOSWTYSU5W3VXSJVHO6ZJSGWBUUMZKTWEV5YMDQONFHN53A, seg_pos={0 0}, upload=444208]: Done uploading piece. err: <nil> Dec 26 17:43:13.277910097 (506h14m13.148993142s) [noise=true , address=145.79.227.22:4511 , piece_id=G7HZFZ7DF6NEWMGRUQQH4UOPUZIYVOVDH22TZNCDWTSQWB3ZEFGQ, seg_pos={0 0}, upload=444208]: Done uploading piece. err: <nil> Dec 26 17:43:13.289773407 (506h14m13.160856452s) [noise=true , address=145.79.227.49:4338 , piece_id=LNEIJJAJA7Q6J6ZGVQRPQFZELDCNKU2VKXX64K4ET6XVJ52LGVDQ, seg_pos={0 0}, upload=444208]: Done uploading piece. err: <nil> Dec 26 17:43:13.290752031 (506h14m13.161835086s) [noise=true , address=145.79.227.48:4302 , piece_id=EGYOF4SZ3PKE3H6QYM3QHHRMITTGH55TPNY5D4NS5FSSTKLTML3Q, seg_pos={0 0}, upload=444208]: Done uploading piece. err: <nil> Dec 26 17:43:13.294786072 (506h14m13.165869117s) [noise=true , address=145.79.227.8:4330 , piece_id=HEEUM6HSWGUYQFB2NP5T4DZST5GUJEDFNTLWDS6GEZC75P54UJQQ, seg_pos={0 0}, upload=444208]: Done uploading piece. err: <nil> Dec 26 17:43:13.489465767 (506h14m13.360548802s) [noise=true , address=45.91.116.12:4228 , piece_id=LCYNTQGR2ZIUX6HBLYVSCUTCLAU6PFGHSA2ICB6DWE2ZFOI22NSA, seg_pos={0 0}, upload=444208]: Done uploading piece. err: <nil> Dec 26 17:43:13.566172019 (506h14m13.437255064s) [noise=true , address=45.91.116.14:4191 , piece_id=DZPD74AMFXUMJYQEFLME4ZX7NRSLDIYE3ZBR5CIKSJE3XPXANUPA, seg_pos={0 0}, upload=444208]: Done uploading piece. err: <nil> Dec 26 17:43:13.725036525 (506h14m13.59611956s) [noise=true , address=45.91.116.6:4974 , piece_id=VKPME5624PBSFP3HLHKANNY7SBEVFL4Z636HEN2KXUQ55XPORJ2A, seg_pos={0 0}, upload=444208]: Done uploading piece. err: <nil> Dec 26 17:43:13.725057735 (506h14m13.59614077s) [seg_pos={0 0}, upload=444208]: Segment reached optimal threshold of 13 pieces. Dec 26 17:43:13.725271673 (506h14m13.596354708s) [noise=true , address=145.79.227.35:4481 , piece_id=AWRLZ2QPXDUTJWC2J6TCOM7QROIZJENTYBVWB6T2KPR2B6YYXS4A, seg_pos={0 0}, upload=444208]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 26 17:43:13.725279558 (506h14m13.596362593s) [noise=true , address=145.79.227.35:4481 , piece_id=AWRLZ2QPXDUTJWC2J6TCOM7QROIZJENTYBVWB6T2KPR2B6YYXS4A, seg_pos={0 0}, upload=444208]: Long tail cancellation due to optimal threshold Dec 26 17:43:13.725338128 (506h14m13.596421174s) [noise=true , address=77.90.31.21:4880 , piece_id=KYMCS7O4EGMIVDSALBAUBNP7Q3TO5ZX4PLQ4KWBMKSWRUC7TM6EQ, seg_pos={0 0}, upload=444208]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: write tcp 188.166.225.170:37372->77.90.31.21:4880: use of closed network connection; write tcp 188.166.225.170:37372->77.90.31.21:4880: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37372->77.90.31.21:4880: use of closed network connection Dec 26 17:43:13.725346515 (506h14m13.59642955s) [noise=true , address=77.90.31.21:4880 , piece_id=KYMCS7O4EGMIVDSALBAUBNP7Q3TO5ZX4PLQ4KWBMKSWRUC7TM6EQ, seg_pos={0 0}, upload=444208]: Long tail cancellation due to optimal threshold Dec 26 17:43:13.725564680 (506h14m13.596647715s) [noise=true , address=45.91.116.8:4048 , piece_id=JPMATYG4OTAACBWLQKX64FUDV7ONG4P4YZLF4SI4EVGBXOYSICLA, seg_pos={0 0}, upload=444208]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: context canceled; context canceled Dec 26 17:43:13.725572675 (506h14m13.59665572s) [noise=true , address=45.91.116.8:4048 , piece_id=JPMATYG4OTAACBWLQKX64FUDV7ONG4P4YZLF4SI4EVGBXOYSICLA, seg_pos={0 0}, upload=444208]: Long tail cancellation due to optimal threshold Dec 26 17:43:13.725590309 (506h14m13.596673354s) [seg_pos={0 0}, upload=444208]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 17:43:13.725667807 (506h14m13.596750842s) [upload=444208]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:43:13.725675932 (506h14m13.596758977s) [upload=444208]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:43:14.020265812 (506h14m13.891348857s) [upload=444208]: Done uploading segments... Dec 26 17:43:14.020287173 (506h14m13.891370219s) [upload=444208]: Upload finished. err: <nil> Duration: 20.53723065s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 44.71101242s trace: https://jaeger.dev.storj.io/trace/665ea65f6e9ccc33 FAIL (2025-12-26 17:44:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== e=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789316979 (506h12m54.660400024s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789326507 (506h12m54.660409552s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789611339 (506h12m54.660694384s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789619275 (506h12m54.66070231s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789914006 (506h12m54.660997041s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789949443 (506h12m54.661032478s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790181566 (506h12m54.661264601s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790295632 (506h12m54.661378668s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790510872 (506h12m54.661593917s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790692980 (506h12m54.661776015s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790787379 (506h12m54.661870414s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790992970 (506h12m54.662076005s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791120053 (506h12m54.662203098s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791275218 (506h12m54.662358253s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791443709 (506h12m54.662526744s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:56.112313145 (506h12m55.98339619s) [upload=444191]: Next returned nil segment Dec 26 17:41:56.112330638 (506h12m55.983413683s) [upload=444191]: Waiting for error group managing segments... Dec 26 17:41:56.485322720 (506h12m56.356405765s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.485637079 (506h12m56.356720124s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.666807539 (506h12m56.537890584s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.678753987 (506h12m56.549837032s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.687788039 (506h12m56.558871074s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.694318023 (506h12m56.565401058s) [noise=true , address=145.79.227.7:4326 , piece_id=DMQ5RPI46X2U7TJTS42ZHQEF32BDJRKJJO5HWOYAPNS6VWVG5B5Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.716581187 (506h12m56.587664232s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.972183007 (506h12m56.843266042s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.011917999 (506h12m56.883001034s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.054168622 (506h12m56.925251667s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.114312777 (506h12m56.985395812s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.499846250 (506h12m57.370929285s) [noise=true , address=145.79.227.34:4919 , piece_id=YRWHW3BJVEGVJRSA6RJB3T7VJQSDTGKEZQKRICFIV2G3T7AN4MIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.612649066 (506h12m57.483732101s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649413733 (506h12m57.520496768s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649440282 (506h12m57.520523328s) [seg_pos={0 0}, upload=444191]: Segment reached optimal threshold of 14 pieces. Dec 26 17:41:57.649674729 (506h12m57.520757774s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G7ty71tPwe7cZSBNK9u74DFUCGomnxDsNvAif4C3SbP7HWwfo): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649684087 (506h12m57.520767142s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649747708 (506h12m57.520830763s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v6zXZuMmyL2MU8qb3q6gg1iwTYvndjzMjVpv8btEpeX9Lnqmu): protocol: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection Dec 26 17:41:57.649785219 (506h12m57.520868274s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649899828 (506h12m57.520982903s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649908775 (506h12m57.52099182s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650074641 (506h12m57.521157696s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.650083287 (506h12m57.521166342s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650132821 (506h12m57.521215866s) [seg_pos={0 0}, upload=444191]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 17:41:57.650220138 (506h12m57.521303183s) [upload=444191]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:41:57.650227171 (506h12m57.521310206s) [upload=444191]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:41:57.958890870 (506h12m57.829973915s) [upload=444191]: Done uploading segments... Dec 26 17:41:57.958911620 (506h12m57.829994665s) [upload=444191]: Upload finished. err: <nil> Duration: 44.71102799s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [.XX..X.X......XX..X....X.X.XX...XX...XX....XX..XX......X.XXX] duration: 30.003819108s trace: https://jaeger.dev.storj.io/trace/3b761e3de974ea48 FAIL (2025-12-26 17:43:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA, Piece ID: LCYNTQGR2ZIUX6HBLYVSCUTCLAU6PFGHSA2ICB6DWE2ZFOI22NSA) context deadline exceeded; piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: DZPD74AMFXUMJYQEFLME4ZX7NRSLDIYE3ZBR5CIKSJE3XPXANUPA) context deadline exceeded ======LOGS====== Dec 26 17:43:10.974317425 (506h14m10.84540046s) [upload=444208]: Starting upload Dec 26 17:43:10.974446491 (506h14m10.845529526s) [upload=444208]: Done starting upload Dec 26 17:43:10.974581899 (506h14m10.845664944s) [upload=444208]: Uploading segments... Dec 26 17:43:10.974665328 (506h14m10.845748383s) [upload=444208]: Got next segment. Inline: false Dec 26 17:43:10.974721515 (506h14m10.84580456s) [upload=444208]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 17:43:10.974752394 (506h14m10.845835439s) [upload=444208]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 17:43:11.225224176 (506h14m11.096307221s) [seg_pos={0 0}, upload=444208]: Begin upload segment... Dec 26 17:43:11.225287356 (506h14m11.096370411s) [seg_pos={0 0}, upload=444208]: Done begin upload segment. Dec 26 17:43:11.225329857 (506h14m11.096412902s) [noise=true , address=145.79.227.46:4601 , piece_id=IM6NZ7Z4XQSEP4XG4WBWT4Y6P5DUX62KAMLQJANCBKSAU6ZVYRLA, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.225917767 (506h14m11.097000802s) [noise=true , address=145.79.227.8:4330 , piece_id=HEEUM6HSWGUYQFB2NP5T4DZST5GUJEDFNTLWDS6GEZC75P54UJQQ, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.231054878 (506h14m11.102137913s) [noise=true , address=145.79.227.49:4338 , piece_id=LNEIJJAJA7Q6J6ZGVQRPQFZELDCNKU2VKXX64K4ET6XVJ52LGVDQ, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.232622754 (506h14m11.103705799s) [noise=true , address=45.91.116.12:4228 , piece_id=LCYNTQGR2ZIUX6HBLYVSCUTCLAU6PFGHSA2ICB6DWE2ZFOI22NSA, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.234863150 (506h14m11.105946186s) [noise=true , address=145.79.227.45:4230 , piece_id=RKFJEJXGFXRGQZ2B3B3U3XZATBUAPFSV3IVYZQVIS63Q34RNCOSQ, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.236499346 (506h14m11.107582411s) [noise=true , address=45.91.116.6:4974 , piece_id=VKPME5624PBSFP3HLHKANNY7SBEVFL4Z636HEN2KXUQ55XPORJ2A, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.238064495 (506h14m11.10914753s) [noise=true , address=145.79.227.35:4481 , piece_id=AWRLZ2QPXDUTJWC2J6TCOM7QROIZJENTYBVWB6T2KPR2B6YYXS4A, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.243205044 (506h14m11.114288089s) [noise=true , address=145.79.227.37:4567 , piece_id=SLFOZCOSWTYSU5W3VXSJVHO6ZJSGWBUUMZKTWEV5YMDQONFHN53A, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.245389073 (506h14m11.116472108s) [noise=true , address=145.79.227.15:4874 , piece_id=TIWMO4OASWXEVFEG4COHHNDNGKHZUWZGTP6D6NNCR5PEZRSBYVIA, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.248134351 (506h14m11.119217426s) [noise=true , address=45.91.116.14:4191 , piece_id=DZPD74AMFXUMJYQEFLME4ZX7NRSLDIYE3ZBR5CIKSJE3XPXANUPA, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.251152537 (506h14m11.122235572s) [noise=true , address=145.79.227.3:4151 , piece_id=4QA7ARRZII6ZRW3GSYBHQVFBGDY4F676MVVJMQHMKWBZ2WS7P5VA, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.253552006 (506h14m11.124635051s) [noise=true , address=77.90.31.21:4880 , piece_id=KYMCS7O4EGMIVDSALBAUBNP7Q3TO5ZX4PLQ4KWBMKSWRUC7TM6EQ, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.255859751 (506h14m11.126942797s) [noise=true , address=145.79.227.22:4511 , piece_id=G7HZFZ7DF6NEWMGRUQQH4UOPUZIYVOVDH22TZNCDWTSQWB3ZEFGQ, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.256186012 (506h14m11.127269057s) [noise=true , address=145.79.227.48:4302 , piece_id=EGYOF4SZ3PKE3H6QYM3QHHRMITTGH55TPNY5D4NS5FSSTKLTML3Q, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.257647495 (506h14m11.12873056s) [noise=true , address=145.79.227.50:4373 , piece_id=DXRYJICGTQJBT7NDE6UUYPDPJZ2HD6MVGRTVZX5GAXXKJFSJ47FA, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.260156352 (506h14m11.131239398s) [noise=true , address=45.91.116.8:4048 , piece_id=JPMATYG4OTAACBWLQKX64FUDV7ONG4P4YZLF4SI4EVGBXOYSICLA, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:12.947257377 (506h14m12.818340412s) [upload=444208]: Next returned nil segment Dec 26 17:43:12.947333081 (506h14m12.818416126s) [upload=444208]: Waiting for error group managing segments... Dec 26 17:43:13.273287306 (506h14m13.144370351s) [noise=true , address=145.79.227.46:4601 , piece_id=IM6NZ7Z4XQSEP4XG4WBWT4Y6P5DUX62KAMLQJANCBKSAU6ZVYRLA, seg_pos={0 0}, upload=444208]: Done uploading piece. err: <nil> Dec 26 17:43:13.273494530 (506h14m13.144577575s) [noise=true , address=145.79.227.3:4151 , piece_id=4QA7ARRZII6ZRW3GSYBHQVFBGDY4F676MVVJMQHMKWBZ2WS7P5VA, seg_pos={0 0}, upload=444208]: Done uploading piece. err: <nil> Dec 26 17:43:13.275357928 (506h14m13.146440963s) [noise=true , address=145.79.227.15:4874 , piece_id=TIWMO4OASWXEVFEG4COHHNDNGKHZUWZGTP6D6NNCR5PEZRSBYVIA, seg_pos={0 0}, upload=444208]: Done uploading piece. err: <nil> Dec 26 17:43:13.276417696 (506h14m13.147500742s) [noise=true , address=145.79.227.45:4230 , piece_id=RKFJEJXGFXRGQZ2B3B3U3XZATBUAPFSV3IVYZQVIS63Q34RNCOSQ, seg_pos={0 0}, upload=444208]: Done uploading piece. err: <nil> Dec 26 17:43:13.277361283 (506h14m13.148444318s) [noise=true , address=145.79.227.50:4373 , piece_id=DXRYJICGTQJBT7NDE6UUYPDPJZ2HD6MVGRTVZX5GAXXKJFSJ47FA, seg_pos={0 0}, upload=444208]: Done uploading piece. err: <nil> Dec 26 17:43:13.277397332 (506h14m13.148480367s) [noise=true , address=145.79.227.37:4567 , piece_id=SLFOZCOSWTYSU5W3VXSJVHO6ZJSGWBUUMZKTWEV5YMDQONFHN53A, seg_pos={0 0}, upload=444208]: Done uploading piece. err: <nil> Dec 26 17:43:13.277910097 (506h14m13.148993142s) [noise=true , address=145.79.227.22:4511 , piece_id=G7HZFZ7DF6NEWMGRUQQH4UOPUZIYVOVDH22TZNCDWTSQWB3ZEFGQ, seg_pos={0 0}, upload=444208]: Done uploading piece. err: <nil> Dec 26 17:43:13.289773407 (506h14m13.160856452s) [noise=true , address=145.79.227.49:4338 , piece_id=LNEIJJAJA7Q6J6ZGVQRPQFZELDCNKU2VKXX64K4ET6XVJ52LGVDQ, seg_pos={0 0}, upload=444208]: Done uploading piece. err: <nil> Dec 26 17:43:13.290752031 (506h14m13.161835086s) [noise=true , address=145.79.227.48:4302 , piece_id=EGYOF4SZ3PKE3H6QYM3QHHRMITTGH55TPNY5D4NS5FSSTKLTML3Q, seg_pos={0 0}, upload=444208]: Done uploading piece. err: <nil> Dec 26 17:43:13.294786072 (506h14m13.165869117s) [noise=true , address=145.79.227.8:4330 , piece_id=HEEUM6HSWGUYQFB2NP5T4DZST5GUJEDFNTLWDS6GEZC75P54UJQQ, seg_pos={0 0}, upload=444208]: Done uploading piece. err: <nil> Dec 26 17:43:13.489465767 (506h14m13.360548802s) [noise=true , address=45.91.116.12:4228 , piece_id=LCYNTQGR2ZIUX6HBLYVSCUTCLAU6PFGHSA2ICB6DWE2ZFOI22NSA, seg_pos={0 0}, upload=444208]: Done uploading piece. err: <nil> Dec 26 17:43:13.566172019 (506h14m13.437255064s) [noise=true , address=45.91.116.14:4191 , piece_id=DZPD74AMFXUMJYQEFLME4ZX7NRSLDIYE3ZBR5CIKSJE3XPXANUPA, seg_pos={0 0}, upload=444208]: Done uploading piece. err: <nil> Dec 26 17:43:13.725036525 (506h14m13.59611956s) [noise=true , address=45.91.116.6:4974 , piece_id=VKPME5624PBSFP3HLHKANNY7SBEVFL4Z636HEN2KXUQ55XPORJ2A, seg_pos={0 0}, upload=444208]: Done uploading piece. err: <nil> Dec 26 17:43:13.725057735 (506h14m13.59614077s) [seg_pos={0 0}, upload=444208]: Segment reached optimal threshold of 13 pieces. Dec 26 17:43:13.725271673 (506h14m13.596354708s) [noise=true , address=145.79.227.35:4481 , piece_id=AWRLZ2QPXDUTJWC2J6TCOM7QROIZJENTYBVWB6T2KPR2B6YYXS4A, seg_pos={0 0}, upload=444208]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 26 17:43:13.725279558 (506h14m13.596362593s) [noise=true , address=145.79.227.35:4481 , piece_id=AWRLZ2QPXDUTJWC2J6TCOM7QROIZJENTYBVWB6T2KPR2B6YYXS4A, seg_pos={0 0}, upload=444208]: Long tail cancellation due to optimal threshold Dec 26 17:43:13.725338128 (506h14m13.596421174s) [noise=true , address=77.90.31.21:4880 , piece_id=KYMCS7O4EGMIVDSALBAUBNP7Q3TO5ZX4PLQ4KWBMKSWRUC7TM6EQ, seg_pos={0 0}, upload=444208]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: write tcp 188.166.225.170:37372->77.90.31.21:4880: use of closed network connection; write tcp 188.166.225.170:37372->77.90.31.21:4880: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37372->77.90.31.21:4880: use of closed network connection Dec 26 17:43:13.725346515 (506h14m13.59642955s) [noise=true , address=77.90.31.21:4880 , piece_id=KYMCS7O4EGMIVDSALBAUBNP7Q3TO5ZX4PLQ4KWBMKSWRUC7TM6EQ, seg_pos={0 0}, upload=444208]: Long tail cancellation due to optimal threshold Dec 26 17:43:13.725564680 (506h14m13.596647715s) [noise=true , address=45.91.116.8:4048 , piece_id=JPMATYG4OTAACBWLQKX64FUDV7ONG4P4YZLF4SI4EVGBXOYSICLA, seg_pos={0 0}, upload=444208]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: context canceled; context canceled Dec 26 17:43:13.725572675 (506h14m13.59665572s) [noise=true , address=45.91.116.8:4048 , piece_id=JPMATYG4OTAACBWLQKX64FUDV7ONG4P4YZLF4SI4EVGBXOYSICLA, seg_pos={0 0}, upload=444208]: Long tail cancellation due to optimal threshold Dec 26 17:43:13.725590309 (506h14m13.596673354s) [seg_pos={0 0}, upload=444208]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 17:43:13.725667807 (506h14m13.596750842s) [upload=444208]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:43:13.725675932 (506h14m13.596758977s) [upload=444208]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:43:14.020265812 (506h14m13.891348857s) [upload=444208]: Done uploading segments... Dec 26 17:43:14.020287173 (506h14m13.891370219s) [upload=444208]: Upload finished. err: <nil> Duration: 30.003844386s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 44.09144737s trace: https://jaeger.dev.storj.io/trace/665ea65f6e9ccc33 FAIL (2025-12-26 17:43:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== e=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789316979 (506h12m54.660400024s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789326507 (506h12m54.660409552s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789611339 (506h12m54.660694384s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789619275 (506h12m54.66070231s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789914006 (506h12m54.660997041s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789949443 (506h12m54.661032478s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790181566 (506h12m54.661264601s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790295632 (506h12m54.661378668s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790510872 (506h12m54.661593917s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790692980 (506h12m54.661776015s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790787379 (506h12m54.661870414s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790992970 (506h12m54.662076005s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791120053 (506h12m54.662203098s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791275218 (506h12m54.662358253s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791443709 (506h12m54.662526744s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:56.112313145 (506h12m55.98339619s) [upload=444191]: Next returned nil segment Dec 26 17:41:56.112330638 (506h12m55.983413683s) [upload=444191]: Waiting for error group managing segments... Dec 26 17:41:56.485322720 (506h12m56.356405765s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.485637079 (506h12m56.356720124s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.666807539 (506h12m56.537890584s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.678753987 (506h12m56.549837032s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.687788039 (506h12m56.558871074s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.694318023 (506h12m56.565401058s) [noise=true , address=145.79.227.7:4326 , piece_id=DMQ5RPI46X2U7TJTS42ZHQEF32BDJRKJJO5HWOYAPNS6VWVG5B5Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.716581187 (506h12m56.587664232s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.972183007 (506h12m56.843266042s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.011917999 (506h12m56.883001034s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.054168622 (506h12m56.925251667s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.114312777 (506h12m56.985395812s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.499846250 (506h12m57.370929285s) [noise=true , address=145.79.227.34:4919 , piece_id=YRWHW3BJVEGVJRSA6RJB3T7VJQSDTGKEZQKRICFIV2G3T7AN4MIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.612649066 (506h12m57.483732101s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649413733 (506h12m57.520496768s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649440282 (506h12m57.520523328s) [seg_pos={0 0}, upload=444191]: Segment reached optimal threshold of 14 pieces. Dec 26 17:41:57.649674729 (506h12m57.520757774s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G7ty71tPwe7cZSBNK9u74DFUCGomnxDsNvAif4C3SbP7HWwfo): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649684087 (506h12m57.520767142s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649747708 (506h12m57.520830763s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v6zXZuMmyL2MU8qb3q6gg1iwTYvndjzMjVpv8btEpeX9Lnqmu): protocol: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection Dec 26 17:41:57.649785219 (506h12m57.520868274s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649899828 (506h12m57.520982903s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649908775 (506h12m57.52099182s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650074641 (506h12m57.521157696s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.650083287 (506h12m57.521166342s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650132821 (506h12m57.521215866s) [seg_pos={0 0}, upload=444191]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 17:41:57.650220138 (506h12m57.521303183s) [upload=444191]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:41:57.650227171 (506h12m57.521310206s) [upload=444191]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:41:57.958890870 (506h12m57.829973915s) [upload=444191]: Done uploading segments... Dec 26 17:41:57.958911620 (506h12m57.829994665s) [upload=444191]: Upload finished. err: <nil> Duration: 44.091463121s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1m0.005108919s trace: https://jaeger.dev.storj.io/trace/665ea65f6e9ccc33 FAIL (2025-12-26 17:42:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== e=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789316979 (506h12m54.660400024s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789326507 (506h12m54.660409552s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789611339 (506h12m54.660694384s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789619275 (506h12m54.66070231s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789914006 (506h12m54.660997041s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789949443 (506h12m54.661032478s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790181566 (506h12m54.661264601s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790295632 (506h12m54.661378668s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790510872 (506h12m54.661593917s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790692980 (506h12m54.661776015s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790787379 (506h12m54.661870414s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790992970 (506h12m54.662076005s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791120053 (506h12m54.662203098s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791275218 (506h12m54.662358253s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791443709 (506h12m54.662526744s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:56.112313145 (506h12m55.98339619s) [upload=444191]: Next returned nil segment Dec 26 17:41:56.112330638 (506h12m55.983413683s) [upload=444191]: Waiting for error group managing segments... Dec 26 17:41:56.485322720 (506h12m56.356405765s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.485637079 (506h12m56.356720124s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.666807539 (506h12m56.537890584s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.678753987 (506h12m56.549837032s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.687788039 (506h12m56.558871074s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.694318023 (506h12m56.565401058s) [noise=true , address=145.79.227.7:4326 , piece_id=DMQ5RPI46X2U7TJTS42ZHQEF32BDJRKJJO5HWOYAPNS6VWVG5B5Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.716581187 (506h12m56.587664232s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.972183007 (506h12m56.843266042s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.011917999 (506h12m56.883001034s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.054168622 (506h12m56.925251667s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.114312777 (506h12m56.985395812s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.499846250 (506h12m57.370929285s) [noise=true , address=145.79.227.34:4919 , piece_id=YRWHW3BJVEGVJRSA6RJB3T7VJQSDTGKEZQKRICFIV2G3T7AN4MIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.612649066 (506h12m57.483732101s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649413733 (506h12m57.520496768s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649440282 (506h12m57.520523328s) [seg_pos={0 0}, upload=444191]: Segment reached optimal threshold of 14 pieces. Dec 26 17:41:57.649674729 (506h12m57.520757774s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G7ty71tPwe7cZSBNK9u74DFUCGomnxDsNvAif4C3SbP7HWwfo): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649684087 (506h12m57.520767142s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649747708 (506h12m57.520830763s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v6zXZuMmyL2MU8qb3q6gg1iwTYvndjzMjVpv8btEpeX9Lnqmu): protocol: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection Dec 26 17:41:57.649785219 (506h12m57.520868274s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649899828 (506h12m57.520982903s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649908775 (506h12m57.52099182s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650074641 (506h12m57.521157696s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.650083287 (506h12m57.521166342s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650132821 (506h12m57.521215866s) [seg_pos={0 0}, upload=444191]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 17:41:57.650220138 (506h12m57.521303183s) [upload=444191]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:41:57.650227171 (506h12m57.521310206s) [upload=444191]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:41:57.958890870 (506h12m57.829973915s) [upload=444191]: Done uploading segments... Dec 26 17:41:57.958911620 (506h12m57.829994665s) [upload=444191]: Upload finished. err: <nil> Duration: 1m0.005132294s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [..XX..X.X......XX..X....X.X.XX...XX...XX....XX..XX......X.XX] duration: 30.003094067s trace: https://jaeger.dev.storj.io/trace/62f7fc0c5a1ad946 FAIL (2025-12-26 17:42:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 128559683s) [upload=444192]: Uploading segments... Dec 26 17:42:02.257710684 (506h13m2.128793759s) [upload=444192]: Got next segment. Inline: false Dec 26 17:42:02.257822557 (506h13m2.128905602s) [upload=444192]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 17:42:02.257831124 (506h13m2.128914169s) [upload=444192]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 17:42:02.509474617 (506h13m2.380557652s) [seg_pos={0 0}, upload=444192]: Begin upload segment... Dec 26 17:42:02.509568656 (506h13m2.380651691s) [seg_pos={0 0}, upload=444192]: Done begin upload segment. Dec 26 17:42:02.509621667 (506h13m2.380704702s) [noise=true , address=145.79.227.39:4013 , piece_id=ZHXQX4KGHHMMJJ2D2FQLSXOB6THUPW26UE6LLD6RC2N2YBDWQP2A, seg_pos={0 0}, upload=444192]: Uploading piece... Dec 26 17:42:02.509664338 (506h13m2.380747373s) [noise=true , address=145.79.227.48:4302 , piece_id=DJPKMZBB77FKNNR6UQAV7G2YH3CHERJGCVKJ54LYU4OHQ7SBBSXA, seg_pos={0 0}, upload=444192]: Uploading piece... Dec 26 17:42:02.510339714 (506h13m2.381422759s) [noise=true , address=145.79.227.13:4803 , piece_id=F2HSVSTCIHL7EGMFGH6MRJWIBJSRCYCB5KIJ67BHNEY3KLQQB2KA, seg_pos={0 0}, upload=444192]: Uploading piece... Dec 26 17:42:02.510921102 (506h13m2.382004147s) [noise=true , address=145.79.227.46:4603 , piece_id=YR7URK5J6XQGPTKU7BEHRF5TJGFTJYMXIUXDVDMXT6A4NRSNV4MQ, seg_pos={0 0}, upload=444192]: Uploading piece... Dec 26 17:42:02.511399051 (506h13m2.382482096s) [noise=true , address=45.91.116.8:4048 , piece_id=Z3EDQ4LSQNHGX7WVEX27YJN3RP7AV3MFCBX7N4RA255PCPOKVBJQ, seg_pos={0 0}, upload=444192]: Uploading piece... Dec 26 17:42:02.511896359 (506h13m2.382979404s) [noise=true , address=145.79.227.47:4267 , piece_id=YUEXWTX5DPO6CWJ277BETHSMZUQOO7Z4KMZIRNIL2VDHBNJ2R3RA, seg_pos={0 0}, upload=444192]: Uploading piece... Dec 26 17:42:02.512392152 (506h13m2.383475218s) [noise=true , address=145.79.227.5:4223 , piece_id=K7HJCSGCIHC234FHEJ2J5DMPNFXOFFLGKJ7DJ5BVHAE3TN4H3IKA, seg_pos={0 0}, upload=444192]: Uploading piece... Dec 26 17:42:02.513842594 (506h13m2.384925669s) [noise=true , address=145.79.227.19:4078 , piece_id=7SJXYBFRRRPSLWA5KOSUZVL2QATOCX47XE44C7KTWQBBDTUQGCMQ, seg_pos={0 0}, upload=444192]: Uploading piece... Dec 26 17:42:02.514505586 (506h13m2.385588631s) [noise=true , address=145.79.227.49:4337 , piece_id=UEQJGQXQIBJFTBGNUNYSTHV7KPH7ICQGZXUF3IDJRFHGPRUOEUUQ, seg_pos={0 0}, upload=444192]: Uploading piece... Dec 26 17:42:02.515700592 (506h13m2.386783637s) [noise=true , address=45.91.116.12:4227 , piece_id=KP2REIHSVRVMUMWN6ELXUTWAHKPW4RJMIBKSOZHRGRHSL6ZAC3WQ, seg_pos={0 0}, upload=444192]: Uploading piece... Dec 26 17:42:02.517455193 (506h13m2.388538238s) [noise=true , address=45.91.116.13:4154 , piece_id=3KPQXHK7I5FQUZOSYO2J2V4RLZZRBU6PDKETK6GOOXS4O32SR4FA, seg_pos={0 0}, upload=444192]: Uploading piece... Dec 26 17:42:02.520346198 (506h13m2.391429243s) [noise=true , address=45.91.116.6:4975 , piece_id=4LIPZX75NBQIBHIPBDSNOV7HGCZW77CLT4AGYBW77UWSK5UJPTWA, seg_pos={0 0}, upload=444192]: Uploading piece... Dec 26 17:42:02.523210573 (506h13m2.394293608s) [noise=true , address=145.79.227.36:4517 , piece_id=2LUCPUCCOP3KCVXT5HO622KJ6K3CVPH6OUBMWXTK2CPLRO3ZKMIA, seg_pos={0 0}, upload=444192]: Uploading piece... Dec 26 17:42:02.525217644 (506h13m2.396300679s) [noise=true , address=145.79.227.14:4839 , piece_id=WWSM7LOWX4BYGFVQA4BL3PGZJTFMJFD5DEJ7FBR5KO55DNB3EAZA, seg_pos={0 0}, upload=444192]: Uploading piece... Dec 26 17:42:02.525491245 (506h13m2.39657429s) [noise=true , address=77.90.31.10:4519 , piece_id=UZNXHJJTYNPOSLTOLSVKX3GWA54TVAIPCHWWIWRCSJMMANCI62HQ, seg_pos={0 0}, upload=444192]: Uploading piece... Dec 26 17:42:02.527686766 (506h13m2.398769811s) [noise=true , address=77.90.31.21:4880 , piece_id=GXZP4U6SQ6TVO6F6KGBDUZIAPL3QLVRFFVHJKOSDCMHL55G4N37A, seg_pos={0 0}, upload=444192]: Uploading piece... Dec 26 17:42:04.223512734 (506h13m4.09459577s) [upload=444192]: Next returned nil segment Dec 26 17:42:04.223527622 (506h13m4.094610667s) [upload=444192]: Waiting for error group managing segments... Dec 26 17:42:04.507405456 (506h13m4.378488501s) [noise=true , address=145.79.227.49:4337 , piece_id=UEQJGQXQIBJFTBGNUNYSTHV7KPH7ICQGZXUF3IDJRFHGPRUOEUUQ, seg_pos={0 0}, upload=444192]: Done uploading piece. err: <nil> Dec 26 17:42:04.516976481 (506h13m4.388059526s) [noise=true , address=145.79.227.48:4302 , piece_id=DJPKMZBB77FKNNR6UQAV7G2YH3CHERJGCVKJ54LYU4OHQ7SBBSXA, seg_pos={0 0}, upload=444192]: Done uploading piece. err: <nil> Dec 26 17:42:04.526249758 (506h13m4.397332803s) [noise=true , address=145.79.227.47:4267 , piece_id=YUEXWTX5DPO6CWJ277BETHSMZUQOO7Z4KMZIRNIL2VDHBNJ2R3RA, seg_pos={0 0}, upload=444192]: Done uploading piece. err: <nil> Dec 26 17:42:04.534707944 (506h13m4.405790989s) [noise=true , address=145.79.227.13:4803 , piece_id=F2HSVSTCIHL7EGMFGH6MRJWIBJSRCYCB5KIJ67BHNEY3KLQQB2KA, seg_pos={0 0}, upload=444192]: Done uploading piece. err: <nil> Dec 26 17:42:04.543854872 (506h13m4.414937917s) [noise=true , address=145.79.227.46:4603 , piece_id=YR7URK5J6XQGPTKU7BEHRF5TJGFTJYMXIUXDVDMXT6A4NRSNV4MQ, seg_pos={0 0}, upload=444192]: Done uploading piece. err: <nil> Dec 26 17:42:04.544624798 (506h13m4.415707843s) [noise=true , address=145.79.227.39:4013 , piece_id=ZHXQX4KGHHMMJJ2D2FQLSXOB6THUPW26UE6LLD6RC2N2YBDWQP2A, seg_pos={0 0}, upload=444192]: Done uploading piece. err: <nil> Dec 26 17:42:04.548648781 (506h13m4.419731816s) [noise=true , address=145.79.227.36:4517 , piece_id=2LUCPUCCOP3KCVXT5HO622KJ6K3CVPH6OUBMWXTK2CPLRO3ZKMIA, seg_pos={0 0}, upload=444192]: Done uploading piece. err: <nil> Dec 26 17:42:04.565514746 (506h13m4.436597781s) [noise=true , address=145.79.227.5:4223 , piece_id=K7HJCSGCIHC234FHEJ2J5DMPNFXOFFLGKJ7DJ5BVHAE3TN4H3IKA, seg_pos={0 0}, upload=444192]: Done uploading piece. err: <nil> Dec 26 17:42:04.576429810 (506h13m4.447512855s) [noise=true , address=145.79.227.14:4839 , piece_id=WWSM7LOWX4BYGFVQA4BL3PGZJTFMJFD5DEJ7FBR5KO55DNB3EAZA, seg_pos={0 0}, upload=444192]: Done uploading piece. err: <nil> Dec 26 17:42:04.714734266 (506h13m4.585817311s) [noise=true , address=77.90.31.21:4880 , piece_id=GXZP4U6SQ6TVO6F6KGBDUZIAPL3QLVRFFVHJKOSDCMHL55G4N37A, seg_pos={0 0}, upload=444192]: Done uploading piece. err: <nil> Dec 26 17:42:04.834789672 (506h13m4.705872707s) [noise=true , address=45.91.116.6:4975 , piece_id=4LIPZX75NBQIBHIPBDSNOV7HGCZW77CLT4AGYBW77UWSK5UJPTWA, seg_pos={0 0}, upload=444192]: Done uploading piece. err: <nil> Dec 26 17:42:04.852580028 (506h13m4.723663073s) [noise=true , address=77.90.31.10:4519 , piece_id=UZNXHJJTYNPOSLTOLSVKX3GWA54TVAIPCHWWIWRCSJMMANCI62HQ, seg_pos={0 0}, upload=444192]: Done uploading piece. err: <nil> Dec 26 17:42:04.929238529 (506h13m4.800321574s) [noise=true , address=45.91.116.13:4154 , piece_id=3KPQXHK7I5FQUZOSYO2J2V4RLZZRBU6PDKETK6GOOXS4O32SR4FA, seg_pos={0 0}, upload=444192]: Done uploading piece. err: <nil> Dec 26 17:42:04.929261032 (506h13m4.800344077s) [seg_pos={0 0}, upload=444192]: Segment reached optimal threshold of 13 pieces. Dec 26 17:42:04.929471012 (506h13m4.800554047s) [noise=true , address=45.91.116.8:4048 , piece_id=Z3EDQ4LSQNHGX7WVEX27YJN3RP7AV3MFCBX7N4RA255PCPOKVBJQ, seg_pos={0 0}, upload=444192]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 26 17:42:04.929479758 (506h13m4.800562803s) [noise=true , address=45.91.116.8:4048 , piece_id=Z3EDQ4LSQNHGX7WVEX27YJN3RP7AV3MFCBX7N4RA255PCPOKVBJQ, seg_pos={0 0}, upload=444192]: Long tail cancellation due to optimal threshold Dec 26 17:42:04.929487975 (506h13m4.80057101s) [noise=true , address=145.79.227.19:4078 , piece_id=7SJXYBFRRRPSLWA5KOSUZVL2QATOCX47XE44C7KTWQBBDTUQGCMQ, seg_pos={0 0}, upload=444192]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: write tcp 188.166.225.170:49786->145.79.227.19:4078: use of closed network connection; write tcp 188.166.225.170:49786->145.79.227.19:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49786->145.79.227.19:4078: use of closed network connection Dec 26 17:42:04.929496270 (506h13m4.800579305s) [noise=true , address=145.79.227.19:4078 , piece_id=7SJXYBFRRRPSLWA5KOSUZVL2QATOCX47XE44C7KTWQBBDTUQGCMQ, seg_pos={0 0}, upload=444192]: Long tail cancellation due to optimal threshold Dec 26 17:42:04.929677445 (506h13m4.80076048s) [noise=true , address=45.91.116.12:4227 , piece_id=KP2REIHSVRVMUMWN6ELXUTWAHKPW4RJMIBKSOZHRGRHSL6ZAC3WQ, seg_pos={0 0}, upload=444192]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: write tcp 188.166.225.170:42854->45.91.116.12:4227: use of closed network connection; write tcp 188.166.225.170:42854->45.91.116.12:4227: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42854->45.91.116.12:4227: use of closed network connection Dec 26 17:42:04.929685991 (506h13m4.800769036s) [noise=true , address=45.91.116.12:4227 , piece_id=KP2REIHSVRVMUMWN6ELXUTWAHKPW4RJMIBKSOZHRGRHSL6ZAC3WQ, seg_pos={0 0}, upload=444192]: Long tail cancellation due to optimal threshold Dec 26 17:42:04.929702323 (506h13m4.800785368s) [seg_pos={0 0}, upload=444192]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 17:42:04.929770513 (506h13m4.800853548s) [upload=444192]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:42:04.929775582 (506h13m4.800858627s) [upload=444192]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:42:05.230190364 (506h13m5.101273409s) [upload=444192]: Done uploading segments... Dec 26 17:42:05.230246571 (506h13m5.101329616s) [upload=444192]: Upload finished. err: <nil> Duration: 30.003113675s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [X..XX..X.X......XX..X....X.X.XX...XX...XX....XX..XX......X.X] duration: 30.001739737s trace: https://jaeger.dev.storj.io/trace/4daf2ff92b9c8b4c FAIL (2025-12-26 17:41:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: TDCV5LZE3KD5K5HGON6QJDR3ZVGOWIINNZSU7GFRPEFYL453EW3Q) context deadline exceeded; piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: CS7JGSEQKFLYTU7COQGR6PF57ZNBT4TLXE2Y5F64YALZAZBD33GA) context deadline exceeded ======LOGS====== Dec 26 17:41:30.342176247 (506h12m30.213259292s) [upload=444184]: Starting upload Dec 26 17:41:30.342266378 (506h12m30.213349423s) [upload=444184]: Done starting upload Dec 26 17:41:30.342349247 (506h12m30.213432282s) [upload=444184]: Uploading segments... Dec 26 17:41:30.342700175 (506h12m30.21378321s) [upload=444184]: Got next segment. Inline: false Dec 26 17:41:30.342735702 (506h12m30.213818737s) [upload=444184]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 17:41:30.342740772 (506h12m30.213823817s) [upload=444184]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 17:41:30.590907063 (506h12m30.461990098s) [seg_pos={0 0}, upload=444184]: Begin upload segment... Dec 26 17:41:30.590979241 (506h12m30.462062276s) [seg_pos={0 0}, upload=444184]: Done begin upload segment. Dec 26 17:41:30.591006603 (506h12m30.462089638s) [noise=true , address=77.90.31.11:4554 , piece_id=4XMYTK5W5YVGAOTGP2SK2SS76YCWGOHNFJDWKHGEYATNROBHGI2A, seg_pos={0 0}, upload=444184]: Uploading piece... Dec 26 17:41:30.591086745 (506h12m30.46216978s) [noise=true , address=45.91.116.8:4048 , piece_id=CS7JGSEQKFLYTU7COQGR6PF57ZNBT4TLXE2Y5F64YALZAZBD33GA, seg_pos={0 0}, upload=444184]: Uploading piece... Dec 26 17:41:30.591675036 (506h12m30.462758071s) [noise=true , address=145.79.227.21:4475 , piece_id=FMLEX77GUSP75ZQRE2FFE6T4UOI44DLSMGC2NYIMDSN3QDJ36WFA, seg_pos={0 0}, upload=444184]: Uploading piece... Dec 26 17:41:30.592077112 (506h12m30.463160147s) [noise=true , address=145.79.227.37:4566 , piece_id=GBQNL6EO7W66TKHUIHEJMWOVKQEQVSPGMDIMESNS7E5MNB6GY42Q, seg_pos={0 0}, upload=444184]: Uploading piece... Dec 26 17:41:30.592383405 (506h12m30.46346644s) [noise=true , address=45.91.116.6:4974 , piece_id=D5JZIKIUXTR5PTHON47D3Z4FE6YBKYVWZ5KTPMLIP33GLTZ55ADQ, seg_pos={0 0}, upload=444184]: Uploading piece... Dec 26 17:41:30.592429613 (506h12m30.463512658s) [noise=true , address=145.79.227.22:4511 , piece_id=3TKSRDFRJBWLK3DA65X6SCVLPJYHFIRZLC5XDXVEP2UGLL735SZA, seg_pos={0 0}, upload=444184]: Uploading piece... Dec 26 17:41:30.592812191 (506h12m30.463895236s) [noise=true , address=45.91.116.12:4226 , piece_id=Z2T4CXPYD5JDRNSUOXP3VR452LPJH7YVUZNSE75JWVCQX22KYOHQ, seg_pos={0 0}, upload=444184]: Uploading piece... Dec 26 17:41:30.593194209 (506h12m30.464277244s) [noise=true , address=145.79.227.13:4804 , piece_id=Y7N25A3HJHTO2527XMFIQDVLY7FWGBHSSWPTKKBT7LQIS2OKHLXA, seg_pos={0 0}, upload=444184]: Uploading piece... Dec 26 17:41:30.593264302 (506h12m30.464347347s) [noise=true , address=145.79.227.49:4338 , piece_id=35KBWBUCDNCTN5FXESYQG4AI67AKC6HB3RWUHPLWULAZLFR6MIPA, seg_pos={0 0}, upload=444184]: Uploading piece... Dec 26 17:41:30.593562921 (506h12m30.464645966s) [noise=true , address=145.79.227.47:4267 , piece_id=QFNETFIAYWOI3O5XIR3IAR6DFURRXQNETDCALTUMZXEL7S5RZOYQ, seg_pos={0 0}, upload=444184]: Uploading piece... Dec 26 17:41:30.594016024 (506h12m30.465099059s) [noise=true , address=145.79.227.17:4970 , piece_id=YF6HQAP2FPLHNRVIOGEIPBZIL6EQRSRTS7IKGW3BN46YZ4MKOSHA, seg_pos={0 0}, upload=444184]: Uploading piece... Dec 26 17:41:30.594023799 (506h12m30.465106834s) [noise=true , address=145.79.227.36:4518 , piece_id=3TILE5SGVL7F2V3PKEPCHWVYJFFY7633PGPV73UP4VPAD2IXWSKQ, seg_pos={0 0}, upload=444184]: Uploading piece... Dec 26 17:41:30.594732518 (506h12m30.465815553s) [noise=true , address=145.79.227.15:4874 , piece_id=FTS327KZ6NTCKUSDA7X73Z72GSHYFR3U3CKHJNJXCUGRVN37RDQA, seg_pos={0 0}, upload=444184]: Uploading piece... Dec 26 17:41:30.595188456 (506h12m30.466271501s) [noise=true , address=77.90.31.13:4627 , piece_id=TDCV5LZE3KD5K5HGON6QJDR3ZVGOWIINNZSU7GFRPEFYL453EW3Q, seg_pos={0 0}, upload=444184]: Uploading piece... Dec 26 17:41:30.595816301 (506h12m30.466899336s) [noise=true , address=77.90.31.15:4700 , piece_id=GKF55DIDDSNFDPTZMZMATPHONO6UCPMPJL7KQ64D3T735Z46PCAA, seg_pos={0 0}, upload=444184]: Uploading piece... Dec 26 17:41:30.597630426 (506h12m30.468713461s) [noise=true , address=145.79.227.50:4375 , piece_id=77QS7NRHT254LFYLEURHB2HA27K4SMSPWH3H2HPXH2GC5GIE7VQA, seg_pos={0 0}, upload=444184]: Uploading piece... Dec 26 17:41:31.676905454 (506h12m31.547988489s) [noise=true , address=77.90.31.11:4554 , piece_id=4XMYTK5W5YVGAOTGP2SK2SS76YCWGOHNFJDWKHGEYATNROBHGI2A, seg_pos={0 0}, upload=444184]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: storage node overloaded, request limit: 2000 Dec 26 17:41:32.217415931 (506h12m32.088498976s) [upload=444184]: Next returned nil segment Dec 26 17:41:32.217428524 (506h12m32.088511559s) [upload=444184]: Waiting for error group managing segments... Dec 26 17:41:32.593052037 (506h12m32.464135072s) [noise=true , address=145.79.227.37:4566 , piece_id=GBQNL6EO7W66TKHUIHEJMWOVKQEQVSPGMDIMESNS7E5MNB6GY42Q, seg_pos={0 0}, upload=444184]: Done uploading piece. err: <nil> Dec 26 17:41:32.601468784 (506h12m32.472551829s) [noise=true , address=145.79.227.36:4518 , piece_id=3TILE5SGVL7F2V3PKEPCHWVYJFFY7633PGPV73UP4VPAD2IXWSKQ, seg_pos={0 0}, upload=444184]: Done uploading piece. err: <nil> Dec 26 17:41:32.602975623 (506h12m32.474058658s) [noise=true , address=145.79.227.15:4874 , piece_id=FTS327KZ6NTCKUSDA7X73Z72GSHYFR3U3CKHJNJXCUGRVN37RDQA, seg_pos={0 0}, upload=444184]: Done uploading piece. err: <nil> Dec 26 17:41:32.604016365 (506h12m32.4750994s) [noise=true , address=145.79.227.13:4804 , piece_id=Y7N25A3HJHTO2527XMFIQDVLY7FWGBHSSWPTKKBT7LQIS2OKHLXA, seg_pos={0 0}, upload=444184]: Done uploading piece. err: <nil> Dec 26 17:41:32.615802728 (506h12m32.486885764s) [noise=true , address=145.79.227.22:4511 , piece_id=3TKSRDFRJBWLK3DA65X6SCVLPJYHFIRZLC5XDXVEP2UGLL735SZA, seg_pos={0 0}, upload=444184]: Done uploading piece. err: <nil> Dec 26 17:41:32.617291944 (506h12m32.488374989s) [noise=true , address=145.79.227.50:4375 , piece_id=77QS7NRHT254LFYLEURHB2HA27K4SMSPWH3H2HPXH2GC5GIE7VQA, seg_pos={0 0}, upload=444184]: Done uploading piece. err: <nil> Dec 26 17:41:32.618485928 (506h12m32.489568963s) [noise=true , address=145.79.227.49:4338 , piece_id=35KBWBUCDNCTN5FXESYQG4AI67AKC6HB3RWUHPLWULAZLFR6MIPA, seg_pos={0 0}, upload=444184]: Done uploading piece. err: <nil> Dec 26 17:41:32.630331403 (506h12m32.501414448s) [noise=true , address=145.79.227.17:4970 , piece_id=YF6HQAP2FPLHNRVIOGEIPBZIL6EQRSRTS7IKGW3BN46YZ4MKOSHA, seg_pos={0 0}, upload=444184]: Done uploading piece. err: <nil> Dec 26 17:41:32.650549482 (506h12m32.521632527s) [noise=true , address=145.79.227.21:4475 , piece_id=FMLEX77GUSP75ZQRE2FFE6T4UOI44DLSMGC2NYIMDSN3QDJ36WFA, seg_pos={0 0}, upload=444184]: Done uploading piece. err: <nil> Dec 26 17:41:32.653130737 (506h12m32.524213772s) [noise=true , address=145.79.227.47:4267 , piece_id=QFNETFIAYWOI3O5XIR3IAR6DFURRXQNETDCALTUMZXEL7S5RZOYQ, seg_pos={0 0}, upload=444184]: Done uploading piece. err: <nil> Dec 26 17:41:32.929651411 (506h12m32.800734446s) [noise=true , address=77.90.31.15:4700 , piece_id=GKF55DIDDSNFDPTZMZMATPHONO6UCPMPJL7KQ64D3T735Z46PCAA, seg_pos={0 0}, upload=444184]: Done uploading piece. err: <nil> Dec 26 17:41:32.956934142 (506h12m32.828017187s) [noise=true , address=45.91.116.8:4048 , piece_id=CS7JGSEQKFLYTU7COQGR6PF57ZNBT4TLXE2Y5F64YALZAZBD33GA, seg_pos={0 0}, upload=444184]: Done uploading piece. err: <nil> Dec 26 17:41:33.082884172 (506h12m32.953967217s) [noise=true , address=77.90.31.13:4627 , piece_id=TDCV5LZE3KD5K5HGON6QJDR3ZVGOWIINNZSU7GFRPEFYL453EW3Q, seg_pos={0 0}, upload=444184]: Done uploading piece. err: <nil> Dec 26 17:41:33.082908609 (506h12m32.953991644s) [seg_pos={0 0}, upload=444184]: Segment reached optimal threshold of 13 pieces. Dec 26 17:41:33.083073163 (506h12m32.954156208s) [noise=true , address=45.91.116.6:4974 , piece_id=D5JZIKIUXTR5PTHON47D3Z4FE6YBKYVWZ5KTPMLIP33GLTZ55ADQ, seg_pos={0 0}, upload=444184]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:33.083080907 (506h12m32.954163952s) [noise=true , address=45.91.116.6:4974 , piece_id=D5JZIKIUXTR5PTHON47D3Z4FE6YBKYVWZ5KTPMLIP33GLTZ55ADQ, seg_pos={0 0}, upload=444184]: Long tail cancellation due to optimal threshold Dec 26 17:41:33.083197118 (506h12m32.954280163s) [noise=true , address=45.91.116.12:4226 , piece_id=Z2T4CXPYD5JDRNSUOXP3VR452LPJH7YVUZNSE75JWVCQX22KYOHQ, seg_pos={0 0}, upload=444184]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: write tcp 188.166.225.170:49340->45.91.116.12:4226: use of closed network connection; write tcp 188.166.225.170:49340->45.91.116.12:4226: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49340->45.91.116.12:4226: use of closed network connection Dec 26 17:41:33.083213610 (506h12m32.954296655s) [noise=true , address=45.91.116.12:4226 , piece_id=Z2T4CXPYD5JDRNSUOXP3VR452LPJH7YVUZNSE75JWVCQX22KYOHQ, seg_pos={0 0}, upload=444184]: Long tail cancellation due to optimal threshold Dec 26 17:41:33.083236613 (506h12m32.954319648s) [seg_pos={0 0}, upload=444184]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 17:41:33.083312879 (506h12m32.954395914s) [upload=444184]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:41:33.083318700 (506h12m32.954401745s) [upload=444184]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:41:33.386458081 (506h12m33.257541146s) [upload=444184]: Done uploading segments... Dec 26 17:41:33.386484981 (506h12m33.257568016s) [upload=444184]: Upload finished. err: <nil> Duration: 30.00177751s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [.X..XX..X.X......XX..X....X.X.XX...XX...XX....XX..XX......X.] duration: 5.786667976s trace: https://jaeger.dev.storj.io/trace/7dcc90e954987633 FAIL (2025-12-26 17:41:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: GUR4DFQ5S6GSYX7CL3O3SWXEZM7RMU6DDXMM4MGOBVCXR6J2YQJQ) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: E4SJ2HNBOZLDVYAUAHOLTDXWTZJECLTDPAUBPZYDZE7QRH3W7JSA) context deadline exceeded; piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: EIOWTJUKMOATRZF2UY3DLKGYESKIBFXS65CUCNAMKGJCSX3UG65Q) context deadline exceeded ======LOGS====== Dec 26 17:40:32.629030395 (506h11m32.50011343s) [upload=444170]: Starting upload Dec 26 17:40:32.629137850 (506h11m32.500220895s) [upload=444170]: Done starting upload Dec 26 17:40:32.629194116 (506h11m32.500277151s) [upload=444170]: Uploading segments... Dec 26 17:40:32.629411992 (506h11m32.500495027s) [upload=444170]: Got next segment. Inline: false Dec 26 17:40:32.629442109 (506h11m32.500525144s) [upload=444170]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 17:40:32.629446438 (506h11m32.500529473s) [upload=444170]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 17:40:32.882053776 (506h11m32.753136811s) [seg_pos={0 0}, upload=444170]: Begin upload segment... Dec 26 17:40:32.882112577 (506h11m32.753195622s) [seg_pos={0 0}, upload=444170]: Done begin upload segment. Dec 26 17:40:32.882167813 (506h11m32.753250858s) [noise=true , address=145.79.227.30:4762 , piece_id=ALSND2PZYHHJK6XR2KIG3WTXHKW4CCFY4HSUYJMB2A4KZW2EAFKA, seg_pos={0 0}, upload=444170]: Uploading piece... Dec 26 17:40:32.882193783 (506h11m32.753276818s) [noise=true , address=145.79.227.49:4338 , piece_id=BFF6GOFAJRXWKYP6ZHOPU3QVE5AYJSKR3XZCJM4XQQOHSW6NG4ZQ, seg_pos={0 0}, upload=444170]: Uploading piece... Dec 26 17:40:32.882272993 (506h11m32.753356028s) [noise=true , address=145.79.227.42:4122 , piece_id=6XAIHO7KXGJOZHXGM2YUKBFIG46LCUAJX76WYXC6Q3MKBMDENIFQ, seg_pos={0 0}, upload=444170]: Uploading piece... Dec 26 17:40:32.882711207 (506h11m32.753794242s) [noise=true , address=145.79.227.51:4410 , piece_id=M6HPWUA5BEBB2ORWPLBRPKJMUH75HPLBI7EUQBEHOXBWP4KLYGBA, seg_pos={0 0}, upload=444170]: Uploading piece... Dec 26 17:40:32.883030676 (506h11m32.754113721s) [noise=true , address=77.90.31.14:4663 , piece_id=GUR4DFQ5S6GSYX7CL3O3SWXEZM7RMU6DDXMM4MGOBVCXR6J2YQJQ, seg_pos={0 0}, upload=444170]: Uploading piece... Dec 26 17:40:32.883362978 (506h11m32.754446013s) [noise=true , address=145.79.227.35:4483 , piece_id=MKPPDDF6YT5QLO5CDY6ZRA7YZXTJ4MKDBGNDNHSSSU4L46BKOF3Q, seg_pos={0 0}, upload=444170]: Uploading piece... Dec 26 17:40:32.883714147 (506h11m32.754797182s) [noise=true , address=145.79.227.37:4565 , piece_id=YNEH6HUSM6ECPJN4JSQEQTRRUVP7VCDHPXT545NPDKAXHFIXY3QA, seg_pos={0 0}, upload=444170]: Uploading piece... Dec 26 17:40:32.884014119 (506h11m32.755097154s) [noise=true , address=145.79.227.48:4303 , piece_id=OFH7UK6PC44SMGM6YW2LYPHLIATMHPA5ZRGXAGXKIMJDZXWAFNLA, seg_pos={0 0}, upload=444170]: Uploading piece... Dec 26 17:40:32.884022635 (506h11m32.75510567s) [noise=true , address=145.79.227.47:4265 , piece_id=QZBDKRMGE3B4PR4LUCZSKSFVGPSGFTKL3CYFE6NY6MYM2DFWMXUA, seg_pos={0 0}, upload=444170]: Uploading piece... Dec 26 17:40:32.884311033 (506h11m32.755394068s) [noise=true , address=45.91.116.5:4300 , piece_id=TA4FHHIWP6UMXNANHDABUBVTQDXHGPMETVHIZZICOYKGAVOIDH3Q, seg_pos={0 0}, upload=444170]: Uploading piece... Dec 26 17:40:32.884607849 (506h11m32.755690884s) [noise=true , address=145.79.227.50:4375 , piece_id=BM7MUQLGCI3SW6RX2UHRP2DVTUYEI4XO3G2ESPTIQ5WID5COJIPQ, seg_pos={0 0}, upload=444170]: Uploading piece... Dec 26 17:40:32.884903872 (506h11m32.755986917s) [noise=true , address=45.91.116.10:4081 , piece_id=JZWRTSMB7S5YNQLD2MQSBVEHPFNDIWJOXYMG2XSI2B2DUYKTSMRA, seg_pos={0 0}, upload=444170]: Uploading piece... Dec 26 17:40:32.885165190 (506h11m32.756248235s) [noise=true , address=45.91.116.6:4974 , piece_id=E4SJ2HNBOZLDVYAUAHOLTDXWTZJECLTDPAUBPZYDZE7QRH3W7JSA, seg_pos={0 0}, upload=444170]: Uploading piece... Dec 26 17:40:32.885490119 (506h11m32.756573164s) [noise=true , address=77.90.31.21:4880 , piece_id=EIOWTJUKMOATRZF2UY3DLKGYESKIBFXS65CUCNAMKGJCSX3UG65Q, seg_pos={0 0}, upload=444170]: Uploading piece... Dec 26 17:40:32.886019257 (506h11m32.757102292s) [noise=true , address=145.79.227.38:4977 , piece_id=NNQ7PYEKLHDBU4TH3CIXDFFCDPSI2AC5OX5EONQHAUN5XMKNFNJA, seg_pos={0 0}, upload=444170]: Uploading piece... Dec 26 17:40:32.886513849 (506h11m32.757596884s) [noise=true , address=145.79.227.32:4836 , piece_id=CTAFA4SVTCPMCUL7H3HST6M3ZXWHYHPODBPMUAC3VURXOQ2DEHIA, seg_pos={0 0}, upload=444170]: Uploading piece... Dec 26 17:40:34.549562373 (506h11m34.420645418s) [upload=444170]: Next returned nil segment Dec 26 17:40:34.549627798 (506h11m34.420710843s) [upload=444170]: Waiting for error group managing segments... Dec 26 17:40:34.870488332 (506h11m34.741571377s) [noise=true , address=145.79.227.50:4375 , piece_id=BM7MUQLGCI3SW6RX2UHRP2DVTUYEI4XO3G2ESPTIQ5WID5COJIPQ, seg_pos={0 0}, upload=444170]: Done uploading piece. err: <nil> Dec 26 17:40:34.880316386 (506h11m34.751399431s) [noise=true , address=145.79.227.51:4410 , piece_id=M6HPWUA5BEBB2ORWPLBRPKJMUH75HPLBI7EUQBEHOXBWP4KLYGBA, seg_pos={0 0}, upload=444170]: Done uploading piece. err: <nil> Dec 26 17:40:34.882049948 (506h11m34.753132993s) [noise=true , address=145.79.227.38:4977 , piece_id=NNQ7PYEKLHDBU4TH3CIXDFFCDPSI2AC5OX5EONQHAUN5XMKNFNJA, seg_pos={0 0}, upload=444170]: Done uploading piece. err: <nil> Dec 26 17:40:34.886928637 (506h11m34.758011672s) [noise=true , address=145.79.227.32:4836 , piece_id=CTAFA4SVTCPMCUL7H3HST6M3ZXWHYHPODBPMUAC3VURXOQ2DEHIA, seg_pos={0 0}, upload=444170]: Done uploading piece. err: <nil> Dec 26 17:40:34.887098591 (506h11m34.758181626s) [noise=true , address=145.79.227.37:4565 , piece_id=YNEH6HUSM6ECPJN4JSQEQTRRUVP7VCDHPXT545NPDKAXHFIXY3QA, seg_pos={0 0}, upload=444170]: Done uploading piece. err: <nil> Dec 26 17:40:34.895364682 (506h11m34.766447717s) [noise=true , address=145.79.227.47:4265 , piece_id=QZBDKRMGE3B4PR4LUCZSKSFVGPSGFTKL3CYFE6NY6MYM2DFWMXUA, seg_pos={0 0}, upload=444170]: Done uploading piece. err: <nil> Dec 26 17:40:34.902481413 (506h11m34.773564448s) [noise=true , address=145.79.227.48:4303 , piece_id=OFH7UK6PC44SMGM6YW2LYPHLIATMHPA5ZRGXAGXKIMJDZXWAFNLA, seg_pos={0 0}, upload=444170]: Done uploading piece. err: <nil> Dec 26 17:40:34.907683850 (506h11m34.778766895s) [noise=true , address=145.79.227.49:4338 , piece_id=BFF6GOFAJRXWKYP6ZHOPU3QVE5AYJSKR3XZCJM4XQQOHSW6NG4ZQ, seg_pos={0 0}, upload=444170]: Done uploading piece. err: <nil> Dec 26 17:40:34.910066416 (506h11m34.781149451s) [noise=true , address=145.79.227.42:4122 , piece_id=6XAIHO7KXGJOZHXGM2YUKBFIG46LCUAJX76WYXC6Q3MKBMDENIFQ, seg_pos={0 0}, upload=444170]: Done uploading piece. err: <nil> Dec 26 17:40:34.910541110 (506h11m34.781624145s) [noise=true , address=145.79.227.35:4483 , piece_id=MKPPDDF6YT5QLO5CDY6ZRA7YZXTJ4MKDBGNDNHSSSU4L46BKOF3Q, seg_pos={0 0}, upload=444170]: Done uploading piece. err: <nil> Dec 26 17:40:35.054049024 (506h11m34.925132059s) [noise=true , address=77.90.31.21:4880 , piece_id=EIOWTJUKMOATRZF2UY3DLKGYESKIBFXS65CUCNAMKGJCSX3UG65Q, seg_pos={0 0}, upload=444170]: Done uploading piece. err: <nil> Dec 26 17:40:35.137537921 (506h11m35.008620966s) [noise=true , address=45.91.116.6:4974 , piece_id=E4SJ2HNBOZLDVYAUAHOLTDXWTZJECLTDPAUBPZYDZE7QRH3W7JSA, seg_pos={0 0}, upload=444170]: Done uploading piece. err: <nil> Dec 26 17:40:35.528738526 (506h11m35.399821561s) [noise=true , address=77.90.31.14:4663 , piece_id=GUR4DFQ5S6GSYX7CL3O3SWXEZM7RMU6DDXMM4MGOBVCXR6J2YQJQ, seg_pos={0 0}, upload=444170]: Done uploading piece. err: <nil> Dec 26 17:40:35.528759806 (506h11m35.399842841s) [seg_pos={0 0}, upload=444170]: Segment reached optimal threshold of 13 pieces. Dec 26 17:40:35.529016585 (506h11m35.400099621s) [noise=true , address=45.91.116.5:4300 , piece_id=TA4FHHIWP6UMXNANHDABUBVTQDXHGPMETVHIZZICOYKGAVOIDH3Q, seg_pos={0 0}, upload=444170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Dec 26 17:40:35.529025081 (506h11m35.400108116s) [noise=true , address=45.91.116.5:4300 , piece_id=TA4FHHIWP6UMXNANHDABUBVTQDXHGPMETVHIZZICOYKGAVOIDH3Q, seg_pos={0 0}, upload=444170]: Long tail cancellation due to optimal threshold Dec 26 17:40:35.529363877 (506h11m35.400446912s) [noise=true , address=45.91.116.10:4081 , piece_id=JZWRTSMB7S5YNQLD2MQSBVEHPFNDIWJOXYMG2XSI2B2DUYKTSMRA, seg_pos={0 0}, upload=444170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: read tcp 188.166.225.170:56048->45.91.116.10:4081: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 17:40:35.529384375 (506h11m35.40046744s) [noise=true , address=145.79.227.30:4762 , piece_id=ALSND2PZYHHJK6XR2KIG3WTXHKW4CCFY4HSUYJMB2A4KZW2EAFKA, seg_pos={0 0}, upload=444170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 26 17:40:35.529396619 (506h11m35.400479664s) [noise=true , address=145.79.227.30:4762 , piece_id=ALSND2PZYHHJK6XR2KIG3WTXHKW4CCFY4HSUYJMB2A4KZW2EAFKA, seg_pos={0 0}, upload=444170]: Long tail cancellation due to optimal threshold Dec 26 17:40:35.529417949 (506h11m35.400501074s) [noise=true , address=45.91.116.10:4081 , piece_id=JZWRTSMB7S5YNQLD2MQSBVEHPFNDIWJOXYMG2XSI2B2DUYKTSMRA, seg_pos={0 0}, upload=444170]: Long tail cancellation due to optimal threshold Dec 26 17:40:35.529439471 (506h11m35.400522516s) [seg_pos={0 0}, upload=444170]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 17:40:35.529523660 (506h11m35.400606706s) [upload=444170]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:40:35.529530263 (506h11m35.400613298s) [upload=444170]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:40:35.824915358 (506h11m35.695998403s) [upload=444170]: Done uploading segments... Dec 26 17:40:35.824934655 (506h11m35.69601769s) [upload=444170]: Upload finished. err: <nil> Duration: 5.786694606s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [X.X..XX..X.X......XX..X....X.X.XX...XX...XX....XX..XX......X] duration: 30.004171249s trace: https://jaeger.dev.storj.io/trace/7dcc90e954987633 FAIL (2025-12-26 17:41:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: GUR4DFQ5S6GSYX7CL3O3SWXEZM7RMU6DDXMM4MGOBVCXR6J2YQJQ) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: E4SJ2HNBOZLDVYAUAHOLTDXWTZJECLTDPAUBPZYDZE7QRH3W7JSA) context deadline exceeded; piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: EIOWTJUKMOATRZF2UY3DLKGYESKIBFXS65CUCNAMKGJCSX3UG65Q) context deadline exceeded ======LOGS====== Dec 26 17:40:32.629030395 (506h11m32.50011343s) [upload=444170]: Starting upload Dec 26 17:40:32.629137850 (506h11m32.500220895s) [upload=444170]: Done starting upload Dec 26 17:40:32.629194116 (506h11m32.500277151s) [upload=444170]: Uploading segments... Dec 26 17:40:32.629411992 (506h11m32.500495027s) [upload=444170]: Got next segment. Inline: false Dec 26 17:40:32.629442109 (506h11m32.500525144s) [upload=444170]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 17:40:32.629446438 (506h11m32.500529473s) [upload=444170]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 17:40:32.882053776 (506h11m32.753136811s) [seg_pos={0 0}, upload=444170]: Begin upload segment... Dec 26 17:40:32.882112577 (506h11m32.753195622s) [seg_pos={0 0}, upload=444170]: Done begin upload segment. Dec 26 17:40:32.882167813 (506h11m32.753250858s) [noise=true , address=145.79.227.30:4762 , piece_id=ALSND2PZYHHJK6XR2KIG3WTXHKW4CCFY4HSUYJMB2A4KZW2EAFKA, seg_pos={0 0}, upload=444170]: Uploading piece... Dec 26 17:40:32.882193783 (506h11m32.753276818s) [noise=true , address=145.79.227.49:4338 , piece_id=BFF6GOFAJRXWKYP6ZHOPU3QVE5AYJSKR3XZCJM4XQQOHSW6NG4ZQ, seg_pos={0 0}, upload=444170]: Uploading piece... Dec 26 17:40:32.882272993 (506h11m32.753356028s) [noise=true , address=145.79.227.42:4122 , piece_id=6XAIHO7KXGJOZHXGM2YUKBFIG46LCUAJX76WYXC6Q3MKBMDENIFQ, seg_pos={0 0}, upload=444170]: Uploading piece... Dec 26 17:40:32.882711207 (506h11m32.753794242s) [noise=true , address=145.79.227.51:4410 , piece_id=M6HPWUA5BEBB2ORWPLBRPKJMUH75HPLBI7EUQBEHOXBWP4KLYGBA, seg_pos={0 0}, upload=444170]: Uploading piece... Dec 26 17:40:32.883030676 (506h11m32.754113721s) [noise=true , address=77.90.31.14:4663 , piece_id=GUR4DFQ5S6GSYX7CL3O3SWXEZM7RMU6DDXMM4MGOBVCXR6J2YQJQ, seg_pos={0 0}, upload=444170]: Uploading piece... Dec 26 17:40:32.883362978 (506h11m32.754446013s) [noise=true , address=145.79.227.35:4483 , piece_id=MKPPDDF6YT5QLO5CDY6ZRA7YZXTJ4MKDBGNDNHSSSU4L46BKOF3Q, seg_pos={0 0}, upload=444170]: Uploading piece... Dec 26 17:40:32.883714147 (506h11m32.754797182s) [noise=true , address=145.79.227.37:4565 , piece_id=YNEH6HUSM6ECPJN4JSQEQTRRUVP7VCDHPXT545NPDKAXHFIXY3QA, seg_pos={0 0}, upload=444170]: Uploading piece... Dec 26 17:40:32.884014119 (506h11m32.755097154s) [noise=true , address=145.79.227.48:4303 , piece_id=OFH7UK6PC44SMGM6YW2LYPHLIATMHPA5ZRGXAGXKIMJDZXWAFNLA, seg_pos={0 0}, upload=444170]: Uploading piece... Dec 26 17:40:32.884022635 (506h11m32.75510567s) [noise=true , address=145.79.227.47:4265 , piece_id=QZBDKRMGE3B4PR4LUCZSKSFVGPSGFTKL3CYFE6NY6MYM2DFWMXUA, seg_pos={0 0}, upload=444170]: Uploading piece... Dec 26 17:40:32.884311033 (506h11m32.755394068s) [noise=true , address=45.91.116.5:4300 , piece_id=TA4FHHIWP6UMXNANHDABUBVTQDXHGPMETVHIZZICOYKGAVOIDH3Q, seg_pos={0 0}, upload=444170]: Uploading piece... Dec 26 17:40:32.884607849 (506h11m32.755690884s) [noise=true , address=145.79.227.50:4375 , piece_id=BM7MUQLGCI3SW6RX2UHRP2DVTUYEI4XO3G2ESPTIQ5WID5COJIPQ, seg_pos={0 0}, upload=444170]: Uploading piece... Dec 26 17:40:32.884903872 (506h11m32.755986917s) [noise=true , address=45.91.116.10:4081 , piece_id=JZWRTSMB7S5YNQLD2MQSBVEHPFNDIWJOXYMG2XSI2B2DUYKTSMRA, seg_pos={0 0}, upload=444170]: Uploading piece... Dec 26 17:40:32.885165190 (506h11m32.756248235s) [noise=true , address=45.91.116.6:4974 , piece_id=E4SJ2HNBOZLDVYAUAHOLTDXWTZJECLTDPAUBPZYDZE7QRH3W7JSA, seg_pos={0 0}, upload=444170]: Uploading piece... Dec 26 17:40:32.885490119 (506h11m32.756573164s) [noise=true , address=77.90.31.21:4880 , piece_id=EIOWTJUKMOATRZF2UY3DLKGYESKIBFXS65CUCNAMKGJCSX3UG65Q, seg_pos={0 0}, upload=444170]: Uploading piece... Dec 26 17:40:32.886019257 (506h11m32.757102292s) [noise=true , address=145.79.227.38:4977 , piece_id=NNQ7PYEKLHDBU4TH3CIXDFFCDPSI2AC5OX5EONQHAUN5XMKNFNJA, seg_pos={0 0}, upload=444170]: Uploading piece... Dec 26 17:40:32.886513849 (506h11m32.757596884s) [noise=true , address=145.79.227.32:4836 , piece_id=CTAFA4SVTCPMCUL7H3HST6M3ZXWHYHPODBPMUAC3VURXOQ2DEHIA, seg_pos={0 0}, upload=444170]: Uploading piece... Dec 26 17:40:34.549562373 (506h11m34.420645418s) [upload=444170]: Next returned nil segment Dec 26 17:40:34.549627798 (506h11m34.420710843s) [upload=444170]: Waiting for error group managing segments... Dec 26 17:40:34.870488332 (506h11m34.741571377s) [noise=true , address=145.79.227.50:4375 , piece_id=BM7MUQLGCI3SW6RX2UHRP2DVTUYEI4XO3G2ESPTIQ5WID5COJIPQ, seg_pos={0 0}, upload=444170]: Done uploading piece. err: <nil> Dec 26 17:40:34.880316386 (506h11m34.751399431s) [noise=true , address=145.79.227.51:4410 , piece_id=M6HPWUA5BEBB2ORWPLBRPKJMUH75HPLBI7EUQBEHOXBWP4KLYGBA, seg_pos={0 0}, upload=444170]: Done uploading piece. err: <nil> Dec 26 17:40:34.882049948 (506h11m34.753132993s) [noise=true , address=145.79.227.38:4977 , piece_id=NNQ7PYEKLHDBU4TH3CIXDFFCDPSI2AC5OX5EONQHAUN5XMKNFNJA, seg_pos={0 0}, upload=444170]: Done uploading piece. err: <nil> Dec 26 17:40:34.886928637 (506h11m34.758011672s) [noise=true , address=145.79.227.32:4836 , piece_id=CTAFA4SVTCPMCUL7H3HST6M3ZXWHYHPODBPMUAC3VURXOQ2DEHIA, seg_pos={0 0}, upload=444170]: Done uploading piece. err: <nil> Dec 26 17:40:34.887098591 (506h11m34.758181626s) [noise=true , address=145.79.227.37:4565 , piece_id=YNEH6HUSM6ECPJN4JSQEQTRRUVP7VCDHPXT545NPDKAXHFIXY3QA, seg_pos={0 0}, upload=444170]: Done uploading piece. err: <nil> Dec 26 17:40:34.895364682 (506h11m34.766447717s) [noise=true , address=145.79.227.47:4265 , piece_id=QZBDKRMGE3B4PR4LUCZSKSFVGPSGFTKL3CYFE6NY6MYM2DFWMXUA, seg_pos={0 0}, upload=444170]: Done uploading piece. err: <nil> Dec 26 17:40:34.902481413 (506h11m34.773564448s) [noise=true , address=145.79.227.48:4303 , piece_id=OFH7UK6PC44SMGM6YW2LYPHLIATMHPA5ZRGXAGXKIMJDZXWAFNLA, seg_pos={0 0}, upload=444170]: Done uploading piece. err: <nil> Dec 26 17:40:34.907683850 (506h11m34.778766895s) [noise=true , address=145.79.227.49:4338 , piece_id=BFF6GOFAJRXWKYP6ZHOPU3QVE5AYJSKR3XZCJM4XQQOHSW6NG4ZQ, seg_pos={0 0}, upload=444170]: Done uploading piece. err: <nil> Dec 26 17:40:34.910066416 (506h11m34.781149451s) [noise=true , address=145.79.227.42:4122 , piece_id=6XAIHO7KXGJOZHXGM2YUKBFIG46LCUAJX76WYXC6Q3MKBMDENIFQ, seg_pos={0 0}, upload=444170]: Done uploading piece. err: <nil> Dec 26 17:40:34.910541110 (506h11m34.781624145s) [noise=true , address=145.79.227.35:4483 , piece_id=MKPPDDF6YT5QLO5CDY6ZRA7YZXTJ4MKDBGNDNHSSSU4L46BKOF3Q, seg_pos={0 0}, upload=444170]: Done uploading piece. err: <nil> Dec 26 17:40:35.054049024 (506h11m34.925132059s) [noise=true , address=77.90.31.21:4880 , piece_id=EIOWTJUKMOATRZF2UY3DLKGYESKIBFXS65CUCNAMKGJCSX3UG65Q, seg_pos={0 0}, upload=444170]: Done uploading piece. err: <nil> Dec 26 17:40:35.137537921 (506h11m35.008620966s) [noise=true , address=45.91.116.6:4974 , piece_id=E4SJ2HNBOZLDVYAUAHOLTDXWTZJECLTDPAUBPZYDZE7QRH3W7JSA, seg_pos={0 0}, upload=444170]: Done uploading piece. err: <nil> Dec 26 17:40:35.528738526 (506h11m35.399821561s) [noise=true , address=77.90.31.14:4663 , piece_id=GUR4DFQ5S6GSYX7CL3O3SWXEZM7RMU6DDXMM4MGOBVCXR6J2YQJQ, seg_pos={0 0}, upload=444170]: Done uploading piece. err: <nil> Dec 26 17:40:35.528759806 (506h11m35.399842841s) [seg_pos={0 0}, upload=444170]: Segment reached optimal threshold of 13 pieces. Dec 26 17:40:35.529016585 (506h11m35.400099621s) [noise=true , address=45.91.116.5:4300 , piece_id=TA4FHHIWP6UMXNANHDABUBVTQDXHGPMETVHIZZICOYKGAVOIDH3Q, seg_pos={0 0}, upload=444170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Dec 26 17:40:35.529025081 (506h11m35.400108116s) [noise=true , address=45.91.116.5:4300 , piece_id=TA4FHHIWP6UMXNANHDABUBVTQDXHGPMETVHIZZICOYKGAVOIDH3Q, seg_pos={0 0}, upload=444170]: Long tail cancellation due to optimal threshold Dec 26 17:40:35.529363877 (506h11m35.400446912s) [noise=true , address=45.91.116.10:4081 , piece_id=JZWRTSMB7S5YNQLD2MQSBVEHPFNDIWJOXYMG2XSI2B2DUYKTSMRA, seg_pos={0 0}, upload=444170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: read tcp 188.166.225.170:56048->45.91.116.10:4081: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 17:40:35.529384375 (506h11m35.40046744s) [noise=true , address=145.79.227.30:4762 , piece_id=ALSND2PZYHHJK6XR2KIG3WTXHKW4CCFY4HSUYJMB2A4KZW2EAFKA, seg_pos={0 0}, upload=444170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 26 17:40:35.529396619 (506h11m35.400479664s) [noise=true , address=145.79.227.30:4762 , piece_id=ALSND2PZYHHJK6XR2KIG3WTXHKW4CCFY4HSUYJMB2A4KZW2EAFKA, seg_pos={0 0}, upload=444170]: Long tail cancellation due to optimal threshold Dec 26 17:40:35.529417949 (506h11m35.400501074s) [noise=true , address=45.91.116.10:4081 , piece_id=JZWRTSMB7S5YNQLD2MQSBVEHPFNDIWJOXYMG2XSI2B2DUYKTSMRA, seg_pos={0 0}, upload=444170]: Long tail cancellation due to optimal threshold Dec 26 17:40:35.529439471 (506h11m35.400522516s) [seg_pos={0 0}, upload=444170]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 17:40:35.529523660 (506h11m35.400606706s) [upload=444170]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:40:35.529530263 (506h11m35.400613298s) [upload=444170]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:40:35.824915358 (506h11m35.695998403s) [upload=444170]: Done uploading segments... Dec 26 17:40:35.824934655 (506h11m35.69601769s) [upload=444170]: Upload finished. err: <nil> Duration: 30.004197779s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [.X.X..XX..X.X......XX..X....X.X.XX...XX...XX....XX..XX......] duration: 5.956572874s trace: https://jaeger.dev.storj.io/trace/5835db663f93224f FAIL (2025-12-26 17:39:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: PZBVVBULP6BCJP6ANFEIKXXHLOSYX55VGV65ST5VV3DJUCIOWV4A) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: LRDA54KUC5RYNCYLFGC7VRXSIXLIVZEEBR5LMBC7L4WQEQW67GRA) context deadline exceeded ======LOGS====== h5m36.320342257s) [upload=444089]: Uploading segments... Dec 26 17:34:36.449470063 (506h5m36.320553108s) [upload=444089]: Got next segment. Inline: false Dec 26 17:34:36.449502284 (506h5m36.320585329s) [upload=444089]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 17:34:36.449507534 (506h5m36.320590569s) [upload=444089]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 17:34:36.704238476 (506h5m36.575321511s) [seg_pos={0 0}, upload=444089]: Begin upload segment... Dec 26 17:34:36.704303771 (506h5m36.575386806s) [seg_pos={0 0}, upload=444089]: Done begin upload segment. Dec 26 17:34:36.704328949 (506h5m36.575411984s) [noise=true , address=145.79.227.39:4015 , piece_id=WPFE4H77F7OFWA5M73NKI7F3ALXVLH63AA6HB3PTQJR4EVEJQ4NQ, seg_pos={0 0}, upload=444089]: Uploading piece... Dec 26 17:34:36.704344969 (506h5m36.575428014s) [noise=true , address=145.79.227.29:4943 , piece_id=ODGVNYKPLIZBUYUNJZUHCJQ2K2ZURD5UPDWZS3F34JQSTBZ5WAAA, seg_pos={0 0}, upload=444089]: Uploading piece... Dec 26 17:34:36.704785899 (506h5m36.575868944s) [noise=true , address=145.79.227.14:4839 , piece_id=XUTEEECYLYYN4X5N7ATZL32B6LSQCCIJWDJZHORYRHAZTHJ3UE7A, seg_pos={0 0}, upload=444089]: Uploading piece... Dec 26 17:34:36.705072314 (506h5m36.576155349s) [noise=true , address=145.79.227.11:4729 , piece_id=H62Q4ZQ2NEQGJ4XQYUORX3TITDWQRJNHSBQ3XNSEBHMVCPUWU5TA, seg_pos={0 0}, upload=444089]: Uploading piece... Dec 26 17:34:36.705347178 (506h5m36.576430213s) [noise=true , address=145.79.227.51:4410 , piece_id=6KF6ECGI7J2YPW3LVDWA2RLOERJXVJR7ADZ7WDWUNAXZBWEO7CIA, seg_pos={0 0}, upload=444089]: Uploading piece... Dec 26 17:34:36.705666385 (506h5m36.57674943s) [noise=true , address=145.79.227.10:4401 , piece_id=LREJ32SPU7ABE5AG347AALAIL7IVFZTYSCISFD2HROEUIASA7CNA, seg_pos={0 0}, upload=444089]: Uploading piece... Dec 26 17:34:36.705672767 (506h5m36.576755802s) [noise=true , address=145.79.227.47:4265 , piece_id=V77EKXQ5GVAGSKL4W6CBJWDQMJOZXK6U67UO4D63ZWSISD7PBUJA, seg_pos={0 0}, upload=444089]: Uploading piece... Dec 26 17:34:36.705947450 (506h5m36.577030485s) [noise=true , address=45.91.116.9:4261 , piece_id=PZBVVBULP6BCJP6ANFEIKXXHLOSYX55VGV65ST5VV3DJUCIOWV4A, seg_pos={0 0}, upload=444089]: Uploading piece... Dec 26 17:34:36.706017995 (506h5m36.57710103s) [noise=true , address=145.79.227.21:4475 , piece_id=G66X2QLH27XZPW5F73COJD5CE5ZYB65TFI2OBNWPQRMQSTNI7XKQ, seg_pos={0 0}, upload=444089]: Uploading piece... Dec 26 17:34:36.706262210 (506h5m36.577345245s) [noise=true , address=145.79.227.49:4337 , piece_id=XS2JRKD7JNCMWPBFA6GFRO4TNQMWUSHJQ73UIDTGFEBSFHDQEWWA, seg_pos={0 0}, upload=444089]: Uploading piece... Dec 26 17:34:36.706304049 (506h5m36.577387084s) [noise=true , address=45.91.116.12:4227 , piece_id=24ITM3BBJS2KUYEWHQXATVWA7AH72V5245IJNCNDBDX4R6XV5BJQ, seg_pos={0 0}, upload=444089]: Uploading piece... Dec 26 17:34:36.706789504 (506h5m36.577872539s) [noise=true , address=45.91.116.6:4975 , piece_id=LRDA54KUC5RYNCYLFGC7VRXSIXLIVZEEBR5LMBC7L4WQEQW67GRA, seg_pos={0 0}, upload=444089]: Uploading piece... Dec 26 17:34:36.707126466 (506h5m36.578209501s) [noise=true , address=145.79.227.38:4978 , piece_id=5YINIP7MUYYNOBNHV4ZGUWE366QEWGVG3TWORNJDOJEPU3AIBJEQ, seg_pos={0 0}, upload=444089]: Uploading piece... Dec 26 17:34:36.707614815 (506h5m36.578697881s) [noise=true , address=77.90.31.14:4663 , piece_id=EB7GNCWQG3KGQ4GQ4I74J5EX2G54YWE62JJCI2HNPRV2BVZQKFPQ, seg_pos={0 0}, upload=444089]: Uploading piece... Dec 26 17:34:36.707973388 (506h5m36.579056423s) [noise=true , address=45.91.116.5:4300 , piece_id=MY7IEOWS4SFF4OYCWTZXBFTYXB3LZRA7EQJS5THZCF5IL62ZVSHQ, seg_pos={0 0}, upload=444089]: Uploading piece... Dec 26 17:34:36.708277437 (506h5m36.579360472s) [noise=true , address=145.79.227.42:4122 , piece_id=HLGY23RXNXCND25LS6JW35YX4TDJHH77OMA2JUE5GSFL76KYQ77Q, seg_pos={0 0}, upload=444089]: Uploading piece... Dec 26 17:34:38.412838239 (506h5m38.283921284s) [upload=444089]: Next returned nil segment Dec 26 17:34:38.412890217 (506h5m38.283973262s) [upload=444089]: Waiting for error group managing segments... Dec 26 17:34:38.702510425 (506h5m38.57359346s) [noise=true , address=145.79.227.38:4978 , piece_id=5YINIP7MUYYNOBNHV4ZGUWE366QEWGVG3TWORNJDOJEPU3AIBJEQ, seg_pos={0 0}, upload=444089]: Done uploading piece. err: <nil> Dec 26 17:34:38.703987367 (506h5m38.575070412s) [noise=true , address=145.79.227.39:4015 , piece_id=WPFE4H77F7OFWA5M73NKI7F3ALXVLH63AA6HB3PTQJR4EVEJQ4NQ, seg_pos={0 0}, upload=444089]: Done uploading piece. err: <nil> Dec 26 17:34:38.708568922 (506h5m38.579651957s) [noise=true , address=145.79.227.11:4729 , piece_id=H62Q4ZQ2NEQGJ4XQYUORX3TITDWQRJNHSBQ3XNSEBHMVCPUWU5TA, seg_pos={0 0}, upload=444089]: Done uploading piece. err: <nil> Dec 26 17:34:38.714031684 (506h5m38.585114719s) [noise=true , address=145.79.227.49:4337 , piece_id=XS2JRKD7JNCMWPBFA6GFRO4TNQMWUSHJQ73UIDTGFEBSFHDQEWWA, seg_pos={0 0}, upload=444089]: Done uploading piece. err: <nil> Dec 26 17:34:38.714566472 (506h5m38.585649507s) [noise=true , address=145.79.227.21:4475 , piece_id=G66X2QLH27XZPW5F73COJD5CE5ZYB65TFI2OBNWPQRMQSTNI7XKQ, seg_pos={0 0}, upload=444089]: Done uploading piece. err: <nil> Dec 26 17:34:38.724676302 (506h5m38.595759347s) [noise=true , address=145.79.227.14:4839 , piece_id=XUTEEECYLYYN4X5N7ATZL32B6LSQCCIJWDJZHORYRHAZTHJ3UE7A, seg_pos={0 0}, upload=444089]: Done uploading piece. err: <nil> Dec 26 17:34:38.726402780 (506h5m38.597485825s) [noise=true , address=145.79.227.10:4401 , piece_id=LREJ32SPU7ABE5AG347AALAIL7IVFZTYSCISFD2HROEUIASA7CNA, seg_pos={0 0}, upload=444089]: Done uploading piece. err: <nil> Dec 26 17:34:38.736250622 (506h5m38.607333677s) [noise=true , address=145.79.227.42:4122 , piece_id=HLGY23RXNXCND25LS6JW35YX4TDJHH77OMA2JUE5GSFL76KYQ77Q, seg_pos={0 0}, upload=444089]: Done uploading piece. err: <nil> Dec 26 17:34:38.740648155 (506h5m38.611731201s) [noise=true , address=145.79.227.51:4410 , piece_id=6KF6ECGI7J2YPW3LVDWA2RLOERJXVJR7ADZ7WDWUNAXZBWEO7CIA, seg_pos={0 0}, upload=444089]: Done uploading piece. err: <nil> Dec 26 17:34:38.744072547 (506h5m38.615155582s) [noise=true , address=145.79.227.47:4265 , piece_id=V77EKXQ5GVAGSKL4W6CBJWDQMJOZXK6U67UO4D63ZWSISD7PBUJA, seg_pos={0 0}, upload=444089]: Done uploading piece. err: <nil> Dec 26 17:34:38.969785032 (506h5m38.840868067s) [noise=true , address=45.91.116.6:4975 , piece_id=LRDA54KUC5RYNCYLFGC7VRXSIXLIVZEEBR5LMBC7L4WQEQW67GRA, seg_pos={0 0}, upload=444089]: Done uploading piece. err: <nil> Dec 26 17:34:39.263027698 (506h5m39.134110733s) [noise=true , address=45.91.116.9:4261 , piece_id=PZBVVBULP6BCJP6ANFEIKXXHLOSYX55VGV65ST5VV3DJUCIOWV4A, seg_pos={0 0}, upload=444089]: Done uploading piece. err: <nil> Dec 26 17:34:39.558290509 (506h5m39.429373544s) [noise=true , address=45.91.116.5:4300 , piece_id=MY7IEOWS4SFF4OYCWTZXBFTYXB3LZRA7EQJS5THZCF5IL62ZVSHQ, seg_pos={0 0}, upload=444089]: Done uploading piece. err: <nil> Dec 26 17:34:39.558412061 (506h5m39.429495096s) [seg_pos={0 0}, upload=444089]: Segment reached optimal threshold of 13 pieces. Dec 26 17:34:39.558680011 (506h5m39.429763046s) [noise=true , address=145.79.227.29:4943 , piece_id=ODGVNYKPLIZBUYUNJZUHCJQ2K2ZURD5UPDWZS3F34JQSTBZ5WAAA, seg_pos={0 0}, upload=444089]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz): protocol: expected piece hash; context canceled; EOF Dec 26 17:34:39.558692886 (506h5m39.429775931s) [noise=true , address=145.79.227.29:4943 , piece_id=ODGVNYKPLIZBUYUNJZUHCJQ2K2ZURD5UPDWZS3F34JQSTBZ5WAAA, seg_pos={0 0}, upload=444089]: Long tail cancellation due to optimal threshold Dec 26 17:34:39.558702083 (506h5m39.429785128s) [noise=true , address=77.90.31.14:4663 , piece_id=EB7GNCWQG3KGQ4GQ4I74J5EX2G54YWE62JJCI2HNPRV2BVZQKFPQ, seg_pos={0 0}, upload=444089]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: write tcp 188.166.225.170:38944->77.90.31.14:4663: use of closed network connection; write tcp 188.166.225.170:38944->77.90.31.14:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38944->77.90.31.14:4663: use of closed network connection Dec 26 17:34:39.558709006 (506h5m39.429792061s) [noise=true , address=77.90.31.14:4663 , piece_id=EB7GNCWQG3KGQ4GQ4I74J5EX2G54YWE62JJCI2HNPRV2BVZQKFPQ, seg_pos={0 0}, upload=444089]: Long tail cancellation due to optimal threshold Dec 26 17:34:39.558885092 (506h5m39.429968127s) [noise=true , address=45.91.116.12:4227 , piece_id=24ITM3BBJS2KUYEWHQXATVWA7AH72V5245IJNCNDBDX4R6XV5BJQ, seg_pos={0 0}, upload=444089]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: write tcp 188.166.225.170:34996->45.91.116.12:4227: use of closed network connection; write tcp 188.166.225.170:34996->45.91.116.12:4227: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34996->45.91.116.12:4227: use of closed network connection Dec 26 17:34:39.558891674 (506h5m39.429974709s) [noise=true , address=45.91.116.12:4227 , piece_id=24ITM3BBJS2KUYEWHQXATVWA7AH72V5245IJNCNDBDX4R6XV5BJQ, seg_pos={0 0}, upload=444089]: Long tail cancellation due to optimal threshold Dec 26 17:34:39.558909198 (506h5m39.429992243s) [seg_pos={0 0}, upload=444089]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 17:34:39.558975133 (506h5m39.430058168s) [upload=444089]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:34:39.558979602 (506h5m39.430062647s) [upload=444089]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:34:39.847668435 (506h5m39.71875147s) [upload=444089]: Done uploading segments... Dec 26 17:34:39.847712239 (506h5m39.718795274s) [upload=444089]: Upload finished. err: <nil> Duration: 5.956599255s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [..X.X..XX..X.X......XX..X....X.X.XX...XX...XX....XX..XX.....] duration: 6.054887002s trace: https://jaeger.dev.storj.io/trace/5835db663f93224f FAIL (2025-12-26 17:38:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: PZBVVBULP6BCJP6ANFEIKXXHLOSYX55VGV65ST5VV3DJUCIOWV4A) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: LRDA54KUC5RYNCYLFGC7VRXSIXLIVZEEBR5LMBC7L4WQEQW67GRA) context deadline exceeded ======LOGS====== h5m36.320342257s) [upload=444089]: Uploading segments... Dec 26 17:34:36.449470063 (506h5m36.320553108s) [upload=444089]: Got next segment. Inline: false Dec 26 17:34:36.449502284 (506h5m36.320585329s) [upload=444089]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 17:34:36.449507534 (506h5m36.320590569s) [upload=444089]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 17:34:36.704238476 (506h5m36.575321511s) [seg_pos={0 0}, upload=444089]: Begin upload segment... Dec 26 17:34:36.704303771 (506h5m36.575386806s) [seg_pos={0 0}, upload=444089]: Done begin upload segment. Dec 26 17:34:36.704328949 (506h5m36.575411984s) [noise=true , address=145.79.227.39:4015 , piece_id=WPFE4H77F7OFWA5M73NKI7F3ALXVLH63AA6HB3PTQJR4EVEJQ4NQ, seg_pos={0 0}, upload=444089]: Uploading piece... Dec 26 17:34:36.704344969 (506h5m36.575428014s) [noise=true , address=145.79.227.29:4943 , piece_id=ODGVNYKPLIZBUYUNJZUHCJQ2K2ZURD5UPDWZS3F34JQSTBZ5WAAA, seg_pos={0 0}, upload=444089]: Uploading piece... Dec 26 17:34:36.704785899 (506h5m36.575868944s) [noise=true , address=145.79.227.14:4839 , piece_id=XUTEEECYLYYN4X5N7ATZL32B6LSQCCIJWDJZHORYRHAZTHJ3UE7A, seg_pos={0 0}, upload=444089]: Uploading piece... Dec 26 17:34:36.705072314 (506h5m36.576155349s) [noise=true , address=145.79.227.11:4729 , piece_id=H62Q4ZQ2NEQGJ4XQYUORX3TITDWQRJNHSBQ3XNSEBHMVCPUWU5TA, seg_pos={0 0}, upload=444089]: Uploading piece... Dec 26 17:34:36.705347178 (506h5m36.576430213s) [noise=true , address=145.79.227.51:4410 , piece_id=6KF6ECGI7J2YPW3LVDWA2RLOERJXVJR7ADZ7WDWUNAXZBWEO7CIA, seg_pos={0 0}, upload=444089]: Uploading piece... Dec 26 17:34:36.705666385 (506h5m36.57674943s) [noise=true , address=145.79.227.10:4401 , piece_id=LREJ32SPU7ABE5AG347AALAIL7IVFZTYSCISFD2HROEUIASA7CNA, seg_pos={0 0}, upload=444089]: Uploading piece... Dec 26 17:34:36.705672767 (506h5m36.576755802s) [noise=true , address=145.79.227.47:4265 , piece_id=V77EKXQ5GVAGSKL4W6CBJWDQMJOZXK6U67UO4D63ZWSISD7PBUJA, seg_pos={0 0}, upload=444089]: Uploading piece... Dec 26 17:34:36.705947450 (506h5m36.577030485s) [noise=true , address=45.91.116.9:4261 , piece_id=PZBVVBULP6BCJP6ANFEIKXXHLOSYX55VGV65ST5VV3DJUCIOWV4A, seg_pos={0 0}, upload=444089]: Uploading piece... Dec 26 17:34:36.706017995 (506h5m36.57710103s) [noise=true , address=145.79.227.21:4475 , piece_id=G66X2QLH27XZPW5F73COJD5CE5ZYB65TFI2OBNWPQRMQSTNI7XKQ, seg_pos={0 0}, upload=444089]: Uploading piece... Dec 26 17:34:36.706262210 (506h5m36.577345245s) [noise=true , address=145.79.227.49:4337 , piece_id=XS2JRKD7JNCMWPBFA6GFRO4TNQMWUSHJQ73UIDTGFEBSFHDQEWWA, seg_pos={0 0}, upload=444089]: Uploading piece... Dec 26 17:34:36.706304049 (506h5m36.577387084s) [noise=true , address=45.91.116.12:4227 , piece_id=24ITM3BBJS2KUYEWHQXATVWA7AH72V5245IJNCNDBDX4R6XV5BJQ, seg_pos={0 0}, upload=444089]: Uploading piece... Dec 26 17:34:36.706789504 (506h5m36.577872539s) [noise=true , address=45.91.116.6:4975 , piece_id=LRDA54KUC5RYNCYLFGC7VRXSIXLIVZEEBR5LMBC7L4WQEQW67GRA, seg_pos={0 0}, upload=444089]: Uploading piece... Dec 26 17:34:36.707126466 (506h5m36.578209501s) [noise=true , address=145.79.227.38:4978 , piece_id=5YINIP7MUYYNOBNHV4ZGUWE366QEWGVG3TWORNJDOJEPU3AIBJEQ, seg_pos={0 0}, upload=444089]: Uploading piece... Dec 26 17:34:36.707614815 (506h5m36.578697881s) [noise=true , address=77.90.31.14:4663 , piece_id=EB7GNCWQG3KGQ4GQ4I74J5EX2G54YWE62JJCI2HNPRV2BVZQKFPQ, seg_pos={0 0}, upload=444089]: Uploading piece... Dec 26 17:34:36.707973388 (506h5m36.579056423s) [noise=true , address=45.91.116.5:4300 , piece_id=MY7IEOWS4SFF4OYCWTZXBFTYXB3LZRA7EQJS5THZCF5IL62ZVSHQ, seg_pos={0 0}, upload=444089]: Uploading piece... Dec 26 17:34:36.708277437 (506h5m36.579360472s) [noise=true , address=145.79.227.42:4122 , piece_id=HLGY23RXNXCND25LS6JW35YX4TDJHH77OMA2JUE5GSFL76KYQ77Q, seg_pos={0 0}, upload=444089]: Uploading piece... Dec 26 17:34:38.412838239 (506h5m38.283921284s) [upload=444089]: Next returned nil segment Dec 26 17:34:38.412890217 (506h5m38.283973262s) [upload=444089]: Waiting for error group managing segments... Dec 26 17:34:38.702510425 (506h5m38.57359346s) [noise=true , address=145.79.227.38:4978 , piece_id=5YINIP7MUYYNOBNHV4ZGUWE366QEWGVG3TWORNJDOJEPU3AIBJEQ, seg_pos={0 0}, upload=444089]: Done uploading piece. err: <nil> Dec 26 17:34:38.703987367 (506h5m38.575070412s) [noise=true , address=145.79.227.39:4015 , piece_id=WPFE4H77F7OFWA5M73NKI7F3ALXVLH63AA6HB3PTQJR4EVEJQ4NQ, seg_pos={0 0}, upload=444089]: Done uploading piece. err: <nil> Dec 26 17:34:38.708568922 (506h5m38.579651957s) [noise=true , address=145.79.227.11:4729 , piece_id=H62Q4ZQ2NEQGJ4XQYUORX3TITDWQRJNHSBQ3XNSEBHMVCPUWU5TA, seg_pos={0 0}, upload=444089]: Done uploading piece. err: <nil> Dec 26 17:34:38.714031684 (506h5m38.585114719s) [noise=true , address=145.79.227.49:4337 , piece_id=XS2JRKD7JNCMWPBFA6GFRO4TNQMWUSHJQ73UIDTGFEBSFHDQEWWA, seg_pos={0 0}, upload=444089]: Done uploading piece. err: <nil> Dec 26 17:34:38.714566472 (506h5m38.585649507s) [noise=true , address=145.79.227.21:4475 , piece_id=G66X2QLH27XZPW5F73COJD5CE5ZYB65TFI2OBNWPQRMQSTNI7XKQ, seg_pos={0 0}, upload=444089]: Done uploading piece. err: <nil> Dec 26 17:34:38.724676302 (506h5m38.595759347s) [noise=true , address=145.79.227.14:4839 , piece_id=XUTEEECYLYYN4X5N7ATZL32B6LSQCCIJWDJZHORYRHAZTHJ3UE7A, seg_pos={0 0}, upload=444089]: Done uploading piece. err: <nil> Dec 26 17:34:38.726402780 (506h5m38.597485825s) [noise=true , address=145.79.227.10:4401 , piece_id=LREJ32SPU7ABE5AG347AALAIL7IVFZTYSCISFD2HROEUIASA7CNA, seg_pos={0 0}, upload=444089]: Done uploading piece. err: <nil> Dec 26 17:34:38.736250622 (506h5m38.607333677s) [noise=true , address=145.79.227.42:4122 , piece_id=HLGY23RXNXCND25LS6JW35YX4TDJHH77OMA2JUE5GSFL76KYQ77Q, seg_pos={0 0}, upload=444089]: Done uploading piece. err: <nil> Dec 26 17:34:38.740648155 (506h5m38.611731201s) [noise=true , address=145.79.227.51:4410 , piece_id=6KF6ECGI7J2YPW3LVDWA2RLOERJXVJR7ADZ7WDWUNAXZBWEO7CIA, seg_pos={0 0}, upload=444089]: Done uploading piece. err: <nil> Dec 26 17:34:38.744072547 (506h5m38.615155582s) [noise=true , address=145.79.227.47:4265 , piece_id=V77EKXQ5GVAGSKL4W6CBJWDQMJOZXK6U67UO4D63ZWSISD7PBUJA, seg_pos={0 0}, upload=444089]: Done uploading piece. err: <nil> Dec 26 17:34:38.969785032 (506h5m38.840868067s) [noise=true , address=45.91.116.6:4975 , piece_id=LRDA54KUC5RYNCYLFGC7VRXSIXLIVZEEBR5LMBC7L4WQEQW67GRA, seg_pos={0 0}, upload=444089]: Done uploading piece. err: <nil> Dec 26 17:34:39.263027698 (506h5m39.134110733s) [noise=true , address=45.91.116.9:4261 , piece_id=PZBVVBULP6BCJP6ANFEIKXXHLOSYX55VGV65ST5VV3DJUCIOWV4A, seg_pos={0 0}, upload=444089]: Done uploading piece. err: <nil> Dec 26 17:34:39.558290509 (506h5m39.429373544s) [noise=true , address=45.91.116.5:4300 , piece_id=MY7IEOWS4SFF4OYCWTZXBFTYXB3LZRA7EQJS5THZCF5IL62ZVSHQ, seg_pos={0 0}, upload=444089]: Done uploading piece. err: <nil> Dec 26 17:34:39.558412061 (506h5m39.429495096s) [seg_pos={0 0}, upload=444089]: Segment reached optimal threshold of 13 pieces. Dec 26 17:34:39.558680011 (506h5m39.429763046s) [noise=true , address=145.79.227.29:4943 , piece_id=ODGVNYKPLIZBUYUNJZUHCJQ2K2ZURD5UPDWZS3F34JQSTBZ5WAAA, seg_pos={0 0}, upload=444089]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz): protocol: expected piece hash; context canceled; EOF Dec 26 17:34:39.558692886 (506h5m39.429775931s) [noise=true , address=145.79.227.29:4943 , piece_id=ODGVNYKPLIZBUYUNJZUHCJQ2K2ZURD5UPDWZS3F34JQSTBZ5WAAA, seg_pos={0 0}, upload=444089]: Long tail cancellation due to optimal threshold Dec 26 17:34:39.558702083 (506h5m39.429785128s) [noise=true , address=77.90.31.14:4663 , piece_id=EB7GNCWQG3KGQ4GQ4I74J5EX2G54YWE62JJCI2HNPRV2BVZQKFPQ, seg_pos={0 0}, upload=444089]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: write tcp 188.166.225.170:38944->77.90.31.14:4663: use of closed network connection; write tcp 188.166.225.170:38944->77.90.31.14:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38944->77.90.31.14:4663: use of closed network connection Dec 26 17:34:39.558709006 (506h5m39.429792061s) [noise=true , address=77.90.31.14:4663 , piece_id=EB7GNCWQG3KGQ4GQ4I74J5EX2G54YWE62JJCI2HNPRV2BVZQKFPQ, seg_pos={0 0}, upload=444089]: Long tail cancellation due to optimal threshold Dec 26 17:34:39.558885092 (506h5m39.429968127s) [noise=true , address=45.91.116.12:4227 , piece_id=24ITM3BBJS2KUYEWHQXATVWA7AH72V5245IJNCNDBDX4R6XV5BJQ, seg_pos={0 0}, upload=444089]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: write tcp 188.166.225.170:34996->45.91.116.12:4227: use of closed network connection; write tcp 188.166.225.170:34996->45.91.116.12:4227: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34996->45.91.116.12:4227: use of closed network connection Dec 26 17:34:39.558891674 (506h5m39.429974709s) [noise=true , address=45.91.116.12:4227 , piece_id=24ITM3BBJS2KUYEWHQXATVWA7AH72V5245IJNCNDBDX4R6XV5BJQ, seg_pos={0 0}, upload=444089]: Long tail cancellation due to optimal threshold Dec 26 17:34:39.558909198 (506h5m39.429992243s) [seg_pos={0 0}, upload=444089]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 17:34:39.558975133 (506h5m39.430058168s) [upload=444089]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:34:39.558979602 (506h5m39.430062647s) [upload=444089]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:34:39.847668435 (506h5m39.71875147s) [upload=444089]: Done uploading segments... Dec 26 17:34:39.847712239 (506h5m39.718795274s) [upload=444089]: Upload finished. err: <nil> Duration: 6.054913261s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [X..X.X..XX..X.X......XX..X....X.X.XX...XX...XX....XX..XX....] duration: 6.274143935s trace: https://jaeger.dev.storj.io/trace/5835db663f93224f FAIL (2025-12-26 17:38:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: PZBVVBULP6BCJP6ANFEIKXXHLOSYX55VGV65ST5VV3DJUCIOWV4A) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: LRDA54KUC5RYNCYLFGC7VRXSIXLIVZEEBR5LMBC7L4WQEQW67GRA) context deadline exceeded ======LOGS====== h5m36.320342257s) [upload=444089]: Uploading segments... Dec 26 17:34:36.449470063 (506h5m36.320553108s) [upload=444089]: Got next segment. Inline: false Dec 26 17:34:36.449502284 (506h5m36.320585329s) [upload=444089]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 17:34:36.449507534 (506h5m36.320590569s) [upload=444089]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 17:34:36.704238476 (506h5m36.575321511s) [seg_pos={0 0}, upload=444089]: Begin upload segment... Dec 26 17:34:36.704303771 (506h5m36.575386806s) [seg_pos={0 0}, upload=444089]: Done begin upload segment. Dec 26 17:34:36.704328949 (506h5m36.575411984s) [noise=true , address=145.79.227.39:4015 , piece_id=WPFE4H77F7OFWA5M73NKI7F3ALXVLH63AA6HB3PTQJR4EVEJQ4NQ, seg_pos={0 0}, upload=444089]: Uploading piece... Dec 26 17:34:36.704344969 (506h5m36.575428014s) [noise=true , address=145.79.227.29:4943 , piece_id=ODGVNYKPLIZBUYUNJZUHCJQ2K2ZURD5UPDWZS3F34JQSTBZ5WAAA, seg_pos={0 0}, upload=444089]: Uploading piece... Dec 26 17:34:36.704785899 (506h5m36.575868944s) [noise=true , address=145.79.227.14:4839 , piece_id=XUTEEECYLYYN4X5N7ATZL32B6LSQCCIJWDJZHORYRHAZTHJ3UE7A, seg_pos={0 0}, upload=444089]: Uploading piece... Dec 26 17:34:36.705072314 (506h5m36.576155349s) [noise=true , address=145.79.227.11:4729 , piece_id=H62Q4ZQ2NEQGJ4XQYUORX3TITDWQRJNHSBQ3XNSEBHMVCPUWU5TA, seg_pos={0 0}, upload=444089]: Uploading piece... Dec 26 17:34:36.705347178 (506h5m36.576430213s) [noise=true , address=145.79.227.51:4410 , piece_id=6KF6ECGI7J2YPW3LVDWA2RLOERJXVJR7ADZ7WDWUNAXZBWEO7CIA, seg_pos={0 0}, upload=444089]: Uploading piece... Dec 26 17:34:36.705666385 (506h5m36.57674943s) [noise=true , address=145.79.227.10:4401 , piece_id=LREJ32SPU7ABE5AG347AALAIL7IVFZTYSCISFD2HROEUIASA7CNA, seg_pos={0 0}, upload=444089]: Uploading piece... Dec 26 17:34:36.705672767 (506h5m36.576755802s) [noise=true , address=145.79.227.47:4265 , piece_id=V77EKXQ5GVAGSKL4W6CBJWDQMJOZXK6U67UO4D63ZWSISD7PBUJA, seg_pos={0 0}, upload=444089]: Uploading piece... Dec 26 17:34:36.705947450 (506h5m36.577030485s) [noise=true , address=45.91.116.9:4261 , piece_id=PZBVVBULP6BCJP6ANFEIKXXHLOSYX55VGV65ST5VV3DJUCIOWV4A, seg_pos={0 0}, upload=444089]: Uploading piece... Dec 26 17:34:36.706017995 (506h5m36.57710103s) [noise=true , address=145.79.227.21:4475 , piece_id=G66X2QLH27XZPW5F73COJD5CE5ZYB65TFI2OBNWPQRMQSTNI7XKQ, seg_pos={0 0}, upload=444089]: Uploading piece... Dec 26 17:34:36.706262210 (506h5m36.577345245s) [noise=true , address=145.79.227.49:4337 , piece_id=XS2JRKD7JNCMWPBFA6GFRO4TNQMWUSHJQ73UIDTGFEBSFHDQEWWA, seg_pos={0 0}, upload=444089]: Uploading piece... Dec 26 17:34:36.706304049 (506h5m36.577387084s) [noise=true , address=45.91.116.12:4227 , piece_id=24ITM3BBJS2KUYEWHQXATVWA7AH72V5245IJNCNDBDX4R6XV5BJQ, seg_pos={0 0}, upload=444089]: Uploading piece... Dec 26 17:34:36.706789504 (506h5m36.577872539s) [noise=true , address=45.91.116.6:4975 , piece_id=LRDA54KUC5RYNCYLFGC7VRXSIXLIVZEEBR5LMBC7L4WQEQW67GRA, seg_pos={0 0}, upload=444089]: Uploading piece... Dec 26 17:34:36.707126466 (506h5m36.578209501s) [noise=true , address=145.79.227.38:4978 , piece_id=5YINIP7MUYYNOBNHV4ZGUWE366QEWGVG3TWORNJDOJEPU3AIBJEQ, seg_pos={0 0}, upload=444089]: Uploading piece... Dec 26 17:34:36.707614815 (506h5m36.578697881s) [noise=true , address=77.90.31.14:4663 , piece_id=EB7GNCWQG3KGQ4GQ4I74J5EX2G54YWE62JJCI2HNPRV2BVZQKFPQ, seg_pos={0 0}, upload=444089]: Uploading piece... Dec 26 17:34:36.707973388 (506h5m36.579056423s) [noise=true , address=45.91.116.5:4300 , piece_id=MY7IEOWS4SFF4OYCWTZXBFTYXB3LZRA7EQJS5THZCF5IL62ZVSHQ, seg_pos={0 0}, upload=444089]: Uploading piece... Dec 26 17:34:36.708277437 (506h5m36.579360472s) [noise=true , address=145.79.227.42:4122 , piece_id=HLGY23RXNXCND25LS6JW35YX4TDJHH77OMA2JUE5GSFL76KYQ77Q, seg_pos={0 0}, upload=444089]: Uploading piece... Dec 26 17:34:38.412838239 (506h5m38.283921284s) [upload=444089]: Next returned nil segment Dec 26 17:34:38.412890217 (506h5m38.283973262s) [upload=444089]: Waiting for error group managing segments... Dec 26 17:34:38.702510425 (506h5m38.57359346s) [noise=true , address=145.79.227.38:4978 , piece_id=5YINIP7MUYYNOBNHV4ZGUWE366QEWGVG3TWORNJDOJEPU3AIBJEQ, seg_pos={0 0}, upload=444089]: Done uploading piece. err: <nil> Dec 26 17:34:38.703987367 (506h5m38.575070412s) [noise=true , address=145.79.227.39:4015 , piece_id=WPFE4H77F7OFWA5M73NKI7F3ALXVLH63AA6HB3PTQJR4EVEJQ4NQ, seg_pos={0 0}, upload=444089]: Done uploading piece. err: <nil> Dec 26 17:34:38.708568922 (506h5m38.579651957s) [noise=true , address=145.79.227.11:4729 , piece_id=H62Q4ZQ2NEQGJ4XQYUORX3TITDWQRJNHSBQ3XNSEBHMVCPUWU5TA, seg_pos={0 0}, upload=444089]: Done uploading piece. err: <nil> Dec 26 17:34:38.714031684 (506h5m38.585114719s) [noise=true , address=145.79.227.49:4337 , piece_id=XS2JRKD7JNCMWPBFA6GFRO4TNQMWUSHJQ73UIDTGFEBSFHDQEWWA, seg_pos={0 0}, upload=444089]: Done uploading piece. err: <nil> Dec 26 17:34:38.714566472 (506h5m38.585649507s) [noise=true , address=145.79.227.21:4475 , piece_id=G66X2QLH27XZPW5F73COJD5CE5ZYB65TFI2OBNWPQRMQSTNI7XKQ, seg_pos={0 0}, upload=444089]: Done uploading piece. err: <nil> Dec 26 17:34:38.724676302 (506h5m38.595759347s) [noise=true , address=145.79.227.14:4839 , piece_id=XUTEEECYLYYN4X5N7ATZL32B6LSQCCIJWDJZHORYRHAZTHJ3UE7A, seg_pos={0 0}, upload=444089]: Done uploading piece. err: <nil> Dec 26 17:34:38.726402780 (506h5m38.597485825s) [noise=true , address=145.79.227.10:4401 , piece_id=LREJ32SPU7ABE5AG347AALAIL7IVFZTYSCISFD2HROEUIASA7CNA, seg_pos={0 0}, upload=444089]: Done uploading piece. err: <nil> Dec 26 17:34:38.736250622 (506h5m38.607333677s) [noise=true , address=145.79.227.42:4122 , piece_id=HLGY23RXNXCND25LS6JW35YX4TDJHH77OMA2JUE5GSFL76KYQ77Q, seg_pos={0 0}, upload=444089]: Done uploading piece. err: <nil> Dec 26 17:34:38.740648155 (506h5m38.611731201s) [noise=true , address=145.79.227.51:4410 , piece_id=6KF6ECGI7J2YPW3LVDWA2RLOERJXVJR7ADZ7WDWUNAXZBWEO7CIA, seg_pos={0 0}, upload=444089]: Done uploading piece. err: <nil> Dec 26 17:34:38.744072547 (506h5m38.615155582s) [noise=true , address=145.79.227.47:4265 , piece_id=V77EKXQ5GVAGSKL4W6CBJWDQMJOZXK6U67UO4D63ZWSISD7PBUJA, seg_pos={0 0}, upload=444089]: Done uploading piece. err: <nil> Dec 26 17:34:38.969785032 (506h5m38.840868067s) [noise=true , address=45.91.116.6:4975 , piece_id=LRDA54KUC5RYNCYLFGC7VRXSIXLIVZEEBR5LMBC7L4WQEQW67GRA, seg_pos={0 0}, upload=444089]: Done uploading piece. err: <nil> Dec 26 17:34:39.263027698 (506h5m39.134110733s) [noise=true , address=45.91.116.9:4261 , piece_id=PZBVVBULP6BCJP6ANFEIKXXHLOSYX55VGV65ST5VV3DJUCIOWV4A, seg_pos={0 0}, upload=444089]: Done uploading piece. err: <nil> Dec 26 17:34:39.558290509 (506h5m39.429373544s) [noise=true , address=45.91.116.5:4300 , piece_id=MY7IEOWS4SFF4OYCWTZXBFTYXB3LZRA7EQJS5THZCF5IL62ZVSHQ, seg_pos={0 0}, upload=444089]: Done uploading piece. err: <nil> Dec 26 17:34:39.558412061 (506h5m39.429495096s) [seg_pos={0 0}, upload=444089]: Segment reached optimal threshold of 13 pieces. Dec 26 17:34:39.558680011 (506h5m39.429763046s) [noise=true , address=145.79.227.29:4943 , piece_id=ODGVNYKPLIZBUYUNJZUHCJQ2K2ZURD5UPDWZS3F34JQSTBZ5WAAA, seg_pos={0 0}, upload=444089]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz): protocol: expected piece hash; context canceled; EOF Dec 26 17:34:39.558692886 (506h5m39.429775931s) [noise=true , address=145.79.227.29:4943 , piece_id=ODGVNYKPLIZBUYUNJZUHCJQ2K2ZURD5UPDWZS3F34JQSTBZ5WAAA, seg_pos={0 0}, upload=444089]: Long tail cancellation due to optimal threshold Dec 26 17:34:39.558702083 (506h5m39.429785128s) [noise=true , address=77.90.31.14:4663 , piece_id=EB7GNCWQG3KGQ4GQ4I74J5EX2G54YWE62JJCI2HNPRV2BVZQKFPQ, seg_pos={0 0}, upload=444089]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: write tcp 188.166.225.170:38944->77.90.31.14:4663: use of closed network connection; write tcp 188.166.225.170:38944->77.90.31.14:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38944->77.90.31.14:4663: use of closed network connection Dec 26 17:34:39.558709006 (506h5m39.429792061s) [noise=true , address=77.90.31.14:4663 , piece_id=EB7GNCWQG3KGQ4GQ4I74J5EX2G54YWE62JJCI2HNPRV2BVZQKFPQ, seg_pos={0 0}, upload=444089]: Long tail cancellation due to optimal threshold Dec 26 17:34:39.558885092 (506h5m39.429968127s) [noise=true , address=45.91.116.12:4227 , piece_id=24ITM3BBJS2KUYEWHQXATVWA7AH72V5245IJNCNDBDX4R6XV5BJQ, seg_pos={0 0}, upload=444089]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: write tcp 188.166.225.170:34996->45.91.116.12:4227: use of closed network connection; write tcp 188.166.225.170:34996->45.91.116.12:4227: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34996->45.91.116.12:4227: use of closed network connection Dec 26 17:34:39.558891674 (506h5m39.429974709s) [noise=true , address=45.91.116.12:4227 , piece_id=24ITM3BBJS2KUYEWHQXATVWA7AH72V5245IJNCNDBDX4R6XV5BJQ, seg_pos={0 0}, upload=444089]: Long tail cancellation due to optimal threshold Dec 26 17:34:39.558909198 (506h5m39.429992243s) [seg_pos={0 0}, upload=444089]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 17:34:39.558975133 (506h5m39.430058168s) [upload=444089]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:34:39.558979602 (506h5m39.430062647s) [upload=444089]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:34:39.847668435 (506h5m39.71875147s) [upload=444089]: Done uploading segments... Dec 26 17:34:39.847712239 (506h5m39.718795274s) [upload=444089]: Upload finished. err: <nil> Duration: 6.274189332s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [XX..X.X..XX..X.X......XX..X....X.X.XX...XX...XX....XX..XX...] duration: 6.761062503s trace: https://jaeger.dev.storj.io/trace/5835db663f93224f FAIL (2025-12-26 17:37:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: PZBVVBULP6BCJP6ANFEIKXXHLOSYX55VGV65ST5VV3DJUCIOWV4A) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: LRDA54KUC5RYNCYLFGC7VRXSIXLIVZEEBR5LMBC7L4WQEQW67GRA) context deadline exceeded ======LOGS====== h5m36.320342257s) [upload=444089]: Uploading segments... Dec 26 17:34:36.449470063 (506h5m36.320553108s) [upload=444089]: Got next segment. Inline: false Dec 26 17:34:36.449502284 (506h5m36.320585329s) [upload=444089]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 17:34:36.449507534 (506h5m36.320590569s) [upload=444089]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 17:34:36.704238476 (506h5m36.575321511s) [seg_pos={0 0}, upload=444089]: Begin upload segment... Dec 26 17:34:36.704303771 (506h5m36.575386806s) [seg_pos={0 0}, upload=444089]: Done begin upload segment. Dec 26 17:34:36.704328949 (506h5m36.575411984s) [noise=true , address=145.79.227.39:4015 , piece_id=WPFE4H77F7OFWA5M73NKI7F3ALXVLH63AA6HB3PTQJR4EVEJQ4NQ, seg_pos={0 0}, upload=444089]: Uploading piece... Dec 26 17:34:36.704344969 (506h5m36.575428014s) [noise=true , address=145.79.227.29:4943 , piece_id=ODGVNYKPLIZBUYUNJZUHCJQ2K2ZURD5UPDWZS3F34JQSTBZ5WAAA, seg_pos={0 0}, upload=444089]: Uploading piece... Dec 26 17:34:36.704785899 (506h5m36.575868944s) [noise=true , address=145.79.227.14:4839 , piece_id=XUTEEECYLYYN4X5N7ATZL32B6LSQCCIJWDJZHORYRHAZTHJ3UE7A, seg_pos={0 0}, upload=444089]: Uploading piece... Dec 26 17:34:36.705072314 (506h5m36.576155349s) [noise=true , address=145.79.227.11:4729 , piece_id=H62Q4ZQ2NEQGJ4XQYUORX3TITDWQRJNHSBQ3XNSEBHMVCPUWU5TA, seg_pos={0 0}, upload=444089]: Uploading piece... Dec 26 17:34:36.705347178 (506h5m36.576430213s) [noise=true , address=145.79.227.51:4410 , piece_id=6KF6ECGI7J2YPW3LVDWA2RLOERJXVJR7ADZ7WDWUNAXZBWEO7CIA, seg_pos={0 0}, upload=444089]: Uploading piece... Dec 26 17:34:36.705666385 (506h5m36.57674943s) [noise=true , address=145.79.227.10:4401 , piece_id=LREJ32SPU7ABE5AG347AALAIL7IVFZTYSCISFD2HROEUIASA7CNA, seg_pos={0 0}, upload=444089]: Uploading piece... Dec 26 17:34:36.705672767 (506h5m36.576755802s) [noise=true , address=145.79.227.47:4265 , piece_id=V77EKXQ5GVAGSKL4W6CBJWDQMJOZXK6U67UO4D63ZWSISD7PBUJA, seg_pos={0 0}, upload=444089]: Uploading piece... Dec 26 17:34:36.705947450 (506h5m36.577030485s) [noise=true , address=45.91.116.9:4261 , piece_id=PZBVVBULP6BCJP6ANFEIKXXHLOSYX55VGV65ST5VV3DJUCIOWV4A, seg_pos={0 0}, upload=444089]: Uploading piece... Dec 26 17:34:36.706017995 (506h5m36.57710103s) [noise=true , address=145.79.227.21:4475 , piece_id=G66X2QLH27XZPW5F73COJD5CE5ZYB65TFI2OBNWPQRMQSTNI7XKQ, seg_pos={0 0}, upload=444089]: Uploading piece... Dec 26 17:34:36.706262210 (506h5m36.577345245s) [noise=true , address=145.79.227.49:4337 , piece_id=XS2JRKD7JNCMWPBFA6GFRO4TNQMWUSHJQ73UIDTGFEBSFHDQEWWA, seg_pos={0 0}, upload=444089]: Uploading piece... Dec 26 17:34:36.706304049 (506h5m36.577387084s) [noise=true , address=45.91.116.12:4227 , piece_id=24ITM3BBJS2KUYEWHQXATVWA7AH72V5245IJNCNDBDX4R6XV5BJQ, seg_pos={0 0}, upload=444089]: Uploading piece... Dec 26 17:34:36.706789504 (506h5m36.577872539s) [noise=true , address=45.91.116.6:4975 , piece_id=LRDA54KUC5RYNCYLFGC7VRXSIXLIVZEEBR5LMBC7L4WQEQW67GRA, seg_pos={0 0}, upload=444089]: Uploading piece... Dec 26 17:34:36.707126466 (506h5m36.578209501s) [noise=true , address=145.79.227.38:4978 , piece_id=5YINIP7MUYYNOBNHV4ZGUWE366QEWGVG3TWORNJDOJEPU3AIBJEQ, seg_pos={0 0}, upload=444089]: Uploading piece... Dec 26 17:34:36.707614815 (506h5m36.578697881s) [noise=true , address=77.90.31.14:4663 , piece_id=EB7GNCWQG3KGQ4GQ4I74J5EX2G54YWE62JJCI2HNPRV2BVZQKFPQ, seg_pos={0 0}, upload=444089]: Uploading piece... Dec 26 17:34:36.707973388 (506h5m36.579056423s) [noise=true , address=45.91.116.5:4300 , piece_id=MY7IEOWS4SFF4OYCWTZXBFTYXB3LZRA7EQJS5THZCF5IL62ZVSHQ, seg_pos={0 0}, upload=444089]: Uploading piece... Dec 26 17:34:36.708277437 (506h5m36.579360472s) [noise=true , address=145.79.227.42:4122 , piece_id=HLGY23RXNXCND25LS6JW35YX4TDJHH77OMA2JUE5GSFL76KYQ77Q, seg_pos={0 0}, upload=444089]: Uploading piece... Dec 26 17:34:38.412838239 (506h5m38.283921284s) [upload=444089]: Next returned nil segment Dec 26 17:34:38.412890217 (506h5m38.283973262s) [upload=444089]: Waiting for error group managing segments... Dec 26 17:34:38.702510425 (506h5m38.57359346s) [noise=true , address=145.79.227.38:4978 , piece_id=5YINIP7MUYYNOBNHV4ZGUWE366QEWGVG3TWORNJDOJEPU3AIBJEQ, seg_pos={0 0}, upload=444089]: Done uploading piece. err: <nil> Dec 26 17:34:38.703987367 (506h5m38.575070412s) [noise=true , address=145.79.227.39:4015 , piece_id=WPFE4H77F7OFWA5M73NKI7F3ALXVLH63AA6HB3PTQJR4EVEJQ4NQ, seg_pos={0 0}, upload=444089]: Done uploading piece. err: <nil> Dec 26 17:34:38.708568922 (506h5m38.579651957s) [noise=true , address=145.79.227.11:4729 , piece_id=H62Q4ZQ2NEQGJ4XQYUORX3TITDWQRJNHSBQ3XNSEBHMVCPUWU5TA, seg_pos={0 0}, upload=444089]: Done uploading piece. err: <nil> Dec 26 17:34:38.714031684 (506h5m38.585114719s) [noise=true , address=145.79.227.49:4337 , piece_id=XS2JRKD7JNCMWPBFA6GFRO4TNQMWUSHJQ73UIDTGFEBSFHDQEWWA, seg_pos={0 0}, upload=444089]: Done uploading piece. err: <nil> Dec 26 17:34:38.714566472 (506h5m38.585649507s) [noise=true , address=145.79.227.21:4475 , piece_id=G66X2QLH27XZPW5F73COJD5CE5ZYB65TFI2OBNWPQRMQSTNI7XKQ, seg_pos={0 0}, upload=444089]: Done uploading piece. err: <nil> Dec 26 17:34:38.724676302 (506h5m38.595759347s) [noise=true , address=145.79.227.14:4839 , piece_id=XUTEEECYLYYN4X5N7ATZL32B6LSQCCIJWDJZHORYRHAZTHJ3UE7A, seg_pos={0 0}, upload=444089]: Done uploading piece. err: <nil> Dec 26 17:34:38.726402780 (506h5m38.597485825s) [noise=true , address=145.79.227.10:4401 , piece_id=LREJ32SPU7ABE5AG347AALAIL7IVFZTYSCISFD2HROEUIASA7CNA, seg_pos={0 0}, upload=444089]: Done uploading piece. err: <nil> Dec 26 17:34:38.736250622 (506h5m38.607333677s) [noise=true , address=145.79.227.42:4122 , piece_id=HLGY23RXNXCND25LS6JW35YX4TDJHH77OMA2JUE5GSFL76KYQ77Q, seg_pos={0 0}, upload=444089]: Done uploading piece. err: <nil> Dec 26 17:34:38.740648155 (506h5m38.611731201s) [noise=true , address=145.79.227.51:4410 , piece_id=6KF6ECGI7J2YPW3LVDWA2RLOERJXVJR7ADZ7WDWUNAXZBWEO7CIA, seg_pos={0 0}, upload=444089]: Done uploading piece. err: <nil> Dec 26 17:34:38.744072547 (506h5m38.615155582s) [noise=true , address=145.79.227.47:4265 , piece_id=V77EKXQ5GVAGSKL4W6CBJWDQMJOZXK6U67UO4D63ZWSISD7PBUJA, seg_pos={0 0}, upload=444089]: Done uploading piece. err: <nil> Dec 26 17:34:38.969785032 (506h5m38.840868067s) [noise=true , address=45.91.116.6:4975 , piece_id=LRDA54KUC5RYNCYLFGC7VRXSIXLIVZEEBR5LMBC7L4WQEQW67GRA, seg_pos={0 0}, upload=444089]: Done uploading piece. err: <nil> Dec 26 17:34:39.263027698 (506h5m39.134110733s) [noise=true , address=45.91.116.9:4261 , piece_id=PZBVVBULP6BCJP6ANFEIKXXHLOSYX55VGV65ST5VV3DJUCIOWV4A, seg_pos={0 0}, upload=444089]: Done uploading piece. err: <nil> Dec 26 17:34:39.558290509 (506h5m39.429373544s) [noise=true , address=45.91.116.5:4300 , piece_id=MY7IEOWS4SFF4OYCWTZXBFTYXB3LZRA7EQJS5THZCF5IL62ZVSHQ, seg_pos={0 0}, upload=444089]: Done uploading piece. err: <nil> Dec 26 17:34:39.558412061 (506h5m39.429495096s) [seg_pos={0 0}, upload=444089]: Segment reached optimal threshold of 13 pieces. Dec 26 17:34:39.558680011 (506h5m39.429763046s) [noise=true , address=145.79.227.29:4943 , piece_id=ODGVNYKPLIZBUYUNJZUHCJQ2K2ZURD5UPDWZS3F34JQSTBZ5WAAA, seg_pos={0 0}, upload=444089]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz): protocol: expected piece hash; context canceled; EOF Dec 26 17:34:39.558692886 (506h5m39.429775931s) [noise=true , address=145.79.227.29:4943 , piece_id=ODGVNYKPLIZBUYUNJZUHCJQ2K2ZURD5UPDWZS3F34JQSTBZ5WAAA, seg_pos={0 0}, upload=444089]: Long tail cancellation due to optimal threshold Dec 26 17:34:39.558702083 (506h5m39.429785128s) [noise=true , address=77.90.31.14:4663 , piece_id=EB7GNCWQG3KGQ4GQ4I74J5EX2G54YWE62JJCI2HNPRV2BVZQKFPQ, seg_pos={0 0}, upload=444089]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: write tcp 188.166.225.170:38944->77.90.31.14:4663: use of closed network connection; write tcp 188.166.225.170:38944->77.90.31.14:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38944->77.90.31.14:4663: use of closed network connection Dec 26 17:34:39.558709006 (506h5m39.429792061s) [noise=true , address=77.90.31.14:4663 , piece_id=EB7GNCWQG3KGQ4GQ4I74J5EX2G54YWE62JJCI2HNPRV2BVZQKFPQ, seg_pos={0 0}, upload=444089]: Long tail cancellation due to optimal threshold Dec 26 17:34:39.558885092 (506h5m39.429968127s) [noise=true , address=45.91.116.12:4227 , piece_id=24ITM3BBJS2KUYEWHQXATVWA7AH72V5245IJNCNDBDX4R6XV5BJQ, seg_pos={0 0}, upload=444089]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: write tcp 188.166.225.170:34996->45.91.116.12:4227: use of closed network connection; write tcp 188.166.225.170:34996->45.91.116.12:4227: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34996->45.91.116.12:4227: use of closed network connection Dec 26 17:34:39.558891674 (506h5m39.429974709s) [noise=true , address=45.91.116.12:4227 , piece_id=24ITM3BBJS2KUYEWHQXATVWA7AH72V5245IJNCNDBDX4R6XV5BJQ, seg_pos={0 0}, upload=444089]: Long tail cancellation due to optimal threshold Dec 26 17:34:39.558909198 (506h5m39.429992243s) [seg_pos={0 0}, upload=444089]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 17:34:39.558975133 (506h5m39.430058168s) [upload=444089]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:34:39.558979602 (506h5m39.430062647s) [upload=444089]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:34:39.847668435 (506h5m39.71875147s) [upload=444089]: Done uploading segments... Dec 26 17:34:39.847712239 (506h5m39.718795274s) [upload=444089]: Upload finished. err: <nil> Duration: 6.761095956s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [.XX..X.X..XX..X.X......XX..X....X.X.XX...XX...XX....XX..XX..] duration: 6.637208082s trace: https://jaeger.dev.storj.io/trace/5835db663f93224f FAIL (2025-12-26 17:36:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: PZBVVBULP6BCJP6ANFEIKXXHLOSYX55VGV65ST5VV3DJUCIOWV4A) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: LRDA54KUC5RYNCYLFGC7VRXSIXLIVZEEBR5LMBC7L4WQEQW67GRA) context deadline exceeded ======LOGS====== h5m36.320342257s) [upload=444089]: Uploading segments... Dec 26 17:34:36.449470063 (506h5m36.320553108s) [upload=444089]: Got next segment. Inline: false Dec 26 17:34:36.449502284 (506h5m36.320585329s) [upload=444089]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 17:34:36.449507534 (506h5m36.320590569s) [upload=444089]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 17:34:36.704238476 (506h5m36.575321511s) [seg_pos={0 0}, upload=444089]: Begin upload segment... Dec 26 17:34:36.704303771 (506h5m36.575386806s) [seg_pos={0 0}, upload=444089]: Done begin upload segment. Dec 26 17:34:36.704328949 (506h5m36.575411984s) [noise=true , address=145.79.227.39:4015 , piece_id=WPFE4H77F7OFWA5M73NKI7F3ALXVLH63AA6HB3PTQJR4EVEJQ4NQ, seg_pos={0 0}, upload=444089]: Uploading piece... Dec 26 17:34:36.704344969 (506h5m36.575428014s) [noise=true , address=145.79.227.29:4943 , piece_id=ODGVNYKPLIZBUYUNJZUHCJQ2K2ZURD5UPDWZS3F34JQSTBZ5WAAA, seg_pos={0 0}, upload=444089]: Uploading piece... Dec 26 17:34:36.704785899 (506h5m36.575868944s) [noise=true , address=145.79.227.14:4839 , piece_id=XUTEEECYLYYN4X5N7ATZL32B6LSQCCIJWDJZHORYRHAZTHJ3UE7A, seg_pos={0 0}, upload=444089]: Uploading piece... Dec 26 17:34:36.705072314 (506h5m36.576155349s) [noise=true , address=145.79.227.11:4729 , piece_id=H62Q4ZQ2NEQGJ4XQYUORX3TITDWQRJNHSBQ3XNSEBHMVCPUWU5TA, seg_pos={0 0}, upload=444089]: Uploading piece... Dec 26 17:34:36.705347178 (506h5m36.576430213s) [noise=true , address=145.79.227.51:4410 , piece_id=6KF6ECGI7J2YPW3LVDWA2RLOERJXVJR7ADZ7WDWUNAXZBWEO7CIA, seg_pos={0 0}, upload=444089]: Uploading piece... Dec 26 17:34:36.705666385 (506h5m36.57674943s) [noise=true , address=145.79.227.10:4401 , piece_id=LREJ32SPU7ABE5AG347AALAIL7IVFZTYSCISFD2HROEUIASA7CNA, seg_pos={0 0}, upload=444089]: Uploading piece... Dec 26 17:34:36.705672767 (506h5m36.576755802s) [noise=true , address=145.79.227.47:4265 , piece_id=V77EKXQ5GVAGSKL4W6CBJWDQMJOZXK6U67UO4D63ZWSISD7PBUJA, seg_pos={0 0}, upload=444089]: Uploading piece... Dec 26 17:34:36.705947450 (506h5m36.577030485s) [noise=true , address=45.91.116.9:4261 , piece_id=PZBVVBULP6BCJP6ANFEIKXXHLOSYX55VGV65ST5VV3DJUCIOWV4A, seg_pos={0 0}, upload=444089]: Uploading piece... Dec 26 17:34:36.706017995 (506h5m36.57710103s) [noise=true , address=145.79.227.21:4475 , piece_id=G66X2QLH27XZPW5F73COJD5CE5ZYB65TFI2OBNWPQRMQSTNI7XKQ, seg_pos={0 0}, upload=444089]: Uploading piece... Dec 26 17:34:36.706262210 (506h5m36.577345245s) [noise=true , address=145.79.227.49:4337 , piece_id=XS2JRKD7JNCMWPBFA6GFRO4TNQMWUSHJQ73UIDTGFEBSFHDQEWWA, seg_pos={0 0}, upload=444089]: Uploading piece... Dec 26 17:34:36.706304049 (506h5m36.577387084s) [noise=true , address=45.91.116.12:4227 , piece_id=24ITM3BBJS2KUYEWHQXATVWA7AH72V5245IJNCNDBDX4R6XV5BJQ, seg_pos={0 0}, upload=444089]: Uploading piece... Dec 26 17:34:36.706789504 (506h5m36.577872539s) [noise=true , address=45.91.116.6:4975 , piece_id=LRDA54KUC5RYNCYLFGC7VRXSIXLIVZEEBR5LMBC7L4WQEQW67GRA, seg_pos={0 0}, upload=444089]: Uploading piece... Dec 26 17:34:36.707126466 (506h5m36.578209501s) [noise=true , address=145.79.227.38:4978 , piece_id=5YINIP7MUYYNOBNHV4ZGUWE366QEWGVG3TWORNJDOJEPU3AIBJEQ, seg_pos={0 0}, upload=444089]: Uploading piece... Dec 26 17:34:36.707614815 (506h5m36.578697881s) [noise=true , address=77.90.31.14:4663 , piece_id=EB7GNCWQG3KGQ4GQ4I74J5EX2G54YWE62JJCI2HNPRV2BVZQKFPQ, seg_pos={0 0}, upload=444089]: Uploading piece... Dec 26 17:34:36.707973388 (506h5m36.579056423s) [noise=true , address=45.91.116.5:4300 , piece_id=MY7IEOWS4SFF4OYCWTZXBFTYXB3LZRA7EQJS5THZCF5IL62ZVSHQ, seg_pos={0 0}, upload=444089]: Uploading piece... Dec 26 17:34:36.708277437 (506h5m36.579360472s) [noise=true , address=145.79.227.42:4122 , piece_id=HLGY23RXNXCND25LS6JW35YX4TDJHH77OMA2JUE5GSFL76KYQ77Q, seg_pos={0 0}, upload=444089]: Uploading piece... Dec 26 17:34:38.412838239 (506h5m38.283921284s) [upload=444089]: Next returned nil segment Dec 26 17:34:38.412890217 (506h5m38.283973262s) [upload=444089]: Waiting for error group managing segments... Dec 26 17:34:38.702510425 (506h5m38.57359346s) [noise=true , address=145.79.227.38:4978 , piece_id=5YINIP7MUYYNOBNHV4ZGUWE366QEWGVG3TWORNJDOJEPU3AIBJEQ, seg_pos={0 0}, upload=444089]: Done uploading piece. err: <nil> Dec 26 17:34:38.703987367 (506h5m38.575070412s) [noise=true , address=145.79.227.39:4015 , piece_id=WPFE4H77F7OFWA5M73NKI7F3ALXVLH63AA6HB3PTQJR4EVEJQ4NQ, seg_pos={0 0}, upload=444089]: Done uploading piece. err: <nil> Dec 26 17:34:38.708568922 (506h5m38.579651957s) [noise=true , address=145.79.227.11:4729 , piece_id=H62Q4ZQ2NEQGJ4XQYUORX3TITDWQRJNHSBQ3XNSEBHMVCPUWU5TA, seg_pos={0 0}, upload=444089]: Done uploading piece. err: <nil> Dec 26 17:34:38.714031684 (506h5m38.585114719s) [noise=true , address=145.79.227.49:4337 , piece_id=XS2JRKD7JNCMWPBFA6GFRO4TNQMWUSHJQ73UIDTGFEBSFHDQEWWA, seg_pos={0 0}, upload=444089]: Done uploading piece. err: <nil> Dec 26 17:34:38.714566472 (506h5m38.585649507s) [noise=true , address=145.79.227.21:4475 , piece_id=G66X2QLH27XZPW5F73COJD5CE5ZYB65TFI2OBNWPQRMQSTNI7XKQ, seg_pos={0 0}, upload=444089]: Done uploading piece. err: <nil> Dec 26 17:34:38.724676302 (506h5m38.595759347s) [noise=true , address=145.79.227.14:4839 , piece_id=XUTEEECYLYYN4X5N7ATZL32B6LSQCCIJWDJZHORYRHAZTHJ3UE7A, seg_pos={0 0}, upload=444089]: Done uploading piece. err: <nil> Dec 26 17:34:38.726402780 (506h5m38.597485825s) [noise=true , address=145.79.227.10:4401 , piece_id=LREJ32SPU7ABE5AG347AALAIL7IVFZTYSCISFD2HROEUIASA7CNA, seg_pos={0 0}, upload=444089]: Done uploading piece. err: <nil> Dec 26 17:34:38.736250622 (506h5m38.607333677s) [noise=true , address=145.79.227.42:4122 , piece_id=HLGY23RXNXCND25LS6JW35YX4TDJHH77OMA2JUE5GSFL76KYQ77Q, seg_pos={0 0}, upload=444089]: Done uploading piece. err: <nil> Dec 26 17:34:38.740648155 (506h5m38.611731201s) [noise=true , address=145.79.227.51:4410 , piece_id=6KF6ECGI7J2YPW3LVDWA2RLOERJXVJR7ADZ7WDWUNAXZBWEO7CIA, seg_pos={0 0}, upload=444089]: Done uploading piece. err: <nil> Dec 26 17:34:38.744072547 (506h5m38.615155582s) [noise=true , address=145.79.227.47:4265 , piece_id=V77EKXQ5GVAGSKL4W6CBJWDQMJOZXK6U67UO4D63ZWSISD7PBUJA, seg_pos={0 0}, upload=444089]: Done uploading piece. err: <nil> Dec 26 17:34:38.969785032 (506h5m38.840868067s) [noise=true , address=45.91.116.6:4975 , piece_id=LRDA54KUC5RYNCYLFGC7VRXSIXLIVZEEBR5LMBC7L4WQEQW67GRA, seg_pos={0 0}, upload=444089]: Done uploading piece. err: <nil> Dec 26 17:34:39.263027698 (506h5m39.134110733s) [noise=true , address=45.91.116.9:4261 , piece_id=PZBVVBULP6BCJP6ANFEIKXXHLOSYX55VGV65ST5VV3DJUCIOWV4A, seg_pos={0 0}, upload=444089]: Done uploading piece. err: <nil> Dec 26 17:34:39.558290509 (506h5m39.429373544s) [noise=true , address=45.91.116.5:4300 , piece_id=MY7IEOWS4SFF4OYCWTZXBFTYXB3LZRA7EQJS5THZCF5IL62ZVSHQ, seg_pos={0 0}, upload=444089]: Done uploading piece. err: <nil> Dec 26 17:34:39.558412061 (506h5m39.429495096s) [seg_pos={0 0}, upload=444089]: Segment reached optimal threshold of 13 pieces. Dec 26 17:34:39.558680011 (506h5m39.429763046s) [noise=true , address=145.79.227.29:4943 , piece_id=ODGVNYKPLIZBUYUNJZUHCJQ2K2ZURD5UPDWZS3F34JQSTBZ5WAAA, seg_pos={0 0}, upload=444089]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz): protocol: expected piece hash; context canceled; EOF Dec 26 17:34:39.558692886 (506h5m39.429775931s) [noise=true , address=145.79.227.29:4943 , piece_id=ODGVNYKPLIZBUYUNJZUHCJQ2K2ZURD5UPDWZS3F34JQSTBZ5WAAA, seg_pos={0 0}, upload=444089]: Long tail cancellation due to optimal threshold Dec 26 17:34:39.558702083 (506h5m39.429785128s) [noise=true , address=77.90.31.14:4663 , piece_id=EB7GNCWQG3KGQ4GQ4I74J5EX2G54YWE62JJCI2HNPRV2BVZQKFPQ, seg_pos={0 0}, upload=444089]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: write tcp 188.166.225.170:38944->77.90.31.14:4663: use of closed network connection; write tcp 188.166.225.170:38944->77.90.31.14:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38944->77.90.31.14:4663: use of closed network connection Dec 26 17:34:39.558709006 (506h5m39.429792061s) [noise=true , address=77.90.31.14:4663 , piece_id=EB7GNCWQG3KGQ4GQ4I74J5EX2G54YWE62JJCI2HNPRV2BVZQKFPQ, seg_pos={0 0}, upload=444089]: Long tail cancellation due to optimal threshold Dec 26 17:34:39.558885092 (506h5m39.429968127s) [noise=true , address=45.91.116.12:4227 , piece_id=24ITM3BBJS2KUYEWHQXATVWA7AH72V5245IJNCNDBDX4R6XV5BJQ, seg_pos={0 0}, upload=444089]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: write tcp 188.166.225.170:34996->45.91.116.12:4227: use of closed network connection; write tcp 188.166.225.170:34996->45.91.116.12:4227: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34996->45.91.116.12:4227: use of closed network connection Dec 26 17:34:39.558891674 (506h5m39.429974709s) [noise=true , address=45.91.116.12:4227 , piece_id=24ITM3BBJS2KUYEWHQXATVWA7AH72V5245IJNCNDBDX4R6XV5BJQ, seg_pos={0 0}, upload=444089]: Long tail cancellation due to optimal threshold Dec 26 17:34:39.558909198 (506h5m39.429992243s) [seg_pos={0 0}, upload=444089]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 17:34:39.558975133 (506h5m39.430058168s) [upload=444089]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:34:39.558979602 (506h5m39.430062647s) [upload=444089]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:34:39.847668435 (506h5m39.71875147s) [upload=444089]: Done uploading segments... Dec 26 17:34:39.847712239 (506h5m39.718795274s) [upload=444089]: Upload finished. err: <nil> Duration: 6.637247058s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 43.443724464s trace: https://jaeger.dev.storj.io/trace/109f3e067e010c2f FAIL (2025-12-26 17:36:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 127c4HGgJu3z16vxq7TYZydCUa86FhCc3TmoPZoheVeqTbnJfne, Piece ID: DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q) context deadline exceeded; piecestore: (Node ID: 1ahTbggGRi6cJa4JPBfSEgLJYymrgVuLxSc15BV3qj5gPWPFiZ, Piece ID: WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA) context deadline exceeded; piecestore: (Node ID: 1m4M9WpEgS8BgLgQ2kwqzb6szQ2k4KMS3pv6DRQsEPS2ujPyLM, Piece ID: TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ) context deadline exceeded ======LOGS====== 26 16:46:05.282897453 (505h17m5.153980498s) [noise=true , address=77.90.31.18:4828 , piece_id=EBH3I5TADBPDMDRYQSG7NQVEFRJVNACURFHR3UTKQL2ZPE7KAFVA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.282954052 (505h17m5.154037097s) [noise=true , address=77.90.31.21:4885 , piece_id=6YIKCRJQZ4Z2EKOB7Y7QZFVS2WIHV7CJUJVOKV5DS64CNLADLMVQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283314878 (505h17m5.154397923s) [noise=true , address=145.79.227.44:4199 , piece_id=7ZE6F5CL4VK6NKCPLM6H7RLIHLEO2RWEG5EGJFXF7X7Y4CSFX6VQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283493097 (505h17m5.154576143s) [noise=true , address=77.90.31.11:4578 , piece_id=TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283720251 (505h17m5.154803296s) [noise=true , address=145.79.227.12:4797 , piece_id=OR6AQ5RKIULPNCDS4TXRRXNXKXZDM4TT2J367SRQJEYZAYYML6LA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283969956 (505h17m5.155053001s) [noise=true , address=77.90.31.16:4758 , piece_id=WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284146873 (505h17m5.155229918s) [noise=true , address=145.79.227.40:4071 , piece_id=IBCGYA4LAJIRKFVJV2E6T2NPRCVADJIP5SAJ75H4QXDIMOQAN2NA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284470589 (505h17m5.155553634s) [noise=true , address=77.90.31.15:4710 , piece_id=25MAPHYJKQV4XQ7X67IVTNXM2ZKPBALDEKK66ZK67TFX3M74SGRQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284553537 (505h17m5.155636582s) [noise=true , address=77.90.31.19:4860 , piece_id=P2FISE3LU4Q5ZVOOGYHZAO52MIAE5DYWLFY6CVXSZIUC5ONL7HYQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284974238 (505h17m5.156057283s) [noise=true , address=77.90.31.10:4530 , piece_id=DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.285009706 (505h17m5.156092751s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.285544504 (505h17m5.156627549s) [noise=true , address=77.90.31.14:4685 , piece_id=4XZFFB7E642K5A4IKKN6Y6NJ42POJWUCIK7534PNM5A332YHTABA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.286800255 (505h17m5.15788337s) [noise=true , address=77.90.31.13:4644 , piece_id=6QLHFIXW7DIL3XRHVODFU6LPMRW25WSM77LRPYGVGUEN36NUALMA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.287403053 (505h17m5.158486098s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.287928925 (505h17m5.15901197s) [noise=true , address=145.79.227.11:4737 , piece_id=FAQBVVD3I2B5ZOPSR2VCZGSRQNVIRSZRVQCX5E33QECVEAOKLHRQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.289150881 (505h17m5.160233926s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:06.768498316 (505h17m6.639581351s) [upload=443378]: Next returned nil segment Dec 26 16:46:06.768512754 (505h17m6.63959579s) [upload=443378]: Waiting for error group managing segments... Dec 26 16:46:07.177478726 (505h17m7.048561771s) [noise=true , address=145.79.227.11:4737 , piece_id=FAQBVVD3I2B5ZOPSR2VCZGSRQNVIRSZRVQCX5E33QECVEAOKLHRQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.187360212 (505h17m7.058443247s) [noise=true , address=145.79.227.12:4797 , piece_id=OR6AQ5RKIULPNCDS4TXRRXNXKXZDM4TT2J367SRQJEYZAYYML6LA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.191564518 (505h17m7.062647563s) [noise=true , address=64.49.13.26:4950 , piece_id=NFO7SRXQGAUISAEPYC6OCLPCUTKFWR5PSOWUJZ5YHSLCYNZQQ7SQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.431132678 (505h17m7.302215713s) [noise=true , address=77.90.31.10:4530 , piece_id=DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.443236505 (505h17m7.31431955s) [noise=true , address=77.90.31.15:4710 , piece_id=25MAPHYJKQV4XQ7X67IVTNXM2ZKPBALDEKK66ZK67TFX3M74SGRQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.515779830 (505h17m7.386862865s) [noise=true , address=77.90.31.21:4885 , piece_id=6YIKCRJQZ4Z2EKOB7Y7QZFVS2WIHV7CJUJVOKV5DS64CNLADLMVQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.030094386 (505h17m7.901177421s) [noise=true , address=145.79.227.40:4071 , piece_id=IBCGYA4LAJIRKFVJV2E6T2NPRCVADJIP5SAJ75H4QXDIMOQAN2NA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.108333867 (505h17m7.979416913s) [noise=true , address=77.90.31.16:4758 , piece_id=WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.146456269 (505h17m8.017539314s) [noise=true , address=145.79.227.44:4199 , piece_id=7ZE6F5CL4VK6NKCPLM6H7RLIHLEO2RWEG5EGJFXF7X7Y4CSFX6VQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.167739325 (505h17m8.03882236s) [noise=true , address=77.90.31.18:4828 , piece_id=EBH3I5TADBPDMDRYQSG7NQVEFRJVNACURFHR3UTKQL2ZPE7KAFVA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.292396866 (505h17m8.163479911s) [noise=true , address=77.90.31.13:4644 , piece_id=6QLHFIXW7DIL3XRHVODFU6LPMRW25WSM77LRPYGVGUEN36NUALMA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.475687322 (505h17m8.346770367s) [noise=true , address=77.90.31.14:4685 , piece_id=4XZFFB7E642K5A4IKKN6Y6NJ42POJWUCIK7534PNM5A332YHTABA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.551777060 (505h17m8.422860105s) [noise=true , address=77.90.31.19:4860 , piece_id=P2FISE3LU4Q5ZVOOGYHZAO52MIAE5DYWLFY6CVXSZIUC5ONL7HYQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.678298409 (505h17m8.549381454s) [noise=true , address=77.90.31.11:4578 , piece_id=TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.678323457 (505h17m8.549406502s) [seg_pos={0 0}, upload=443378]: Segment reached optimal threshold of 14 pieces. Dec 26 16:46:08.678549747 (505h17m8.549632793s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1T9uQjynbTYcaixhvHWCKzatRWpFxMonHFi5idiMQtvivupKSu): protocol: write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection; write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection Dec 26 16:46:08.678558464 (505h17m8.549641509s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.678891048 (505h17m8.549974093s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ptydBd7T89noaveRPbmziYKKX9C1eNKaYG8BEfz2Zk4T9XtKz): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.678907879 (505h17m8.549990924s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127aXyQbxPkETL9tjbKQodpMMteTiZjBQ5C4muNnXELbuy6H5TM): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.678919432 (505h17m8.550002467s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.678931354 (505h17m8.550014399s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.679190638 (505h17m8.550273714s) [noise=true , address=64.49.13.140:4508 , piece_id=WKGTL6PEL62NGB4T3SZ4VH4YDG3523UJDLUCFRQRWJXVXDWXOMXA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.679200026 (505h17m8.550283061s) [noise=true , address=64.49.13.140:4508 , piece_id=WKGTL6PEL62NGB4T3SZ4VH4YDG3523UJDLUCFRQRWJXVXDWXOMXA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.679263507 (505h17m8.550346552s) [seg_pos={0 0}, upload=443378]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 16:46:08.679352386 (505h17m8.550435431s) [upload=443378]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:46:08.679359249 (505h17m8.550442294s) [upload=443378]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:46:08.985467612 (505h17m8.856550657s) [upload=443378]: Done uploading segments... Dec 26 16:46:08.985487951 (505h17m8.856570997s) [upload=443378]: Upload finished. err: <nil> Duration: 43.443752467s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [..XX..X.X..XX..X.X......XX..X....X.X.XX...XX...XX....XX..XX.] duration: 24.968542733s trace: https://jaeger.dev.storj.io/trace/5835db663f93224f FAIL (2025-12-26 17:36:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: PZBVVBULP6BCJP6ANFEIKXXHLOSYX55VGV65ST5VV3DJUCIOWV4A) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: LRDA54KUC5RYNCYLFGC7VRXSIXLIVZEEBR5LMBC7L4WQEQW67GRA) context deadline exceeded ======LOGS====== h5m36.320342257s) [upload=444089]: Uploading segments... Dec 26 17:34:36.449470063 (506h5m36.320553108s) [upload=444089]: Got next segment. Inline: false Dec 26 17:34:36.449502284 (506h5m36.320585329s) [upload=444089]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 17:34:36.449507534 (506h5m36.320590569s) [upload=444089]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 17:34:36.704238476 (506h5m36.575321511s) [seg_pos={0 0}, upload=444089]: Begin upload segment... Dec 26 17:34:36.704303771 (506h5m36.575386806s) [seg_pos={0 0}, upload=444089]: Done begin upload segment. Dec 26 17:34:36.704328949 (506h5m36.575411984s) [noise=true , address=145.79.227.39:4015 , piece_id=WPFE4H77F7OFWA5M73NKI7F3ALXVLH63AA6HB3PTQJR4EVEJQ4NQ, seg_pos={0 0}, upload=444089]: Uploading piece... Dec 26 17:34:36.704344969 (506h5m36.575428014s) [noise=true , address=145.79.227.29:4943 , piece_id=ODGVNYKPLIZBUYUNJZUHCJQ2K2ZURD5UPDWZS3F34JQSTBZ5WAAA, seg_pos={0 0}, upload=444089]: Uploading piece... Dec 26 17:34:36.704785899 (506h5m36.575868944s) [noise=true , address=145.79.227.14:4839 , piece_id=XUTEEECYLYYN4X5N7ATZL32B6LSQCCIJWDJZHORYRHAZTHJ3UE7A, seg_pos={0 0}, upload=444089]: Uploading piece... Dec 26 17:34:36.705072314 (506h5m36.576155349s) [noise=true , address=145.79.227.11:4729 , piece_id=H62Q4ZQ2NEQGJ4XQYUORX3TITDWQRJNHSBQ3XNSEBHMVCPUWU5TA, seg_pos={0 0}, upload=444089]: Uploading piece... Dec 26 17:34:36.705347178 (506h5m36.576430213s) [noise=true , address=145.79.227.51:4410 , piece_id=6KF6ECGI7J2YPW3LVDWA2RLOERJXVJR7ADZ7WDWUNAXZBWEO7CIA, seg_pos={0 0}, upload=444089]: Uploading piece... Dec 26 17:34:36.705666385 (506h5m36.57674943s) [noise=true , address=145.79.227.10:4401 , piece_id=LREJ32SPU7ABE5AG347AALAIL7IVFZTYSCISFD2HROEUIASA7CNA, seg_pos={0 0}, upload=444089]: Uploading piece... Dec 26 17:34:36.705672767 (506h5m36.576755802s) [noise=true , address=145.79.227.47:4265 , piece_id=V77EKXQ5GVAGSKL4W6CBJWDQMJOZXK6U67UO4D63ZWSISD7PBUJA, seg_pos={0 0}, upload=444089]: Uploading piece... Dec 26 17:34:36.705947450 (506h5m36.577030485s) [noise=true , address=45.91.116.9:4261 , piece_id=PZBVVBULP6BCJP6ANFEIKXXHLOSYX55VGV65ST5VV3DJUCIOWV4A, seg_pos={0 0}, upload=444089]: Uploading piece... Dec 26 17:34:36.706017995 (506h5m36.57710103s) [noise=true , address=145.79.227.21:4475 , piece_id=G66X2QLH27XZPW5F73COJD5CE5ZYB65TFI2OBNWPQRMQSTNI7XKQ, seg_pos={0 0}, upload=444089]: Uploading piece... Dec 26 17:34:36.706262210 (506h5m36.577345245s) [noise=true , address=145.79.227.49:4337 , piece_id=XS2JRKD7JNCMWPBFA6GFRO4TNQMWUSHJQ73UIDTGFEBSFHDQEWWA, seg_pos={0 0}, upload=444089]: Uploading piece... Dec 26 17:34:36.706304049 (506h5m36.577387084s) [noise=true , address=45.91.116.12:4227 , piece_id=24ITM3BBJS2KUYEWHQXATVWA7AH72V5245IJNCNDBDX4R6XV5BJQ, seg_pos={0 0}, upload=444089]: Uploading piece... Dec 26 17:34:36.706789504 (506h5m36.577872539s) [noise=true , address=45.91.116.6:4975 , piece_id=LRDA54KUC5RYNCYLFGC7VRXSIXLIVZEEBR5LMBC7L4WQEQW67GRA, seg_pos={0 0}, upload=444089]: Uploading piece... Dec 26 17:34:36.707126466 (506h5m36.578209501s) [noise=true , address=145.79.227.38:4978 , piece_id=5YINIP7MUYYNOBNHV4ZGUWE366QEWGVG3TWORNJDOJEPU3AIBJEQ, seg_pos={0 0}, upload=444089]: Uploading piece... Dec 26 17:34:36.707614815 (506h5m36.578697881s) [noise=true , address=77.90.31.14:4663 , piece_id=EB7GNCWQG3KGQ4GQ4I74J5EX2G54YWE62JJCI2HNPRV2BVZQKFPQ, seg_pos={0 0}, upload=444089]: Uploading piece... Dec 26 17:34:36.707973388 (506h5m36.579056423s) [noise=true , address=45.91.116.5:4300 , piece_id=MY7IEOWS4SFF4OYCWTZXBFTYXB3LZRA7EQJS5THZCF5IL62ZVSHQ, seg_pos={0 0}, upload=444089]: Uploading piece... Dec 26 17:34:36.708277437 (506h5m36.579360472s) [noise=true , address=145.79.227.42:4122 , piece_id=HLGY23RXNXCND25LS6JW35YX4TDJHH77OMA2JUE5GSFL76KYQ77Q, seg_pos={0 0}, upload=444089]: Uploading piece... Dec 26 17:34:38.412838239 (506h5m38.283921284s) [upload=444089]: Next returned nil segment Dec 26 17:34:38.412890217 (506h5m38.283973262s) [upload=444089]: Waiting for error group managing segments... Dec 26 17:34:38.702510425 (506h5m38.57359346s) [noise=true , address=145.79.227.38:4978 , piece_id=5YINIP7MUYYNOBNHV4ZGUWE366QEWGVG3TWORNJDOJEPU3AIBJEQ, seg_pos={0 0}, upload=444089]: Done uploading piece. err: <nil> Dec 26 17:34:38.703987367 (506h5m38.575070412s) [noise=true , address=145.79.227.39:4015 , piece_id=WPFE4H77F7OFWA5M73NKI7F3ALXVLH63AA6HB3PTQJR4EVEJQ4NQ, seg_pos={0 0}, upload=444089]: Done uploading piece. err: <nil> Dec 26 17:34:38.708568922 (506h5m38.579651957s) [noise=true , address=145.79.227.11:4729 , piece_id=H62Q4ZQ2NEQGJ4XQYUORX3TITDWQRJNHSBQ3XNSEBHMVCPUWU5TA, seg_pos={0 0}, upload=444089]: Done uploading piece. err: <nil> Dec 26 17:34:38.714031684 (506h5m38.585114719s) [noise=true , address=145.79.227.49:4337 , piece_id=XS2JRKD7JNCMWPBFA6GFRO4TNQMWUSHJQ73UIDTGFEBSFHDQEWWA, seg_pos={0 0}, upload=444089]: Done uploading piece. err: <nil> Dec 26 17:34:38.714566472 (506h5m38.585649507s) [noise=true , address=145.79.227.21:4475 , piece_id=G66X2QLH27XZPW5F73COJD5CE5ZYB65TFI2OBNWPQRMQSTNI7XKQ, seg_pos={0 0}, upload=444089]: Done uploading piece. err: <nil> Dec 26 17:34:38.724676302 (506h5m38.595759347s) [noise=true , address=145.79.227.14:4839 , piece_id=XUTEEECYLYYN4X5N7ATZL32B6LSQCCIJWDJZHORYRHAZTHJ3UE7A, seg_pos={0 0}, upload=444089]: Done uploading piece. err: <nil> Dec 26 17:34:38.726402780 (506h5m38.597485825s) [noise=true , address=145.79.227.10:4401 , piece_id=LREJ32SPU7ABE5AG347AALAIL7IVFZTYSCISFD2HROEUIASA7CNA, seg_pos={0 0}, upload=444089]: Done uploading piece. err: <nil> Dec 26 17:34:38.736250622 (506h5m38.607333677s) [noise=true , address=145.79.227.42:4122 , piece_id=HLGY23RXNXCND25LS6JW35YX4TDJHH77OMA2JUE5GSFL76KYQ77Q, seg_pos={0 0}, upload=444089]: Done uploading piece. err: <nil> Dec 26 17:34:38.740648155 (506h5m38.611731201s) [noise=true , address=145.79.227.51:4410 , piece_id=6KF6ECGI7J2YPW3LVDWA2RLOERJXVJR7ADZ7WDWUNAXZBWEO7CIA, seg_pos={0 0}, upload=444089]: Done uploading piece. err: <nil> Dec 26 17:34:38.744072547 (506h5m38.615155582s) [noise=true , address=145.79.227.47:4265 , piece_id=V77EKXQ5GVAGSKL4W6CBJWDQMJOZXK6U67UO4D63ZWSISD7PBUJA, seg_pos={0 0}, upload=444089]: Done uploading piece. err: <nil> Dec 26 17:34:38.969785032 (506h5m38.840868067s) [noise=true , address=45.91.116.6:4975 , piece_id=LRDA54KUC5RYNCYLFGC7VRXSIXLIVZEEBR5LMBC7L4WQEQW67GRA, seg_pos={0 0}, upload=444089]: Done uploading piece. err: <nil> Dec 26 17:34:39.263027698 (506h5m39.134110733s) [noise=true , address=45.91.116.9:4261 , piece_id=PZBVVBULP6BCJP6ANFEIKXXHLOSYX55VGV65ST5VV3DJUCIOWV4A, seg_pos={0 0}, upload=444089]: Done uploading piece. err: <nil> Dec 26 17:34:39.558290509 (506h5m39.429373544s) [noise=true , address=45.91.116.5:4300 , piece_id=MY7IEOWS4SFF4OYCWTZXBFTYXB3LZRA7EQJS5THZCF5IL62ZVSHQ, seg_pos={0 0}, upload=444089]: Done uploading piece. err: <nil> Dec 26 17:34:39.558412061 (506h5m39.429495096s) [seg_pos={0 0}, upload=444089]: Segment reached optimal threshold of 13 pieces. Dec 26 17:34:39.558680011 (506h5m39.429763046s) [noise=true , address=145.79.227.29:4943 , piece_id=ODGVNYKPLIZBUYUNJZUHCJQ2K2ZURD5UPDWZS3F34JQSTBZ5WAAA, seg_pos={0 0}, upload=444089]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz): protocol: expected piece hash; context canceled; EOF Dec 26 17:34:39.558692886 (506h5m39.429775931s) [noise=true , address=145.79.227.29:4943 , piece_id=ODGVNYKPLIZBUYUNJZUHCJQ2K2ZURD5UPDWZS3F34JQSTBZ5WAAA, seg_pos={0 0}, upload=444089]: Long tail cancellation due to optimal threshold Dec 26 17:34:39.558702083 (506h5m39.429785128s) [noise=true , address=77.90.31.14:4663 , piece_id=EB7GNCWQG3KGQ4GQ4I74J5EX2G54YWE62JJCI2HNPRV2BVZQKFPQ, seg_pos={0 0}, upload=444089]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: write tcp 188.166.225.170:38944->77.90.31.14:4663: use of closed network connection; write tcp 188.166.225.170:38944->77.90.31.14:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38944->77.90.31.14:4663: use of closed network connection Dec 26 17:34:39.558709006 (506h5m39.429792061s) [noise=true , address=77.90.31.14:4663 , piece_id=EB7GNCWQG3KGQ4GQ4I74J5EX2G54YWE62JJCI2HNPRV2BVZQKFPQ, seg_pos={0 0}, upload=444089]: Long tail cancellation due to optimal threshold Dec 26 17:34:39.558885092 (506h5m39.429968127s) [noise=true , address=45.91.116.12:4227 , piece_id=24ITM3BBJS2KUYEWHQXATVWA7AH72V5245IJNCNDBDX4R6XV5BJQ, seg_pos={0 0}, upload=444089]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: write tcp 188.166.225.170:34996->45.91.116.12:4227: use of closed network connection; write tcp 188.166.225.170:34996->45.91.116.12:4227: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34996->45.91.116.12:4227: use of closed network connection Dec 26 17:34:39.558891674 (506h5m39.429974709s) [noise=true , address=45.91.116.12:4227 , piece_id=24ITM3BBJS2KUYEWHQXATVWA7AH72V5245IJNCNDBDX4R6XV5BJQ, seg_pos={0 0}, upload=444089]: Long tail cancellation due to optimal threshold Dec 26 17:34:39.558909198 (506h5m39.429992243s) [seg_pos={0 0}, upload=444089]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 17:34:39.558975133 (506h5m39.430058168s) [upload=444089]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:34:39.558979602 (506h5m39.430062647s) [upload=444089]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:34:39.847668435 (506h5m39.71875147s) [upload=444089]: Done uploading segments... Dec 26 17:34:39.847712239 (506h5m39.718795274s) [upload=444089]: Upload finished. err: <nil> Duration: 24.968569985s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 31.181779741s trace: https://jaeger.dev.storj.io/trace/109f3e067e010c2f FAIL (2025-12-26 17:36:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 127c4HGgJu3z16vxq7TYZydCUa86FhCc3TmoPZoheVeqTbnJfne, Piece ID: DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q) context deadline exceeded; piecestore: (Node ID: 1ahTbggGRi6cJa4JPBfSEgLJYymrgVuLxSc15BV3qj5gPWPFiZ, Piece ID: WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA) context deadline exceeded; piecestore: (Node ID: 1m4M9WpEgS8BgLgQ2kwqzb6szQ2k4KMS3pv6DRQsEPS2ujPyLM, Piece ID: TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ) context deadline exceeded ======LOGS====== 26 16:46:05.282897453 (505h17m5.153980498s) [noise=true , address=77.90.31.18:4828 , piece_id=EBH3I5TADBPDMDRYQSG7NQVEFRJVNACURFHR3UTKQL2ZPE7KAFVA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.282954052 (505h17m5.154037097s) [noise=true , address=77.90.31.21:4885 , piece_id=6YIKCRJQZ4Z2EKOB7Y7QZFVS2WIHV7CJUJVOKV5DS64CNLADLMVQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283314878 (505h17m5.154397923s) [noise=true , address=145.79.227.44:4199 , piece_id=7ZE6F5CL4VK6NKCPLM6H7RLIHLEO2RWEG5EGJFXF7X7Y4CSFX6VQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283493097 (505h17m5.154576143s) [noise=true , address=77.90.31.11:4578 , piece_id=TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283720251 (505h17m5.154803296s) [noise=true , address=145.79.227.12:4797 , piece_id=OR6AQ5RKIULPNCDS4TXRRXNXKXZDM4TT2J367SRQJEYZAYYML6LA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283969956 (505h17m5.155053001s) [noise=true , address=77.90.31.16:4758 , piece_id=WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284146873 (505h17m5.155229918s) [noise=true , address=145.79.227.40:4071 , piece_id=IBCGYA4LAJIRKFVJV2E6T2NPRCVADJIP5SAJ75H4QXDIMOQAN2NA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284470589 (505h17m5.155553634s) [noise=true , address=77.90.31.15:4710 , piece_id=25MAPHYJKQV4XQ7X67IVTNXM2ZKPBALDEKK66ZK67TFX3M74SGRQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284553537 (505h17m5.155636582s) [noise=true , address=77.90.31.19:4860 , piece_id=P2FISE3LU4Q5ZVOOGYHZAO52MIAE5DYWLFY6CVXSZIUC5ONL7HYQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284974238 (505h17m5.156057283s) [noise=true , address=77.90.31.10:4530 , piece_id=DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.285009706 (505h17m5.156092751s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.285544504 (505h17m5.156627549s) [noise=true , address=77.90.31.14:4685 , piece_id=4XZFFB7E642K5A4IKKN6Y6NJ42POJWUCIK7534PNM5A332YHTABA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.286800255 (505h17m5.15788337s) [noise=true , address=77.90.31.13:4644 , piece_id=6QLHFIXW7DIL3XRHVODFU6LPMRW25WSM77LRPYGVGUEN36NUALMA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.287403053 (505h17m5.158486098s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.287928925 (505h17m5.15901197s) [noise=true , address=145.79.227.11:4737 , piece_id=FAQBVVD3I2B5ZOPSR2VCZGSRQNVIRSZRVQCX5E33QECVEAOKLHRQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.289150881 (505h17m5.160233926s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:06.768498316 (505h17m6.639581351s) [upload=443378]: Next returned nil segment Dec 26 16:46:06.768512754 (505h17m6.63959579s) [upload=443378]: Waiting for error group managing segments... Dec 26 16:46:07.177478726 (505h17m7.048561771s) [noise=true , address=145.79.227.11:4737 , piece_id=FAQBVVD3I2B5ZOPSR2VCZGSRQNVIRSZRVQCX5E33QECVEAOKLHRQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.187360212 (505h17m7.058443247s) [noise=true , address=145.79.227.12:4797 , piece_id=OR6AQ5RKIULPNCDS4TXRRXNXKXZDM4TT2J367SRQJEYZAYYML6LA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.191564518 (505h17m7.062647563s) [noise=true , address=64.49.13.26:4950 , piece_id=NFO7SRXQGAUISAEPYC6OCLPCUTKFWR5PSOWUJZ5YHSLCYNZQQ7SQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.431132678 (505h17m7.302215713s) [noise=true , address=77.90.31.10:4530 , piece_id=DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.443236505 (505h17m7.31431955s) [noise=true , address=77.90.31.15:4710 , piece_id=25MAPHYJKQV4XQ7X67IVTNXM2ZKPBALDEKK66ZK67TFX3M74SGRQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.515779830 (505h17m7.386862865s) [noise=true , address=77.90.31.21:4885 , piece_id=6YIKCRJQZ4Z2EKOB7Y7QZFVS2WIHV7CJUJVOKV5DS64CNLADLMVQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.030094386 (505h17m7.901177421s) [noise=true , address=145.79.227.40:4071 , piece_id=IBCGYA4LAJIRKFVJV2E6T2NPRCVADJIP5SAJ75H4QXDIMOQAN2NA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.108333867 (505h17m7.979416913s) [noise=true , address=77.90.31.16:4758 , piece_id=WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.146456269 (505h17m8.017539314s) [noise=true , address=145.79.227.44:4199 , piece_id=7ZE6F5CL4VK6NKCPLM6H7RLIHLEO2RWEG5EGJFXF7X7Y4CSFX6VQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.167739325 (505h17m8.03882236s) [noise=true , address=77.90.31.18:4828 , piece_id=EBH3I5TADBPDMDRYQSG7NQVEFRJVNACURFHR3UTKQL2ZPE7KAFVA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.292396866 (505h17m8.163479911s) [noise=true , address=77.90.31.13:4644 , piece_id=6QLHFIXW7DIL3XRHVODFU6LPMRW25WSM77LRPYGVGUEN36NUALMA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.475687322 (505h17m8.346770367s) [noise=true , address=77.90.31.14:4685 , piece_id=4XZFFB7E642K5A4IKKN6Y6NJ42POJWUCIK7534PNM5A332YHTABA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.551777060 (505h17m8.422860105s) [noise=true , address=77.90.31.19:4860 , piece_id=P2FISE3LU4Q5ZVOOGYHZAO52MIAE5DYWLFY6CVXSZIUC5ONL7HYQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.678298409 (505h17m8.549381454s) [noise=true , address=77.90.31.11:4578 , piece_id=TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.678323457 (505h17m8.549406502s) [seg_pos={0 0}, upload=443378]: Segment reached optimal threshold of 14 pieces. Dec 26 16:46:08.678549747 (505h17m8.549632793s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1T9uQjynbTYcaixhvHWCKzatRWpFxMonHFi5idiMQtvivupKSu): protocol: write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection; write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection Dec 26 16:46:08.678558464 (505h17m8.549641509s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.678891048 (505h17m8.549974093s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ptydBd7T89noaveRPbmziYKKX9C1eNKaYG8BEfz2Zk4T9XtKz): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.678907879 (505h17m8.549990924s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127aXyQbxPkETL9tjbKQodpMMteTiZjBQ5C4muNnXELbuy6H5TM): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.678919432 (505h17m8.550002467s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.678931354 (505h17m8.550014399s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.679190638 (505h17m8.550273714s) [noise=true , address=64.49.13.140:4508 , piece_id=WKGTL6PEL62NGB4T3SZ4VH4YDG3523UJDLUCFRQRWJXVXDWXOMXA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.679200026 (505h17m8.550283061s) [noise=true , address=64.49.13.140:4508 , piece_id=WKGTL6PEL62NGB4T3SZ4VH4YDG3523UJDLUCFRQRWJXVXDWXOMXA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.679263507 (505h17m8.550346552s) [seg_pos={0 0}, upload=443378]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 16:46:08.679352386 (505h17m8.550435431s) [upload=443378]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:46:08.679359249 (505h17m8.550442294s) [upload=443378]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:46:08.985467612 (505h17m8.856550657s) [upload=443378]: Done uploading segments... Dec 26 16:46:08.985487951 (505h17m8.856570997s) [upload=443378]: Upload finished. err: <nil> Duration: 31.18179524s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 49.321588452s trace: https://jaeger.dev.storj.io/trace/109f3e067e010c2f FAIL (2025-12-26 17:35:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 127c4HGgJu3z16vxq7TYZydCUa86FhCc3TmoPZoheVeqTbnJfne, Piece ID: DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q) context deadline exceeded; piecestore: (Node ID: 1ahTbggGRi6cJa4JPBfSEgLJYymrgVuLxSc15BV3qj5gPWPFiZ, Piece ID: WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA) context deadline exceeded; piecestore: (Node ID: 1m4M9WpEgS8BgLgQ2kwqzb6szQ2k4KMS3pv6DRQsEPS2ujPyLM, Piece ID: TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ) context deadline exceeded ======LOGS====== 26 16:46:05.282897453 (505h17m5.153980498s) [noise=true , address=77.90.31.18:4828 , piece_id=EBH3I5TADBPDMDRYQSG7NQVEFRJVNACURFHR3UTKQL2ZPE7KAFVA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.282954052 (505h17m5.154037097s) [noise=true , address=77.90.31.21:4885 , piece_id=6YIKCRJQZ4Z2EKOB7Y7QZFVS2WIHV7CJUJVOKV5DS64CNLADLMVQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283314878 (505h17m5.154397923s) [noise=true , address=145.79.227.44:4199 , piece_id=7ZE6F5CL4VK6NKCPLM6H7RLIHLEO2RWEG5EGJFXF7X7Y4CSFX6VQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283493097 (505h17m5.154576143s) [noise=true , address=77.90.31.11:4578 , piece_id=TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283720251 (505h17m5.154803296s) [noise=true , address=145.79.227.12:4797 , piece_id=OR6AQ5RKIULPNCDS4TXRRXNXKXZDM4TT2J367SRQJEYZAYYML6LA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283969956 (505h17m5.155053001s) [noise=true , address=77.90.31.16:4758 , piece_id=WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284146873 (505h17m5.155229918s) [noise=true , address=145.79.227.40:4071 , piece_id=IBCGYA4LAJIRKFVJV2E6T2NPRCVADJIP5SAJ75H4QXDIMOQAN2NA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284470589 (505h17m5.155553634s) [noise=true , address=77.90.31.15:4710 , piece_id=25MAPHYJKQV4XQ7X67IVTNXM2ZKPBALDEKK66ZK67TFX3M74SGRQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284553537 (505h17m5.155636582s) [noise=true , address=77.90.31.19:4860 , piece_id=P2FISE3LU4Q5ZVOOGYHZAO52MIAE5DYWLFY6CVXSZIUC5ONL7HYQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284974238 (505h17m5.156057283s) [noise=true , address=77.90.31.10:4530 , piece_id=DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.285009706 (505h17m5.156092751s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.285544504 (505h17m5.156627549s) [noise=true , address=77.90.31.14:4685 , piece_id=4XZFFB7E642K5A4IKKN6Y6NJ42POJWUCIK7534PNM5A332YHTABA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.286800255 (505h17m5.15788337s) [noise=true , address=77.90.31.13:4644 , piece_id=6QLHFIXW7DIL3XRHVODFU6LPMRW25WSM77LRPYGVGUEN36NUALMA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.287403053 (505h17m5.158486098s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.287928925 (505h17m5.15901197s) [noise=true , address=145.79.227.11:4737 , piece_id=FAQBVVD3I2B5ZOPSR2VCZGSRQNVIRSZRVQCX5E33QECVEAOKLHRQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.289150881 (505h17m5.160233926s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:06.768498316 (505h17m6.639581351s) [upload=443378]: Next returned nil segment Dec 26 16:46:06.768512754 (505h17m6.63959579s) [upload=443378]: Waiting for error group managing segments... Dec 26 16:46:07.177478726 (505h17m7.048561771s) [noise=true , address=145.79.227.11:4737 , piece_id=FAQBVVD3I2B5ZOPSR2VCZGSRQNVIRSZRVQCX5E33QECVEAOKLHRQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.187360212 (505h17m7.058443247s) [noise=true , address=145.79.227.12:4797 , piece_id=OR6AQ5RKIULPNCDS4TXRRXNXKXZDM4TT2J367SRQJEYZAYYML6LA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.191564518 (505h17m7.062647563s) [noise=true , address=64.49.13.26:4950 , piece_id=NFO7SRXQGAUISAEPYC6OCLPCUTKFWR5PSOWUJZ5YHSLCYNZQQ7SQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.431132678 (505h17m7.302215713s) [noise=true , address=77.90.31.10:4530 , piece_id=DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.443236505 (505h17m7.31431955s) [noise=true , address=77.90.31.15:4710 , piece_id=25MAPHYJKQV4XQ7X67IVTNXM2ZKPBALDEKK66ZK67TFX3M74SGRQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.515779830 (505h17m7.386862865s) [noise=true , address=77.90.31.21:4885 , piece_id=6YIKCRJQZ4Z2EKOB7Y7QZFVS2WIHV7CJUJVOKV5DS64CNLADLMVQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.030094386 (505h17m7.901177421s) [noise=true , address=145.79.227.40:4071 , piece_id=IBCGYA4LAJIRKFVJV2E6T2NPRCVADJIP5SAJ75H4QXDIMOQAN2NA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.108333867 (505h17m7.979416913s) [noise=true , address=77.90.31.16:4758 , piece_id=WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.146456269 (505h17m8.017539314s) [noise=true , address=145.79.227.44:4199 , piece_id=7ZE6F5CL4VK6NKCPLM6H7RLIHLEO2RWEG5EGJFXF7X7Y4CSFX6VQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.167739325 (505h17m8.03882236s) [noise=true , address=77.90.31.18:4828 , piece_id=EBH3I5TADBPDMDRYQSG7NQVEFRJVNACURFHR3UTKQL2ZPE7KAFVA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.292396866 (505h17m8.163479911s) [noise=true , address=77.90.31.13:4644 , piece_id=6QLHFIXW7DIL3XRHVODFU6LPMRW25WSM77LRPYGVGUEN36NUALMA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.475687322 (505h17m8.346770367s) [noise=true , address=77.90.31.14:4685 , piece_id=4XZFFB7E642K5A4IKKN6Y6NJ42POJWUCIK7534PNM5A332YHTABA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.551777060 (505h17m8.422860105s) [noise=true , address=77.90.31.19:4860 , piece_id=P2FISE3LU4Q5ZVOOGYHZAO52MIAE5DYWLFY6CVXSZIUC5ONL7HYQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.678298409 (505h17m8.549381454s) [noise=true , address=77.90.31.11:4578 , piece_id=TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.678323457 (505h17m8.549406502s) [seg_pos={0 0}, upload=443378]: Segment reached optimal threshold of 14 pieces. Dec 26 16:46:08.678549747 (505h17m8.549632793s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1T9uQjynbTYcaixhvHWCKzatRWpFxMonHFi5idiMQtvivupKSu): protocol: write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection; write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection Dec 26 16:46:08.678558464 (505h17m8.549641509s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.678891048 (505h17m8.549974093s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ptydBd7T89noaveRPbmziYKKX9C1eNKaYG8BEfz2Zk4T9XtKz): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.678907879 (505h17m8.549990924s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127aXyQbxPkETL9tjbKQodpMMteTiZjBQ5C4muNnXELbuy6H5TM): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.678919432 (505h17m8.550002467s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.678931354 (505h17m8.550014399s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.679190638 (505h17m8.550273714s) [noise=true , address=64.49.13.140:4508 , piece_id=WKGTL6PEL62NGB4T3SZ4VH4YDG3523UJDLUCFRQRWJXVXDWXOMXA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.679200026 (505h17m8.550283061s) [noise=true , address=64.49.13.140:4508 , piece_id=WKGTL6PEL62NGB4T3SZ4VH4YDG3523UJDLUCFRQRWJXVXDWXOMXA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.679263507 (505h17m8.550346552s) [seg_pos={0 0}, upload=443378]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 16:46:08.679352386 (505h17m8.550435431s) [upload=443378]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:46:08.679359249 (505h17m8.550442294s) [upload=443378]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:46:08.985467612 (505h17m8.856550657s) [upload=443378]: Done uploading segments... Dec 26 16:46:08.985487951 (505h17m8.856570997s) [upload=443378]: Upload finished. err: <nil> Duration: 49.321608912s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [...XX..X.X..XX..X.X......XX..X....X.X.XX...XX...XX....XX..XX] duration: 30.0046186s trace: https://jaeger.dev.storj.io/trace/5835db663f93224f FAIL (2025-12-26 17:35:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: PZBVVBULP6BCJP6ANFEIKXXHLOSYX55VGV65ST5VV3DJUCIOWV4A) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: LRDA54KUC5RYNCYLFGC7VRXSIXLIVZEEBR5LMBC7L4WQEQW67GRA) context deadline exceeded ======LOGS====== h5m36.320342257s) [upload=444089]: Uploading segments... Dec 26 17:34:36.449470063 (506h5m36.320553108s) [upload=444089]: Got next segment. Inline: false Dec 26 17:34:36.449502284 (506h5m36.320585329s) [upload=444089]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 17:34:36.449507534 (506h5m36.320590569s) [upload=444089]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 17:34:36.704238476 (506h5m36.575321511s) [seg_pos={0 0}, upload=444089]: Begin upload segment... Dec 26 17:34:36.704303771 (506h5m36.575386806s) [seg_pos={0 0}, upload=444089]: Done begin upload segment. Dec 26 17:34:36.704328949 (506h5m36.575411984s) [noise=true , address=145.79.227.39:4015 , piece_id=WPFE4H77F7OFWA5M73NKI7F3ALXVLH63AA6HB3PTQJR4EVEJQ4NQ, seg_pos={0 0}, upload=444089]: Uploading piece... Dec 26 17:34:36.704344969 (506h5m36.575428014s) [noise=true , address=145.79.227.29:4943 , piece_id=ODGVNYKPLIZBUYUNJZUHCJQ2K2ZURD5UPDWZS3F34JQSTBZ5WAAA, seg_pos={0 0}, upload=444089]: Uploading piece... Dec 26 17:34:36.704785899 (506h5m36.575868944s) [noise=true , address=145.79.227.14:4839 , piece_id=XUTEEECYLYYN4X5N7ATZL32B6LSQCCIJWDJZHORYRHAZTHJ3UE7A, seg_pos={0 0}, upload=444089]: Uploading piece... Dec 26 17:34:36.705072314 (506h5m36.576155349s) [noise=true , address=145.79.227.11:4729 , piece_id=H62Q4ZQ2NEQGJ4XQYUORX3TITDWQRJNHSBQ3XNSEBHMVCPUWU5TA, seg_pos={0 0}, upload=444089]: Uploading piece... Dec 26 17:34:36.705347178 (506h5m36.576430213s) [noise=true , address=145.79.227.51:4410 , piece_id=6KF6ECGI7J2YPW3LVDWA2RLOERJXVJR7ADZ7WDWUNAXZBWEO7CIA, seg_pos={0 0}, upload=444089]: Uploading piece... Dec 26 17:34:36.705666385 (506h5m36.57674943s) [noise=true , address=145.79.227.10:4401 , piece_id=LREJ32SPU7ABE5AG347AALAIL7IVFZTYSCISFD2HROEUIASA7CNA, seg_pos={0 0}, upload=444089]: Uploading piece... Dec 26 17:34:36.705672767 (506h5m36.576755802s) [noise=true , address=145.79.227.47:4265 , piece_id=V77EKXQ5GVAGSKL4W6CBJWDQMJOZXK6U67UO4D63ZWSISD7PBUJA, seg_pos={0 0}, upload=444089]: Uploading piece... Dec 26 17:34:36.705947450 (506h5m36.577030485s) [noise=true , address=45.91.116.9:4261 , piece_id=PZBVVBULP6BCJP6ANFEIKXXHLOSYX55VGV65ST5VV3DJUCIOWV4A, seg_pos={0 0}, upload=444089]: Uploading piece... Dec 26 17:34:36.706017995 (506h5m36.57710103s) [noise=true , address=145.79.227.21:4475 , piece_id=G66X2QLH27XZPW5F73COJD5CE5ZYB65TFI2OBNWPQRMQSTNI7XKQ, seg_pos={0 0}, upload=444089]: Uploading piece... Dec 26 17:34:36.706262210 (506h5m36.577345245s) [noise=true , address=145.79.227.49:4337 , piece_id=XS2JRKD7JNCMWPBFA6GFRO4TNQMWUSHJQ73UIDTGFEBSFHDQEWWA, seg_pos={0 0}, upload=444089]: Uploading piece... Dec 26 17:34:36.706304049 (506h5m36.577387084s) [noise=true , address=45.91.116.12:4227 , piece_id=24ITM3BBJS2KUYEWHQXATVWA7AH72V5245IJNCNDBDX4R6XV5BJQ, seg_pos={0 0}, upload=444089]: Uploading piece... Dec 26 17:34:36.706789504 (506h5m36.577872539s) [noise=true , address=45.91.116.6:4975 , piece_id=LRDA54KUC5RYNCYLFGC7VRXSIXLIVZEEBR5LMBC7L4WQEQW67GRA, seg_pos={0 0}, upload=444089]: Uploading piece... Dec 26 17:34:36.707126466 (506h5m36.578209501s) [noise=true , address=145.79.227.38:4978 , piece_id=5YINIP7MUYYNOBNHV4ZGUWE366QEWGVG3TWORNJDOJEPU3AIBJEQ, seg_pos={0 0}, upload=444089]: Uploading piece... Dec 26 17:34:36.707614815 (506h5m36.578697881s) [noise=true , address=77.90.31.14:4663 , piece_id=EB7GNCWQG3KGQ4GQ4I74J5EX2G54YWE62JJCI2HNPRV2BVZQKFPQ, seg_pos={0 0}, upload=444089]: Uploading piece... Dec 26 17:34:36.707973388 (506h5m36.579056423s) [noise=true , address=45.91.116.5:4300 , piece_id=MY7IEOWS4SFF4OYCWTZXBFTYXB3LZRA7EQJS5THZCF5IL62ZVSHQ, seg_pos={0 0}, upload=444089]: Uploading piece... Dec 26 17:34:36.708277437 (506h5m36.579360472s) [noise=true , address=145.79.227.42:4122 , piece_id=HLGY23RXNXCND25LS6JW35YX4TDJHH77OMA2JUE5GSFL76KYQ77Q, seg_pos={0 0}, upload=444089]: Uploading piece... Dec 26 17:34:38.412838239 (506h5m38.283921284s) [upload=444089]: Next returned nil segment Dec 26 17:34:38.412890217 (506h5m38.283973262s) [upload=444089]: Waiting for error group managing segments... Dec 26 17:34:38.702510425 (506h5m38.57359346s) [noise=true , address=145.79.227.38:4978 , piece_id=5YINIP7MUYYNOBNHV4ZGUWE366QEWGVG3TWORNJDOJEPU3AIBJEQ, seg_pos={0 0}, upload=444089]: Done uploading piece. err: <nil> Dec 26 17:34:38.703987367 (506h5m38.575070412s) [noise=true , address=145.79.227.39:4015 , piece_id=WPFE4H77F7OFWA5M73NKI7F3ALXVLH63AA6HB3PTQJR4EVEJQ4NQ, seg_pos={0 0}, upload=444089]: Done uploading piece. err: <nil> Dec 26 17:34:38.708568922 (506h5m38.579651957s) [noise=true , address=145.79.227.11:4729 , piece_id=H62Q4ZQ2NEQGJ4XQYUORX3TITDWQRJNHSBQ3XNSEBHMVCPUWU5TA, seg_pos={0 0}, upload=444089]: Done uploading piece. err: <nil> Dec 26 17:34:38.714031684 (506h5m38.585114719s) [noise=true , address=145.79.227.49:4337 , piece_id=XS2JRKD7JNCMWPBFA6GFRO4TNQMWUSHJQ73UIDTGFEBSFHDQEWWA, seg_pos={0 0}, upload=444089]: Done uploading piece. err: <nil> Dec 26 17:34:38.714566472 (506h5m38.585649507s) [noise=true , address=145.79.227.21:4475 , piece_id=G66X2QLH27XZPW5F73COJD5CE5ZYB65TFI2OBNWPQRMQSTNI7XKQ, seg_pos={0 0}, upload=444089]: Done uploading piece. err: <nil> Dec 26 17:34:38.724676302 (506h5m38.595759347s) [noise=true , address=145.79.227.14:4839 , piece_id=XUTEEECYLYYN4X5N7ATZL32B6LSQCCIJWDJZHORYRHAZTHJ3UE7A, seg_pos={0 0}, upload=444089]: Done uploading piece. err: <nil> Dec 26 17:34:38.726402780 (506h5m38.597485825s) [noise=true , address=145.79.227.10:4401 , piece_id=LREJ32SPU7ABE5AG347AALAIL7IVFZTYSCISFD2HROEUIASA7CNA, seg_pos={0 0}, upload=444089]: Done uploading piece. err: <nil> Dec 26 17:34:38.736250622 (506h5m38.607333677s) [noise=true , address=145.79.227.42:4122 , piece_id=HLGY23RXNXCND25LS6JW35YX4TDJHH77OMA2JUE5GSFL76KYQ77Q, seg_pos={0 0}, upload=444089]: Done uploading piece. err: <nil> Dec 26 17:34:38.740648155 (506h5m38.611731201s) [noise=true , address=145.79.227.51:4410 , piece_id=6KF6ECGI7J2YPW3LVDWA2RLOERJXVJR7ADZ7WDWUNAXZBWEO7CIA, seg_pos={0 0}, upload=444089]: Done uploading piece. err: <nil> Dec 26 17:34:38.744072547 (506h5m38.615155582s) [noise=true , address=145.79.227.47:4265 , piece_id=V77EKXQ5GVAGSKL4W6CBJWDQMJOZXK6U67UO4D63ZWSISD7PBUJA, seg_pos={0 0}, upload=444089]: Done uploading piece. err: <nil> Dec 26 17:34:38.969785032 (506h5m38.840868067s) [noise=true , address=45.91.116.6:4975 , piece_id=LRDA54KUC5RYNCYLFGC7VRXSIXLIVZEEBR5LMBC7L4WQEQW67GRA, seg_pos={0 0}, upload=444089]: Done uploading piece. err: <nil> Dec 26 17:34:39.263027698 (506h5m39.134110733s) [noise=true , address=45.91.116.9:4261 , piece_id=PZBVVBULP6BCJP6ANFEIKXXHLOSYX55VGV65ST5VV3DJUCIOWV4A, seg_pos={0 0}, upload=444089]: Done uploading piece. err: <nil> Dec 26 17:34:39.558290509 (506h5m39.429373544s) [noise=true , address=45.91.116.5:4300 , piece_id=MY7IEOWS4SFF4OYCWTZXBFTYXB3LZRA7EQJS5THZCF5IL62ZVSHQ, seg_pos={0 0}, upload=444089]: Done uploading piece. err: <nil> Dec 26 17:34:39.558412061 (506h5m39.429495096s) [seg_pos={0 0}, upload=444089]: Segment reached optimal threshold of 13 pieces. Dec 26 17:34:39.558680011 (506h5m39.429763046s) [noise=true , address=145.79.227.29:4943 , piece_id=ODGVNYKPLIZBUYUNJZUHCJQ2K2ZURD5UPDWZS3F34JQSTBZ5WAAA, seg_pos={0 0}, upload=444089]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz): protocol: expected piece hash; context canceled; EOF Dec 26 17:34:39.558692886 (506h5m39.429775931s) [noise=true , address=145.79.227.29:4943 , piece_id=ODGVNYKPLIZBUYUNJZUHCJQ2K2ZURD5UPDWZS3F34JQSTBZ5WAAA, seg_pos={0 0}, upload=444089]: Long tail cancellation due to optimal threshold Dec 26 17:34:39.558702083 (506h5m39.429785128s) [noise=true , address=77.90.31.14:4663 , piece_id=EB7GNCWQG3KGQ4GQ4I74J5EX2G54YWE62JJCI2HNPRV2BVZQKFPQ, seg_pos={0 0}, upload=444089]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: write tcp 188.166.225.170:38944->77.90.31.14:4663: use of closed network connection; write tcp 188.166.225.170:38944->77.90.31.14:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38944->77.90.31.14:4663: use of closed network connection Dec 26 17:34:39.558709006 (506h5m39.429792061s) [noise=true , address=77.90.31.14:4663 , piece_id=EB7GNCWQG3KGQ4GQ4I74J5EX2G54YWE62JJCI2HNPRV2BVZQKFPQ, seg_pos={0 0}, upload=444089]: Long tail cancellation due to optimal threshold Dec 26 17:34:39.558885092 (506h5m39.429968127s) [noise=true , address=45.91.116.12:4227 , piece_id=24ITM3BBJS2KUYEWHQXATVWA7AH72V5245IJNCNDBDX4R6XV5BJQ, seg_pos={0 0}, upload=444089]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: write tcp 188.166.225.170:34996->45.91.116.12:4227: use of closed network connection; write tcp 188.166.225.170:34996->45.91.116.12:4227: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34996->45.91.116.12:4227: use of closed network connection Dec 26 17:34:39.558891674 (506h5m39.429974709s) [noise=true , address=45.91.116.12:4227 , piece_id=24ITM3BBJS2KUYEWHQXATVWA7AH72V5245IJNCNDBDX4R6XV5BJQ, seg_pos={0 0}, upload=444089]: Long tail cancellation due to optimal threshold Dec 26 17:34:39.558909198 (506h5m39.429992243s) [seg_pos={0 0}, upload=444089]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 17:34:39.558975133 (506h5m39.430058168s) [upload=444089]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:34:39.558979602 (506h5m39.430062647s) [upload=444089]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:34:39.847668435 (506h5m39.71875147s) [upload=444089]: Done uploading segments... Dec 26 17:34:39.847712239 (506h5m39.718795274s) [upload=444089]: Upload finished. err: <nil> Duration: 30.004645923s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [....XX..X.X..XX..X.X......XX..X....X.X.XX...XX...XX....XX..X] duration: 30.003507985s trace: https://jaeger.dev.storj.io/trace/2462c1f4bbf67250 FAIL (2025-12-26 17:34:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== m47.965226237s) [upload=444078]: Uploading segments... Dec 26 17:33:48.094221381 (506h4m47.965304426s) [upload=444078]: Got next segment. Inline: false Dec 26 17:33:48.094261527 (506h4m47.965344572s) [upload=444078]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 17:33:48.094269362 (506h4m47.965352407s) [upload=444078]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 17:33:48.345195179 (506h4m48.216278224s) [seg_pos={0 0}, upload=444078]: Begin upload segment... Dec 26 17:33:48.345260363 (506h4m48.216343398s) [seg_pos={0 0}, upload=444078]: Done begin upload segment. Dec 26 17:33:48.345308023 (506h4m48.216391068s) [noise=true , address=145.79.227.45:4231 , piece_id=SKLFLVNMBKTJR54HTEVOTLCNCCEUX2EECYI6M5NAGYWLVCCB4CDQ, seg_pos={0 0}, upload=444078]: Uploading piece... Dec 26 17:33:48.345315627 (506h4m48.216398673s) [noise=true , address=145.79.227.36:4519 , piece_id=YZG2YQVSUCE6S6WY4SYMZVRRZOONN6IXXYMHLZKZOHCKSCMMUR3Q, seg_pos={0 0}, upload=444078]: Uploading piece... Dec 26 17:33:48.345953172 (506h4m48.217036207s) [noise=true , address=45.91.116.11:4119 , piece_id=L6Z7I36XGAOH2GI5EFVGSK3H5W4ZDE44ZJHWBW6LTFI6GKDUX4YA, seg_pos={0 0}, upload=444078]: Uploading piece... Dec 26 17:33:48.346052481 (506h4m48.217135516s) [noise=true , address=45.91.116.14:4192 , piece_id=4XCMEBSR4RU4GSURWQGK3TSNMDW6P4YHKUU3J2IWI4E6JHUJUFJA, seg_pos={0 0}, upload=444078]: Uploading piece... Dec 26 17:33:48.346601797 (506h4m48.217684832s) [noise=true , address=145.79.227.39:4013 , piece_id=M5VBSGHOG3LMLCCSTJNKTHO6AHJQQKHQHBGPPGOXACHHIBCAQ2UQ, seg_pos={0 0}, upload=444078]: Uploading piece... Dec 26 17:33:48.347135934 (506h4m48.218218979s) [noise=true , address=145.79.227.31:4799 , piece_id=YRJKEULMYLOZLGMQ4HTBBL2MN5QZCKPIYQG5NFCJVTGJOWM27Z6Q, seg_pos={0 0}, upload=444078]: Uploading piece... Dec 26 17:33:48.347342999 (506h4m48.218426034s) [noise=true , address=145.79.227.48:4302 , piece_id=7KGTIIYDWSD2RKNQY5BDK6LCS24Y66IPBGY5GGWSK5YQKO7KXUAQ, seg_pos={0 0}, upload=444078]: Uploading piece... Dec 26 17:33:48.347956837 (506h4m48.219039872s) [noise=true , address=145.79.227.6:4259 , piece_id=WMZ5OC67HDHI4Z7BK53GVEC5RF523JW6WMSRASIIDJLGFXLL34ZQ, seg_pos={0 0}, upload=444078]: Uploading piece... Dec 26 17:33:48.348440528 (506h4m48.219523563s) [noise=true , address=145.79.227.2:4114 , piece_id=PU5U427YCRS4IXDQWKWBRJBWO43IRQ73RQWBX7STOG4ES6L5HPXQ, seg_pos={0 0}, upload=444078]: Uploading piece... Dec 26 17:33:48.348897909 (506h4m48.219980954s) [noise=true , address=45.91.116.13:4153 , piece_id=6YONETK4ZYPFX6UYFDC7RSTCYUZFSH63LJLNY4AINCFER6S6A4PQ, seg_pos={0 0}, upload=444078]: Uploading piece... Dec 26 17:33:48.349336715 (506h4m48.22041975s) [noise=true , address=145.79.227.10:4401 , piece_id=6JADHOOL6KRW6OP5SQFA2DNEMMCHDPLSPVMGLX2Q5H6KW3LZIQEA, seg_pos={0 0}, upload=444078]: Uploading piece... Dec 26 17:33:48.349767976 (506h4m48.220851021s) [noise=true , address=45.91.116.5:4300 , piece_id=5V2NQG4HLWNV7PFLKUS5JUDGPEN43LOUXCMFNEV6AJYI5GZ5CG7Q, seg_pos={0 0}, upload=444078]: Uploading piece... Dec 26 17:33:48.354253928 (506h4m48.225336973s) [noise=true , address=145.79.227.43:4158 , piece_id=HEQN7NI3JPJCQUR37H7AGSSVV6W6Q3TRRD2N4FR6XHESZY3UZRNA, seg_pos={0 0}, upload=444078]: Uploading piece... Dec 26 17:33:48.354747427 (506h4m48.225830472s) [noise=true , address=45.91.116.4:4940 , piece_id=WHLW6DCYHQI22E52R72HLWBI2TWWKOII5SZWFJJJ2WNMNSH7LB5A, seg_pos={0 0}, upload=444078]: Uploading piece... Dec 26 17:33:48.355105911 (506h4m48.226188946s) [noise=true , address=145.79.227.47:4266 , piece_id=N3FQGEWIYG4V53Z2O6KHNUSRA2XSVGK7DKIWGSJ3MI4S3MDXAICA, seg_pos={0 0}, upload=444078]: Uploading piece... Dec 26 17:33:48.355634136 (506h4m48.226717171s) [noise=true , address=145.79.227.15:4874 , piece_id=WVZ2SD6EYNO54LHOIVPEYD67TAZWENKDBZUOS24AE7HPL6BGQY7A, seg_pos={0 0}, upload=444078]: Uploading piece... Dec 26 17:33:49.959552677 (506h4m49.830635722s) [upload=444078]: Next returned nil segment Dec 26 17:33:49.959568888 (506h4m49.830651923s) [upload=444078]: Waiting for error group managing segments... Dec 26 17:33:50.350933845 (506h4m50.22201689s) [noise=true , address=145.79.227.48:4302 , piece_id=7KGTIIYDWSD2RKNQY5BDK6LCS24Y66IPBGY5GGWSK5YQKO7KXUAQ, seg_pos={0 0}, upload=444078]: Done uploading piece. err: <nil> Dec 26 17:33:50.367973973 (506h4m50.239057008s) [noise=true , address=145.79.227.47:4266 , piece_id=N3FQGEWIYG4V53Z2O6KHNUSRA2XSVGK7DKIWGSJ3MI4S3MDXAICA, seg_pos={0 0}, upload=444078]: Done uploading piece. err: <nil> Dec 26 17:33:50.368310242 (506h4m50.239393277s) [noise=true , address=145.79.227.45:4231 , piece_id=SKLFLVNMBKTJR54HTEVOTLCNCCEUX2EECYI6M5NAGYWLVCCB4CDQ, seg_pos={0 0}, upload=444078]: Done uploading piece. err: <nil> Dec 26 17:33:50.368435080 (506h4m50.239518116s) [noise=true , address=145.79.227.15:4874 , piece_id=WVZ2SD6EYNO54LHOIVPEYD67TAZWENKDBZUOS24AE7HPL6BGQY7A, seg_pos={0 0}, upload=444078]: Done uploading piece. err: <nil> Dec 26 17:33:50.375071156 (506h4m50.246154191s) [noise=true , address=145.79.227.36:4519 , piece_id=YZG2YQVSUCE6S6WY4SYMZVRRZOONN6IXXYMHLZKZOHCKSCMMUR3Q, seg_pos={0 0}, upload=444078]: Done uploading piece. err: <nil> Dec 26 17:33:50.375891739 (506h4m50.246974784s) [noise=true , address=145.79.227.39:4013 , piece_id=M5VBSGHOG3LMLCCSTJNKTHO6AHJQQKHQHBGPPGOXACHHIBCAQ2UQ, seg_pos={0 0}, upload=444078]: Done uploading piece. err: <nil> Dec 26 17:33:50.379671697 (506h4m50.250754742s) [noise=true , address=145.79.227.31:4799 , piece_id=YRJKEULMYLOZLGMQ4HTBBL2MN5QZCKPIYQG5NFCJVTGJOWM27Z6Q, seg_pos={0 0}, upload=444078]: Done uploading piece. err: <nil> Dec 26 17:33:50.380148013 (506h4m50.251231059s) [noise=true , address=145.79.227.10:4401 , piece_id=6JADHOOL6KRW6OP5SQFA2DNEMMCHDPLSPVMGLX2Q5H6KW3LZIQEA, seg_pos={0 0}, upload=444078]: Done uploading piece. err: <nil> Dec 26 17:33:50.721472315 (506h4m50.59255535s) [noise=true , address=45.91.116.14:4192 , piece_id=4XCMEBSR4RU4GSURWQGK3TSNMDW6P4YHKUU3J2IWI4E6JHUJUFJA, seg_pos={0 0}, upload=444078]: Done uploading piece. err: <nil> Dec 26 17:33:50.947604721 (506h4m50.818687766s) [noise=true , address=45.91.116.11:4119 , piece_id=L6Z7I36XGAOH2GI5EFVGSK3H5W4ZDE44ZJHWBW6LTFI6GKDUX4YA, seg_pos={0 0}, upload=444078]: Done uploading piece. err: <nil> Dec 26 17:33:51.480151352 (506h4m51.351234397s) [noise=true , address=45.91.116.5:4300 , piece_id=5V2NQG4HLWNV7PFLKUS5JUDGPEN43LOUXCMFNEV6AJYI5GZ5CG7Q, seg_pos={0 0}, upload=444078]: Done uploading piece. err: <nil> Dec 26 17:33:51.763774360 (506h4m51.634857425s) [noise=true , address=45.91.116.13:4153 , piece_id=6YONETK4ZYPFX6UYFDC7RSTCYUZFSH63LJLNY4AINCFER6S6A4PQ, seg_pos={0 0}, upload=444078]: Done uploading piece. err: <nil> Dec 26 17:33:51.804243651 (506h4m51.675326696s) [noise=true , address=45.91.116.4:4940 , piece_id=WHLW6DCYHQI22E52R72HLWBI2TWWKOII5SZWFJJJ2WNMNSH7LB5A, seg_pos={0 0}, upload=444078]: Done uploading piece. err: <nil> Dec 26 17:33:51.804269098 (506h4m51.675352144s) [seg_pos={0 0}, upload=444078]: Segment reached optimal threshold of 13 pieces. Dec 26 17:33:51.804485521 (506h4m51.675568556s) [noise=true , address=145.79.227.2:4114 , piece_id=PU5U427YCRS4IXDQWKWBRJBWO43IRQ73RQWBX7STOG4ES6L5HPXQ, seg_pos={0 0}, upload=444078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN): protocol: write tcp 188.166.225.170:57138->145.79.227.2:4114: use of closed network connection; write tcp 188.166.225.170:57138->145.79.227.2:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57138->145.79.227.2:4114: use of closed network connection Dec 26 17:33:51.804492444 (506h4m51.675575489s) [noise=true , address=145.79.227.2:4114 , piece_id=PU5U427YCRS4IXDQWKWBRJBWO43IRQ73RQWBX7STOG4ES6L5HPXQ, seg_pos={0 0}, upload=444078]: Long tail cancellation due to optimal threshold Dec 26 17:33:51.804500179 (506h4m51.675583224s) [noise=true , address=145.79.227.6:4259 , piece_id=WMZ5OC67HDHI4Z7BK53GVEC5RF523JW6WMSRASIIDJLGFXLL34ZQ, seg_pos={0 0}, upload=444078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y): protocol: write tcp 188.166.225.170:38530->145.79.227.6:4259: use of closed network connection; write tcp 188.166.225.170:38530->145.79.227.6:4259: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38530->145.79.227.6:4259: use of closed network connection Dec 26 17:33:51.804511440 (506h4m51.675594475s) [noise=true , address=145.79.227.6:4259 , piece_id=WMZ5OC67HDHI4Z7BK53GVEC5RF523JW6WMSRASIIDJLGFXLL34ZQ, seg_pos={0 0}, upload=444078]: Long tail cancellation due to optimal threshold Dec 26 17:33:51.804686423 (506h4m51.675769468s) [noise=true , address=145.79.227.43:4158 , piece_id=HEQN7NI3JPJCQUR37H7AGSSVV6W6Q3TRRD2N4FR6XHESZY3UZRNA, seg_pos={0 0}, upload=444078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2): protocol: expected piece hash; context canceled; EOF Dec 26 17:33:51.804692596 (506h4m51.675775631s) [noise=true , address=145.79.227.43:4158 , piece_id=HEQN7NI3JPJCQUR37H7AGSSVV6W6Q3TRRD2N4FR6XHESZY3UZRNA, seg_pos={0 0}, upload=444078]: Long tail cancellation due to optimal threshold Dec 26 17:33:51.804710259 (506h4m51.675793304s) [seg_pos={0 0}, upload=444078]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 17:33:51.804789089 (506h4m51.675872124s) [upload=444078]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:33:51.804793838 (506h4m51.675876873s) [upload=444078]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:33:52.118290901 (506h4m51.989373936s) [upload=444078]: Done uploading segments... Dec 26 17:33:52.118432561 (506h4m51.989515606s) [upload=444078]: Upload finished. err: <nil> Duration: 30.003523655s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 20.35917898s trace: https://jaeger.dev.storj.io/trace/109f3e067e010c2f FAIL (2025-12-26 17:34:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 127c4HGgJu3z16vxq7TYZydCUa86FhCc3TmoPZoheVeqTbnJfne, Piece ID: DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q) context deadline exceeded; piecestore: (Node ID: 1ahTbggGRi6cJa4JPBfSEgLJYymrgVuLxSc15BV3qj5gPWPFiZ, Piece ID: WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA) context deadline exceeded; piecestore: (Node ID: 1m4M9WpEgS8BgLgQ2kwqzb6szQ2k4KMS3pv6DRQsEPS2ujPyLM, Piece ID: TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ) context deadline exceeded ======LOGS====== 26 16:46:05.282897453 (505h17m5.153980498s) [noise=true , address=77.90.31.18:4828 , piece_id=EBH3I5TADBPDMDRYQSG7NQVEFRJVNACURFHR3UTKQL2ZPE7KAFVA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.282954052 (505h17m5.154037097s) [noise=true , address=77.90.31.21:4885 , piece_id=6YIKCRJQZ4Z2EKOB7Y7QZFVS2WIHV7CJUJVOKV5DS64CNLADLMVQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283314878 (505h17m5.154397923s) [noise=true , address=145.79.227.44:4199 , piece_id=7ZE6F5CL4VK6NKCPLM6H7RLIHLEO2RWEG5EGJFXF7X7Y4CSFX6VQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283493097 (505h17m5.154576143s) [noise=true , address=77.90.31.11:4578 , piece_id=TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283720251 (505h17m5.154803296s) [noise=true , address=145.79.227.12:4797 , piece_id=OR6AQ5RKIULPNCDS4TXRRXNXKXZDM4TT2J367SRQJEYZAYYML6LA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283969956 (505h17m5.155053001s) [noise=true , address=77.90.31.16:4758 , piece_id=WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284146873 (505h17m5.155229918s) [noise=true , address=145.79.227.40:4071 , piece_id=IBCGYA4LAJIRKFVJV2E6T2NPRCVADJIP5SAJ75H4QXDIMOQAN2NA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284470589 (505h17m5.155553634s) [noise=true , address=77.90.31.15:4710 , piece_id=25MAPHYJKQV4XQ7X67IVTNXM2ZKPBALDEKK66ZK67TFX3M74SGRQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284553537 (505h17m5.155636582s) [noise=true , address=77.90.31.19:4860 , piece_id=P2FISE3LU4Q5ZVOOGYHZAO52MIAE5DYWLFY6CVXSZIUC5ONL7HYQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284974238 (505h17m5.156057283s) [noise=true , address=77.90.31.10:4530 , piece_id=DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.285009706 (505h17m5.156092751s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.285544504 (505h17m5.156627549s) [noise=true , address=77.90.31.14:4685 , piece_id=4XZFFB7E642K5A4IKKN6Y6NJ42POJWUCIK7534PNM5A332YHTABA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.286800255 (505h17m5.15788337s) [noise=true , address=77.90.31.13:4644 , piece_id=6QLHFIXW7DIL3XRHVODFU6LPMRW25WSM77LRPYGVGUEN36NUALMA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.287403053 (505h17m5.158486098s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.287928925 (505h17m5.15901197s) [noise=true , address=145.79.227.11:4737 , piece_id=FAQBVVD3I2B5ZOPSR2VCZGSRQNVIRSZRVQCX5E33QECVEAOKLHRQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.289150881 (505h17m5.160233926s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:06.768498316 (505h17m6.639581351s) [upload=443378]: Next returned nil segment Dec 26 16:46:06.768512754 (505h17m6.63959579s) [upload=443378]: Waiting for error group managing segments... Dec 26 16:46:07.177478726 (505h17m7.048561771s) [noise=true , address=145.79.227.11:4737 , piece_id=FAQBVVD3I2B5ZOPSR2VCZGSRQNVIRSZRVQCX5E33QECVEAOKLHRQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.187360212 (505h17m7.058443247s) [noise=true , address=145.79.227.12:4797 , piece_id=OR6AQ5RKIULPNCDS4TXRRXNXKXZDM4TT2J367SRQJEYZAYYML6LA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.191564518 (505h17m7.062647563s) [noise=true , address=64.49.13.26:4950 , piece_id=NFO7SRXQGAUISAEPYC6OCLPCUTKFWR5PSOWUJZ5YHSLCYNZQQ7SQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.431132678 (505h17m7.302215713s) [noise=true , address=77.90.31.10:4530 , piece_id=DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.443236505 (505h17m7.31431955s) [noise=true , address=77.90.31.15:4710 , piece_id=25MAPHYJKQV4XQ7X67IVTNXM2ZKPBALDEKK66ZK67TFX3M74SGRQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.515779830 (505h17m7.386862865s) [noise=true , address=77.90.31.21:4885 , piece_id=6YIKCRJQZ4Z2EKOB7Y7QZFVS2WIHV7CJUJVOKV5DS64CNLADLMVQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.030094386 (505h17m7.901177421s) [noise=true , address=145.79.227.40:4071 , piece_id=IBCGYA4LAJIRKFVJV2E6T2NPRCVADJIP5SAJ75H4QXDIMOQAN2NA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.108333867 (505h17m7.979416913s) [noise=true , address=77.90.31.16:4758 , piece_id=WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.146456269 (505h17m8.017539314s) [noise=true , address=145.79.227.44:4199 , piece_id=7ZE6F5CL4VK6NKCPLM6H7RLIHLEO2RWEG5EGJFXF7X7Y4CSFX6VQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.167739325 (505h17m8.03882236s) [noise=true , address=77.90.31.18:4828 , piece_id=EBH3I5TADBPDMDRYQSG7NQVEFRJVNACURFHR3UTKQL2ZPE7KAFVA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.292396866 (505h17m8.163479911s) [noise=true , address=77.90.31.13:4644 , piece_id=6QLHFIXW7DIL3XRHVODFU6LPMRW25WSM77LRPYGVGUEN36NUALMA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.475687322 (505h17m8.346770367s) [noise=true , address=77.90.31.14:4685 , piece_id=4XZFFB7E642K5A4IKKN6Y6NJ42POJWUCIK7534PNM5A332YHTABA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.551777060 (505h17m8.422860105s) [noise=true , address=77.90.31.19:4860 , piece_id=P2FISE3LU4Q5ZVOOGYHZAO52MIAE5DYWLFY6CVXSZIUC5ONL7HYQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.678298409 (505h17m8.549381454s) [noise=true , address=77.90.31.11:4578 , piece_id=TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.678323457 (505h17m8.549406502s) [seg_pos={0 0}, upload=443378]: Segment reached optimal threshold of 14 pieces. Dec 26 16:46:08.678549747 (505h17m8.549632793s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1T9uQjynbTYcaixhvHWCKzatRWpFxMonHFi5idiMQtvivupKSu): protocol: write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection; write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection Dec 26 16:46:08.678558464 (505h17m8.549641509s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.678891048 (505h17m8.549974093s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ptydBd7T89noaveRPbmziYKKX9C1eNKaYG8BEfz2Zk4T9XtKz): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.678907879 (505h17m8.549990924s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127aXyQbxPkETL9tjbKQodpMMteTiZjBQ5C4muNnXELbuy6H5TM): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.678919432 (505h17m8.550002467s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.678931354 (505h17m8.550014399s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.679190638 (505h17m8.550273714s) [noise=true , address=64.49.13.140:4508 , piece_id=WKGTL6PEL62NGB4T3SZ4VH4YDG3523UJDLUCFRQRWJXVXDWXOMXA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.679200026 (505h17m8.550283061s) [noise=true , address=64.49.13.140:4508 , piece_id=WKGTL6PEL62NGB4T3SZ4VH4YDG3523UJDLUCFRQRWJXVXDWXOMXA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.679263507 (505h17m8.550346552s) [seg_pos={0 0}, upload=443378]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 16:46:08.679352386 (505h17m8.550435431s) [upload=443378]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:46:08.679359249 (505h17m8.550442294s) [upload=443378]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:46:08.985467612 (505h17m8.856550657s) [upload=443378]: Done uploading segments... Dec 26 16:46:08.985487951 (505h17m8.856570997s) [upload=443378]: Upload finished. err: <nil> Duration: 20.359207574s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 39.464722949s trace: https://jaeger.dev.storj.io/trace/109f3e067e010c2f FAIL (2025-12-26 17:33:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 127c4HGgJu3z16vxq7TYZydCUa86FhCc3TmoPZoheVeqTbnJfne, Piece ID: DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q) context deadline exceeded; piecestore: (Node ID: 1ahTbggGRi6cJa4JPBfSEgLJYymrgVuLxSc15BV3qj5gPWPFiZ, Piece ID: WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA) context deadline exceeded; piecestore: (Node ID: 1m4M9WpEgS8BgLgQ2kwqzb6szQ2k4KMS3pv6DRQsEPS2ujPyLM, Piece ID: TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ) context deadline exceeded ======LOGS====== 26 16:46:05.282897453 (505h17m5.153980498s) [noise=true , address=77.90.31.18:4828 , piece_id=EBH3I5TADBPDMDRYQSG7NQVEFRJVNACURFHR3UTKQL2ZPE7KAFVA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.282954052 (505h17m5.154037097s) [noise=true , address=77.90.31.21:4885 , piece_id=6YIKCRJQZ4Z2EKOB7Y7QZFVS2WIHV7CJUJVOKV5DS64CNLADLMVQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283314878 (505h17m5.154397923s) [noise=true , address=145.79.227.44:4199 , piece_id=7ZE6F5CL4VK6NKCPLM6H7RLIHLEO2RWEG5EGJFXF7X7Y4CSFX6VQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283493097 (505h17m5.154576143s) [noise=true , address=77.90.31.11:4578 , piece_id=TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283720251 (505h17m5.154803296s) [noise=true , address=145.79.227.12:4797 , piece_id=OR6AQ5RKIULPNCDS4TXRRXNXKXZDM4TT2J367SRQJEYZAYYML6LA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283969956 (505h17m5.155053001s) [noise=true , address=77.90.31.16:4758 , piece_id=WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284146873 (505h17m5.155229918s) [noise=true , address=145.79.227.40:4071 , piece_id=IBCGYA4LAJIRKFVJV2E6T2NPRCVADJIP5SAJ75H4QXDIMOQAN2NA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284470589 (505h17m5.155553634s) [noise=true , address=77.90.31.15:4710 , piece_id=25MAPHYJKQV4XQ7X67IVTNXM2ZKPBALDEKK66ZK67TFX3M74SGRQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284553537 (505h17m5.155636582s) [noise=true , address=77.90.31.19:4860 , piece_id=P2FISE3LU4Q5ZVOOGYHZAO52MIAE5DYWLFY6CVXSZIUC5ONL7HYQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284974238 (505h17m5.156057283s) [noise=true , address=77.90.31.10:4530 , piece_id=DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.285009706 (505h17m5.156092751s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.285544504 (505h17m5.156627549s) [noise=true , address=77.90.31.14:4685 , piece_id=4XZFFB7E642K5A4IKKN6Y6NJ42POJWUCIK7534PNM5A332YHTABA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.286800255 (505h17m5.15788337s) [noise=true , address=77.90.31.13:4644 , piece_id=6QLHFIXW7DIL3XRHVODFU6LPMRW25WSM77LRPYGVGUEN36NUALMA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.287403053 (505h17m5.158486098s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.287928925 (505h17m5.15901197s) [noise=true , address=145.79.227.11:4737 , piece_id=FAQBVVD3I2B5ZOPSR2VCZGSRQNVIRSZRVQCX5E33QECVEAOKLHRQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.289150881 (505h17m5.160233926s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:06.768498316 (505h17m6.639581351s) [upload=443378]: Next returned nil segment Dec 26 16:46:06.768512754 (505h17m6.63959579s) [upload=443378]: Waiting for error group managing segments... Dec 26 16:46:07.177478726 (505h17m7.048561771s) [noise=true , address=145.79.227.11:4737 , piece_id=FAQBVVD3I2B5ZOPSR2VCZGSRQNVIRSZRVQCX5E33QECVEAOKLHRQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.187360212 (505h17m7.058443247s) [noise=true , address=145.79.227.12:4797 , piece_id=OR6AQ5RKIULPNCDS4TXRRXNXKXZDM4TT2J367SRQJEYZAYYML6LA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.191564518 (505h17m7.062647563s) [noise=true , address=64.49.13.26:4950 , piece_id=NFO7SRXQGAUISAEPYC6OCLPCUTKFWR5PSOWUJZ5YHSLCYNZQQ7SQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.431132678 (505h17m7.302215713s) [noise=true , address=77.90.31.10:4530 , piece_id=DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.443236505 (505h17m7.31431955s) [noise=true , address=77.90.31.15:4710 , piece_id=25MAPHYJKQV4XQ7X67IVTNXM2ZKPBALDEKK66ZK67TFX3M74SGRQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.515779830 (505h17m7.386862865s) [noise=true , address=77.90.31.21:4885 , piece_id=6YIKCRJQZ4Z2EKOB7Y7QZFVS2WIHV7CJUJVOKV5DS64CNLADLMVQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.030094386 (505h17m7.901177421s) [noise=true , address=145.79.227.40:4071 , piece_id=IBCGYA4LAJIRKFVJV2E6T2NPRCVADJIP5SAJ75H4QXDIMOQAN2NA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.108333867 (505h17m7.979416913s) [noise=true , address=77.90.31.16:4758 , piece_id=WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.146456269 (505h17m8.017539314s) [noise=true , address=145.79.227.44:4199 , piece_id=7ZE6F5CL4VK6NKCPLM6H7RLIHLEO2RWEG5EGJFXF7X7Y4CSFX6VQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.167739325 (505h17m8.03882236s) [noise=true , address=77.90.31.18:4828 , piece_id=EBH3I5TADBPDMDRYQSG7NQVEFRJVNACURFHR3UTKQL2ZPE7KAFVA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.292396866 (505h17m8.163479911s) [noise=true , address=77.90.31.13:4644 , piece_id=6QLHFIXW7DIL3XRHVODFU6LPMRW25WSM77LRPYGVGUEN36NUALMA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.475687322 (505h17m8.346770367s) [noise=true , address=77.90.31.14:4685 , piece_id=4XZFFB7E642K5A4IKKN6Y6NJ42POJWUCIK7534PNM5A332YHTABA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.551777060 (505h17m8.422860105s) [noise=true , address=77.90.31.19:4860 , piece_id=P2FISE3LU4Q5ZVOOGYHZAO52MIAE5DYWLFY6CVXSZIUC5ONL7HYQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.678298409 (505h17m8.549381454s) [noise=true , address=77.90.31.11:4578 , piece_id=TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.678323457 (505h17m8.549406502s) [seg_pos={0 0}, upload=443378]: Segment reached optimal threshold of 14 pieces. Dec 26 16:46:08.678549747 (505h17m8.549632793s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1T9uQjynbTYcaixhvHWCKzatRWpFxMonHFi5idiMQtvivupKSu): protocol: write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection; write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection Dec 26 16:46:08.678558464 (505h17m8.549641509s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.678891048 (505h17m8.549974093s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ptydBd7T89noaveRPbmziYKKX9C1eNKaYG8BEfz2Zk4T9XtKz): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.678907879 (505h17m8.549990924s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127aXyQbxPkETL9tjbKQodpMMteTiZjBQ5C4muNnXELbuy6H5TM): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.678919432 (505h17m8.550002467s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.678931354 (505h17m8.550014399s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.679190638 (505h17m8.550273714s) [noise=true , address=64.49.13.140:4508 , piece_id=WKGTL6PEL62NGB4T3SZ4VH4YDG3523UJDLUCFRQRWJXVXDWXOMXA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.679200026 (505h17m8.550283061s) [noise=true , address=64.49.13.140:4508 , piece_id=WKGTL6PEL62NGB4T3SZ4VH4YDG3523UJDLUCFRQRWJXVXDWXOMXA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.679263507 (505h17m8.550346552s) [seg_pos={0 0}, upload=443378]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 16:46:08.679352386 (505h17m8.550435431s) [upload=443378]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:46:08.679359249 (505h17m8.550442294s) [upload=443378]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:46:08.985467612 (505h17m8.856550657s) [upload=443378]: Done uploading segments... Dec 26 16:46:08.985487951 (505h17m8.856570997s) [upload=443378]: Upload finished. err: <nil> Duration: 39.464790959s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)