Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [.XX...XX...XX....XX..XX......X.XXX.....XX.X.XXX.........X..X] duration: 30.003618969s trace: https://jaeger.dev.storj.io/trace/243f66ab99efb3ee FAIL (2025-12-26 18:03:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: UVYN4DXLISSVHNJFRXG6IZYN2LW7V6ZKFQRHNTB24T2UEAG2KW4Q) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: 6NSZ2L7LU5TU3OXOM6G7FVDUBDONHLESTCHJCILFY7WS7DU7YEKA) context deadline exceeded ======LOGS====== Dec 26 18:03:10.947413330 (506h34m10.818496385s) [upload=444504]: Starting upload Dec 26 18:03:10.947509102 (506h34m10.818592137s) [upload=444504]: Done starting upload Dec 26 18:03:10.947593964 (506h34m10.818676999s) [upload=444504]: Uploading segments... Dec 26 18:03:10.947770049 (506h34m10.818853084s) [upload=444504]: Got next segment. Inline: false Dec 26 18:03:10.947835494 (506h34m10.818918539s) [upload=444504]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 18:03:10.947841495 (506h34m10.81892453s) [upload=444504]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 18:03:11.196337715 (506h34m11.06742076s) [seg_pos={0 0}, upload=444504]: Begin upload segment... Dec 26 18:03:11.196399363 (506h34m11.067482398s) [seg_pos={0 0}, upload=444504]: Done begin upload segment. Dec 26 18:03:11.196462302 (506h34m11.067545347s) [noise=true , address=145.79.227.39:4015 , piece_id=3EIJJ2FW3KFAE7OHXOH3M2BARFKUPV6AER472K5F7KHJFLXTG63Q, seg_pos={0 0}, upload=444504]: Uploading piece... Dec 26 18:03:11.196638999 (506h34m11.067722044s) [noise=true , address=145.79.227.4:4186 , piece_id=HHX2ZHYKTMM55GULLNOMQJ2WPZRP444XFAO65HU2RVAYWUCDZZ7A, seg_pos={0 0}, upload=444504]: Uploading piece... Dec 26 18:03:11.197357437 (506h34m11.068440482s) [noise=true , address=45.91.116.9:4264 , piece_id=V6G45QGAWGT3MWIYQGPGHTF5XBCKD4G4G3FBXXFYGOKQBHE33LRQ, seg_pos={0 0}, upload=444504]: Uploading piece... Dec 26 18:03:11.197503895 (506h34m11.06858694s) [noise=true , address=77.90.31.11:4553 , piece_id=W3JRDU7L66LTYHTGHMK3PE554MUYSOIQMSSRBL34K4IPOSAE474A, seg_pos={0 0}, upload=444504]: Uploading piece... Dec 26 18:03:11.198008657 (506h34m11.069091702s) [noise=true , address=145.79.227.49:4339 , piece_id=2M7VCO7OY3MU3WWZNM7FCYQHJMTQO35BGT57SSYDMHJCPX2LZT2A, seg_pos={0 0}, upload=444504]: Uploading piece... Dec 26 18:03:11.198082277 (506h34m11.069165322s) [noise=true , address=145.79.227.41:4086 , piece_id=RAUVGKJR7TNL5YJHPNAUIPI72CX5GCHNBT4HOR3IUB4AKKCLMO5Q, seg_pos={0 0}, upload=444504]: Uploading piece... Dec 26 18:03:11.198472169 (506h34m11.069555214s) [noise=true , address=77.90.31.13:4627 , piece_id=FYB4X6MKKBN3L4RW3UKMR3YAX4WOQN63LQCN2MWHZJG3TVESVRQQ, seg_pos={0 0}, upload=444504]: Uploading piece... Dec 26 18:03:11.198538466 (506h34m11.069621501s) [noise=true , address=145.79.227.14:4839 , piece_id=EF5AB6ZDBM5O5DKMUY3JAG2QC7YT2DW5ODTK2Z7IUSKTBCCTXFSA, seg_pos={0 0}, upload=444504]: Uploading piece... Dec 26 18:03:11.199136184 (506h34m11.070219229s) [noise=true , address=145.79.227.47:4267 , piece_id=KWHHCDIZ6KRBJK335BTJ2XN7DTZVIRF3LSNLRTLZSFKKRRHKPHUA, seg_pos={0 0}, upload=444504]: Uploading piece... Dec 26 18:03:11.199257004 (506h34m11.070340049s) [noise=true , address=145.79.227.15:4874 , piece_id=6JDOK6U5OIFFQTEDA5IVGHFFZVHUXMRF4GTAM5ETPZNYGDIV7LPQ, seg_pos={0 0}, upload=444504]: Uploading piece... Dec 26 18:03:11.199660883 (506h34m11.070743928s) [noise=true , address=45.91.116.13:4156 , piece_id=UVYN4DXLISSVHNJFRXG6IZYN2LW7V6ZKFQRHNTB24T2UEAG2KW4Q, seg_pos={0 0}, upload=444504]: Uploading piece... Dec 26 18:03:11.199676052 (506h34m11.070759097s) [noise=true , address=45.91.116.10:4081 , piece_id=QC77DKAG4LZRC3UFN5UBTFBY7EC7P6Q5RUVITKZNUDVXL4LDTI6A, seg_pos={0 0}, upload=444504]: Uploading piece... Dec 26 18:03:11.200151407 (506h34m11.071234452s) [noise=true , address=145.79.227.45:4229 , piece_id=4QVKJHC6IXIKWJIW6XHTLWJGIHUW5HZ3ORB4B65527NCRNQURUNA, seg_pos={0 0}, upload=444504]: Uploading piece... Dec 26 18:03:11.200184359 (506h34m11.071267404s) [noise=true , address=45.91.116.14:4191 , piece_id=LG4MVMFR5622ZJOOMYG3IPLMMOPSWSSMHWOFCU76WXIAJIW4XUBQ, seg_pos={0 0}, upload=444504]: Uploading piece... Dec 26 18:03:11.200606214 (506h34m11.071689259s) [noise=true , address=45.91.116.8:4045 , piece_id=6NSZ2L7LU5TU3OXOM6G7FVDUBDONHLESTCHJCILFY7WS7DU7YEKA, seg_pos={0 0}, upload=444504]: Uploading piece... Dec 26 18:03:11.201053194 (506h34m11.072136239s) [noise=true , address=145.79.227.46:4601 , piece_id=JFFS2GUNTINTQBO7EEB324PK37ZXJ2ARH6XJ4L53GQV25WYQRAUA, seg_pos={0 0}, upload=444504]: Uploading piece... Dec 26 18:03:12.350290563 (506h34m12.221373608s) [noise=true , address=77.90.31.11:4553 , piece_id=W3JRDU7L66LTYHTGHMK3PE554MUYSOIQMSSRBL34K4IPOSAE474A, seg_pos={0 0}, upload=444504]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: storage node overloaded, request limit: 2000 Dec 26 18:03:12.915003815 (506h34m12.78608685s) [upload=444504]: Next returned nil segment Dec 26 18:03:12.915016951 (506h34m12.786099996s) [upload=444504]: Waiting for error group managing segments... Dec 26 18:03:13.194034659 (506h34m13.065117694s) [noise=true , address=145.79.227.15:4874 , piece_id=6JDOK6U5OIFFQTEDA5IVGHFFZVHUXMRF4GTAM5ETPZNYGDIV7LPQ, seg_pos={0 0}, upload=444504]: Done uploading piece. err: <nil> Dec 26 18:03:13.199551184 (506h34m13.070634219s) [noise=true , address=145.79.227.46:4601 , piece_id=JFFS2GUNTINTQBO7EEB324PK37ZXJ2ARH6XJ4L53GQV25WYQRAUA, seg_pos={0 0}, upload=444504]: Done uploading piece. err: <nil> Dec 26 18:03:13.204584307 (506h34m13.075667352s) [noise=true , address=145.79.227.49:4339 , piece_id=2M7VCO7OY3MU3WWZNM7FCYQHJMTQO35BGT57SSYDMHJCPX2LZT2A, seg_pos={0 0}, upload=444504]: Done uploading piece. err: <nil> Dec 26 18:03:13.206994768 (506h34m13.078077813s) [noise=true , address=145.79.227.39:4015 , piece_id=3EIJJ2FW3KFAE7OHXOH3M2BARFKUPV6AER472K5F7KHJFLXTG63Q, seg_pos={0 0}, upload=444504]: Done uploading piece. err: <nil> Dec 26 18:03:13.219091742 (506h34m13.090174777s) [noise=true , address=145.79.227.47:4267 , piece_id=KWHHCDIZ6KRBJK335BTJ2XN7DTZVIRF3LSNLRTLZSFKKRRHKPHUA, seg_pos={0 0}, upload=444504]: Done uploading piece. err: <nil> Dec 26 18:03:13.220706828 (506h34m13.091789863s) [noise=true , address=145.79.227.45:4229 , piece_id=4QVKJHC6IXIKWJIW6XHTLWJGIHUW5HZ3ORB4B65527NCRNQURUNA, seg_pos={0 0}, upload=444504]: Done uploading piece. err: <nil> Dec 26 18:03:13.230304272 (506h34m13.101387307s) [noise=true , address=145.79.227.14:4839 , piece_id=EF5AB6ZDBM5O5DKMUY3JAG2QC7YT2DW5ODTK2Z7IUSKTBCCTXFSA, seg_pos={0 0}, upload=444504]: Done uploading piece. err: <nil> Dec 26 18:03:13.232301426 (506h34m13.103384461s) [noise=true , address=145.79.227.4:4186 , piece_id=HHX2ZHYKTMM55GULLNOMQJ2WPZRP444XFAO65HU2RVAYWUCDZZ7A, seg_pos={0 0}, upload=444504]: Done uploading piece. err: <nil> Dec 26 18:03:13.439749002 (506h34m13.310832047s) [noise=true , address=45.91.116.9:4264 , piece_id=V6G45QGAWGT3MWIYQGPGHTF5XBCKD4G4G3FBXXFYGOKQBHE33LRQ, seg_pos={0 0}, upload=444504]: Done uploading piece. err: <nil> Dec 26 18:03:13.491708832 (506h34m13.362791878s) [noise=true , address=45.91.116.13:4156 , piece_id=UVYN4DXLISSVHNJFRXG6IZYN2LW7V6ZKFQRHNTB24T2UEAG2KW4Q, seg_pos={0 0}, upload=444504]: Done uploading piece. err: <nil> Dec 26 18:03:13.514759485 (506h34m13.38584253s) [noise=true , address=77.90.31.13:4627 , piece_id=FYB4X6MKKBN3L4RW3UKMR3YAX4WOQN63LQCN2MWHZJG3TVESVRQQ, seg_pos={0 0}, upload=444504]: Done uploading piece. err: <nil> Dec 26 18:03:13.585088304 (506h34m13.456171349s) [noise=true , address=45.91.116.14:4191 , piece_id=LG4MVMFR5622ZJOOMYG3IPLMMOPSWSSMHWOFCU76WXIAJIW4XUBQ, seg_pos={0 0}, upload=444504]: Done uploading piece. err: <nil> Dec 26 18:03:14.293444978 (506h34m14.164528033s) [noise=true , address=45.91.116.8:4045 , piece_id=6NSZ2L7LU5TU3OXOM6G7FVDUBDONHLESTCHJCILFY7WS7DU7YEKA, seg_pos={0 0}, upload=444504]: Done uploading piece. err: <nil> Dec 26 18:03:14.293466870 (506h34m14.164549925s) [seg_pos={0 0}, upload=444504]: Segment reached optimal threshold of 13 pieces. Dec 26 18:03:14.293642154 (506h34m14.164725219s) [noise=true , address=145.79.227.41:4086 , piece_id=RAUVGKJR7TNL5YJHPNAUIPI72CX5GCHNBT4HOR3IUB4AKKCLMO5Q, seg_pos={0 0}, upload=444504]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm): protocol: expected piece hash; context canceled; EOF Dec 26 18:03:14.293652554 (506h34m14.164735589s) [noise=true , address=145.79.227.41:4086 , piece_id=RAUVGKJR7TNL5YJHPNAUIPI72CX5GCHNBT4HOR3IUB4AKKCLMO5Q, seg_pos={0 0}, upload=444504]: Long tail cancellation due to optimal threshold Dec 26 18:03:14.294191048 (506h34m14.165274094s) [noise=true , address=45.91.116.10:4081 , piece_id=QC77DKAG4LZRC3UFN5UBTFBY7EC7P6Q5RUVITKZNUDVXL4LDTI6A, seg_pos={0 0}, upload=444504]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: context canceled; context canceled Dec 26 18:03:14.294210877 (506h34m14.165293922s) [noise=true , address=45.91.116.10:4081 , piece_id=QC77DKAG4LZRC3UFN5UBTFBY7EC7P6Q5RUVITKZNUDVXL4LDTI6A, seg_pos={0 0}, upload=444504]: Long tail cancellation due to optimal threshold Dec 26 18:03:14.294238509 (506h34m14.165321555s) [seg_pos={0 0}, upload=444504]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 18:03:14.294369770 (506h34m14.165452815s) [upload=444504]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 18:03:14.294377755 (506h34m14.1654608s) [upload=444504]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 18:03:14.588297970 (506h34m14.459381005s) [upload=444504]: Done uploading segments... Dec 26 18:03:14.588319040 (506h34m14.459402075s) [upload=444504]: Upload finished. err: <nil> Duration: 30.003642884s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 15.419372427s trace: https://jaeger.dev.storj.io/trace/665ea65f6e9ccc33 FAIL (2025-12-26 18:03:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== e=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789316979 (506h12m54.660400024s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789326507 (506h12m54.660409552s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789611339 (506h12m54.660694384s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789619275 (506h12m54.66070231s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789914006 (506h12m54.660997041s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789949443 (506h12m54.661032478s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790181566 (506h12m54.661264601s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790295632 (506h12m54.661378668s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790510872 (506h12m54.661593917s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790692980 (506h12m54.661776015s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790787379 (506h12m54.661870414s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790992970 (506h12m54.662076005s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791120053 (506h12m54.662203098s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791275218 (506h12m54.662358253s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791443709 (506h12m54.662526744s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:56.112313145 (506h12m55.98339619s) [upload=444191]: Next returned nil segment Dec 26 17:41:56.112330638 (506h12m55.983413683s) [upload=444191]: Waiting for error group managing segments... Dec 26 17:41:56.485322720 (506h12m56.356405765s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.485637079 (506h12m56.356720124s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.666807539 (506h12m56.537890584s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.678753987 (506h12m56.549837032s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.687788039 (506h12m56.558871074s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.694318023 (506h12m56.565401058s) [noise=true , address=145.79.227.7:4326 , piece_id=DMQ5RPI46X2U7TJTS42ZHQEF32BDJRKJJO5HWOYAPNS6VWVG5B5Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.716581187 (506h12m56.587664232s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.972183007 (506h12m56.843266042s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.011917999 (506h12m56.883001034s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.054168622 (506h12m56.925251667s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.114312777 (506h12m56.985395812s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.499846250 (506h12m57.370929285s) [noise=true , address=145.79.227.34:4919 , piece_id=YRWHW3BJVEGVJRSA6RJB3T7VJQSDTGKEZQKRICFIV2G3T7AN4MIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.612649066 (506h12m57.483732101s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649413733 (506h12m57.520496768s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649440282 (506h12m57.520523328s) [seg_pos={0 0}, upload=444191]: Segment reached optimal threshold of 14 pieces. Dec 26 17:41:57.649674729 (506h12m57.520757774s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G7ty71tPwe7cZSBNK9u74DFUCGomnxDsNvAif4C3SbP7HWwfo): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649684087 (506h12m57.520767142s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649747708 (506h12m57.520830763s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v6zXZuMmyL2MU8qb3q6gg1iwTYvndjzMjVpv8btEpeX9Lnqmu): protocol: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection Dec 26 17:41:57.649785219 (506h12m57.520868274s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649899828 (506h12m57.520982903s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649908775 (506h12m57.52099182s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650074641 (506h12m57.521157696s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.650083287 (506h12m57.521166342s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650132821 (506h12m57.521215866s) [seg_pos={0 0}, upload=444191]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 17:41:57.650220138 (506h12m57.521303183s) [upload=444191]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:41:57.650227171 (506h12m57.521310206s) [upload=444191]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:41:57.958890870 (506h12m57.829973915s) [upload=444191]: Done uploading segments... Dec 26 17:41:57.958911620 (506h12m57.829994665s) [upload=444191]: Upload finished. err: <nil> Duration: 15.419397193s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 15.534535388s trace: https://jaeger.dev.storj.io/trace/665ea65f6e9ccc33 FAIL (2025-12-26 18:03:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== e=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789316979 (506h12m54.660400024s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789326507 (506h12m54.660409552s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789611339 (506h12m54.660694384s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789619275 (506h12m54.66070231s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789914006 (506h12m54.660997041s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789949443 (506h12m54.661032478s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790181566 (506h12m54.661264601s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790295632 (506h12m54.661378668s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790510872 (506h12m54.661593917s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790692980 (506h12m54.661776015s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790787379 (506h12m54.661870414s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790992970 (506h12m54.662076005s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791120053 (506h12m54.662203098s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791275218 (506h12m54.662358253s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791443709 (506h12m54.662526744s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:56.112313145 (506h12m55.98339619s) [upload=444191]: Next returned nil segment Dec 26 17:41:56.112330638 (506h12m55.983413683s) [upload=444191]: Waiting for error group managing segments... Dec 26 17:41:56.485322720 (506h12m56.356405765s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.485637079 (506h12m56.356720124s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.666807539 (506h12m56.537890584s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.678753987 (506h12m56.549837032s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.687788039 (506h12m56.558871074s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.694318023 (506h12m56.565401058s) [noise=true , address=145.79.227.7:4326 , piece_id=DMQ5RPI46X2U7TJTS42ZHQEF32BDJRKJJO5HWOYAPNS6VWVG5B5Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.716581187 (506h12m56.587664232s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.972183007 (506h12m56.843266042s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.011917999 (506h12m56.883001034s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.054168622 (506h12m56.925251667s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.114312777 (506h12m56.985395812s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.499846250 (506h12m57.370929285s) [noise=true , address=145.79.227.34:4919 , piece_id=YRWHW3BJVEGVJRSA6RJB3T7VJQSDTGKEZQKRICFIV2G3T7AN4MIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.612649066 (506h12m57.483732101s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649413733 (506h12m57.520496768s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649440282 (506h12m57.520523328s) [seg_pos={0 0}, upload=444191]: Segment reached optimal threshold of 14 pieces. Dec 26 17:41:57.649674729 (506h12m57.520757774s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G7ty71tPwe7cZSBNK9u74DFUCGomnxDsNvAif4C3SbP7HWwfo): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649684087 (506h12m57.520767142s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649747708 (506h12m57.520830763s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v6zXZuMmyL2MU8qb3q6gg1iwTYvndjzMjVpv8btEpeX9Lnqmu): protocol: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection Dec 26 17:41:57.649785219 (506h12m57.520868274s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649899828 (506h12m57.520982903s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649908775 (506h12m57.52099182s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650074641 (506h12m57.521157696s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.650083287 (506h12m57.521166342s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650132821 (506h12m57.521215866s) [seg_pos={0 0}, upload=444191]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 17:41:57.650220138 (506h12m57.521303183s) [upload=444191]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:41:57.650227171 (506h12m57.521310206s) [upload=444191]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:41:57.958890870 (506h12m57.829973915s) [upload=444191]: Done uploading segments... Dec 26 17:41:57.958911620 (506h12m57.829994665s) [upload=444191]: Upload finished. err: <nil> Duration: 15.534587367s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [X.XX...XX...XX....XX..XX......X.XXX.....XX.X.XXX.........X..] duration: 5.72947971s trace: https://jaeger.dev.storj.io/trace/6c2ceda91b8dc6ef FAIL (2025-12-26 18:02:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: MGFBY7Y657TW2OEYGUEMDUVN6DYYWK5GTHQCGVACSYY33HMMCHJQ) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: JV6GLRMSHCB345DOFAMII6KWS74WJDZT3IGPLGMRZM2NUWYOB5PQ) context deadline exceeded ======LOGS====== 6 18:01:44.349873787 (506h32m44.220956832s) [upload=444482]: Starting upload Dec 26 18:01:44.349943950 (506h32m44.221026995s) [upload=444482]: Done starting upload Dec 26 18:01:44.350214976 (506h32m44.221298012s) [upload=444482]: Uploading segments... Dec 26 18:01:44.350283717 (506h32m44.221366773s) [upload=444482]: Got next segment. Inline: false Dec 26 18:01:44.350333432 (506h32m44.221416477s) [upload=444482]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 18:01:44.350339253 (506h32m44.221422298s) [upload=444482]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 18:01:44.601277424 (506h32m44.472360479s) [seg_pos={0 0}, upload=444482]: Begin upload segment... Dec 26 18:01:44.601356514 (506h32m44.472439559s) [seg_pos={0 0}, upload=444482]: Done begin upload segment. Dec 26 18:01:44.601440984 (506h32m44.472524019s) [noise=true , address=145.79.227.39:4013 , piece_id=EZHHDHZD3USFUSURBFORTHKGONZ6YJ2BCRUCUUKT4HUVSCL2LPTQ, seg_pos={0 0}, upload=444482]: Uploading piece... Dec 26 18:01:44.601479578 (506h32m44.472562613s) [noise=true , address=145.79.227.45:4231 , piece_id=XFMDANT7ZCIGASWV56DKO3OPAFAQNEJTRGEI563O37EO4YZSBELA, seg_pos={0 0}, upload=444482]: Uploading piece... Dec 26 18:01:44.602076505 (506h32m44.47315954s) [noise=true , address=45.91.116.14:4189 , piece_id=KUVACSJCPXTLHDNOL26KBRJA7KZMYXOFRU6S7ABZLCRYOOOMZD3Q, seg_pos={0 0}, upload=444482]: Uploading piece... Dec 26 18:01:44.602477328 (506h32m44.473560383s) [noise=true , address=145.79.227.51:4411 , piece_id=YKV7HVSLN4L7TPCJ3TPAKGMZ2ZRZXKC3B7NDJNBOFAQGMF4DDTRQ, seg_pos={0 0}, upload=444482]: Uploading piece... Dec 26 18:01:44.602736631 (506h32m44.473819666s) [noise=true , address=145.79.227.10:4401 , piece_id=XCYA3E6OVOJETWBYL34FP2NGVN47K75ZZTJRXPPH6NOKQ4XMJB7Q, seg_pos={0 0}, upload=444482]: Uploading piece... Dec 26 18:01:44.602955559 (506h32m44.474038604s) [noise=true , address=45.91.116.9:4263 , piece_id=MGFBY7Y657TW2OEYGUEMDUVN6DYYWK5GTHQCGVACSYY33HMMCHJQ, seg_pos={0 0}, upload=444482]: Uploading piece... Dec 26 18:01:44.603278083 (506h32m44.474361118s) [noise=true , address=77.90.31.19:4844 , piece_id=QC3UCTWSJDI2MDBDO3XWWFYRXL2APO7OIHY4BR7PAJSXPOZZYIDA, seg_pos={0 0}, upload=444482]: Uploading piece... Dec 26 18:01:44.603350230 (506h32m44.474433275s) [noise=true , address=145.79.227.8:4330 , piece_id=3YMCHIGG6LOX47OQ265IL4X6FJSK6RMZAWKMDF5C2LX7RDUBEINA, seg_pos={0 0}, upload=444482]: Uploading piece... Dec 26 18:01:44.603707049 (506h32m44.474790094s) [noise=true , address=77.90.31.14:4663 , piece_id=SESXAVIAYHPZ5KIQFD6QWNTI6NXKKCMSSYN6ZXDM2DUDN3CY7SIA, seg_pos={0 0}, upload=444482]: Uploading piece... Dec 26 18:01:44.604197894 (506h32m44.475280929s) [noise=true , address=77.90.31.10:4519 , piece_id=3WLJHO37GY3JLLHFJ2UDYXMOVR6VQ6T3U52YYPMJDAHHCE5LOPAA, seg_pos={0 0}, upload=444482]: Uploading piece... Dec 26 18:01:44.604605240 (506h32m44.475688275s) [noise=true , address=45.91.116.4:4937 , piece_id=GSM7OS24YKFP3VIINOAEGGX4EOBL2VQRDD6YKGU5T5GKXVNZ74HA, seg_pos={0 0}, upload=444482]: Uploading piece... Dec 26 18:01:44.604981677 (506h32m44.476064722s) [noise=true , address=145.79.227.11:4732 , piece_id=JGKV3PVFUSTOEK6VX3IX7J46TIJA22ZPDZNEBCKD6RI2FP45ZEGA, seg_pos={0 0}, upload=444482]: Uploading piece... Dec 26 18:01:44.605299181 (506h32m44.476382226s) [noise=true , address=45.91.116.6:4975 , piece_id=JV6GLRMSHCB345DOFAMII6KWS74WJDZT3IGPLGMRZM2NUWYOB5PQ, seg_pos={0 0}, upload=444482]: Uploading piece... Dec 26 18:01:44.605492490 (506h32m44.476575535s) [noise=true , address=145.79.227.31:4799 , piece_id=WEVOSTZDZUXV2WQ54RBGQHHG4GWVWNVS6JNPWPCCBO5RSLWX72OQ, seg_pos={0 0}, upload=444482]: Uploading piece... Dec 26 18:01:44.606080369 (506h32m44.477163414s) [noise=true , address=145.79.227.21:4474 , piece_id=HELCVB43WKB3ZDONRBS6BR3DMTYO76AWYORTIQ2NCV7SHI4KAMRQ, seg_pos={0 0}, upload=444482]: Uploading piece... Dec 26 18:01:44.606580110 (506h32m44.477663156s) [noise=true , address=145.79.227.4:4186 , piece_id=RWWTBRHMNLLAXYQBNHBGSCAPVNQVW4GBGNO3CKREZZU5WLOR3T7A, seg_pos={0 0}, upload=444482]: Uploading piece... Dec 26 18:01:46.319626871 (506h32m46.190709906s) [upload=444482]: Next returned nil segment Dec 26 18:01:46.319642711 (506h32m46.190725746s) [upload=444482]: Waiting for error group managing segments... Dec 26 18:01:46.607291835 (506h32m46.47837488s) [noise=true , address=145.79.227.10:4401 , piece_id=XCYA3E6OVOJETWBYL34FP2NGVN47K75ZZTJRXPPH6NOKQ4XMJB7Q, seg_pos={0 0}, upload=444482]: Done uploading piece. err: <nil> Dec 26 18:01:46.610693453 (506h32m46.481776488s) [noise=true , address=145.79.227.39:4013 , piece_id=EZHHDHZD3USFUSURBFORTHKGONZ6YJ2BCRUCUUKT4HUVSCL2LPTQ, seg_pos={0 0}, upload=444482]: Done uploading piece. err: <nil> Dec 26 18:01:46.611012420 (506h32m46.482095465s) [noise=true , address=145.79.227.8:4330 , piece_id=3YMCHIGG6LOX47OQ265IL4X6FJSK6RMZAWKMDF5C2LX7RDUBEINA, seg_pos={0 0}, upload=444482]: Done uploading piece. err: <nil> Dec 26 18:01:46.621508626 (506h32m46.492591671s) [noise=true , address=145.79.227.31:4799 , piece_id=WEVOSTZDZUXV2WQ54RBGQHHG4GWVWNVS6JNPWPCCBO5RSLWX72OQ, seg_pos={0 0}, upload=444482]: Done uploading piece. err: <nil> Dec 26 18:01:46.625365981 (506h32m46.496449016s) [noise=true , address=145.79.227.45:4231 , piece_id=XFMDANT7ZCIGASWV56DKO3OPAFAQNEJTRGEI563O37EO4YZSBELA, seg_pos={0 0}, upload=444482]: Done uploading piece. err: <nil> Dec 26 18:01:46.629445088 (506h32m46.500528133s) [noise=true , address=145.79.227.51:4411 , piece_id=YKV7HVSLN4L7TPCJ3TPAKGMZ2ZRZXKC3B7NDJNBOFAQGMF4DDTRQ, seg_pos={0 0}, upload=444482]: Done uploading piece. err: <nil> Dec 26 18:01:46.640495721 (506h32m46.511578756s) [noise=true , address=145.79.227.4:4186 , piece_id=RWWTBRHMNLLAXYQBNHBGSCAPVNQVW4GBGNO3CKREZZU5WLOR3T7A, seg_pos={0 0}, upload=444482]: Done uploading piece. err: <nil> Dec 26 18:01:46.645928436 (506h32m46.517011481s) [noise=true , address=145.79.227.21:4474 , piece_id=HELCVB43WKB3ZDONRBS6BR3DMTYO76AWYORTIQ2NCV7SHI4KAMRQ, seg_pos={0 0}, upload=444482]: Done uploading piece. err: <nil> Dec 26 18:01:46.654201760 (506h32m46.525284795s) [noise=true , address=145.79.227.11:4732 , piece_id=JGKV3PVFUSTOEK6VX3IX7J46TIJA22ZPDZNEBCKD6RI2FP45ZEGA, seg_pos={0 0}, upload=444482]: Done uploading piece. err: <nil> Dec 26 18:01:46.850051632 (506h32m46.721134677s) [noise=true , address=45.91.116.9:4263 , piece_id=MGFBY7Y657TW2OEYGUEMDUVN6DYYWK5GTHQCGVACSYY33HMMCHJQ, seg_pos={0 0}, upload=444482]: Done uploading piece. err: <nil> Dec 26 18:01:46.892001602 (506h32m46.763084637s) [noise=true , address=45.91.116.6:4975 , piece_id=JV6GLRMSHCB345DOFAMII6KWS74WJDZT3IGPLGMRZM2NUWYOB5PQ, seg_pos={0 0}, upload=444482]: Done uploading piece. err: <nil> Dec 26 18:01:46.896117650 (506h32m46.767200695s) [noise=true , address=45.91.116.14:4189 , piece_id=KUVACSJCPXTLHDNOL26KBRJA7KZMYXOFRU6S7ABZLCRYOOOMZD3Q, seg_pos={0 0}, upload=444482]: Done uploading piece. err: <nil> Dec 26 18:01:46.908261072 (506h32m46.779344118s) [noise=true , address=77.90.31.14:4663 , piece_id=SESXAVIAYHPZ5KIQFD6QWNTI6NXKKCMSSYN6ZXDM2DUDN3CY7SIA, seg_pos={0 0}, upload=444482]: Done uploading piece. err: <nil> Dec 26 18:01:46.908285720 (506h32m46.779368755s) [seg_pos={0 0}, upload=444482]: Segment reached optimal threshold of 13 pieces. Dec 26 18:01:46.908564050 (506h32m46.779647085s) [noise=true , address=77.90.31.19:4844 , piece_id=QC3UCTWSJDI2MDBDO3XWWFYRXL2APO7OIHY4BR7PAJSXPOZZYIDA, seg_pos={0 0}, upload=444482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: expected piece hash; context canceled; EOF Dec 26 18:01:46.908575322 (506h32m46.779658367s) [noise=true , address=77.90.31.19:4844 , piece_id=QC3UCTWSJDI2MDBDO3XWWFYRXL2APO7OIHY4BR7PAJSXPOZZYIDA, seg_pos={0 0}, upload=444482]: Long tail cancellation due to optimal threshold Dec 26 18:01:46.911452860 (506h32m46.782535926s) [noise=true , address=77.90.31.10:4519 , piece_id=3WLJHO37GY3JLLHFJ2UDYXMOVR6VQ6T3U52YYPMJDAHHCE5LOPAA, seg_pos={0 0}, upload=444482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: write tcp 188.166.225.170:50298->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:50298->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50298->77.90.31.10:4519: use of closed network connection Dec 26 18:01:46.911470074 (506h32m46.782553129s) [noise=true , address=77.90.31.10:4519 , piece_id=3WLJHO37GY3JLLHFJ2UDYXMOVR6VQ6T3U52YYPMJDAHHCE5LOPAA, seg_pos={0 0}, upload=444482]: Long tail cancellation due to optimal threshold Dec 26 18:01:46.911673732 (506h32m46.782756777s) [noise=true , address=45.91.116.4:4937 , piece_id=GSM7OS24YKFP3VIINOAEGGX4EOBL2VQRDD6YKGU5T5GKXVNZ74HA, seg_pos={0 0}, upload=444482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 26 18:01:46.911683430 (506h32m46.782766475s) [noise=true , address=45.91.116.4:4937 , piece_id=GSM7OS24YKFP3VIINOAEGGX4EOBL2VQRDD6YKGU5T5GKXVNZ74HA, seg_pos={0 0}, upload=444482]: Long tail cancellation due to optimal threshold Dec 26 18:01:46.911707225 (506h32m46.78279026s) [seg_pos={0 0}, upload=444482]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 18:01:46.911789291 (506h32m46.782872337s) [upload=444482]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 18:01:46.911795543 (506h32m46.782878588s) [upload=444482]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 18:01:47.214392370 (506h32m47.085475415s) [upload=444482]: Done uploading segments... Dec 26 18:01:47.214413361 (506h32m47.085496406s) [upload=444482]: Upload finished. err: <nil> Duration: 5.729504196s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [.X.XX...XX...XX....XX..XX......X.XXX.....XX.X.XXX.........X.] duration: 6.043270952s trace: https://jaeger.dev.storj.io/trace/6c2ceda91b8dc6ef FAIL (2025-12-26 18:02:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: MGFBY7Y657TW2OEYGUEMDUVN6DYYWK5GTHQCGVACSYY33HMMCHJQ) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: JV6GLRMSHCB345DOFAMII6KWS74WJDZT3IGPLGMRZM2NUWYOB5PQ) context deadline exceeded ======LOGS====== 6 18:01:44.349873787 (506h32m44.220956832s) [upload=444482]: Starting upload Dec 26 18:01:44.349943950 (506h32m44.221026995s) [upload=444482]: Done starting upload Dec 26 18:01:44.350214976 (506h32m44.221298012s) [upload=444482]: Uploading segments... Dec 26 18:01:44.350283717 (506h32m44.221366773s) [upload=444482]: Got next segment. Inline: false Dec 26 18:01:44.350333432 (506h32m44.221416477s) [upload=444482]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 18:01:44.350339253 (506h32m44.221422298s) [upload=444482]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 18:01:44.601277424 (506h32m44.472360479s) [seg_pos={0 0}, upload=444482]: Begin upload segment... Dec 26 18:01:44.601356514 (506h32m44.472439559s) [seg_pos={0 0}, upload=444482]: Done begin upload segment. Dec 26 18:01:44.601440984 (506h32m44.472524019s) [noise=true , address=145.79.227.39:4013 , piece_id=EZHHDHZD3USFUSURBFORTHKGONZ6YJ2BCRUCUUKT4HUVSCL2LPTQ, seg_pos={0 0}, upload=444482]: Uploading piece... Dec 26 18:01:44.601479578 (506h32m44.472562613s) [noise=true , address=145.79.227.45:4231 , piece_id=XFMDANT7ZCIGASWV56DKO3OPAFAQNEJTRGEI563O37EO4YZSBELA, seg_pos={0 0}, upload=444482]: Uploading piece... Dec 26 18:01:44.602076505 (506h32m44.47315954s) [noise=true , address=45.91.116.14:4189 , piece_id=KUVACSJCPXTLHDNOL26KBRJA7KZMYXOFRU6S7ABZLCRYOOOMZD3Q, seg_pos={0 0}, upload=444482]: Uploading piece... Dec 26 18:01:44.602477328 (506h32m44.473560383s) [noise=true , address=145.79.227.51:4411 , piece_id=YKV7HVSLN4L7TPCJ3TPAKGMZ2ZRZXKC3B7NDJNBOFAQGMF4DDTRQ, seg_pos={0 0}, upload=444482]: Uploading piece... Dec 26 18:01:44.602736631 (506h32m44.473819666s) [noise=true , address=145.79.227.10:4401 , piece_id=XCYA3E6OVOJETWBYL34FP2NGVN47K75ZZTJRXPPH6NOKQ4XMJB7Q, seg_pos={0 0}, upload=444482]: Uploading piece... Dec 26 18:01:44.602955559 (506h32m44.474038604s) [noise=true , address=45.91.116.9:4263 , piece_id=MGFBY7Y657TW2OEYGUEMDUVN6DYYWK5GTHQCGVACSYY33HMMCHJQ, seg_pos={0 0}, upload=444482]: Uploading piece... Dec 26 18:01:44.603278083 (506h32m44.474361118s) [noise=true , address=77.90.31.19:4844 , piece_id=QC3UCTWSJDI2MDBDO3XWWFYRXL2APO7OIHY4BR7PAJSXPOZZYIDA, seg_pos={0 0}, upload=444482]: Uploading piece... Dec 26 18:01:44.603350230 (506h32m44.474433275s) [noise=true , address=145.79.227.8:4330 , piece_id=3YMCHIGG6LOX47OQ265IL4X6FJSK6RMZAWKMDF5C2LX7RDUBEINA, seg_pos={0 0}, upload=444482]: Uploading piece... Dec 26 18:01:44.603707049 (506h32m44.474790094s) [noise=true , address=77.90.31.14:4663 , piece_id=SESXAVIAYHPZ5KIQFD6QWNTI6NXKKCMSSYN6ZXDM2DUDN3CY7SIA, seg_pos={0 0}, upload=444482]: Uploading piece... Dec 26 18:01:44.604197894 (506h32m44.475280929s) [noise=true , address=77.90.31.10:4519 , piece_id=3WLJHO37GY3JLLHFJ2UDYXMOVR6VQ6T3U52YYPMJDAHHCE5LOPAA, seg_pos={0 0}, upload=444482]: Uploading piece... Dec 26 18:01:44.604605240 (506h32m44.475688275s) [noise=true , address=45.91.116.4:4937 , piece_id=GSM7OS24YKFP3VIINOAEGGX4EOBL2VQRDD6YKGU5T5GKXVNZ74HA, seg_pos={0 0}, upload=444482]: Uploading piece... Dec 26 18:01:44.604981677 (506h32m44.476064722s) [noise=true , address=145.79.227.11:4732 , piece_id=JGKV3PVFUSTOEK6VX3IX7J46TIJA22ZPDZNEBCKD6RI2FP45ZEGA, seg_pos={0 0}, upload=444482]: Uploading piece... Dec 26 18:01:44.605299181 (506h32m44.476382226s) [noise=true , address=45.91.116.6:4975 , piece_id=JV6GLRMSHCB345DOFAMII6KWS74WJDZT3IGPLGMRZM2NUWYOB5PQ, seg_pos={0 0}, upload=444482]: Uploading piece... Dec 26 18:01:44.605492490 (506h32m44.476575535s) [noise=true , address=145.79.227.31:4799 , piece_id=WEVOSTZDZUXV2WQ54RBGQHHG4GWVWNVS6JNPWPCCBO5RSLWX72OQ, seg_pos={0 0}, upload=444482]: Uploading piece... Dec 26 18:01:44.606080369 (506h32m44.477163414s) [noise=true , address=145.79.227.21:4474 , piece_id=HELCVB43WKB3ZDONRBS6BR3DMTYO76AWYORTIQ2NCV7SHI4KAMRQ, seg_pos={0 0}, upload=444482]: Uploading piece... Dec 26 18:01:44.606580110 (506h32m44.477663156s) [noise=true , address=145.79.227.4:4186 , piece_id=RWWTBRHMNLLAXYQBNHBGSCAPVNQVW4GBGNO3CKREZZU5WLOR3T7A, seg_pos={0 0}, upload=444482]: Uploading piece... Dec 26 18:01:46.319626871 (506h32m46.190709906s) [upload=444482]: Next returned nil segment Dec 26 18:01:46.319642711 (506h32m46.190725746s) [upload=444482]: Waiting for error group managing segments... Dec 26 18:01:46.607291835 (506h32m46.47837488s) [noise=true , address=145.79.227.10:4401 , piece_id=XCYA3E6OVOJETWBYL34FP2NGVN47K75ZZTJRXPPH6NOKQ4XMJB7Q, seg_pos={0 0}, upload=444482]: Done uploading piece. err: <nil> Dec 26 18:01:46.610693453 (506h32m46.481776488s) [noise=true , address=145.79.227.39:4013 , piece_id=EZHHDHZD3USFUSURBFORTHKGONZ6YJ2BCRUCUUKT4HUVSCL2LPTQ, seg_pos={0 0}, upload=444482]: Done uploading piece. err: <nil> Dec 26 18:01:46.611012420 (506h32m46.482095465s) [noise=true , address=145.79.227.8:4330 , piece_id=3YMCHIGG6LOX47OQ265IL4X6FJSK6RMZAWKMDF5C2LX7RDUBEINA, seg_pos={0 0}, upload=444482]: Done uploading piece. err: <nil> Dec 26 18:01:46.621508626 (506h32m46.492591671s) [noise=true , address=145.79.227.31:4799 , piece_id=WEVOSTZDZUXV2WQ54RBGQHHG4GWVWNVS6JNPWPCCBO5RSLWX72OQ, seg_pos={0 0}, upload=444482]: Done uploading piece. err: <nil> Dec 26 18:01:46.625365981 (506h32m46.496449016s) [noise=true , address=145.79.227.45:4231 , piece_id=XFMDANT7ZCIGASWV56DKO3OPAFAQNEJTRGEI563O37EO4YZSBELA, seg_pos={0 0}, upload=444482]: Done uploading piece. err: <nil> Dec 26 18:01:46.629445088 (506h32m46.500528133s) [noise=true , address=145.79.227.51:4411 , piece_id=YKV7HVSLN4L7TPCJ3TPAKGMZ2ZRZXKC3B7NDJNBOFAQGMF4DDTRQ, seg_pos={0 0}, upload=444482]: Done uploading piece. err: <nil> Dec 26 18:01:46.640495721 (506h32m46.511578756s) [noise=true , address=145.79.227.4:4186 , piece_id=RWWTBRHMNLLAXYQBNHBGSCAPVNQVW4GBGNO3CKREZZU5WLOR3T7A, seg_pos={0 0}, upload=444482]: Done uploading piece. err: <nil> Dec 26 18:01:46.645928436 (506h32m46.517011481s) [noise=true , address=145.79.227.21:4474 , piece_id=HELCVB43WKB3ZDONRBS6BR3DMTYO76AWYORTIQ2NCV7SHI4KAMRQ, seg_pos={0 0}, upload=444482]: Done uploading piece. err: <nil> Dec 26 18:01:46.654201760 (506h32m46.525284795s) [noise=true , address=145.79.227.11:4732 , piece_id=JGKV3PVFUSTOEK6VX3IX7J46TIJA22ZPDZNEBCKD6RI2FP45ZEGA, seg_pos={0 0}, upload=444482]: Done uploading piece. err: <nil> Dec 26 18:01:46.850051632 (506h32m46.721134677s) [noise=true , address=45.91.116.9:4263 , piece_id=MGFBY7Y657TW2OEYGUEMDUVN6DYYWK5GTHQCGVACSYY33HMMCHJQ, seg_pos={0 0}, upload=444482]: Done uploading piece. err: <nil> Dec 26 18:01:46.892001602 (506h32m46.763084637s) [noise=true , address=45.91.116.6:4975 , piece_id=JV6GLRMSHCB345DOFAMII6KWS74WJDZT3IGPLGMRZM2NUWYOB5PQ, seg_pos={0 0}, upload=444482]: Done uploading piece. err: <nil> Dec 26 18:01:46.896117650 (506h32m46.767200695s) [noise=true , address=45.91.116.14:4189 , piece_id=KUVACSJCPXTLHDNOL26KBRJA7KZMYXOFRU6S7ABZLCRYOOOMZD3Q, seg_pos={0 0}, upload=444482]: Done uploading piece. err: <nil> Dec 26 18:01:46.908261072 (506h32m46.779344118s) [noise=true , address=77.90.31.14:4663 , piece_id=SESXAVIAYHPZ5KIQFD6QWNTI6NXKKCMSSYN6ZXDM2DUDN3CY7SIA, seg_pos={0 0}, upload=444482]: Done uploading piece. err: <nil> Dec 26 18:01:46.908285720 (506h32m46.779368755s) [seg_pos={0 0}, upload=444482]: Segment reached optimal threshold of 13 pieces. Dec 26 18:01:46.908564050 (506h32m46.779647085s) [noise=true , address=77.90.31.19:4844 , piece_id=QC3UCTWSJDI2MDBDO3XWWFYRXL2APO7OIHY4BR7PAJSXPOZZYIDA, seg_pos={0 0}, upload=444482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: expected piece hash; context canceled; EOF Dec 26 18:01:46.908575322 (506h32m46.779658367s) [noise=true , address=77.90.31.19:4844 , piece_id=QC3UCTWSJDI2MDBDO3XWWFYRXL2APO7OIHY4BR7PAJSXPOZZYIDA, seg_pos={0 0}, upload=444482]: Long tail cancellation due to optimal threshold Dec 26 18:01:46.911452860 (506h32m46.782535926s) [noise=true , address=77.90.31.10:4519 , piece_id=3WLJHO37GY3JLLHFJ2UDYXMOVR6VQ6T3U52YYPMJDAHHCE5LOPAA, seg_pos={0 0}, upload=444482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: write tcp 188.166.225.170:50298->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:50298->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50298->77.90.31.10:4519: use of closed network connection Dec 26 18:01:46.911470074 (506h32m46.782553129s) [noise=true , address=77.90.31.10:4519 , piece_id=3WLJHO37GY3JLLHFJ2UDYXMOVR6VQ6T3U52YYPMJDAHHCE5LOPAA, seg_pos={0 0}, upload=444482]: Long tail cancellation due to optimal threshold Dec 26 18:01:46.911673732 (506h32m46.782756777s) [noise=true , address=45.91.116.4:4937 , piece_id=GSM7OS24YKFP3VIINOAEGGX4EOBL2VQRDD6YKGU5T5GKXVNZ74HA, seg_pos={0 0}, upload=444482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 26 18:01:46.911683430 (506h32m46.782766475s) [noise=true , address=45.91.116.4:4937 , piece_id=GSM7OS24YKFP3VIINOAEGGX4EOBL2VQRDD6YKGU5T5GKXVNZ74HA, seg_pos={0 0}, upload=444482]: Long tail cancellation due to optimal threshold Dec 26 18:01:46.911707225 (506h32m46.78279026s) [seg_pos={0 0}, upload=444482]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 18:01:46.911789291 (506h32m46.782872337s) [upload=444482]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 18:01:46.911795543 (506h32m46.782878588s) [upload=444482]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 18:01:47.214392370 (506h32m47.085475415s) [upload=444482]: Done uploading segments... Dec 26 18:01:47.214413361 (506h32m47.085496406s) [upload=444482]: Upload finished. err: <nil> Duration: 6.043302452s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 20.419914382s trace: https://jaeger.dev.storj.io/trace/665ea65f6e9ccc33 FAIL (2025-12-26 18:02:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== e=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789316979 (506h12m54.660400024s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789326507 (506h12m54.660409552s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789611339 (506h12m54.660694384s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789619275 (506h12m54.66070231s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789914006 (506h12m54.660997041s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789949443 (506h12m54.661032478s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790181566 (506h12m54.661264601s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790295632 (506h12m54.661378668s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790510872 (506h12m54.661593917s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790692980 (506h12m54.661776015s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790787379 (506h12m54.661870414s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790992970 (506h12m54.662076005s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791120053 (506h12m54.662203098s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791275218 (506h12m54.662358253s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791443709 (506h12m54.662526744s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:56.112313145 (506h12m55.98339619s) [upload=444191]: Next returned nil segment Dec 26 17:41:56.112330638 (506h12m55.983413683s) [upload=444191]: Waiting for error group managing segments... Dec 26 17:41:56.485322720 (506h12m56.356405765s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.485637079 (506h12m56.356720124s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.666807539 (506h12m56.537890584s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.678753987 (506h12m56.549837032s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.687788039 (506h12m56.558871074s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.694318023 (506h12m56.565401058s) [noise=true , address=145.79.227.7:4326 , piece_id=DMQ5RPI46X2U7TJTS42ZHQEF32BDJRKJJO5HWOYAPNS6VWVG5B5Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.716581187 (506h12m56.587664232s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.972183007 (506h12m56.843266042s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.011917999 (506h12m56.883001034s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.054168622 (506h12m56.925251667s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.114312777 (506h12m56.985395812s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.499846250 (506h12m57.370929285s) [noise=true , address=145.79.227.34:4919 , piece_id=YRWHW3BJVEGVJRSA6RJB3T7VJQSDTGKEZQKRICFIV2G3T7AN4MIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.612649066 (506h12m57.483732101s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649413733 (506h12m57.520496768s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649440282 (506h12m57.520523328s) [seg_pos={0 0}, upload=444191]: Segment reached optimal threshold of 14 pieces. Dec 26 17:41:57.649674729 (506h12m57.520757774s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G7ty71tPwe7cZSBNK9u74DFUCGomnxDsNvAif4C3SbP7HWwfo): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649684087 (506h12m57.520767142s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649747708 (506h12m57.520830763s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v6zXZuMmyL2MU8qb3q6gg1iwTYvndjzMjVpv8btEpeX9Lnqmu): protocol: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection Dec 26 17:41:57.649785219 (506h12m57.520868274s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649899828 (506h12m57.520982903s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649908775 (506h12m57.52099182s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650074641 (506h12m57.521157696s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.650083287 (506h12m57.521166342s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650132821 (506h12m57.521215866s) [seg_pos={0 0}, upload=444191]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 17:41:57.650220138 (506h12m57.521303183s) [upload=444191]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:41:57.650227171 (506h12m57.521310206s) [upload=444191]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:41:57.958890870 (506h12m57.829973915s) [upload=444191]: Done uploading segments... Dec 26 17:41:57.958911620 (506h12m57.829994665s) [upload=444191]: Upload finished. err: <nil> Duration: 20.419936203s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [X.X.XX...XX...XX....XX..XX......X.XXX.....XX.X.XXX.........X] duration: 30.003482929s trace: https://jaeger.dev.storj.io/trace/6c2ceda91b8dc6ef FAIL (2025-12-26 18:02:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: MGFBY7Y657TW2OEYGUEMDUVN6DYYWK5GTHQCGVACSYY33HMMCHJQ) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: JV6GLRMSHCB345DOFAMII6KWS74WJDZT3IGPLGMRZM2NUWYOB5PQ) context deadline exceeded ======LOGS====== 6 18:01:44.349873787 (506h32m44.220956832s) [upload=444482]: Starting upload Dec 26 18:01:44.349943950 (506h32m44.221026995s) [upload=444482]: Done starting upload Dec 26 18:01:44.350214976 (506h32m44.221298012s) [upload=444482]: Uploading segments... Dec 26 18:01:44.350283717 (506h32m44.221366773s) [upload=444482]: Got next segment. Inline: false Dec 26 18:01:44.350333432 (506h32m44.221416477s) [upload=444482]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 18:01:44.350339253 (506h32m44.221422298s) [upload=444482]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 18:01:44.601277424 (506h32m44.472360479s) [seg_pos={0 0}, upload=444482]: Begin upload segment... Dec 26 18:01:44.601356514 (506h32m44.472439559s) [seg_pos={0 0}, upload=444482]: Done begin upload segment. Dec 26 18:01:44.601440984 (506h32m44.472524019s) [noise=true , address=145.79.227.39:4013 , piece_id=EZHHDHZD3USFUSURBFORTHKGONZ6YJ2BCRUCUUKT4HUVSCL2LPTQ, seg_pos={0 0}, upload=444482]: Uploading piece... Dec 26 18:01:44.601479578 (506h32m44.472562613s) [noise=true , address=145.79.227.45:4231 , piece_id=XFMDANT7ZCIGASWV56DKO3OPAFAQNEJTRGEI563O37EO4YZSBELA, seg_pos={0 0}, upload=444482]: Uploading piece... Dec 26 18:01:44.602076505 (506h32m44.47315954s) [noise=true , address=45.91.116.14:4189 , piece_id=KUVACSJCPXTLHDNOL26KBRJA7KZMYXOFRU6S7ABZLCRYOOOMZD3Q, seg_pos={0 0}, upload=444482]: Uploading piece... Dec 26 18:01:44.602477328 (506h32m44.473560383s) [noise=true , address=145.79.227.51:4411 , piece_id=YKV7HVSLN4L7TPCJ3TPAKGMZ2ZRZXKC3B7NDJNBOFAQGMF4DDTRQ, seg_pos={0 0}, upload=444482]: Uploading piece... Dec 26 18:01:44.602736631 (506h32m44.473819666s) [noise=true , address=145.79.227.10:4401 , piece_id=XCYA3E6OVOJETWBYL34FP2NGVN47K75ZZTJRXPPH6NOKQ4XMJB7Q, seg_pos={0 0}, upload=444482]: Uploading piece... Dec 26 18:01:44.602955559 (506h32m44.474038604s) [noise=true , address=45.91.116.9:4263 , piece_id=MGFBY7Y657TW2OEYGUEMDUVN6DYYWK5GTHQCGVACSYY33HMMCHJQ, seg_pos={0 0}, upload=444482]: Uploading piece... Dec 26 18:01:44.603278083 (506h32m44.474361118s) [noise=true , address=77.90.31.19:4844 , piece_id=QC3UCTWSJDI2MDBDO3XWWFYRXL2APO7OIHY4BR7PAJSXPOZZYIDA, seg_pos={0 0}, upload=444482]: Uploading piece... Dec 26 18:01:44.603350230 (506h32m44.474433275s) [noise=true , address=145.79.227.8:4330 , piece_id=3YMCHIGG6LOX47OQ265IL4X6FJSK6RMZAWKMDF5C2LX7RDUBEINA, seg_pos={0 0}, upload=444482]: Uploading piece... Dec 26 18:01:44.603707049 (506h32m44.474790094s) [noise=true , address=77.90.31.14:4663 , piece_id=SESXAVIAYHPZ5KIQFD6QWNTI6NXKKCMSSYN6ZXDM2DUDN3CY7SIA, seg_pos={0 0}, upload=444482]: Uploading piece... Dec 26 18:01:44.604197894 (506h32m44.475280929s) [noise=true , address=77.90.31.10:4519 , piece_id=3WLJHO37GY3JLLHFJ2UDYXMOVR6VQ6T3U52YYPMJDAHHCE5LOPAA, seg_pos={0 0}, upload=444482]: Uploading piece... Dec 26 18:01:44.604605240 (506h32m44.475688275s) [noise=true , address=45.91.116.4:4937 , piece_id=GSM7OS24YKFP3VIINOAEGGX4EOBL2VQRDD6YKGU5T5GKXVNZ74HA, seg_pos={0 0}, upload=444482]: Uploading piece... Dec 26 18:01:44.604981677 (506h32m44.476064722s) [noise=true , address=145.79.227.11:4732 , piece_id=JGKV3PVFUSTOEK6VX3IX7J46TIJA22ZPDZNEBCKD6RI2FP45ZEGA, seg_pos={0 0}, upload=444482]: Uploading piece... Dec 26 18:01:44.605299181 (506h32m44.476382226s) [noise=true , address=45.91.116.6:4975 , piece_id=JV6GLRMSHCB345DOFAMII6KWS74WJDZT3IGPLGMRZM2NUWYOB5PQ, seg_pos={0 0}, upload=444482]: Uploading piece... Dec 26 18:01:44.605492490 (506h32m44.476575535s) [noise=true , address=145.79.227.31:4799 , piece_id=WEVOSTZDZUXV2WQ54RBGQHHG4GWVWNVS6JNPWPCCBO5RSLWX72OQ, seg_pos={0 0}, upload=444482]: Uploading piece... Dec 26 18:01:44.606080369 (506h32m44.477163414s) [noise=true , address=145.79.227.21:4474 , piece_id=HELCVB43WKB3ZDONRBS6BR3DMTYO76AWYORTIQ2NCV7SHI4KAMRQ, seg_pos={0 0}, upload=444482]: Uploading piece... Dec 26 18:01:44.606580110 (506h32m44.477663156s) [noise=true , address=145.79.227.4:4186 , piece_id=RWWTBRHMNLLAXYQBNHBGSCAPVNQVW4GBGNO3CKREZZU5WLOR3T7A, seg_pos={0 0}, upload=444482]: Uploading piece... Dec 26 18:01:46.319626871 (506h32m46.190709906s) [upload=444482]: Next returned nil segment Dec 26 18:01:46.319642711 (506h32m46.190725746s) [upload=444482]: Waiting for error group managing segments... Dec 26 18:01:46.607291835 (506h32m46.47837488s) [noise=true , address=145.79.227.10:4401 , piece_id=XCYA3E6OVOJETWBYL34FP2NGVN47K75ZZTJRXPPH6NOKQ4XMJB7Q, seg_pos={0 0}, upload=444482]: Done uploading piece. err: <nil> Dec 26 18:01:46.610693453 (506h32m46.481776488s) [noise=true , address=145.79.227.39:4013 , piece_id=EZHHDHZD3USFUSURBFORTHKGONZ6YJ2BCRUCUUKT4HUVSCL2LPTQ, seg_pos={0 0}, upload=444482]: Done uploading piece. err: <nil> Dec 26 18:01:46.611012420 (506h32m46.482095465s) [noise=true , address=145.79.227.8:4330 , piece_id=3YMCHIGG6LOX47OQ265IL4X6FJSK6RMZAWKMDF5C2LX7RDUBEINA, seg_pos={0 0}, upload=444482]: Done uploading piece. err: <nil> Dec 26 18:01:46.621508626 (506h32m46.492591671s) [noise=true , address=145.79.227.31:4799 , piece_id=WEVOSTZDZUXV2WQ54RBGQHHG4GWVWNVS6JNPWPCCBO5RSLWX72OQ, seg_pos={0 0}, upload=444482]: Done uploading piece. err: <nil> Dec 26 18:01:46.625365981 (506h32m46.496449016s) [noise=true , address=145.79.227.45:4231 , piece_id=XFMDANT7ZCIGASWV56DKO3OPAFAQNEJTRGEI563O37EO4YZSBELA, seg_pos={0 0}, upload=444482]: Done uploading piece. err: <nil> Dec 26 18:01:46.629445088 (506h32m46.500528133s) [noise=true , address=145.79.227.51:4411 , piece_id=YKV7HVSLN4L7TPCJ3TPAKGMZ2ZRZXKC3B7NDJNBOFAQGMF4DDTRQ, seg_pos={0 0}, upload=444482]: Done uploading piece. err: <nil> Dec 26 18:01:46.640495721 (506h32m46.511578756s) [noise=true , address=145.79.227.4:4186 , piece_id=RWWTBRHMNLLAXYQBNHBGSCAPVNQVW4GBGNO3CKREZZU5WLOR3T7A, seg_pos={0 0}, upload=444482]: Done uploading piece. err: <nil> Dec 26 18:01:46.645928436 (506h32m46.517011481s) [noise=true , address=145.79.227.21:4474 , piece_id=HELCVB43WKB3ZDONRBS6BR3DMTYO76AWYORTIQ2NCV7SHI4KAMRQ, seg_pos={0 0}, upload=444482]: Done uploading piece. err: <nil> Dec 26 18:01:46.654201760 (506h32m46.525284795s) [noise=true , address=145.79.227.11:4732 , piece_id=JGKV3PVFUSTOEK6VX3IX7J46TIJA22ZPDZNEBCKD6RI2FP45ZEGA, seg_pos={0 0}, upload=444482]: Done uploading piece. err: <nil> Dec 26 18:01:46.850051632 (506h32m46.721134677s) [noise=true , address=45.91.116.9:4263 , piece_id=MGFBY7Y657TW2OEYGUEMDUVN6DYYWK5GTHQCGVACSYY33HMMCHJQ, seg_pos={0 0}, upload=444482]: Done uploading piece. err: <nil> Dec 26 18:01:46.892001602 (506h32m46.763084637s) [noise=true , address=45.91.116.6:4975 , piece_id=JV6GLRMSHCB345DOFAMII6KWS74WJDZT3IGPLGMRZM2NUWYOB5PQ, seg_pos={0 0}, upload=444482]: Done uploading piece. err: <nil> Dec 26 18:01:46.896117650 (506h32m46.767200695s) [noise=true , address=45.91.116.14:4189 , piece_id=KUVACSJCPXTLHDNOL26KBRJA7KZMYXOFRU6S7ABZLCRYOOOMZD3Q, seg_pos={0 0}, upload=444482]: Done uploading piece. err: <nil> Dec 26 18:01:46.908261072 (506h32m46.779344118s) [noise=true , address=77.90.31.14:4663 , piece_id=SESXAVIAYHPZ5KIQFD6QWNTI6NXKKCMSSYN6ZXDM2DUDN3CY7SIA, seg_pos={0 0}, upload=444482]: Done uploading piece. err: <nil> Dec 26 18:01:46.908285720 (506h32m46.779368755s) [seg_pos={0 0}, upload=444482]: Segment reached optimal threshold of 13 pieces. Dec 26 18:01:46.908564050 (506h32m46.779647085s) [noise=true , address=77.90.31.19:4844 , piece_id=QC3UCTWSJDI2MDBDO3XWWFYRXL2APO7OIHY4BR7PAJSXPOZZYIDA, seg_pos={0 0}, upload=444482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: expected piece hash; context canceled; EOF Dec 26 18:01:46.908575322 (506h32m46.779658367s) [noise=true , address=77.90.31.19:4844 , piece_id=QC3UCTWSJDI2MDBDO3XWWFYRXL2APO7OIHY4BR7PAJSXPOZZYIDA, seg_pos={0 0}, upload=444482]: Long tail cancellation due to optimal threshold Dec 26 18:01:46.911452860 (506h32m46.782535926s) [noise=true , address=77.90.31.10:4519 , piece_id=3WLJHO37GY3JLLHFJ2UDYXMOVR6VQ6T3U52YYPMJDAHHCE5LOPAA, seg_pos={0 0}, upload=444482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: write tcp 188.166.225.170:50298->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:50298->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50298->77.90.31.10:4519: use of closed network connection Dec 26 18:01:46.911470074 (506h32m46.782553129s) [noise=true , address=77.90.31.10:4519 , piece_id=3WLJHO37GY3JLLHFJ2UDYXMOVR6VQ6T3U52YYPMJDAHHCE5LOPAA, seg_pos={0 0}, upload=444482]: Long tail cancellation due to optimal threshold Dec 26 18:01:46.911673732 (506h32m46.782756777s) [noise=true , address=45.91.116.4:4937 , piece_id=GSM7OS24YKFP3VIINOAEGGX4EOBL2VQRDD6YKGU5T5GKXVNZ74HA, seg_pos={0 0}, upload=444482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 26 18:01:46.911683430 (506h32m46.782766475s) [noise=true , address=45.91.116.4:4937 , piece_id=GSM7OS24YKFP3VIINOAEGGX4EOBL2VQRDD6YKGU5T5GKXVNZ74HA, seg_pos={0 0}, upload=444482]: Long tail cancellation due to optimal threshold Dec 26 18:01:46.911707225 (506h32m46.78279026s) [seg_pos={0 0}, upload=444482]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 18:01:46.911789291 (506h32m46.782872337s) [upload=444482]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 18:01:46.911795543 (506h32m46.782878588s) [upload=444482]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 18:01:47.214392370 (506h32m47.085475415s) [upload=444482]: Done uploading segments... Dec 26 18:01:47.214413361 (506h32m47.085496406s) [upload=444482]: Upload finished. err: <nil> Duration: 30.003506404s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 19.274835321s trace: https://jaeger.dev.storj.io/trace/665ea65f6e9ccc33 FAIL (2025-12-26 18:01:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== e=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789316979 (506h12m54.660400024s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789326507 (506h12m54.660409552s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789611339 (506h12m54.660694384s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789619275 (506h12m54.66070231s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789914006 (506h12m54.660997041s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789949443 (506h12m54.661032478s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790181566 (506h12m54.661264601s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790295632 (506h12m54.661378668s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790510872 (506h12m54.661593917s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790692980 (506h12m54.661776015s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790787379 (506h12m54.661870414s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790992970 (506h12m54.662076005s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791120053 (506h12m54.662203098s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791275218 (506h12m54.662358253s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791443709 (506h12m54.662526744s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:56.112313145 (506h12m55.98339619s) [upload=444191]: Next returned nil segment Dec 26 17:41:56.112330638 (506h12m55.983413683s) [upload=444191]: Waiting for error group managing segments... Dec 26 17:41:56.485322720 (506h12m56.356405765s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.485637079 (506h12m56.356720124s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.666807539 (506h12m56.537890584s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.678753987 (506h12m56.549837032s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.687788039 (506h12m56.558871074s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.694318023 (506h12m56.565401058s) [noise=true , address=145.79.227.7:4326 , piece_id=DMQ5RPI46X2U7TJTS42ZHQEF32BDJRKJJO5HWOYAPNS6VWVG5B5Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.716581187 (506h12m56.587664232s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.972183007 (506h12m56.843266042s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.011917999 (506h12m56.883001034s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.054168622 (506h12m56.925251667s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.114312777 (506h12m56.985395812s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.499846250 (506h12m57.370929285s) [noise=true , address=145.79.227.34:4919 , piece_id=YRWHW3BJVEGVJRSA6RJB3T7VJQSDTGKEZQKRICFIV2G3T7AN4MIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.612649066 (506h12m57.483732101s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649413733 (506h12m57.520496768s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649440282 (506h12m57.520523328s) [seg_pos={0 0}, upload=444191]: Segment reached optimal threshold of 14 pieces. Dec 26 17:41:57.649674729 (506h12m57.520757774s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G7ty71tPwe7cZSBNK9u74DFUCGomnxDsNvAif4C3SbP7HWwfo): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649684087 (506h12m57.520767142s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649747708 (506h12m57.520830763s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v6zXZuMmyL2MU8qb3q6gg1iwTYvndjzMjVpv8btEpeX9Lnqmu): protocol: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection Dec 26 17:41:57.649785219 (506h12m57.520868274s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649899828 (506h12m57.520982903s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649908775 (506h12m57.52099182s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650074641 (506h12m57.521157696s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.650083287 (506h12m57.521166342s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650132821 (506h12m57.521215866s) [seg_pos={0 0}, upload=444191]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 17:41:57.650220138 (506h12m57.521303183s) [upload=444191]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:41:57.650227171 (506h12m57.521310206s) [upload=444191]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:41:57.958890870 (506h12m57.829973915s) [upload=444191]: Done uploading segments... Dec 26 17:41:57.958911620 (506h12m57.829994665s) [upload=444191]: Upload finished. err: <nil> Duration: 19.274863425s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [.X.X.XX...XX...XX....XX..XX......X.XXX.....XX.X.XXX.........] duration: 5.84582447s trace: https://jaeger.dev.storj.io/trace/66e1eaafd7692f5a FAIL (2025-12-26 18:00:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: VVU5B5KNVMOIGGZABQAAI22IQ2CZTWJURAJWQQ2JGY3ABTBPFK3A) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: AC62U7GFNGWMPNEFW5H2C4IMSFT6TSFNH5EHUCX24SWZXCPRKOQQ) context deadline exceeded ======LOGS====== Dec 26 17:54:05.860888315 (506h25m5.73197135s) [upload=444374]: Starting upload Dec 26 17:54:05.860960913 (506h25m5.732043958s) [upload=444374]: Done starting upload Dec 26 17:54:05.861243932 (506h25m5.732326967s) [upload=444374]: Uploading segments... Dec 26 17:54:05.861348571 (506h25m5.732431616s) [upload=444374]: Got next segment. Inline: false Dec 26 17:54:05.861399419 (506h25m5.732482464s) [upload=444374]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 17:54:05.861404999 (506h25m5.732488034s) [upload=444374]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 17:54:06.104522897 (506h25m5.975605933s) [seg_pos={0 0}, upload=444374]: Begin upload segment... Dec 26 17:54:06.104572262 (506h25m5.975655297s) [seg_pos={0 0}, upload=444374]: Done begin upload segment. Dec 26 17:54:06.104602710 (506h25m5.975685755s) [noise=true , address=77.90.31.12:4589 , piece_id=CJJUEAIEWP66VHOKZTYLIZUHN3OET6SAFJHXK73EBDFNB5QFE5RA, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.104713491 (506h25m5.975796536s) [noise=true , address=145.79.227.21:4475 , piece_id=UP7LFRPWG3GFBUFT6Y3TOMII7C2UE5BJEY7VCDKHWPYN7NHOL25A, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.105352147 (506h25m5.976435192s) [noise=true , address=145.79.227.51:4410 , piece_id=NJSE6VNZ2ICW5KYFAVC7TXUKOHPAQSHATMU2WW2XMXJF77VTZLVQ, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.105704788 (506h25m5.976787823s) [noise=true , address=77.90.31.15:4700 , piece_id=AC62U7GFNGWMPNEFW5H2C4IMSFT6TSFNH5EHUCX24SWZXCPRKOQQ, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.106020589 (506h25m5.977103624s) [noise=true , address=77.90.31.14:4663 , piece_id=WOQS65FTAS6NTIR7QVGUIFPIJY6POSNUFZTBGYCDZW67TKWZ2GZQ, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.106328646 (506h25m5.977411681s) [noise=true , address=145.79.227.42:4123 , piece_id=T5UOINXTMKBAACABHEVV5NGEMB52SGKWNSJPJ7Y6UKCQXNOUOJMA, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.106787600 (506h25m5.977870635s) [noise=true , address=45.91.116.6:4974 , piece_id=23YEXTSWEM6GOICKLZM5QVYLA7OK2HE55TIA52YG4YO26WBFK4LA, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.107122217 (506h25m5.978205252s) [noise=true , address=145.79.227.49:4339 , piece_id=NHMAMURTEDXM7SZDKB6OP4NU6SCIIKVLIHTVAOT2DVU7UAIT6V4Q, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.108570165 (506h25m5.97965324s) [noise=true , address=145.79.227.15:4874 , piece_id=7PRWX53PMTA2U7GAUIKHBWJXO6RANYJHXUIUFMG3UNASTUTNFIXQ, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.108781597 (506h25m5.979864633s) [noise=true , address=45.91.116.5:4297 , piece_id=I3IPV24ECC3YV4V3QRQDP2ZZT3JOO2CN6M6BISNPENMQZNSDKVAQ, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.110063228 (506h25m5.981146263s) [noise=true , address=145.79.227.11:4732 , piece_id=5FVE6U6ISMCIMDESG3OQ6SHFBGTWT2PPZZC7XRZAPSNK47HZIBWQ, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.111752484 (506h25m5.982835529s) [noise=true , address=145.79.227.36:4519 , piece_id=FJEF6GROIQF7AWQYSJHUEINR5FTBIO7USLJGFWEEEDRJSGW2JG4Q, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.113340689 (506h25m5.984423734s) [noise=true , address=145.79.227.10:4401 , piece_id=ZI4WZERG34DPXHUOLN6QZ55O6SJPZT5LPXPPUNMGGTYAZCAOTORA, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.114643860 (506h25m5.985726905s) [noise=true , address=45.91.116.13:4153 , piece_id=VVU5B5KNVMOIGGZABQAAI22IQ2CZTWJURAJWQQ2JGY3ABTBPFK3A, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.115852932 (506h25m5.986935967s) [noise=true , address=77.90.31.19:4844 , piece_id=AVEWXCOCGI7U437EYT37M4VY3TH73JESDRWAXRLHQKONPFPLAHTQ, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.116921628 (506h25m5.988004673s) [noise=true , address=145.79.227.39:4015 , piece_id=XD5CWP335WRWKXUP3WKMIO7VOFE5G4RN4EGONIRXB6HI43GOCJFQ, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:07.294144135 (506h25m7.16522718s) [noise=true , address=77.90.31.12:4589 , piece_id=CJJUEAIEWP66VHOKZTYLIZUHN3OET6SAFJHXK73EBDFNB5QFE5RA, seg_pos={0 0}, upload=444374]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 26 17:54:07.828164492 (506h25m7.699247537s) [upload=444374]: Next returned nil segment Dec 26 17:54:07.828181555 (506h25m7.6992646s) [upload=444374]: Waiting for error group managing segments... Dec 26 17:54:08.107862847 (506h25m7.978945892s) [noise=true , address=145.79.227.21:4475 , piece_id=UP7LFRPWG3GFBUFT6Y3TOMII7C2UE5BJEY7VCDKHWPYN7NHOL25A, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.112828612 (506h25m7.983911657s) [noise=true , address=145.79.227.51:4410 , piece_id=NJSE6VNZ2ICW5KYFAVC7TXUKOHPAQSHATMU2WW2XMXJF77VTZLVQ, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.119737338 (506h25m7.990820383s) [noise=true , address=145.79.227.10:4401 , piece_id=ZI4WZERG34DPXHUOLN6QZ55O6SJPZT5LPXPPUNMGGTYAZCAOTORA, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.121777863 (506h25m7.992860908s) [noise=true , address=145.79.227.11:4732 , piece_id=5FVE6U6ISMCIMDESG3OQ6SHFBGTWT2PPZZC7XRZAPSNK47HZIBWQ, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.130879434 (506h25m8.00196248s) [noise=true , address=145.79.227.39:4015 , piece_id=XD5CWP335WRWKXUP3WKMIO7VOFE5G4RN4EGONIRXB6HI43GOCJFQ, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.132848083 (506h25m8.003931118s) [noise=true , address=145.79.227.42:4123 , piece_id=T5UOINXTMKBAACABHEVV5NGEMB52SGKWNSJPJ7Y6UKCQXNOUOJMA, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.135816906 (506h25m8.006899952s) [noise=true , address=145.79.227.49:4339 , piece_id=NHMAMURTEDXM7SZDKB6OP4NU6SCIIKVLIHTVAOT2DVU7UAIT6V4Q, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.139078326 (506h25m8.010161361s) [noise=true , address=145.79.227.15:4874 , piece_id=7PRWX53PMTA2U7GAUIKHBWJXO6RANYJHXUIUFMG3UNASTUTNFIXQ, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.139833234 (506h25m8.010916279s) [noise=true , address=145.79.227.36:4519 , piece_id=FJEF6GROIQF7AWQYSJHUEINR5FTBIO7USLJGFWEEEDRJSGW2JG4Q, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.299232928 (506h25m8.170315963s) [noise=true , address=77.90.31.15:4700 , piece_id=AC62U7GFNGWMPNEFW5H2C4IMSFT6TSFNH5EHUCX24SWZXCPRKOQQ, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.362334064 (506h25m8.233417109s) [noise=true , address=45.91.116.13:4153 , piece_id=VVU5B5KNVMOIGGZABQAAI22IQ2CZTWJURAJWQQ2JGY3ABTBPFK3A, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.462770483 (506h25m8.333853538s) [noise=true , address=77.90.31.19:4844 , piece_id=AVEWXCOCGI7U437EYT37M4VY3TH73JESDRWAXRLHQKONPFPLAHTQ, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.474990001 (506h25m8.346073046s) [noise=true , address=45.91.116.6:4974 , piece_id=23YEXTSWEM6GOICKLZM5QVYLA7OK2HE55TIA52YG4YO26WBFK4LA, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.475014728 (506h25m8.346097793s) [seg_pos={0 0}, upload=444374]: Segment reached optimal threshold of 13 pieces. Dec 26 17:54:08.475260065 (506h25m8.34634311s) [noise=true , address=45.91.116.5:4297 , piece_id=I3IPV24ECC3YV4V3QRQDP2ZZT3JOO2CN6M6BISNPENMQZNSDKVAQ, seg_pos={0 0}, upload=444374]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 26 17:54:08.475266929 (506h25m8.346349964s) [noise=true , address=45.91.116.5:4297 , piece_id=I3IPV24ECC3YV4V3QRQDP2ZZT3JOO2CN6M6BISNPENMQZNSDKVAQ, seg_pos={0 0}, upload=444374]: Long tail cancellation due to optimal threshold Dec 26 17:54:08.475601164 (506h25m8.346684209s) [noise=true , address=77.90.31.14:4663 , piece_id=WOQS65FTAS6NTIR7QVGUIFPIJY6POSNUFZTBGYCDZW67TKWZ2GZQ, seg_pos={0 0}, upload=444374]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: expected piece hash; context canceled; EOF Dec 26 17:54:08.475628697 (506h25m8.346711732s) [noise=true , address=77.90.31.14:4663 , piece_id=WOQS65FTAS6NTIR7QVGUIFPIJY6POSNUFZTBGYCDZW67TKWZ2GZQ, seg_pos={0 0}, upload=444374]: Long tail cancellation due to optimal threshold Dec 26 17:54:08.475658875 (506h25m8.34674192s) [seg_pos={0 0}, upload=444374]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 17:54:08.475744387 (506h25m8.346827422s) [upload=444374]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:54:08.475750088 (506h25m8.346833133s) [upload=444374]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:54:08.767674804 (506h25m8.638757859s) [upload=444374]: Done uploading segments... Dec 26 17:54:08.767701635 (506h25m8.63878468s) [upload=444374]: Upload finished. err: <nil> Duration: 5.845842494s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [..X.X.XX...XX...XX....XX..XX......X.XXX.....XX.X.XXX........] duration: 6.64904854s trace: https://jaeger.dev.storj.io/trace/66e1eaafd7692f5a FAIL (2025-12-26 18:00:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: VVU5B5KNVMOIGGZABQAAI22IQ2CZTWJURAJWQQ2JGY3ABTBPFK3A) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: AC62U7GFNGWMPNEFW5H2C4IMSFT6TSFNH5EHUCX24SWZXCPRKOQQ) context deadline exceeded ======LOGS====== Dec 26 17:54:05.860888315 (506h25m5.73197135s) [upload=444374]: Starting upload Dec 26 17:54:05.860960913 (506h25m5.732043958s) [upload=444374]: Done starting upload Dec 26 17:54:05.861243932 (506h25m5.732326967s) [upload=444374]: Uploading segments... Dec 26 17:54:05.861348571 (506h25m5.732431616s) [upload=444374]: Got next segment. Inline: false Dec 26 17:54:05.861399419 (506h25m5.732482464s) [upload=444374]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 17:54:05.861404999 (506h25m5.732488034s) [upload=444374]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 17:54:06.104522897 (506h25m5.975605933s) [seg_pos={0 0}, upload=444374]: Begin upload segment... Dec 26 17:54:06.104572262 (506h25m5.975655297s) [seg_pos={0 0}, upload=444374]: Done begin upload segment. Dec 26 17:54:06.104602710 (506h25m5.975685755s) [noise=true , address=77.90.31.12:4589 , piece_id=CJJUEAIEWP66VHOKZTYLIZUHN3OET6SAFJHXK73EBDFNB5QFE5RA, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.104713491 (506h25m5.975796536s) [noise=true , address=145.79.227.21:4475 , piece_id=UP7LFRPWG3GFBUFT6Y3TOMII7C2UE5BJEY7VCDKHWPYN7NHOL25A, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.105352147 (506h25m5.976435192s) [noise=true , address=145.79.227.51:4410 , piece_id=NJSE6VNZ2ICW5KYFAVC7TXUKOHPAQSHATMU2WW2XMXJF77VTZLVQ, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.105704788 (506h25m5.976787823s) [noise=true , address=77.90.31.15:4700 , piece_id=AC62U7GFNGWMPNEFW5H2C4IMSFT6TSFNH5EHUCX24SWZXCPRKOQQ, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.106020589 (506h25m5.977103624s) [noise=true , address=77.90.31.14:4663 , piece_id=WOQS65FTAS6NTIR7QVGUIFPIJY6POSNUFZTBGYCDZW67TKWZ2GZQ, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.106328646 (506h25m5.977411681s) [noise=true , address=145.79.227.42:4123 , piece_id=T5UOINXTMKBAACABHEVV5NGEMB52SGKWNSJPJ7Y6UKCQXNOUOJMA, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.106787600 (506h25m5.977870635s) [noise=true , address=45.91.116.6:4974 , piece_id=23YEXTSWEM6GOICKLZM5QVYLA7OK2HE55TIA52YG4YO26WBFK4LA, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.107122217 (506h25m5.978205252s) [noise=true , address=145.79.227.49:4339 , piece_id=NHMAMURTEDXM7SZDKB6OP4NU6SCIIKVLIHTVAOT2DVU7UAIT6V4Q, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.108570165 (506h25m5.97965324s) [noise=true , address=145.79.227.15:4874 , piece_id=7PRWX53PMTA2U7GAUIKHBWJXO6RANYJHXUIUFMG3UNASTUTNFIXQ, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.108781597 (506h25m5.979864633s) [noise=true , address=45.91.116.5:4297 , piece_id=I3IPV24ECC3YV4V3QRQDP2ZZT3JOO2CN6M6BISNPENMQZNSDKVAQ, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.110063228 (506h25m5.981146263s) [noise=true , address=145.79.227.11:4732 , piece_id=5FVE6U6ISMCIMDESG3OQ6SHFBGTWT2PPZZC7XRZAPSNK47HZIBWQ, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.111752484 (506h25m5.982835529s) [noise=true , address=145.79.227.36:4519 , piece_id=FJEF6GROIQF7AWQYSJHUEINR5FTBIO7USLJGFWEEEDRJSGW2JG4Q, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.113340689 (506h25m5.984423734s) [noise=true , address=145.79.227.10:4401 , piece_id=ZI4WZERG34DPXHUOLN6QZ55O6SJPZT5LPXPPUNMGGTYAZCAOTORA, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.114643860 (506h25m5.985726905s) [noise=true , address=45.91.116.13:4153 , piece_id=VVU5B5KNVMOIGGZABQAAI22IQ2CZTWJURAJWQQ2JGY3ABTBPFK3A, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.115852932 (506h25m5.986935967s) [noise=true , address=77.90.31.19:4844 , piece_id=AVEWXCOCGI7U437EYT37M4VY3TH73JESDRWAXRLHQKONPFPLAHTQ, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.116921628 (506h25m5.988004673s) [noise=true , address=145.79.227.39:4015 , piece_id=XD5CWP335WRWKXUP3WKMIO7VOFE5G4RN4EGONIRXB6HI43GOCJFQ, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:07.294144135 (506h25m7.16522718s) [noise=true , address=77.90.31.12:4589 , piece_id=CJJUEAIEWP66VHOKZTYLIZUHN3OET6SAFJHXK73EBDFNB5QFE5RA, seg_pos={0 0}, upload=444374]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 26 17:54:07.828164492 (506h25m7.699247537s) [upload=444374]: Next returned nil segment Dec 26 17:54:07.828181555 (506h25m7.6992646s) [upload=444374]: Waiting for error group managing segments... Dec 26 17:54:08.107862847 (506h25m7.978945892s) [noise=true , address=145.79.227.21:4475 , piece_id=UP7LFRPWG3GFBUFT6Y3TOMII7C2UE5BJEY7VCDKHWPYN7NHOL25A, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.112828612 (506h25m7.983911657s) [noise=true , address=145.79.227.51:4410 , piece_id=NJSE6VNZ2ICW5KYFAVC7TXUKOHPAQSHATMU2WW2XMXJF77VTZLVQ, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.119737338 (506h25m7.990820383s) [noise=true , address=145.79.227.10:4401 , piece_id=ZI4WZERG34DPXHUOLN6QZ55O6SJPZT5LPXPPUNMGGTYAZCAOTORA, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.121777863 (506h25m7.992860908s) [noise=true , address=145.79.227.11:4732 , piece_id=5FVE6U6ISMCIMDESG3OQ6SHFBGTWT2PPZZC7XRZAPSNK47HZIBWQ, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.130879434 (506h25m8.00196248s) [noise=true , address=145.79.227.39:4015 , piece_id=XD5CWP335WRWKXUP3WKMIO7VOFE5G4RN4EGONIRXB6HI43GOCJFQ, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.132848083 (506h25m8.003931118s) [noise=true , address=145.79.227.42:4123 , piece_id=T5UOINXTMKBAACABHEVV5NGEMB52SGKWNSJPJ7Y6UKCQXNOUOJMA, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.135816906 (506h25m8.006899952s) [noise=true , address=145.79.227.49:4339 , piece_id=NHMAMURTEDXM7SZDKB6OP4NU6SCIIKVLIHTVAOT2DVU7UAIT6V4Q, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.139078326 (506h25m8.010161361s) [noise=true , address=145.79.227.15:4874 , piece_id=7PRWX53PMTA2U7GAUIKHBWJXO6RANYJHXUIUFMG3UNASTUTNFIXQ, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.139833234 (506h25m8.010916279s) [noise=true , address=145.79.227.36:4519 , piece_id=FJEF6GROIQF7AWQYSJHUEINR5FTBIO7USLJGFWEEEDRJSGW2JG4Q, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.299232928 (506h25m8.170315963s) [noise=true , address=77.90.31.15:4700 , piece_id=AC62U7GFNGWMPNEFW5H2C4IMSFT6TSFNH5EHUCX24SWZXCPRKOQQ, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.362334064 (506h25m8.233417109s) [noise=true , address=45.91.116.13:4153 , piece_id=VVU5B5KNVMOIGGZABQAAI22IQ2CZTWJURAJWQQ2JGY3ABTBPFK3A, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.462770483 (506h25m8.333853538s) [noise=true , address=77.90.31.19:4844 , piece_id=AVEWXCOCGI7U437EYT37M4VY3TH73JESDRWAXRLHQKONPFPLAHTQ, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.474990001 (506h25m8.346073046s) [noise=true , address=45.91.116.6:4974 , piece_id=23YEXTSWEM6GOICKLZM5QVYLA7OK2HE55TIA52YG4YO26WBFK4LA, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.475014728 (506h25m8.346097793s) [seg_pos={0 0}, upload=444374]: Segment reached optimal threshold of 13 pieces. Dec 26 17:54:08.475260065 (506h25m8.34634311s) [noise=true , address=45.91.116.5:4297 , piece_id=I3IPV24ECC3YV4V3QRQDP2ZZT3JOO2CN6M6BISNPENMQZNSDKVAQ, seg_pos={0 0}, upload=444374]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 26 17:54:08.475266929 (506h25m8.346349964s) [noise=true , address=45.91.116.5:4297 , piece_id=I3IPV24ECC3YV4V3QRQDP2ZZT3JOO2CN6M6BISNPENMQZNSDKVAQ, seg_pos={0 0}, upload=444374]: Long tail cancellation due to optimal threshold Dec 26 17:54:08.475601164 (506h25m8.346684209s) [noise=true , address=77.90.31.14:4663 , piece_id=WOQS65FTAS6NTIR7QVGUIFPIJY6POSNUFZTBGYCDZW67TKWZ2GZQ, seg_pos={0 0}, upload=444374]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: expected piece hash; context canceled; EOF Dec 26 17:54:08.475628697 (506h25m8.346711732s) [noise=true , address=77.90.31.14:4663 , piece_id=WOQS65FTAS6NTIR7QVGUIFPIJY6POSNUFZTBGYCDZW67TKWZ2GZQ, seg_pos={0 0}, upload=444374]: Long tail cancellation due to optimal threshold Dec 26 17:54:08.475658875 (506h25m8.34674192s) [seg_pos={0 0}, upload=444374]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 17:54:08.475744387 (506h25m8.346827422s) [upload=444374]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:54:08.475750088 (506h25m8.346833133s) [upload=444374]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:54:08.767674804 (506h25m8.638757859s) [upload=444374]: Done uploading segments... Dec 26 17:54:08.767701635 (506h25m8.63878468s) [upload=444374]: Upload finished. err: <nil> Duration: 6.649079819s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 26.017312004s trace: https://jaeger.dev.storj.io/trace/665ea65f6e9ccc33 FAIL (2025-12-26 18:00:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== e=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789316979 (506h12m54.660400024s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789326507 (506h12m54.660409552s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789611339 (506h12m54.660694384s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789619275 (506h12m54.66070231s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789914006 (506h12m54.660997041s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789949443 (506h12m54.661032478s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790181566 (506h12m54.661264601s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790295632 (506h12m54.661378668s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790510872 (506h12m54.661593917s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790692980 (506h12m54.661776015s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790787379 (506h12m54.661870414s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790992970 (506h12m54.662076005s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791120053 (506h12m54.662203098s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791275218 (506h12m54.662358253s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791443709 (506h12m54.662526744s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:56.112313145 (506h12m55.98339619s) [upload=444191]: Next returned nil segment Dec 26 17:41:56.112330638 (506h12m55.983413683s) [upload=444191]: Waiting for error group managing segments... Dec 26 17:41:56.485322720 (506h12m56.356405765s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.485637079 (506h12m56.356720124s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.666807539 (506h12m56.537890584s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.678753987 (506h12m56.549837032s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.687788039 (506h12m56.558871074s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.694318023 (506h12m56.565401058s) [noise=true , address=145.79.227.7:4326 , piece_id=DMQ5RPI46X2U7TJTS42ZHQEF32BDJRKJJO5HWOYAPNS6VWVG5B5Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.716581187 (506h12m56.587664232s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.972183007 (506h12m56.843266042s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.011917999 (506h12m56.883001034s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.054168622 (506h12m56.925251667s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.114312777 (506h12m56.985395812s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.499846250 (506h12m57.370929285s) [noise=true , address=145.79.227.34:4919 , piece_id=YRWHW3BJVEGVJRSA6RJB3T7VJQSDTGKEZQKRICFIV2G3T7AN4MIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.612649066 (506h12m57.483732101s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649413733 (506h12m57.520496768s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649440282 (506h12m57.520523328s) [seg_pos={0 0}, upload=444191]: Segment reached optimal threshold of 14 pieces. Dec 26 17:41:57.649674729 (506h12m57.520757774s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G7ty71tPwe7cZSBNK9u74DFUCGomnxDsNvAif4C3SbP7HWwfo): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649684087 (506h12m57.520767142s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649747708 (506h12m57.520830763s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v6zXZuMmyL2MU8qb3q6gg1iwTYvndjzMjVpv8btEpeX9Lnqmu): protocol: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection Dec 26 17:41:57.649785219 (506h12m57.520868274s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649899828 (506h12m57.520982903s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649908775 (506h12m57.52099182s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650074641 (506h12m57.521157696s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.650083287 (506h12m57.521166342s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650132821 (506h12m57.521215866s) [seg_pos={0 0}, upload=444191]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 17:41:57.650220138 (506h12m57.521303183s) [upload=444191]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:41:57.650227171 (506h12m57.521310206s) [upload=444191]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:41:57.958890870 (506h12m57.829973915s) [upload=444191]: Done uploading segments... Dec 26 17:41:57.958911620 (506h12m57.829994665s) [upload=444191]: Upload finished. err: <nil> Duration: 26.01734699s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 40.433925294s trace: https://jaeger.dev.storj.io/trace/665ea65f6e9ccc33 FAIL (2025-12-26 18:00:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== e=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789316979 (506h12m54.660400024s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789326507 (506h12m54.660409552s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789611339 (506h12m54.660694384s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789619275 (506h12m54.66070231s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789914006 (506h12m54.660997041s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789949443 (506h12m54.661032478s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790181566 (506h12m54.661264601s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790295632 (506h12m54.661378668s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790510872 (506h12m54.661593917s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790692980 (506h12m54.661776015s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790787379 (506h12m54.661870414s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790992970 (506h12m54.662076005s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791120053 (506h12m54.662203098s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791275218 (506h12m54.662358253s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791443709 (506h12m54.662526744s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:56.112313145 (506h12m55.98339619s) [upload=444191]: Next returned nil segment Dec 26 17:41:56.112330638 (506h12m55.983413683s) [upload=444191]: Waiting for error group managing segments... Dec 26 17:41:56.485322720 (506h12m56.356405765s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.485637079 (506h12m56.356720124s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.666807539 (506h12m56.537890584s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.678753987 (506h12m56.549837032s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.687788039 (506h12m56.558871074s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.694318023 (506h12m56.565401058s) [noise=true , address=145.79.227.7:4326 , piece_id=DMQ5RPI46X2U7TJTS42ZHQEF32BDJRKJJO5HWOYAPNS6VWVG5B5Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.716581187 (506h12m56.587664232s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.972183007 (506h12m56.843266042s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.011917999 (506h12m56.883001034s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.054168622 (506h12m56.925251667s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.114312777 (506h12m56.985395812s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.499846250 (506h12m57.370929285s) [noise=true , address=145.79.227.34:4919 , piece_id=YRWHW3BJVEGVJRSA6RJB3T7VJQSDTGKEZQKRICFIV2G3T7AN4MIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.612649066 (506h12m57.483732101s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649413733 (506h12m57.520496768s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649440282 (506h12m57.520523328s) [seg_pos={0 0}, upload=444191]: Segment reached optimal threshold of 14 pieces. Dec 26 17:41:57.649674729 (506h12m57.520757774s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G7ty71tPwe7cZSBNK9u74DFUCGomnxDsNvAif4C3SbP7HWwfo): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649684087 (506h12m57.520767142s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649747708 (506h12m57.520830763s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v6zXZuMmyL2MU8qb3q6gg1iwTYvndjzMjVpv8btEpeX9Lnqmu): protocol: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection Dec 26 17:41:57.649785219 (506h12m57.520868274s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649899828 (506h12m57.520982903s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649908775 (506h12m57.52099182s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650074641 (506h12m57.521157696s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.650083287 (506h12m57.521166342s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650132821 (506h12m57.521215866s) [seg_pos={0 0}, upload=444191]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 17:41:57.650220138 (506h12m57.521303183s) [upload=444191]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:41:57.650227171 (506h12m57.521310206s) [upload=444191]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:41:57.958890870 (506h12m57.829973915s) [upload=444191]: Done uploading segments... Dec 26 17:41:57.958911620 (506h12m57.829994665s) [upload=444191]: Upload finished. err: <nil> Duration: 40.433944379s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [...X.X.XX...XX...XX....XX..XX......X.XXX.....XX.X.XXX.......] duration: 6.229446153s trace: https://jaeger.dev.storj.io/trace/66e1eaafd7692f5a FAIL (2025-12-26 17:59:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: VVU5B5KNVMOIGGZABQAAI22IQ2CZTWJURAJWQQ2JGY3ABTBPFK3A) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: AC62U7GFNGWMPNEFW5H2C4IMSFT6TSFNH5EHUCX24SWZXCPRKOQQ) context deadline exceeded ======LOGS====== Dec 26 17:54:05.860888315 (506h25m5.73197135s) [upload=444374]: Starting upload Dec 26 17:54:05.860960913 (506h25m5.732043958s) [upload=444374]: Done starting upload Dec 26 17:54:05.861243932 (506h25m5.732326967s) [upload=444374]: Uploading segments... Dec 26 17:54:05.861348571 (506h25m5.732431616s) [upload=444374]: Got next segment. Inline: false Dec 26 17:54:05.861399419 (506h25m5.732482464s) [upload=444374]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 17:54:05.861404999 (506h25m5.732488034s) [upload=444374]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 17:54:06.104522897 (506h25m5.975605933s) [seg_pos={0 0}, upload=444374]: Begin upload segment... Dec 26 17:54:06.104572262 (506h25m5.975655297s) [seg_pos={0 0}, upload=444374]: Done begin upload segment. Dec 26 17:54:06.104602710 (506h25m5.975685755s) [noise=true , address=77.90.31.12:4589 , piece_id=CJJUEAIEWP66VHOKZTYLIZUHN3OET6SAFJHXK73EBDFNB5QFE5RA, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.104713491 (506h25m5.975796536s) [noise=true , address=145.79.227.21:4475 , piece_id=UP7LFRPWG3GFBUFT6Y3TOMII7C2UE5BJEY7VCDKHWPYN7NHOL25A, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.105352147 (506h25m5.976435192s) [noise=true , address=145.79.227.51:4410 , piece_id=NJSE6VNZ2ICW5KYFAVC7TXUKOHPAQSHATMU2WW2XMXJF77VTZLVQ, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.105704788 (506h25m5.976787823s) [noise=true , address=77.90.31.15:4700 , piece_id=AC62U7GFNGWMPNEFW5H2C4IMSFT6TSFNH5EHUCX24SWZXCPRKOQQ, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.106020589 (506h25m5.977103624s) [noise=true , address=77.90.31.14:4663 , piece_id=WOQS65FTAS6NTIR7QVGUIFPIJY6POSNUFZTBGYCDZW67TKWZ2GZQ, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.106328646 (506h25m5.977411681s) [noise=true , address=145.79.227.42:4123 , piece_id=T5UOINXTMKBAACABHEVV5NGEMB52SGKWNSJPJ7Y6UKCQXNOUOJMA, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.106787600 (506h25m5.977870635s) [noise=true , address=45.91.116.6:4974 , piece_id=23YEXTSWEM6GOICKLZM5QVYLA7OK2HE55TIA52YG4YO26WBFK4LA, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.107122217 (506h25m5.978205252s) [noise=true , address=145.79.227.49:4339 , piece_id=NHMAMURTEDXM7SZDKB6OP4NU6SCIIKVLIHTVAOT2DVU7UAIT6V4Q, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.108570165 (506h25m5.97965324s) [noise=true , address=145.79.227.15:4874 , piece_id=7PRWX53PMTA2U7GAUIKHBWJXO6RANYJHXUIUFMG3UNASTUTNFIXQ, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.108781597 (506h25m5.979864633s) [noise=true , address=45.91.116.5:4297 , piece_id=I3IPV24ECC3YV4V3QRQDP2ZZT3JOO2CN6M6BISNPENMQZNSDKVAQ, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.110063228 (506h25m5.981146263s) [noise=true , address=145.79.227.11:4732 , piece_id=5FVE6U6ISMCIMDESG3OQ6SHFBGTWT2PPZZC7XRZAPSNK47HZIBWQ, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.111752484 (506h25m5.982835529s) [noise=true , address=145.79.227.36:4519 , piece_id=FJEF6GROIQF7AWQYSJHUEINR5FTBIO7USLJGFWEEEDRJSGW2JG4Q, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.113340689 (506h25m5.984423734s) [noise=true , address=145.79.227.10:4401 , piece_id=ZI4WZERG34DPXHUOLN6QZ55O6SJPZT5LPXPPUNMGGTYAZCAOTORA, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.114643860 (506h25m5.985726905s) [noise=true , address=45.91.116.13:4153 , piece_id=VVU5B5KNVMOIGGZABQAAI22IQ2CZTWJURAJWQQ2JGY3ABTBPFK3A, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.115852932 (506h25m5.986935967s) [noise=true , address=77.90.31.19:4844 , piece_id=AVEWXCOCGI7U437EYT37M4VY3TH73JESDRWAXRLHQKONPFPLAHTQ, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.116921628 (506h25m5.988004673s) [noise=true , address=145.79.227.39:4015 , piece_id=XD5CWP335WRWKXUP3WKMIO7VOFE5G4RN4EGONIRXB6HI43GOCJFQ, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:07.294144135 (506h25m7.16522718s) [noise=true , address=77.90.31.12:4589 , piece_id=CJJUEAIEWP66VHOKZTYLIZUHN3OET6SAFJHXK73EBDFNB5QFE5RA, seg_pos={0 0}, upload=444374]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 26 17:54:07.828164492 (506h25m7.699247537s) [upload=444374]: Next returned nil segment Dec 26 17:54:07.828181555 (506h25m7.6992646s) [upload=444374]: Waiting for error group managing segments... Dec 26 17:54:08.107862847 (506h25m7.978945892s) [noise=true , address=145.79.227.21:4475 , piece_id=UP7LFRPWG3GFBUFT6Y3TOMII7C2UE5BJEY7VCDKHWPYN7NHOL25A, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.112828612 (506h25m7.983911657s) [noise=true , address=145.79.227.51:4410 , piece_id=NJSE6VNZ2ICW5KYFAVC7TXUKOHPAQSHATMU2WW2XMXJF77VTZLVQ, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.119737338 (506h25m7.990820383s) [noise=true , address=145.79.227.10:4401 , piece_id=ZI4WZERG34DPXHUOLN6QZ55O6SJPZT5LPXPPUNMGGTYAZCAOTORA, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.121777863 (506h25m7.992860908s) [noise=true , address=145.79.227.11:4732 , piece_id=5FVE6U6ISMCIMDESG3OQ6SHFBGTWT2PPZZC7XRZAPSNK47HZIBWQ, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.130879434 (506h25m8.00196248s) [noise=true , address=145.79.227.39:4015 , piece_id=XD5CWP335WRWKXUP3WKMIO7VOFE5G4RN4EGONIRXB6HI43GOCJFQ, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.132848083 (506h25m8.003931118s) [noise=true , address=145.79.227.42:4123 , piece_id=T5UOINXTMKBAACABHEVV5NGEMB52SGKWNSJPJ7Y6UKCQXNOUOJMA, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.135816906 (506h25m8.006899952s) [noise=true , address=145.79.227.49:4339 , piece_id=NHMAMURTEDXM7SZDKB6OP4NU6SCIIKVLIHTVAOT2DVU7UAIT6V4Q, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.139078326 (506h25m8.010161361s) [noise=true , address=145.79.227.15:4874 , piece_id=7PRWX53PMTA2U7GAUIKHBWJXO6RANYJHXUIUFMG3UNASTUTNFIXQ, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.139833234 (506h25m8.010916279s) [noise=true , address=145.79.227.36:4519 , piece_id=FJEF6GROIQF7AWQYSJHUEINR5FTBIO7USLJGFWEEEDRJSGW2JG4Q, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.299232928 (506h25m8.170315963s) [noise=true , address=77.90.31.15:4700 , piece_id=AC62U7GFNGWMPNEFW5H2C4IMSFT6TSFNH5EHUCX24SWZXCPRKOQQ, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.362334064 (506h25m8.233417109s) [noise=true , address=45.91.116.13:4153 , piece_id=VVU5B5KNVMOIGGZABQAAI22IQ2CZTWJURAJWQQ2JGY3ABTBPFK3A, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.462770483 (506h25m8.333853538s) [noise=true , address=77.90.31.19:4844 , piece_id=AVEWXCOCGI7U437EYT37M4VY3TH73JESDRWAXRLHQKONPFPLAHTQ, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.474990001 (506h25m8.346073046s) [noise=true , address=45.91.116.6:4974 , piece_id=23YEXTSWEM6GOICKLZM5QVYLA7OK2HE55TIA52YG4YO26WBFK4LA, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.475014728 (506h25m8.346097793s) [seg_pos={0 0}, upload=444374]: Segment reached optimal threshold of 13 pieces. Dec 26 17:54:08.475260065 (506h25m8.34634311s) [noise=true , address=45.91.116.5:4297 , piece_id=I3IPV24ECC3YV4V3QRQDP2ZZT3JOO2CN6M6BISNPENMQZNSDKVAQ, seg_pos={0 0}, upload=444374]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 26 17:54:08.475266929 (506h25m8.346349964s) [noise=true , address=45.91.116.5:4297 , piece_id=I3IPV24ECC3YV4V3QRQDP2ZZT3JOO2CN6M6BISNPENMQZNSDKVAQ, seg_pos={0 0}, upload=444374]: Long tail cancellation due to optimal threshold Dec 26 17:54:08.475601164 (506h25m8.346684209s) [noise=true , address=77.90.31.14:4663 , piece_id=WOQS65FTAS6NTIR7QVGUIFPIJY6POSNUFZTBGYCDZW67TKWZ2GZQ, seg_pos={0 0}, upload=444374]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: expected piece hash; context canceled; EOF Dec 26 17:54:08.475628697 (506h25m8.346711732s) [noise=true , address=77.90.31.14:4663 , piece_id=WOQS65FTAS6NTIR7QVGUIFPIJY6POSNUFZTBGYCDZW67TKWZ2GZQ, seg_pos={0 0}, upload=444374]: Long tail cancellation due to optimal threshold Dec 26 17:54:08.475658875 (506h25m8.34674192s) [seg_pos={0 0}, upload=444374]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 17:54:08.475744387 (506h25m8.346827422s) [upload=444374]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:54:08.475750088 (506h25m8.346833133s) [upload=444374]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:54:08.767674804 (506h25m8.638757859s) [upload=444374]: Done uploading segments... Dec 26 17:54:08.767701635 (506h25m8.63878468s) [upload=444374]: Upload finished. err: <nil> Duration: 6.229477613s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 31.77153047s trace: https://jaeger.dev.storj.io/trace/665ea65f6e9ccc33 FAIL (2025-12-26 17:59:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== e=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789316979 (506h12m54.660400024s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789326507 (506h12m54.660409552s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789611339 (506h12m54.660694384s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789619275 (506h12m54.66070231s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789914006 (506h12m54.660997041s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789949443 (506h12m54.661032478s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790181566 (506h12m54.661264601s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790295632 (506h12m54.661378668s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790510872 (506h12m54.661593917s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790692980 (506h12m54.661776015s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790787379 (506h12m54.661870414s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790992970 (506h12m54.662076005s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791120053 (506h12m54.662203098s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791275218 (506h12m54.662358253s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791443709 (506h12m54.662526744s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:56.112313145 (506h12m55.98339619s) [upload=444191]: Next returned nil segment Dec 26 17:41:56.112330638 (506h12m55.983413683s) [upload=444191]: Waiting for error group managing segments... Dec 26 17:41:56.485322720 (506h12m56.356405765s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.485637079 (506h12m56.356720124s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.666807539 (506h12m56.537890584s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.678753987 (506h12m56.549837032s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.687788039 (506h12m56.558871074s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.694318023 (506h12m56.565401058s) [noise=true , address=145.79.227.7:4326 , piece_id=DMQ5RPI46X2U7TJTS42ZHQEF32BDJRKJJO5HWOYAPNS6VWVG5B5Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.716581187 (506h12m56.587664232s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.972183007 (506h12m56.843266042s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.011917999 (506h12m56.883001034s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.054168622 (506h12m56.925251667s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.114312777 (506h12m56.985395812s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.499846250 (506h12m57.370929285s) [noise=true , address=145.79.227.34:4919 , piece_id=YRWHW3BJVEGVJRSA6RJB3T7VJQSDTGKEZQKRICFIV2G3T7AN4MIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.612649066 (506h12m57.483732101s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649413733 (506h12m57.520496768s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649440282 (506h12m57.520523328s) [seg_pos={0 0}, upload=444191]: Segment reached optimal threshold of 14 pieces. Dec 26 17:41:57.649674729 (506h12m57.520757774s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G7ty71tPwe7cZSBNK9u74DFUCGomnxDsNvAif4C3SbP7HWwfo): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649684087 (506h12m57.520767142s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649747708 (506h12m57.520830763s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v6zXZuMmyL2MU8qb3q6gg1iwTYvndjzMjVpv8btEpeX9Lnqmu): protocol: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection Dec 26 17:41:57.649785219 (506h12m57.520868274s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649899828 (506h12m57.520982903s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649908775 (506h12m57.52099182s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650074641 (506h12m57.521157696s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.650083287 (506h12m57.521166342s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650132821 (506h12m57.521215866s) [seg_pos={0 0}, upload=444191]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 17:41:57.650220138 (506h12m57.521303183s) [upload=444191]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:41:57.650227171 (506h12m57.521310206s) [upload=444191]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:41:57.958890870 (506h12m57.829973915s) [upload=444191]: Done uploading segments... Dec 26 17:41:57.958911620 (506h12m57.829994665s) [upload=444191]: Upload finished. err: <nil> Duration: 31.771558423s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [....X.X.XX...XX...XX....XX..XX......X.XXX.....XX.X.XXX......] duration: 6.025751272s trace: https://jaeger.dev.storj.io/trace/66e1eaafd7692f5a FAIL (2025-12-26 17:58:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: VVU5B5KNVMOIGGZABQAAI22IQ2CZTWJURAJWQQ2JGY3ABTBPFK3A) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: AC62U7GFNGWMPNEFW5H2C4IMSFT6TSFNH5EHUCX24SWZXCPRKOQQ) context deadline exceeded ======LOGS====== Dec 26 17:54:05.860888315 (506h25m5.73197135s) [upload=444374]: Starting upload Dec 26 17:54:05.860960913 (506h25m5.732043958s) [upload=444374]: Done starting upload Dec 26 17:54:05.861243932 (506h25m5.732326967s) [upload=444374]: Uploading segments... Dec 26 17:54:05.861348571 (506h25m5.732431616s) [upload=444374]: Got next segment. Inline: false Dec 26 17:54:05.861399419 (506h25m5.732482464s) [upload=444374]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 17:54:05.861404999 (506h25m5.732488034s) [upload=444374]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 17:54:06.104522897 (506h25m5.975605933s) [seg_pos={0 0}, upload=444374]: Begin upload segment... Dec 26 17:54:06.104572262 (506h25m5.975655297s) [seg_pos={0 0}, upload=444374]: Done begin upload segment. Dec 26 17:54:06.104602710 (506h25m5.975685755s) [noise=true , address=77.90.31.12:4589 , piece_id=CJJUEAIEWP66VHOKZTYLIZUHN3OET6SAFJHXK73EBDFNB5QFE5RA, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.104713491 (506h25m5.975796536s) [noise=true , address=145.79.227.21:4475 , piece_id=UP7LFRPWG3GFBUFT6Y3TOMII7C2UE5BJEY7VCDKHWPYN7NHOL25A, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.105352147 (506h25m5.976435192s) [noise=true , address=145.79.227.51:4410 , piece_id=NJSE6VNZ2ICW5KYFAVC7TXUKOHPAQSHATMU2WW2XMXJF77VTZLVQ, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.105704788 (506h25m5.976787823s) [noise=true , address=77.90.31.15:4700 , piece_id=AC62U7GFNGWMPNEFW5H2C4IMSFT6TSFNH5EHUCX24SWZXCPRKOQQ, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.106020589 (506h25m5.977103624s) [noise=true , address=77.90.31.14:4663 , piece_id=WOQS65FTAS6NTIR7QVGUIFPIJY6POSNUFZTBGYCDZW67TKWZ2GZQ, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.106328646 (506h25m5.977411681s) [noise=true , address=145.79.227.42:4123 , piece_id=T5UOINXTMKBAACABHEVV5NGEMB52SGKWNSJPJ7Y6UKCQXNOUOJMA, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.106787600 (506h25m5.977870635s) [noise=true , address=45.91.116.6:4974 , piece_id=23YEXTSWEM6GOICKLZM5QVYLA7OK2HE55TIA52YG4YO26WBFK4LA, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.107122217 (506h25m5.978205252s) [noise=true , address=145.79.227.49:4339 , piece_id=NHMAMURTEDXM7SZDKB6OP4NU6SCIIKVLIHTVAOT2DVU7UAIT6V4Q, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.108570165 (506h25m5.97965324s) [noise=true , address=145.79.227.15:4874 , piece_id=7PRWX53PMTA2U7GAUIKHBWJXO6RANYJHXUIUFMG3UNASTUTNFIXQ, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.108781597 (506h25m5.979864633s) [noise=true , address=45.91.116.5:4297 , piece_id=I3IPV24ECC3YV4V3QRQDP2ZZT3JOO2CN6M6BISNPENMQZNSDKVAQ, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.110063228 (506h25m5.981146263s) [noise=true , address=145.79.227.11:4732 , piece_id=5FVE6U6ISMCIMDESG3OQ6SHFBGTWT2PPZZC7XRZAPSNK47HZIBWQ, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.111752484 (506h25m5.982835529s) [noise=true , address=145.79.227.36:4519 , piece_id=FJEF6GROIQF7AWQYSJHUEINR5FTBIO7USLJGFWEEEDRJSGW2JG4Q, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.113340689 (506h25m5.984423734s) [noise=true , address=145.79.227.10:4401 , piece_id=ZI4WZERG34DPXHUOLN6QZ55O6SJPZT5LPXPPUNMGGTYAZCAOTORA, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.114643860 (506h25m5.985726905s) [noise=true , address=45.91.116.13:4153 , piece_id=VVU5B5KNVMOIGGZABQAAI22IQ2CZTWJURAJWQQ2JGY3ABTBPFK3A, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.115852932 (506h25m5.986935967s) [noise=true , address=77.90.31.19:4844 , piece_id=AVEWXCOCGI7U437EYT37M4VY3TH73JESDRWAXRLHQKONPFPLAHTQ, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.116921628 (506h25m5.988004673s) [noise=true , address=145.79.227.39:4015 , piece_id=XD5CWP335WRWKXUP3WKMIO7VOFE5G4RN4EGONIRXB6HI43GOCJFQ, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:07.294144135 (506h25m7.16522718s) [noise=true , address=77.90.31.12:4589 , piece_id=CJJUEAIEWP66VHOKZTYLIZUHN3OET6SAFJHXK73EBDFNB5QFE5RA, seg_pos={0 0}, upload=444374]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 26 17:54:07.828164492 (506h25m7.699247537s) [upload=444374]: Next returned nil segment Dec 26 17:54:07.828181555 (506h25m7.6992646s) [upload=444374]: Waiting for error group managing segments... Dec 26 17:54:08.107862847 (506h25m7.978945892s) [noise=true , address=145.79.227.21:4475 , piece_id=UP7LFRPWG3GFBUFT6Y3TOMII7C2UE5BJEY7VCDKHWPYN7NHOL25A, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.112828612 (506h25m7.983911657s) [noise=true , address=145.79.227.51:4410 , piece_id=NJSE6VNZ2ICW5KYFAVC7TXUKOHPAQSHATMU2WW2XMXJF77VTZLVQ, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.119737338 (506h25m7.990820383s) [noise=true , address=145.79.227.10:4401 , piece_id=ZI4WZERG34DPXHUOLN6QZ55O6SJPZT5LPXPPUNMGGTYAZCAOTORA, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.121777863 (506h25m7.992860908s) [noise=true , address=145.79.227.11:4732 , piece_id=5FVE6U6ISMCIMDESG3OQ6SHFBGTWT2PPZZC7XRZAPSNK47HZIBWQ, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.130879434 (506h25m8.00196248s) [noise=true , address=145.79.227.39:4015 , piece_id=XD5CWP335WRWKXUP3WKMIO7VOFE5G4RN4EGONIRXB6HI43GOCJFQ, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.132848083 (506h25m8.003931118s) [noise=true , address=145.79.227.42:4123 , piece_id=T5UOINXTMKBAACABHEVV5NGEMB52SGKWNSJPJ7Y6UKCQXNOUOJMA, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.135816906 (506h25m8.006899952s) [noise=true , address=145.79.227.49:4339 , piece_id=NHMAMURTEDXM7SZDKB6OP4NU6SCIIKVLIHTVAOT2DVU7UAIT6V4Q, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.139078326 (506h25m8.010161361s) [noise=true , address=145.79.227.15:4874 , piece_id=7PRWX53PMTA2U7GAUIKHBWJXO6RANYJHXUIUFMG3UNASTUTNFIXQ, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.139833234 (506h25m8.010916279s) [noise=true , address=145.79.227.36:4519 , piece_id=FJEF6GROIQF7AWQYSJHUEINR5FTBIO7USLJGFWEEEDRJSGW2JG4Q, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.299232928 (506h25m8.170315963s) [noise=true , address=77.90.31.15:4700 , piece_id=AC62U7GFNGWMPNEFW5H2C4IMSFT6TSFNH5EHUCX24SWZXCPRKOQQ, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.362334064 (506h25m8.233417109s) [noise=true , address=45.91.116.13:4153 , piece_id=VVU5B5KNVMOIGGZABQAAI22IQ2CZTWJURAJWQQ2JGY3ABTBPFK3A, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.462770483 (506h25m8.333853538s) [noise=true , address=77.90.31.19:4844 , piece_id=AVEWXCOCGI7U437EYT37M4VY3TH73JESDRWAXRLHQKONPFPLAHTQ, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.474990001 (506h25m8.346073046s) [noise=true , address=45.91.116.6:4974 , piece_id=23YEXTSWEM6GOICKLZM5QVYLA7OK2HE55TIA52YG4YO26WBFK4LA, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.475014728 (506h25m8.346097793s) [seg_pos={0 0}, upload=444374]: Segment reached optimal threshold of 13 pieces. Dec 26 17:54:08.475260065 (506h25m8.34634311s) [noise=true , address=45.91.116.5:4297 , piece_id=I3IPV24ECC3YV4V3QRQDP2ZZT3JOO2CN6M6BISNPENMQZNSDKVAQ, seg_pos={0 0}, upload=444374]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 26 17:54:08.475266929 (506h25m8.346349964s) [noise=true , address=45.91.116.5:4297 , piece_id=I3IPV24ECC3YV4V3QRQDP2ZZT3JOO2CN6M6BISNPENMQZNSDKVAQ, seg_pos={0 0}, upload=444374]: Long tail cancellation due to optimal threshold Dec 26 17:54:08.475601164 (506h25m8.346684209s) [noise=true , address=77.90.31.14:4663 , piece_id=WOQS65FTAS6NTIR7QVGUIFPIJY6POSNUFZTBGYCDZW67TKWZ2GZQ, seg_pos={0 0}, upload=444374]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: expected piece hash; context canceled; EOF Dec 26 17:54:08.475628697 (506h25m8.346711732s) [noise=true , address=77.90.31.14:4663 , piece_id=WOQS65FTAS6NTIR7QVGUIFPIJY6POSNUFZTBGYCDZW67TKWZ2GZQ, seg_pos={0 0}, upload=444374]: Long tail cancellation due to optimal threshold Dec 26 17:54:08.475658875 (506h25m8.34674192s) [seg_pos={0 0}, upload=444374]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 17:54:08.475744387 (506h25m8.346827422s) [upload=444374]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:54:08.475750088 (506h25m8.346833133s) [upload=444374]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:54:08.767674804 (506h25m8.638757859s) [upload=444374]: Done uploading segments... Dec 26 17:54:08.767701635 (506h25m8.63878468s) [upload=444374]: Upload finished. err: <nil> Duration: 6.025777032s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 29.659075597s trace: https://jaeger.dev.storj.io/trace/665ea65f6e9ccc33 FAIL (2025-12-26 17:57:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== e=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789316979 (506h12m54.660400024s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789326507 (506h12m54.660409552s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789611339 (506h12m54.660694384s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789619275 (506h12m54.66070231s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789914006 (506h12m54.660997041s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789949443 (506h12m54.661032478s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790181566 (506h12m54.661264601s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790295632 (506h12m54.661378668s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790510872 (506h12m54.661593917s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790692980 (506h12m54.661776015s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790787379 (506h12m54.661870414s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790992970 (506h12m54.662076005s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791120053 (506h12m54.662203098s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791275218 (506h12m54.662358253s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791443709 (506h12m54.662526744s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:56.112313145 (506h12m55.98339619s) [upload=444191]: Next returned nil segment Dec 26 17:41:56.112330638 (506h12m55.983413683s) [upload=444191]: Waiting for error group managing segments... Dec 26 17:41:56.485322720 (506h12m56.356405765s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.485637079 (506h12m56.356720124s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.666807539 (506h12m56.537890584s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.678753987 (506h12m56.549837032s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.687788039 (506h12m56.558871074s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.694318023 (506h12m56.565401058s) [noise=true , address=145.79.227.7:4326 , piece_id=DMQ5RPI46X2U7TJTS42ZHQEF32BDJRKJJO5HWOYAPNS6VWVG5B5Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.716581187 (506h12m56.587664232s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.972183007 (506h12m56.843266042s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.011917999 (506h12m56.883001034s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.054168622 (506h12m56.925251667s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.114312777 (506h12m56.985395812s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.499846250 (506h12m57.370929285s) [noise=true , address=145.79.227.34:4919 , piece_id=YRWHW3BJVEGVJRSA6RJB3T7VJQSDTGKEZQKRICFIV2G3T7AN4MIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.612649066 (506h12m57.483732101s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649413733 (506h12m57.520496768s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649440282 (506h12m57.520523328s) [seg_pos={0 0}, upload=444191]: Segment reached optimal threshold of 14 pieces. Dec 26 17:41:57.649674729 (506h12m57.520757774s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G7ty71tPwe7cZSBNK9u74DFUCGomnxDsNvAif4C3SbP7HWwfo): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649684087 (506h12m57.520767142s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649747708 (506h12m57.520830763s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v6zXZuMmyL2MU8qb3q6gg1iwTYvndjzMjVpv8btEpeX9Lnqmu): protocol: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection Dec 26 17:41:57.649785219 (506h12m57.520868274s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649899828 (506h12m57.520982903s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649908775 (506h12m57.52099182s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650074641 (506h12m57.521157696s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.650083287 (506h12m57.521166342s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650132821 (506h12m57.521215866s) [seg_pos={0 0}, upload=444191]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 17:41:57.650220138 (506h12m57.521303183s) [upload=444191]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:41:57.650227171 (506h12m57.521310206s) [upload=444191]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:41:57.958890870 (506h12m57.829973915s) [upload=444191]: Done uploading segments... Dec 26 17:41:57.958911620 (506h12m57.829994665s) [upload=444191]: Upload finished. err: <nil> Duration: 29.659095604s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [X....X.X.XX...XX...XX....XX..XX......X.XXX.....XX.X.XXX.....] duration: 6.161584864s trace: https://jaeger.dev.storj.io/trace/66e1eaafd7692f5a FAIL (2025-12-26 17:57:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: VVU5B5KNVMOIGGZABQAAI22IQ2CZTWJURAJWQQ2JGY3ABTBPFK3A) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: AC62U7GFNGWMPNEFW5H2C4IMSFT6TSFNH5EHUCX24SWZXCPRKOQQ) context deadline exceeded ======LOGS====== Dec 26 17:54:05.860888315 (506h25m5.73197135s) [upload=444374]: Starting upload Dec 26 17:54:05.860960913 (506h25m5.732043958s) [upload=444374]: Done starting upload Dec 26 17:54:05.861243932 (506h25m5.732326967s) [upload=444374]: Uploading segments... Dec 26 17:54:05.861348571 (506h25m5.732431616s) [upload=444374]: Got next segment. Inline: false Dec 26 17:54:05.861399419 (506h25m5.732482464s) [upload=444374]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 17:54:05.861404999 (506h25m5.732488034s) [upload=444374]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 17:54:06.104522897 (506h25m5.975605933s) [seg_pos={0 0}, upload=444374]: Begin upload segment... Dec 26 17:54:06.104572262 (506h25m5.975655297s) [seg_pos={0 0}, upload=444374]: Done begin upload segment. Dec 26 17:54:06.104602710 (506h25m5.975685755s) [noise=true , address=77.90.31.12:4589 , piece_id=CJJUEAIEWP66VHOKZTYLIZUHN3OET6SAFJHXK73EBDFNB5QFE5RA, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.104713491 (506h25m5.975796536s) [noise=true , address=145.79.227.21:4475 , piece_id=UP7LFRPWG3GFBUFT6Y3TOMII7C2UE5BJEY7VCDKHWPYN7NHOL25A, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.105352147 (506h25m5.976435192s) [noise=true , address=145.79.227.51:4410 , piece_id=NJSE6VNZ2ICW5KYFAVC7TXUKOHPAQSHATMU2WW2XMXJF77VTZLVQ, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.105704788 (506h25m5.976787823s) [noise=true , address=77.90.31.15:4700 , piece_id=AC62U7GFNGWMPNEFW5H2C4IMSFT6TSFNH5EHUCX24SWZXCPRKOQQ, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.106020589 (506h25m5.977103624s) [noise=true , address=77.90.31.14:4663 , piece_id=WOQS65FTAS6NTIR7QVGUIFPIJY6POSNUFZTBGYCDZW67TKWZ2GZQ, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.106328646 (506h25m5.977411681s) [noise=true , address=145.79.227.42:4123 , piece_id=T5UOINXTMKBAACABHEVV5NGEMB52SGKWNSJPJ7Y6UKCQXNOUOJMA, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.106787600 (506h25m5.977870635s) [noise=true , address=45.91.116.6:4974 , piece_id=23YEXTSWEM6GOICKLZM5QVYLA7OK2HE55TIA52YG4YO26WBFK4LA, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.107122217 (506h25m5.978205252s) [noise=true , address=145.79.227.49:4339 , piece_id=NHMAMURTEDXM7SZDKB6OP4NU6SCIIKVLIHTVAOT2DVU7UAIT6V4Q, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.108570165 (506h25m5.97965324s) [noise=true , address=145.79.227.15:4874 , piece_id=7PRWX53PMTA2U7GAUIKHBWJXO6RANYJHXUIUFMG3UNASTUTNFIXQ, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.108781597 (506h25m5.979864633s) [noise=true , address=45.91.116.5:4297 , piece_id=I3IPV24ECC3YV4V3QRQDP2ZZT3JOO2CN6M6BISNPENMQZNSDKVAQ, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.110063228 (506h25m5.981146263s) [noise=true , address=145.79.227.11:4732 , piece_id=5FVE6U6ISMCIMDESG3OQ6SHFBGTWT2PPZZC7XRZAPSNK47HZIBWQ, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.111752484 (506h25m5.982835529s) [noise=true , address=145.79.227.36:4519 , piece_id=FJEF6GROIQF7AWQYSJHUEINR5FTBIO7USLJGFWEEEDRJSGW2JG4Q, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.113340689 (506h25m5.984423734s) [noise=true , address=145.79.227.10:4401 , piece_id=ZI4WZERG34DPXHUOLN6QZ55O6SJPZT5LPXPPUNMGGTYAZCAOTORA, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.114643860 (506h25m5.985726905s) [noise=true , address=45.91.116.13:4153 , piece_id=VVU5B5KNVMOIGGZABQAAI22IQ2CZTWJURAJWQQ2JGY3ABTBPFK3A, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.115852932 (506h25m5.986935967s) [noise=true , address=77.90.31.19:4844 , piece_id=AVEWXCOCGI7U437EYT37M4VY3TH73JESDRWAXRLHQKONPFPLAHTQ, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.116921628 (506h25m5.988004673s) [noise=true , address=145.79.227.39:4015 , piece_id=XD5CWP335WRWKXUP3WKMIO7VOFE5G4RN4EGONIRXB6HI43GOCJFQ, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:07.294144135 (506h25m7.16522718s) [noise=true , address=77.90.31.12:4589 , piece_id=CJJUEAIEWP66VHOKZTYLIZUHN3OET6SAFJHXK73EBDFNB5QFE5RA, seg_pos={0 0}, upload=444374]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 26 17:54:07.828164492 (506h25m7.699247537s) [upload=444374]: Next returned nil segment Dec 26 17:54:07.828181555 (506h25m7.6992646s) [upload=444374]: Waiting for error group managing segments... Dec 26 17:54:08.107862847 (506h25m7.978945892s) [noise=true , address=145.79.227.21:4475 , piece_id=UP7LFRPWG3GFBUFT6Y3TOMII7C2UE5BJEY7VCDKHWPYN7NHOL25A, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.112828612 (506h25m7.983911657s) [noise=true , address=145.79.227.51:4410 , piece_id=NJSE6VNZ2ICW5KYFAVC7TXUKOHPAQSHATMU2WW2XMXJF77VTZLVQ, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.119737338 (506h25m7.990820383s) [noise=true , address=145.79.227.10:4401 , piece_id=ZI4WZERG34DPXHUOLN6QZ55O6SJPZT5LPXPPUNMGGTYAZCAOTORA, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.121777863 (506h25m7.992860908s) [noise=true , address=145.79.227.11:4732 , piece_id=5FVE6U6ISMCIMDESG3OQ6SHFBGTWT2PPZZC7XRZAPSNK47HZIBWQ, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.130879434 (506h25m8.00196248s) [noise=true , address=145.79.227.39:4015 , piece_id=XD5CWP335WRWKXUP3WKMIO7VOFE5G4RN4EGONIRXB6HI43GOCJFQ, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.132848083 (506h25m8.003931118s) [noise=true , address=145.79.227.42:4123 , piece_id=T5UOINXTMKBAACABHEVV5NGEMB52SGKWNSJPJ7Y6UKCQXNOUOJMA, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.135816906 (506h25m8.006899952s) [noise=true , address=145.79.227.49:4339 , piece_id=NHMAMURTEDXM7SZDKB6OP4NU6SCIIKVLIHTVAOT2DVU7UAIT6V4Q, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.139078326 (506h25m8.010161361s) [noise=true , address=145.79.227.15:4874 , piece_id=7PRWX53PMTA2U7GAUIKHBWJXO6RANYJHXUIUFMG3UNASTUTNFIXQ, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.139833234 (506h25m8.010916279s) [noise=true , address=145.79.227.36:4519 , piece_id=FJEF6GROIQF7AWQYSJHUEINR5FTBIO7USLJGFWEEEDRJSGW2JG4Q, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.299232928 (506h25m8.170315963s) [noise=true , address=77.90.31.15:4700 , piece_id=AC62U7GFNGWMPNEFW5H2C4IMSFT6TSFNH5EHUCX24SWZXCPRKOQQ, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.362334064 (506h25m8.233417109s) [noise=true , address=45.91.116.13:4153 , piece_id=VVU5B5KNVMOIGGZABQAAI22IQ2CZTWJURAJWQQ2JGY3ABTBPFK3A, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.462770483 (506h25m8.333853538s) [noise=true , address=77.90.31.19:4844 , piece_id=AVEWXCOCGI7U437EYT37M4VY3TH73JESDRWAXRLHQKONPFPLAHTQ, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.474990001 (506h25m8.346073046s) [noise=true , address=45.91.116.6:4974 , piece_id=23YEXTSWEM6GOICKLZM5QVYLA7OK2HE55TIA52YG4YO26WBFK4LA, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.475014728 (506h25m8.346097793s) [seg_pos={0 0}, upload=444374]: Segment reached optimal threshold of 13 pieces. Dec 26 17:54:08.475260065 (506h25m8.34634311s) [noise=true , address=45.91.116.5:4297 , piece_id=I3IPV24ECC3YV4V3QRQDP2ZZT3JOO2CN6M6BISNPENMQZNSDKVAQ, seg_pos={0 0}, upload=444374]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 26 17:54:08.475266929 (506h25m8.346349964s) [noise=true , address=45.91.116.5:4297 , piece_id=I3IPV24ECC3YV4V3QRQDP2ZZT3JOO2CN6M6BISNPENMQZNSDKVAQ, seg_pos={0 0}, upload=444374]: Long tail cancellation due to optimal threshold Dec 26 17:54:08.475601164 (506h25m8.346684209s) [noise=true , address=77.90.31.14:4663 , piece_id=WOQS65FTAS6NTIR7QVGUIFPIJY6POSNUFZTBGYCDZW67TKWZ2GZQ, seg_pos={0 0}, upload=444374]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: expected piece hash; context canceled; EOF Dec 26 17:54:08.475628697 (506h25m8.346711732s) [noise=true , address=77.90.31.14:4663 , piece_id=WOQS65FTAS6NTIR7QVGUIFPIJY6POSNUFZTBGYCDZW67TKWZ2GZQ, seg_pos={0 0}, upload=444374]: Long tail cancellation due to optimal threshold Dec 26 17:54:08.475658875 (506h25m8.34674192s) [seg_pos={0 0}, upload=444374]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 17:54:08.475744387 (506h25m8.346827422s) [upload=444374]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:54:08.475750088 (506h25m8.346833133s) [upload=444374]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:54:08.767674804 (506h25m8.638757859s) [upload=444374]: Done uploading segments... Dec 26 17:54:08.767701635 (506h25m8.63878468s) [upload=444374]: Upload finished. err: <nil> Duration: 6.161609771s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [.X....X.X.XX...XX...XX....XX..XX......X.XXX.....XX.X.XXX....] duration: 6.111093157s trace: https://jaeger.dev.storj.io/trace/66e1eaafd7692f5a FAIL (2025-12-26 17:57:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: VVU5B5KNVMOIGGZABQAAI22IQ2CZTWJURAJWQQ2JGY3ABTBPFK3A) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: AC62U7GFNGWMPNEFW5H2C4IMSFT6TSFNH5EHUCX24SWZXCPRKOQQ) context deadline exceeded ======LOGS====== Dec 26 17:54:05.860888315 (506h25m5.73197135s) [upload=444374]: Starting upload Dec 26 17:54:05.860960913 (506h25m5.732043958s) [upload=444374]: Done starting upload Dec 26 17:54:05.861243932 (506h25m5.732326967s) [upload=444374]: Uploading segments... Dec 26 17:54:05.861348571 (506h25m5.732431616s) [upload=444374]: Got next segment. Inline: false Dec 26 17:54:05.861399419 (506h25m5.732482464s) [upload=444374]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 17:54:05.861404999 (506h25m5.732488034s) [upload=444374]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 17:54:06.104522897 (506h25m5.975605933s) [seg_pos={0 0}, upload=444374]: Begin upload segment... Dec 26 17:54:06.104572262 (506h25m5.975655297s) [seg_pos={0 0}, upload=444374]: Done begin upload segment. Dec 26 17:54:06.104602710 (506h25m5.975685755s) [noise=true , address=77.90.31.12:4589 , piece_id=CJJUEAIEWP66VHOKZTYLIZUHN3OET6SAFJHXK73EBDFNB5QFE5RA, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.104713491 (506h25m5.975796536s) [noise=true , address=145.79.227.21:4475 , piece_id=UP7LFRPWG3GFBUFT6Y3TOMII7C2UE5BJEY7VCDKHWPYN7NHOL25A, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.105352147 (506h25m5.976435192s) [noise=true , address=145.79.227.51:4410 , piece_id=NJSE6VNZ2ICW5KYFAVC7TXUKOHPAQSHATMU2WW2XMXJF77VTZLVQ, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.105704788 (506h25m5.976787823s) [noise=true , address=77.90.31.15:4700 , piece_id=AC62U7GFNGWMPNEFW5H2C4IMSFT6TSFNH5EHUCX24SWZXCPRKOQQ, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.106020589 (506h25m5.977103624s) [noise=true , address=77.90.31.14:4663 , piece_id=WOQS65FTAS6NTIR7QVGUIFPIJY6POSNUFZTBGYCDZW67TKWZ2GZQ, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.106328646 (506h25m5.977411681s) [noise=true , address=145.79.227.42:4123 , piece_id=T5UOINXTMKBAACABHEVV5NGEMB52SGKWNSJPJ7Y6UKCQXNOUOJMA, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.106787600 (506h25m5.977870635s) [noise=true , address=45.91.116.6:4974 , piece_id=23YEXTSWEM6GOICKLZM5QVYLA7OK2HE55TIA52YG4YO26WBFK4LA, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.107122217 (506h25m5.978205252s) [noise=true , address=145.79.227.49:4339 , piece_id=NHMAMURTEDXM7SZDKB6OP4NU6SCIIKVLIHTVAOT2DVU7UAIT6V4Q, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.108570165 (506h25m5.97965324s) [noise=true , address=145.79.227.15:4874 , piece_id=7PRWX53PMTA2U7GAUIKHBWJXO6RANYJHXUIUFMG3UNASTUTNFIXQ, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.108781597 (506h25m5.979864633s) [noise=true , address=45.91.116.5:4297 , piece_id=I3IPV24ECC3YV4V3QRQDP2ZZT3JOO2CN6M6BISNPENMQZNSDKVAQ, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.110063228 (506h25m5.981146263s) [noise=true , address=145.79.227.11:4732 , piece_id=5FVE6U6ISMCIMDESG3OQ6SHFBGTWT2PPZZC7XRZAPSNK47HZIBWQ, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.111752484 (506h25m5.982835529s) [noise=true , address=145.79.227.36:4519 , piece_id=FJEF6GROIQF7AWQYSJHUEINR5FTBIO7USLJGFWEEEDRJSGW2JG4Q, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.113340689 (506h25m5.984423734s) [noise=true , address=145.79.227.10:4401 , piece_id=ZI4WZERG34DPXHUOLN6QZ55O6SJPZT5LPXPPUNMGGTYAZCAOTORA, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.114643860 (506h25m5.985726905s) [noise=true , address=45.91.116.13:4153 , piece_id=VVU5B5KNVMOIGGZABQAAI22IQ2CZTWJURAJWQQ2JGY3ABTBPFK3A, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.115852932 (506h25m5.986935967s) [noise=true , address=77.90.31.19:4844 , piece_id=AVEWXCOCGI7U437EYT37M4VY3TH73JESDRWAXRLHQKONPFPLAHTQ, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.116921628 (506h25m5.988004673s) [noise=true , address=145.79.227.39:4015 , piece_id=XD5CWP335WRWKXUP3WKMIO7VOFE5G4RN4EGONIRXB6HI43GOCJFQ, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:07.294144135 (506h25m7.16522718s) [noise=true , address=77.90.31.12:4589 , piece_id=CJJUEAIEWP66VHOKZTYLIZUHN3OET6SAFJHXK73EBDFNB5QFE5RA, seg_pos={0 0}, upload=444374]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 26 17:54:07.828164492 (506h25m7.699247537s) [upload=444374]: Next returned nil segment Dec 26 17:54:07.828181555 (506h25m7.6992646s) [upload=444374]: Waiting for error group managing segments... Dec 26 17:54:08.107862847 (506h25m7.978945892s) [noise=true , address=145.79.227.21:4475 , piece_id=UP7LFRPWG3GFBUFT6Y3TOMII7C2UE5BJEY7VCDKHWPYN7NHOL25A, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.112828612 (506h25m7.983911657s) [noise=true , address=145.79.227.51:4410 , piece_id=NJSE6VNZ2ICW5KYFAVC7TXUKOHPAQSHATMU2WW2XMXJF77VTZLVQ, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.119737338 (506h25m7.990820383s) [noise=true , address=145.79.227.10:4401 , piece_id=ZI4WZERG34DPXHUOLN6QZ55O6SJPZT5LPXPPUNMGGTYAZCAOTORA, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.121777863 (506h25m7.992860908s) [noise=true , address=145.79.227.11:4732 , piece_id=5FVE6U6ISMCIMDESG3OQ6SHFBGTWT2PPZZC7XRZAPSNK47HZIBWQ, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.130879434 (506h25m8.00196248s) [noise=true , address=145.79.227.39:4015 , piece_id=XD5CWP335WRWKXUP3WKMIO7VOFE5G4RN4EGONIRXB6HI43GOCJFQ, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.132848083 (506h25m8.003931118s) [noise=true , address=145.79.227.42:4123 , piece_id=T5UOINXTMKBAACABHEVV5NGEMB52SGKWNSJPJ7Y6UKCQXNOUOJMA, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.135816906 (506h25m8.006899952s) [noise=true , address=145.79.227.49:4339 , piece_id=NHMAMURTEDXM7SZDKB6OP4NU6SCIIKVLIHTVAOT2DVU7UAIT6V4Q, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.139078326 (506h25m8.010161361s) [noise=true , address=145.79.227.15:4874 , piece_id=7PRWX53PMTA2U7GAUIKHBWJXO6RANYJHXUIUFMG3UNASTUTNFIXQ, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.139833234 (506h25m8.010916279s) [noise=true , address=145.79.227.36:4519 , piece_id=FJEF6GROIQF7AWQYSJHUEINR5FTBIO7USLJGFWEEEDRJSGW2JG4Q, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.299232928 (506h25m8.170315963s) [noise=true , address=77.90.31.15:4700 , piece_id=AC62U7GFNGWMPNEFW5H2C4IMSFT6TSFNH5EHUCX24SWZXCPRKOQQ, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.362334064 (506h25m8.233417109s) [noise=true , address=45.91.116.13:4153 , piece_id=VVU5B5KNVMOIGGZABQAAI22IQ2CZTWJURAJWQQ2JGY3ABTBPFK3A, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.462770483 (506h25m8.333853538s) [noise=true , address=77.90.31.19:4844 , piece_id=AVEWXCOCGI7U437EYT37M4VY3TH73JESDRWAXRLHQKONPFPLAHTQ, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.474990001 (506h25m8.346073046s) [noise=true , address=45.91.116.6:4974 , piece_id=23YEXTSWEM6GOICKLZM5QVYLA7OK2HE55TIA52YG4YO26WBFK4LA, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.475014728 (506h25m8.346097793s) [seg_pos={0 0}, upload=444374]: Segment reached optimal threshold of 13 pieces. Dec 26 17:54:08.475260065 (506h25m8.34634311s) [noise=true , address=45.91.116.5:4297 , piece_id=I3IPV24ECC3YV4V3QRQDP2ZZT3JOO2CN6M6BISNPENMQZNSDKVAQ, seg_pos={0 0}, upload=444374]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 26 17:54:08.475266929 (506h25m8.346349964s) [noise=true , address=45.91.116.5:4297 , piece_id=I3IPV24ECC3YV4V3QRQDP2ZZT3JOO2CN6M6BISNPENMQZNSDKVAQ, seg_pos={0 0}, upload=444374]: Long tail cancellation due to optimal threshold Dec 26 17:54:08.475601164 (506h25m8.346684209s) [noise=true , address=77.90.31.14:4663 , piece_id=WOQS65FTAS6NTIR7QVGUIFPIJY6POSNUFZTBGYCDZW67TKWZ2GZQ, seg_pos={0 0}, upload=444374]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: expected piece hash; context canceled; EOF Dec 26 17:54:08.475628697 (506h25m8.346711732s) [noise=true , address=77.90.31.14:4663 , piece_id=WOQS65FTAS6NTIR7QVGUIFPIJY6POSNUFZTBGYCDZW67TKWZ2GZQ, seg_pos={0 0}, upload=444374]: Long tail cancellation due to optimal threshold Dec 26 17:54:08.475658875 (506h25m8.34674192s) [seg_pos={0 0}, upload=444374]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 17:54:08.475744387 (506h25m8.346827422s) [upload=444374]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:54:08.475750088 (506h25m8.346833133s) [upload=444374]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:54:08.767674804 (506h25m8.638757859s) [upload=444374]: Done uploading segments... Dec 26 17:54:08.767701635 (506h25m8.63878468s) [upload=444374]: Upload finished. err: <nil> Duration: 6.111127062s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 15.416782194s trace: https://jaeger.dev.storj.io/trace/665ea65f6e9ccc33 FAIL (2025-12-26 17:56:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== e=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789316979 (506h12m54.660400024s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789326507 (506h12m54.660409552s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789611339 (506h12m54.660694384s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789619275 (506h12m54.66070231s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789914006 (506h12m54.660997041s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789949443 (506h12m54.661032478s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790181566 (506h12m54.661264601s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790295632 (506h12m54.661378668s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790510872 (506h12m54.661593917s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790692980 (506h12m54.661776015s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790787379 (506h12m54.661870414s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790992970 (506h12m54.662076005s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791120053 (506h12m54.662203098s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791275218 (506h12m54.662358253s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791443709 (506h12m54.662526744s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:56.112313145 (506h12m55.98339619s) [upload=444191]: Next returned nil segment Dec 26 17:41:56.112330638 (506h12m55.983413683s) [upload=444191]: Waiting for error group managing segments... Dec 26 17:41:56.485322720 (506h12m56.356405765s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.485637079 (506h12m56.356720124s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.666807539 (506h12m56.537890584s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.678753987 (506h12m56.549837032s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.687788039 (506h12m56.558871074s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.694318023 (506h12m56.565401058s) [noise=true , address=145.79.227.7:4326 , piece_id=DMQ5RPI46X2U7TJTS42ZHQEF32BDJRKJJO5HWOYAPNS6VWVG5B5Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.716581187 (506h12m56.587664232s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.972183007 (506h12m56.843266042s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.011917999 (506h12m56.883001034s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.054168622 (506h12m56.925251667s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.114312777 (506h12m56.985395812s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.499846250 (506h12m57.370929285s) [noise=true , address=145.79.227.34:4919 , piece_id=YRWHW3BJVEGVJRSA6RJB3T7VJQSDTGKEZQKRICFIV2G3T7AN4MIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.612649066 (506h12m57.483732101s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649413733 (506h12m57.520496768s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649440282 (506h12m57.520523328s) [seg_pos={0 0}, upload=444191]: Segment reached optimal threshold of 14 pieces. Dec 26 17:41:57.649674729 (506h12m57.520757774s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G7ty71tPwe7cZSBNK9u74DFUCGomnxDsNvAif4C3SbP7HWwfo): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649684087 (506h12m57.520767142s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649747708 (506h12m57.520830763s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v6zXZuMmyL2MU8qb3q6gg1iwTYvndjzMjVpv8btEpeX9Lnqmu): protocol: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection Dec 26 17:41:57.649785219 (506h12m57.520868274s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649899828 (506h12m57.520982903s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649908775 (506h12m57.52099182s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650074641 (506h12m57.521157696s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.650083287 (506h12m57.521166342s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650132821 (506h12m57.521215866s) [seg_pos={0 0}, upload=444191]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 17:41:57.650220138 (506h12m57.521303183s) [upload=444191]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:41:57.650227171 (506h12m57.521310206s) [upload=444191]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:41:57.958890870 (506h12m57.829973915s) [upload=444191]: Done uploading segments... Dec 26 17:41:57.958911620 (506h12m57.829994665s) [upload=444191]: Upload finished. err: <nil> Duration: 15.416811441s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [..X....X.X.XX...XX...XX....XX..XX......X.XXX.....XX.X.XXX...] duration: 5.786459418s trace: https://jaeger.dev.storj.io/trace/66e1eaafd7692f5a FAIL (2025-12-26 17:56:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: VVU5B5KNVMOIGGZABQAAI22IQ2CZTWJURAJWQQ2JGY3ABTBPFK3A) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: AC62U7GFNGWMPNEFW5H2C4IMSFT6TSFNH5EHUCX24SWZXCPRKOQQ) context deadline exceeded ======LOGS====== Dec 26 17:54:05.860888315 (506h25m5.73197135s) [upload=444374]: Starting upload Dec 26 17:54:05.860960913 (506h25m5.732043958s) [upload=444374]: Done starting upload Dec 26 17:54:05.861243932 (506h25m5.732326967s) [upload=444374]: Uploading segments... Dec 26 17:54:05.861348571 (506h25m5.732431616s) [upload=444374]: Got next segment. Inline: false Dec 26 17:54:05.861399419 (506h25m5.732482464s) [upload=444374]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 17:54:05.861404999 (506h25m5.732488034s) [upload=444374]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 17:54:06.104522897 (506h25m5.975605933s) [seg_pos={0 0}, upload=444374]: Begin upload segment... Dec 26 17:54:06.104572262 (506h25m5.975655297s) [seg_pos={0 0}, upload=444374]: Done begin upload segment. Dec 26 17:54:06.104602710 (506h25m5.975685755s) [noise=true , address=77.90.31.12:4589 , piece_id=CJJUEAIEWP66VHOKZTYLIZUHN3OET6SAFJHXK73EBDFNB5QFE5RA, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.104713491 (506h25m5.975796536s) [noise=true , address=145.79.227.21:4475 , piece_id=UP7LFRPWG3GFBUFT6Y3TOMII7C2UE5BJEY7VCDKHWPYN7NHOL25A, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.105352147 (506h25m5.976435192s) [noise=true , address=145.79.227.51:4410 , piece_id=NJSE6VNZ2ICW5KYFAVC7TXUKOHPAQSHATMU2WW2XMXJF77VTZLVQ, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.105704788 (506h25m5.976787823s) [noise=true , address=77.90.31.15:4700 , piece_id=AC62U7GFNGWMPNEFW5H2C4IMSFT6TSFNH5EHUCX24SWZXCPRKOQQ, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.106020589 (506h25m5.977103624s) [noise=true , address=77.90.31.14:4663 , piece_id=WOQS65FTAS6NTIR7QVGUIFPIJY6POSNUFZTBGYCDZW67TKWZ2GZQ, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.106328646 (506h25m5.977411681s) [noise=true , address=145.79.227.42:4123 , piece_id=T5UOINXTMKBAACABHEVV5NGEMB52SGKWNSJPJ7Y6UKCQXNOUOJMA, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.106787600 (506h25m5.977870635s) [noise=true , address=45.91.116.6:4974 , piece_id=23YEXTSWEM6GOICKLZM5QVYLA7OK2HE55TIA52YG4YO26WBFK4LA, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.107122217 (506h25m5.978205252s) [noise=true , address=145.79.227.49:4339 , piece_id=NHMAMURTEDXM7SZDKB6OP4NU6SCIIKVLIHTVAOT2DVU7UAIT6V4Q, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.108570165 (506h25m5.97965324s) [noise=true , address=145.79.227.15:4874 , piece_id=7PRWX53PMTA2U7GAUIKHBWJXO6RANYJHXUIUFMG3UNASTUTNFIXQ, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.108781597 (506h25m5.979864633s) [noise=true , address=45.91.116.5:4297 , piece_id=I3IPV24ECC3YV4V3QRQDP2ZZT3JOO2CN6M6BISNPENMQZNSDKVAQ, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.110063228 (506h25m5.981146263s) [noise=true , address=145.79.227.11:4732 , piece_id=5FVE6U6ISMCIMDESG3OQ6SHFBGTWT2PPZZC7XRZAPSNK47HZIBWQ, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.111752484 (506h25m5.982835529s) [noise=true , address=145.79.227.36:4519 , piece_id=FJEF6GROIQF7AWQYSJHUEINR5FTBIO7USLJGFWEEEDRJSGW2JG4Q, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.113340689 (506h25m5.984423734s) [noise=true , address=145.79.227.10:4401 , piece_id=ZI4WZERG34DPXHUOLN6QZ55O6SJPZT5LPXPPUNMGGTYAZCAOTORA, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.114643860 (506h25m5.985726905s) [noise=true , address=45.91.116.13:4153 , piece_id=VVU5B5KNVMOIGGZABQAAI22IQ2CZTWJURAJWQQ2JGY3ABTBPFK3A, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.115852932 (506h25m5.986935967s) [noise=true , address=77.90.31.19:4844 , piece_id=AVEWXCOCGI7U437EYT37M4VY3TH73JESDRWAXRLHQKONPFPLAHTQ, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.116921628 (506h25m5.988004673s) [noise=true , address=145.79.227.39:4015 , piece_id=XD5CWP335WRWKXUP3WKMIO7VOFE5G4RN4EGONIRXB6HI43GOCJFQ, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:07.294144135 (506h25m7.16522718s) [noise=true , address=77.90.31.12:4589 , piece_id=CJJUEAIEWP66VHOKZTYLIZUHN3OET6SAFJHXK73EBDFNB5QFE5RA, seg_pos={0 0}, upload=444374]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 26 17:54:07.828164492 (506h25m7.699247537s) [upload=444374]: Next returned nil segment Dec 26 17:54:07.828181555 (506h25m7.6992646s) [upload=444374]: Waiting for error group managing segments... Dec 26 17:54:08.107862847 (506h25m7.978945892s) [noise=true , address=145.79.227.21:4475 , piece_id=UP7LFRPWG3GFBUFT6Y3TOMII7C2UE5BJEY7VCDKHWPYN7NHOL25A, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.112828612 (506h25m7.983911657s) [noise=true , address=145.79.227.51:4410 , piece_id=NJSE6VNZ2ICW5KYFAVC7TXUKOHPAQSHATMU2WW2XMXJF77VTZLVQ, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.119737338 (506h25m7.990820383s) [noise=true , address=145.79.227.10:4401 , piece_id=ZI4WZERG34DPXHUOLN6QZ55O6SJPZT5LPXPPUNMGGTYAZCAOTORA, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.121777863 (506h25m7.992860908s) [noise=true , address=145.79.227.11:4732 , piece_id=5FVE6U6ISMCIMDESG3OQ6SHFBGTWT2PPZZC7XRZAPSNK47HZIBWQ, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.130879434 (506h25m8.00196248s) [noise=true , address=145.79.227.39:4015 , piece_id=XD5CWP335WRWKXUP3WKMIO7VOFE5G4RN4EGONIRXB6HI43GOCJFQ, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.132848083 (506h25m8.003931118s) [noise=true , address=145.79.227.42:4123 , piece_id=T5UOINXTMKBAACABHEVV5NGEMB52SGKWNSJPJ7Y6UKCQXNOUOJMA, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.135816906 (506h25m8.006899952s) [noise=true , address=145.79.227.49:4339 , piece_id=NHMAMURTEDXM7SZDKB6OP4NU6SCIIKVLIHTVAOT2DVU7UAIT6V4Q, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.139078326 (506h25m8.010161361s) [noise=true , address=145.79.227.15:4874 , piece_id=7PRWX53PMTA2U7GAUIKHBWJXO6RANYJHXUIUFMG3UNASTUTNFIXQ, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.139833234 (506h25m8.010916279s) [noise=true , address=145.79.227.36:4519 , piece_id=FJEF6GROIQF7AWQYSJHUEINR5FTBIO7USLJGFWEEEDRJSGW2JG4Q, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.299232928 (506h25m8.170315963s) [noise=true , address=77.90.31.15:4700 , piece_id=AC62U7GFNGWMPNEFW5H2C4IMSFT6TSFNH5EHUCX24SWZXCPRKOQQ, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.362334064 (506h25m8.233417109s) [noise=true , address=45.91.116.13:4153 , piece_id=VVU5B5KNVMOIGGZABQAAI22IQ2CZTWJURAJWQQ2JGY3ABTBPFK3A, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.462770483 (506h25m8.333853538s) [noise=true , address=77.90.31.19:4844 , piece_id=AVEWXCOCGI7U437EYT37M4VY3TH73JESDRWAXRLHQKONPFPLAHTQ, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.474990001 (506h25m8.346073046s) [noise=true , address=45.91.116.6:4974 , piece_id=23YEXTSWEM6GOICKLZM5QVYLA7OK2HE55TIA52YG4YO26WBFK4LA, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.475014728 (506h25m8.346097793s) [seg_pos={0 0}, upload=444374]: Segment reached optimal threshold of 13 pieces. Dec 26 17:54:08.475260065 (506h25m8.34634311s) [noise=true , address=45.91.116.5:4297 , piece_id=I3IPV24ECC3YV4V3QRQDP2ZZT3JOO2CN6M6BISNPENMQZNSDKVAQ, seg_pos={0 0}, upload=444374]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 26 17:54:08.475266929 (506h25m8.346349964s) [noise=true , address=45.91.116.5:4297 , piece_id=I3IPV24ECC3YV4V3QRQDP2ZZT3JOO2CN6M6BISNPENMQZNSDKVAQ, seg_pos={0 0}, upload=444374]: Long tail cancellation due to optimal threshold Dec 26 17:54:08.475601164 (506h25m8.346684209s) [noise=true , address=77.90.31.14:4663 , piece_id=WOQS65FTAS6NTIR7QVGUIFPIJY6POSNUFZTBGYCDZW67TKWZ2GZQ, seg_pos={0 0}, upload=444374]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: expected piece hash; context canceled; EOF Dec 26 17:54:08.475628697 (506h25m8.346711732s) [noise=true , address=77.90.31.14:4663 , piece_id=WOQS65FTAS6NTIR7QVGUIFPIJY6POSNUFZTBGYCDZW67TKWZ2GZQ, seg_pos={0 0}, upload=444374]: Long tail cancellation due to optimal threshold Dec 26 17:54:08.475658875 (506h25m8.34674192s) [seg_pos={0 0}, upload=444374]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 17:54:08.475744387 (506h25m8.346827422s) [upload=444374]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:54:08.475750088 (506h25m8.346833133s) [upload=444374]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:54:08.767674804 (506h25m8.638757859s) [upload=444374]: Done uploading segments... Dec 26 17:54:08.767701635 (506h25m8.63878468s) [upload=444374]: Upload finished. err: <nil> Duration: 5.786489808s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 35.17438565s trace: https://jaeger.dev.storj.io/trace/665ea65f6e9ccc33 FAIL (2025-12-26 17:55:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== e=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789316979 (506h12m54.660400024s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789326507 (506h12m54.660409552s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789611339 (506h12m54.660694384s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789619275 (506h12m54.66070231s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789914006 (506h12m54.660997041s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789949443 (506h12m54.661032478s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790181566 (506h12m54.661264601s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790295632 (506h12m54.661378668s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790510872 (506h12m54.661593917s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790692980 (506h12m54.661776015s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790787379 (506h12m54.661870414s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790992970 (506h12m54.662076005s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791120053 (506h12m54.662203098s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791275218 (506h12m54.662358253s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791443709 (506h12m54.662526744s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:56.112313145 (506h12m55.98339619s) [upload=444191]: Next returned nil segment Dec 26 17:41:56.112330638 (506h12m55.983413683s) [upload=444191]: Waiting for error group managing segments... Dec 26 17:41:56.485322720 (506h12m56.356405765s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.485637079 (506h12m56.356720124s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.666807539 (506h12m56.537890584s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.678753987 (506h12m56.549837032s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.687788039 (506h12m56.558871074s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.694318023 (506h12m56.565401058s) [noise=true , address=145.79.227.7:4326 , piece_id=DMQ5RPI46X2U7TJTS42ZHQEF32BDJRKJJO5HWOYAPNS6VWVG5B5Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.716581187 (506h12m56.587664232s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.972183007 (506h12m56.843266042s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.011917999 (506h12m56.883001034s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.054168622 (506h12m56.925251667s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.114312777 (506h12m56.985395812s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.499846250 (506h12m57.370929285s) [noise=true , address=145.79.227.34:4919 , piece_id=YRWHW3BJVEGVJRSA6RJB3T7VJQSDTGKEZQKRICFIV2G3T7AN4MIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.612649066 (506h12m57.483732101s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649413733 (506h12m57.520496768s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649440282 (506h12m57.520523328s) [seg_pos={0 0}, upload=444191]: Segment reached optimal threshold of 14 pieces. Dec 26 17:41:57.649674729 (506h12m57.520757774s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G7ty71tPwe7cZSBNK9u74DFUCGomnxDsNvAif4C3SbP7HWwfo): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649684087 (506h12m57.520767142s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649747708 (506h12m57.520830763s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v6zXZuMmyL2MU8qb3q6gg1iwTYvndjzMjVpv8btEpeX9Lnqmu): protocol: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection Dec 26 17:41:57.649785219 (506h12m57.520868274s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649899828 (506h12m57.520982903s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649908775 (506h12m57.52099182s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650074641 (506h12m57.521157696s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.650083287 (506h12m57.521166342s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650132821 (506h12m57.521215866s) [seg_pos={0 0}, upload=444191]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 17:41:57.650220138 (506h12m57.521303183s) [upload=444191]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:41:57.650227171 (506h12m57.521310206s) [upload=444191]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:41:57.958890870 (506h12m57.829973915s) [upload=444191]: Done uploading segments... Dec 26 17:41:57.958911620 (506h12m57.829994665s) [upload=444191]: Upload finished. err: <nil> Duration: 35.174400509s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [X..X....X.X.XX...XX...XX....XX..XX......X.XXX.....XX.X.XXX..] duration: 5.77130468s trace: https://jaeger.dev.storj.io/trace/66e1eaafd7692f5a FAIL (2025-12-26 17:55:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: VVU5B5KNVMOIGGZABQAAI22IQ2CZTWJURAJWQQ2JGY3ABTBPFK3A) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: AC62U7GFNGWMPNEFW5H2C4IMSFT6TSFNH5EHUCX24SWZXCPRKOQQ) context deadline exceeded ======LOGS====== Dec 26 17:54:05.860888315 (506h25m5.73197135s) [upload=444374]: Starting upload Dec 26 17:54:05.860960913 (506h25m5.732043958s) [upload=444374]: Done starting upload Dec 26 17:54:05.861243932 (506h25m5.732326967s) [upload=444374]: Uploading segments... Dec 26 17:54:05.861348571 (506h25m5.732431616s) [upload=444374]: Got next segment. Inline: false Dec 26 17:54:05.861399419 (506h25m5.732482464s) [upload=444374]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 17:54:05.861404999 (506h25m5.732488034s) [upload=444374]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 17:54:06.104522897 (506h25m5.975605933s) [seg_pos={0 0}, upload=444374]: Begin upload segment... Dec 26 17:54:06.104572262 (506h25m5.975655297s) [seg_pos={0 0}, upload=444374]: Done begin upload segment. Dec 26 17:54:06.104602710 (506h25m5.975685755s) [noise=true , address=77.90.31.12:4589 , piece_id=CJJUEAIEWP66VHOKZTYLIZUHN3OET6SAFJHXK73EBDFNB5QFE5RA, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.104713491 (506h25m5.975796536s) [noise=true , address=145.79.227.21:4475 , piece_id=UP7LFRPWG3GFBUFT6Y3TOMII7C2UE5BJEY7VCDKHWPYN7NHOL25A, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.105352147 (506h25m5.976435192s) [noise=true , address=145.79.227.51:4410 , piece_id=NJSE6VNZ2ICW5KYFAVC7TXUKOHPAQSHATMU2WW2XMXJF77VTZLVQ, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.105704788 (506h25m5.976787823s) [noise=true , address=77.90.31.15:4700 , piece_id=AC62U7GFNGWMPNEFW5H2C4IMSFT6TSFNH5EHUCX24SWZXCPRKOQQ, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.106020589 (506h25m5.977103624s) [noise=true , address=77.90.31.14:4663 , piece_id=WOQS65FTAS6NTIR7QVGUIFPIJY6POSNUFZTBGYCDZW67TKWZ2GZQ, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.106328646 (506h25m5.977411681s) [noise=true , address=145.79.227.42:4123 , piece_id=T5UOINXTMKBAACABHEVV5NGEMB52SGKWNSJPJ7Y6UKCQXNOUOJMA, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.106787600 (506h25m5.977870635s) [noise=true , address=45.91.116.6:4974 , piece_id=23YEXTSWEM6GOICKLZM5QVYLA7OK2HE55TIA52YG4YO26WBFK4LA, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.107122217 (506h25m5.978205252s) [noise=true , address=145.79.227.49:4339 , piece_id=NHMAMURTEDXM7SZDKB6OP4NU6SCIIKVLIHTVAOT2DVU7UAIT6V4Q, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.108570165 (506h25m5.97965324s) [noise=true , address=145.79.227.15:4874 , piece_id=7PRWX53PMTA2U7GAUIKHBWJXO6RANYJHXUIUFMG3UNASTUTNFIXQ, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.108781597 (506h25m5.979864633s) [noise=true , address=45.91.116.5:4297 , piece_id=I3IPV24ECC3YV4V3QRQDP2ZZT3JOO2CN6M6BISNPENMQZNSDKVAQ, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.110063228 (506h25m5.981146263s) [noise=true , address=145.79.227.11:4732 , piece_id=5FVE6U6ISMCIMDESG3OQ6SHFBGTWT2PPZZC7XRZAPSNK47HZIBWQ, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.111752484 (506h25m5.982835529s) [noise=true , address=145.79.227.36:4519 , piece_id=FJEF6GROIQF7AWQYSJHUEINR5FTBIO7USLJGFWEEEDRJSGW2JG4Q, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.113340689 (506h25m5.984423734s) [noise=true , address=145.79.227.10:4401 , piece_id=ZI4WZERG34DPXHUOLN6QZ55O6SJPZT5LPXPPUNMGGTYAZCAOTORA, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.114643860 (506h25m5.985726905s) [noise=true , address=45.91.116.13:4153 , piece_id=VVU5B5KNVMOIGGZABQAAI22IQ2CZTWJURAJWQQ2JGY3ABTBPFK3A, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.115852932 (506h25m5.986935967s) [noise=true , address=77.90.31.19:4844 , piece_id=AVEWXCOCGI7U437EYT37M4VY3TH73JESDRWAXRLHQKONPFPLAHTQ, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.116921628 (506h25m5.988004673s) [noise=true , address=145.79.227.39:4015 , piece_id=XD5CWP335WRWKXUP3WKMIO7VOFE5G4RN4EGONIRXB6HI43GOCJFQ, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:07.294144135 (506h25m7.16522718s) [noise=true , address=77.90.31.12:4589 , piece_id=CJJUEAIEWP66VHOKZTYLIZUHN3OET6SAFJHXK73EBDFNB5QFE5RA, seg_pos={0 0}, upload=444374]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 26 17:54:07.828164492 (506h25m7.699247537s) [upload=444374]: Next returned nil segment Dec 26 17:54:07.828181555 (506h25m7.6992646s) [upload=444374]: Waiting for error group managing segments... Dec 26 17:54:08.107862847 (506h25m7.978945892s) [noise=true , address=145.79.227.21:4475 , piece_id=UP7LFRPWG3GFBUFT6Y3TOMII7C2UE5BJEY7VCDKHWPYN7NHOL25A, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.112828612 (506h25m7.983911657s) [noise=true , address=145.79.227.51:4410 , piece_id=NJSE6VNZ2ICW5KYFAVC7TXUKOHPAQSHATMU2WW2XMXJF77VTZLVQ, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.119737338 (506h25m7.990820383s) [noise=true , address=145.79.227.10:4401 , piece_id=ZI4WZERG34DPXHUOLN6QZ55O6SJPZT5LPXPPUNMGGTYAZCAOTORA, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.121777863 (506h25m7.992860908s) [noise=true , address=145.79.227.11:4732 , piece_id=5FVE6U6ISMCIMDESG3OQ6SHFBGTWT2PPZZC7XRZAPSNK47HZIBWQ, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.130879434 (506h25m8.00196248s) [noise=true , address=145.79.227.39:4015 , piece_id=XD5CWP335WRWKXUP3WKMIO7VOFE5G4RN4EGONIRXB6HI43GOCJFQ, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.132848083 (506h25m8.003931118s) [noise=true , address=145.79.227.42:4123 , piece_id=T5UOINXTMKBAACABHEVV5NGEMB52SGKWNSJPJ7Y6UKCQXNOUOJMA, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.135816906 (506h25m8.006899952s) [noise=true , address=145.79.227.49:4339 , piece_id=NHMAMURTEDXM7SZDKB6OP4NU6SCIIKVLIHTVAOT2DVU7UAIT6V4Q, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.139078326 (506h25m8.010161361s) [noise=true , address=145.79.227.15:4874 , piece_id=7PRWX53PMTA2U7GAUIKHBWJXO6RANYJHXUIUFMG3UNASTUTNFIXQ, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.139833234 (506h25m8.010916279s) [noise=true , address=145.79.227.36:4519 , piece_id=FJEF6GROIQF7AWQYSJHUEINR5FTBIO7USLJGFWEEEDRJSGW2JG4Q, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.299232928 (506h25m8.170315963s) [noise=true , address=77.90.31.15:4700 , piece_id=AC62U7GFNGWMPNEFW5H2C4IMSFT6TSFNH5EHUCX24SWZXCPRKOQQ, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.362334064 (506h25m8.233417109s) [noise=true , address=45.91.116.13:4153 , piece_id=VVU5B5KNVMOIGGZABQAAI22IQ2CZTWJURAJWQQ2JGY3ABTBPFK3A, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.462770483 (506h25m8.333853538s) [noise=true , address=77.90.31.19:4844 , piece_id=AVEWXCOCGI7U437EYT37M4VY3TH73JESDRWAXRLHQKONPFPLAHTQ, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.474990001 (506h25m8.346073046s) [noise=true , address=45.91.116.6:4974 , piece_id=23YEXTSWEM6GOICKLZM5QVYLA7OK2HE55TIA52YG4YO26WBFK4LA, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.475014728 (506h25m8.346097793s) [seg_pos={0 0}, upload=444374]: Segment reached optimal threshold of 13 pieces. Dec 26 17:54:08.475260065 (506h25m8.34634311s) [noise=true , address=45.91.116.5:4297 , piece_id=I3IPV24ECC3YV4V3QRQDP2ZZT3JOO2CN6M6BISNPENMQZNSDKVAQ, seg_pos={0 0}, upload=444374]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 26 17:54:08.475266929 (506h25m8.346349964s) [noise=true , address=45.91.116.5:4297 , piece_id=I3IPV24ECC3YV4V3QRQDP2ZZT3JOO2CN6M6BISNPENMQZNSDKVAQ, seg_pos={0 0}, upload=444374]: Long tail cancellation due to optimal threshold Dec 26 17:54:08.475601164 (506h25m8.346684209s) [noise=true , address=77.90.31.14:4663 , piece_id=WOQS65FTAS6NTIR7QVGUIFPIJY6POSNUFZTBGYCDZW67TKWZ2GZQ, seg_pos={0 0}, upload=444374]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: expected piece hash; context canceled; EOF Dec 26 17:54:08.475628697 (506h25m8.346711732s) [noise=true , address=77.90.31.14:4663 , piece_id=WOQS65FTAS6NTIR7QVGUIFPIJY6POSNUFZTBGYCDZW67TKWZ2GZQ, seg_pos={0 0}, upload=444374]: Long tail cancellation due to optimal threshold Dec 26 17:54:08.475658875 (506h25m8.34674192s) [seg_pos={0 0}, upload=444374]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 17:54:08.475744387 (506h25m8.346827422s) [upload=444374]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:54:08.475750088 (506h25m8.346833133s) [upload=444374]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:54:08.767674804 (506h25m8.638757859s) [upload=444374]: Done uploading segments... Dec 26 17:54:08.767701635 (506h25m8.63878468s) [upload=444374]: Upload finished. err: <nil> Duration: 5.771331303s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 27.366989456s trace: https://jaeger.dev.storj.io/trace/665ea65f6e9ccc33 FAIL (2025-12-26 17:55:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== e=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789316979 (506h12m54.660400024s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789326507 (506h12m54.660409552s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789611339 (506h12m54.660694384s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789619275 (506h12m54.66070231s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789914006 (506h12m54.660997041s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789949443 (506h12m54.661032478s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790181566 (506h12m54.661264601s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790295632 (506h12m54.661378668s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790510872 (506h12m54.661593917s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790692980 (506h12m54.661776015s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790787379 (506h12m54.661870414s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790992970 (506h12m54.662076005s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791120053 (506h12m54.662203098s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791275218 (506h12m54.662358253s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791443709 (506h12m54.662526744s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:56.112313145 (506h12m55.98339619s) [upload=444191]: Next returned nil segment Dec 26 17:41:56.112330638 (506h12m55.983413683s) [upload=444191]: Waiting for error group managing segments... Dec 26 17:41:56.485322720 (506h12m56.356405765s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.485637079 (506h12m56.356720124s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.666807539 (506h12m56.537890584s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.678753987 (506h12m56.549837032s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.687788039 (506h12m56.558871074s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.694318023 (506h12m56.565401058s) [noise=true , address=145.79.227.7:4326 , piece_id=DMQ5RPI46X2U7TJTS42ZHQEF32BDJRKJJO5HWOYAPNS6VWVG5B5Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.716581187 (506h12m56.587664232s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.972183007 (506h12m56.843266042s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.011917999 (506h12m56.883001034s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.054168622 (506h12m56.925251667s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.114312777 (506h12m56.985395812s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.499846250 (506h12m57.370929285s) [noise=true , address=145.79.227.34:4919 , piece_id=YRWHW3BJVEGVJRSA6RJB3T7VJQSDTGKEZQKRICFIV2G3T7AN4MIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.612649066 (506h12m57.483732101s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649413733 (506h12m57.520496768s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649440282 (506h12m57.520523328s) [seg_pos={0 0}, upload=444191]: Segment reached optimal threshold of 14 pieces. Dec 26 17:41:57.649674729 (506h12m57.520757774s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G7ty71tPwe7cZSBNK9u74DFUCGomnxDsNvAif4C3SbP7HWwfo): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649684087 (506h12m57.520767142s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649747708 (506h12m57.520830763s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v6zXZuMmyL2MU8qb3q6gg1iwTYvndjzMjVpv8btEpeX9Lnqmu): protocol: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection Dec 26 17:41:57.649785219 (506h12m57.520868274s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649899828 (506h12m57.520982903s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649908775 (506h12m57.52099182s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650074641 (506h12m57.521157696s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.650083287 (506h12m57.521166342s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650132821 (506h12m57.521215866s) [seg_pos={0 0}, upload=444191]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 17:41:57.650220138 (506h12m57.521303183s) [upload=444191]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:41:57.650227171 (506h12m57.521310206s) [upload=444191]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:41:57.958890870 (506h12m57.829973915s) [upload=444191]: Done uploading segments... Dec 26 17:41:57.958911620 (506h12m57.829994665s) [upload=444191]: Upload finished. err: <nil> Duration: 27.367016037s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [XX..X....X.X.XX...XX...XX....XX..XX......X.XXX.....XX.X.XXX.] duration: 6.32380054s trace: https://jaeger.dev.storj.io/trace/66e1eaafd7692f5a FAIL (2025-12-26 17:54:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: VVU5B5KNVMOIGGZABQAAI22IQ2CZTWJURAJWQQ2JGY3ABTBPFK3A) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: AC62U7GFNGWMPNEFW5H2C4IMSFT6TSFNH5EHUCX24SWZXCPRKOQQ) context deadline exceeded ======LOGS====== Dec 26 17:54:05.860888315 (506h25m5.73197135s) [upload=444374]: Starting upload Dec 26 17:54:05.860960913 (506h25m5.732043958s) [upload=444374]: Done starting upload Dec 26 17:54:05.861243932 (506h25m5.732326967s) [upload=444374]: Uploading segments... Dec 26 17:54:05.861348571 (506h25m5.732431616s) [upload=444374]: Got next segment. Inline: false Dec 26 17:54:05.861399419 (506h25m5.732482464s) [upload=444374]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 17:54:05.861404999 (506h25m5.732488034s) [upload=444374]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 17:54:06.104522897 (506h25m5.975605933s) [seg_pos={0 0}, upload=444374]: Begin upload segment... Dec 26 17:54:06.104572262 (506h25m5.975655297s) [seg_pos={0 0}, upload=444374]: Done begin upload segment. Dec 26 17:54:06.104602710 (506h25m5.975685755s) [noise=true , address=77.90.31.12:4589 , piece_id=CJJUEAIEWP66VHOKZTYLIZUHN3OET6SAFJHXK73EBDFNB5QFE5RA, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.104713491 (506h25m5.975796536s) [noise=true , address=145.79.227.21:4475 , piece_id=UP7LFRPWG3GFBUFT6Y3TOMII7C2UE5BJEY7VCDKHWPYN7NHOL25A, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.105352147 (506h25m5.976435192s) [noise=true , address=145.79.227.51:4410 , piece_id=NJSE6VNZ2ICW5KYFAVC7TXUKOHPAQSHATMU2WW2XMXJF77VTZLVQ, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.105704788 (506h25m5.976787823s) [noise=true , address=77.90.31.15:4700 , piece_id=AC62U7GFNGWMPNEFW5H2C4IMSFT6TSFNH5EHUCX24SWZXCPRKOQQ, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.106020589 (506h25m5.977103624s) [noise=true , address=77.90.31.14:4663 , piece_id=WOQS65FTAS6NTIR7QVGUIFPIJY6POSNUFZTBGYCDZW67TKWZ2GZQ, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.106328646 (506h25m5.977411681s) [noise=true , address=145.79.227.42:4123 , piece_id=T5UOINXTMKBAACABHEVV5NGEMB52SGKWNSJPJ7Y6UKCQXNOUOJMA, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.106787600 (506h25m5.977870635s) [noise=true , address=45.91.116.6:4974 , piece_id=23YEXTSWEM6GOICKLZM5QVYLA7OK2HE55TIA52YG4YO26WBFK4LA, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.107122217 (506h25m5.978205252s) [noise=true , address=145.79.227.49:4339 , piece_id=NHMAMURTEDXM7SZDKB6OP4NU6SCIIKVLIHTVAOT2DVU7UAIT6V4Q, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.108570165 (506h25m5.97965324s) [noise=true , address=145.79.227.15:4874 , piece_id=7PRWX53PMTA2U7GAUIKHBWJXO6RANYJHXUIUFMG3UNASTUTNFIXQ, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.108781597 (506h25m5.979864633s) [noise=true , address=45.91.116.5:4297 , piece_id=I3IPV24ECC3YV4V3QRQDP2ZZT3JOO2CN6M6BISNPENMQZNSDKVAQ, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.110063228 (506h25m5.981146263s) [noise=true , address=145.79.227.11:4732 , piece_id=5FVE6U6ISMCIMDESG3OQ6SHFBGTWT2PPZZC7XRZAPSNK47HZIBWQ, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.111752484 (506h25m5.982835529s) [noise=true , address=145.79.227.36:4519 , piece_id=FJEF6GROIQF7AWQYSJHUEINR5FTBIO7USLJGFWEEEDRJSGW2JG4Q, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.113340689 (506h25m5.984423734s) [noise=true , address=145.79.227.10:4401 , piece_id=ZI4WZERG34DPXHUOLN6QZ55O6SJPZT5LPXPPUNMGGTYAZCAOTORA, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.114643860 (506h25m5.985726905s) [noise=true , address=45.91.116.13:4153 , piece_id=VVU5B5KNVMOIGGZABQAAI22IQ2CZTWJURAJWQQ2JGY3ABTBPFK3A, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.115852932 (506h25m5.986935967s) [noise=true , address=77.90.31.19:4844 , piece_id=AVEWXCOCGI7U437EYT37M4VY3TH73JESDRWAXRLHQKONPFPLAHTQ, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.116921628 (506h25m5.988004673s) [noise=true , address=145.79.227.39:4015 , piece_id=XD5CWP335WRWKXUP3WKMIO7VOFE5G4RN4EGONIRXB6HI43GOCJFQ, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:07.294144135 (506h25m7.16522718s) [noise=true , address=77.90.31.12:4589 , piece_id=CJJUEAIEWP66VHOKZTYLIZUHN3OET6SAFJHXK73EBDFNB5QFE5RA, seg_pos={0 0}, upload=444374]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 26 17:54:07.828164492 (506h25m7.699247537s) [upload=444374]: Next returned nil segment Dec 26 17:54:07.828181555 (506h25m7.6992646s) [upload=444374]: Waiting for error group managing segments... Dec 26 17:54:08.107862847 (506h25m7.978945892s) [noise=true , address=145.79.227.21:4475 , piece_id=UP7LFRPWG3GFBUFT6Y3TOMII7C2UE5BJEY7VCDKHWPYN7NHOL25A, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.112828612 (506h25m7.983911657s) [noise=true , address=145.79.227.51:4410 , piece_id=NJSE6VNZ2ICW5KYFAVC7TXUKOHPAQSHATMU2WW2XMXJF77VTZLVQ, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.119737338 (506h25m7.990820383s) [noise=true , address=145.79.227.10:4401 , piece_id=ZI4WZERG34DPXHUOLN6QZ55O6SJPZT5LPXPPUNMGGTYAZCAOTORA, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.121777863 (506h25m7.992860908s) [noise=true , address=145.79.227.11:4732 , piece_id=5FVE6U6ISMCIMDESG3OQ6SHFBGTWT2PPZZC7XRZAPSNK47HZIBWQ, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.130879434 (506h25m8.00196248s) [noise=true , address=145.79.227.39:4015 , piece_id=XD5CWP335WRWKXUP3WKMIO7VOFE5G4RN4EGONIRXB6HI43GOCJFQ, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.132848083 (506h25m8.003931118s) [noise=true , address=145.79.227.42:4123 , piece_id=T5UOINXTMKBAACABHEVV5NGEMB52SGKWNSJPJ7Y6UKCQXNOUOJMA, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.135816906 (506h25m8.006899952s) [noise=true , address=145.79.227.49:4339 , piece_id=NHMAMURTEDXM7SZDKB6OP4NU6SCIIKVLIHTVAOT2DVU7UAIT6V4Q, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.139078326 (506h25m8.010161361s) [noise=true , address=145.79.227.15:4874 , piece_id=7PRWX53PMTA2U7GAUIKHBWJXO6RANYJHXUIUFMG3UNASTUTNFIXQ, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.139833234 (506h25m8.010916279s) [noise=true , address=145.79.227.36:4519 , piece_id=FJEF6GROIQF7AWQYSJHUEINR5FTBIO7USLJGFWEEEDRJSGW2JG4Q, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.299232928 (506h25m8.170315963s) [noise=true , address=77.90.31.15:4700 , piece_id=AC62U7GFNGWMPNEFW5H2C4IMSFT6TSFNH5EHUCX24SWZXCPRKOQQ, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.362334064 (506h25m8.233417109s) [noise=true , address=45.91.116.13:4153 , piece_id=VVU5B5KNVMOIGGZABQAAI22IQ2CZTWJURAJWQQ2JGY3ABTBPFK3A, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.462770483 (506h25m8.333853538s) [noise=true , address=77.90.31.19:4844 , piece_id=AVEWXCOCGI7U437EYT37M4VY3TH73JESDRWAXRLHQKONPFPLAHTQ, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.474990001 (506h25m8.346073046s) [noise=true , address=45.91.116.6:4974 , piece_id=23YEXTSWEM6GOICKLZM5QVYLA7OK2HE55TIA52YG4YO26WBFK4LA, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.475014728 (506h25m8.346097793s) [seg_pos={0 0}, upload=444374]: Segment reached optimal threshold of 13 pieces. Dec 26 17:54:08.475260065 (506h25m8.34634311s) [noise=true , address=45.91.116.5:4297 , piece_id=I3IPV24ECC3YV4V3QRQDP2ZZT3JOO2CN6M6BISNPENMQZNSDKVAQ, seg_pos={0 0}, upload=444374]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 26 17:54:08.475266929 (506h25m8.346349964s) [noise=true , address=45.91.116.5:4297 , piece_id=I3IPV24ECC3YV4V3QRQDP2ZZT3JOO2CN6M6BISNPENMQZNSDKVAQ, seg_pos={0 0}, upload=444374]: Long tail cancellation due to optimal threshold Dec 26 17:54:08.475601164 (506h25m8.346684209s) [noise=true , address=77.90.31.14:4663 , piece_id=WOQS65FTAS6NTIR7QVGUIFPIJY6POSNUFZTBGYCDZW67TKWZ2GZQ, seg_pos={0 0}, upload=444374]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: expected piece hash; context canceled; EOF Dec 26 17:54:08.475628697 (506h25m8.346711732s) [noise=true , address=77.90.31.14:4663 , piece_id=WOQS65FTAS6NTIR7QVGUIFPIJY6POSNUFZTBGYCDZW67TKWZ2GZQ, seg_pos={0 0}, upload=444374]: Long tail cancellation due to optimal threshold Dec 26 17:54:08.475658875 (506h25m8.34674192s) [seg_pos={0 0}, upload=444374]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 17:54:08.475744387 (506h25m8.346827422s) [upload=444374]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:54:08.475750088 (506h25m8.346833133s) [upload=444374]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:54:08.767674804 (506h25m8.638757859s) [upload=444374]: Done uploading segments... Dec 26 17:54:08.767701635 (506h25m8.63878468s) [upload=444374]: Upload finished. err: <nil> Duration: 6.323831138s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [.XX..X....X.X.XX...XX...XX....XX..XX......X.XXX.....XX.X.XXX] duration: 30.003706616s trace: https://jaeger.dev.storj.io/trace/66e1eaafd7692f5a FAIL (2025-12-26 17:54:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: VVU5B5KNVMOIGGZABQAAI22IQ2CZTWJURAJWQQ2JGY3ABTBPFK3A) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: AC62U7GFNGWMPNEFW5H2C4IMSFT6TSFNH5EHUCX24SWZXCPRKOQQ) context deadline exceeded ======LOGS====== Dec 26 17:54:05.860888315 (506h25m5.73197135s) [upload=444374]: Starting upload Dec 26 17:54:05.860960913 (506h25m5.732043958s) [upload=444374]: Done starting upload Dec 26 17:54:05.861243932 (506h25m5.732326967s) [upload=444374]: Uploading segments... Dec 26 17:54:05.861348571 (506h25m5.732431616s) [upload=444374]: Got next segment. Inline: false Dec 26 17:54:05.861399419 (506h25m5.732482464s) [upload=444374]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 17:54:05.861404999 (506h25m5.732488034s) [upload=444374]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 17:54:06.104522897 (506h25m5.975605933s) [seg_pos={0 0}, upload=444374]: Begin upload segment... Dec 26 17:54:06.104572262 (506h25m5.975655297s) [seg_pos={0 0}, upload=444374]: Done begin upload segment. Dec 26 17:54:06.104602710 (506h25m5.975685755s) [noise=true , address=77.90.31.12:4589 , piece_id=CJJUEAIEWP66VHOKZTYLIZUHN3OET6SAFJHXK73EBDFNB5QFE5RA, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.104713491 (506h25m5.975796536s) [noise=true , address=145.79.227.21:4475 , piece_id=UP7LFRPWG3GFBUFT6Y3TOMII7C2UE5BJEY7VCDKHWPYN7NHOL25A, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.105352147 (506h25m5.976435192s) [noise=true , address=145.79.227.51:4410 , piece_id=NJSE6VNZ2ICW5KYFAVC7TXUKOHPAQSHATMU2WW2XMXJF77VTZLVQ, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.105704788 (506h25m5.976787823s) [noise=true , address=77.90.31.15:4700 , piece_id=AC62U7GFNGWMPNEFW5H2C4IMSFT6TSFNH5EHUCX24SWZXCPRKOQQ, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.106020589 (506h25m5.977103624s) [noise=true , address=77.90.31.14:4663 , piece_id=WOQS65FTAS6NTIR7QVGUIFPIJY6POSNUFZTBGYCDZW67TKWZ2GZQ, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.106328646 (506h25m5.977411681s) [noise=true , address=145.79.227.42:4123 , piece_id=T5UOINXTMKBAACABHEVV5NGEMB52SGKWNSJPJ7Y6UKCQXNOUOJMA, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.106787600 (506h25m5.977870635s) [noise=true , address=45.91.116.6:4974 , piece_id=23YEXTSWEM6GOICKLZM5QVYLA7OK2HE55TIA52YG4YO26WBFK4LA, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.107122217 (506h25m5.978205252s) [noise=true , address=145.79.227.49:4339 , piece_id=NHMAMURTEDXM7SZDKB6OP4NU6SCIIKVLIHTVAOT2DVU7UAIT6V4Q, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.108570165 (506h25m5.97965324s) [noise=true , address=145.79.227.15:4874 , piece_id=7PRWX53PMTA2U7GAUIKHBWJXO6RANYJHXUIUFMG3UNASTUTNFIXQ, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.108781597 (506h25m5.979864633s) [noise=true , address=45.91.116.5:4297 , piece_id=I3IPV24ECC3YV4V3QRQDP2ZZT3JOO2CN6M6BISNPENMQZNSDKVAQ, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.110063228 (506h25m5.981146263s) [noise=true , address=145.79.227.11:4732 , piece_id=5FVE6U6ISMCIMDESG3OQ6SHFBGTWT2PPZZC7XRZAPSNK47HZIBWQ, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.111752484 (506h25m5.982835529s) [noise=true , address=145.79.227.36:4519 , piece_id=FJEF6GROIQF7AWQYSJHUEINR5FTBIO7USLJGFWEEEDRJSGW2JG4Q, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.113340689 (506h25m5.984423734s) [noise=true , address=145.79.227.10:4401 , piece_id=ZI4WZERG34DPXHUOLN6QZ55O6SJPZT5LPXPPUNMGGTYAZCAOTORA, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.114643860 (506h25m5.985726905s) [noise=true , address=45.91.116.13:4153 , piece_id=VVU5B5KNVMOIGGZABQAAI22IQ2CZTWJURAJWQQ2JGY3ABTBPFK3A, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.115852932 (506h25m5.986935967s) [noise=true , address=77.90.31.19:4844 , piece_id=AVEWXCOCGI7U437EYT37M4VY3TH73JESDRWAXRLHQKONPFPLAHTQ, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:06.116921628 (506h25m5.988004673s) [noise=true , address=145.79.227.39:4015 , piece_id=XD5CWP335WRWKXUP3WKMIO7VOFE5G4RN4EGONIRXB6HI43GOCJFQ, seg_pos={0 0}, upload=444374]: Uploading piece... Dec 26 17:54:07.294144135 (506h25m7.16522718s) [noise=true , address=77.90.31.12:4589 , piece_id=CJJUEAIEWP66VHOKZTYLIZUHN3OET6SAFJHXK73EBDFNB5QFE5RA, seg_pos={0 0}, upload=444374]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 26 17:54:07.828164492 (506h25m7.699247537s) [upload=444374]: Next returned nil segment Dec 26 17:54:07.828181555 (506h25m7.6992646s) [upload=444374]: Waiting for error group managing segments... Dec 26 17:54:08.107862847 (506h25m7.978945892s) [noise=true , address=145.79.227.21:4475 , piece_id=UP7LFRPWG3GFBUFT6Y3TOMII7C2UE5BJEY7VCDKHWPYN7NHOL25A, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.112828612 (506h25m7.983911657s) [noise=true , address=145.79.227.51:4410 , piece_id=NJSE6VNZ2ICW5KYFAVC7TXUKOHPAQSHATMU2WW2XMXJF77VTZLVQ, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.119737338 (506h25m7.990820383s) [noise=true , address=145.79.227.10:4401 , piece_id=ZI4WZERG34DPXHUOLN6QZ55O6SJPZT5LPXPPUNMGGTYAZCAOTORA, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.121777863 (506h25m7.992860908s) [noise=true , address=145.79.227.11:4732 , piece_id=5FVE6U6ISMCIMDESG3OQ6SHFBGTWT2PPZZC7XRZAPSNK47HZIBWQ, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.130879434 (506h25m8.00196248s) [noise=true , address=145.79.227.39:4015 , piece_id=XD5CWP335WRWKXUP3WKMIO7VOFE5G4RN4EGONIRXB6HI43GOCJFQ, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.132848083 (506h25m8.003931118s) [noise=true , address=145.79.227.42:4123 , piece_id=T5UOINXTMKBAACABHEVV5NGEMB52SGKWNSJPJ7Y6UKCQXNOUOJMA, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.135816906 (506h25m8.006899952s) [noise=true , address=145.79.227.49:4339 , piece_id=NHMAMURTEDXM7SZDKB6OP4NU6SCIIKVLIHTVAOT2DVU7UAIT6V4Q, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.139078326 (506h25m8.010161361s) [noise=true , address=145.79.227.15:4874 , piece_id=7PRWX53PMTA2U7GAUIKHBWJXO6RANYJHXUIUFMG3UNASTUTNFIXQ, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.139833234 (506h25m8.010916279s) [noise=true , address=145.79.227.36:4519 , piece_id=FJEF6GROIQF7AWQYSJHUEINR5FTBIO7USLJGFWEEEDRJSGW2JG4Q, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.299232928 (506h25m8.170315963s) [noise=true , address=77.90.31.15:4700 , piece_id=AC62U7GFNGWMPNEFW5H2C4IMSFT6TSFNH5EHUCX24SWZXCPRKOQQ, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.362334064 (506h25m8.233417109s) [noise=true , address=45.91.116.13:4153 , piece_id=VVU5B5KNVMOIGGZABQAAI22IQ2CZTWJURAJWQQ2JGY3ABTBPFK3A, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.462770483 (506h25m8.333853538s) [noise=true , address=77.90.31.19:4844 , piece_id=AVEWXCOCGI7U437EYT37M4VY3TH73JESDRWAXRLHQKONPFPLAHTQ, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.474990001 (506h25m8.346073046s) [noise=true , address=45.91.116.6:4974 , piece_id=23YEXTSWEM6GOICKLZM5QVYLA7OK2HE55TIA52YG4YO26WBFK4LA, seg_pos={0 0}, upload=444374]: Done uploading piece. err: <nil> Dec 26 17:54:08.475014728 (506h25m8.346097793s) [seg_pos={0 0}, upload=444374]: Segment reached optimal threshold of 13 pieces. Dec 26 17:54:08.475260065 (506h25m8.34634311s) [noise=true , address=45.91.116.5:4297 , piece_id=I3IPV24ECC3YV4V3QRQDP2ZZT3JOO2CN6M6BISNPENMQZNSDKVAQ, seg_pos={0 0}, upload=444374]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 26 17:54:08.475266929 (506h25m8.346349964s) [noise=true , address=45.91.116.5:4297 , piece_id=I3IPV24ECC3YV4V3QRQDP2ZZT3JOO2CN6M6BISNPENMQZNSDKVAQ, seg_pos={0 0}, upload=444374]: Long tail cancellation due to optimal threshold Dec 26 17:54:08.475601164 (506h25m8.346684209s) [noise=true , address=77.90.31.14:4663 , piece_id=WOQS65FTAS6NTIR7QVGUIFPIJY6POSNUFZTBGYCDZW67TKWZ2GZQ, seg_pos={0 0}, upload=444374]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: expected piece hash; context canceled; EOF Dec 26 17:54:08.475628697 (506h25m8.346711732s) [noise=true , address=77.90.31.14:4663 , piece_id=WOQS65FTAS6NTIR7QVGUIFPIJY6POSNUFZTBGYCDZW67TKWZ2GZQ, seg_pos={0 0}, upload=444374]: Long tail cancellation due to optimal threshold Dec 26 17:54:08.475658875 (506h25m8.34674192s) [seg_pos={0 0}, upload=444374]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 17:54:08.475744387 (506h25m8.346827422s) [upload=444374]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:54:08.475750088 (506h25m8.346833133s) [upload=444374]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:54:08.767674804 (506h25m8.638757859s) [upload=444374]: Done uploading segments... Dec 26 17:54:08.767701635 (506h25m8.63878468s) [upload=444374]: Upload finished. err: <nil> Duration: 30.003862322s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 36.592387464s trace: https://jaeger.dev.storj.io/trace/665ea65f6e9ccc33 FAIL (2025-12-26 17:54:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== e=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789316979 (506h12m54.660400024s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789326507 (506h12m54.660409552s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789611339 (506h12m54.660694384s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789619275 (506h12m54.66070231s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789914006 (506h12m54.660997041s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789949443 (506h12m54.661032478s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790181566 (506h12m54.661264601s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790295632 (506h12m54.661378668s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790510872 (506h12m54.661593917s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790692980 (506h12m54.661776015s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790787379 (506h12m54.661870414s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790992970 (506h12m54.662076005s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791120053 (506h12m54.662203098s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791275218 (506h12m54.662358253s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791443709 (506h12m54.662526744s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:56.112313145 (506h12m55.98339619s) [upload=444191]: Next returned nil segment Dec 26 17:41:56.112330638 (506h12m55.983413683s) [upload=444191]: Waiting for error group managing segments... Dec 26 17:41:56.485322720 (506h12m56.356405765s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.485637079 (506h12m56.356720124s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.666807539 (506h12m56.537890584s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.678753987 (506h12m56.549837032s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.687788039 (506h12m56.558871074s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.694318023 (506h12m56.565401058s) [noise=true , address=145.79.227.7:4326 , piece_id=DMQ5RPI46X2U7TJTS42ZHQEF32BDJRKJJO5HWOYAPNS6VWVG5B5Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.716581187 (506h12m56.587664232s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.972183007 (506h12m56.843266042s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.011917999 (506h12m56.883001034s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.054168622 (506h12m56.925251667s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.114312777 (506h12m56.985395812s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.499846250 (506h12m57.370929285s) [noise=true , address=145.79.227.34:4919 , piece_id=YRWHW3BJVEGVJRSA6RJB3T7VJQSDTGKEZQKRICFIV2G3T7AN4MIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.612649066 (506h12m57.483732101s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649413733 (506h12m57.520496768s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649440282 (506h12m57.520523328s) [seg_pos={0 0}, upload=444191]: Segment reached optimal threshold of 14 pieces. Dec 26 17:41:57.649674729 (506h12m57.520757774s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G7ty71tPwe7cZSBNK9u74DFUCGomnxDsNvAif4C3SbP7HWwfo): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649684087 (506h12m57.520767142s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649747708 (506h12m57.520830763s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v6zXZuMmyL2MU8qb3q6gg1iwTYvndjzMjVpv8btEpeX9Lnqmu): protocol: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection Dec 26 17:41:57.649785219 (506h12m57.520868274s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649899828 (506h12m57.520982903s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649908775 (506h12m57.52099182s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650074641 (506h12m57.521157696s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.650083287 (506h12m57.521166342s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650132821 (506h12m57.521215866s) [seg_pos={0 0}, upload=444191]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 17:41:57.650220138 (506h12m57.521303183s) [upload=444191]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:41:57.650227171 (506h12m57.521310206s) [upload=444191]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:41:57.958890870 (506h12m57.829973915s) [upload=444191]: Done uploading segments... Dec 26 17:41:57.958911620 (506h12m57.829994665s) [upload=444191]: Upload finished. err: <nil> Duration: 36.592416298s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [..XX..X....X.X.XX...XX...XX....XX..XX......X.XXX.....XX.X.XX] duration: 30.003100922s trace: https://jaeger.dev.storj.io/trace/2abebcbf6fa38e21 FAIL (2025-12-26 17:53:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== segments... Dec 26 17:53:24.656416197 (506h24m24.527499242s) [upload=444359]: Got next segment. Inline: false Dec 26 17:53:24.656456674 (506h24m24.527539729s) [upload=444359]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 17:53:24.656463027 (506h24m24.527546072s) [upload=444359]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 17:53:24.901285551 (506h24m24.772368596s) [seg_pos={0 0}, upload=444359]: Begin upload segment... Dec 26 17:53:24.901343923 (506h24m24.772426968s) [seg_pos={0 0}, upload=444359]: Done begin upload segment. Dec 26 17:53:24.901379551 (506h24m24.772462586s) [noise=true , address=45.91.116.7:4009 , piece_id=R2HC5FKC4KK2B246ZENUMQUGXMVFARRQ7644XRZOWYL27LAOS5JA, seg_pos={0 0}, upload=444359]: Uploading piece... Dec 26 17:53:24.901441237 (506h24m24.772524272s) [noise=true , address=145.79.227.32:4836 , piece_id=D2RII4MFOPZAEKQEJ5GFCH3MX6REPVWH36CKUUWVOI4PLLPQHNBA, seg_pos={0 0}, upload=444359]: Uploading piece... Dec 26 17:53:24.901986626 (506h24m24.773069661s) [noise=true , address=145.79.227.4:4185 , piece_id=2RCCJHTRWUZQEK2VWMSK6PM3KA2TG7WRI54I2WAIKHEI7TWK3C7A, seg_pos={0 0}, upload=444359]: Uploading piece... Dec 26 17:53:24.902285636 (506h24m24.773368671s) [noise=true , address=145.79.227.16:4911 , piece_id=ZAYREVMEEG4JCGG2TVYLIDM2DGJXUM2EK2M52RUOF233ELXEQHTQ, seg_pos={0 0}, upload=444359]: Uploading piece... Dec 26 17:53:24.902669958 (506h24m24.773752993s) [noise=true , address=145.79.227.45:4230 , piece_id=H27S2V5TF3GLQKPSO3MPE77MAQV7VD5QW5OIAEAZVTY3YJJEY6PQ, seg_pos={0 0}, upload=444359]: Uploading piece... Dec 26 17:53:24.903002761 (506h24m24.774085806s) [noise=true , address=45.91.116.6:4973 , piece_id=J6Q6VACQLGWWZV7KUUT3C3NZPQGQQUQCB3LQ5J2GZYXM7HCILOIA, seg_pos={0 0}, upload=444359]: Uploading piece... Dec 26 17:53:24.903313272 (506h24m24.774396307s) [noise=true , address=145.79.227.51:4409 , piece_id=VGK7WKNBJZMTGUFKEJJWQUMPAPCHWAQU6KQKSHVYO7YLYYXG7GTA, seg_pos={0 0}, upload=444359]: Uploading piece... Dec 26 17:53:24.903687525 (506h24m24.77477056s) [noise=true , address=145.79.227.39:4015 , piece_id=F5GEWAIHX5YXBHHM4IBYQURKVVKCMWH7UNKRP3FDWJNRGR3TAKLA, seg_pos={0 0}, upload=444359]: Uploading piece... Dec 26 17:53:24.904125209 (506h24m24.775208244s) [noise=true , address=77.90.31.10:4518 , piece_id=AJK75E2YYNTMDOVO35YDGZAFYKDFYU6XJ72APGPFIX2FVOU5ZUXQ, seg_pos={0 0}, upload=444359]: Uploading piece... Dec 26 17:53:24.905924605 (506h24m24.77700764s) [noise=true , address=145.79.227.29:4943 , piece_id=EUP2HCWNINL6OASYTIDCFBKQ6WIWOWGYK2VVJWDYXUNUG75CNAWQ, seg_pos={0 0}, upload=444359]: Uploading piece... Dec 26 17:53:24.907712139 (506h24m24.778795174s) [noise=true , address=45.91.116.5:4299 , piece_id=K6XBSLACN43NEW5DVXXZFUCJ7DCKF22ZX2DZCHLVGR3WCM4URMOA, seg_pos={0 0}, upload=444359]: Uploading piece... Dec 26 17:53:24.910007821 (506h24m24.781090856s) [noise=true , address=145.79.227.3:4151 , piece_id=SABCK6BFDPCMOIJ3PUR2VHWR4AFY3ZKSQDKQHGDLSR5WBEMYPFKA, seg_pos={0 0}, upload=444359]: Uploading piece... Dec 26 17:53:24.914044075 (506h24m24.78512711s) [noise=true , address=77.90.31.13:4629 , piece_id=YVJPN7JYN3I5PX3T7ATU3FB7TM5PO3PKVSXQUITB7VTFFEYCK3EQ, seg_pos={0 0}, upload=444359]: Uploading piece... Dec 26 17:53:24.916211383 (506h24m24.787294428s) [noise=true , address=145.79.227.50:4375 , piece_id=KXDIJRYERUBJ6AFBF7QQHORGRQGJUV74DEQEXWC3W5KBCOF4MRPQ, seg_pos={0 0}, upload=444359]: Uploading piece... Dec 26 17:53:24.918543624 (506h24m24.789626669s) [noise=true , address=145.79.227.37:4566 , piece_id=OFCCD3RUSOMNNHYSWUEPHK2WJFTBYOAZ4AZE2PESISRNMYCYB6GA, seg_pos={0 0}, upload=444359]: Uploading piece... Dec 26 17:53:24.920895542 (506h24m24.791978587s) [noise=true , address=145.79.227.36:4518 , piece_id=ELG4HYZQXS3FBVZH7WF4EPMHB3RMOVL623N6HHCKC56VITSCLDEQ, seg_pos={0 0}, upload=444359]: Uploading piece... Dec 26 17:53:26.613838321 (506h24m26.484921366s) [upload=444359]: Next returned nil segment Dec 26 17:53:26.613855213 (506h24m26.484938258s) [upload=444359]: Waiting for error group managing segments... Dec 26 17:53:26.902624851 (506h24m26.773707896s) [noise=true , address=145.79.227.45:4230 , piece_id=H27S2V5TF3GLQKPSO3MPE77MAQV7VD5QW5OIAEAZVTY3YJJEY6PQ, seg_pos={0 0}, upload=444359]: Done uploading piece. err: <nil> Dec 26 17:53:26.908198324 (506h24m26.779281369s) [noise=true , address=145.79.227.39:4015 , piece_id=F5GEWAIHX5YXBHHM4IBYQURKVVKCMWH7UNKRP3FDWJNRGR3TAKLA, seg_pos={0 0}, upload=444359]: Done uploading piece. err: <nil> Dec 26 17:53:26.913355935 (506h24m26.784438971s) [noise=true , address=145.79.227.16:4911 , piece_id=ZAYREVMEEG4JCGG2TVYLIDM2DGJXUM2EK2M52RUOF233ELXEQHTQ, seg_pos={0 0}, upload=444359]: Done uploading piece. err: <nil> Dec 26 17:53:26.916699462 (506h24m26.787782507s) [noise=true , address=145.79.227.37:4566 , piece_id=OFCCD3RUSOMNNHYSWUEPHK2WJFTBYOAZ4AZE2PESISRNMYCYB6GA, seg_pos={0 0}, upload=444359]: Done uploading piece. err: <nil> Dec 26 17:53:26.920272716 (506h24m26.791355751s) [noise=true , address=145.79.227.4:4185 , piece_id=2RCCJHTRWUZQEK2VWMSK6PM3KA2TG7WRI54I2WAIKHEI7TWK3C7A, seg_pos={0 0}, upload=444359]: Done uploading piece. err: <nil> Dec 26 17:53:26.924613201 (506h24m26.795696246s) [noise=true , address=145.79.227.36:4518 , piece_id=ELG4HYZQXS3FBVZH7WF4EPMHB3RMOVL623N6HHCKC56VITSCLDEQ, seg_pos={0 0}, upload=444359]: Done uploading piece. err: <nil> Dec 26 17:53:26.925364292 (506h24m26.796447337s) [noise=true , address=145.79.227.32:4836 , piece_id=D2RII4MFOPZAEKQEJ5GFCH3MX6REPVWH36CKUUWVOI4PLLPQHNBA, seg_pos={0 0}, upload=444359]: Done uploading piece. err: <nil> Dec 26 17:53:26.929735014 (506h24m26.800818049s) [noise=true , address=145.79.227.51:4409 , piece_id=VGK7WKNBJZMTGUFKEJJWQUMPAPCHWAQU6KQKSHVYO7YLYYXG7GTA, seg_pos={0 0}, upload=444359]: Done uploading piece. err: <nil> Dec 26 17:53:26.933299241 (506h24m26.804382296s) [noise=true , address=145.79.227.50:4375 , piece_id=KXDIJRYERUBJ6AFBF7QQHORGRQGJUV74DEQEXWC3W5KBCOF4MRPQ, seg_pos={0 0}, upload=444359]: Done uploading piece. err: <nil> Dec 26 17:53:26.938819322 (506h24m26.809902357s) [noise=true , address=145.79.227.3:4151 , piece_id=SABCK6BFDPCMOIJ3PUR2VHWR4AFY3ZKSQDKQHGDLSR5WBEMYPFKA, seg_pos={0 0}, upload=444359]: Done uploading piece. err: <nil> Dec 26 17:53:27.118792242 (506h24m26.989875287s) [noise=true , address=77.90.31.13:4629 , piece_id=YVJPN7JYN3I5PX3T7ATU3FB7TM5PO3PKVSXQUITB7VTFFEYCK3EQ, seg_pos={0 0}, upload=444359]: Done uploading piece. err: <nil> Dec 26 17:53:27.149000683 (506h24m27.020083719s) [noise=true , address=45.91.116.6:4973 , piece_id=J6Q6VACQLGWWZV7KUUT3C3NZPQGQQUQCB3LQ5J2GZYXM7HCILOIA, seg_pos={0 0}, upload=444359]: Done uploading piece. err: <nil> Dec 26 17:53:27.174838293 (506h24m27.045921339s) [noise=true , address=45.91.116.7:4009 , piece_id=R2HC5FKC4KK2B246ZENUMQUGXMVFARRQ7644XRZOWYL27LAOS5JA, seg_pos={0 0}, upload=444359]: Done uploading piece. err: <nil> Dec 26 17:53:27.174893578 (506h24m27.045976623s) [seg_pos={0 0}, upload=444359]: Segment reached optimal threshold of 13 pieces. Dec 26 17:53:27.175136361 (506h24m27.046219406s) [noise=true , address=77.90.31.10:4518 , piece_id=AJK75E2YYNTMDOVO35YDGZAFYKDFYU6XJ72APGPFIX2FVOU5ZUXQ, seg_pos={0 0}, upload=444359]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: write tcp 188.166.225.170:49692->77.90.31.10:4518: use of closed network connection; write tcp 188.166.225.170:49692->77.90.31.10:4518: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49692->77.90.31.10:4518: use of closed network connection Dec 26 17:53:27.175145960 (506h24m27.046229005s) [noise=true , address=77.90.31.10:4518 , piece_id=AJK75E2YYNTMDOVO35YDGZAFYKDFYU6XJ72APGPFIX2FVOU5ZUXQ, seg_pos={0 0}, upload=444359]: Long tail cancellation due to optimal threshold Dec 26 17:53:27.175241230 (506h24m27.046324275s) [noise=true , address=145.79.227.29:4943 , piece_id=EUP2HCWNINL6OASYTIDCFBKQ6WIWOWGYK2VVJWDYXUNUG75CNAWQ, seg_pos={0 0}, upload=444359]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz): protocol: expected piece hash; context canceled; EOF Dec 26 17:53:27.175253544 (506h24m27.046336589s) [noise=true , address=145.79.227.29:4943 , piece_id=EUP2HCWNINL6OASYTIDCFBKQ6WIWOWGYK2VVJWDYXUNUG75CNAWQ, seg_pos={0 0}, upload=444359]: Long tail cancellation due to optimal threshold Dec 26 17:53:27.175297768 (506h24m27.046380813s) [noise=true , address=45.91.116.5:4299 , piece_id=K6XBSLACN43NEW5DVXXZFUCJ7DCKF22ZX2DZCHLVGR3WCM4URMOA, seg_pos={0 0}, upload=444359]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: write tcp 188.166.225.170:45058->45.91.116.5:4299: use of closed network connection; write tcp 188.166.225.170:45058->45.91.116.5:4299: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45058->45.91.116.5:4299: use of closed network connection Dec 26 17:53:27.175306825 (506h24m27.04638987s) [noise=true , address=45.91.116.5:4299 , piece_id=K6XBSLACN43NEW5DVXXZFUCJ7DCKF22ZX2DZCHLVGR3WCM4URMOA, seg_pos={0 0}, upload=444359]: Long tail cancellation due to optimal threshold Dec 26 17:53:27.175407186 (506h24m27.046490221s) [seg_pos={0 0}, upload=444359]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 17:53:27.175499202 (506h24m27.046582247s) [upload=444359]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:53:27.175505534 (506h24m27.046588579s) [upload=444359]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:53:27.472394672 (506h24m27.343477717s) [upload=444359]: Done uploading segments... Dec 26 17:53:27.472418067 (506h24m27.343501112s) [upload=444359]: Upload finished. err: <nil> Duration: 30.00312607s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 29.874032573s trace: https://jaeger.dev.storj.io/trace/665ea65f6e9ccc33 FAIL (2025-12-26 17:53:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== e=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789316979 (506h12m54.660400024s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789326507 (506h12m54.660409552s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789611339 (506h12m54.660694384s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789619275 (506h12m54.66070231s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789914006 (506h12m54.660997041s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789949443 (506h12m54.661032478s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790181566 (506h12m54.661264601s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790295632 (506h12m54.661378668s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790510872 (506h12m54.661593917s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790692980 (506h12m54.661776015s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790787379 (506h12m54.661870414s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790992970 (506h12m54.662076005s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791120053 (506h12m54.662203098s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791275218 (506h12m54.662358253s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791443709 (506h12m54.662526744s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:56.112313145 (506h12m55.98339619s) [upload=444191]: Next returned nil segment Dec 26 17:41:56.112330638 (506h12m55.983413683s) [upload=444191]: Waiting for error group managing segments... Dec 26 17:41:56.485322720 (506h12m56.356405765s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.485637079 (506h12m56.356720124s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.666807539 (506h12m56.537890584s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.678753987 (506h12m56.549837032s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.687788039 (506h12m56.558871074s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.694318023 (506h12m56.565401058s) [noise=true , address=145.79.227.7:4326 , piece_id=DMQ5RPI46X2U7TJTS42ZHQEF32BDJRKJJO5HWOYAPNS6VWVG5B5Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.716581187 (506h12m56.587664232s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.972183007 (506h12m56.843266042s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.011917999 (506h12m56.883001034s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.054168622 (506h12m56.925251667s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.114312777 (506h12m56.985395812s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.499846250 (506h12m57.370929285s) [noise=true , address=145.79.227.34:4919 , piece_id=YRWHW3BJVEGVJRSA6RJB3T7VJQSDTGKEZQKRICFIV2G3T7AN4MIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.612649066 (506h12m57.483732101s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649413733 (506h12m57.520496768s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649440282 (506h12m57.520523328s) [seg_pos={0 0}, upload=444191]: Segment reached optimal threshold of 14 pieces. Dec 26 17:41:57.649674729 (506h12m57.520757774s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G7ty71tPwe7cZSBNK9u74DFUCGomnxDsNvAif4C3SbP7HWwfo): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649684087 (506h12m57.520767142s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649747708 (506h12m57.520830763s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v6zXZuMmyL2MU8qb3q6gg1iwTYvndjzMjVpv8btEpeX9Lnqmu): protocol: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection Dec 26 17:41:57.649785219 (506h12m57.520868274s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649899828 (506h12m57.520982903s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649908775 (506h12m57.52099182s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650074641 (506h12m57.521157696s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.650083287 (506h12m57.521166342s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650132821 (506h12m57.521215866s) [seg_pos={0 0}, upload=444191]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 17:41:57.650220138 (506h12m57.521303183s) [upload=444191]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:41:57.650227171 (506h12m57.521310206s) [upload=444191]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:41:57.958890870 (506h12m57.829973915s) [upload=444191]: Done uploading segments... Dec 26 17:41:57.958911620 (506h12m57.829994665s) [upload=444191]: Upload finished. err: <nil> Duration: 29.874049165s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [...XX..X....X.X.XX...XX...XX....XX..XX......X.XXX.....XX.X.X] duration: 30.002837059s trace: https://jaeger.dev.storj.io/trace/4c3ec4ef955343e5 FAIL (2025-12-26 17:53:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ... Dec 26 17:52:36.958132980 (506h23m36.829216045s) [upload=444347]: Got next segment. Inline: false Dec 26 17:52:36.958200809 (506h23m36.829283854s) [upload=444347]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 17:52:36.958207101 (506h23m36.829290136s) [upload=444347]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 17:52:37.205467347 (506h23m37.076550392s) [seg_pos={0 0}, upload=444347]: Begin upload segment... Dec 26 17:52:37.205541009 (506h23m37.076624044s) [seg_pos={0 0}, upload=444347]: Done begin upload segment. Dec 26 17:52:37.205578229 (506h23m37.076661274s) [noise=true , address=145.79.227.22:4511 , piece_id=CZWOZGJW2MBO637ILPITYTJISIGZSF5GAPA5SWPJSTFJ4I65SDZQ, seg_pos={0 0}, upload=444347]: Uploading piece... Dec 26 17:52:37.205666727 (506h23m37.076749762s) [noise=true , address=145.79.227.14:4839 , piece_id=NEU4CR4NHWJOX6CCIKVDRLYQGTVXIMLX7SAE4FESU4P7UPYEFL5Q, seg_pos={0 0}, upload=444347]: Uploading piece... Dec 26 17:52:37.206173813 (506h23m37.077256848s) [noise=true , address=45.91.116.6:4976 , piece_id=CDAFBV45VWVVV7WGZMIGYVKKTXENYXVODWBKTTOLL7SWHNUNRW3A, seg_pos={0 0}, upload=444347]: Uploading piece... Dec 26 17:52:37.206579515 (506h23m37.07766256s) [noise=true , address=77.90.31.21:4877 , piece_id=V54ODCUPFXSZUMFVXQ42NK4W6J3UWAWKPVGZBP2ATFJGZBTYXQZQ, seg_pos={0 0}, upload=444347]: Uploading piece... Dec 26 17:52:37.206588353 (506h23m37.077671398s) [noise=true , address=145.79.227.37:4566 , piece_id=G3NISFVXPUV42L5QIJVUVLJCRS7T4NKRIZRWRCT4QH6BCI6IW24Q, seg_pos={0 0}, upload=444347]: Uploading piece... Dec 26 17:52:37.206898984 (506h23m37.077982019s) [noise=true , address=145.79.227.11:4732 , piece_id=IFSGOZFLB64ULDCT7J6QVLPIRHE27URV22DQIZDR4FQBXMYSARFQ, seg_pos={0 0}, upload=444347]: Uploading piece... Dec 26 17:52:37.207009805 (506h23m37.07809284s) [noise=true , address=145.79.227.39:4013 , piece_id=SNSGNLGZSZQD6O5OTO2P5JGMGOKIXXUIFDEY7TH6JXBNSUVSW6XA, seg_pos={0 0}, upload=444347]: Uploading piece... Dec 26 17:52:37.207290690 (506h23m37.078373725s) [noise=true , address=145.79.227.10:4401 , piece_id=F6FIC5TD5NN4STBRXRK2UAWTJNHINXUSJ66ANOV5GUZ5SBXPHL7Q, seg_pos={0 0}, upload=444347]: Uploading piece... Dec 26 17:52:37.207350514 (506h23m37.078433549s) [noise=true , address=45.91.116.9:4262 , piece_id=NMN4LMG2QDXPP5LVMS26XVIKDQ6GFYFXJVFKCRL2W5CBO3NECYQA, seg_pos={0 0}, upload=444347]: Uploading piece... Dec 26 17:52:37.207681514 (506h23m37.078764559s) [noise=true , address=45.91.116.13:4155 , piece_id=UNORBCTRBMUSUNF5AO6YXBN7RY3URSONP2DNLPLFV76K5447NUNA, seg_pos={0 0}, upload=444347]: Uploading piece... Dec 26 17:52:37.207824446 (506h23m37.078907491s) [noise=true , address=45.91.116.5:4299 , piece_id=SSAE35TRUWLFCWUO4Y7VR6DOLNWXW4ZEYOEURFATDV6TXLV5KQ5A, seg_pos={0 0}, upload=444347]: Uploading piece... Dec 26 17:52:37.208125339 (506h23m37.079208374s) [noise=true , address=77.90.31.14:4661 , piece_id=S2O7XFHU26A7E5FS77VS7SVKL24NQXX7NKTAEOZVSYQC6JXDZOYQ, seg_pos={0 0}, upload=444347]: Uploading piece... Dec 26 17:52:37.208298830 (506h23m37.079381875s) [noise=true , address=145.79.227.36:4518 , piece_id=JXEVAWLBWPAOWC2BY6PRDUKOF2I3BY3X36DEFV6LTKS6GLVFVRAQ, seg_pos={0 0}, upload=444347]: Uploading piece... Dec 26 17:52:37.208472110 (506h23m37.079555145s) [noise=true , address=145.79.227.50:4374 , piece_id=WTIUMPRW6PDT3O3C5LIZLVN2UCKH574A62FQK56OJKX2DNULGNCQ, seg_pos={0 0}, upload=444347]: Uploading piece... Dec 26 17:52:37.208661981 (506h23m37.079745016s) [noise=true , address=45.91.116.11:4117 , piece_id=V7AAPZPBTFZ627FY7YOGDBGQONAMAXCKQJAFITBTRTLJFXJIOX2Q, seg_pos={0 0}, upload=444347]: Uploading piece... Dec 26 17:52:37.208869466 (506h23m37.079952501s) [noise=true , address=45.91.116.12:4226 , piece_id=5676LLY5CGKKJEYGQN7PW326A5YYK225YT3FZ5TBUQXCKNS3HHOQ, seg_pos={0 0}, upload=444347]: Uploading piece... Dec 26 17:52:38.918139605 (506h23m38.78922264s) [upload=444347]: Next returned nil segment Dec 26 17:52:38.918156828 (506h23m38.789239893s) [upload=444347]: Waiting for error group managing segments... Dec 26 17:52:39.208498188 (506h23m39.079581223s) [noise=true , address=145.79.227.37:4566 , piece_id=G3NISFVXPUV42L5QIJVUVLJCRS7T4NKRIZRWRCT4QH6BCI6IW24Q, seg_pos={0 0}, upload=444347]: Done uploading piece. err: <nil> Dec 26 17:52:39.213152029 (506h23m39.084235064s) [noise=true , address=145.79.227.11:4732 , piece_id=IFSGOZFLB64ULDCT7J6QVLPIRHE27URV22DQIZDR4FQBXMYSARFQ, seg_pos={0 0}, upload=444347]: Done uploading piece. err: <nil> Dec 26 17:52:39.219232758 (506h23m39.090315793s) [noise=true , address=145.79.227.22:4511 , piece_id=CZWOZGJW2MBO637ILPITYTJISIGZSF5GAPA5SWPJSTFJ4I65SDZQ, seg_pos={0 0}, upload=444347]: Done uploading piece. err: <nil> Dec 26 17:52:39.220396134 (506h23m39.091479169s) [noise=true , address=145.79.227.39:4013 , piece_id=SNSGNLGZSZQD6O5OTO2P5JGMGOKIXXUIFDEY7TH6JXBNSUVSW6XA, seg_pos={0 0}, upload=444347]: Done uploading piece. err: <nil> Dec 26 17:52:39.226717931 (506h23m39.097800987s) [noise=true , address=145.79.227.36:4518 , piece_id=JXEVAWLBWPAOWC2BY6PRDUKOF2I3BY3X36DEFV6LTKS6GLVFVRAQ, seg_pos={0 0}, upload=444347]: Done uploading piece. err: <nil> Dec 26 17:52:39.237771881 (506h23m39.108854916s) [noise=true , address=145.79.227.50:4374 , piece_id=WTIUMPRW6PDT3O3C5LIZLVN2UCKH574A62FQK56OJKX2DNULGNCQ, seg_pos={0 0}, upload=444347]: Done uploading piece. err: <nil> Dec 26 17:52:39.240742547 (506h23m39.111825582s) [noise=true , address=145.79.227.10:4401 , piece_id=F6FIC5TD5NN4STBRXRK2UAWTJNHINXUSJ66ANOV5GUZ5SBXPHL7Q, seg_pos={0 0}, upload=444347]: Done uploading piece. err: <nil> Dec 26 17:52:39.403754260 (506h23m39.274837295s) [noise=true , address=77.90.31.21:4877 , piece_id=V54ODCUPFXSZUMFVXQ42NK4W6J3UWAWKPVGZBP2ATFJGZBTYXQZQ, seg_pos={0 0}, upload=444347]: Done uploading piece. err: <nil> Dec 26 17:52:39.415742668 (506h23m39.286825713s) [noise=true , address=77.90.31.14:4661 , piece_id=S2O7XFHU26A7E5FS77VS7SVKL24NQXX7NKTAEOZVSYQC6JXDZOYQ, seg_pos={0 0}, upload=444347]: Done uploading piece. err: <nil> Dec 26 17:52:39.510946703 (506h23m39.382029738s) [noise=true , address=45.91.116.5:4299 , piece_id=SSAE35TRUWLFCWUO4Y7VR6DOLNWXW4ZEYOEURFATDV6TXLV5KQ5A, seg_pos={0 0}, upload=444347]: Done uploading piece. err: <nil> Dec 26 17:52:39.516619004 (506h23m39.387702039s) [noise=true , address=45.91.116.13:4155 , piece_id=UNORBCTRBMUSUNF5AO6YXBN7RY3URSONP2DNLPLFV76K5447NUNA, seg_pos={0 0}, upload=444347]: Done uploading piece. err: <nil> Dec 26 17:52:39.527490505 (506h23m39.39857354s) [noise=true , address=45.91.116.6:4976 , piece_id=CDAFBV45VWVVV7WGZMIGYVKKTXENYXVODWBKTTOLL7SWHNUNRW3A, seg_pos={0 0}, upload=444347]: Done uploading piece. err: <nil> Dec 26 17:52:39.563650190 (506h23m39.434733235s) [noise=true , address=45.91.116.11:4117 , piece_id=V7AAPZPBTFZ627FY7YOGDBGQONAMAXCKQJAFITBTRTLJFXJIOX2Q, seg_pos={0 0}, upload=444347]: Done uploading piece. err: <nil> Dec 26 17:52:39.563675488 (506h23m39.434758523s) [seg_pos={0 0}, upload=444347]: Segment reached optimal threshold of 13 pieces. Dec 26 17:52:39.563875107 (506h23m39.434958153s) [noise=true , address=45.91.116.12:4226 , piece_id=5676LLY5CGKKJEYGQN7PW326A5YYK225YT3FZ5TBUQXCKNS3HHOQ, seg_pos={0 0}, upload=444347]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: write tcp 188.166.225.170:45890->45.91.116.12:4226: use of closed network connection; write tcp 188.166.225.170:45890->45.91.116.12:4226: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45890->45.91.116.12:4226: use of closed network connection Dec 26 17:52:39.563883012 (506h23m39.434966047s) [noise=true , address=45.91.116.12:4226 , piece_id=5676LLY5CGKKJEYGQN7PW326A5YYK225YT3FZ5TBUQXCKNS3HHOQ, seg_pos={0 0}, upload=444347]: Long tail cancellation due to optimal threshold Dec 26 17:52:39.563939470 (506h23m39.435022505s) [noise=true , address=45.91.116.9:4262 , piece_id=NMN4LMG2QDXPP5LVMS26XVIKDQ6GFYFXJVFKCRL2W5CBO3NECYQA, seg_pos={0 0}, upload=444347]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 26 17:52:39.563948437 (506h23m39.435031472s) [noise=true , address=45.91.116.9:4262 , piece_id=NMN4LMG2QDXPP5LVMS26XVIKDQ6GFYFXJVFKCRL2W5CBO3NECYQA, seg_pos={0 0}, upload=444347]: Long tail cancellation due to optimal threshold Dec 26 17:52:39.564134823 (506h23m39.435217868s) [noise=true , address=145.79.227.14:4839 , piece_id=NEU4CR4NHWJOX6CCIKVDRLYQGTVXIMLX7SAE4FESU4P7UPYEFL5Q, seg_pos={0 0}, upload=444347]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv): protocol: write tcp 188.166.225.170:45666->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:45666->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45666->145.79.227.14:4839: use of closed network connection Dec 26 17:52:39.564144300 (506h23m39.435227335s) [noise=true , address=145.79.227.14:4839 , piece_id=NEU4CR4NHWJOX6CCIKVDRLYQGTVXIMLX7SAE4FESU4P7UPYEFL5Q, seg_pos={0 0}, upload=444347]: Long tail cancellation due to optimal threshold Dec 26 17:52:39.564167124 (506h23m39.435250159s) [seg_pos={0 0}, upload=444347]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 17:52:39.564251625 (506h23m39.43533467s) [upload=444347]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:52:39.564259249 (506h23m39.435342294s) [upload=444347]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:52:39.863068274 (506h23m39.734151329s) [upload=444347]: Done uploading segments... Dec 26 17:52:39.863091257 (506h23m39.734174302s) [upload=444347]: Upload finished. err: <nil> Duration: 30.002848742s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 38.360230105s trace: https://jaeger.dev.storj.io/trace/665ea65f6e9ccc33 FAIL (2025-12-26 17:52:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== e=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789316979 (506h12m54.660400024s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789326507 (506h12m54.660409552s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789611339 (506h12m54.660694384s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789619275 (506h12m54.66070231s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789914006 (506h12m54.660997041s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789949443 (506h12m54.661032478s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790181566 (506h12m54.661264601s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790295632 (506h12m54.661378668s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790510872 (506h12m54.661593917s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790692980 (506h12m54.661776015s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790787379 (506h12m54.661870414s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790992970 (506h12m54.662076005s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791120053 (506h12m54.662203098s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791275218 (506h12m54.662358253s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791443709 (506h12m54.662526744s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:56.112313145 (506h12m55.98339619s) [upload=444191]: Next returned nil segment Dec 26 17:41:56.112330638 (506h12m55.983413683s) [upload=444191]: Waiting for error group managing segments... Dec 26 17:41:56.485322720 (506h12m56.356405765s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.485637079 (506h12m56.356720124s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.666807539 (506h12m56.537890584s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.678753987 (506h12m56.549837032s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.687788039 (506h12m56.558871074s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.694318023 (506h12m56.565401058s) [noise=true , address=145.79.227.7:4326 , piece_id=DMQ5RPI46X2U7TJTS42ZHQEF32BDJRKJJO5HWOYAPNS6VWVG5B5Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.716581187 (506h12m56.587664232s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.972183007 (506h12m56.843266042s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.011917999 (506h12m56.883001034s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.054168622 (506h12m56.925251667s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.114312777 (506h12m56.985395812s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.499846250 (506h12m57.370929285s) [noise=true , address=145.79.227.34:4919 , piece_id=YRWHW3BJVEGVJRSA6RJB3T7VJQSDTGKEZQKRICFIV2G3T7AN4MIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.612649066 (506h12m57.483732101s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649413733 (506h12m57.520496768s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649440282 (506h12m57.520523328s) [seg_pos={0 0}, upload=444191]: Segment reached optimal threshold of 14 pieces. Dec 26 17:41:57.649674729 (506h12m57.520757774s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G7ty71tPwe7cZSBNK9u74DFUCGomnxDsNvAif4C3SbP7HWwfo): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649684087 (506h12m57.520767142s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649747708 (506h12m57.520830763s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v6zXZuMmyL2MU8qb3q6gg1iwTYvndjzMjVpv8btEpeX9Lnqmu): protocol: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection Dec 26 17:41:57.649785219 (506h12m57.520868274s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649899828 (506h12m57.520982903s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649908775 (506h12m57.52099182s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650074641 (506h12m57.521157696s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.650083287 (506h12m57.521166342s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650132821 (506h12m57.521215866s) [seg_pos={0 0}, upload=444191]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 17:41:57.650220138 (506h12m57.521303183s) [upload=444191]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:41:57.650227171 (506h12m57.521310206s) [upload=444191]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:41:57.958890870 (506h12m57.829973915s) [upload=444191]: Done uploading segments... Dec 26 17:41:57.958911620 (506h12m57.829994665s) [upload=444191]: Upload finished. err: <nil> Duration: 38.360249262s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [....XX..X....X.X.XX...XX...XX....XX..XX......X.XXX.....XX.X.] duration: 22.460526183s trace: https://jaeger.dev.storj.io/trace/6779a1c1690f0818 FAIL (2025-12-26 17:52:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 17:51:44.005473953 (506h22m43.876556998s) [upload=444333]: Starting upload Dec 26 17:51:44.005550680 (506h22m43.876633725s) [upload=444333]: Done starting upload Dec 26 17:51:44.005629710 (506h22m43.876712745s) [upload=444333]: Uploading segments... Dec 26 17:51:44.005815313 (506h22m43.876898358s) [upload=444333]: Got next segment. Inline: false Dec 26 17:51:44.006334462 (506h22m43.877417497s) [upload=444333]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 17:51:44.006342637 (506h22m43.877425682s) [upload=444333]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 17:51:44.253388747 (506h22m44.124471782s) [seg_pos={0 0}, upload=444333]: Begin upload segment... Dec 26 17:51:44.253449862 (506h22m44.124532897s) [seg_pos={0 0}, upload=444333]: Done begin upload segment. Dec 26 17:51:44.253524976 (506h22m44.124608031s) [noise=true , address=145.79.227.45:4229 , piece_id=AZJFYG26H6TS33VDSJLOZAMLB4IXDHNNCGNEYPVM5GAPI2RX3P6Q, seg_pos={0 0}, upload=444333]: Uploading piece... Dec 26 17:51:44.253945116 (506h22m44.125028151s) [noise=true , address=145.79.227.46:4602 , piece_id=ECGPTTVPWENOLT2KGHKS3LVH5UBIRSOJ7MOQOIVNPFANSUYQCMSA, seg_pos={0 0}, upload=444333]: Uploading piece... Dec 26 17:51:44.254587920 (506h22m44.125670955s) [noise=true , address=45.91.116.8:4045 , piece_id=LEBFSGXN42BETMF7Y2A7BGF4IRTZS5E4M3UQGPXBAKUPYC2EJZZQ, seg_pos={0 0}, upload=444333]: Uploading piece... Dec 26 17:51:44.255123360 (506h22m44.126206405s) [noise=true , address=145.79.227.42:4121 , piece_id=SVSSFSUNTQW7RAQWWTRL5I5GIQVHZZJC72JE67T4MOERTUQIU6VA, seg_pos={0 0}, upload=444333]: Uploading piece... Dec 26 17:51:44.255301148 (506h22m44.126384203s) [noise=true , address=145.79.227.8:4330 , piece_id=VFRBYMLXXLKNWXJ55AS3LY5OJ4NDYX63RO75IIGFNUJSVRUYG4BQ, seg_pos={0 0}, upload=444333]: Uploading piece... Dec 26 17:51:44.255791632 (506h22m44.126874677s) [noise=true , address=77.90.31.17:4770 , piece_id=MV7MNB7IEULOCTO2HP2SXMM3NEYA2OZGPITNVCKNNBAAWBDLGRQA, seg_pos={0 0}, upload=444333]: Uploading piece... Dec 26 17:51:44.256058289 (506h22m44.127141334s) [noise=true , address=77.90.31.14:4663 , piece_id=UPPD3AXAZGG4P5TTY4TEU2O6JHC6LUBJAVHKC74LCWJHUW3MAOBA, seg_pos={0 0}, upload=444333]: Uploading piece... Dec 26 17:51:44.256323264 (506h22m44.127406309s) [noise=true , address=145.79.227.41:4087 , piece_id=TBTU3YCEOW5MBSY3EEN5FOMMAXSVYYT77ZETHVOJIXKVLCNGDQ7Q, seg_pos={0 0}, upload=444333]: Uploading piece... Dec 26 17:51:44.256526151 (506h22m44.127609196s) [noise=true , address=145.79.227.38:4979 , piece_id=3VOAEVN4NVEESMVPIISWOBSCTJU76REM3REKJWSUGLQCYW5D7ZHA, seg_pos={0 0}, upload=444333]: Uploading piece... Dec 26 17:51:44.256901556 (506h22m44.127984601s) [noise=true , address=145.79.227.11:4730 , piece_id=WVALFZ65W45RU7FLO4IOE377B6CKJPZCZULUHQFFBSOMSFTV5VWQ, seg_pos={0 0}, upload=444333]: Uploading piece... Dec 26 17:51:44.256990264 (506h22m44.128073319s) [noise=true , address=145.79.227.15:4874 , piece_id=BW27X34FKIWARBEHQFIYWCKM5MIYG3BY4OTX37SF7RNOKNFVHY2Q, seg_pos={0 0}, upload=444333]: Uploading piece... Dec 26 17:51:44.257506828 (506h22m44.128589873s) [noise=true , address=145.79.227.47:4267 , piece_id=MY3VGFEQNDCFFCGX4EKY5SRDAEP52HDS5EPR3XXAJCAXP34TKQBQ, seg_pos={0 0}, upload=444333]: Uploading piece... Dec 26 17:51:44.257872845 (506h22m44.12895588s) [noise=true , address=45.91.116.14:4190 , piece_id=IZDDTQ6ZDIQINXQGNS3GD6RYVLI5TFDB4TCQT7RPPOHCSGIECNRA, seg_pos={0 0}, upload=444333]: Uploading piece... Dec 26 17:51:44.258130345 (506h22m44.12921339s) [noise=true , address=145.79.227.32:4836 , piece_id=BIDZSYZH7JZ4ARTHIPWDZQFQUPY2QDRQY2FKV7J4DEEJW2S2K3NQ, seg_pos={0 0}, upload=444333]: Uploading piece... Dec 26 17:51:44.258270612 (506h22m44.129353657s) [noise=true , address=77.90.31.18:4805 , piece_id=4BHLLSGBF6B74I6OOICNNFRJMM2WQXDVDSJVPXZGVZBB5IJ2XRQQ, seg_pos={0 0}, upload=444333]: Uploading piece... Dec 26 17:51:44.258961107 (506h22m44.130044152s) [noise=true , address=145.79.227.12:4765 , piece_id=ZT5AWKLZU2V7JS7BXIPAOJQQ3WFJDNIR7TRSD4QMH66KA6AVS44A, seg_pos={0 0}, upload=444333]: Uploading piece... Dec 26 17:51:45.429528151 (506h22m45.300611196s) [noise=true , address=77.90.31.18:4805 , piece_id=4BHLLSGBF6B74I6OOICNNFRJMM2WQXDVDSJVPXZGVZBB5IJ2XRQQ, seg_pos={0 0}, upload=444333]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: storage node overloaded, request limit: 2000 Dec 26 17:51:45.959357869 (506h22m45.830440914s) [upload=444333]: Next returned nil segment Dec 26 17:51:45.959375462 (506h22m45.830458507s) [upload=444333]: Waiting for error group managing segments... Dec 26 17:51:46.247253764 (506h22m46.118336799s) [noise=true , address=145.79.227.46:4602 , piece_id=ECGPTTVPWENOLT2KGHKS3LVH5UBIRSOJ7MOQOIVNPFANSUYQCMSA, seg_pos={0 0}, upload=444333]: Done uploading piece. err: <nil> Dec 26 17:51:46.250896720 (506h22m46.121979755s) [noise=true , address=145.79.227.45:4229 , piece_id=AZJFYG26H6TS33VDSJLOZAMLB4IXDHNNCGNEYPVM5GAPI2RX3P6Q, seg_pos={0 0}, upload=444333]: Done uploading piece. err: <nil> Dec 26 17:51:46.253591341 (506h22m46.124674386s) [noise=true , address=145.79.227.8:4330 , piece_id=VFRBYMLXXLKNWXJ55AS3LY5OJ4NDYX63RO75IIGFNUJSVRUYG4BQ, seg_pos={0 0}, upload=444333]: Done uploading piece. err: <nil> Dec 26 17:51:46.256258740 (506h22m46.127341775s) [noise=true , address=145.79.227.15:4874 , piece_id=BW27X34FKIWARBEHQFIYWCKM5MIYG3BY4OTX37SF7RNOKNFVHY2Q, seg_pos={0 0}, upload=444333]: Done uploading piece. err: <nil> Dec 26 17:51:46.279230472 (506h22m46.150313517s) [noise=true , address=145.79.227.32:4836 , piece_id=BIDZSYZH7JZ4ARTHIPWDZQFQUPY2QDRQY2FKV7J4DEEJW2S2K3NQ, seg_pos={0 0}, upload=444333]: Done uploading piece. err: <nil> Dec 26 17:51:46.280002032 (506h22m46.151085067s) [noise=true , address=145.79.227.38:4979 , piece_id=3VOAEVN4NVEESMVPIISWOBSCTJU76REM3REKJWSUGLQCYW5D7ZHA, seg_pos={0 0}, upload=444333]: Done uploading piece. err: <nil> Dec 26 17:51:46.281328107 (506h22m46.152411142s) [noise=true , address=145.79.227.47:4267 , piece_id=MY3VGFEQNDCFFCGX4EKY5SRDAEP52HDS5EPR3XXAJCAXP34TKQBQ, seg_pos={0 0}, upload=444333]: Done uploading piece. err: <nil> Dec 26 17:51:46.286020221 (506h22m46.157103256s) [noise=true , address=145.79.227.11:4730 , piece_id=WVALFZ65W45RU7FLO4IOE377B6CKJPZCZULUHQFFBSOMSFTV5VWQ, seg_pos={0 0}, upload=444333]: Done uploading piece. err: <nil> Dec 26 17:51:46.288375025 (506h22m46.15945806s) [noise=true , address=145.79.227.42:4121 , piece_id=SVSSFSUNTQW7RAQWWTRL5I5GIQVHZZJC72JE67T4MOERTUQIU6VA, seg_pos={0 0}, upload=444333]: Done uploading piece. err: <nil> Dec 26 17:51:46.588588915 (506h22m46.45967195s) [noise=true , address=77.90.31.14:4663 , piece_id=UPPD3AXAZGG4P5TTY4TEU2O6JHC6LUBJAVHKC74LCWJHUW3MAOBA, seg_pos={0 0}, upload=444333]: Done uploading piece. err: <nil> Dec 26 17:51:46.590231473 (506h22m46.461314518s) [noise=true , address=45.91.116.14:4190 , piece_id=IZDDTQ6ZDIQINXQGNS3GD6RYVLI5TFDB4TCQT7RPPOHCSGIECNRA, seg_pos={0 0}, upload=444333]: Done uploading piece. err: <nil> Dec 26 17:51:46.603359420 (506h22m46.474442465s) [noise=true , address=77.90.31.17:4770 , piece_id=MV7MNB7IEULOCTO2HP2SXMM3NEYA2OZGPITNVCKNNBAAWBDLGRQA, seg_pos={0 0}, upload=444333]: Done uploading piece. err: <nil> Dec 26 17:51:46.606138262 (506h22m46.477221307s) [noise=true , address=45.91.116.8:4045 , piece_id=LEBFSGXN42BETMF7Y2A7BGF4IRTZS5E4M3UQGPXBAKUPYC2EJZZQ, seg_pos={0 0}, upload=444333]: Done uploading piece. err: <nil> Dec 26 17:51:46.606159363 (506h22m46.477242408s) [seg_pos={0 0}, upload=444333]: Segment reached optimal threshold of 13 pieces. Dec 26 17:51:46.606376275 (506h22m46.47745932s) [noise=true , address=145.79.227.41:4087 , piece_id=TBTU3YCEOW5MBSY3EEN5FOMMAXSVYYT77ZETHVOJIXKVLCNGDQ7Q, seg_pos={0 0}, upload=444333]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR): protocol: write tcp 188.166.225.170:59534->145.79.227.41:4087: use of closed network connection; write tcp 188.166.225.170:59534->145.79.227.41:4087: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59534->145.79.227.41:4087: use of closed network connection Dec 26 17:51:46.606383268 (506h22m46.477466304s) [noise=true , address=145.79.227.41:4087 , piece_id=TBTU3YCEOW5MBSY3EEN5FOMMAXSVYYT77ZETHVOJIXKVLCNGDQ7Q, seg_pos={0 0}, upload=444333]: Long tail cancellation due to optimal threshold Dec 26 17:51:46.606386765 (506h22m46.47746981s) [noise=true , address=145.79.227.12:4765 , piece_id=ZT5AWKLZU2V7JS7BXIPAOJQQ3WFJDNIR7TRSD4QMH66KA6AVS44A, seg_pos={0 0}, upload=444333]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV): protocol: write tcp 188.166.225.170:40256->145.79.227.12:4765: use of closed network connection; write tcp 188.166.225.170:40256->145.79.227.12:4765: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40256->145.79.227.12:4765: use of closed network connection Dec 26 17:51:46.606397215 (506h22m46.47748027s) [noise=true , address=145.79.227.12:4765 , piece_id=ZT5AWKLZU2V7JS7BXIPAOJQQ3WFJDNIR7TRSD4QMH66KA6AVS44A, seg_pos={0 0}, upload=444333]: Long tail cancellation due to optimal threshold Dec 26 17:51:46.606452631 (506h22m46.477535676s) [seg_pos={0 0}, upload=444333]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 17:51:46.606720050 (506h22m46.477803095s) [upload=444333]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:51:46.606726051 (506h22m46.477809096s) [upload=444333]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:51:46.895225284 (506h22m46.766308329s) [upload=444333]: Done uploading segments... Dec 26 17:51:46.895243990 (506h22m46.766327025s) [upload=444333]: Upload finished. err: <nil> Duration: 22.460553225s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [.....XX..X....X.X.XX...XX...XX....XX..XX......X.XXX.....XX.X] duration: 30.003077226s trace: https://jaeger.dev.storj.io/trace/6779a1c1690f0818 FAIL (2025-12-26 17:52:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 17:51:44.005473953 (506h22m43.876556998s) [upload=444333]: Starting upload Dec 26 17:51:44.005550680 (506h22m43.876633725s) [upload=444333]: Done starting upload Dec 26 17:51:44.005629710 (506h22m43.876712745s) [upload=444333]: Uploading segments... Dec 26 17:51:44.005815313 (506h22m43.876898358s) [upload=444333]: Got next segment. Inline: false Dec 26 17:51:44.006334462 (506h22m43.877417497s) [upload=444333]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 17:51:44.006342637 (506h22m43.877425682s) [upload=444333]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 17:51:44.253388747 (506h22m44.124471782s) [seg_pos={0 0}, upload=444333]: Begin upload segment... Dec 26 17:51:44.253449862 (506h22m44.124532897s) [seg_pos={0 0}, upload=444333]: Done begin upload segment. Dec 26 17:51:44.253524976 (506h22m44.124608031s) [noise=true , address=145.79.227.45:4229 , piece_id=AZJFYG26H6TS33VDSJLOZAMLB4IXDHNNCGNEYPVM5GAPI2RX3P6Q, seg_pos={0 0}, upload=444333]: Uploading piece... Dec 26 17:51:44.253945116 (506h22m44.125028151s) [noise=true , address=145.79.227.46:4602 , piece_id=ECGPTTVPWENOLT2KGHKS3LVH5UBIRSOJ7MOQOIVNPFANSUYQCMSA, seg_pos={0 0}, upload=444333]: Uploading piece... Dec 26 17:51:44.254587920 (506h22m44.125670955s) [noise=true , address=45.91.116.8:4045 , piece_id=LEBFSGXN42BETMF7Y2A7BGF4IRTZS5E4M3UQGPXBAKUPYC2EJZZQ, seg_pos={0 0}, upload=444333]: Uploading piece... Dec 26 17:51:44.255123360 (506h22m44.126206405s) [noise=true , address=145.79.227.42:4121 , piece_id=SVSSFSUNTQW7RAQWWTRL5I5GIQVHZZJC72JE67T4MOERTUQIU6VA, seg_pos={0 0}, upload=444333]: Uploading piece... Dec 26 17:51:44.255301148 (506h22m44.126384203s) [noise=true , address=145.79.227.8:4330 , piece_id=VFRBYMLXXLKNWXJ55AS3LY5OJ4NDYX63RO75IIGFNUJSVRUYG4BQ, seg_pos={0 0}, upload=444333]: Uploading piece... Dec 26 17:51:44.255791632 (506h22m44.126874677s) [noise=true , address=77.90.31.17:4770 , piece_id=MV7MNB7IEULOCTO2HP2SXMM3NEYA2OZGPITNVCKNNBAAWBDLGRQA, seg_pos={0 0}, upload=444333]: Uploading piece... Dec 26 17:51:44.256058289 (506h22m44.127141334s) [noise=true , address=77.90.31.14:4663 , piece_id=UPPD3AXAZGG4P5TTY4TEU2O6JHC6LUBJAVHKC74LCWJHUW3MAOBA, seg_pos={0 0}, upload=444333]: Uploading piece... Dec 26 17:51:44.256323264 (506h22m44.127406309s) [noise=true , address=145.79.227.41:4087 , piece_id=TBTU3YCEOW5MBSY3EEN5FOMMAXSVYYT77ZETHVOJIXKVLCNGDQ7Q, seg_pos={0 0}, upload=444333]: Uploading piece... Dec 26 17:51:44.256526151 (506h22m44.127609196s) [noise=true , address=145.79.227.38:4979 , piece_id=3VOAEVN4NVEESMVPIISWOBSCTJU76REM3REKJWSUGLQCYW5D7ZHA, seg_pos={0 0}, upload=444333]: Uploading piece... Dec 26 17:51:44.256901556 (506h22m44.127984601s) [noise=true , address=145.79.227.11:4730 , piece_id=WVALFZ65W45RU7FLO4IOE377B6CKJPZCZULUHQFFBSOMSFTV5VWQ, seg_pos={0 0}, upload=444333]: Uploading piece... Dec 26 17:51:44.256990264 (506h22m44.128073319s) [noise=true , address=145.79.227.15:4874 , piece_id=BW27X34FKIWARBEHQFIYWCKM5MIYG3BY4OTX37SF7RNOKNFVHY2Q, seg_pos={0 0}, upload=444333]: Uploading piece... Dec 26 17:51:44.257506828 (506h22m44.128589873s) [noise=true , address=145.79.227.47:4267 , piece_id=MY3VGFEQNDCFFCGX4EKY5SRDAEP52HDS5EPR3XXAJCAXP34TKQBQ, seg_pos={0 0}, upload=444333]: Uploading piece... Dec 26 17:51:44.257872845 (506h22m44.12895588s) [noise=true , address=45.91.116.14:4190 , piece_id=IZDDTQ6ZDIQINXQGNS3GD6RYVLI5TFDB4TCQT7RPPOHCSGIECNRA, seg_pos={0 0}, upload=444333]: Uploading piece... Dec 26 17:51:44.258130345 (506h22m44.12921339s) [noise=true , address=145.79.227.32:4836 , piece_id=BIDZSYZH7JZ4ARTHIPWDZQFQUPY2QDRQY2FKV7J4DEEJW2S2K3NQ, seg_pos={0 0}, upload=444333]: Uploading piece... Dec 26 17:51:44.258270612 (506h22m44.129353657s) [noise=true , address=77.90.31.18:4805 , piece_id=4BHLLSGBF6B74I6OOICNNFRJMM2WQXDVDSJVPXZGVZBB5IJ2XRQQ, seg_pos={0 0}, upload=444333]: Uploading piece... Dec 26 17:51:44.258961107 (506h22m44.130044152s) [noise=true , address=145.79.227.12:4765 , piece_id=ZT5AWKLZU2V7JS7BXIPAOJQQ3WFJDNIR7TRSD4QMH66KA6AVS44A, seg_pos={0 0}, upload=444333]: Uploading piece... Dec 26 17:51:45.429528151 (506h22m45.300611196s) [noise=true , address=77.90.31.18:4805 , piece_id=4BHLLSGBF6B74I6OOICNNFRJMM2WQXDVDSJVPXZGVZBB5IJ2XRQQ, seg_pos={0 0}, upload=444333]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: storage node overloaded, request limit: 2000 Dec 26 17:51:45.959357869 (506h22m45.830440914s) [upload=444333]: Next returned nil segment Dec 26 17:51:45.959375462 (506h22m45.830458507s) [upload=444333]: Waiting for error group managing segments... Dec 26 17:51:46.247253764 (506h22m46.118336799s) [noise=true , address=145.79.227.46:4602 , piece_id=ECGPTTVPWENOLT2KGHKS3LVH5UBIRSOJ7MOQOIVNPFANSUYQCMSA, seg_pos={0 0}, upload=444333]: Done uploading piece. err: <nil> Dec 26 17:51:46.250896720 (506h22m46.121979755s) [noise=true , address=145.79.227.45:4229 , piece_id=AZJFYG26H6TS33VDSJLOZAMLB4IXDHNNCGNEYPVM5GAPI2RX3P6Q, seg_pos={0 0}, upload=444333]: Done uploading piece. err: <nil> Dec 26 17:51:46.253591341 (506h22m46.124674386s) [noise=true , address=145.79.227.8:4330 , piece_id=VFRBYMLXXLKNWXJ55AS3LY5OJ4NDYX63RO75IIGFNUJSVRUYG4BQ, seg_pos={0 0}, upload=444333]: Done uploading piece. err: <nil> Dec 26 17:51:46.256258740 (506h22m46.127341775s) [noise=true , address=145.79.227.15:4874 , piece_id=BW27X34FKIWARBEHQFIYWCKM5MIYG3BY4OTX37SF7RNOKNFVHY2Q, seg_pos={0 0}, upload=444333]: Done uploading piece. err: <nil> Dec 26 17:51:46.279230472 (506h22m46.150313517s) [noise=true , address=145.79.227.32:4836 , piece_id=BIDZSYZH7JZ4ARTHIPWDZQFQUPY2QDRQY2FKV7J4DEEJW2S2K3NQ, seg_pos={0 0}, upload=444333]: Done uploading piece. err: <nil> Dec 26 17:51:46.280002032 (506h22m46.151085067s) [noise=true , address=145.79.227.38:4979 , piece_id=3VOAEVN4NVEESMVPIISWOBSCTJU76REM3REKJWSUGLQCYW5D7ZHA, seg_pos={0 0}, upload=444333]: Done uploading piece. err: <nil> Dec 26 17:51:46.281328107 (506h22m46.152411142s) [noise=true , address=145.79.227.47:4267 , piece_id=MY3VGFEQNDCFFCGX4EKY5SRDAEP52HDS5EPR3XXAJCAXP34TKQBQ, seg_pos={0 0}, upload=444333]: Done uploading piece. err: <nil> Dec 26 17:51:46.286020221 (506h22m46.157103256s) [noise=true , address=145.79.227.11:4730 , piece_id=WVALFZ65W45RU7FLO4IOE377B6CKJPZCZULUHQFFBSOMSFTV5VWQ, seg_pos={0 0}, upload=444333]: Done uploading piece. err: <nil> Dec 26 17:51:46.288375025 (506h22m46.15945806s) [noise=true , address=145.79.227.42:4121 , piece_id=SVSSFSUNTQW7RAQWWTRL5I5GIQVHZZJC72JE67T4MOERTUQIU6VA, seg_pos={0 0}, upload=444333]: Done uploading piece. err: <nil> Dec 26 17:51:46.588588915 (506h22m46.45967195s) [noise=true , address=77.90.31.14:4663 , piece_id=UPPD3AXAZGG4P5TTY4TEU2O6JHC6LUBJAVHKC74LCWJHUW3MAOBA, seg_pos={0 0}, upload=444333]: Done uploading piece. err: <nil> Dec 26 17:51:46.590231473 (506h22m46.461314518s) [noise=true , address=45.91.116.14:4190 , piece_id=IZDDTQ6ZDIQINXQGNS3GD6RYVLI5TFDB4TCQT7RPPOHCSGIECNRA, seg_pos={0 0}, upload=444333]: Done uploading piece. err: <nil> Dec 26 17:51:46.603359420 (506h22m46.474442465s) [noise=true , address=77.90.31.17:4770 , piece_id=MV7MNB7IEULOCTO2HP2SXMM3NEYA2OZGPITNVCKNNBAAWBDLGRQA, seg_pos={0 0}, upload=444333]: Done uploading piece. err: <nil> Dec 26 17:51:46.606138262 (506h22m46.477221307s) [noise=true , address=45.91.116.8:4045 , piece_id=LEBFSGXN42BETMF7Y2A7BGF4IRTZS5E4M3UQGPXBAKUPYC2EJZZQ, seg_pos={0 0}, upload=444333]: Done uploading piece. err: <nil> Dec 26 17:51:46.606159363 (506h22m46.477242408s) [seg_pos={0 0}, upload=444333]: Segment reached optimal threshold of 13 pieces. Dec 26 17:51:46.606376275 (506h22m46.47745932s) [noise=true , address=145.79.227.41:4087 , piece_id=TBTU3YCEOW5MBSY3EEN5FOMMAXSVYYT77ZETHVOJIXKVLCNGDQ7Q, seg_pos={0 0}, upload=444333]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR): protocol: write tcp 188.166.225.170:59534->145.79.227.41:4087: use of closed network connection; write tcp 188.166.225.170:59534->145.79.227.41:4087: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59534->145.79.227.41:4087: use of closed network connection Dec 26 17:51:46.606383268 (506h22m46.477466304s) [noise=true , address=145.79.227.41:4087 , piece_id=TBTU3YCEOW5MBSY3EEN5FOMMAXSVYYT77ZETHVOJIXKVLCNGDQ7Q, seg_pos={0 0}, upload=444333]: Long tail cancellation due to optimal threshold Dec 26 17:51:46.606386765 (506h22m46.47746981s) [noise=true , address=145.79.227.12:4765 , piece_id=ZT5AWKLZU2V7JS7BXIPAOJQQ3WFJDNIR7TRSD4QMH66KA6AVS44A, seg_pos={0 0}, upload=444333]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV): protocol: write tcp 188.166.225.170:40256->145.79.227.12:4765: use of closed network connection; write tcp 188.166.225.170:40256->145.79.227.12:4765: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40256->145.79.227.12:4765: use of closed network connection Dec 26 17:51:46.606397215 (506h22m46.47748027s) [noise=true , address=145.79.227.12:4765 , piece_id=ZT5AWKLZU2V7JS7BXIPAOJQQ3WFJDNIR7TRSD4QMH66KA6AVS44A, seg_pos={0 0}, upload=444333]: Long tail cancellation due to optimal threshold Dec 26 17:51:46.606452631 (506h22m46.477535676s) [seg_pos={0 0}, upload=444333]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 17:51:46.606720050 (506h22m46.477803095s) [upload=444333]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:51:46.606726051 (506h22m46.477809096s) [upload=444333]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:51:46.895225284 (506h22m46.766308329s) [upload=444333]: Done uploading segments... Dec 26 17:51:46.895243990 (506h22m46.766327025s) [upload=444333]: Upload finished. err: <nil> Duration: 30.003101001s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 40.889787223s trace: https://jaeger.dev.storj.io/trace/665ea65f6e9ccc33 FAIL (2025-12-26 17:52:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== e=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789316979 (506h12m54.660400024s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789326507 (506h12m54.660409552s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789611339 (506h12m54.660694384s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789619275 (506h12m54.66070231s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789914006 (506h12m54.660997041s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789949443 (506h12m54.661032478s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790181566 (506h12m54.661264601s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790295632 (506h12m54.661378668s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790510872 (506h12m54.661593917s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790692980 (506h12m54.661776015s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790787379 (506h12m54.661870414s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790992970 (506h12m54.662076005s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791120053 (506h12m54.662203098s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791275218 (506h12m54.662358253s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791443709 (506h12m54.662526744s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:56.112313145 (506h12m55.98339619s) [upload=444191]: Next returned nil segment Dec 26 17:41:56.112330638 (506h12m55.983413683s) [upload=444191]: Waiting for error group managing segments... Dec 26 17:41:56.485322720 (506h12m56.356405765s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.485637079 (506h12m56.356720124s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.666807539 (506h12m56.537890584s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.678753987 (506h12m56.549837032s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.687788039 (506h12m56.558871074s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.694318023 (506h12m56.565401058s) [noise=true , address=145.79.227.7:4326 , piece_id=DMQ5RPI46X2U7TJTS42ZHQEF32BDJRKJJO5HWOYAPNS6VWVG5B5Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.716581187 (506h12m56.587664232s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.972183007 (506h12m56.843266042s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.011917999 (506h12m56.883001034s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.054168622 (506h12m56.925251667s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.114312777 (506h12m56.985395812s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.499846250 (506h12m57.370929285s) [noise=true , address=145.79.227.34:4919 , piece_id=YRWHW3BJVEGVJRSA6RJB3T7VJQSDTGKEZQKRICFIV2G3T7AN4MIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.612649066 (506h12m57.483732101s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649413733 (506h12m57.520496768s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649440282 (506h12m57.520523328s) [seg_pos={0 0}, upload=444191]: Segment reached optimal threshold of 14 pieces. Dec 26 17:41:57.649674729 (506h12m57.520757774s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G7ty71tPwe7cZSBNK9u74DFUCGomnxDsNvAif4C3SbP7HWwfo): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649684087 (506h12m57.520767142s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649747708 (506h12m57.520830763s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v6zXZuMmyL2MU8qb3q6gg1iwTYvndjzMjVpv8btEpeX9Lnqmu): protocol: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection Dec 26 17:41:57.649785219 (506h12m57.520868274s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649899828 (506h12m57.520982903s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649908775 (506h12m57.52099182s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650074641 (506h12m57.521157696s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.650083287 (506h12m57.521166342s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650132821 (506h12m57.521215866s) [seg_pos={0 0}, upload=444191]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 17:41:57.650220138 (506h12m57.521303183s) [upload=444191]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:41:57.650227171 (506h12m57.521310206s) [upload=444191]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:41:57.958890870 (506h12m57.829973915s) [upload=444191]: Done uploading segments... Dec 26 17:41:57.958911620 (506h12m57.829994665s) [upload=444191]: Upload finished. err: <nil> Duration: 40.889818943s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 22.912489619s trace: https://jaeger.dev.storj.io/trace/665ea65f6e9ccc33 FAIL (2025-12-26 17:51:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== e=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789316979 (506h12m54.660400024s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789326507 (506h12m54.660409552s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789611339 (506h12m54.660694384s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789619275 (506h12m54.66070231s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789914006 (506h12m54.660997041s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789949443 (506h12m54.661032478s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790181566 (506h12m54.661264601s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790295632 (506h12m54.661378668s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790510872 (506h12m54.661593917s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790692980 (506h12m54.661776015s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790787379 (506h12m54.661870414s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790992970 (506h12m54.662076005s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791120053 (506h12m54.662203098s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791275218 (506h12m54.662358253s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791443709 (506h12m54.662526744s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:56.112313145 (506h12m55.98339619s) [upload=444191]: Next returned nil segment Dec 26 17:41:56.112330638 (506h12m55.983413683s) [upload=444191]: Waiting for error group managing segments... Dec 26 17:41:56.485322720 (506h12m56.356405765s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.485637079 (506h12m56.356720124s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.666807539 (506h12m56.537890584s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.678753987 (506h12m56.549837032s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.687788039 (506h12m56.558871074s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.694318023 (506h12m56.565401058s) [noise=true , address=145.79.227.7:4326 , piece_id=DMQ5RPI46X2U7TJTS42ZHQEF32BDJRKJJO5HWOYAPNS6VWVG5B5Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.716581187 (506h12m56.587664232s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.972183007 (506h12m56.843266042s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.011917999 (506h12m56.883001034s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.054168622 (506h12m56.925251667s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.114312777 (506h12m56.985395812s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.499846250 (506h12m57.370929285s) [noise=true , address=145.79.227.34:4919 , piece_id=YRWHW3BJVEGVJRSA6RJB3T7VJQSDTGKEZQKRICFIV2G3T7AN4MIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.612649066 (506h12m57.483732101s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649413733 (506h12m57.520496768s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649440282 (506h12m57.520523328s) [seg_pos={0 0}, upload=444191]: Segment reached optimal threshold of 14 pieces. Dec 26 17:41:57.649674729 (506h12m57.520757774s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G7ty71tPwe7cZSBNK9u74DFUCGomnxDsNvAif4C3SbP7HWwfo): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649684087 (506h12m57.520767142s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649747708 (506h12m57.520830763s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v6zXZuMmyL2MU8qb3q6gg1iwTYvndjzMjVpv8btEpeX9Lnqmu): protocol: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection Dec 26 17:41:57.649785219 (506h12m57.520868274s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649899828 (506h12m57.520982903s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649908775 (506h12m57.52099182s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650074641 (506h12m57.521157696s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.650083287 (506h12m57.521166342s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650132821 (506h12m57.521215866s) [seg_pos={0 0}, upload=444191]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 17:41:57.650220138 (506h12m57.521303183s) [upload=444191]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:41:57.650227171 (506h12m57.521310206s) [upload=444191]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:41:57.958890870 (506h12m57.829973915s) [upload=444191]: Done uploading segments... Dec 26 17:41:57.958911620 (506h12m57.829994665s) [upload=444191]: Upload finished. err: <nil> Duration: 22.912520167s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 38.303469472s trace: https://jaeger.dev.storj.io/trace/665ea65f6e9ccc33 FAIL (2025-12-26 17:50:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== e=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789316979 (506h12m54.660400024s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789326507 (506h12m54.660409552s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789611339 (506h12m54.660694384s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789619275 (506h12m54.66070231s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789914006 (506h12m54.660997041s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789949443 (506h12m54.661032478s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790181566 (506h12m54.661264601s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790295632 (506h12m54.661378668s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790510872 (506h12m54.661593917s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790692980 (506h12m54.661776015s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790787379 (506h12m54.661870414s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790992970 (506h12m54.662076005s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791120053 (506h12m54.662203098s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791275218 (506h12m54.662358253s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791443709 (506h12m54.662526744s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:56.112313145 (506h12m55.98339619s) [upload=444191]: Next returned nil segment Dec 26 17:41:56.112330638 (506h12m55.983413683s) [upload=444191]: Waiting for error group managing segments... Dec 26 17:41:56.485322720 (506h12m56.356405765s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.485637079 (506h12m56.356720124s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.666807539 (506h12m56.537890584s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.678753987 (506h12m56.549837032s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.687788039 (506h12m56.558871074s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.694318023 (506h12m56.565401058s) [noise=true , address=145.79.227.7:4326 , piece_id=DMQ5RPI46X2U7TJTS42ZHQEF32BDJRKJJO5HWOYAPNS6VWVG5B5Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.716581187 (506h12m56.587664232s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.972183007 (506h12m56.843266042s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.011917999 (506h12m56.883001034s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.054168622 (506h12m56.925251667s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.114312777 (506h12m56.985395812s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.499846250 (506h12m57.370929285s) [noise=true , address=145.79.227.34:4919 , piece_id=YRWHW3BJVEGVJRSA6RJB3T7VJQSDTGKEZQKRICFIV2G3T7AN4MIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.612649066 (506h12m57.483732101s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649413733 (506h12m57.520496768s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649440282 (506h12m57.520523328s) [seg_pos={0 0}, upload=444191]: Segment reached optimal threshold of 14 pieces. Dec 26 17:41:57.649674729 (506h12m57.520757774s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G7ty71tPwe7cZSBNK9u74DFUCGomnxDsNvAif4C3SbP7HWwfo): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649684087 (506h12m57.520767142s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649747708 (506h12m57.520830763s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v6zXZuMmyL2MU8qb3q6gg1iwTYvndjzMjVpv8btEpeX9Lnqmu): protocol: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection Dec 26 17:41:57.649785219 (506h12m57.520868274s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649899828 (506h12m57.520982903s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649908775 (506h12m57.52099182s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650074641 (506h12m57.521157696s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.650083287 (506h12m57.521166342s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650132821 (506h12m57.521215866s) [seg_pos={0 0}, upload=444191]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 17:41:57.650220138 (506h12m57.521303183s) [upload=444191]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:41:57.650227171 (506h12m57.521310206s) [upload=444191]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:41:57.958890870 (506h12m57.829973915s) [upload=444191]: Done uploading segments... Dec 26 17:41:57.958911620 (506h12m57.829994665s) [upload=444191]: Upload finished. err: <nil> Duration: 38.303485894s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [......XX..X....X.X.XX...XX...XX....XX..XX......X.XXX.....XX.] duration: 5.817956153s trace: https://jaeger.dev.storj.io/trace/35e50250d8ec4d18 FAIL (2025-12-26 17:50:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 49:45.546933233 (506h20m45.418016268s) [upload=444303]: Starting upload Dec 26 17:49:45.547023725 (506h20m45.41810678s) [upload=444303]: Done starting upload Dec 26 17:49:45.547161267 (506h20m45.418244302s) [upload=444303]: Uploading segments... Dec 26 17:49:45.547605252 (506h20m45.418688297s) [upload=444303]: Got next segment. Inline: false Dec 26 17:49:45.547667982 (506h20m45.418751027s) [upload=444303]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 17:49:45.547672871 (506h20m45.418755906s) [upload=444303]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 17:49:45.798759755 (506h20m45.66984279s) [seg_pos={0 0}, upload=444303]: Begin upload segment... Dec 26 17:49:45.798803067 (506h20m45.669886102s) [seg_pos={0 0}, upload=444303]: Done begin upload segment. Dec 26 17:49:45.798827774 (506h20m45.669910809s) [noise=true , address=145.79.227.42:4123 , piece_id=PCF57B3KN5GCF66PIC4W25NWKPAREHUATMLNICZW2QORLZF5JXFA, seg_pos={0 0}, upload=444303]: Uploading piece... Dec 26 17:49:45.798988950 (506h20m45.670071985s) [noise=true , address=145.79.227.17:4970 , piece_id=5PBTPJ7CASBOGJHB6RNOKKB4QORQTNOCPRJHLSWHDKNFCJNRAVRQ, seg_pos={0 0}, upload=444303]: Uploading piece... Dec 26 17:49:45.799347293 (506h20m45.670430339s) [noise=true , address=145.79.227.41:4085 , piece_id=O2QUW63WBL5QCPWJMPKXLR7JFYKXOHDTE7FONSIO3C4KSWSG7FIA, seg_pos={0 0}, upload=444303]: Uploading piece... Dec 26 17:49:45.799818831 (506h20m45.670901876s) [noise=true , address=145.79.227.45:4231 , piece_id=YJNSD2UPVC44FPSZOSF2JJXUUFEKDT7ILHPHTNMRBJ6YTFNAUSCA, seg_pos={0 0}, upload=444303]: Uploading piece... Dec 26 17:49:45.800132238 (506h20m45.671215283s) [noise=true , address=45.91.116.11:4119 , piece_id=BQB6HVHKCV5MIX3J3N4IEG4K6PTQXHFVBFXHUC2RCHLSDYP6G4BA, seg_pos={0 0}, upload=444303]: Uploading piece... Dec 26 17:49:45.800396832 (506h20m45.671479867s) [noise=true , address=145.79.227.37:4567 , piece_id=HZLKTO2RYIIWJUYDKLBCBXM4OODKRWYVN7TX6S3NC4ALBDDSOK5A, seg_pos={0 0}, upload=444303]: Uploading piece... Dec 26 17:49:45.800665002 (506h20m45.671748037s) [noise=true , address=145.79.227.22:4511 , piece_id=HIFEE7NRC6URJD73GSDPWWQKQZ53ATEX3ILPGI3UWX4HAINHQZFQ, seg_pos={0 0}, upload=444303]: Uploading piece... Dec 26 17:49:45.800938272 (506h20m45.672021307s) [noise=true , address=145.79.227.35:4482 , piece_id=BDEH4XYBMLMEU7Y6C3R3C22NC5Z5SNIUALARXRZ3Y5LL4JX45K2Q, seg_pos={0 0}, upload=444303]: Uploading piece... Dec 26 17:49:45.801177298 (506h20m45.672260333s) [noise=true , address=145.79.227.38:4979 , piece_id=6OXP7ITZMGRMQPZGRKYMSWJPRFLAGV32TUK5V2PWW6SYVXMFY4JA, seg_pos={0 0}, upload=444303]: Uploading piece... Dec 26 17:49:45.801436361 (506h20m45.672519396s) [noise=true , address=45.91.116.12:4225 , piece_id=2JBAXZKMZPPGUACKEBK7M2QOU77X5G3A7NUNXDGRMWIS62COFKPA, seg_pos={0 0}, upload=444303]: Uploading piece... Dec 26 17:49:45.802369117 (506h20m45.673452152s) [noise=true , address=145.79.227.44:4194 , piece_id=DN36VWFP3N2CMTWMOXAMX7ZZPLBS5ZCGPDLMIAYIG3QIWXBGO4QA, seg_pos={0 0}, upload=444303]: Uploading piece... Dec 26 17:49:45.802775130 (506h20m45.673858175s) [noise=true , address=77.90.31.16:4736 , piece_id=IARA5RUI2HGVCDMQBC2E66OHLSA2NAPOS2GLZ3SMUPX536PDYOCA, seg_pos={0 0}, upload=444303]: Uploading piece... Dec 26 17:49:45.803110639 (506h20m45.674193674s) [noise=true , address=145.79.227.51:4411 , piece_id=U3MYVOJ5NCRZZ277CVQMSQYABZYHOLEHQOZI4NJACKRXLFOX2TNA, seg_pos={0 0}, upload=444303]: Uploading piece... Dec 26 17:49:45.803459403 (506h20m45.674542439s) [noise=true , address=45.91.116.5:4297 , piece_id=U7IN34XCBKJ6UUOCW3YPYLJ7YNYPVAYACUO5H254KZFWSSTMZ63Q, seg_pos={0 0}, upload=444303]: Uploading piece... Dec 26 17:49:45.803820701 (506h20m45.674903736s) [noise=true , address=145.79.227.15:4874 , piece_id=DL7H3UQQZZ6DWHL23YN7VN5RFLJ7CY6PLSOAJ2UMCBV2TPW3NKPA, seg_pos={0 0}, upload=444303]: Uploading piece... Dec 26 17:49:45.804169205 (506h20m45.67525224s) [noise=true , address=145.79.227.48:4303 , piece_id=GPUCXYNHJKXZFNTQAPMPMLF65X3W5MWLHTGQMDQL3K6UCHJDAY2A, seg_pos={0 0}, upload=444303]: Uploading piece... Dec 26 17:49:47.502662332 (506h20m47.373745377s) [upload=444303]: Next returned nil segment Dec 26 17:49:47.502676088 (506h20m47.373759133s) [upload=444303]: Waiting for error group managing segments... Dec 26 17:49:47.790650903 (506h20m47.661733938s) [noise=true , address=145.79.227.42:4123 , piece_id=PCF57B3KN5GCF66PIC4W25NWKPAREHUATMLNICZW2QORLZF5JXFA, seg_pos={0 0}, upload=444303]: Done uploading piece. err: <nil> Dec 26 17:49:47.805161804 (506h20m47.676244839s) [noise=true , address=145.79.227.35:4482 , piece_id=BDEH4XYBMLMEU7Y6C3R3C22NC5Z5SNIUALARXRZ3Y5LL4JX45K2Q, seg_pos={0 0}, upload=444303]: Done uploading piece. err: <nil> Dec 26 17:49:47.805680091 (506h20m47.676763126s) [noise=true , address=145.79.227.37:4567 , piece_id=HZLKTO2RYIIWJUYDKLBCBXM4OODKRWYVN7TX6S3NC4ALBDDSOK5A, seg_pos={0 0}, upload=444303]: Done uploading piece. err: <nil> Dec 26 17:49:47.808020628 (506h20m47.679103663s) [noise=true , address=145.79.227.15:4874 , piece_id=DL7H3UQQZZ6DWHL23YN7VN5RFLJ7CY6PLSOAJ2UMCBV2TPW3NKPA, seg_pos={0 0}, upload=444303]: Done uploading piece. err: <nil> Dec 26 17:49:47.821567703 (506h20m47.692650738s) [noise=true , address=145.79.227.45:4231 , piece_id=YJNSD2UPVC44FPSZOSF2JJXUUFEKDT7ILHPHTNMRBJ6YTFNAUSCA, seg_pos={0 0}, upload=444303]: Done uploading piece. err: <nil> Dec 26 17:49:47.822683879 (506h20m47.693766914s) [noise=true , address=145.79.227.51:4411 , piece_id=U3MYVOJ5NCRZZ277CVQMSQYABZYHOLEHQOZI4NJACKRXLFOX2TNA, seg_pos={0 0}, upload=444303]: Done uploading piece. err: <nil> Dec 26 17:49:47.823197246 (506h20m47.694280282s) [noise=true , address=145.79.227.22:4511 , piece_id=HIFEE7NRC6URJD73GSDPWWQKQZ53ATEX3ILPGI3UWX4HAINHQZFQ, seg_pos={0 0}, upload=444303]: Done uploading piece. err: <nil> Dec 26 17:49:47.836677435 (506h20m47.70776048s) [noise=true , address=145.79.227.48:4303 , piece_id=GPUCXYNHJKXZFNTQAPMPMLF65X3W5MWLHTGQMDQL3K6UCHJDAY2A, seg_pos={0 0}, upload=444303]: Done uploading piece. err: <nil> Dec 26 17:49:47.843337307 (506h20m47.714420342s) [noise=true , address=145.79.227.38:4979 , piece_id=6OXP7ITZMGRMQPZGRKYMSWJPRFLAGV32TUK5V2PWW6SYVXMFY4JA, seg_pos={0 0}, upload=444303]: Done uploading piece. err: <nil> Dec 26 17:49:48.083197271 (506h20m47.954280307s) [noise=true , address=77.90.31.16:4736 , piece_id=IARA5RUI2HGVCDMQBC2E66OHLSA2NAPOS2GLZ3SMUPX536PDYOCA, seg_pos={0 0}, upload=444303]: Done uploading piece. err: <nil> Dec 26 17:49:48.141985204 (506h20m48.013068249s) [noise=true , address=45.91.116.12:4225 , piece_id=2JBAXZKMZPPGUACKEBK7M2QOU77X5G3A7NUNXDGRMWIS62COFKPA, seg_pos={0 0}, upload=444303]: Done uploading piece. err: <nil> Dec 26 17:49:48.144369094 (506h20m48.015452139s) [noise=true , address=145.79.227.41:4085 , piece_id=O2QUW63WBL5QCPWJMPKXLR7JFYKXOHDTE7FONSIO3C4KSWSG7FIA, seg_pos={0 0}, upload=444303]: Done uploading piece. err: <nil> Dec 26 17:49:48.171456162 (506h20m48.042539207s) [noise=true , address=45.91.116.11:4119 , piece_id=BQB6HVHKCV5MIX3J3N4IEG4K6PTQXHFVBFXHUC2RCHLSDYP6G4BA, seg_pos={0 0}, upload=444303]: Done uploading piece. err: <nil> Dec 26 17:49:48.171484226 (506h20m48.042567261s) [seg_pos={0 0}, upload=444303]: Segment reached optimal threshold of 13 pieces. Dec 26 17:49:48.171729252 (506h20m48.042812297s) [noise=true , address=145.79.227.17:4970 , piece_id=5PBTPJ7CASBOGJHB6RNOKKB4QORQTNOCPRJHLSWHDKNFCJNRAVRQ, seg_pos={0 0}, upload=444303]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe): protocol: write tcp 188.166.225.170:34658->145.79.227.17:4970: use of closed network connection; write tcp 188.166.225.170:34658->145.79.227.17:4970: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34658->145.79.227.17:4970: use of closed network connection Dec 26 17:49:48.171740664 (506h20m48.042823709s) [noise=true , address=145.79.227.17:4970 , piece_id=5PBTPJ7CASBOGJHB6RNOKKB4QORQTNOCPRJHLSWHDKNFCJNRAVRQ, seg_pos={0 0}, upload=444303]: Long tail cancellation due to optimal threshold Dec 26 17:49:48.171789948 (506h20m48.042872983s) [noise=true , address=145.79.227.44:4194 , piece_id=DN36VWFP3N2CMTWMOXAMX7ZZPLBS5ZCGPDLMIAYIG3QIWXBGO4QA, seg_pos={0 0}, upload=444303]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 26 17:49:48.171797983 (506h20m48.042881018s) [noise=true , address=145.79.227.44:4194 , piece_id=DN36VWFP3N2CMTWMOXAMX7ZZPLBS5ZCGPDLMIAYIG3QIWXBGO4QA, seg_pos={0 0}, upload=444303]: Long tail cancellation due to optimal threshold Dec 26 17:49:48.172073217 (506h20m48.043156253s) [noise=true , address=45.91.116.5:4297 , piece_id=U7IN34XCBKJ6UUOCW3YPYLJ7YNYPVAYACUO5H254KZFWSSTMZ63Q, seg_pos={0 0}, upload=444303]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 26 17:49:48.172080601 (506h20m48.043163636s) [noise=true , address=45.91.116.5:4297 , piece_id=U7IN34XCBKJ6UUOCW3YPYLJ7YNYPVAYACUO5H254KZFWSSTMZ63Q, seg_pos={0 0}, upload=444303]: Long tail cancellation due to optimal threshold Dec 26 17:49:48.172097663 (506h20m48.043180708s) [seg_pos={0 0}, upload=444303]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 17:49:48.172200780 (506h20m48.043283815s) [upload=444303]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:49:48.172210990 (506h20m48.043294035s) [upload=444303]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:49:48.474685714 (506h20m48.345768769s) [upload=444303]: Done uploading segments... Dec 26 17:49:48.474708668 (506h20m48.345791713s) [upload=444303]: Upload finished. err: <nil> Duration: 5.817986792s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 47.075496393s trace: https://jaeger.dev.storj.io/trace/665ea65f6e9ccc33 FAIL (2025-12-26 17:50:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== e=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789316979 (506h12m54.660400024s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789326507 (506h12m54.660409552s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789611339 (506h12m54.660694384s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789619275 (506h12m54.66070231s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789914006 (506h12m54.660997041s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789949443 (506h12m54.661032478s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790181566 (506h12m54.661264601s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790295632 (506h12m54.661378668s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790510872 (506h12m54.661593917s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790692980 (506h12m54.661776015s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790787379 (506h12m54.661870414s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790992970 (506h12m54.662076005s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791120053 (506h12m54.662203098s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791275218 (506h12m54.662358253s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791443709 (506h12m54.662526744s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:56.112313145 (506h12m55.98339619s) [upload=444191]: Next returned nil segment Dec 26 17:41:56.112330638 (506h12m55.983413683s) [upload=444191]: Waiting for error group managing segments... Dec 26 17:41:56.485322720 (506h12m56.356405765s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.485637079 (506h12m56.356720124s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.666807539 (506h12m56.537890584s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.678753987 (506h12m56.549837032s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.687788039 (506h12m56.558871074s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.694318023 (506h12m56.565401058s) [noise=true , address=145.79.227.7:4326 , piece_id=DMQ5RPI46X2U7TJTS42ZHQEF32BDJRKJJO5HWOYAPNS6VWVG5B5Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.716581187 (506h12m56.587664232s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.972183007 (506h12m56.843266042s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.011917999 (506h12m56.883001034s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.054168622 (506h12m56.925251667s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.114312777 (506h12m56.985395812s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.499846250 (506h12m57.370929285s) [noise=true , address=145.79.227.34:4919 , piece_id=YRWHW3BJVEGVJRSA6RJB3T7VJQSDTGKEZQKRICFIV2G3T7AN4MIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.612649066 (506h12m57.483732101s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649413733 (506h12m57.520496768s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649440282 (506h12m57.520523328s) [seg_pos={0 0}, upload=444191]: Segment reached optimal threshold of 14 pieces. Dec 26 17:41:57.649674729 (506h12m57.520757774s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G7ty71tPwe7cZSBNK9u74DFUCGomnxDsNvAif4C3SbP7HWwfo): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649684087 (506h12m57.520767142s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649747708 (506h12m57.520830763s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v6zXZuMmyL2MU8qb3q6gg1iwTYvndjzMjVpv8btEpeX9Lnqmu): protocol: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection Dec 26 17:41:57.649785219 (506h12m57.520868274s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649899828 (506h12m57.520982903s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649908775 (506h12m57.52099182s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650074641 (506h12m57.521157696s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.650083287 (506h12m57.521166342s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650132821 (506h12m57.521215866s) [seg_pos={0 0}, upload=444191]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 17:41:57.650220138 (506h12m57.521303183s) [upload=444191]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:41:57.650227171 (506h12m57.521310206s) [upload=444191]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:41:57.958890870 (506h12m57.829973915s) [upload=444191]: Done uploading segments... Dec 26 17:41:57.958911620 (506h12m57.829994665s) [upload=444191]: Upload finished. err: <nil> Duration: 47.075520208s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [X......XX..X....X.X.XX...XX...XX....XX..XX......X.XXX.....XX] duration: 30.002812252s trace: https://jaeger.dev.storj.io/trace/35e50250d8ec4d18 FAIL (2025-12-26 17:50:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 49:45.546933233 (506h20m45.418016268s) [upload=444303]: Starting upload Dec 26 17:49:45.547023725 (506h20m45.41810678s) [upload=444303]: Done starting upload Dec 26 17:49:45.547161267 (506h20m45.418244302s) [upload=444303]: Uploading segments... Dec 26 17:49:45.547605252 (506h20m45.418688297s) [upload=444303]: Got next segment. Inline: false Dec 26 17:49:45.547667982 (506h20m45.418751027s) [upload=444303]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 17:49:45.547672871 (506h20m45.418755906s) [upload=444303]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 17:49:45.798759755 (506h20m45.66984279s) [seg_pos={0 0}, upload=444303]: Begin upload segment... Dec 26 17:49:45.798803067 (506h20m45.669886102s) [seg_pos={0 0}, upload=444303]: Done begin upload segment. Dec 26 17:49:45.798827774 (506h20m45.669910809s) [noise=true , address=145.79.227.42:4123 , piece_id=PCF57B3KN5GCF66PIC4W25NWKPAREHUATMLNICZW2QORLZF5JXFA, seg_pos={0 0}, upload=444303]: Uploading piece... Dec 26 17:49:45.798988950 (506h20m45.670071985s) [noise=true , address=145.79.227.17:4970 , piece_id=5PBTPJ7CASBOGJHB6RNOKKB4QORQTNOCPRJHLSWHDKNFCJNRAVRQ, seg_pos={0 0}, upload=444303]: Uploading piece... Dec 26 17:49:45.799347293 (506h20m45.670430339s) [noise=true , address=145.79.227.41:4085 , piece_id=O2QUW63WBL5QCPWJMPKXLR7JFYKXOHDTE7FONSIO3C4KSWSG7FIA, seg_pos={0 0}, upload=444303]: Uploading piece... Dec 26 17:49:45.799818831 (506h20m45.670901876s) [noise=true , address=145.79.227.45:4231 , piece_id=YJNSD2UPVC44FPSZOSF2JJXUUFEKDT7ILHPHTNMRBJ6YTFNAUSCA, seg_pos={0 0}, upload=444303]: Uploading piece... Dec 26 17:49:45.800132238 (506h20m45.671215283s) [noise=true , address=45.91.116.11:4119 , piece_id=BQB6HVHKCV5MIX3J3N4IEG4K6PTQXHFVBFXHUC2RCHLSDYP6G4BA, seg_pos={0 0}, upload=444303]: Uploading piece... Dec 26 17:49:45.800396832 (506h20m45.671479867s) [noise=true , address=145.79.227.37:4567 , piece_id=HZLKTO2RYIIWJUYDKLBCBXM4OODKRWYVN7TX6S3NC4ALBDDSOK5A, seg_pos={0 0}, upload=444303]: Uploading piece... Dec 26 17:49:45.800665002 (506h20m45.671748037s) [noise=true , address=145.79.227.22:4511 , piece_id=HIFEE7NRC6URJD73GSDPWWQKQZ53ATEX3ILPGI3UWX4HAINHQZFQ, seg_pos={0 0}, upload=444303]: Uploading piece... Dec 26 17:49:45.800938272 (506h20m45.672021307s) [noise=true , address=145.79.227.35:4482 , piece_id=BDEH4XYBMLMEU7Y6C3R3C22NC5Z5SNIUALARXRZ3Y5LL4JX45K2Q, seg_pos={0 0}, upload=444303]: Uploading piece... Dec 26 17:49:45.801177298 (506h20m45.672260333s) [noise=true , address=145.79.227.38:4979 , piece_id=6OXP7ITZMGRMQPZGRKYMSWJPRFLAGV32TUK5V2PWW6SYVXMFY4JA, seg_pos={0 0}, upload=444303]: Uploading piece... Dec 26 17:49:45.801436361 (506h20m45.672519396s) [noise=true , address=45.91.116.12:4225 , piece_id=2JBAXZKMZPPGUACKEBK7M2QOU77X5G3A7NUNXDGRMWIS62COFKPA, seg_pos={0 0}, upload=444303]: Uploading piece... Dec 26 17:49:45.802369117 (506h20m45.673452152s) [noise=true , address=145.79.227.44:4194 , piece_id=DN36VWFP3N2CMTWMOXAMX7ZZPLBS5ZCGPDLMIAYIG3QIWXBGO4QA, seg_pos={0 0}, upload=444303]: Uploading piece... Dec 26 17:49:45.802775130 (506h20m45.673858175s) [noise=true , address=77.90.31.16:4736 , piece_id=IARA5RUI2HGVCDMQBC2E66OHLSA2NAPOS2GLZ3SMUPX536PDYOCA, seg_pos={0 0}, upload=444303]: Uploading piece... Dec 26 17:49:45.803110639 (506h20m45.674193674s) [noise=true , address=145.79.227.51:4411 , piece_id=U3MYVOJ5NCRZZ277CVQMSQYABZYHOLEHQOZI4NJACKRXLFOX2TNA, seg_pos={0 0}, upload=444303]: Uploading piece... Dec 26 17:49:45.803459403 (506h20m45.674542439s) [noise=true , address=45.91.116.5:4297 , piece_id=U7IN34XCBKJ6UUOCW3YPYLJ7YNYPVAYACUO5H254KZFWSSTMZ63Q, seg_pos={0 0}, upload=444303]: Uploading piece... Dec 26 17:49:45.803820701 (506h20m45.674903736s) [noise=true , address=145.79.227.15:4874 , piece_id=DL7H3UQQZZ6DWHL23YN7VN5RFLJ7CY6PLSOAJ2UMCBV2TPW3NKPA, seg_pos={0 0}, upload=444303]: Uploading piece... Dec 26 17:49:45.804169205 (506h20m45.67525224s) [noise=true , address=145.79.227.48:4303 , piece_id=GPUCXYNHJKXZFNTQAPMPMLF65X3W5MWLHTGQMDQL3K6UCHJDAY2A, seg_pos={0 0}, upload=444303]: Uploading piece... Dec 26 17:49:47.502662332 (506h20m47.373745377s) [upload=444303]: Next returned nil segment Dec 26 17:49:47.502676088 (506h20m47.373759133s) [upload=444303]: Waiting for error group managing segments... Dec 26 17:49:47.790650903 (506h20m47.661733938s) [noise=true , address=145.79.227.42:4123 , piece_id=PCF57B3KN5GCF66PIC4W25NWKPAREHUATMLNICZW2QORLZF5JXFA, seg_pos={0 0}, upload=444303]: Done uploading piece. err: <nil> Dec 26 17:49:47.805161804 (506h20m47.676244839s) [noise=true , address=145.79.227.35:4482 , piece_id=BDEH4XYBMLMEU7Y6C3R3C22NC5Z5SNIUALARXRZ3Y5LL4JX45K2Q, seg_pos={0 0}, upload=444303]: Done uploading piece. err: <nil> Dec 26 17:49:47.805680091 (506h20m47.676763126s) [noise=true , address=145.79.227.37:4567 , piece_id=HZLKTO2RYIIWJUYDKLBCBXM4OODKRWYVN7TX6S3NC4ALBDDSOK5A, seg_pos={0 0}, upload=444303]: Done uploading piece. err: <nil> Dec 26 17:49:47.808020628 (506h20m47.679103663s) [noise=true , address=145.79.227.15:4874 , piece_id=DL7H3UQQZZ6DWHL23YN7VN5RFLJ7CY6PLSOAJ2UMCBV2TPW3NKPA, seg_pos={0 0}, upload=444303]: Done uploading piece. err: <nil> Dec 26 17:49:47.821567703 (506h20m47.692650738s) [noise=true , address=145.79.227.45:4231 , piece_id=YJNSD2UPVC44FPSZOSF2JJXUUFEKDT7ILHPHTNMRBJ6YTFNAUSCA, seg_pos={0 0}, upload=444303]: Done uploading piece. err: <nil> Dec 26 17:49:47.822683879 (506h20m47.693766914s) [noise=true , address=145.79.227.51:4411 , piece_id=U3MYVOJ5NCRZZ277CVQMSQYABZYHOLEHQOZI4NJACKRXLFOX2TNA, seg_pos={0 0}, upload=444303]: Done uploading piece. err: <nil> Dec 26 17:49:47.823197246 (506h20m47.694280282s) [noise=true , address=145.79.227.22:4511 , piece_id=HIFEE7NRC6URJD73GSDPWWQKQZ53ATEX3ILPGI3UWX4HAINHQZFQ, seg_pos={0 0}, upload=444303]: Done uploading piece. err: <nil> Dec 26 17:49:47.836677435 (506h20m47.70776048s) [noise=true , address=145.79.227.48:4303 , piece_id=GPUCXYNHJKXZFNTQAPMPMLF65X3W5MWLHTGQMDQL3K6UCHJDAY2A, seg_pos={0 0}, upload=444303]: Done uploading piece. err: <nil> Dec 26 17:49:47.843337307 (506h20m47.714420342s) [noise=true , address=145.79.227.38:4979 , piece_id=6OXP7ITZMGRMQPZGRKYMSWJPRFLAGV32TUK5V2PWW6SYVXMFY4JA, seg_pos={0 0}, upload=444303]: Done uploading piece. err: <nil> Dec 26 17:49:48.083197271 (506h20m47.954280307s) [noise=true , address=77.90.31.16:4736 , piece_id=IARA5RUI2HGVCDMQBC2E66OHLSA2NAPOS2GLZ3SMUPX536PDYOCA, seg_pos={0 0}, upload=444303]: Done uploading piece. err: <nil> Dec 26 17:49:48.141985204 (506h20m48.013068249s) [noise=true , address=45.91.116.12:4225 , piece_id=2JBAXZKMZPPGUACKEBK7M2QOU77X5G3A7NUNXDGRMWIS62COFKPA, seg_pos={0 0}, upload=444303]: Done uploading piece. err: <nil> Dec 26 17:49:48.144369094 (506h20m48.015452139s) [noise=true , address=145.79.227.41:4085 , piece_id=O2QUW63WBL5QCPWJMPKXLR7JFYKXOHDTE7FONSIO3C4KSWSG7FIA, seg_pos={0 0}, upload=444303]: Done uploading piece. err: <nil> Dec 26 17:49:48.171456162 (506h20m48.042539207s) [noise=true , address=45.91.116.11:4119 , piece_id=BQB6HVHKCV5MIX3J3N4IEG4K6PTQXHFVBFXHUC2RCHLSDYP6G4BA, seg_pos={0 0}, upload=444303]: Done uploading piece. err: <nil> Dec 26 17:49:48.171484226 (506h20m48.042567261s) [seg_pos={0 0}, upload=444303]: Segment reached optimal threshold of 13 pieces. Dec 26 17:49:48.171729252 (506h20m48.042812297s) [noise=true , address=145.79.227.17:4970 , piece_id=5PBTPJ7CASBOGJHB6RNOKKB4QORQTNOCPRJHLSWHDKNFCJNRAVRQ, seg_pos={0 0}, upload=444303]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe): protocol: write tcp 188.166.225.170:34658->145.79.227.17:4970: use of closed network connection; write tcp 188.166.225.170:34658->145.79.227.17:4970: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34658->145.79.227.17:4970: use of closed network connection Dec 26 17:49:48.171740664 (506h20m48.042823709s) [noise=true , address=145.79.227.17:4970 , piece_id=5PBTPJ7CASBOGJHB6RNOKKB4QORQTNOCPRJHLSWHDKNFCJNRAVRQ, seg_pos={0 0}, upload=444303]: Long tail cancellation due to optimal threshold Dec 26 17:49:48.171789948 (506h20m48.042872983s) [noise=true , address=145.79.227.44:4194 , piece_id=DN36VWFP3N2CMTWMOXAMX7ZZPLBS5ZCGPDLMIAYIG3QIWXBGO4QA, seg_pos={0 0}, upload=444303]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 26 17:49:48.171797983 (506h20m48.042881018s) [noise=true , address=145.79.227.44:4194 , piece_id=DN36VWFP3N2CMTWMOXAMX7ZZPLBS5ZCGPDLMIAYIG3QIWXBGO4QA, seg_pos={0 0}, upload=444303]: Long tail cancellation due to optimal threshold Dec 26 17:49:48.172073217 (506h20m48.043156253s) [noise=true , address=45.91.116.5:4297 , piece_id=U7IN34XCBKJ6UUOCW3YPYLJ7YNYPVAYACUO5H254KZFWSSTMZ63Q, seg_pos={0 0}, upload=444303]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 26 17:49:48.172080601 (506h20m48.043163636s) [noise=true , address=45.91.116.5:4297 , piece_id=U7IN34XCBKJ6UUOCW3YPYLJ7YNYPVAYACUO5H254KZFWSSTMZ63Q, seg_pos={0 0}, upload=444303]: Long tail cancellation due to optimal threshold Dec 26 17:49:48.172097663 (506h20m48.043180708s) [seg_pos={0 0}, upload=444303]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 17:49:48.172200780 (506h20m48.043283815s) [upload=444303]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:49:48.172210990 (506h20m48.043294035s) [upload=444303]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:49:48.474685714 (506h20m48.345768769s) [upload=444303]: Done uploading segments... Dec 26 17:49:48.474708668 (506h20m48.345791713s) [upload=444303]: Upload finished. err: <nil> Duration: 30.002823563s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [.X......XX..X....X.X.XX...XX...XX....XX..XX......X.XXX.....X] duration: 30.003380744s trace: https://jaeger.dev.storj.io/trace/3924adfdb49627e7 FAIL (2025-12-26 17:49:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: MTL6PGPEIXILZ7QNHK7CJQL7VEV7MPZD4ZPDF5OWLGLJPONHXVKA) context deadline exceeded; piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: 5GUNH3O2NJBNOUQIH3YJNLA7YI2RKWA2MEFK24KJLJLBFPRV7GLQ) context deadline exceeded ======LOGS====== Dec 26 17:49:15.289574008 (506h20m15.160657053s) [upload=444298]: Starting upload Dec 26 17:49:15.289667135 (506h20m15.16075017s) [upload=444298]: Done starting upload Dec 26 17:49:15.289718463 (506h20m15.160801498s) [upload=444298]: Uploading segments... Dec 26 17:49:15.289954463 (506h20m15.161037508s) [upload=444298]: Got next segment. Inline: false Dec 26 17:49:15.290000941 (506h20m15.161083996s) [upload=444298]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 17:49:15.290006361 (506h20m15.161089396s) [upload=444298]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 17:49:15.536008492 (506h20m15.407091547s) [seg_pos={0 0}, upload=444298]: Begin upload segment... Dec 26 17:49:15.536070210 (506h20m15.407153245s) [seg_pos={0 0}, upload=444298]: Done begin upload segment. Dec 26 17:49:15.536088765 (506h20m15.40717181s) [noise=true , address=45.91.116.9:4263 , piece_id=5GUNH3O2NJBNOUQIH3YJNLA7YI2RKWA2MEFK24KJLJLBFPRV7GLQ, seg_pos={0 0}, upload=444298]: Uploading piece... Dec 26 17:49:15.536114233 (506h20m15.407197268s) [noise=true , address=145.79.227.50:4375 , piece_id=HY5DOH636DYPS2UPKUPTA5L7O6BXUJVU2SVCPPYUQVAIQXXD7QJA, seg_pos={0 0}, upload=444298]: Uploading piece... Dec 26 17:49:15.536173667 (506h20m15.407256702s) [noise=true , address=145.79.227.49:4337 , piece_id=3FDPV4WTBLXQLAVKM5AZG6K57PZI3H7UNB5EYJE6SQSCL5DZLQWQ, seg_pos={0 0}, upload=444298]: Uploading piece... Dec 26 17:49:15.536609557 (506h20m15.407692602s) [noise=true , address=145.79.227.22:4511 , piece_id=IN72QA7IUWXLFIIPRA7ZDRY5OSVXCEZPBGB6DXWRBDXHFZB2GFHA, seg_pos={0 0}, upload=444298]: Uploading piece... Dec 26 17:49:15.537449126 (506h20m15.408532161s) [noise=true , address=45.91.116.6:4976 , piece_id=YZILLOJJQHM22YI63N6RQGLCKDFUBUMRM3H475YSDQJG3WBG3FJA, seg_pos={0 0}, upload=444298]: Uploading piece... Dec 26 17:49:15.537898542 (506h20m15.408981587s) [noise=true , address=77.90.31.14:4662 , piece_id=QN4JZEDFD6O73SOHQXGBV4HSJ5GXWUFN5KMA4NG64ICN5CMDC5LA, seg_pos={0 0}, upload=444298]: Uploading piece... Dec 26 17:49:15.538312891 (506h20m15.409395926s) [noise=true , address=145.79.227.46:4601 , piece_id=JOBBOXGBU2NMNYMBYK7A7ZSOWO375VKUY2RXQS7F5JATSOJBLIKA, seg_pos={0 0}, upload=444298]: Uploading piece... Dec 26 17:49:15.538786131 (506h20m15.409869176s) [noise=true , address=45.91.116.5:4300 , piece_id=MTL6PGPEIXILZ7QNHK7CJQL7VEV7MPZD4ZPDF5OWLGLJPONHXVKA, seg_pos={0 0}, upload=444298]: Uploading piece... Dec 26 17:49:15.539287657 (506h20m15.410370702s) [noise=true , address=145.79.227.4:4185 , piece_id=VTUPQDZ4ZE33VRBTERIJLVED7UI2Q5SWOQU7HSSAB63L4444DHOA, seg_pos={0 0}, upload=444298]: Uploading piece... Dec 26 17:49:15.539808367 (506h20m15.410891412s) [noise=true , address=145.79.227.51:4410 , piece_id=OBOHTWXWELFMY6RELPQEC46MQOSZWNTJXGKXKC43VM7LACHSZHDA, seg_pos={0 0}, upload=444298]: Uploading piece... Dec 26 17:49:15.540943649 (506h20m15.412026695s) [noise=true , address=45.91.116.10:4082 , piece_id=I7SNOQZP2ABOQCTJ4SJRM3UZKD32AZ2LENWSRTPEESX4IVNXVITQ, seg_pos={0 0}, upload=444298]: Uploading piece... Dec 26 17:49:15.541215267 (506h20m15.412298302s) [noise=true , address=145.79.227.14:4839 , piece_id=GHIQWMQPBJOY6UZP6S2BK5LIGAKBMOMBJGVTAVH47DWJPB5E53IA, seg_pos={0 0}, upload=444298]: Uploading piece... Dec 26 17:49:15.542706527 (506h20m15.413789572s) [noise=true , address=145.79.227.21:4475 , piece_id=TRQTFZ4MLBMU5VLWPEUVBR7IRTMFSEE6JQBQYMVLWHTNIVLDRWMQ, seg_pos={0 0}, upload=444298]: Uploading piece... Dec 26 17:49:15.542949750 (506h20m15.414032785s) [noise=true , address=145.79.227.37:4565 , piece_id=TBKSYNWDGU7LRU2P5NEJCZZDWUAJXJS7WK7TD66E6W5MZHXZ2FOA, seg_pos={0 0}, upload=444298]: Uploading piece... Dec 26 17:49:15.544793941 (506h20m15.415876987s) [noise=true , address=145.79.227.39:4013 , piece_id=F6QHPEDTSFUWOUI6CJNVH5WX76WM5FYJEHDT2OVMRLLESLNEONOQ, seg_pos={0 0}, upload=444298]: Uploading piece... Dec 26 17:49:15.546619567 (506h20m15.417702612s) [noise=true , address=145.79.227.16:4909 , piece_id=EYTTQTIDKRRJOCKR6G423QKGNT6XNL67RPZ3NO23HYL4HTGUX75Q, seg_pos={0 0}, upload=444298]: Uploading piece... Dec 26 17:49:17.246937189 (506h20m17.118020234s) [upload=444298]: Next returned nil segment Dec 26 17:49:17.246957578 (506h20m17.118040623s) [upload=444298]: Waiting for error group managing segments... Dec 26 17:49:17.529375160 (506h20m17.400458205s) [noise=true , address=145.79.227.50:4375 , piece_id=HY5DOH636DYPS2UPKUPTA5L7O6BXUJVU2SVCPPYUQVAIQXXD7QJA, seg_pos={0 0}, upload=444298]: Done uploading piece. err: <nil> Dec 26 17:49:17.542613168 (506h20m17.413696203s) [noise=true , address=145.79.227.49:4337 , piece_id=3FDPV4WTBLXQLAVKM5AZG6K57PZI3H7UNB5EYJE6SQSCL5DZLQWQ, seg_pos={0 0}, upload=444298]: Done uploading piece. err: <nil> Dec 26 17:49:17.545456011 (506h20m17.416539056s) [noise=true , address=145.79.227.39:4013 , piece_id=F6QHPEDTSFUWOUI6CJNVH5WX76WM5FYJEHDT2OVMRLLESLNEONOQ, seg_pos={0 0}, upload=444298]: Done uploading piece. err: <nil> Dec 26 17:49:17.546061995 (506h20m17.41714504s) [noise=true , address=145.79.227.21:4475 , piece_id=TRQTFZ4MLBMU5VLWPEUVBR7IRTMFSEE6JQBQYMVLWHTNIVLDRWMQ, seg_pos={0 0}, upload=444298]: Done uploading piece. err: <nil> Dec 26 17:49:17.546381954 (506h20m17.41746499s) [noise=true , address=145.79.227.51:4410 , piece_id=OBOHTWXWELFMY6RELPQEC46MQOSZWNTJXGKXKC43VM7LACHSZHDA, seg_pos={0 0}, upload=444298]: Done uploading piece. err: <nil> Dec 26 17:49:17.550722269 (506h20m17.421805305s) [noise=true , address=145.79.227.4:4185 , piece_id=VTUPQDZ4ZE33VRBTERIJLVED7UI2Q5SWOQU7HSSAB63L4444DHOA, seg_pos={0 0}, upload=444298]: Done uploading piece. err: <nil> Dec 26 17:49:17.554602247 (506h20m17.425685282s) [noise=true , address=145.79.227.22:4511 , piece_id=IN72QA7IUWXLFIIPRA7ZDRY5OSVXCEZPBGB6DXWRBDXHFZB2GFHA, seg_pos={0 0}, upload=444298]: Done uploading piece. err: <nil> Dec 26 17:49:17.568298828 (506h20m17.439381873s) [noise=true , address=145.79.227.37:4565 , piece_id=TBKSYNWDGU7LRU2P5NEJCZZDWUAJXJS7WK7TD66E6W5MZHXZ2FOA, seg_pos={0 0}, upload=444298]: Done uploading piece. err: <nil> Dec 26 17:49:17.570158719 (506h20m17.441241755s) [noise=true , address=145.79.227.46:4601 , piece_id=JOBBOXGBU2NMNYMBYK7A7ZSOWO375VKUY2RXQS7F5JATSOJBLIKA, seg_pos={0 0}, upload=444298]: Done uploading piece. err: <nil> Dec 26 17:49:17.585858535 (506h20m17.45694158s) [noise=true , address=145.79.227.16:4909 , piece_id=EYTTQTIDKRRJOCKR6G423QKGNT6XNL67RPZ3NO23HYL4HTGUX75Q, seg_pos={0 0}, upload=444298]: Done uploading piece. err: <nil> Dec 26 17:49:17.636854440 (506h20m17.507937475s) [noise=true , address=145.79.227.14:4839 , piece_id=GHIQWMQPBJOY6UZP6S2BK5LIGAKBMOMBJGVTAVH47DWJPB5E53IA, seg_pos={0 0}, upload=444298]: Done uploading piece. err: <nil> Dec 26 17:49:17.768189515 (506h20m17.63927255s) [noise=true , address=45.91.116.5:4300 , piece_id=MTL6PGPEIXILZ7QNHK7CJQL7VEV7MPZD4ZPDF5OWLGLJPONHXVKA, seg_pos={0 0}, upload=444298]: Done uploading piece. err: <nil> Dec 26 17:49:17.772556360 (506h20m17.643639395s) [noise=true , address=45.91.116.9:4263 , piece_id=5GUNH3O2NJBNOUQIH3YJNLA7YI2RKWA2MEFK24KJLJLBFPRV7GLQ, seg_pos={0 0}, upload=444298]: Done uploading piece. err: <nil> Dec 26 17:49:17.772575025 (506h20m17.64365807s) [seg_pos={0 0}, upload=444298]: Segment reached optimal threshold of 13 pieces. Dec 26 17:49:17.772782631 (506h20m17.643865676s) [noise=true , address=45.91.116.10:4082 , piece_id=I7SNOQZP2ABOQCTJ4SJRM3UZKD32AZ2LENWSRTPEESX4IVNXVITQ, seg_pos={0 0}, upload=444298]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 26 17:49:17.772790626 (506h20m17.643873661s) [noise=true , address=45.91.116.10:4082 , piece_id=I7SNOQZP2ABOQCTJ4SJRM3UZKD32AZ2LENWSRTPEESX4IVNXVITQ, seg_pos={0 0}, upload=444298]: Long tail cancellation due to optimal threshold Dec 26 17:49:17.772852263 (506h20m17.643935308s) [noise=true , address=45.91.116.6:4976 , piece_id=YZILLOJJQHM22YI63N6RQGLCKDFUBUMRM3H475YSDQJG3WBG3FJA, seg_pos={0 0}, upload=444298]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Dec 26 17:49:17.772863325 (506h20m17.64394637s) [noise=true , address=45.91.116.6:4976 , piece_id=YZILLOJJQHM22YI63N6RQGLCKDFUBUMRM3H475YSDQJG3WBG3FJA, seg_pos={0 0}, upload=444298]: Long tail cancellation due to optimal threshold Dec 26 17:49:17.773038038 (506h20m17.644121073s) [noise=true , address=77.90.31.14:4662 , piece_id=QN4JZEDFD6O73SOHQXGBV4HSJ5GXWUFN5KMA4NG64ICN5CMDC5LA, seg_pos={0 0}, upload=444298]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Dec 26 17:49:17.773045361 (506h20m17.644128396s) [noise=true , address=77.90.31.14:4662 , piece_id=QN4JZEDFD6O73SOHQXGBV4HSJ5GXWUFN5KMA4NG64ICN5CMDC5LA, seg_pos={0 0}, upload=444298]: Long tail cancellation due to optimal threshold Dec 26 17:49:17.773096729 (506h20m17.644179774s) [seg_pos={0 0}, upload=444298]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 17:49:17.773192491 (506h20m17.644275576s) [upload=444298]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:49:17.773206097 (506h20m17.644289182s) [upload=444298]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:49:18.065456723 (506h20m17.936539758s) [upload=444298]: Done uploading segments... Dec 26 17:49:18.065475199 (506h20m17.936558244s) [upload=444298]: Upload finished. err: <nil> Duration: 30.00341454s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 25.784887869s trace: https://jaeger.dev.storj.io/trace/665ea65f6e9ccc33 FAIL (2025-12-26 17:48:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== e=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789316979 (506h12m54.660400024s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789326507 (506h12m54.660409552s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789611339 (506h12m54.660694384s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789619275 (506h12m54.66070231s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789914006 (506h12m54.660997041s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789949443 (506h12m54.661032478s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790181566 (506h12m54.661264601s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790295632 (506h12m54.661378668s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790510872 (506h12m54.661593917s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790692980 (506h12m54.661776015s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790787379 (506h12m54.661870414s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790992970 (506h12m54.662076005s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791120053 (506h12m54.662203098s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791275218 (506h12m54.662358253s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791443709 (506h12m54.662526744s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:56.112313145 (506h12m55.98339619s) [upload=444191]: Next returned nil segment Dec 26 17:41:56.112330638 (506h12m55.983413683s) [upload=444191]: Waiting for error group managing segments... Dec 26 17:41:56.485322720 (506h12m56.356405765s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.485637079 (506h12m56.356720124s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.666807539 (506h12m56.537890584s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.678753987 (506h12m56.549837032s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.687788039 (506h12m56.558871074s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.694318023 (506h12m56.565401058s) [noise=true , address=145.79.227.7:4326 , piece_id=DMQ5RPI46X2U7TJTS42ZHQEF32BDJRKJJO5HWOYAPNS6VWVG5B5Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.716581187 (506h12m56.587664232s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.972183007 (506h12m56.843266042s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.011917999 (506h12m56.883001034s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.054168622 (506h12m56.925251667s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.114312777 (506h12m56.985395812s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.499846250 (506h12m57.370929285s) [noise=true , address=145.79.227.34:4919 , piece_id=YRWHW3BJVEGVJRSA6RJB3T7VJQSDTGKEZQKRICFIV2G3T7AN4MIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.612649066 (506h12m57.483732101s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649413733 (506h12m57.520496768s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649440282 (506h12m57.520523328s) [seg_pos={0 0}, upload=444191]: Segment reached optimal threshold of 14 pieces. Dec 26 17:41:57.649674729 (506h12m57.520757774s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G7ty71tPwe7cZSBNK9u74DFUCGomnxDsNvAif4C3SbP7HWwfo): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649684087 (506h12m57.520767142s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649747708 (506h12m57.520830763s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v6zXZuMmyL2MU8qb3q6gg1iwTYvndjzMjVpv8btEpeX9Lnqmu): protocol: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection Dec 26 17:41:57.649785219 (506h12m57.520868274s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649899828 (506h12m57.520982903s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649908775 (506h12m57.52099182s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650074641 (506h12m57.521157696s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.650083287 (506h12m57.521166342s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650132821 (506h12m57.521215866s) [seg_pos={0 0}, upload=444191]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 17:41:57.650220138 (506h12m57.521303183s) [upload=444191]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:41:57.650227171 (506h12m57.521310206s) [upload=444191]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:41:57.958890870 (506h12m57.829973915s) [upload=444191]: Done uploading segments... Dec 26 17:41:57.958911620 (506h12m57.829994665s) [upload=444191]: Upload finished. err: <nil> Duration: 25.784911104s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 33.5971771s trace: https://jaeger.dev.storj.io/trace/665ea65f6e9ccc33 FAIL (2025-12-26 17:48:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== e=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789316979 (506h12m54.660400024s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789326507 (506h12m54.660409552s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789611339 (506h12m54.660694384s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789619275 (506h12m54.66070231s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789914006 (506h12m54.660997041s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789949443 (506h12m54.661032478s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790181566 (506h12m54.661264601s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790295632 (506h12m54.661378668s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790510872 (506h12m54.661593917s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790692980 (506h12m54.661776015s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790787379 (506h12m54.661870414s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790992970 (506h12m54.662076005s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791120053 (506h12m54.662203098s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791275218 (506h12m54.662358253s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791443709 (506h12m54.662526744s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:56.112313145 (506h12m55.98339619s) [upload=444191]: Next returned nil segment Dec 26 17:41:56.112330638 (506h12m55.983413683s) [upload=444191]: Waiting for error group managing segments... Dec 26 17:41:56.485322720 (506h12m56.356405765s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.485637079 (506h12m56.356720124s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.666807539 (506h12m56.537890584s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.678753987 (506h12m56.549837032s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.687788039 (506h12m56.558871074s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.694318023 (506h12m56.565401058s) [noise=true , address=145.79.227.7:4326 , piece_id=DMQ5RPI46X2U7TJTS42ZHQEF32BDJRKJJO5HWOYAPNS6VWVG5B5Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.716581187 (506h12m56.587664232s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.972183007 (506h12m56.843266042s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.011917999 (506h12m56.883001034s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.054168622 (506h12m56.925251667s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.114312777 (506h12m56.985395812s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.499846250 (506h12m57.370929285s) [noise=true , address=145.79.227.34:4919 , piece_id=YRWHW3BJVEGVJRSA6RJB3T7VJQSDTGKEZQKRICFIV2G3T7AN4MIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.612649066 (506h12m57.483732101s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649413733 (506h12m57.520496768s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649440282 (506h12m57.520523328s) [seg_pos={0 0}, upload=444191]: Segment reached optimal threshold of 14 pieces. Dec 26 17:41:57.649674729 (506h12m57.520757774s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G7ty71tPwe7cZSBNK9u74DFUCGomnxDsNvAif4C3SbP7HWwfo): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649684087 (506h12m57.520767142s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649747708 (506h12m57.520830763s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v6zXZuMmyL2MU8qb3q6gg1iwTYvndjzMjVpv8btEpeX9Lnqmu): protocol: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection Dec 26 17:41:57.649785219 (506h12m57.520868274s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649899828 (506h12m57.520982903s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649908775 (506h12m57.52099182s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650074641 (506h12m57.521157696s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.650083287 (506h12m57.521166342s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650132821 (506h12m57.521215866s) [seg_pos={0 0}, upload=444191]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 17:41:57.650220138 (506h12m57.521303183s) [upload=444191]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:41:57.650227171 (506h12m57.521310206s) [upload=444191]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:41:57.958890870 (506h12m57.829973915s) [upload=444191]: Done uploading segments... Dec 26 17:41:57.958911620 (506h12m57.829994665s) [upload=444191]: Upload finished. err: <nil> Duration: 33.597191157s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [X.X......XX..X....X.X.XX...XX...XX....XX..XX......X.XXX.....] duration: 6.075010131s trace: https://jaeger.dev.storj.io/trace/3b761e3de974ea48 FAIL (2025-12-26 17:48:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA, Piece ID: LCYNTQGR2ZIUX6HBLYVSCUTCLAU6PFGHSA2ICB6DWE2ZFOI22NSA) context deadline exceeded; piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: DZPD74AMFXUMJYQEFLME4ZX7NRSLDIYE3ZBR5CIKSJE3XPXANUPA) context deadline exceeded ======LOGS====== Dec 26 17:43:10.974317425 (506h14m10.84540046s) [upload=444208]: Starting upload Dec 26 17:43:10.974446491 (506h14m10.845529526s) [upload=444208]: Done starting upload Dec 26 17:43:10.974581899 (506h14m10.845664944s) [upload=444208]: Uploading segments... Dec 26 17:43:10.974665328 (506h14m10.845748383s) [upload=444208]: Got next segment. Inline: false Dec 26 17:43:10.974721515 (506h14m10.84580456s) [upload=444208]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 17:43:10.974752394 (506h14m10.845835439s) [upload=444208]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 17:43:11.225224176 (506h14m11.096307221s) [seg_pos={0 0}, upload=444208]: Begin upload segment... Dec 26 17:43:11.225287356 (506h14m11.096370411s) [seg_pos={0 0}, upload=444208]: Done begin upload segment. Dec 26 17:43:11.225329857 (506h14m11.096412902s) [noise=true , address=145.79.227.46:4601 , piece_id=IM6NZ7Z4XQSEP4XG4WBWT4Y6P5DUX62KAMLQJANCBKSAU6ZVYRLA, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.225917767 (506h14m11.097000802s) [noise=true , address=145.79.227.8:4330 , piece_id=HEEUM6HSWGUYQFB2NP5T4DZST5GUJEDFNTLWDS6GEZC75P54UJQQ, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.231054878 (506h14m11.102137913s) [noise=true , address=145.79.227.49:4338 , piece_id=LNEIJJAJA7Q6J6ZGVQRPQFZELDCNKU2VKXX64K4ET6XVJ52LGVDQ, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.232622754 (506h14m11.103705799s) [noise=true , address=45.91.116.12:4228 , piece_id=LCYNTQGR2ZIUX6HBLYVSCUTCLAU6PFGHSA2ICB6DWE2ZFOI22NSA, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.234863150 (506h14m11.105946186s) [noise=true , address=145.79.227.45:4230 , piece_id=RKFJEJXGFXRGQZ2B3B3U3XZATBUAPFSV3IVYZQVIS63Q34RNCOSQ, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.236499346 (506h14m11.107582411s) [noise=true , address=45.91.116.6:4974 , piece_id=VKPME5624PBSFP3HLHKANNY7SBEVFL4Z636HEN2KXUQ55XPORJ2A, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.238064495 (506h14m11.10914753s) [noise=true , address=145.79.227.35:4481 , piece_id=AWRLZ2QPXDUTJWC2J6TCOM7QROIZJENTYBVWB6T2KPR2B6YYXS4A, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.243205044 (506h14m11.114288089s) [noise=true , address=145.79.227.37:4567 , piece_id=SLFOZCOSWTYSU5W3VXSJVHO6ZJSGWBUUMZKTWEV5YMDQONFHN53A, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.245389073 (506h14m11.116472108s) [noise=true , address=145.79.227.15:4874 , piece_id=TIWMO4OASWXEVFEG4COHHNDNGKHZUWZGTP6D6NNCR5PEZRSBYVIA, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.248134351 (506h14m11.119217426s) [noise=true , address=45.91.116.14:4191 , piece_id=DZPD74AMFXUMJYQEFLME4ZX7NRSLDIYE3ZBR5CIKSJE3XPXANUPA, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.251152537 (506h14m11.122235572s) [noise=true , address=145.79.227.3:4151 , piece_id=4QA7ARRZII6ZRW3GSYBHQVFBGDY4F676MVVJMQHMKWBZ2WS7P5VA, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.253552006 (506h14m11.124635051s) [noise=true , address=77.90.31.21:4880 , piece_id=KYMCS7O4EGMIVDSALBAUBNP7Q3TO5ZX4PLQ4KWBMKSWRUC7TM6EQ, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.255859751 (506h14m11.126942797s) [noise=true , address=145.79.227.22:4511 , piece_id=G7HZFZ7DF6NEWMGRUQQH4UOPUZIYVOVDH22TZNCDWTSQWB3ZEFGQ, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.256186012 (506h14m11.127269057s) [noise=true , address=145.79.227.48:4302 , piece_id=EGYOF4SZ3PKE3H6QYM3QHHRMITTGH55TPNY5D4NS5FSSTKLTML3Q, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.257647495 (506h14m11.12873056s) [noise=true , address=145.79.227.50:4373 , piece_id=DXRYJICGTQJBT7NDE6UUYPDPJZ2HD6MVGRTVZX5GAXXKJFSJ47FA, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.260156352 (506h14m11.131239398s) [noise=true , address=45.91.116.8:4048 , piece_id=JPMATYG4OTAACBWLQKX64FUDV7ONG4P4YZLF4SI4EVGBXOYSICLA, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:12.947257377 (506h14m12.818340412s) [upload=444208]: Next returned nil segment Dec 26 17:43:12.947333081 (506h14m12.818416126s) [upload=444208]: Waiting for error group managing segments... Dec 26 17:43:13.273287306 (506h14m13.144370351s) [noise=true , address=145.79.227.46:4601 , piece_id=IM6NZ7Z4XQSEP4XG4WBWT4Y6P5DUX62KAMLQJANCBKSAU6ZVYRLA, seg_pos={0 0}, upload=444208]: Done uploading piece. err: <nil> Dec 26 17:43:13.273494530 (506h14m13.144577575s) [noise=true , address=145.79.227.3:4151 , piece_id=4QA7ARRZII6ZRW3GSYBHQVFBGDY4F676MVVJMQHMKWBZ2WS7P5VA, seg_pos={0 0}, upload=444208]: Done uploading piece. err: <nil> Dec 26 17:43:13.275357928 (506h14m13.146440963s) [noise=true , address=145.79.227.15:4874 , piece_id=TIWMO4OASWXEVFEG4COHHNDNGKHZUWZGTP6D6NNCR5PEZRSBYVIA, seg_pos={0 0}, upload=444208]: Done uploading piece. err: <nil> Dec 26 17:43:13.276417696 (506h14m13.147500742s) [noise=true , address=145.79.227.45:4230 , piece_id=RKFJEJXGFXRGQZ2B3B3U3XZATBUAPFSV3IVYZQVIS63Q34RNCOSQ, seg_pos={0 0}, upload=444208]: Done uploading piece. err: <nil> Dec 26 17:43:13.277361283 (506h14m13.148444318s) [noise=true , address=145.79.227.50:4373 , piece_id=DXRYJICGTQJBT7NDE6UUYPDPJZ2HD6MVGRTVZX5GAXXKJFSJ47FA, seg_pos={0 0}, upload=444208]: Done uploading piece. err: <nil> Dec 26 17:43:13.277397332 (506h14m13.148480367s) [noise=true , address=145.79.227.37:4567 , piece_id=SLFOZCOSWTYSU5W3VXSJVHO6ZJSGWBUUMZKTWEV5YMDQONFHN53A, seg_pos={0 0}, upload=444208]: Done uploading piece. err: <nil> Dec 26 17:43:13.277910097 (506h14m13.148993142s) [noise=true , address=145.79.227.22:4511 , piece_id=G7HZFZ7DF6NEWMGRUQQH4UOPUZIYVOVDH22TZNCDWTSQWB3ZEFGQ, seg_pos={0 0}, upload=444208]: Done uploading piece. err: <nil> Dec 26 17:43:13.289773407 (506h14m13.160856452s) [noise=true , address=145.79.227.49:4338 , piece_id=LNEIJJAJA7Q6J6ZGVQRPQFZELDCNKU2VKXX64K4ET6XVJ52LGVDQ, seg_pos={0 0}, upload=444208]: Done uploading piece. err: <nil> Dec 26 17:43:13.290752031 (506h14m13.161835086s) [noise=true , address=145.79.227.48:4302 , piece_id=EGYOF4SZ3PKE3H6QYM3QHHRMITTGH55TPNY5D4NS5FSSTKLTML3Q, seg_pos={0 0}, upload=444208]: Done uploading piece. err: <nil> Dec 26 17:43:13.294786072 (506h14m13.165869117s) [noise=true , address=145.79.227.8:4330 , piece_id=HEEUM6HSWGUYQFB2NP5T4DZST5GUJEDFNTLWDS6GEZC75P54UJQQ, seg_pos={0 0}, upload=444208]: Done uploading piece. err: <nil> Dec 26 17:43:13.489465767 (506h14m13.360548802s) [noise=true , address=45.91.116.12:4228 , piece_id=LCYNTQGR2ZIUX6HBLYVSCUTCLAU6PFGHSA2ICB6DWE2ZFOI22NSA, seg_pos={0 0}, upload=444208]: Done uploading piece. err: <nil> Dec 26 17:43:13.566172019 (506h14m13.437255064s) [noise=true , address=45.91.116.14:4191 , piece_id=DZPD74AMFXUMJYQEFLME4ZX7NRSLDIYE3ZBR5CIKSJE3XPXANUPA, seg_pos={0 0}, upload=444208]: Done uploading piece. err: <nil> Dec 26 17:43:13.725036525 (506h14m13.59611956s) [noise=true , address=45.91.116.6:4974 , piece_id=VKPME5624PBSFP3HLHKANNY7SBEVFL4Z636HEN2KXUQ55XPORJ2A, seg_pos={0 0}, upload=444208]: Done uploading piece. err: <nil> Dec 26 17:43:13.725057735 (506h14m13.59614077s) [seg_pos={0 0}, upload=444208]: Segment reached optimal threshold of 13 pieces. Dec 26 17:43:13.725271673 (506h14m13.596354708s) [noise=true , address=145.79.227.35:4481 , piece_id=AWRLZ2QPXDUTJWC2J6TCOM7QROIZJENTYBVWB6T2KPR2B6YYXS4A, seg_pos={0 0}, upload=444208]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 26 17:43:13.725279558 (506h14m13.596362593s) [noise=true , address=145.79.227.35:4481 , piece_id=AWRLZ2QPXDUTJWC2J6TCOM7QROIZJENTYBVWB6T2KPR2B6YYXS4A, seg_pos={0 0}, upload=444208]: Long tail cancellation due to optimal threshold Dec 26 17:43:13.725338128 (506h14m13.596421174s) [noise=true , address=77.90.31.21:4880 , piece_id=KYMCS7O4EGMIVDSALBAUBNP7Q3TO5ZX4PLQ4KWBMKSWRUC7TM6EQ, seg_pos={0 0}, upload=444208]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: write tcp 188.166.225.170:37372->77.90.31.21:4880: use of closed network connection; write tcp 188.166.225.170:37372->77.90.31.21:4880: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37372->77.90.31.21:4880: use of closed network connection Dec 26 17:43:13.725346515 (506h14m13.59642955s) [noise=true , address=77.90.31.21:4880 , piece_id=KYMCS7O4EGMIVDSALBAUBNP7Q3TO5ZX4PLQ4KWBMKSWRUC7TM6EQ, seg_pos={0 0}, upload=444208]: Long tail cancellation due to optimal threshold Dec 26 17:43:13.725564680 (506h14m13.596647715s) [noise=true , address=45.91.116.8:4048 , piece_id=JPMATYG4OTAACBWLQKX64FUDV7ONG4P4YZLF4SI4EVGBXOYSICLA, seg_pos={0 0}, upload=444208]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: context canceled; context canceled Dec 26 17:43:13.725572675 (506h14m13.59665572s) [noise=true , address=45.91.116.8:4048 , piece_id=JPMATYG4OTAACBWLQKX64FUDV7ONG4P4YZLF4SI4EVGBXOYSICLA, seg_pos={0 0}, upload=444208]: Long tail cancellation due to optimal threshold Dec 26 17:43:13.725590309 (506h14m13.596673354s) [seg_pos={0 0}, upload=444208]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 17:43:13.725667807 (506h14m13.596750842s) [upload=444208]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:43:13.725675932 (506h14m13.596758977s) [upload=444208]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:43:14.020265812 (506h14m13.891348857s) [upload=444208]: Done uploading segments... Dec 26 17:43:14.020287173 (506h14m13.891370219s) [upload=444208]: Upload finished. err: <nil> Duration: 6.075079571s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 32.202732924s trace: https://jaeger.dev.storj.io/trace/665ea65f6e9ccc33 FAIL (2025-12-26 17:47:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== e=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789316979 (506h12m54.660400024s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789326507 (506h12m54.660409552s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789611339 (506h12m54.660694384s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789619275 (506h12m54.66070231s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789914006 (506h12m54.660997041s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789949443 (506h12m54.661032478s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790181566 (506h12m54.661264601s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790295632 (506h12m54.661378668s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790510872 (506h12m54.661593917s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790692980 (506h12m54.661776015s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790787379 (506h12m54.661870414s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790992970 (506h12m54.662076005s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791120053 (506h12m54.662203098s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791275218 (506h12m54.662358253s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791443709 (506h12m54.662526744s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:56.112313145 (506h12m55.98339619s) [upload=444191]: Next returned nil segment Dec 26 17:41:56.112330638 (506h12m55.983413683s) [upload=444191]: Waiting for error group managing segments... Dec 26 17:41:56.485322720 (506h12m56.356405765s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.485637079 (506h12m56.356720124s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.666807539 (506h12m56.537890584s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.678753987 (506h12m56.549837032s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.687788039 (506h12m56.558871074s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.694318023 (506h12m56.565401058s) [noise=true , address=145.79.227.7:4326 , piece_id=DMQ5RPI46X2U7TJTS42ZHQEF32BDJRKJJO5HWOYAPNS6VWVG5B5Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.716581187 (506h12m56.587664232s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.972183007 (506h12m56.843266042s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.011917999 (506h12m56.883001034s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.054168622 (506h12m56.925251667s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.114312777 (506h12m56.985395812s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.499846250 (506h12m57.370929285s) [noise=true , address=145.79.227.34:4919 , piece_id=YRWHW3BJVEGVJRSA6RJB3T7VJQSDTGKEZQKRICFIV2G3T7AN4MIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.612649066 (506h12m57.483732101s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649413733 (506h12m57.520496768s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649440282 (506h12m57.520523328s) [seg_pos={0 0}, upload=444191]: Segment reached optimal threshold of 14 pieces. Dec 26 17:41:57.649674729 (506h12m57.520757774s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G7ty71tPwe7cZSBNK9u74DFUCGomnxDsNvAif4C3SbP7HWwfo): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649684087 (506h12m57.520767142s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649747708 (506h12m57.520830763s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v6zXZuMmyL2MU8qb3q6gg1iwTYvndjzMjVpv8btEpeX9Lnqmu): protocol: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection Dec 26 17:41:57.649785219 (506h12m57.520868274s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649899828 (506h12m57.520982903s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649908775 (506h12m57.52099182s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650074641 (506h12m57.521157696s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.650083287 (506h12m57.521166342s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650132821 (506h12m57.521215866s) [seg_pos={0 0}, upload=444191]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 17:41:57.650220138 (506h12m57.521303183s) [upload=444191]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:41:57.650227171 (506h12m57.521310206s) [upload=444191]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:41:57.958890870 (506h12m57.829973915s) [upload=444191]: Done uploading segments... Dec 26 17:41:57.958911620 (506h12m57.829994665s) [upload=444191]: Upload finished. err: <nil> Duration: 32.202764103s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [.X.X......XX..X....X.X.XX...XX...XX....XX..XX......X.XXX....] duration: 6.854931927s trace: https://jaeger.dev.storj.io/trace/3b761e3de974ea48 FAIL (2025-12-26 17:47:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA, Piece ID: LCYNTQGR2ZIUX6HBLYVSCUTCLAU6PFGHSA2ICB6DWE2ZFOI22NSA) context deadline exceeded; piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: DZPD74AMFXUMJYQEFLME4ZX7NRSLDIYE3ZBR5CIKSJE3XPXANUPA) context deadline exceeded ======LOGS====== Dec 26 17:43:10.974317425 (506h14m10.84540046s) [upload=444208]: Starting upload Dec 26 17:43:10.974446491 (506h14m10.845529526s) [upload=444208]: Done starting upload Dec 26 17:43:10.974581899 (506h14m10.845664944s) [upload=444208]: Uploading segments... Dec 26 17:43:10.974665328 (506h14m10.845748383s) [upload=444208]: Got next segment. Inline: false Dec 26 17:43:10.974721515 (506h14m10.84580456s) [upload=444208]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 17:43:10.974752394 (506h14m10.845835439s) [upload=444208]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 17:43:11.225224176 (506h14m11.096307221s) [seg_pos={0 0}, upload=444208]: Begin upload segment... Dec 26 17:43:11.225287356 (506h14m11.096370411s) [seg_pos={0 0}, upload=444208]: Done begin upload segment. Dec 26 17:43:11.225329857 (506h14m11.096412902s) [noise=true , address=145.79.227.46:4601 , piece_id=IM6NZ7Z4XQSEP4XG4WBWT4Y6P5DUX62KAMLQJANCBKSAU6ZVYRLA, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.225917767 (506h14m11.097000802s) [noise=true , address=145.79.227.8:4330 , piece_id=HEEUM6HSWGUYQFB2NP5T4DZST5GUJEDFNTLWDS6GEZC75P54UJQQ, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.231054878 (506h14m11.102137913s) [noise=true , address=145.79.227.49:4338 , piece_id=LNEIJJAJA7Q6J6ZGVQRPQFZELDCNKU2VKXX64K4ET6XVJ52LGVDQ, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.232622754 (506h14m11.103705799s) [noise=true , address=45.91.116.12:4228 , piece_id=LCYNTQGR2ZIUX6HBLYVSCUTCLAU6PFGHSA2ICB6DWE2ZFOI22NSA, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.234863150 (506h14m11.105946186s) [noise=true , address=145.79.227.45:4230 , piece_id=RKFJEJXGFXRGQZ2B3B3U3XZATBUAPFSV3IVYZQVIS63Q34RNCOSQ, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.236499346 (506h14m11.107582411s) [noise=true , address=45.91.116.6:4974 , piece_id=VKPME5624PBSFP3HLHKANNY7SBEVFL4Z636HEN2KXUQ55XPORJ2A, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.238064495 (506h14m11.10914753s) [noise=true , address=145.79.227.35:4481 , piece_id=AWRLZ2QPXDUTJWC2J6TCOM7QROIZJENTYBVWB6T2KPR2B6YYXS4A, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.243205044 (506h14m11.114288089s) [noise=true , address=145.79.227.37:4567 , piece_id=SLFOZCOSWTYSU5W3VXSJVHO6ZJSGWBUUMZKTWEV5YMDQONFHN53A, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.245389073 (506h14m11.116472108s) [noise=true , address=145.79.227.15:4874 , piece_id=TIWMO4OASWXEVFEG4COHHNDNGKHZUWZGTP6D6NNCR5PEZRSBYVIA, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.248134351 (506h14m11.119217426s) [noise=true , address=45.91.116.14:4191 , piece_id=DZPD74AMFXUMJYQEFLME4ZX7NRSLDIYE3ZBR5CIKSJE3XPXANUPA, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.251152537 (506h14m11.122235572s) [noise=true , address=145.79.227.3:4151 , piece_id=4QA7ARRZII6ZRW3GSYBHQVFBGDY4F676MVVJMQHMKWBZ2WS7P5VA, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.253552006 (506h14m11.124635051s) [noise=true , address=77.90.31.21:4880 , piece_id=KYMCS7O4EGMIVDSALBAUBNP7Q3TO5ZX4PLQ4KWBMKSWRUC7TM6EQ, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.255859751 (506h14m11.126942797s) [noise=true , address=145.79.227.22:4511 , piece_id=G7HZFZ7DF6NEWMGRUQQH4UOPUZIYVOVDH22TZNCDWTSQWB3ZEFGQ, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.256186012 (506h14m11.127269057s) [noise=true , address=145.79.227.48:4302 , piece_id=EGYOF4SZ3PKE3H6QYM3QHHRMITTGH55TPNY5D4NS5FSSTKLTML3Q, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.257647495 (506h14m11.12873056s) [noise=true , address=145.79.227.50:4373 , piece_id=DXRYJICGTQJBT7NDE6UUYPDPJZ2HD6MVGRTVZX5GAXXKJFSJ47FA, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.260156352 (506h14m11.131239398s) [noise=true , address=45.91.116.8:4048 , piece_id=JPMATYG4OTAACBWLQKX64FUDV7ONG4P4YZLF4SI4EVGBXOYSICLA, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:12.947257377 (506h14m12.818340412s) [upload=444208]: Next returned nil segment Dec 26 17:43:12.947333081 (506h14m12.818416126s) [upload=444208]: Waiting for error group managing segments... Dec 26 17:43:13.273287306 (506h14m13.144370351s) [noise=true , address=145.79.227.46:4601 , piece_id=IM6NZ7Z4XQSEP4XG4WBWT4Y6P5DUX62KAMLQJANCBKSAU6ZVYRLA, seg_pos={0 0}, upload=444208]: Done uploading piece. err: <nil> Dec 26 17:43:13.273494530 (506h14m13.144577575s) [noise=true , address=145.79.227.3:4151 , piece_id=4QA7ARRZII6ZRW3GSYBHQVFBGDY4F676MVVJMQHMKWBZ2WS7P5VA, seg_pos={0 0}, upload=444208]: Done uploading piece. err: <nil> Dec 26 17:43:13.275357928 (506h14m13.146440963s) [noise=true , address=145.79.227.15:4874 , piece_id=TIWMO4OASWXEVFEG4COHHNDNGKHZUWZGTP6D6NNCR5PEZRSBYVIA, seg_pos={0 0}, upload=444208]: Done uploading piece. err: <nil> Dec 26 17:43:13.276417696 (506h14m13.147500742s) [noise=true , address=145.79.227.45:4230 , piece_id=RKFJEJXGFXRGQZ2B3B3U3XZATBUAPFSV3IVYZQVIS63Q34RNCOSQ, seg_pos={0 0}, upload=444208]: Done uploading piece. err: <nil> Dec 26 17:43:13.277361283 (506h14m13.148444318s) [noise=true , address=145.79.227.50:4373 , piece_id=DXRYJICGTQJBT7NDE6UUYPDPJZ2HD6MVGRTVZX5GAXXKJFSJ47FA, seg_pos={0 0}, upload=444208]: Done uploading piece. err: <nil> Dec 26 17:43:13.277397332 (506h14m13.148480367s) [noise=true , address=145.79.227.37:4567 , piece_id=SLFOZCOSWTYSU5W3VXSJVHO6ZJSGWBUUMZKTWEV5YMDQONFHN53A, seg_pos={0 0}, upload=444208]: Done uploading piece. err: <nil> Dec 26 17:43:13.277910097 (506h14m13.148993142s) [noise=true , address=145.79.227.22:4511 , piece_id=G7HZFZ7DF6NEWMGRUQQH4UOPUZIYVOVDH22TZNCDWTSQWB3ZEFGQ, seg_pos={0 0}, upload=444208]: Done uploading piece. err: <nil> Dec 26 17:43:13.289773407 (506h14m13.160856452s) [noise=true , address=145.79.227.49:4338 , piece_id=LNEIJJAJA7Q6J6ZGVQRPQFZELDCNKU2VKXX64K4ET6XVJ52LGVDQ, seg_pos={0 0}, upload=444208]: Done uploading piece. err: <nil> Dec 26 17:43:13.290752031 (506h14m13.161835086s) [noise=true , address=145.79.227.48:4302 , piece_id=EGYOF4SZ3PKE3H6QYM3QHHRMITTGH55TPNY5D4NS5FSSTKLTML3Q, seg_pos={0 0}, upload=444208]: Done uploading piece. err: <nil> Dec 26 17:43:13.294786072 (506h14m13.165869117s) [noise=true , address=145.79.227.8:4330 , piece_id=HEEUM6HSWGUYQFB2NP5T4DZST5GUJEDFNTLWDS6GEZC75P54UJQQ, seg_pos={0 0}, upload=444208]: Done uploading piece. err: <nil> Dec 26 17:43:13.489465767 (506h14m13.360548802s) [noise=true , address=45.91.116.12:4228 , piece_id=LCYNTQGR2ZIUX6HBLYVSCUTCLAU6PFGHSA2ICB6DWE2ZFOI22NSA, seg_pos={0 0}, upload=444208]: Done uploading piece. err: <nil> Dec 26 17:43:13.566172019 (506h14m13.437255064s) [noise=true , address=45.91.116.14:4191 , piece_id=DZPD74AMFXUMJYQEFLME4ZX7NRSLDIYE3ZBR5CIKSJE3XPXANUPA, seg_pos={0 0}, upload=444208]: Done uploading piece. err: <nil> Dec 26 17:43:13.725036525 (506h14m13.59611956s) [noise=true , address=45.91.116.6:4974 , piece_id=VKPME5624PBSFP3HLHKANNY7SBEVFL4Z636HEN2KXUQ55XPORJ2A, seg_pos={0 0}, upload=444208]: Done uploading piece. err: <nil> Dec 26 17:43:13.725057735 (506h14m13.59614077s) [seg_pos={0 0}, upload=444208]: Segment reached optimal threshold of 13 pieces. Dec 26 17:43:13.725271673 (506h14m13.596354708s) [noise=true , address=145.79.227.35:4481 , piece_id=AWRLZ2QPXDUTJWC2J6TCOM7QROIZJENTYBVWB6T2KPR2B6YYXS4A, seg_pos={0 0}, upload=444208]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 26 17:43:13.725279558 (506h14m13.596362593s) [noise=true , address=145.79.227.35:4481 , piece_id=AWRLZ2QPXDUTJWC2J6TCOM7QROIZJENTYBVWB6T2KPR2B6YYXS4A, seg_pos={0 0}, upload=444208]: Long tail cancellation due to optimal threshold Dec 26 17:43:13.725338128 (506h14m13.596421174s) [noise=true , address=77.90.31.21:4880 , piece_id=KYMCS7O4EGMIVDSALBAUBNP7Q3TO5ZX4PLQ4KWBMKSWRUC7TM6EQ, seg_pos={0 0}, upload=444208]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: write tcp 188.166.225.170:37372->77.90.31.21:4880: use of closed network connection; write tcp 188.166.225.170:37372->77.90.31.21:4880: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37372->77.90.31.21:4880: use of closed network connection Dec 26 17:43:13.725346515 (506h14m13.59642955s) [noise=true , address=77.90.31.21:4880 , piece_id=KYMCS7O4EGMIVDSALBAUBNP7Q3TO5ZX4PLQ4KWBMKSWRUC7TM6EQ, seg_pos={0 0}, upload=444208]: Long tail cancellation due to optimal threshold Dec 26 17:43:13.725564680 (506h14m13.596647715s) [noise=true , address=45.91.116.8:4048 , piece_id=JPMATYG4OTAACBWLQKX64FUDV7ONG4P4YZLF4SI4EVGBXOYSICLA, seg_pos={0 0}, upload=444208]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: context canceled; context canceled Dec 26 17:43:13.725572675 (506h14m13.59665572s) [noise=true , address=45.91.116.8:4048 , piece_id=JPMATYG4OTAACBWLQKX64FUDV7ONG4P4YZLF4SI4EVGBXOYSICLA, seg_pos={0 0}, upload=444208]: Long tail cancellation due to optimal threshold Dec 26 17:43:13.725590309 (506h14m13.596673354s) [seg_pos={0 0}, upload=444208]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 17:43:13.725667807 (506h14m13.596750842s) [upload=444208]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:43:13.725675932 (506h14m13.596758977s) [upload=444208]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:43:14.020265812 (506h14m13.891348857s) [upload=444208]: Done uploading segments... Dec 26 17:43:14.020287173 (506h14m13.891370219s) [upload=444208]: Upload finished. err: <nil> Duration: 6.8550234s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 45.875667435s trace: https://jaeger.dev.storj.io/trace/665ea65f6e9ccc33 FAIL (2025-12-26 17:46:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== e=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789316979 (506h12m54.660400024s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789326507 (506h12m54.660409552s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789611339 (506h12m54.660694384s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789619275 (506h12m54.66070231s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789914006 (506h12m54.660997041s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789949443 (506h12m54.661032478s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790181566 (506h12m54.661264601s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790295632 (506h12m54.661378668s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790510872 (506h12m54.661593917s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790692980 (506h12m54.661776015s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790787379 (506h12m54.661870414s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790992970 (506h12m54.662076005s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791120053 (506h12m54.662203098s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791275218 (506h12m54.662358253s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791443709 (506h12m54.662526744s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:56.112313145 (506h12m55.98339619s) [upload=444191]: Next returned nil segment Dec 26 17:41:56.112330638 (506h12m55.983413683s) [upload=444191]: Waiting for error group managing segments... Dec 26 17:41:56.485322720 (506h12m56.356405765s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.485637079 (506h12m56.356720124s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.666807539 (506h12m56.537890584s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.678753987 (506h12m56.549837032s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.687788039 (506h12m56.558871074s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.694318023 (506h12m56.565401058s) [noise=true , address=145.79.227.7:4326 , piece_id=DMQ5RPI46X2U7TJTS42ZHQEF32BDJRKJJO5HWOYAPNS6VWVG5B5Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.716581187 (506h12m56.587664232s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.972183007 (506h12m56.843266042s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.011917999 (506h12m56.883001034s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.054168622 (506h12m56.925251667s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.114312777 (506h12m56.985395812s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.499846250 (506h12m57.370929285s) [noise=true , address=145.79.227.34:4919 , piece_id=YRWHW3BJVEGVJRSA6RJB3T7VJQSDTGKEZQKRICFIV2G3T7AN4MIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.612649066 (506h12m57.483732101s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649413733 (506h12m57.520496768s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649440282 (506h12m57.520523328s) [seg_pos={0 0}, upload=444191]: Segment reached optimal threshold of 14 pieces. Dec 26 17:41:57.649674729 (506h12m57.520757774s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G7ty71tPwe7cZSBNK9u74DFUCGomnxDsNvAif4C3SbP7HWwfo): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649684087 (506h12m57.520767142s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649747708 (506h12m57.520830763s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v6zXZuMmyL2MU8qb3q6gg1iwTYvndjzMjVpv8btEpeX9Lnqmu): protocol: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection Dec 26 17:41:57.649785219 (506h12m57.520868274s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649899828 (506h12m57.520982903s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649908775 (506h12m57.52099182s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650074641 (506h12m57.521157696s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.650083287 (506h12m57.521166342s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650132821 (506h12m57.521215866s) [seg_pos={0 0}, upload=444191]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 17:41:57.650220138 (506h12m57.521303183s) [upload=444191]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:41:57.650227171 (506h12m57.521310206s) [upload=444191]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:41:57.958890870 (506h12m57.829973915s) [upload=444191]: Done uploading segments... Dec 26 17:41:57.958911620 (506h12m57.829994665s) [upload=444191]: Upload finished. err: <nil> Duration: 45.875693586s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [..X.X......XX..X....X.X.XX...XX...XX....XX..XX......X.XXX...] duration: 5.98046039s trace: https://jaeger.dev.storj.io/trace/3b761e3de974ea48 FAIL (2025-12-26 17:46:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA, Piece ID: LCYNTQGR2ZIUX6HBLYVSCUTCLAU6PFGHSA2ICB6DWE2ZFOI22NSA) context deadline exceeded; piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: DZPD74AMFXUMJYQEFLME4ZX7NRSLDIYE3ZBR5CIKSJE3XPXANUPA) context deadline exceeded ======LOGS====== Dec 26 17:43:10.974317425 (506h14m10.84540046s) [upload=444208]: Starting upload Dec 26 17:43:10.974446491 (506h14m10.845529526s) [upload=444208]: Done starting upload Dec 26 17:43:10.974581899 (506h14m10.845664944s) [upload=444208]: Uploading segments... Dec 26 17:43:10.974665328 (506h14m10.845748383s) [upload=444208]: Got next segment. Inline: false Dec 26 17:43:10.974721515 (506h14m10.84580456s) [upload=444208]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 17:43:10.974752394 (506h14m10.845835439s) [upload=444208]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 17:43:11.225224176 (506h14m11.096307221s) [seg_pos={0 0}, upload=444208]: Begin upload segment... Dec 26 17:43:11.225287356 (506h14m11.096370411s) [seg_pos={0 0}, upload=444208]: Done begin upload segment. Dec 26 17:43:11.225329857 (506h14m11.096412902s) [noise=true , address=145.79.227.46:4601 , piece_id=IM6NZ7Z4XQSEP4XG4WBWT4Y6P5DUX62KAMLQJANCBKSAU6ZVYRLA, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.225917767 (506h14m11.097000802s) [noise=true , address=145.79.227.8:4330 , piece_id=HEEUM6HSWGUYQFB2NP5T4DZST5GUJEDFNTLWDS6GEZC75P54UJQQ, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.231054878 (506h14m11.102137913s) [noise=true , address=145.79.227.49:4338 , piece_id=LNEIJJAJA7Q6J6ZGVQRPQFZELDCNKU2VKXX64K4ET6XVJ52LGVDQ, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.232622754 (506h14m11.103705799s) [noise=true , address=45.91.116.12:4228 , piece_id=LCYNTQGR2ZIUX6HBLYVSCUTCLAU6PFGHSA2ICB6DWE2ZFOI22NSA, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.234863150 (506h14m11.105946186s) [noise=true , address=145.79.227.45:4230 , piece_id=RKFJEJXGFXRGQZ2B3B3U3XZATBUAPFSV3IVYZQVIS63Q34RNCOSQ, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.236499346 (506h14m11.107582411s) [noise=true , address=45.91.116.6:4974 , piece_id=VKPME5624PBSFP3HLHKANNY7SBEVFL4Z636HEN2KXUQ55XPORJ2A, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.238064495 (506h14m11.10914753s) [noise=true , address=145.79.227.35:4481 , piece_id=AWRLZ2QPXDUTJWC2J6TCOM7QROIZJENTYBVWB6T2KPR2B6YYXS4A, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.243205044 (506h14m11.114288089s) [noise=true , address=145.79.227.37:4567 , piece_id=SLFOZCOSWTYSU5W3VXSJVHO6ZJSGWBUUMZKTWEV5YMDQONFHN53A, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.245389073 (506h14m11.116472108s) [noise=true , address=145.79.227.15:4874 , piece_id=TIWMO4OASWXEVFEG4COHHNDNGKHZUWZGTP6D6NNCR5PEZRSBYVIA, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.248134351 (506h14m11.119217426s) [noise=true , address=45.91.116.14:4191 , piece_id=DZPD74AMFXUMJYQEFLME4ZX7NRSLDIYE3ZBR5CIKSJE3XPXANUPA, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.251152537 (506h14m11.122235572s) [noise=true , address=145.79.227.3:4151 , piece_id=4QA7ARRZII6ZRW3GSYBHQVFBGDY4F676MVVJMQHMKWBZ2WS7P5VA, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.253552006 (506h14m11.124635051s) [noise=true , address=77.90.31.21:4880 , piece_id=KYMCS7O4EGMIVDSALBAUBNP7Q3TO5ZX4PLQ4KWBMKSWRUC7TM6EQ, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.255859751 (506h14m11.126942797s) [noise=true , address=145.79.227.22:4511 , piece_id=G7HZFZ7DF6NEWMGRUQQH4UOPUZIYVOVDH22TZNCDWTSQWB3ZEFGQ, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.256186012 (506h14m11.127269057s) [noise=true , address=145.79.227.48:4302 , piece_id=EGYOF4SZ3PKE3H6QYM3QHHRMITTGH55TPNY5D4NS5FSSTKLTML3Q, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.257647495 (506h14m11.12873056s) [noise=true , address=145.79.227.50:4373 , piece_id=DXRYJICGTQJBT7NDE6UUYPDPJZ2HD6MVGRTVZX5GAXXKJFSJ47FA, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.260156352 (506h14m11.131239398s) [noise=true , address=45.91.116.8:4048 , piece_id=JPMATYG4OTAACBWLQKX64FUDV7ONG4P4YZLF4SI4EVGBXOYSICLA, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:12.947257377 (506h14m12.818340412s) [upload=444208]: Next returned nil segment Dec 26 17:43:12.947333081 (506h14m12.818416126s) [upload=444208]: Waiting for error group managing segments... Dec 26 17:43:13.273287306 (506h14m13.144370351s) [noise=true , address=145.79.227.46:4601 , piece_id=IM6NZ7Z4XQSEP4XG4WBWT4Y6P5DUX62KAMLQJANCBKSAU6ZVYRLA, seg_pos={0 0}, upload=444208]: Done uploading piece. err: <nil> Dec 26 17:43:13.273494530 (506h14m13.144577575s) [noise=true , address=145.79.227.3:4151 , piece_id=4QA7ARRZII6ZRW3GSYBHQVFBGDY4F676MVVJMQHMKWBZ2WS7P5VA, seg_pos={0 0}, upload=444208]: Done uploading piece. err: <nil> Dec 26 17:43:13.275357928 (506h14m13.146440963s) [noise=true , address=145.79.227.15:4874 , piece_id=TIWMO4OASWXEVFEG4COHHNDNGKHZUWZGTP6D6NNCR5PEZRSBYVIA, seg_pos={0 0}, upload=444208]: Done uploading piece. err: <nil> Dec 26 17:43:13.276417696 (506h14m13.147500742s) [noise=true , address=145.79.227.45:4230 , piece_id=RKFJEJXGFXRGQZ2B3B3U3XZATBUAPFSV3IVYZQVIS63Q34RNCOSQ, seg_pos={0 0}, upload=444208]: Done uploading piece. err: <nil> Dec 26 17:43:13.277361283 (506h14m13.148444318s) [noise=true , address=145.79.227.50:4373 , piece_id=DXRYJICGTQJBT7NDE6UUYPDPJZ2HD6MVGRTVZX5GAXXKJFSJ47FA, seg_pos={0 0}, upload=444208]: Done uploading piece. err: <nil> Dec 26 17:43:13.277397332 (506h14m13.148480367s) [noise=true , address=145.79.227.37:4567 , piece_id=SLFOZCOSWTYSU5W3VXSJVHO6ZJSGWBUUMZKTWEV5YMDQONFHN53A, seg_pos={0 0}, upload=444208]: Done uploading piece. err: <nil> Dec 26 17:43:13.277910097 (506h14m13.148993142s) [noise=true , address=145.79.227.22:4511 , piece_id=G7HZFZ7DF6NEWMGRUQQH4UOPUZIYVOVDH22TZNCDWTSQWB3ZEFGQ, seg_pos={0 0}, upload=444208]: Done uploading piece. err: <nil> Dec 26 17:43:13.289773407 (506h14m13.160856452s) [noise=true , address=145.79.227.49:4338 , piece_id=LNEIJJAJA7Q6J6ZGVQRPQFZELDCNKU2VKXX64K4ET6XVJ52LGVDQ, seg_pos={0 0}, upload=444208]: Done uploading piece. err: <nil> Dec 26 17:43:13.290752031 (506h14m13.161835086s) [noise=true , address=145.79.227.48:4302 , piece_id=EGYOF4SZ3PKE3H6QYM3QHHRMITTGH55TPNY5D4NS5FSSTKLTML3Q, seg_pos={0 0}, upload=444208]: Done uploading piece. err: <nil> Dec 26 17:43:13.294786072 (506h14m13.165869117s) [noise=true , address=145.79.227.8:4330 , piece_id=HEEUM6HSWGUYQFB2NP5T4DZST5GUJEDFNTLWDS6GEZC75P54UJQQ, seg_pos={0 0}, upload=444208]: Done uploading piece. err: <nil> Dec 26 17:43:13.489465767 (506h14m13.360548802s) [noise=true , address=45.91.116.12:4228 , piece_id=LCYNTQGR2ZIUX6HBLYVSCUTCLAU6PFGHSA2ICB6DWE2ZFOI22NSA, seg_pos={0 0}, upload=444208]: Done uploading piece. err: <nil> Dec 26 17:43:13.566172019 (506h14m13.437255064s) [noise=true , address=45.91.116.14:4191 , piece_id=DZPD74AMFXUMJYQEFLME4ZX7NRSLDIYE3ZBR5CIKSJE3XPXANUPA, seg_pos={0 0}, upload=444208]: Done uploading piece. err: <nil> Dec 26 17:43:13.725036525 (506h14m13.59611956s) [noise=true , address=45.91.116.6:4974 , piece_id=VKPME5624PBSFP3HLHKANNY7SBEVFL4Z636HEN2KXUQ55XPORJ2A, seg_pos={0 0}, upload=444208]: Done uploading piece. err: <nil> Dec 26 17:43:13.725057735 (506h14m13.59614077s) [seg_pos={0 0}, upload=444208]: Segment reached optimal threshold of 13 pieces. Dec 26 17:43:13.725271673 (506h14m13.596354708s) [noise=true , address=145.79.227.35:4481 , piece_id=AWRLZ2QPXDUTJWC2J6TCOM7QROIZJENTYBVWB6T2KPR2B6YYXS4A, seg_pos={0 0}, upload=444208]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 26 17:43:13.725279558 (506h14m13.596362593s) [noise=true , address=145.79.227.35:4481 , piece_id=AWRLZ2QPXDUTJWC2J6TCOM7QROIZJENTYBVWB6T2KPR2B6YYXS4A, seg_pos={0 0}, upload=444208]: Long tail cancellation due to optimal threshold Dec 26 17:43:13.725338128 (506h14m13.596421174s) [noise=true , address=77.90.31.21:4880 , piece_id=KYMCS7O4EGMIVDSALBAUBNP7Q3TO5ZX4PLQ4KWBMKSWRUC7TM6EQ, seg_pos={0 0}, upload=444208]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: write tcp 188.166.225.170:37372->77.90.31.21:4880: use of closed network connection; write tcp 188.166.225.170:37372->77.90.31.21:4880: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37372->77.90.31.21:4880: use of closed network connection Dec 26 17:43:13.725346515 (506h14m13.59642955s) [noise=true , address=77.90.31.21:4880 , piece_id=KYMCS7O4EGMIVDSALBAUBNP7Q3TO5ZX4PLQ4KWBMKSWRUC7TM6EQ, seg_pos={0 0}, upload=444208]: Long tail cancellation due to optimal threshold Dec 26 17:43:13.725564680 (506h14m13.596647715s) [noise=true , address=45.91.116.8:4048 , piece_id=JPMATYG4OTAACBWLQKX64FUDV7ONG4P4YZLF4SI4EVGBXOYSICLA, seg_pos={0 0}, upload=444208]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: context canceled; context canceled Dec 26 17:43:13.725572675 (506h14m13.59665572s) [noise=true , address=45.91.116.8:4048 , piece_id=JPMATYG4OTAACBWLQKX64FUDV7ONG4P4YZLF4SI4EVGBXOYSICLA, seg_pos={0 0}, upload=444208]: Long tail cancellation due to optimal threshold Dec 26 17:43:13.725590309 (506h14m13.596673354s) [seg_pos={0 0}, upload=444208]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 17:43:13.725667807 (506h14m13.596750842s) [upload=444208]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:43:13.725675932 (506h14m13.596758977s) [upload=444208]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:43:14.020265812 (506h14m13.891348857s) [upload=444208]: Done uploading segments... Dec 26 17:43:14.020287173 (506h14m13.891370219s) [upload=444208]: Upload finished. err: <nil> Duration: 5.980493884s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 29.486971962s trace: https://jaeger.dev.storj.io/trace/665ea65f6e9ccc33 FAIL (2025-12-26 17:46:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== e=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789316979 (506h12m54.660400024s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789326507 (506h12m54.660409552s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789611339 (506h12m54.660694384s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789619275 (506h12m54.66070231s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789914006 (506h12m54.660997041s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789949443 (506h12m54.661032478s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790181566 (506h12m54.661264601s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790295632 (506h12m54.661378668s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790510872 (506h12m54.661593917s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790692980 (506h12m54.661776015s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790787379 (506h12m54.661870414s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790992970 (506h12m54.662076005s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791120053 (506h12m54.662203098s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791275218 (506h12m54.662358253s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791443709 (506h12m54.662526744s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:56.112313145 (506h12m55.98339619s) [upload=444191]: Next returned nil segment Dec 26 17:41:56.112330638 (506h12m55.983413683s) [upload=444191]: Waiting for error group managing segments... Dec 26 17:41:56.485322720 (506h12m56.356405765s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.485637079 (506h12m56.356720124s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.666807539 (506h12m56.537890584s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.678753987 (506h12m56.549837032s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.687788039 (506h12m56.558871074s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.694318023 (506h12m56.565401058s) [noise=true , address=145.79.227.7:4326 , piece_id=DMQ5RPI46X2U7TJTS42ZHQEF32BDJRKJJO5HWOYAPNS6VWVG5B5Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.716581187 (506h12m56.587664232s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.972183007 (506h12m56.843266042s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.011917999 (506h12m56.883001034s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.054168622 (506h12m56.925251667s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.114312777 (506h12m56.985395812s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.499846250 (506h12m57.370929285s) [noise=true , address=145.79.227.34:4919 , piece_id=YRWHW3BJVEGVJRSA6RJB3T7VJQSDTGKEZQKRICFIV2G3T7AN4MIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.612649066 (506h12m57.483732101s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649413733 (506h12m57.520496768s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649440282 (506h12m57.520523328s) [seg_pos={0 0}, upload=444191]: Segment reached optimal threshold of 14 pieces. Dec 26 17:41:57.649674729 (506h12m57.520757774s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G7ty71tPwe7cZSBNK9u74DFUCGomnxDsNvAif4C3SbP7HWwfo): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649684087 (506h12m57.520767142s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649747708 (506h12m57.520830763s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v6zXZuMmyL2MU8qb3q6gg1iwTYvndjzMjVpv8btEpeX9Lnqmu): protocol: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection Dec 26 17:41:57.649785219 (506h12m57.520868274s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649899828 (506h12m57.520982903s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649908775 (506h12m57.52099182s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650074641 (506h12m57.521157696s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.650083287 (506h12m57.521166342s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650132821 (506h12m57.521215866s) [seg_pos={0 0}, upload=444191]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 17:41:57.650220138 (506h12m57.521303183s) [upload=444191]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:41:57.650227171 (506h12m57.521310206s) [upload=444191]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:41:57.958890870 (506h12m57.829973915s) [upload=444191]: Done uploading segments... Dec 26 17:41:57.958911620 (506h12m57.829994665s) [upload=444191]: Upload finished. err: <nil> Duration: 29.486991309s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [X..X.X......XX..X....X.X.XX...XX...XX....XX..XX......X.XXX..] duration: 6.038367596s trace: https://jaeger.dev.storj.io/trace/3b761e3de974ea48 FAIL (2025-12-26 17:45:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA, Piece ID: LCYNTQGR2ZIUX6HBLYVSCUTCLAU6PFGHSA2ICB6DWE2ZFOI22NSA) context deadline exceeded; piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: DZPD74AMFXUMJYQEFLME4ZX7NRSLDIYE3ZBR5CIKSJE3XPXANUPA) context deadline exceeded ======LOGS====== Dec 26 17:43:10.974317425 (506h14m10.84540046s) [upload=444208]: Starting upload Dec 26 17:43:10.974446491 (506h14m10.845529526s) [upload=444208]: Done starting upload Dec 26 17:43:10.974581899 (506h14m10.845664944s) [upload=444208]: Uploading segments... Dec 26 17:43:10.974665328 (506h14m10.845748383s) [upload=444208]: Got next segment. Inline: false Dec 26 17:43:10.974721515 (506h14m10.84580456s) [upload=444208]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 17:43:10.974752394 (506h14m10.845835439s) [upload=444208]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 17:43:11.225224176 (506h14m11.096307221s) [seg_pos={0 0}, upload=444208]: Begin upload segment... Dec 26 17:43:11.225287356 (506h14m11.096370411s) [seg_pos={0 0}, upload=444208]: Done begin upload segment. Dec 26 17:43:11.225329857 (506h14m11.096412902s) [noise=true , address=145.79.227.46:4601 , piece_id=IM6NZ7Z4XQSEP4XG4WBWT4Y6P5DUX62KAMLQJANCBKSAU6ZVYRLA, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.225917767 (506h14m11.097000802s) [noise=true , address=145.79.227.8:4330 , piece_id=HEEUM6HSWGUYQFB2NP5T4DZST5GUJEDFNTLWDS6GEZC75P54UJQQ, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.231054878 (506h14m11.102137913s) [noise=true , address=145.79.227.49:4338 , piece_id=LNEIJJAJA7Q6J6ZGVQRPQFZELDCNKU2VKXX64K4ET6XVJ52LGVDQ, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.232622754 (506h14m11.103705799s) [noise=true , address=45.91.116.12:4228 , piece_id=LCYNTQGR2ZIUX6HBLYVSCUTCLAU6PFGHSA2ICB6DWE2ZFOI22NSA, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.234863150 (506h14m11.105946186s) [noise=true , address=145.79.227.45:4230 , piece_id=RKFJEJXGFXRGQZ2B3B3U3XZATBUAPFSV3IVYZQVIS63Q34RNCOSQ, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.236499346 (506h14m11.107582411s) [noise=true , address=45.91.116.6:4974 , piece_id=VKPME5624PBSFP3HLHKANNY7SBEVFL4Z636HEN2KXUQ55XPORJ2A, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.238064495 (506h14m11.10914753s) [noise=true , address=145.79.227.35:4481 , piece_id=AWRLZ2QPXDUTJWC2J6TCOM7QROIZJENTYBVWB6T2KPR2B6YYXS4A, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.243205044 (506h14m11.114288089s) [noise=true , address=145.79.227.37:4567 , piece_id=SLFOZCOSWTYSU5W3VXSJVHO6ZJSGWBUUMZKTWEV5YMDQONFHN53A, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.245389073 (506h14m11.116472108s) [noise=true , address=145.79.227.15:4874 , piece_id=TIWMO4OASWXEVFEG4COHHNDNGKHZUWZGTP6D6NNCR5PEZRSBYVIA, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.248134351 (506h14m11.119217426s) [noise=true , address=45.91.116.14:4191 , piece_id=DZPD74AMFXUMJYQEFLME4ZX7NRSLDIYE3ZBR5CIKSJE3XPXANUPA, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.251152537 (506h14m11.122235572s) [noise=true , address=145.79.227.3:4151 , piece_id=4QA7ARRZII6ZRW3GSYBHQVFBGDY4F676MVVJMQHMKWBZ2WS7P5VA, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.253552006 (506h14m11.124635051s) [noise=true , address=77.90.31.21:4880 , piece_id=KYMCS7O4EGMIVDSALBAUBNP7Q3TO5ZX4PLQ4KWBMKSWRUC7TM6EQ, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.255859751 (506h14m11.126942797s) [noise=true , address=145.79.227.22:4511 , piece_id=G7HZFZ7DF6NEWMGRUQQH4UOPUZIYVOVDH22TZNCDWTSQWB3ZEFGQ, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.256186012 (506h14m11.127269057s) [noise=true , address=145.79.227.48:4302 , piece_id=EGYOF4SZ3PKE3H6QYM3QHHRMITTGH55TPNY5D4NS5FSSTKLTML3Q, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.257647495 (506h14m11.12873056s) [noise=true , address=145.79.227.50:4373 , piece_id=DXRYJICGTQJBT7NDE6UUYPDPJZ2HD6MVGRTVZX5GAXXKJFSJ47FA, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.260156352 (506h14m11.131239398s) [noise=true , address=45.91.116.8:4048 , piece_id=JPMATYG4OTAACBWLQKX64FUDV7ONG4P4YZLF4SI4EVGBXOYSICLA, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:12.947257377 (506h14m12.818340412s) [upload=444208]: Next returned nil segment Dec 26 17:43:12.947333081 (506h14m12.818416126s) [upload=444208]: Waiting for error group managing segments... Dec 26 17:43:13.273287306 (506h14m13.144370351s) [noise=true , address=145.79.227.46:4601 , piece_id=IM6NZ7Z4XQSEP4XG4WBWT4Y6P5DUX62KAMLQJANCBKSAU6ZVYRLA, seg_pos={0 0}, upload=444208]: Done uploading piece. err: <nil> Dec 26 17:43:13.273494530 (506h14m13.144577575s) [noise=true , address=145.79.227.3:4151 , piece_id=4QA7ARRZII6ZRW3GSYBHQVFBGDY4F676MVVJMQHMKWBZ2WS7P5VA, seg_pos={0 0}, upload=444208]: Done uploading piece. err: <nil> Dec 26 17:43:13.275357928 (506h14m13.146440963s) [noise=true , address=145.79.227.15:4874 , piece_id=TIWMO4OASWXEVFEG4COHHNDNGKHZUWZGTP6D6NNCR5PEZRSBYVIA, seg_pos={0 0}, upload=444208]: Done uploading piece. err: <nil> Dec 26 17:43:13.276417696 (506h14m13.147500742s) [noise=true , address=145.79.227.45:4230 , piece_id=RKFJEJXGFXRGQZ2B3B3U3XZATBUAPFSV3IVYZQVIS63Q34RNCOSQ, seg_pos={0 0}, upload=444208]: Done uploading piece. err: <nil> Dec 26 17:43:13.277361283 (506h14m13.148444318s) [noise=true , address=145.79.227.50:4373 , piece_id=DXRYJICGTQJBT7NDE6UUYPDPJZ2HD6MVGRTVZX5GAXXKJFSJ47FA, seg_pos={0 0}, upload=444208]: Done uploading piece. err: <nil> Dec 26 17:43:13.277397332 (506h14m13.148480367s) [noise=true , address=145.79.227.37:4567 , piece_id=SLFOZCOSWTYSU5W3VXSJVHO6ZJSGWBUUMZKTWEV5YMDQONFHN53A, seg_pos={0 0}, upload=444208]: Done uploading piece. err: <nil> Dec 26 17:43:13.277910097 (506h14m13.148993142s) [noise=true , address=145.79.227.22:4511 , piece_id=G7HZFZ7DF6NEWMGRUQQH4UOPUZIYVOVDH22TZNCDWTSQWB3ZEFGQ, seg_pos={0 0}, upload=444208]: Done uploading piece. err: <nil> Dec 26 17:43:13.289773407 (506h14m13.160856452s) [noise=true , address=145.79.227.49:4338 , piece_id=LNEIJJAJA7Q6J6ZGVQRPQFZELDCNKU2VKXX64K4ET6XVJ52LGVDQ, seg_pos={0 0}, upload=444208]: Done uploading piece. err: <nil> Dec 26 17:43:13.290752031 (506h14m13.161835086s) [noise=true , address=145.79.227.48:4302 , piece_id=EGYOF4SZ3PKE3H6QYM3QHHRMITTGH55TPNY5D4NS5FSSTKLTML3Q, seg_pos={0 0}, upload=444208]: Done uploading piece. err: <nil> Dec 26 17:43:13.294786072 (506h14m13.165869117s) [noise=true , address=145.79.227.8:4330 , piece_id=HEEUM6HSWGUYQFB2NP5T4DZST5GUJEDFNTLWDS6GEZC75P54UJQQ, seg_pos={0 0}, upload=444208]: Done uploading piece. err: <nil> Dec 26 17:43:13.489465767 (506h14m13.360548802s) [noise=true , address=45.91.116.12:4228 , piece_id=LCYNTQGR2ZIUX6HBLYVSCUTCLAU6PFGHSA2ICB6DWE2ZFOI22NSA, seg_pos={0 0}, upload=444208]: Done uploading piece. err: <nil> Dec 26 17:43:13.566172019 (506h14m13.437255064s) [noise=true , address=45.91.116.14:4191 , piece_id=DZPD74AMFXUMJYQEFLME4ZX7NRSLDIYE3ZBR5CIKSJE3XPXANUPA, seg_pos={0 0}, upload=444208]: Done uploading piece. err: <nil> Dec 26 17:43:13.725036525 (506h14m13.59611956s) [noise=true , address=45.91.116.6:4974 , piece_id=VKPME5624PBSFP3HLHKANNY7SBEVFL4Z636HEN2KXUQ55XPORJ2A, seg_pos={0 0}, upload=444208]: Done uploading piece. err: <nil> Dec 26 17:43:13.725057735 (506h14m13.59614077s) [seg_pos={0 0}, upload=444208]: Segment reached optimal threshold of 13 pieces. Dec 26 17:43:13.725271673 (506h14m13.596354708s) [noise=true , address=145.79.227.35:4481 , piece_id=AWRLZ2QPXDUTJWC2J6TCOM7QROIZJENTYBVWB6T2KPR2B6YYXS4A, seg_pos={0 0}, upload=444208]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 26 17:43:13.725279558 (506h14m13.596362593s) [noise=true , address=145.79.227.35:4481 , piece_id=AWRLZ2QPXDUTJWC2J6TCOM7QROIZJENTYBVWB6T2KPR2B6YYXS4A, seg_pos={0 0}, upload=444208]: Long tail cancellation due to optimal threshold Dec 26 17:43:13.725338128 (506h14m13.596421174s) [noise=true , address=77.90.31.21:4880 , piece_id=KYMCS7O4EGMIVDSALBAUBNP7Q3TO5ZX4PLQ4KWBMKSWRUC7TM6EQ, seg_pos={0 0}, upload=444208]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: write tcp 188.166.225.170:37372->77.90.31.21:4880: use of closed network connection; write tcp 188.166.225.170:37372->77.90.31.21:4880: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37372->77.90.31.21:4880: use of closed network connection Dec 26 17:43:13.725346515 (506h14m13.59642955s) [noise=true , address=77.90.31.21:4880 , piece_id=KYMCS7O4EGMIVDSALBAUBNP7Q3TO5ZX4PLQ4KWBMKSWRUC7TM6EQ, seg_pos={0 0}, upload=444208]: Long tail cancellation due to optimal threshold Dec 26 17:43:13.725564680 (506h14m13.596647715s) [noise=true , address=45.91.116.8:4048 , piece_id=JPMATYG4OTAACBWLQKX64FUDV7ONG4P4YZLF4SI4EVGBXOYSICLA, seg_pos={0 0}, upload=444208]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: context canceled; context canceled Dec 26 17:43:13.725572675 (506h14m13.59665572s) [noise=true , address=45.91.116.8:4048 , piece_id=JPMATYG4OTAACBWLQKX64FUDV7ONG4P4YZLF4SI4EVGBXOYSICLA, seg_pos={0 0}, upload=444208]: Long tail cancellation due to optimal threshold Dec 26 17:43:13.725590309 (506h14m13.596673354s) [seg_pos={0 0}, upload=444208]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 17:43:13.725667807 (506h14m13.596750842s) [upload=444208]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:43:13.725675932 (506h14m13.596758977s) [upload=444208]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:43:14.020265812 (506h14m13.891348857s) [upload=444208]: Done uploading segments... Dec 26 17:43:14.020287173 (506h14m13.891370219s) [upload=444208]: Upload finished. err: <nil> Duration: 6.03840155s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 39.346101655s trace: https://jaeger.dev.storj.io/trace/665ea65f6e9ccc33 FAIL (2025-12-26 17:45:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== e=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789316979 (506h12m54.660400024s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789326507 (506h12m54.660409552s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789611339 (506h12m54.660694384s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789619275 (506h12m54.66070231s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789914006 (506h12m54.660997041s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789949443 (506h12m54.661032478s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790181566 (506h12m54.661264601s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790295632 (506h12m54.661378668s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790510872 (506h12m54.661593917s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790692980 (506h12m54.661776015s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790787379 (506h12m54.661870414s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790992970 (506h12m54.662076005s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791120053 (506h12m54.662203098s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791275218 (506h12m54.662358253s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791443709 (506h12m54.662526744s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:56.112313145 (506h12m55.98339619s) [upload=444191]: Next returned nil segment Dec 26 17:41:56.112330638 (506h12m55.983413683s) [upload=444191]: Waiting for error group managing segments... Dec 26 17:41:56.485322720 (506h12m56.356405765s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.485637079 (506h12m56.356720124s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.666807539 (506h12m56.537890584s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.678753987 (506h12m56.549837032s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.687788039 (506h12m56.558871074s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.694318023 (506h12m56.565401058s) [noise=true , address=145.79.227.7:4326 , piece_id=DMQ5RPI46X2U7TJTS42ZHQEF32BDJRKJJO5HWOYAPNS6VWVG5B5Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.716581187 (506h12m56.587664232s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.972183007 (506h12m56.843266042s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.011917999 (506h12m56.883001034s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.054168622 (506h12m56.925251667s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.114312777 (506h12m56.985395812s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.499846250 (506h12m57.370929285s) [noise=true , address=145.79.227.34:4919 , piece_id=YRWHW3BJVEGVJRSA6RJB3T7VJQSDTGKEZQKRICFIV2G3T7AN4MIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.612649066 (506h12m57.483732101s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649413733 (506h12m57.520496768s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649440282 (506h12m57.520523328s) [seg_pos={0 0}, upload=444191]: Segment reached optimal threshold of 14 pieces. Dec 26 17:41:57.649674729 (506h12m57.520757774s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G7ty71tPwe7cZSBNK9u74DFUCGomnxDsNvAif4C3SbP7HWwfo): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649684087 (506h12m57.520767142s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649747708 (506h12m57.520830763s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v6zXZuMmyL2MU8qb3q6gg1iwTYvndjzMjVpv8btEpeX9Lnqmu): protocol: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection Dec 26 17:41:57.649785219 (506h12m57.520868274s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649899828 (506h12m57.520982903s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649908775 (506h12m57.52099182s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650074641 (506h12m57.521157696s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.650083287 (506h12m57.521166342s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650132821 (506h12m57.521215866s) [seg_pos={0 0}, upload=444191]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 17:41:57.650220138 (506h12m57.521303183s) [upload=444191]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:41:57.650227171 (506h12m57.521310206s) [upload=444191]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:41:57.958890870 (506h12m57.829973915s) [upload=444191]: Done uploading segments... Dec 26 17:41:57.958911620 (506h12m57.829994665s) [upload=444191]: Upload finished. err: <nil> Duration: 39.34613097s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [XX..X.X......XX..X....X.X.XX...XX...XX....XX..XX......X.XXX.] duration: 20.537201736s trace: https://jaeger.dev.storj.io/trace/3b761e3de974ea48 FAIL (2025-12-26 17:44:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA, Piece ID: LCYNTQGR2ZIUX6HBLYVSCUTCLAU6PFGHSA2ICB6DWE2ZFOI22NSA) context deadline exceeded; piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: DZPD74AMFXUMJYQEFLME4ZX7NRSLDIYE3ZBR5CIKSJE3XPXANUPA) context deadline exceeded ======LOGS====== Dec 26 17:43:10.974317425 (506h14m10.84540046s) [upload=444208]: Starting upload Dec 26 17:43:10.974446491 (506h14m10.845529526s) [upload=444208]: Done starting upload Dec 26 17:43:10.974581899 (506h14m10.845664944s) [upload=444208]: Uploading segments... Dec 26 17:43:10.974665328 (506h14m10.845748383s) [upload=444208]: Got next segment. Inline: false Dec 26 17:43:10.974721515 (506h14m10.84580456s) [upload=444208]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 17:43:10.974752394 (506h14m10.845835439s) [upload=444208]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 17:43:11.225224176 (506h14m11.096307221s) [seg_pos={0 0}, upload=444208]: Begin upload segment... Dec 26 17:43:11.225287356 (506h14m11.096370411s) [seg_pos={0 0}, upload=444208]: Done begin upload segment. Dec 26 17:43:11.225329857 (506h14m11.096412902s) [noise=true , address=145.79.227.46:4601 , piece_id=IM6NZ7Z4XQSEP4XG4WBWT4Y6P5DUX62KAMLQJANCBKSAU6ZVYRLA, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.225917767 (506h14m11.097000802s) [noise=true , address=145.79.227.8:4330 , piece_id=HEEUM6HSWGUYQFB2NP5T4DZST5GUJEDFNTLWDS6GEZC75P54UJQQ, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.231054878 (506h14m11.102137913s) [noise=true , address=145.79.227.49:4338 , piece_id=LNEIJJAJA7Q6J6ZGVQRPQFZELDCNKU2VKXX64K4ET6XVJ52LGVDQ, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.232622754 (506h14m11.103705799s) [noise=true , address=45.91.116.12:4228 , piece_id=LCYNTQGR2ZIUX6HBLYVSCUTCLAU6PFGHSA2ICB6DWE2ZFOI22NSA, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.234863150 (506h14m11.105946186s) [noise=true , address=145.79.227.45:4230 , piece_id=RKFJEJXGFXRGQZ2B3B3U3XZATBUAPFSV3IVYZQVIS63Q34RNCOSQ, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.236499346 (506h14m11.107582411s) [noise=true , address=45.91.116.6:4974 , piece_id=VKPME5624PBSFP3HLHKANNY7SBEVFL4Z636HEN2KXUQ55XPORJ2A, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.238064495 (506h14m11.10914753s) [noise=true , address=145.79.227.35:4481 , piece_id=AWRLZ2QPXDUTJWC2J6TCOM7QROIZJENTYBVWB6T2KPR2B6YYXS4A, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.243205044 (506h14m11.114288089s) [noise=true , address=145.79.227.37:4567 , piece_id=SLFOZCOSWTYSU5W3VXSJVHO6ZJSGWBUUMZKTWEV5YMDQONFHN53A, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.245389073 (506h14m11.116472108s) [noise=true , address=145.79.227.15:4874 , piece_id=TIWMO4OASWXEVFEG4COHHNDNGKHZUWZGTP6D6NNCR5PEZRSBYVIA, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.248134351 (506h14m11.119217426s) [noise=true , address=45.91.116.14:4191 , piece_id=DZPD74AMFXUMJYQEFLME4ZX7NRSLDIYE3ZBR5CIKSJE3XPXANUPA, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.251152537 (506h14m11.122235572s) [noise=true , address=145.79.227.3:4151 , piece_id=4QA7ARRZII6ZRW3GSYBHQVFBGDY4F676MVVJMQHMKWBZ2WS7P5VA, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.253552006 (506h14m11.124635051s) [noise=true , address=77.90.31.21:4880 , piece_id=KYMCS7O4EGMIVDSALBAUBNP7Q3TO5ZX4PLQ4KWBMKSWRUC7TM6EQ, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.255859751 (506h14m11.126942797s) [noise=true , address=145.79.227.22:4511 , piece_id=G7HZFZ7DF6NEWMGRUQQH4UOPUZIYVOVDH22TZNCDWTSQWB3ZEFGQ, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.256186012 (506h14m11.127269057s) [noise=true , address=145.79.227.48:4302 , piece_id=EGYOF4SZ3PKE3H6QYM3QHHRMITTGH55TPNY5D4NS5FSSTKLTML3Q, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.257647495 (506h14m11.12873056s) [noise=true , address=145.79.227.50:4373 , piece_id=DXRYJICGTQJBT7NDE6UUYPDPJZ2HD6MVGRTVZX5GAXXKJFSJ47FA, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:11.260156352 (506h14m11.131239398s) [noise=true , address=45.91.116.8:4048 , piece_id=JPMATYG4OTAACBWLQKX64FUDV7ONG4P4YZLF4SI4EVGBXOYSICLA, seg_pos={0 0}, upload=444208]: Uploading piece... Dec 26 17:43:12.947257377 (506h14m12.818340412s) [upload=444208]: Next returned nil segment Dec 26 17:43:12.947333081 (506h14m12.818416126s) [upload=444208]: Waiting for error group managing segments... Dec 26 17:43:13.273287306 (506h14m13.144370351s) [noise=true , address=145.79.227.46:4601 , piece_id=IM6NZ7Z4XQSEP4XG4WBWT4Y6P5DUX62KAMLQJANCBKSAU6ZVYRLA, seg_pos={0 0}, upload=444208]: Done uploading piece. err: <nil> Dec 26 17:43:13.273494530 (506h14m13.144577575s) [noise=true , address=145.79.227.3:4151 , piece_id=4QA7ARRZII6ZRW3GSYBHQVFBGDY4F676MVVJMQHMKWBZ2WS7P5VA, seg_pos={0 0}, upload=444208]: Done uploading piece. err: <nil> Dec 26 17:43:13.275357928 (506h14m13.146440963s) [noise=true , address=145.79.227.15:4874 , piece_id=TIWMO4OASWXEVFEG4COHHNDNGKHZUWZGTP6D6NNCR5PEZRSBYVIA, seg_pos={0 0}, upload=444208]: Done uploading piece. err: <nil> Dec 26 17:43:13.276417696 (506h14m13.147500742s) [noise=true , address=145.79.227.45:4230 , piece_id=RKFJEJXGFXRGQZ2B3B3U3XZATBUAPFSV3IVYZQVIS63Q34RNCOSQ, seg_pos={0 0}, upload=444208]: Done uploading piece. err: <nil> Dec 26 17:43:13.277361283 (506h14m13.148444318s) [noise=true , address=145.79.227.50:4373 , piece_id=DXRYJICGTQJBT7NDE6UUYPDPJZ2HD6MVGRTVZX5GAXXKJFSJ47FA, seg_pos={0 0}, upload=444208]: Done uploading piece. err: <nil> Dec 26 17:43:13.277397332 (506h14m13.148480367s) [noise=true , address=145.79.227.37:4567 , piece_id=SLFOZCOSWTYSU5W3VXSJVHO6ZJSGWBUUMZKTWEV5YMDQONFHN53A, seg_pos={0 0}, upload=444208]: Done uploading piece. err: <nil> Dec 26 17:43:13.277910097 (506h14m13.148993142s) [noise=true , address=145.79.227.22:4511 , piece_id=G7HZFZ7DF6NEWMGRUQQH4UOPUZIYVOVDH22TZNCDWTSQWB3ZEFGQ, seg_pos={0 0}, upload=444208]: Done uploading piece. err: <nil> Dec 26 17:43:13.289773407 (506h14m13.160856452s) [noise=true , address=145.79.227.49:4338 , piece_id=LNEIJJAJA7Q6J6ZGVQRPQFZELDCNKU2VKXX64K4ET6XVJ52LGVDQ, seg_pos={0 0}, upload=444208]: Done uploading piece. err: <nil> Dec 26 17:43:13.290752031 (506h14m13.161835086s) [noise=true , address=145.79.227.48:4302 , piece_id=EGYOF4SZ3PKE3H6QYM3QHHRMITTGH55TPNY5D4NS5FSSTKLTML3Q, seg_pos={0 0}, upload=444208]: Done uploading piece. err: <nil> Dec 26 17:43:13.294786072 (506h14m13.165869117s) [noise=true , address=145.79.227.8:4330 , piece_id=HEEUM6HSWGUYQFB2NP5T4DZST5GUJEDFNTLWDS6GEZC75P54UJQQ, seg_pos={0 0}, upload=444208]: Done uploading piece. err: <nil> Dec 26 17:43:13.489465767 (506h14m13.360548802s) [noise=true , address=45.91.116.12:4228 , piece_id=LCYNTQGR2ZIUX6HBLYVSCUTCLAU6PFGHSA2ICB6DWE2ZFOI22NSA, seg_pos={0 0}, upload=444208]: Done uploading piece. err: <nil> Dec 26 17:43:13.566172019 (506h14m13.437255064s) [noise=true , address=45.91.116.14:4191 , piece_id=DZPD74AMFXUMJYQEFLME4ZX7NRSLDIYE3ZBR5CIKSJE3XPXANUPA, seg_pos={0 0}, upload=444208]: Done uploading piece. err: <nil> Dec 26 17:43:13.725036525 (506h14m13.59611956s) [noise=true , address=45.91.116.6:4974 , piece_id=VKPME5624PBSFP3HLHKANNY7SBEVFL4Z636HEN2KXUQ55XPORJ2A, seg_pos={0 0}, upload=444208]: Done uploading piece. err: <nil> Dec 26 17:43:13.725057735 (506h14m13.59614077s) [seg_pos={0 0}, upload=444208]: Segment reached optimal threshold of 13 pieces. Dec 26 17:43:13.725271673 (506h14m13.596354708s) [noise=true , address=145.79.227.35:4481 , piece_id=AWRLZ2QPXDUTJWC2J6TCOM7QROIZJENTYBVWB6T2KPR2B6YYXS4A, seg_pos={0 0}, upload=444208]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 26 17:43:13.725279558 (506h14m13.596362593s) [noise=true , address=145.79.227.35:4481 , piece_id=AWRLZ2QPXDUTJWC2J6TCOM7QROIZJENTYBVWB6T2KPR2B6YYXS4A, seg_pos={0 0}, upload=444208]: Long tail cancellation due to optimal threshold Dec 26 17:43:13.725338128 (506h14m13.596421174s) [noise=true , address=77.90.31.21:4880 , piece_id=KYMCS7O4EGMIVDSALBAUBNP7Q3TO5ZX4PLQ4KWBMKSWRUC7TM6EQ, seg_pos={0 0}, upload=444208]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: write tcp 188.166.225.170:37372->77.90.31.21:4880: use of closed network connection; write tcp 188.166.225.170:37372->77.90.31.21:4880: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37372->77.90.31.21:4880: use of closed network connection Dec 26 17:43:13.725346515 (506h14m13.59642955s) [noise=true , address=77.90.31.21:4880 , piece_id=KYMCS7O4EGMIVDSALBAUBNP7Q3TO5ZX4PLQ4KWBMKSWRUC7TM6EQ, seg_pos={0 0}, upload=444208]: Long tail cancellation due to optimal threshold Dec 26 17:43:13.725564680 (506h14m13.596647715s) [noise=true , address=45.91.116.8:4048 , piece_id=JPMATYG4OTAACBWLQKX64FUDV7ONG4P4YZLF4SI4EVGBXOYSICLA, seg_pos={0 0}, upload=444208]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: context canceled; context canceled Dec 26 17:43:13.725572675 (506h14m13.59665572s) [noise=true , address=45.91.116.8:4048 , piece_id=JPMATYG4OTAACBWLQKX64FUDV7ONG4P4YZLF4SI4EVGBXOYSICLA, seg_pos={0 0}, upload=444208]: Long tail cancellation due to optimal threshold Dec 26 17:43:13.725590309 (506h14m13.596673354s) [seg_pos={0 0}, upload=444208]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 17:43:13.725667807 (506h14m13.596750842s) [upload=444208]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:43:13.725675932 (506h14m13.596758977s) [upload=444208]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:43:14.020265812 (506h14m13.891348857s) [upload=444208]: Done uploading segments... Dec 26 17:43:14.020287173 (506h14m13.891370219s) [upload=444208]: Upload finished. err: <nil> Duration: 20.53723065s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)