Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....X............................X.........................X] duration: 30.004676256s trace: https://jaeger.dev.storj.io/trace/50cd070c446f9532 FAIL (2025-12-29 22:21:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: FJ54ZRYEBVWLLMHEZTZWT4RGNDFWIAFUQLUKGGNE6R3NUK33DQ5A) context deadline exceeded; piecestore: (Node ID: 12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, Piece ID: BGPEL5PH6AY77T3TNM6DYCQQZL6I7YAUH6J6SV3GO4KB6FF3S7TA) context deadline exceeded ======LOGS====== Dec 29 22:20:38.003755724 (582h51m37.874838759s) [upload=510934]: Starting upload Dec 29 22:20:38.003809547 (582h51m37.874892592s) [upload=510934]: Done starting upload Dec 29 22:20:38.003861455 (582h51m37.87494449s) [upload=510934]: Uploading segments... Dec 29 22:20:38.003995491 (582h51m37.875078526s) [upload=510934]: Got next segment. Inline: false Dec 29 22:20:38.004039846 (582h51m37.875122881s) [upload=510934]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 22:20:38.004047089 (582h51m37.875130124s) [upload=510934]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 22:20:38.253256798 (582h51m38.124339843s) [seg_pos={0 0}, upload=510934]: Begin upload segment... Dec 29 22:20:38.253324216 (582h51m38.124407251s) [seg_pos={0 0}, upload=510934]: Done begin upload segment. Dec 29 22:20:38.253359373 (582h51m38.124442418s) [noise=true , address=145.79.227.31:4797 , piece_id=5GDJZFDCJZCTK23N7JERDCSKEJGZLS6LP6AU73CIVRDAJG6ZHE5Q, seg_pos={0 0}, upload=510934]: Uploading piece... Dec 29 22:20:38.253389000 (582h51m38.124472045s) [noise=true , address=145.79.227.5:4221 , piece_id=BQW7FU3WVPKOSFN4YI4SIFGQQREVX3ANEEFVZRA4UAHTOJ67YN4A, seg_pos={0 0}, upload=510934]: Uploading piece... Dec 29 22:20:38.254399654 (582h51m38.125482689s) [noise=true , address=45.91.116.9:4264 , piece_id=CXCM5HK2R63AS323NRU74LPUFUR6XM2N3FZEDET5WITNXA32PXMA, seg_pos={0 0}, upload=510934]: Uploading piece... Dec 29 22:20:38.255022320 (582h51m38.126105385s) [noise=true , address=145.79.227.15:4874 , piece_id=QETCOOBNUALMPNO6D4H4YSFFQYQWEJUGWIHLIOXK65C6K5L7LJEQ, seg_pos={0 0}, upload=510934]: Uploading piece... Dec 29 22:20:38.255623524 (582h51m38.126706569s) [noise=true , address=145.79.227.36:4517 , piece_id=QCLMEHUNXCGDMIM2I6N7GOV65O7XGYH6NSOTOU3G2OEHLZPGOJVA, seg_pos={0 0}, upload=510934]: Uploading piece... Dec 29 22:20:38.255639395 (582h51m38.12672243s) [noise=true , address=145.79.227.47:4265 , piece_id=ZBZOWDIJX2LAQBUI23ZN6EU3TG6HL3DXBTB3SWV23APVUU2XRJCQ, seg_pos={0 0}, upload=510934]: Uploading piece... Dec 29 22:20:38.256173752 (582h51m38.127256797s) [noise=true , address=145.79.227.46:4601 , piece_id=UUGD4OACIRIGHQJJTMAFVSBB3VPRLEBZKWOCWUFEZTVCYE5FTU7Q, seg_pos={0 0}, upload=510934]: Uploading piece... Dec 29 22:20:38.256278241 (582h51m38.127361276s) [noise=true , address=145.79.227.42:4121 , piece_id=7PMBOHIQKT2T54Z4UWFH2ELTXKU4GY5NQ64CYJJB7DU5LPDIGF3Q, seg_pos={0 0}, upload=510934]: Uploading piece... Dec 29 22:20:38.256604232 (582h51m38.127687267s) [noise=true , address=145.79.227.12:4767 , piece_id=KWBXJ3XU5TSSJB4U6YZ75AA2VY3KQESHOEODRQCWTRVYDV2FQPHQ, seg_pos={0 0}, upload=510934]: Uploading piece... Dec 29 22:20:38.256837957 (582h51m38.127920992s) [noise=true , address=77.90.31.15:4700 , piece_id=FJ54ZRYEBVWLLMHEZTZWT4RGNDFWIAFUQLUKGGNE6R3NUK33DQ5A, seg_pos={0 0}, upload=510934]: Uploading piece... Dec 29 22:20:38.257074728 (582h51m38.128157773s) [noise=true , address=45.91.116.13:4156 , piece_id=4L2LAVITURELAOGSGTZJVFJ3A4Y4OORLW6MGBPDOA3LFBUMAGLTQ, seg_pos={0 0}, upload=510934]: Uploading piece... Dec 29 22:20:38.257379468 (582h51m38.128462513s) [noise=true , address=145.79.227.11:4729 , piece_id=U2KL2THUN6ZP2GOQZTG7AZBU7PGDNQRQZKFY4QI6ZNFXHVO2P7UQ, seg_pos={0 0}, upload=510934]: Uploading piece... Dec 29 22:20:38.257566414 (582h51m38.128649459s) [noise=true , address=77.90.31.21:4880 , piece_id=4WSJJMCPRUTTYY2XEGKWSDPMOQDQGGW7NYZNZG3TG3QTPKIQW4TQ, seg_pos={0 0}, upload=510934]: Uploading piece... Dec 29 22:20:38.258046438 (582h51m38.129129473s) [noise=true , address=145.79.227.48:4301 , piece_id=4JB2GAAAXXPFNWJAXOPLGUH3GB6VAI56OSFNIQTW4LLW5PJRDOYA, seg_pos={0 0}, upload=510934]: Uploading piece... Dec 29 22:20:38.258587398 (582h51m38.129670433s) [noise=true , address=77.90.31.14:4661 , piece_id=BGPEL5PH6AY77T3TNM6DYCQQZL6I7YAUH6J6SV3GO4KB6FF3S7TA, seg_pos={0 0}, upload=510934]: Uploading piece... Dec 29 22:20:38.259062062 (582h51m38.130145097s) [noise=true , address=77.90.31.10:4517 , piece_id=F3LR3WBWE5BYEUGOKSELTAOKT6R6IZPUHCH5XCHYNT65WPJXPAEQ, seg_pos={0 0}, upload=510934]: Uploading piece... Dec 29 22:20:39.938508372 (582h51m39.809591407s) [upload=510934]: Next returned nil segment Dec 29 22:20:39.938519794 (582h51m39.809602829s) [upload=510934]: Waiting for error group managing segments... Dec 29 22:20:40.236728734 (582h51m40.107811779s) [noise=true , address=145.79.227.42:4121 , piece_id=7PMBOHIQKT2T54Z4UWFH2ELTXKU4GY5NQ64CYJJB7DU5LPDIGF3Q, seg_pos={0 0}, upload=510934]: Done uploading piece. err: <nil> Dec 29 22:20:40.238046043 (582h51m40.109129078s) [noise=true , address=145.79.227.11:4729 , piece_id=U2KL2THUN6ZP2GOQZTG7AZBU7PGDNQRQZKFY4QI6ZNFXHVO2P7UQ, seg_pos={0 0}, upload=510934]: Done uploading piece. err: <nil> Dec 29 22:20:40.246341979 (582h51m40.117425024s) [noise=true , address=145.79.227.31:4797 , piece_id=5GDJZFDCJZCTK23N7JERDCSKEJGZLS6LP6AU73CIVRDAJG6ZHE5Q, seg_pos={0 0}, upload=510934]: Done uploading piece. err: <nil> Dec 29 22:20:40.247622127 (582h51m40.118705162s) [noise=true , address=145.79.227.48:4301 , piece_id=4JB2GAAAXXPFNWJAXOPLGUH3GB6VAI56OSFNIQTW4LLW5PJRDOYA, seg_pos={0 0}, upload=510934]: Done uploading piece. err: <nil> Dec 29 22:20:40.251962321 (582h51m40.123045356s) [noise=true , address=145.79.227.46:4601 , piece_id=UUGD4OACIRIGHQJJTMAFVSBB3VPRLEBZKWOCWUFEZTVCYE5FTU7Q, seg_pos={0 0}, upload=510934]: Done uploading piece. err: <nil> Dec 29 22:20:40.255370801 (582h51m40.126453836s) [noise=true , address=145.79.227.47:4265 , piece_id=ZBZOWDIJX2LAQBUI23ZN6EU3TG6HL3DXBTB3SWV23APVUU2XRJCQ, seg_pos={0 0}, upload=510934]: Done uploading piece. err: <nil> Dec 29 22:20:40.256021209 (582h51m40.127104244s) [noise=true , address=145.79.227.15:4874 , piece_id=QETCOOBNUALMPNO6D4H4YSFFQYQWEJUGWIHLIOXK65C6K5L7LJEQ, seg_pos={0 0}, upload=510934]: Done uploading piece. err: <nil> Dec 29 22:20:40.256716132 (582h51m40.127799167s) [noise=true , address=145.79.227.36:4517 , piece_id=QCLMEHUNXCGDMIM2I6N7GOV65O7XGYH6NSOTOU3G2OEHLZPGOJVA, seg_pos={0 0}, upload=510934]: Done uploading piece. err: <nil> Dec 29 22:20:40.275478619 (582h51m40.146561654s) [noise=true , address=145.79.227.5:4221 , piece_id=BQW7FU3WVPKOSFN4YI4SIFGQQREVX3ANEEFVZRA4UAHTOJ67YN4A, seg_pos={0 0}, upload=510934]: Done uploading piece. err: <nil> Dec 29 22:20:40.287513205 (582h51m40.15859624s) [noise=true , address=145.79.227.12:4767 , piece_id=KWBXJ3XU5TSSJB4U6YZ75AA2VY3KQESHOEODRQCWTRVYDV2FQPHQ, seg_pos={0 0}, upload=510934]: Done uploading piece. err: <nil> Dec 29 22:20:40.427649182 (582h51m40.298732217s) [noise=true , address=77.90.31.15:4700 , piece_id=FJ54ZRYEBVWLLMHEZTZWT4RGNDFWIAFUQLUKGGNE6R3NUK33DQ5A, seg_pos={0 0}, upload=510934]: Done uploading piece. err: <nil> Dec 29 22:20:40.510504820 (582h51m40.381587875s) [noise=true , address=77.90.31.14:4661 , piece_id=BGPEL5PH6AY77T3TNM6DYCQQZL6I7YAUH6J6SV3GO4KB6FF3S7TA, seg_pos={0 0}, upload=510934]: Done uploading piece. err: <nil> Dec 29 22:20:40.544136709 (582h51m40.415219744s) [noise=true , address=77.90.31.21:4880 , piece_id=4WSJJMCPRUTTYY2XEGKWSDPMOQDQGGW7NYZNZG3TG3QTPKIQW4TQ, seg_pos={0 0}, upload=510934]: Done uploading piece. err: <nil> Dec 29 22:20:40.544156237 (582h51m40.415239282s) [seg_pos={0 0}, upload=510934]: Segment reached optimal threshold of 13 pieces. Dec 29 22:20:40.544344815 (582h51m40.41542785s) [noise=true , address=77.90.31.10:4517 , piece_id=F3LR3WBWE5BYEUGOKSELTAOKT6R6IZPUHCH5XCHYNT65WPJXPAEQ, seg_pos={0 0}, upload=510934]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 29 22:20:40.544351499 (582h51m40.415434544s) [noise=true , address=77.90.31.10:4517 , piece_id=F3LR3WBWE5BYEUGOKSELTAOKT6R6IZPUHCH5XCHYNT65WPJXPAEQ, seg_pos={0 0}, upload=510934]: Long tail cancellation due to optimal threshold Dec 29 22:20:40.544505341 (582h51m40.415588387s) [noise=true , address=45.91.116.9:4264 , piece_id=CXCM5HK2R63AS323NRU74LPUFUR6XM2N3FZEDET5WITNXA32PXMA, seg_pos={0 0}, upload=510934]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 29 22:20:40.544546820 (582h51m40.415629865s) [noise=true , address=45.91.116.9:4264 , piece_id=CXCM5HK2R63AS323NRU74LPUFUR6XM2N3FZEDET5WITNXA32PXMA, seg_pos={0 0}, upload=510934]: Long tail cancellation due to optimal threshold Dec 29 22:20:40.544584402 (582h51m40.415667437s) [noise=true , address=45.91.116.13:4156 , piece_id=4L2LAVITURELAOGSGTZJVFJ3A4Y4OORLW6MGBPDOA3LFBUMAGLTQ, seg_pos={0 0}, upload=510934]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 29 22:20:40.544590794 (582h51m40.415673829s) [noise=true , address=45.91.116.13:4156 , piece_id=4L2LAVITURELAOGSGTZJVFJ3A4Y4OORLW6MGBPDOA3LFBUMAGLTQ, seg_pos={0 0}, upload=510934]: Long tail cancellation due to optimal threshold Dec 29 22:20:40.544641000 (582h51m40.415724035s) [seg_pos={0 0}, upload=510934]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 22:20:40.544714600 (582h51m40.415797635s) [upload=510934]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 22:20:40.544719519 (582h51m40.415802554s) [upload=510934]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 22:20:40.845710266 (582h51m40.716793301s) [upload=510934]: Done uploading segments... Dec 29 22:20:40.845727149 (582h51m40.716810184s) [upload=510934]: Upload finished. err: <nil> Duration: 30.004699171s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............X............................X.................] duration: 17.323212513s trace: https://jaeger.dev.storj.io/trace/7096862780f29049 FAIL (2025-12-29 22:21:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== oise=true , address=145.79.227.30:4766 , piece_id=YWDMUGNTYVXWOCMSN63VIC6HOTWHPY7VJ7QWT76P63UAJIBS2XLA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.060493636 (582h37m18.931576671s) [noise=true , address=5.231.73.5:4022 , piece_id=5WVAJH3Y6TLATXY4IYB6ZADKKJDXDS52B2ZN6FJAZ7UHFVRYRRAA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.060685291 (582h37m18.931768326s) [noise=true , address=77.90.31.11:4579 , piece_id=DEJPYHGYNAP52CZ45BKGNXJM5MLIA2OQZ4BRKN5PBADNRVE3VLEA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.060749954 (582h37m18.931832989s) [noise=true , address=145.79.227.29:4976 , piece_id=YRCRXSSQOXPOJGVX46L7WCB3TRDZ5QB755SEOTNHSCHN4ZDMCOUQ, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.061033203 (582h37m18.932116248s) [noise=true , address=193.149.164.143:4698 , piece_id=6X5KB4TXFV4ZOF5JTPNRW4Q3GWNCVCXCA4JSB3AZURBHXY2VHTRQ, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.061248855 (582h37m18.9323319s) [noise=true , address=145.79.227.49:4347 , piece_id=75B3DRRHEPDL6EK6ZUXTKASXUKWGAVUWBPSTNRYT4BJMAU5ALXRA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.061382729 (582h37m18.932465764s) [noise=true , address=5.231.73.2:4004 , piece_id=PTHHO3XEPGGCIRCUBIWHN4LVNBLMISJM5LESQGR2ERRESFK3S4RA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.061645560 (582h37m18.932728595s) [noise=true , address=5.231.73.3:4007 , piece_id=6AF3CEPLUMRO4CDW3SRWZSYHWEXP35AVSM5M3LH4KCRARSZ6NFNA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.061769094 (582h37m18.932852139s) [noise=true , address=145.79.227.7:4321 , piece_id=SPWPZ5MR5ZCQNYVYYQQEZC4OURAM5VAH3WN5UP6HMGCCDMCH6ZGA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.061970298 (582h37m18.933053333s) [noise=true , address=145.79.227.35:4507 , piece_id=ZZSVIHOAN2SWXI4NS3TRD4B64NDB4D43J63DAS74TCPEE6MPGRXA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.062237577 (582h37m18.933320612s) [noise=true , address=145.79.227.35:4489 , piece_id=BM7ZQESTHEUIRJWAPORTKDMT5TVLEXMOHULGFVEQKHT4AJV3HT5A, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.062304304 (582h37m18.933387349s) [noise=true , address=145.79.227.34:4931 , piece_id=XJHLE3B5HIF4RC4LV32SBCNMVTOTEXWBBCXBVPQFN6N5HFQ6ZS7A, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.062522640 (582h37m18.933605685s) [noise=true , address=145.79.227.48:4329 , piece_id=YWCVTYLCAGU57SE3ID3WW2JLCJGTJ7CPDRCZENAMO42BVPIAVLIQ, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.062904968 (582h37m18.933988003s) [noise=true , address=77.90.31.17:4801 , piece_id=K3QD7FPVRQHZL5R57EKKAC27RGDJQH6HDCND7FIVXSZBRH42WSIA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.063420419 (582h37m18.934503464s) [noise=true , address=145.79.227.17:4995 , piece_id=JJTVPVBAMCTDGHGWNPUVPYFVKD4SMTDAHF4VWVIW7JFUAMBIPRHQ, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.063943495 (582h37m18.93502654s) [noise=true , address=5.231.73.4:4018 , piece_id=X45H47HER27IM5FEQSEAMXSDX7UQG3AEOY65Q4WCCUOX7SVFZ5KA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:20.386418430 (582h37m20.257501465s) [upload=510734]: Next returned nil segment Dec 29 22:06:20.386428990 (582h37m20.257512025s) [upload=510734]: Waiting for error group managing segments... Dec 29 22:06:20.935543714 (582h37m20.806626749s) [noise=true , address=145.79.227.48:4329 , piece_id=YWCVTYLCAGU57SE3ID3WW2JLCJGTJ7CPDRCZENAMO42BVPIAVLIQ, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:20.978073277 (582h37m20.849156312s) [noise=true , address=145.79.227.39:4022 , piece_id=5IJCSVRVXCICRNRB3TTJAVNRF5FTUQMMY4KL7ZFF4SBKILFGHJWA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.079004013 (582h37m20.950087058s) [noise=true , address=5.231.73.4:4018 , piece_id=X45H47HER27IM5FEQSEAMXSDX7UQG3AEOY65Q4WCCUOX7SVFZ5KA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.101144552 (582h37m20.972227597s) [noise=true , address=77.90.31.11:4579 , piece_id=DEJPYHGYNAP52CZ45BKGNXJM5MLIA2OQZ4BRKN5PBADNRVE3VLEA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.138766369 (582h37m21.009849405s) [noise=true , address=77.90.31.17:4801 , piece_id=K3QD7FPVRQHZL5R57EKKAC27RGDJQH6HDCND7FIVXSZBRH42WSIA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.231047035 (582h37m21.10213007s) [noise=true , address=77.90.31.19:4857 , piece_id=DMJ4O3TWZYWKWVLJFSQBULHDFWZCNLVDBIGYJEOBAVVCU5UGG27Q, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.272062706 (582h37m21.143145741s) [noise=true , address=5.231.73.3:4007 , piece_id=6AF3CEPLUMRO4CDW3SRWZSYHWEXP35AVSM5M3LH4KCRARSZ6NFNA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.303520666 (582h37m21.174603711s) [noise=true , address=145.79.227.29:4976 , piece_id=YRCRXSSQOXPOJGVX46L7WCB3TRDZ5QB755SEOTNHSCHN4ZDMCOUQ, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.340640177 (582h37m21.211723212s) [noise=true , address=5.231.73.5:4022 , piece_id=5WVAJH3Y6TLATXY4IYB6ZADKKJDXDS52B2ZN6FJAZ7UHFVRYRRAA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.481118228 (582h37m21.352201263s) [noise=true , address=5.231.73.2:4004 , piece_id=PTHHO3XEPGGCIRCUBIWHN4LVNBLMISJM5LESQGR2ERRESFK3S4RA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.896953769 (582h37m21.768036804s) [noise=true , address=145.79.227.49:4347 , piece_id=75B3DRRHEPDL6EK6ZUXTKASXUKWGAVUWBPSTNRYT4BJMAU5ALXRA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.911314353 (582h37m21.782397399s) [noise=true , address=145.79.227.34:4931 , piece_id=XJHLE3B5HIF4RC4LV32SBCNMVTOTEXWBBCXBVPQFN6N5HFQ6ZS7A, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.915845371 (582h37m21.786928406s) [noise=true , address=145.79.227.30:4766 , piece_id=YWDMUGNTYVXWOCMSN63VIC6HOTWHPY7VJ7QWT76P63UAJIBS2XLA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.928785972 (582h37m21.799869017s) [noise=true , address=145.79.227.35:4489 , piece_id=BM7ZQESTHEUIRJWAPORTKDMT5TVLEXMOHULGFVEQKHT4AJV3HT5A, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.928810859 (582h37m21.799893904s) [seg_pos={0 0}, upload=510734]: Segment reached optimal threshold of 14 pieces. Dec 29 22:06:21.929016471 (582h37m21.800099516s) [noise=true , address=145.79.227.7:4321 , piece_id=SPWPZ5MR5ZCQNYVYYQQEZC4OURAM5VAH3WN5UP6HMGCCDMCH6ZGA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12edsHqD9iRBDgMU8vYhSM1EWG8Sn57GZ8TPiTytri3rp7W9Eae): protocol: expected piece hash; context canceled; EOF Dec 29 22:06:21.929028644 (582h37m21.800111709s) [noise=true , address=145.79.227.7:4321 , piece_id=SPWPZ5MR5ZCQNYVYYQQEZC4OURAM5VAH3WN5UP6HMGCCDMCH6ZGA, seg_pos={0 0}, upload=510734]: Long tail cancellation due to optimal threshold Dec 29 22:06:21.929093137 (582h37m21.800176192s) [noise=true , address=145.79.227.35:4507 , piece_id=ZZSVIHOAN2SWXI4NS3TRD4B64NDB4D43J63DAS74TCPEE6MPGRXA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14jurWWPpmxwiBS3XmZysEUwugsceYJqVXFj7BbYpx8UFcTR8R): protocol: expected piece hash; context canceled; EOF Dec 29 22:06:21.929106793 (582h37m21.800189838s) [noise=true , address=145.79.227.35:4507 , piece_id=ZZSVIHOAN2SWXI4NS3TRD4B64NDB4D43J63DAS74TCPEE6MPGRXA, seg_pos={0 0}, upload=510734]: Long tail cancellation due to optimal threshold Dec 29 22:06:21.929206612 (582h37m21.800289657s) [noise=true , address=145.79.227.17:4995 , piece_id=JJTVPVBAMCTDGHGWNPUVPYFVKD4SMTDAHF4VWVIW7JFUAMBIPRHQ, seg_pos={0 0}, upload=510734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tvoTD1LoSj3q2rLiDckfT56dXNKKwdVu1ZAsBZExNFVTom9tH): protocol: write tcp 188.166.225.170:47696->145.79.227.17:4995: use of closed network connection; write tcp 188.166.225.170:47696->145.79.227.17:4995: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47696->145.79.227.17:4995: use of closed network connection Dec 29 22:06:21.929215911 (582h37m21.800298976s) [noise=true , address=145.79.227.17:4995 , piece_id=JJTVPVBAMCTDGHGWNPUVPYFVKD4SMTDAHF4VWVIW7JFUAMBIPRHQ, seg_pos={0 0}, upload=510734]: Long tail cancellation due to optimal threshold Dec 29 22:06:21.929387247 (582h37m21.800470292s) [noise=true , address=193.149.164.143:4698 , piece_id=6X5KB4TXFV4ZOF5JTPNRW4Q3GWNCVCXCA4JSB3AZURBHXY2VHTRQ, seg_pos={0 0}, upload=510734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F4fJAGJM2ZPRt7vJH2cZmEbYeujmxcfi7aPZ2g9VPTNYBYRAi): protocol: expected piece hash; context canceled; EOF Dec 29 22:06:21.929396174 (582h37m21.800479219s) [noise=true , address=193.149.164.143:4698 , piece_id=6X5KB4TXFV4ZOF5JTPNRW4Q3GWNCVCXCA4JSB3AZURBHXY2VHTRQ, seg_pos={0 0}, upload=510734]: Long tail cancellation due to optimal threshold Dec 29 22:06:21.929491536 (582h37m21.800574571s) [seg_pos={0 0}, upload=510734]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 22:06:21.929573792 (582h37m21.800656837s) [upload=510734]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 22:06:21.929580555 (582h37m21.8006636s) [upload=510734]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 22:06:22.233045366 (582h37m22.104128401s) [upload=510734]: Done uploading segments... Dec 29 22:06:22.233064963 (582h37m22.104147998s) [upload=510734]: Upload finished. err: <nil> Duration: 17.323228894s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............X............................X................] duration: 28.577709782s trace: https://jaeger.dev.storj.io/trace/7096862780f29049 FAIL (2025-12-29 22:20:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== oise=true , address=145.79.227.30:4766 , piece_id=YWDMUGNTYVXWOCMSN63VIC6HOTWHPY7VJ7QWT76P63UAJIBS2XLA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.060493636 (582h37m18.931576671s) [noise=true , address=5.231.73.5:4022 , piece_id=5WVAJH3Y6TLATXY4IYB6ZADKKJDXDS52B2ZN6FJAZ7UHFVRYRRAA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.060685291 (582h37m18.931768326s) [noise=true , address=77.90.31.11:4579 , piece_id=DEJPYHGYNAP52CZ45BKGNXJM5MLIA2OQZ4BRKN5PBADNRVE3VLEA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.060749954 (582h37m18.931832989s) [noise=true , address=145.79.227.29:4976 , piece_id=YRCRXSSQOXPOJGVX46L7WCB3TRDZ5QB755SEOTNHSCHN4ZDMCOUQ, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.061033203 (582h37m18.932116248s) [noise=true , address=193.149.164.143:4698 , piece_id=6X5KB4TXFV4ZOF5JTPNRW4Q3GWNCVCXCA4JSB3AZURBHXY2VHTRQ, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.061248855 (582h37m18.9323319s) [noise=true , address=145.79.227.49:4347 , piece_id=75B3DRRHEPDL6EK6ZUXTKASXUKWGAVUWBPSTNRYT4BJMAU5ALXRA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.061382729 (582h37m18.932465764s) [noise=true , address=5.231.73.2:4004 , piece_id=PTHHO3XEPGGCIRCUBIWHN4LVNBLMISJM5LESQGR2ERRESFK3S4RA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.061645560 (582h37m18.932728595s) [noise=true , address=5.231.73.3:4007 , piece_id=6AF3CEPLUMRO4CDW3SRWZSYHWEXP35AVSM5M3LH4KCRARSZ6NFNA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.061769094 (582h37m18.932852139s) [noise=true , address=145.79.227.7:4321 , piece_id=SPWPZ5MR5ZCQNYVYYQQEZC4OURAM5VAH3WN5UP6HMGCCDMCH6ZGA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.061970298 (582h37m18.933053333s) [noise=true , address=145.79.227.35:4507 , piece_id=ZZSVIHOAN2SWXI4NS3TRD4B64NDB4D43J63DAS74TCPEE6MPGRXA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.062237577 (582h37m18.933320612s) [noise=true , address=145.79.227.35:4489 , piece_id=BM7ZQESTHEUIRJWAPORTKDMT5TVLEXMOHULGFVEQKHT4AJV3HT5A, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.062304304 (582h37m18.933387349s) [noise=true , address=145.79.227.34:4931 , piece_id=XJHLE3B5HIF4RC4LV32SBCNMVTOTEXWBBCXBVPQFN6N5HFQ6ZS7A, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.062522640 (582h37m18.933605685s) [noise=true , address=145.79.227.48:4329 , piece_id=YWCVTYLCAGU57SE3ID3WW2JLCJGTJ7CPDRCZENAMO42BVPIAVLIQ, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.062904968 (582h37m18.933988003s) [noise=true , address=77.90.31.17:4801 , piece_id=K3QD7FPVRQHZL5R57EKKAC27RGDJQH6HDCND7FIVXSZBRH42WSIA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.063420419 (582h37m18.934503464s) [noise=true , address=145.79.227.17:4995 , piece_id=JJTVPVBAMCTDGHGWNPUVPYFVKD4SMTDAHF4VWVIW7JFUAMBIPRHQ, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.063943495 (582h37m18.93502654s) [noise=true , address=5.231.73.4:4018 , piece_id=X45H47HER27IM5FEQSEAMXSDX7UQG3AEOY65Q4WCCUOX7SVFZ5KA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:20.386418430 (582h37m20.257501465s) [upload=510734]: Next returned nil segment Dec 29 22:06:20.386428990 (582h37m20.257512025s) [upload=510734]: Waiting for error group managing segments... Dec 29 22:06:20.935543714 (582h37m20.806626749s) [noise=true , address=145.79.227.48:4329 , piece_id=YWCVTYLCAGU57SE3ID3WW2JLCJGTJ7CPDRCZENAMO42BVPIAVLIQ, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:20.978073277 (582h37m20.849156312s) [noise=true , address=145.79.227.39:4022 , piece_id=5IJCSVRVXCICRNRB3TTJAVNRF5FTUQMMY4KL7ZFF4SBKILFGHJWA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.079004013 (582h37m20.950087058s) [noise=true , address=5.231.73.4:4018 , piece_id=X45H47HER27IM5FEQSEAMXSDX7UQG3AEOY65Q4WCCUOX7SVFZ5KA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.101144552 (582h37m20.972227597s) [noise=true , address=77.90.31.11:4579 , piece_id=DEJPYHGYNAP52CZ45BKGNXJM5MLIA2OQZ4BRKN5PBADNRVE3VLEA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.138766369 (582h37m21.009849405s) [noise=true , address=77.90.31.17:4801 , piece_id=K3QD7FPVRQHZL5R57EKKAC27RGDJQH6HDCND7FIVXSZBRH42WSIA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.231047035 (582h37m21.10213007s) [noise=true , address=77.90.31.19:4857 , piece_id=DMJ4O3TWZYWKWVLJFSQBULHDFWZCNLVDBIGYJEOBAVVCU5UGG27Q, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.272062706 (582h37m21.143145741s) [noise=true , address=5.231.73.3:4007 , piece_id=6AF3CEPLUMRO4CDW3SRWZSYHWEXP35AVSM5M3LH4KCRARSZ6NFNA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.303520666 (582h37m21.174603711s) [noise=true , address=145.79.227.29:4976 , piece_id=YRCRXSSQOXPOJGVX46L7WCB3TRDZ5QB755SEOTNHSCHN4ZDMCOUQ, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.340640177 (582h37m21.211723212s) [noise=true , address=5.231.73.5:4022 , piece_id=5WVAJH3Y6TLATXY4IYB6ZADKKJDXDS52B2ZN6FJAZ7UHFVRYRRAA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.481118228 (582h37m21.352201263s) [noise=true , address=5.231.73.2:4004 , piece_id=PTHHO3XEPGGCIRCUBIWHN4LVNBLMISJM5LESQGR2ERRESFK3S4RA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.896953769 (582h37m21.768036804s) [noise=true , address=145.79.227.49:4347 , piece_id=75B3DRRHEPDL6EK6ZUXTKASXUKWGAVUWBPSTNRYT4BJMAU5ALXRA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.911314353 (582h37m21.782397399s) [noise=true , address=145.79.227.34:4931 , piece_id=XJHLE3B5HIF4RC4LV32SBCNMVTOTEXWBBCXBVPQFN6N5HFQ6ZS7A, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.915845371 (582h37m21.786928406s) [noise=true , address=145.79.227.30:4766 , piece_id=YWDMUGNTYVXWOCMSN63VIC6HOTWHPY7VJ7QWT76P63UAJIBS2XLA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.928785972 (582h37m21.799869017s) [noise=true , address=145.79.227.35:4489 , piece_id=BM7ZQESTHEUIRJWAPORTKDMT5TVLEXMOHULGFVEQKHT4AJV3HT5A, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.928810859 (582h37m21.799893904s) [seg_pos={0 0}, upload=510734]: Segment reached optimal threshold of 14 pieces. Dec 29 22:06:21.929016471 (582h37m21.800099516s) [noise=true , address=145.79.227.7:4321 , piece_id=SPWPZ5MR5ZCQNYVYYQQEZC4OURAM5VAH3WN5UP6HMGCCDMCH6ZGA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12edsHqD9iRBDgMU8vYhSM1EWG8Sn57GZ8TPiTytri3rp7W9Eae): protocol: expected piece hash; context canceled; EOF Dec 29 22:06:21.929028644 (582h37m21.800111709s) [noise=true , address=145.79.227.7:4321 , piece_id=SPWPZ5MR5ZCQNYVYYQQEZC4OURAM5VAH3WN5UP6HMGCCDMCH6ZGA, seg_pos={0 0}, upload=510734]: Long tail cancellation due to optimal threshold Dec 29 22:06:21.929093137 (582h37m21.800176192s) [noise=true , address=145.79.227.35:4507 , piece_id=ZZSVIHOAN2SWXI4NS3TRD4B64NDB4D43J63DAS74TCPEE6MPGRXA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14jurWWPpmxwiBS3XmZysEUwugsceYJqVXFj7BbYpx8UFcTR8R): protocol: expected piece hash; context canceled; EOF Dec 29 22:06:21.929106793 (582h37m21.800189838s) [noise=true , address=145.79.227.35:4507 , piece_id=ZZSVIHOAN2SWXI4NS3TRD4B64NDB4D43J63DAS74TCPEE6MPGRXA, seg_pos={0 0}, upload=510734]: Long tail cancellation due to optimal threshold Dec 29 22:06:21.929206612 (582h37m21.800289657s) [noise=true , address=145.79.227.17:4995 , piece_id=JJTVPVBAMCTDGHGWNPUVPYFVKD4SMTDAHF4VWVIW7JFUAMBIPRHQ, seg_pos={0 0}, upload=510734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tvoTD1LoSj3q2rLiDckfT56dXNKKwdVu1ZAsBZExNFVTom9tH): protocol: write tcp 188.166.225.170:47696->145.79.227.17:4995: use of closed network connection; write tcp 188.166.225.170:47696->145.79.227.17:4995: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47696->145.79.227.17:4995: use of closed network connection Dec 29 22:06:21.929215911 (582h37m21.800298976s) [noise=true , address=145.79.227.17:4995 , piece_id=JJTVPVBAMCTDGHGWNPUVPYFVKD4SMTDAHF4VWVIW7JFUAMBIPRHQ, seg_pos={0 0}, upload=510734]: Long tail cancellation due to optimal threshold Dec 29 22:06:21.929387247 (582h37m21.800470292s) [noise=true , address=193.149.164.143:4698 , piece_id=6X5KB4TXFV4ZOF5JTPNRW4Q3GWNCVCXCA4JSB3AZURBHXY2VHTRQ, seg_pos={0 0}, upload=510734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F4fJAGJM2ZPRt7vJH2cZmEbYeujmxcfi7aPZ2g9VPTNYBYRAi): protocol: expected piece hash; context canceled; EOF Dec 29 22:06:21.929396174 (582h37m21.800479219s) [noise=true , address=193.149.164.143:4698 , piece_id=6X5KB4TXFV4ZOF5JTPNRW4Q3GWNCVCXCA4JSB3AZURBHXY2VHTRQ, seg_pos={0 0}, upload=510734]: Long tail cancellation due to optimal threshold Dec 29 22:06:21.929491536 (582h37m21.800574571s) [seg_pos={0 0}, upload=510734]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 22:06:21.929573792 (582h37m21.800656837s) [upload=510734]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 22:06:21.929580555 (582h37m21.8006636s) [upload=510734]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 22:06:22.233045366 (582h37m22.104128401s) [upload=510734]: Done uploading segments... Dec 29 22:06:22.233064963 (582h37m22.104147998s) [upload=510734]: Upload finished. err: <nil> Duration: 28.577739337s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....X............................X.........................] duration: 7.042585322s trace: https://jaeger.dev.storj.io/trace/5209b3d035311fd FAIL (2025-12-29 22:19:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :00:17.043136250 (582h31m16.914219305s) [upload=510642]: Starting upload Dec 29 22:00:17.043227904 (582h31m16.91431096s) [upload=510642]: Done starting upload Dec 29 22:00:17.043335189 (582h31m16.914418234s) [upload=510642]: Uploading segments... Dec 29 22:00:17.043587749 (582h31m16.914670824s) [upload=510642]: Got next segment. Inline: false Dec 29 22:00:17.043644478 (582h31m16.914727523s) [upload=510642]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 22:00:17.043686427 (582h31m16.914769503s) [upload=510642]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 22:00:17.294384363 (582h31m17.165467408s) [seg_pos={0 0}, upload=510642]: Begin upload segment... Dec 29 22:00:17.294445219 (582h31m17.165528264s) [seg_pos={0 0}, upload=510642]: Done begin upload segment. Dec 29 22:00:17.294484133 (582h31m17.165567178s) [noise=true , address=145.79.227.31:4797 , piece_id=NI3M5NVVVX5HUXZ2SOMQD4S5ZOSSNPCD2F3Z5X2BA2VXMGT57AYQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.294560979 (582h31m17.165644014s) [noise=true , address=77.90.31.11:4553 , piece_id=3Y5HSD73DLZ6ALJ5TFRIVTLOE4ZNZ256YSZ2W3X5LPKL5J5LX2EQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.295149941 (582h31m17.166232976s) [noise=true , address=45.91.116.5:4297 , piece_id=PLUQ7Q32BTGYFQRGVJFAJWHPZCPQTC5RX5UX27S5HJ5NNJPLDMPQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.295419574 (582h31m17.166502609s) [noise=true , address=145.79.227.2:4115 , piece_id=P2AS5B3VYO2K2AWSM5W4BXAXBHSXQQWM7JOXFPOV64NTKDFLT62Q, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.295483997 (582h31m17.166567042s) [noise=true , address=145.79.227.51:4411 , piece_id=HTZQ6SW6KSACLQF7G4CFGQA572J5JVHMLGEWQYSSGZRYX7FJ6DXA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.295762207 (582h31m17.166845252s) [noise=true , address=145.79.227.5:4221 , piece_id=LD3FENLBM2L5N7QDOAMAYJUVH54QVSJQJFP7E36UAH2ZD7R65CCQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.295928003 (582h31m17.167011048s) [noise=true , address=145.79.227.16:4909 , piece_id=CF7WHNKC4LFJQRVFBPFV62JGEBLD4KVPE6Q5JBWQVADC6ZAFFLVQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.296039615 (582h31m17.16712266s) [noise=true , address=45.91.116.13:4154 , piece_id=FE4NFODQ3TWGFXARFYIL64TKK6X7H6PPLIG6LO44LKNAZOEU7DJA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.296272429 (582h31m17.167355474s) [noise=true , address=45.91.116.10:4084 , piece_id=5BFUYVGIFOYXC6RCPU7VBUM36LSM32TVBLS3W54OWFNR4SM6GCSA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.296403588 (582h31m17.167486633s) [noise=true , address=77.90.31.19:4844 , piece_id=CDOWECXLSRPACV4HMAH57UK4EEIYNOMH6GYV6HFHV7CWSEAJ44FA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.296646560 (582h31m17.167729596s) [noise=true , address=77.90.31.14:4661 , piece_id=B7IDSCDHBUXUIPSUT3HMQSCMPGBMYASZSQO2EJBXRR2DQHCLWGYA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.296906235 (582h31m17.167989291s) [noise=true , address=145.79.227.47:4265 , piece_id=FYZA2LRIFKLKVUCUAWMNWTUH3VYR5VFAA7XKAYIMD2E2VOJTI7PA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.296989804 (582h31m17.168072849s) [noise=true , address=145.79.227.46:4601 , piece_id=YJTMZID24VBM5ZZSUUHOCZSZSSNGTFHOSKK6OI5QP3A4G6ZWVWQA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.297235221 (582h31m17.168318267s) [noise=true , address=145.79.227.3:4150 , piece_id=HS4S5UDYD2VWI2N4MH6EWEYQ5G6KCFCJBDZR7AVWPXRYALFXP4FQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.297424432 (582h31m17.168507467s) [noise=true , address=145.79.227.13:4803 , piece_id=YIEUZEDLSC5C67SCFEPCHHZSTYP6MYLYJZJTGZF3UP6JWDTGUOPQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.297606900 (582h31m17.168689945s) [noise=true , address=45.91.116.14:4190 , piece_id=FIL6AIXGE7DE2M4QBVSIO53KGY43VFPZQ2FTXSWJFPJU4VZCKUXQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:18.995364155 (582h31m18.866447201s) [upload=510642]: Next returned nil segment Dec 29 22:00:18.995383603 (582h31m18.866466648s) [upload=510642]: Waiting for error group managing segments... Dec 29 22:00:19.284009408 (582h31m19.155092453s) [noise=true , address=145.79.227.51:4411 , piece_id=HTZQ6SW6KSACLQF7G4CFGQA572J5JVHMLGEWQYSSGZRYX7FJ6DXA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.295985232 (582h31m19.167068277s) [noise=true , address=145.79.227.13:4803 , piece_id=YIEUZEDLSC5C67SCFEPCHHZSTYP6MYLYJZJTGZF3UP6JWDTGUOPQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.296135458 (582h31m19.167218503s) [noise=true , address=145.79.227.16:4909 , piece_id=CF7WHNKC4LFJQRVFBPFV62JGEBLD4KVPE6Q5JBWQVADC6ZAFFLVQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.310119916 (582h31m19.181202971s) [noise=true , address=145.79.227.46:4601 , piece_id=YJTMZID24VBM5ZZSUUHOCZSZSSNGTFHOSKK6OI5QP3A4G6ZWVWQA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.310998939 (582h31m19.182081984s) [noise=true , address=145.79.227.31:4797 , piece_id=NI3M5NVVVX5HUXZ2SOMQD4S5ZOSSNPCD2F3Z5X2BA2VXMGT57AYQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.318707138 (582h31m19.189790183s) [noise=true , address=145.79.227.47:4265 , piece_id=FYZA2LRIFKLKVUCUAWMNWTUH3VYR5VFAA7XKAYIMD2E2VOJTI7PA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.322059381 (582h31m19.193142426s) [noise=true , address=145.79.227.5:4221 , piece_id=LD3FENLBM2L5N7QDOAMAYJUVH54QVSJQJFP7E36UAH2ZD7R65CCQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.506372108 (582h31m19.377455153s) [noise=true , address=77.90.31.19:4844 , piece_id=CDOWECXLSRPACV4HMAH57UK4EEIYNOMH6GYV6HFHV7CWSEAJ44FA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.507172602 (582h31m19.378255648s) [noise=true , address=77.90.31.11:4553 , piece_id=3Y5HSD73DLZ6ALJ5TFRIVTLOE4ZNZ256YSZ2W3X5LPKL5J5LX2EQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.527548842 (582h31m19.398631887s) [noise=true , address=77.90.31.14:4661 , piece_id=B7IDSCDHBUXUIPSUT3HMQSCMPGBMYASZSQO2EJBXRR2DQHCLWGYA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.531443508 (582h31m19.402526553s) [noise=true , address=45.91.116.5:4297 , piece_id=PLUQ7Q32BTGYFQRGVJFAJWHPZCPQTC5RX5UX27S5HJ5NNJPLDMPQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.552364906 (582h31m19.423447951s) [noise=true , address=45.91.116.14:4190 , piece_id=FIL6AIXGE7DE2M4QBVSIO53KGY43VFPZQ2FTXSWJFPJU4VZCKUXQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.564056769 (582h31m19.435139814s) [noise=true , address=45.91.116.10:4084 , piece_id=5BFUYVGIFOYXC6RCPU7VBUM36LSM32TVBLS3W54OWFNR4SM6GCSA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.564087016 (582h31m19.435170051s) [seg_pos={0 0}, upload=510642]: Segment reached optimal threshold of 13 pieces. Dec 29 22:00:19.564294431 (582h31m19.435377466s) [noise=true , address=145.79.227.3:4150 , piece_id=HS4S5UDYD2VWI2N4MH6EWEYQ5G6KCFCJBDZR7AVWPXRYALFXP4FQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK): protocol: expected piece hash; context canceled; EOF Dec 29 22:00:19.564302366 (582h31m19.435385411s) [noise=true , address=145.79.227.3:4150 , piece_id=HS4S5UDYD2VWI2N4MH6EWEYQ5G6KCFCJBDZR7AVWPXRYALFXP4FQ, seg_pos={0 0}, upload=510642]: Long tail cancellation due to optimal threshold Dec 29 22:00:19.564388932 (582h31m19.435471977s) [noise=true , address=45.91.116.13:4154 , piece_id=FE4NFODQ3TWGFXARFYIL64TKK6X7H6PPLIG6LO44LKNAZOEU7DJA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Dec 29 22:00:19.564401876 (582h31m19.435484911s) [noise=true , address=45.91.116.13:4154 , piece_id=FE4NFODQ3TWGFXARFYIL64TKK6X7H6PPLIG6LO44LKNAZOEU7DJA, seg_pos={0 0}, upload=510642]: Long tail cancellation due to optimal threshold Dec 29 22:00:19.564558945 (582h31m19.435641981s) [noise=true , address=145.79.227.2:4115 , piece_id=P2AS5B3VYO2K2AWSM5W4BXAXBHSXQQWM7JOXFPOV64NTKDFLT62Q, seg_pos={0 0}, upload=510642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: write tcp 188.166.225.170:38246->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:38246->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38246->145.79.227.2:4115: use of closed network connection Dec 29 22:00:19.564570738 (582h31m19.435653783s) [noise=true , address=145.79.227.2:4115 , piece_id=P2AS5B3VYO2K2AWSM5W4BXAXBHSXQQWM7JOXFPOV64NTKDFLT62Q, seg_pos={0 0}, upload=510642]: Long tail cancellation due to optimal threshold Dec 29 22:00:19.564601426 (582h31m19.435684471s) [seg_pos={0 0}, upload=510642]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 22:00:19.564683973 (582h31m19.435767018s) [upload=510642]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 22:00:19.564690004 (582h31m19.435773049s) [upload=510642]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 22:00:19.871860472 (582h31m19.742943527s) [upload=510642]: Done uploading segments... Dec 29 22:00:19.871892413 (582h31m19.742975468s) [upload=510642]: Upload finished. err: <nil> Duration: 7.042615839s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............X............................X...............] duration: 15.440533586s trace: https://jaeger.dev.storj.io/trace/7096862780f29049 FAIL (2025-12-29 22:19:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== oise=true , address=145.79.227.30:4766 , piece_id=YWDMUGNTYVXWOCMSN63VIC6HOTWHPY7VJ7QWT76P63UAJIBS2XLA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.060493636 (582h37m18.931576671s) [noise=true , address=5.231.73.5:4022 , piece_id=5WVAJH3Y6TLATXY4IYB6ZADKKJDXDS52B2ZN6FJAZ7UHFVRYRRAA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.060685291 (582h37m18.931768326s) [noise=true , address=77.90.31.11:4579 , piece_id=DEJPYHGYNAP52CZ45BKGNXJM5MLIA2OQZ4BRKN5PBADNRVE3VLEA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.060749954 (582h37m18.931832989s) [noise=true , address=145.79.227.29:4976 , piece_id=YRCRXSSQOXPOJGVX46L7WCB3TRDZ5QB755SEOTNHSCHN4ZDMCOUQ, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.061033203 (582h37m18.932116248s) [noise=true , address=193.149.164.143:4698 , piece_id=6X5KB4TXFV4ZOF5JTPNRW4Q3GWNCVCXCA4JSB3AZURBHXY2VHTRQ, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.061248855 (582h37m18.9323319s) [noise=true , address=145.79.227.49:4347 , piece_id=75B3DRRHEPDL6EK6ZUXTKASXUKWGAVUWBPSTNRYT4BJMAU5ALXRA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.061382729 (582h37m18.932465764s) [noise=true , address=5.231.73.2:4004 , piece_id=PTHHO3XEPGGCIRCUBIWHN4LVNBLMISJM5LESQGR2ERRESFK3S4RA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.061645560 (582h37m18.932728595s) [noise=true , address=5.231.73.3:4007 , piece_id=6AF3CEPLUMRO4CDW3SRWZSYHWEXP35AVSM5M3LH4KCRARSZ6NFNA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.061769094 (582h37m18.932852139s) [noise=true , address=145.79.227.7:4321 , piece_id=SPWPZ5MR5ZCQNYVYYQQEZC4OURAM5VAH3WN5UP6HMGCCDMCH6ZGA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.061970298 (582h37m18.933053333s) [noise=true , address=145.79.227.35:4507 , piece_id=ZZSVIHOAN2SWXI4NS3TRD4B64NDB4D43J63DAS74TCPEE6MPGRXA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.062237577 (582h37m18.933320612s) [noise=true , address=145.79.227.35:4489 , piece_id=BM7ZQESTHEUIRJWAPORTKDMT5TVLEXMOHULGFVEQKHT4AJV3HT5A, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.062304304 (582h37m18.933387349s) [noise=true , address=145.79.227.34:4931 , piece_id=XJHLE3B5HIF4RC4LV32SBCNMVTOTEXWBBCXBVPQFN6N5HFQ6ZS7A, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.062522640 (582h37m18.933605685s) [noise=true , address=145.79.227.48:4329 , piece_id=YWCVTYLCAGU57SE3ID3WW2JLCJGTJ7CPDRCZENAMO42BVPIAVLIQ, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.062904968 (582h37m18.933988003s) [noise=true , address=77.90.31.17:4801 , piece_id=K3QD7FPVRQHZL5R57EKKAC27RGDJQH6HDCND7FIVXSZBRH42WSIA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.063420419 (582h37m18.934503464s) [noise=true , address=145.79.227.17:4995 , piece_id=JJTVPVBAMCTDGHGWNPUVPYFVKD4SMTDAHF4VWVIW7JFUAMBIPRHQ, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.063943495 (582h37m18.93502654s) [noise=true , address=5.231.73.4:4018 , piece_id=X45H47HER27IM5FEQSEAMXSDX7UQG3AEOY65Q4WCCUOX7SVFZ5KA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:20.386418430 (582h37m20.257501465s) [upload=510734]: Next returned nil segment Dec 29 22:06:20.386428990 (582h37m20.257512025s) [upload=510734]: Waiting for error group managing segments... Dec 29 22:06:20.935543714 (582h37m20.806626749s) [noise=true , address=145.79.227.48:4329 , piece_id=YWCVTYLCAGU57SE3ID3WW2JLCJGTJ7CPDRCZENAMO42BVPIAVLIQ, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:20.978073277 (582h37m20.849156312s) [noise=true , address=145.79.227.39:4022 , piece_id=5IJCSVRVXCICRNRB3TTJAVNRF5FTUQMMY4KL7ZFF4SBKILFGHJWA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.079004013 (582h37m20.950087058s) [noise=true , address=5.231.73.4:4018 , piece_id=X45H47HER27IM5FEQSEAMXSDX7UQG3AEOY65Q4WCCUOX7SVFZ5KA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.101144552 (582h37m20.972227597s) [noise=true , address=77.90.31.11:4579 , piece_id=DEJPYHGYNAP52CZ45BKGNXJM5MLIA2OQZ4BRKN5PBADNRVE3VLEA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.138766369 (582h37m21.009849405s) [noise=true , address=77.90.31.17:4801 , piece_id=K3QD7FPVRQHZL5R57EKKAC27RGDJQH6HDCND7FIVXSZBRH42WSIA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.231047035 (582h37m21.10213007s) [noise=true , address=77.90.31.19:4857 , piece_id=DMJ4O3TWZYWKWVLJFSQBULHDFWZCNLVDBIGYJEOBAVVCU5UGG27Q, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.272062706 (582h37m21.143145741s) [noise=true , address=5.231.73.3:4007 , piece_id=6AF3CEPLUMRO4CDW3SRWZSYHWEXP35AVSM5M3LH4KCRARSZ6NFNA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.303520666 (582h37m21.174603711s) [noise=true , address=145.79.227.29:4976 , piece_id=YRCRXSSQOXPOJGVX46L7WCB3TRDZ5QB755SEOTNHSCHN4ZDMCOUQ, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.340640177 (582h37m21.211723212s) [noise=true , address=5.231.73.5:4022 , piece_id=5WVAJH3Y6TLATXY4IYB6ZADKKJDXDS52B2ZN6FJAZ7UHFVRYRRAA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.481118228 (582h37m21.352201263s) [noise=true , address=5.231.73.2:4004 , piece_id=PTHHO3XEPGGCIRCUBIWHN4LVNBLMISJM5LESQGR2ERRESFK3S4RA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.896953769 (582h37m21.768036804s) [noise=true , address=145.79.227.49:4347 , piece_id=75B3DRRHEPDL6EK6ZUXTKASXUKWGAVUWBPSTNRYT4BJMAU5ALXRA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.911314353 (582h37m21.782397399s) [noise=true , address=145.79.227.34:4931 , piece_id=XJHLE3B5HIF4RC4LV32SBCNMVTOTEXWBBCXBVPQFN6N5HFQ6ZS7A, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.915845371 (582h37m21.786928406s) [noise=true , address=145.79.227.30:4766 , piece_id=YWDMUGNTYVXWOCMSN63VIC6HOTWHPY7VJ7QWT76P63UAJIBS2XLA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.928785972 (582h37m21.799869017s) [noise=true , address=145.79.227.35:4489 , piece_id=BM7ZQESTHEUIRJWAPORTKDMT5TVLEXMOHULGFVEQKHT4AJV3HT5A, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.928810859 (582h37m21.799893904s) [seg_pos={0 0}, upload=510734]: Segment reached optimal threshold of 14 pieces. Dec 29 22:06:21.929016471 (582h37m21.800099516s) [noise=true , address=145.79.227.7:4321 , piece_id=SPWPZ5MR5ZCQNYVYYQQEZC4OURAM5VAH3WN5UP6HMGCCDMCH6ZGA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12edsHqD9iRBDgMU8vYhSM1EWG8Sn57GZ8TPiTytri3rp7W9Eae): protocol: expected piece hash; context canceled; EOF Dec 29 22:06:21.929028644 (582h37m21.800111709s) [noise=true , address=145.79.227.7:4321 , piece_id=SPWPZ5MR5ZCQNYVYYQQEZC4OURAM5VAH3WN5UP6HMGCCDMCH6ZGA, seg_pos={0 0}, upload=510734]: Long tail cancellation due to optimal threshold Dec 29 22:06:21.929093137 (582h37m21.800176192s) [noise=true , address=145.79.227.35:4507 , piece_id=ZZSVIHOAN2SWXI4NS3TRD4B64NDB4D43J63DAS74TCPEE6MPGRXA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14jurWWPpmxwiBS3XmZysEUwugsceYJqVXFj7BbYpx8UFcTR8R): protocol: expected piece hash; context canceled; EOF Dec 29 22:06:21.929106793 (582h37m21.800189838s) [noise=true , address=145.79.227.35:4507 , piece_id=ZZSVIHOAN2SWXI4NS3TRD4B64NDB4D43J63DAS74TCPEE6MPGRXA, seg_pos={0 0}, upload=510734]: Long tail cancellation due to optimal threshold Dec 29 22:06:21.929206612 (582h37m21.800289657s) [noise=true , address=145.79.227.17:4995 , piece_id=JJTVPVBAMCTDGHGWNPUVPYFVKD4SMTDAHF4VWVIW7JFUAMBIPRHQ, seg_pos={0 0}, upload=510734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tvoTD1LoSj3q2rLiDckfT56dXNKKwdVu1ZAsBZExNFVTom9tH): protocol: write tcp 188.166.225.170:47696->145.79.227.17:4995: use of closed network connection; write tcp 188.166.225.170:47696->145.79.227.17:4995: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47696->145.79.227.17:4995: use of closed network connection Dec 29 22:06:21.929215911 (582h37m21.800298976s) [noise=true , address=145.79.227.17:4995 , piece_id=JJTVPVBAMCTDGHGWNPUVPYFVKD4SMTDAHF4VWVIW7JFUAMBIPRHQ, seg_pos={0 0}, upload=510734]: Long tail cancellation due to optimal threshold Dec 29 22:06:21.929387247 (582h37m21.800470292s) [noise=true , address=193.149.164.143:4698 , piece_id=6X5KB4TXFV4ZOF5JTPNRW4Q3GWNCVCXCA4JSB3AZURBHXY2VHTRQ, seg_pos={0 0}, upload=510734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F4fJAGJM2ZPRt7vJH2cZmEbYeujmxcfi7aPZ2g9VPTNYBYRAi): protocol: expected piece hash; context canceled; EOF Dec 29 22:06:21.929396174 (582h37m21.800479219s) [noise=true , address=193.149.164.143:4698 , piece_id=6X5KB4TXFV4ZOF5JTPNRW4Q3GWNCVCXCA4JSB3AZURBHXY2VHTRQ, seg_pos={0 0}, upload=510734]: Long tail cancellation due to optimal threshold Dec 29 22:06:21.929491536 (582h37m21.800574571s) [seg_pos={0 0}, upload=510734]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 22:06:21.929573792 (582h37m21.800656837s) [upload=510734]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 22:06:21.929580555 (582h37m21.8006636s) [upload=510734]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 22:06:22.233045366 (582h37m22.104128401s) [upload=510734]: Done uploading segments... Dec 29 22:06:22.233064963 (582h37m22.104147998s) [upload=510734]: Upload finished. err: <nil> Duration: 15.440562552s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [................X............................X..............] duration: 16.381032775s trace: https://jaeger.dev.storj.io/trace/7096862780f29049 FAIL (2025-12-29 22:18:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== oise=true , address=145.79.227.30:4766 , piece_id=YWDMUGNTYVXWOCMSN63VIC6HOTWHPY7VJ7QWT76P63UAJIBS2XLA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.060493636 (582h37m18.931576671s) [noise=true , address=5.231.73.5:4022 , piece_id=5WVAJH3Y6TLATXY4IYB6ZADKKJDXDS52B2ZN6FJAZ7UHFVRYRRAA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.060685291 (582h37m18.931768326s) [noise=true , address=77.90.31.11:4579 , piece_id=DEJPYHGYNAP52CZ45BKGNXJM5MLIA2OQZ4BRKN5PBADNRVE3VLEA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.060749954 (582h37m18.931832989s) [noise=true , address=145.79.227.29:4976 , piece_id=YRCRXSSQOXPOJGVX46L7WCB3TRDZ5QB755SEOTNHSCHN4ZDMCOUQ, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.061033203 (582h37m18.932116248s) [noise=true , address=193.149.164.143:4698 , piece_id=6X5KB4TXFV4ZOF5JTPNRW4Q3GWNCVCXCA4JSB3AZURBHXY2VHTRQ, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.061248855 (582h37m18.9323319s) [noise=true , address=145.79.227.49:4347 , piece_id=75B3DRRHEPDL6EK6ZUXTKASXUKWGAVUWBPSTNRYT4BJMAU5ALXRA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.061382729 (582h37m18.932465764s) [noise=true , address=5.231.73.2:4004 , piece_id=PTHHO3XEPGGCIRCUBIWHN4LVNBLMISJM5LESQGR2ERRESFK3S4RA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.061645560 (582h37m18.932728595s) [noise=true , address=5.231.73.3:4007 , piece_id=6AF3CEPLUMRO4CDW3SRWZSYHWEXP35AVSM5M3LH4KCRARSZ6NFNA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.061769094 (582h37m18.932852139s) [noise=true , address=145.79.227.7:4321 , piece_id=SPWPZ5MR5ZCQNYVYYQQEZC4OURAM5VAH3WN5UP6HMGCCDMCH6ZGA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.061970298 (582h37m18.933053333s) [noise=true , address=145.79.227.35:4507 , piece_id=ZZSVIHOAN2SWXI4NS3TRD4B64NDB4D43J63DAS74TCPEE6MPGRXA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.062237577 (582h37m18.933320612s) [noise=true , address=145.79.227.35:4489 , piece_id=BM7ZQESTHEUIRJWAPORTKDMT5TVLEXMOHULGFVEQKHT4AJV3HT5A, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.062304304 (582h37m18.933387349s) [noise=true , address=145.79.227.34:4931 , piece_id=XJHLE3B5HIF4RC4LV32SBCNMVTOTEXWBBCXBVPQFN6N5HFQ6ZS7A, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.062522640 (582h37m18.933605685s) [noise=true , address=145.79.227.48:4329 , piece_id=YWCVTYLCAGU57SE3ID3WW2JLCJGTJ7CPDRCZENAMO42BVPIAVLIQ, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.062904968 (582h37m18.933988003s) [noise=true , address=77.90.31.17:4801 , piece_id=K3QD7FPVRQHZL5R57EKKAC27RGDJQH6HDCND7FIVXSZBRH42WSIA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.063420419 (582h37m18.934503464s) [noise=true , address=145.79.227.17:4995 , piece_id=JJTVPVBAMCTDGHGWNPUVPYFVKD4SMTDAHF4VWVIW7JFUAMBIPRHQ, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.063943495 (582h37m18.93502654s) [noise=true , address=5.231.73.4:4018 , piece_id=X45H47HER27IM5FEQSEAMXSDX7UQG3AEOY65Q4WCCUOX7SVFZ5KA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:20.386418430 (582h37m20.257501465s) [upload=510734]: Next returned nil segment Dec 29 22:06:20.386428990 (582h37m20.257512025s) [upload=510734]: Waiting for error group managing segments... Dec 29 22:06:20.935543714 (582h37m20.806626749s) [noise=true , address=145.79.227.48:4329 , piece_id=YWCVTYLCAGU57SE3ID3WW2JLCJGTJ7CPDRCZENAMO42BVPIAVLIQ, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:20.978073277 (582h37m20.849156312s) [noise=true , address=145.79.227.39:4022 , piece_id=5IJCSVRVXCICRNRB3TTJAVNRF5FTUQMMY4KL7ZFF4SBKILFGHJWA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.079004013 (582h37m20.950087058s) [noise=true , address=5.231.73.4:4018 , piece_id=X45H47HER27IM5FEQSEAMXSDX7UQG3AEOY65Q4WCCUOX7SVFZ5KA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.101144552 (582h37m20.972227597s) [noise=true , address=77.90.31.11:4579 , piece_id=DEJPYHGYNAP52CZ45BKGNXJM5MLIA2OQZ4BRKN5PBADNRVE3VLEA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.138766369 (582h37m21.009849405s) [noise=true , address=77.90.31.17:4801 , piece_id=K3QD7FPVRQHZL5R57EKKAC27RGDJQH6HDCND7FIVXSZBRH42WSIA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.231047035 (582h37m21.10213007s) [noise=true , address=77.90.31.19:4857 , piece_id=DMJ4O3TWZYWKWVLJFSQBULHDFWZCNLVDBIGYJEOBAVVCU5UGG27Q, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.272062706 (582h37m21.143145741s) [noise=true , address=5.231.73.3:4007 , piece_id=6AF3CEPLUMRO4CDW3SRWZSYHWEXP35AVSM5M3LH4KCRARSZ6NFNA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.303520666 (582h37m21.174603711s) [noise=true , address=145.79.227.29:4976 , piece_id=YRCRXSSQOXPOJGVX46L7WCB3TRDZ5QB755SEOTNHSCHN4ZDMCOUQ, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.340640177 (582h37m21.211723212s) [noise=true , address=5.231.73.5:4022 , piece_id=5WVAJH3Y6TLATXY4IYB6ZADKKJDXDS52B2ZN6FJAZ7UHFVRYRRAA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.481118228 (582h37m21.352201263s) [noise=true , address=5.231.73.2:4004 , piece_id=PTHHO3XEPGGCIRCUBIWHN4LVNBLMISJM5LESQGR2ERRESFK3S4RA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.896953769 (582h37m21.768036804s) [noise=true , address=145.79.227.49:4347 , piece_id=75B3DRRHEPDL6EK6ZUXTKASXUKWGAVUWBPSTNRYT4BJMAU5ALXRA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.911314353 (582h37m21.782397399s) [noise=true , address=145.79.227.34:4931 , piece_id=XJHLE3B5HIF4RC4LV32SBCNMVTOTEXWBBCXBVPQFN6N5HFQ6ZS7A, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.915845371 (582h37m21.786928406s) [noise=true , address=145.79.227.30:4766 , piece_id=YWDMUGNTYVXWOCMSN63VIC6HOTWHPY7VJ7QWT76P63UAJIBS2XLA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.928785972 (582h37m21.799869017s) [noise=true , address=145.79.227.35:4489 , piece_id=BM7ZQESTHEUIRJWAPORTKDMT5TVLEXMOHULGFVEQKHT4AJV3HT5A, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.928810859 (582h37m21.799893904s) [seg_pos={0 0}, upload=510734]: Segment reached optimal threshold of 14 pieces. Dec 29 22:06:21.929016471 (582h37m21.800099516s) [noise=true , address=145.79.227.7:4321 , piece_id=SPWPZ5MR5ZCQNYVYYQQEZC4OURAM5VAH3WN5UP6HMGCCDMCH6ZGA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12edsHqD9iRBDgMU8vYhSM1EWG8Sn57GZ8TPiTytri3rp7W9Eae): protocol: expected piece hash; context canceled; EOF Dec 29 22:06:21.929028644 (582h37m21.800111709s) [noise=true , address=145.79.227.7:4321 , piece_id=SPWPZ5MR5ZCQNYVYYQQEZC4OURAM5VAH3WN5UP6HMGCCDMCH6ZGA, seg_pos={0 0}, upload=510734]: Long tail cancellation due to optimal threshold Dec 29 22:06:21.929093137 (582h37m21.800176192s) [noise=true , address=145.79.227.35:4507 , piece_id=ZZSVIHOAN2SWXI4NS3TRD4B64NDB4D43J63DAS74TCPEE6MPGRXA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14jurWWPpmxwiBS3XmZysEUwugsceYJqVXFj7BbYpx8UFcTR8R): protocol: expected piece hash; context canceled; EOF Dec 29 22:06:21.929106793 (582h37m21.800189838s) [noise=true , address=145.79.227.35:4507 , piece_id=ZZSVIHOAN2SWXI4NS3TRD4B64NDB4D43J63DAS74TCPEE6MPGRXA, seg_pos={0 0}, upload=510734]: Long tail cancellation due to optimal threshold Dec 29 22:06:21.929206612 (582h37m21.800289657s) [noise=true , address=145.79.227.17:4995 , piece_id=JJTVPVBAMCTDGHGWNPUVPYFVKD4SMTDAHF4VWVIW7JFUAMBIPRHQ, seg_pos={0 0}, upload=510734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tvoTD1LoSj3q2rLiDckfT56dXNKKwdVu1ZAsBZExNFVTom9tH): protocol: write tcp 188.166.225.170:47696->145.79.227.17:4995: use of closed network connection; write tcp 188.166.225.170:47696->145.79.227.17:4995: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47696->145.79.227.17:4995: use of closed network connection Dec 29 22:06:21.929215911 (582h37m21.800298976s) [noise=true , address=145.79.227.17:4995 , piece_id=JJTVPVBAMCTDGHGWNPUVPYFVKD4SMTDAHF4VWVIW7JFUAMBIPRHQ, seg_pos={0 0}, upload=510734]: Long tail cancellation due to optimal threshold Dec 29 22:06:21.929387247 (582h37m21.800470292s) [noise=true , address=193.149.164.143:4698 , piece_id=6X5KB4TXFV4ZOF5JTPNRW4Q3GWNCVCXCA4JSB3AZURBHXY2VHTRQ, seg_pos={0 0}, upload=510734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F4fJAGJM2ZPRt7vJH2cZmEbYeujmxcfi7aPZ2g9VPTNYBYRAi): protocol: expected piece hash; context canceled; EOF Dec 29 22:06:21.929396174 (582h37m21.800479219s) [noise=true , address=193.149.164.143:4698 , piece_id=6X5KB4TXFV4ZOF5JTPNRW4Q3GWNCVCXCA4JSB3AZURBHXY2VHTRQ, seg_pos={0 0}, upload=510734]: Long tail cancellation due to optimal threshold Dec 29 22:06:21.929491536 (582h37m21.800574571s) [seg_pos={0 0}, upload=510734]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 22:06:21.929573792 (582h37m21.800656837s) [upload=510734]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 22:06:21.929580555 (582h37m21.8006636s) [upload=510734]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 22:06:22.233045366 (582h37m22.104128401s) [upload=510734]: Done uploading segments... Dec 29 22:06:22.233064963 (582h37m22.104147998s) [upload=510734]: Upload finished. err: <nil> Duration: 16.381065908s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......X............................X........................] duration: 6.021479312s trace: https://jaeger.dev.storj.io/trace/5209b3d035311fd FAIL (2025-12-29 22:18:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :00:17.043136250 (582h31m16.914219305s) [upload=510642]: Starting upload Dec 29 22:00:17.043227904 (582h31m16.91431096s) [upload=510642]: Done starting upload Dec 29 22:00:17.043335189 (582h31m16.914418234s) [upload=510642]: Uploading segments... Dec 29 22:00:17.043587749 (582h31m16.914670824s) [upload=510642]: Got next segment. Inline: false Dec 29 22:00:17.043644478 (582h31m16.914727523s) [upload=510642]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 22:00:17.043686427 (582h31m16.914769503s) [upload=510642]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 22:00:17.294384363 (582h31m17.165467408s) [seg_pos={0 0}, upload=510642]: Begin upload segment... Dec 29 22:00:17.294445219 (582h31m17.165528264s) [seg_pos={0 0}, upload=510642]: Done begin upload segment. Dec 29 22:00:17.294484133 (582h31m17.165567178s) [noise=true , address=145.79.227.31:4797 , piece_id=NI3M5NVVVX5HUXZ2SOMQD4S5ZOSSNPCD2F3Z5X2BA2VXMGT57AYQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.294560979 (582h31m17.165644014s) [noise=true , address=77.90.31.11:4553 , piece_id=3Y5HSD73DLZ6ALJ5TFRIVTLOE4ZNZ256YSZ2W3X5LPKL5J5LX2EQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.295149941 (582h31m17.166232976s) [noise=true , address=45.91.116.5:4297 , piece_id=PLUQ7Q32BTGYFQRGVJFAJWHPZCPQTC5RX5UX27S5HJ5NNJPLDMPQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.295419574 (582h31m17.166502609s) [noise=true , address=145.79.227.2:4115 , piece_id=P2AS5B3VYO2K2AWSM5W4BXAXBHSXQQWM7JOXFPOV64NTKDFLT62Q, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.295483997 (582h31m17.166567042s) [noise=true , address=145.79.227.51:4411 , piece_id=HTZQ6SW6KSACLQF7G4CFGQA572J5JVHMLGEWQYSSGZRYX7FJ6DXA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.295762207 (582h31m17.166845252s) [noise=true , address=145.79.227.5:4221 , piece_id=LD3FENLBM2L5N7QDOAMAYJUVH54QVSJQJFP7E36UAH2ZD7R65CCQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.295928003 (582h31m17.167011048s) [noise=true , address=145.79.227.16:4909 , piece_id=CF7WHNKC4LFJQRVFBPFV62JGEBLD4KVPE6Q5JBWQVADC6ZAFFLVQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.296039615 (582h31m17.16712266s) [noise=true , address=45.91.116.13:4154 , piece_id=FE4NFODQ3TWGFXARFYIL64TKK6X7H6PPLIG6LO44LKNAZOEU7DJA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.296272429 (582h31m17.167355474s) [noise=true , address=45.91.116.10:4084 , piece_id=5BFUYVGIFOYXC6RCPU7VBUM36LSM32TVBLS3W54OWFNR4SM6GCSA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.296403588 (582h31m17.167486633s) [noise=true , address=77.90.31.19:4844 , piece_id=CDOWECXLSRPACV4HMAH57UK4EEIYNOMH6GYV6HFHV7CWSEAJ44FA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.296646560 (582h31m17.167729596s) [noise=true , address=77.90.31.14:4661 , piece_id=B7IDSCDHBUXUIPSUT3HMQSCMPGBMYASZSQO2EJBXRR2DQHCLWGYA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.296906235 (582h31m17.167989291s) [noise=true , address=145.79.227.47:4265 , piece_id=FYZA2LRIFKLKVUCUAWMNWTUH3VYR5VFAA7XKAYIMD2E2VOJTI7PA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.296989804 (582h31m17.168072849s) [noise=true , address=145.79.227.46:4601 , piece_id=YJTMZID24VBM5ZZSUUHOCZSZSSNGTFHOSKK6OI5QP3A4G6ZWVWQA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.297235221 (582h31m17.168318267s) [noise=true , address=145.79.227.3:4150 , piece_id=HS4S5UDYD2VWI2N4MH6EWEYQ5G6KCFCJBDZR7AVWPXRYALFXP4FQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.297424432 (582h31m17.168507467s) [noise=true , address=145.79.227.13:4803 , piece_id=YIEUZEDLSC5C67SCFEPCHHZSTYP6MYLYJZJTGZF3UP6JWDTGUOPQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.297606900 (582h31m17.168689945s) [noise=true , address=45.91.116.14:4190 , piece_id=FIL6AIXGE7DE2M4QBVSIO53KGY43VFPZQ2FTXSWJFPJU4VZCKUXQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:18.995364155 (582h31m18.866447201s) [upload=510642]: Next returned nil segment Dec 29 22:00:18.995383603 (582h31m18.866466648s) [upload=510642]: Waiting for error group managing segments... Dec 29 22:00:19.284009408 (582h31m19.155092453s) [noise=true , address=145.79.227.51:4411 , piece_id=HTZQ6SW6KSACLQF7G4CFGQA572J5JVHMLGEWQYSSGZRYX7FJ6DXA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.295985232 (582h31m19.167068277s) [noise=true , address=145.79.227.13:4803 , piece_id=YIEUZEDLSC5C67SCFEPCHHZSTYP6MYLYJZJTGZF3UP6JWDTGUOPQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.296135458 (582h31m19.167218503s) [noise=true , address=145.79.227.16:4909 , piece_id=CF7WHNKC4LFJQRVFBPFV62JGEBLD4KVPE6Q5JBWQVADC6ZAFFLVQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.310119916 (582h31m19.181202971s) [noise=true , address=145.79.227.46:4601 , piece_id=YJTMZID24VBM5ZZSUUHOCZSZSSNGTFHOSKK6OI5QP3A4G6ZWVWQA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.310998939 (582h31m19.182081984s) [noise=true , address=145.79.227.31:4797 , piece_id=NI3M5NVVVX5HUXZ2SOMQD4S5ZOSSNPCD2F3Z5X2BA2VXMGT57AYQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.318707138 (582h31m19.189790183s) [noise=true , address=145.79.227.47:4265 , piece_id=FYZA2LRIFKLKVUCUAWMNWTUH3VYR5VFAA7XKAYIMD2E2VOJTI7PA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.322059381 (582h31m19.193142426s) [noise=true , address=145.79.227.5:4221 , piece_id=LD3FENLBM2L5N7QDOAMAYJUVH54QVSJQJFP7E36UAH2ZD7R65CCQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.506372108 (582h31m19.377455153s) [noise=true , address=77.90.31.19:4844 , piece_id=CDOWECXLSRPACV4HMAH57UK4EEIYNOMH6GYV6HFHV7CWSEAJ44FA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.507172602 (582h31m19.378255648s) [noise=true , address=77.90.31.11:4553 , piece_id=3Y5HSD73DLZ6ALJ5TFRIVTLOE4ZNZ256YSZ2W3X5LPKL5J5LX2EQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.527548842 (582h31m19.398631887s) [noise=true , address=77.90.31.14:4661 , piece_id=B7IDSCDHBUXUIPSUT3HMQSCMPGBMYASZSQO2EJBXRR2DQHCLWGYA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.531443508 (582h31m19.402526553s) [noise=true , address=45.91.116.5:4297 , piece_id=PLUQ7Q32BTGYFQRGVJFAJWHPZCPQTC5RX5UX27S5HJ5NNJPLDMPQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.552364906 (582h31m19.423447951s) [noise=true , address=45.91.116.14:4190 , piece_id=FIL6AIXGE7DE2M4QBVSIO53KGY43VFPZQ2FTXSWJFPJU4VZCKUXQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.564056769 (582h31m19.435139814s) [noise=true , address=45.91.116.10:4084 , piece_id=5BFUYVGIFOYXC6RCPU7VBUM36LSM32TVBLS3W54OWFNR4SM6GCSA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.564087016 (582h31m19.435170051s) [seg_pos={0 0}, upload=510642]: Segment reached optimal threshold of 13 pieces. Dec 29 22:00:19.564294431 (582h31m19.435377466s) [noise=true , address=145.79.227.3:4150 , piece_id=HS4S5UDYD2VWI2N4MH6EWEYQ5G6KCFCJBDZR7AVWPXRYALFXP4FQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK): protocol: expected piece hash; context canceled; EOF Dec 29 22:00:19.564302366 (582h31m19.435385411s) [noise=true , address=145.79.227.3:4150 , piece_id=HS4S5UDYD2VWI2N4MH6EWEYQ5G6KCFCJBDZR7AVWPXRYALFXP4FQ, seg_pos={0 0}, upload=510642]: Long tail cancellation due to optimal threshold Dec 29 22:00:19.564388932 (582h31m19.435471977s) [noise=true , address=45.91.116.13:4154 , piece_id=FE4NFODQ3TWGFXARFYIL64TKK6X7H6PPLIG6LO44LKNAZOEU7DJA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Dec 29 22:00:19.564401876 (582h31m19.435484911s) [noise=true , address=45.91.116.13:4154 , piece_id=FE4NFODQ3TWGFXARFYIL64TKK6X7H6PPLIG6LO44LKNAZOEU7DJA, seg_pos={0 0}, upload=510642]: Long tail cancellation due to optimal threshold Dec 29 22:00:19.564558945 (582h31m19.435641981s) [noise=true , address=145.79.227.2:4115 , piece_id=P2AS5B3VYO2K2AWSM5W4BXAXBHSXQQWM7JOXFPOV64NTKDFLT62Q, seg_pos={0 0}, upload=510642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: write tcp 188.166.225.170:38246->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:38246->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38246->145.79.227.2:4115: use of closed network connection Dec 29 22:00:19.564570738 (582h31m19.435653783s) [noise=true , address=145.79.227.2:4115 , piece_id=P2AS5B3VYO2K2AWSM5W4BXAXBHSXQQWM7JOXFPOV64NTKDFLT62Q, seg_pos={0 0}, upload=510642]: Long tail cancellation due to optimal threshold Dec 29 22:00:19.564601426 (582h31m19.435684471s) [seg_pos={0 0}, upload=510642]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 22:00:19.564683973 (582h31m19.435767018s) [upload=510642]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 22:00:19.564690004 (582h31m19.435773049s) [upload=510642]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 22:00:19.871860472 (582h31m19.742943527s) [upload=510642]: Done uploading segments... Dec 29 22:00:19.871892413 (582h31m19.742975468s) [upload=510642]: Upload finished. err: <nil> Duration: 6.021534527s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................X............................X.............] duration: 29.847009854s trace: https://jaeger.dev.storj.io/trace/7096862780f29049 FAIL (2025-12-29 22:17:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== oise=true , address=145.79.227.30:4766 , piece_id=YWDMUGNTYVXWOCMSN63VIC6HOTWHPY7VJ7QWT76P63UAJIBS2XLA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.060493636 (582h37m18.931576671s) [noise=true , address=5.231.73.5:4022 , piece_id=5WVAJH3Y6TLATXY4IYB6ZADKKJDXDS52B2ZN6FJAZ7UHFVRYRRAA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.060685291 (582h37m18.931768326s) [noise=true , address=77.90.31.11:4579 , piece_id=DEJPYHGYNAP52CZ45BKGNXJM5MLIA2OQZ4BRKN5PBADNRVE3VLEA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.060749954 (582h37m18.931832989s) [noise=true , address=145.79.227.29:4976 , piece_id=YRCRXSSQOXPOJGVX46L7WCB3TRDZ5QB755SEOTNHSCHN4ZDMCOUQ, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.061033203 (582h37m18.932116248s) [noise=true , address=193.149.164.143:4698 , piece_id=6X5KB4TXFV4ZOF5JTPNRW4Q3GWNCVCXCA4JSB3AZURBHXY2VHTRQ, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.061248855 (582h37m18.9323319s) [noise=true , address=145.79.227.49:4347 , piece_id=75B3DRRHEPDL6EK6ZUXTKASXUKWGAVUWBPSTNRYT4BJMAU5ALXRA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.061382729 (582h37m18.932465764s) [noise=true , address=5.231.73.2:4004 , piece_id=PTHHO3XEPGGCIRCUBIWHN4LVNBLMISJM5LESQGR2ERRESFK3S4RA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.061645560 (582h37m18.932728595s) [noise=true , address=5.231.73.3:4007 , piece_id=6AF3CEPLUMRO4CDW3SRWZSYHWEXP35AVSM5M3LH4KCRARSZ6NFNA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.061769094 (582h37m18.932852139s) [noise=true , address=145.79.227.7:4321 , piece_id=SPWPZ5MR5ZCQNYVYYQQEZC4OURAM5VAH3WN5UP6HMGCCDMCH6ZGA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.061970298 (582h37m18.933053333s) [noise=true , address=145.79.227.35:4507 , piece_id=ZZSVIHOAN2SWXI4NS3TRD4B64NDB4D43J63DAS74TCPEE6MPGRXA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.062237577 (582h37m18.933320612s) [noise=true , address=145.79.227.35:4489 , piece_id=BM7ZQESTHEUIRJWAPORTKDMT5TVLEXMOHULGFVEQKHT4AJV3HT5A, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.062304304 (582h37m18.933387349s) [noise=true , address=145.79.227.34:4931 , piece_id=XJHLE3B5HIF4RC4LV32SBCNMVTOTEXWBBCXBVPQFN6N5HFQ6ZS7A, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.062522640 (582h37m18.933605685s) [noise=true , address=145.79.227.48:4329 , piece_id=YWCVTYLCAGU57SE3ID3WW2JLCJGTJ7CPDRCZENAMO42BVPIAVLIQ, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.062904968 (582h37m18.933988003s) [noise=true , address=77.90.31.17:4801 , piece_id=K3QD7FPVRQHZL5R57EKKAC27RGDJQH6HDCND7FIVXSZBRH42WSIA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.063420419 (582h37m18.934503464s) [noise=true , address=145.79.227.17:4995 , piece_id=JJTVPVBAMCTDGHGWNPUVPYFVKD4SMTDAHF4VWVIW7JFUAMBIPRHQ, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.063943495 (582h37m18.93502654s) [noise=true , address=5.231.73.4:4018 , piece_id=X45H47HER27IM5FEQSEAMXSDX7UQG3AEOY65Q4WCCUOX7SVFZ5KA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:20.386418430 (582h37m20.257501465s) [upload=510734]: Next returned nil segment Dec 29 22:06:20.386428990 (582h37m20.257512025s) [upload=510734]: Waiting for error group managing segments... Dec 29 22:06:20.935543714 (582h37m20.806626749s) [noise=true , address=145.79.227.48:4329 , piece_id=YWCVTYLCAGU57SE3ID3WW2JLCJGTJ7CPDRCZENAMO42BVPIAVLIQ, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:20.978073277 (582h37m20.849156312s) [noise=true , address=145.79.227.39:4022 , piece_id=5IJCSVRVXCICRNRB3TTJAVNRF5FTUQMMY4KL7ZFF4SBKILFGHJWA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.079004013 (582h37m20.950087058s) [noise=true , address=5.231.73.4:4018 , piece_id=X45H47HER27IM5FEQSEAMXSDX7UQG3AEOY65Q4WCCUOX7SVFZ5KA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.101144552 (582h37m20.972227597s) [noise=true , address=77.90.31.11:4579 , piece_id=DEJPYHGYNAP52CZ45BKGNXJM5MLIA2OQZ4BRKN5PBADNRVE3VLEA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.138766369 (582h37m21.009849405s) [noise=true , address=77.90.31.17:4801 , piece_id=K3QD7FPVRQHZL5R57EKKAC27RGDJQH6HDCND7FIVXSZBRH42WSIA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.231047035 (582h37m21.10213007s) [noise=true , address=77.90.31.19:4857 , piece_id=DMJ4O3TWZYWKWVLJFSQBULHDFWZCNLVDBIGYJEOBAVVCU5UGG27Q, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.272062706 (582h37m21.143145741s) [noise=true , address=5.231.73.3:4007 , piece_id=6AF3CEPLUMRO4CDW3SRWZSYHWEXP35AVSM5M3LH4KCRARSZ6NFNA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.303520666 (582h37m21.174603711s) [noise=true , address=145.79.227.29:4976 , piece_id=YRCRXSSQOXPOJGVX46L7WCB3TRDZ5QB755SEOTNHSCHN4ZDMCOUQ, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.340640177 (582h37m21.211723212s) [noise=true , address=5.231.73.5:4022 , piece_id=5WVAJH3Y6TLATXY4IYB6ZADKKJDXDS52B2ZN6FJAZ7UHFVRYRRAA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.481118228 (582h37m21.352201263s) [noise=true , address=5.231.73.2:4004 , piece_id=PTHHO3XEPGGCIRCUBIWHN4LVNBLMISJM5LESQGR2ERRESFK3S4RA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.896953769 (582h37m21.768036804s) [noise=true , address=145.79.227.49:4347 , piece_id=75B3DRRHEPDL6EK6ZUXTKASXUKWGAVUWBPSTNRYT4BJMAU5ALXRA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.911314353 (582h37m21.782397399s) [noise=true , address=145.79.227.34:4931 , piece_id=XJHLE3B5HIF4RC4LV32SBCNMVTOTEXWBBCXBVPQFN6N5HFQ6ZS7A, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.915845371 (582h37m21.786928406s) [noise=true , address=145.79.227.30:4766 , piece_id=YWDMUGNTYVXWOCMSN63VIC6HOTWHPY7VJ7QWT76P63UAJIBS2XLA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.928785972 (582h37m21.799869017s) [noise=true , address=145.79.227.35:4489 , piece_id=BM7ZQESTHEUIRJWAPORTKDMT5TVLEXMOHULGFVEQKHT4AJV3HT5A, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.928810859 (582h37m21.799893904s) [seg_pos={0 0}, upload=510734]: Segment reached optimal threshold of 14 pieces. Dec 29 22:06:21.929016471 (582h37m21.800099516s) [noise=true , address=145.79.227.7:4321 , piece_id=SPWPZ5MR5ZCQNYVYYQQEZC4OURAM5VAH3WN5UP6HMGCCDMCH6ZGA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12edsHqD9iRBDgMU8vYhSM1EWG8Sn57GZ8TPiTytri3rp7W9Eae): protocol: expected piece hash; context canceled; EOF Dec 29 22:06:21.929028644 (582h37m21.800111709s) [noise=true , address=145.79.227.7:4321 , piece_id=SPWPZ5MR5ZCQNYVYYQQEZC4OURAM5VAH3WN5UP6HMGCCDMCH6ZGA, seg_pos={0 0}, upload=510734]: Long tail cancellation due to optimal threshold Dec 29 22:06:21.929093137 (582h37m21.800176192s) [noise=true , address=145.79.227.35:4507 , piece_id=ZZSVIHOAN2SWXI4NS3TRD4B64NDB4D43J63DAS74TCPEE6MPGRXA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14jurWWPpmxwiBS3XmZysEUwugsceYJqVXFj7BbYpx8UFcTR8R): protocol: expected piece hash; context canceled; EOF Dec 29 22:06:21.929106793 (582h37m21.800189838s) [noise=true , address=145.79.227.35:4507 , piece_id=ZZSVIHOAN2SWXI4NS3TRD4B64NDB4D43J63DAS74TCPEE6MPGRXA, seg_pos={0 0}, upload=510734]: Long tail cancellation due to optimal threshold Dec 29 22:06:21.929206612 (582h37m21.800289657s) [noise=true , address=145.79.227.17:4995 , piece_id=JJTVPVBAMCTDGHGWNPUVPYFVKD4SMTDAHF4VWVIW7JFUAMBIPRHQ, seg_pos={0 0}, upload=510734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tvoTD1LoSj3q2rLiDckfT56dXNKKwdVu1ZAsBZExNFVTom9tH): protocol: write tcp 188.166.225.170:47696->145.79.227.17:4995: use of closed network connection; write tcp 188.166.225.170:47696->145.79.227.17:4995: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47696->145.79.227.17:4995: use of closed network connection Dec 29 22:06:21.929215911 (582h37m21.800298976s) [noise=true , address=145.79.227.17:4995 , piece_id=JJTVPVBAMCTDGHGWNPUVPYFVKD4SMTDAHF4VWVIW7JFUAMBIPRHQ, seg_pos={0 0}, upload=510734]: Long tail cancellation due to optimal threshold Dec 29 22:06:21.929387247 (582h37m21.800470292s) [noise=true , address=193.149.164.143:4698 , piece_id=6X5KB4TXFV4ZOF5JTPNRW4Q3GWNCVCXCA4JSB3AZURBHXY2VHTRQ, seg_pos={0 0}, upload=510734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F4fJAGJM2ZPRt7vJH2cZmEbYeujmxcfi7aPZ2g9VPTNYBYRAi): protocol: expected piece hash; context canceled; EOF Dec 29 22:06:21.929396174 (582h37m21.800479219s) [noise=true , address=193.149.164.143:4698 , piece_id=6X5KB4TXFV4ZOF5JTPNRW4Q3GWNCVCXCA4JSB3AZURBHXY2VHTRQ, seg_pos={0 0}, upload=510734]: Long tail cancellation due to optimal threshold Dec 29 22:06:21.929491536 (582h37m21.800574571s) [seg_pos={0 0}, upload=510734]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 22:06:21.929573792 (582h37m21.800656837s) [upload=510734]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 22:06:21.929580555 (582h37m21.8006636s) [upload=510734]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 22:06:22.233045366 (582h37m22.104128401s) [upload=510734]: Done uploading segments... Dec 29 22:06:22.233064963 (582h37m22.104147998s) [upload=510734]: Upload finished. err: <nil> Duration: 29.847036935s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......X............................X.......................] duration: 5.840203173s trace: https://jaeger.dev.storj.io/trace/5209b3d035311fd FAIL (2025-12-29 22:17:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :00:17.043136250 (582h31m16.914219305s) [upload=510642]: Starting upload Dec 29 22:00:17.043227904 (582h31m16.91431096s) [upload=510642]: Done starting upload Dec 29 22:00:17.043335189 (582h31m16.914418234s) [upload=510642]: Uploading segments... Dec 29 22:00:17.043587749 (582h31m16.914670824s) [upload=510642]: Got next segment. Inline: false Dec 29 22:00:17.043644478 (582h31m16.914727523s) [upload=510642]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 22:00:17.043686427 (582h31m16.914769503s) [upload=510642]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 22:00:17.294384363 (582h31m17.165467408s) [seg_pos={0 0}, upload=510642]: Begin upload segment... Dec 29 22:00:17.294445219 (582h31m17.165528264s) [seg_pos={0 0}, upload=510642]: Done begin upload segment. Dec 29 22:00:17.294484133 (582h31m17.165567178s) [noise=true , address=145.79.227.31:4797 , piece_id=NI3M5NVVVX5HUXZ2SOMQD4S5ZOSSNPCD2F3Z5X2BA2VXMGT57AYQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.294560979 (582h31m17.165644014s) [noise=true , address=77.90.31.11:4553 , piece_id=3Y5HSD73DLZ6ALJ5TFRIVTLOE4ZNZ256YSZ2W3X5LPKL5J5LX2EQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.295149941 (582h31m17.166232976s) [noise=true , address=45.91.116.5:4297 , piece_id=PLUQ7Q32BTGYFQRGVJFAJWHPZCPQTC5RX5UX27S5HJ5NNJPLDMPQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.295419574 (582h31m17.166502609s) [noise=true , address=145.79.227.2:4115 , piece_id=P2AS5B3VYO2K2AWSM5W4BXAXBHSXQQWM7JOXFPOV64NTKDFLT62Q, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.295483997 (582h31m17.166567042s) [noise=true , address=145.79.227.51:4411 , piece_id=HTZQ6SW6KSACLQF7G4CFGQA572J5JVHMLGEWQYSSGZRYX7FJ6DXA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.295762207 (582h31m17.166845252s) [noise=true , address=145.79.227.5:4221 , piece_id=LD3FENLBM2L5N7QDOAMAYJUVH54QVSJQJFP7E36UAH2ZD7R65CCQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.295928003 (582h31m17.167011048s) [noise=true , address=145.79.227.16:4909 , piece_id=CF7WHNKC4LFJQRVFBPFV62JGEBLD4KVPE6Q5JBWQVADC6ZAFFLVQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.296039615 (582h31m17.16712266s) [noise=true , address=45.91.116.13:4154 , piece_id=FE4NFODQ3TWGFXARFYIL64TKK6X7H6PPLIG6LO44LKNAZOEU7DJA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.296272429 (582h31m17.167355474s) [noise=true , address=45.91.116.10:4084 , piece_id=5BFUYVGIFOYXC6RCPU7VBUM36LSM32TVBLS3W54OWFNR4SM6GCSA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.296403588 (582h31m17.167486633s) [noise=true , address=77.90.31.19:4844 , piece_id=CDOWECXLSRPACV4HMAH57UK4EEIYNOMH6GYV6HFHV7CWSEAJ44FA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.296646560 (582h31m17.167729596s) [noise=true , address=77.90.31.14:4661 , piece_id=B7IDSCDHBUXUIPSUT3HMQSCMPGBMYASZSQO2EJBXRR2DQHCLWGYA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.296906235 (582h31m17.167989291s) [noise=true , address=145.79.227.47:4265 , piece_id=FYZA2LRIFKLKVUCUAWMNWTUH3VYR5VFAA7XKAYIMD2E2VOJTI7PA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.296989804 (582h31m17.168072849s) [noise=true , address=145.79.227.46:4601 , piece_id=YJTMZID24VBM5ZZSUUHOCZSZSSNGTFHOSKK6OI5QP3A4G6ZWVWQA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.297235221 (582h31m17.168318267s) [noise=true , address=145.79.227.3:4150 , piece_id=HS4S5UDYD2VWI2N4MH6EWEYQ5G6KCFCJBDZR7AVWPXRYALFXP4FQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.297424432 (582h31m17.168507467s) [noise=true , address=145.79.227.13:4803 , piece_id=YIEUZEDLSC5C67SCFEPCHHZSTYP6MYLYJZJTGZF3UP6JWDTGUOPQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.297606900 (582h31m17.168689945s) [noise=true , address=45.91.116.14:4190 , piece_id=FIL6AIXGE7DE2M4QBVSIO53KGY43VFPZQ2FTXSWJFPJU4VZCKUXQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:18.995364155 (582h31m18.866447201s) [upload=510642]: Next returned nil segment Dec 29 22:00:18.995383603 (582h31m18.866466648s) [upload=510642]: Waiting for error group managing segments... Dec 29 22:00:19.284009408 (582h31m19.155092453s) [noise=true , address=145.79.227.51:4411 , piece_id=HTZQ6SW6KSACLQF7G4CFGQA572J5JVHMLGEWQYSSGZRYX7FJ6DXA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.295985232 (582h31m19.167068277s) [noise=true , address=145.79.227.13:4803 , piece_id=YIEUZEDLSC5C67SCFEPCHHZSTYP6MYLYJZJTGZF3UP6JWDTGUOPQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.296135458 (582h31m19.167218503s) [noise=true , address=145.79.227.16:4909 , piece_id=CF7WHNKC4LFJQRVFBPFV62JGEBLD4KVPE6Q5JBWQVADC6ZAFFLVQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.310119916 (582h31m19.181202971s) [noise=true , address=145.79.227.46:4601 , piece_id=YJTMZID24VBM5ZZSUUHOCZSZSSNGTFHOSKK6OI5QP3A4G6ZWVWQA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.310998939 (582h31m19.182081984s) [noise=true , address=145.79.227.31:4797 , piece_id=NI3M5NVVVX5HUXZ2SOMQD4S5ZOSSNPCD2F3Z5X2BA2VXMGT57AYQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.318707138 (582h31m19.189790183s) [noise=true , address=145.79.227.47:4265 , piece_id=FYZA2LRIFKLKVUCUAWMNWTUH3VYR5VFAA7XKAYIMD2E2VOJTI7PA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.322059381 (582h31m19.193142426s) [noise=true , address=145.79.227.5:4221 , piece_id=LD3FENLBM2L5N7QDOAMAYJUVH54QVSJQJFP7E36UAH2ZD7R65CCQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.506372108 (582h31m19.377455153s) [noise=true , address=77.90.31.19:4844 , piece_id=CDOWECXLSRPACV4HMAH57UK4EEIYNOMH6GYV6HFHV7CWSEAJ44FA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.507172602 (582h31m19.378255648s) [noise=true , address=77.90.31.11:4553 , piece_id=3Y5HSD73DLZ6ALJ5TFRIVTLOE4ZNZ256YSZ2W3X5LPKL5J5LX2EQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.527548842 (582h31m19.398631887s) [noise=true , address=77.90.31.14:4661 , piece_id=B7IDSCDHBUXUIPSUT3HMQSCMPGBMYASZSQO2EJBXRR2DQHCLWGYA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.531443508 (582h31m19.402526553s) [noise=true , address=45.91.116.5:4297 , piece_id=PLUQ7Q32BTGYFQRGVJFAJWHPZCPQTC5RX5UX27S5HJ5NNJPLDMPQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.552364906 (582h31m19.423447951s) [noise=true , address=45.91.116.14:4190 , piece_id=FIL6AIXGE7DE2M4QBVSIO53KGY43VFPZQ2FTXSWJFPJU4VZCKUXQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.564056769 (582h31m19.435139814s) [noise=true , address=45.91.116.10:4084 , piece_id=5BFUYVGIFOYXC6RCPU7VBUM36LSM32TVBLS3W54OWFNR4SM6GCSA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.564087016 (582h31m19.435170051s) [seg_pos={0 0}, upload=510642]: Segment reached optimal threshold of 13 pieces. Dec 29 22:00:19.564294431 (582h31m19.435377466s) [noise=true , address=145.79.227.3:4150 , piece_id=HS4S5UDYD2VWI2N4MH6EWEYQ5G6KCFCJBDZR7AVWPXRYALFXP4FQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK): protocol: expected piece hash; context canceled; EOF Dec 29 22:00:19.564302366 (582h31m19.435385411s) [noise=true , address=145.79.227.3:4150 , piece_id=HS4S5UDYD2VWI2N4MH6EWEYQ5G6KCFCJBDZR7AVWPXRYALFXP4FQ, seg_pos={0 0}, upload=510642]: Long tail cancellation due to optimal threshold Dec 29 22:00:19.564388932 (582h31m19.435471977s) [noise=true , address=45.91.116.13:4154 , piece_id=FE4NFODQ3TWGFXARFYIL64TKK6X7H6PPLIG6LO44LKNAZOEU7DJA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Dec 29 22:00:19.564401876 (582h31m19.435484911s) [noise=true , address=45.91.116.13:4154 , piece_id=FE4NFODQ3TWGFXARFYIL64TKK6X7H6PPLIG6LO44LKNAZOEU7DJA, seg_pos={0 0}, upload=510642]: Long tail cancellation due to optimal threshold Dec 29 22:00:19.564558945 (582h31m19.435641981s) [noise=true , address=145.79.227.2:4115 , piece_id=P2AS5B3VYO2K2AWSM5W4BXAXBHSXQQWM7JOXFPOV64NTKDFLT62Q, seg_pos={0 0}, upload=510642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: write tcp 188.166.225.170:38246->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:38246->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38246->145.79.227.2:4115: use of closed network connection Dec 29 22:00:19.564570738 (582h31m19.435653783s) [noise=true , address=145.79.227.2:4115 , piece_id=P2AS5B3VYO2K2AWSM5W4BXAXBHSXQQWM7JOXFPOV64NTKDFLT62Q, seg_pos={0 0}, upload=510642]: Long tail cancellation due to optimal threshold Dec 29 22:00:19.564601426 (582h31m19.435684471s) [seg_pos={0 0}, upload=510642]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 22:00:19.564683973 (582h31m19.435767018s) [upload=510642]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 22:00:19.564690004 (582h31m19.435773049s) [upload=510642]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 22:00:19.871860472 (582h31m19.742943527s) [upload=510642]: Done uploading segments... Dec 29 22:00:19.871892413 (582h31m19.742975468s) [upload=510642]: Upload finished. err: <nil> Duration: 5.84022895s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................X............................X............] duration: 15.487777007s trace: https://jaeger.dev.storj.io/trace/7096862780f29049 FAIL (2025-12-29 22:17:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== oise=true , address=145.79.227.30:4766 , piece_id=YWDMUGNTYVXWOCMSN63VIC6HOTWHPY7VJ7QWT76P63UAJIBS2XLA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.060493636 (582h37m18.931576671s) [noise=true , address=5.231.73.5:4022 , piece_id=5WVAJH3Y6TLATXY4IYB6ZADKKJDXDS52B2ZN6FJAZ7UHFVRYRRAA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.060685291 (582h37m18.931768326s) [noise=true , address=77.90.31.11:4579 , piece_id=DEJPYHGYNAP52CZ45BKGNXJM5MLIA2OQZ4BRKN5PBADNRVE3VLEA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.060749954 (582h37m18.931832989s) [noise=true , address=145.79.227.29:4976 , piece_id=YRCRXSSQOXPOJGVX46L7WCB3TRDZ5QB755SEOTNHSCHN4ZDMCOUQ, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.061033203 (582h37m18.932116248s) [noise=true , address=193.149.164.143:4698 , piece_id=6X5KB4TXFV4ZOF5JTPNRW4Q3GWNCVCXCA4JSB3AZURBHXY2VHTRQ, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.061248855 (582h37m18.9323319s) [noise=true , address=145.79.227.49:4347 , piece_id=75B3DRRHEPDL6EK6ZUXTKASXUKWGAVUWBPSTNRYT4BJMAU5ALXRA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.061382729 (582h37m18.932465764s) [noise=true , address=5.231.73.2:4004 , piece_id=PTHHO3XEPGGCIRCUBIWHN4LVNBLMISJM5LESQGR2ERRESFK3S4RA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.061645560 (582h37m18.932728595s) [noise=true , address=5.231.73.3:4007 , piece_id=6AF3CEPLUMRO4CDW3SRWZSYHWEXP35AVSM5M3LH4KCRARSZ6NFNA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.061769094 (582h37m18.932852139s) [noise=true , address=145.79.227.7:4321 , piece_id=SPWPZ5MR5ZCQNYVYYQQEZC4OURAM5VAH3WN5UP6HMGCCDMCH6ZGA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.061970298 (582h37m18.933053333s) [noise=true , address=145.79.227.35:4507 , piece_id=ZZSVIHOAN2SWXI4NS3TRD4B64NDB4D43J63DAS74TCPEE6MPGRXA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.062237577 (582h37m18.933320612s) [noise=true , address=145.79.227.35:4489 , piece_id=BM7ZQESTHEUIRJWAPORTKDMT5TVLEXMOHULGFVEQKHT4AJV3HT5A, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.062304304 (582h37m18.933387349s) [noise=true , address=145.79.227.34:4931 , piece_id=XJHLE3B5HIF4RC4LV32SBCNMVTOTEXWBBCXBVPQFN6N5HFQ6ZS7A, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.062522640 (582h37m18.933605685s) [noise=true , address=145.79.227.48:4329 , piece_id=YWCVTYLCAGU57SE3ID3WW2JLCJGTJ7CPDRCZENAMO42BVPIAVLIQ, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.062904968 (582h37m18.933988003s) [noise=true , address=77.90.31.17:4801 , piece_id=K3QD7FPVRQHZL5R57EKKAC27RGDJQH6HDCND7FIVXSZBRH42WSIA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.063420419 (582h37m18.934503464s) [noise=true , address=145.79.227.17:4995 , piece_id=JJTVPVBAMCTDGHGWNPUVPYFVKD4SMTDAHF4VWVIW7JFUAMBIPRHQ, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.063943495 (582h37m18.93502654s) [noise=true , address=5.231.73.4:4018 , piece_id=X45H47HER27IM5FEQSEAMXSDX7UQG3AEOY65Q4WCCUOX7SVFZ5KA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:20.386418430 (582h37m20.257501465s) [upload=510734]: Next returned nil segment Dec 29 22:06:20.386428990 (582h37m20.257512025s) [upload=510734]: Waiting for error group managing segments... Dec 29 22:06:20.935543714 (582h37m20.806626749s) [noise=true , address=145.79.227.48:4329 , piece_id=YWCVTYLCAGU57SE3ID3WW2JLCJGTJ7CPDRCZENAMO42BVPIAVLIQ, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:20.978073277 (582h37m20.849156312s) [noise=true , address=145.79.227.39:4022 , piece_id=5IJCSVRVXCICRNRB3TTJAVNRF5FTUQMMY4KL7ZFF4SBKILFGHJWA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.079004013 (582h37m20.950087058s) [noise=true , address=5.231.73.4:4018 , piece_id=X45H47HER27IM5FEQSEAMXSDX7UQG3AEOY65Q4WCCUOX7SVFZ5KA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.101144552 (582h37m20.972227597s) [noise=true , address=77.90.31.11:4579 , piece_id=DEJPYHGYNAP52CZ45BKGNXJM5MLIA2OQZ4BRKN5PBADNRVE3VLEA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.138766369 (582h37m21.009849405s) [noise=true , address=77.90.31.17:4801 , piece_id=K3QD7FPVRQHZL5R57EKKAC27RGDJQH6HDCND7FIVXSZBRH42WSIA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.231047035 (582h37m21.10213007s) [noise=true , address=77.90.31.19:4857 , piece_id=DMJ4O3TWZYWKWVLJFSQBULHDFWZCNLVDBIGYJEOBAVVCU5UGG27Q, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.272062706 (582h37m21.143145741s) [noise=true , address=5.231.73.3:4007 , piece_id=6AF3CEPLUMRO4CDW3SRWZSYHWEXP35AVSM5M3LH4KCRARSZ6NFNA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.303520666 (582h37m21.174603711s) [noise=true , address=145.79.227.29:4976 , piece_id=YRCRXSSQOXPOJGVX46L7WCB3TRDZ5QB755SEOTNHSCHN4ZDMCOUQ, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.340640177 (582h37m21.211723212s) [noise=true , address=5.231.73.5:4022 , piece_id=5WVAJH3Y6TLATXY4IYB6ZADKKJDXDS52B2ZN6FJAZ7UHFVRYRRAA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.481118228 (582h37m21.352201263s) [noise=true , address=5.231.73.2:4004 , piece_id=PTHHO3XEPGGCIRCUBIWHN4LVNBLMISJM5LESQGR2ERRESFK3S4RA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.896953769 (582h37m21.768036804s) [noise=true , address=145.79.227.49:4347 , piece_id=75B3DRRHEPDL6EK6ZUXTKASXUKWGAVUWBPSTNRYT4BJMAU5ALXRA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.911314353 (582h37m21.782397399s) [noise=true , address=145.79.227.34:4931 , piece_id=XJHLE3B5HIF4RC4LV32SBCNMVTOTEXWBBCXBVPQFN6N5HFQ6ZS7A, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.915845371 (582h37m21.786928406s) [noise=true , address=145.79.227.30:4766 , piece_id=YWDMUGNTYVXWOCMSN63VIC6HOTWHPY7VJ7QWT76P63UAJIBS2XLA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.928785972 (582h37m21.799869017s) [noise=true , address=145.79.227.35:4489 , piece_id=BM7ZQESTHEUIRJWAPORTKDMT5TVLEXMOHULGFVEQKHT4AJV3HT5A, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.928810859 (582h37m21.799893904s) [seg_pos={0 0}, upload=510734]: Segment reached optimal threshold of 14 pieces. Dec 29 22:06:21.929016471 (582h37m21.800099516s) [noise=true , address=145.79.227.7:4321 , piece_id=SPWPZ5MR5ZCQNYVYYQQEZC4OURAM5VAH3WN5UP6HMGCCDMCH6ZGA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12edsHqD9iRBDgMU8vYhSM1EWG8Sn57GZ8TPiTytri3rp7W9Eae): protocol: expected piece hash; context canceled; EOF Dec 29 22:06:21.929028644 (582h37m21.800111709s) [noise=true , address=145.79.227.7:4321 , piece_id=SPWPZ5MR5ZCQNYVYYQQEZC4OURAM5VAH3WN5UP6HMGCCDMCH6ZGA, seg_pos={0 0}, upload=510734]: Long tail cancellation due to optimal threshold Dec 29 22:06:21.929093137 (582h37m21.800176192s) [noise=true , address=145.79.227.35:4507 , piece_id=ZZSVIHOAN2SWXI4NS3TRD4B64NDB4D43J63DAS74TCPEE6MPGRXA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14jurWWPpmxwiBS3XmZysEUwugsceYJqVXFj7BbYpx8UFcTR8R): protocol: expected piece hash; context canceled; EOF Dec 29 22:06:21.929106793 (582h37m21.800189838s) [noise=true , address=145.79.227.35:4507 , piece_id=ZZSVIHOAN2SWXI4NS3TRD4B64NDB4D43J63DAS74TCPEE6MPGRXA, seg_pos={0 0}, upload=510734]: Long tail cancellation due to optimal threshold Dec 29 22:06:21.929206612 (582h37m21.800289657s) [noise=true , address=145.79.227.17:4995 , piece_id=JJTVPVBAMCTDGHGWNPUVPYFVKD4SMTDAHF4VWVIW7JFUAMBIPRHQ, seg_pos={0 0}, upload=510734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tvoTD1LoSj3q2rLiDckfT56dXNKKwdVu1ZAsBZExNFVTom9tH): protocol: write tcp 188.166.225.170:47696->145.79.227.17:4995: use of closed network connection; write tcp 188.166.225.170:47696->145.79.227.17:4995: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47696->145.79.227.17:4995: use of closed network connection Dec 29 22:06:21.929215911 (582h37m21.800298976s) [noise=true , address=145.79.227.17:4995 , piece_id=JJTVPVBAMCTDGHGWNPUVPYFVKD4SMTDAHF4VWVIW7JFUAMBIPRHQ, seg_pos={0 0}, upload=510734]: Long tail cancellation due to optimal threshold Dec 29 22:06:21.929387247 (582h37m21.800470292s) [noise=true , address=193.149.164.143:4698 , piece_id=6X5KB4TXFV4ZOF5JTPNRW4Q3GWNCVCXCA4JSB3AZURBHXY2VHTRQ, seg_pos={0 0}, upload=510734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F4fJAGJM2ZPRt7vJH2cZmEbYeujmxcfi7aPZ2g9VPTNYBYRAi): protocol: expected piece hash; context canceled; EOF Dec 29 22:06:21.929396174 (582h37m21.800479219s) [noise=true , address=193.149.164.143:4698 , piece_id=6X5KB4TXFV4ZOF5JTPNRW4Q3GWNCVCXCA4JSB3AZURBHXY2VHTRQ, seg_pos={0 0}, upload=510734]: Long tail cancellation due to optimal threshold Dec 29 22:06:21.929491536 (582h37m21.800574571s) [seg_pos={0 0}, upload=510734]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 22:06:21.929573792 (582h37m21.800656837s) [upload=510734]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 22:06:21.929580555 (582h37m21.8006636s) [upload=510734]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 22:06:22.233045366 (582h37m22.104128401s) [upload=510734]: Done uploading segments... Dec 29 22:06:22.233064963 (582h37m22.104147998s) [upload=510734]: Upload finished. err: <nil> Duration: 15.487800421s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........X............................X......................] duration: 6.069115256s trace: https://jaeger.dev.storj.io/trace/5209b3d035311fd FAIL (2025-12-29 22:16:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :00:17.043136250 (582h31m16.914219305s) [upload=510642]: Starting upload Dec 29 22:00:17.043227904 (582h31m16.91431096s) [upload=510642]: Done starting upload Dec 29 22:00:17.043335189 (582h31m16.914418234s) [upload=510642]: Uploading segments... Dec 29 22:00:17.043587749 (582h31m16.914670824s) [upload=510642]: Got next segment. Inline: false Dec 29 22:00:17.043644478 (582h31m16.914727523s) [upload=510642]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 22:00:17.043686427 (582h31m16.914769503s) [upload=510642]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 22:00:17.294384363 (582h31m17.165467408s) [seg_pos={0 0}, upload=510642]: Begin upload segment... Dec 29 22:00:17.294445219 (582h31m17.165528264s) [seg_pos={0 0}, upload=510642]: Done begin upload segment. Dec 29 22:00:17.294484133 (582h31m17.165567178s) [noise=true , address=145.79.227.31:4797 , piece_id=NI3M5NVVVX5HUXZ2SOMQD4S5ZOSSNPCD2F3Z5X2BA2VXMGT57AYQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.294560979 (582h31m17.165644014s) [noise=true , address=77.90.31.11:4553 , piece_id=3Y5HSD73DLZ6ALJ5TFRIVTLOE4ZNZ256YSZ2W3X5LPKL5J5LX2EQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.295149941 (582h31m17.166232976s) [noise=true , address=45.91.116.5:4297 , piece_id=PLUQ7Q32BTGYFQRGVJFAJWHPZCPQTC5RX5UX27S5HJ5NNJPLDMPQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.295419574 (582h31m17.166502609s) [noise=true , address=145.79.227.2:4115 , piece_id=P2AS5B3VYO2K2AWSM5W4BXAXBHSXQQWM7JOXFPOV64NTKDFLT62Q, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.295483997 (582h31m17.166567042s) [noise=true , address=145.79.227.51:4411 , piece_id=HTZQ6SW6KSACLQF7G4CFGQA572J5JVHMLGEWQYSSGZRYX7FJ6DXA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.295762207 (582h31m17.166845252s) [noise=true , address=145.79.227.5:4221 , piece_id=LD3FENLBM2L5N7QDOAMAYJUVH54QVSJQJFP7E36UAH2ZD7R65CCQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.295928003 (582h31m17.167011048s) [noise=true , address=145.79.227.16:4909 , piece_id=CF7WHNKC4LFJQRVFBPFV62JGEBLD4KVPE6Q5JBWQVADC6ZAFFLVQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.296039615 (582h31m17.16712266s) [noise=true , address=45.91.116.13:4154 , piece_id=FE4NFODQ3TWGFXARFYIL64TKK6X7H6PPLIG6LO44LKNAZOEU7DJA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.296272429 (582h31m17.167355474s) [noise=true , address=45.91.116.10:4084 , piece_id=5BFUYVGIFOYXC6RCPU7VBUM36LSM32TVBLS3W54OWFNR4SM6GCSA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.296403588 (582h31m17.167486633s) [noise=true , address=77.90.31.19:4844 , piece_id=CDOWECXLSRPACV4HMAH57UK4EEIYNOMH6GYV6HFHV7CWSEAJ44FA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.296646560 (582h31m17.167729596s) [noise=true , address=77.90.31.14:4661 , piece_id=B7IDSCDHBUXUIPSUT3HMQSCMPGBMYASZSQO2EJBXRR2DQHCLWGYA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.296906235 (582h31m17.167989291s) [noise=true , address=145.79.227.47:4265 , piece_id=FYZA2LRIFKLKVUCUAWMNWTUH3VYR5VFAA7XKAYIMD2E2VOJTI7PA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.296989804 (582h31m17.168072849s) [noise=true , address=145.79.227.46:4601 , piece_id=YJTMZID24VBM5ZZSUUHOCZSZSSNGTFHOSKK6OI5QP3A4G6ZWVWQA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.297235221 (582h31m17.168318267s) [noise=true , address=145.79.227.3:4150 , piece_id=HS4S5UDYD2VWI2N4MH6EWEYQ5G6KCFCJBDZR7AVWPXRYALFXP4FQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.297424432 (582h31m17.168507467s) [noise=true , address=145.79.227.13:4803 , piece_id=YIEUZEDLSC5C67SCFEPCHHZSTYP6MYLYJZJTGZF3UP6JWDTGUOPQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.297606900 (582h31m17.168689945s) [noise=true , address=45.91.116.14:4190 , piece_id=FIL6AIXGE7DE2M4QBVSIO53KGY43VFPZQ2FTXSWJFPJU4VZCKUXQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:18.995364155 (582h31m18.866447201s) [upload=510642]: Next returned nil segment Dec 29 22:00:18.995383603 (582h31m18.866466648s) [upload=510642]: Waiting for error group managing segments... Dec 29 22:00:19.284009408 (582h31m19.155092453s) [noise=true , address=145.79.227.51:4411 , piece_id=HTZQ6SW6KSACLQF7G4CFGQA572J5JVHMLGEWQYSSGZRYX7FJ6DXA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.295985232 (582h31m19.167068277s) [noise=true , address=145.79.227.13:4803 , piece_id=YIEUZEDLSC5C67SCFEPCHHZSTYP6MYLYJZJTGZF3UP6JWDTGUOPQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.296135458 (582h31m19.167218503s) [noise=true , address=145.79.227.16:4909 , piece_id=CF7WHNKC4LFJQRVFBPFV62JGEBLD4KVPE6Q5JBWQVADC6ZAFFLVQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.310119916 (582h31m19.181202971s) [noise=true , address=145.79.227.46:4601 , piece_id=YJTMZID24VBM5ZZSUUHOCZSZSSNGTFHOSKK6OI5QP3A4G6ZWVWQA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.310998939 (582h31m19.182081984s) [noise=true , address=145.79.227.31:4797 , piece_id=NI3M5NVVVX5HUXZ2SOMQD4S5ZOSSNPCD2F3Z5X2BA2VXMGT57AYQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.318707138 (582h31m19.189790183s) [noise=true , address=145.79.227.47:4265 , piece_id=FYZA2LRIFKLKVUCUAWMNWTUH3VYR5VFAA7XKAYIMD2E2VOJTI7PA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.322059381 (582h31m19.193142426s) [noise=true , address=145.79.227.5:4221 , piece_id=LD3FENLBM2L5N7QDOAMAYJUVH54QVSJQJFP7E36UAH2ZD7R65CCQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.506372108 (582h31m19.377455153s) [noise=true , address=77.90.31.19:4844 , piece_id=CDOWECXLSRPACV4HMAH57UK4EEIYNOMH6GYV6HFHV7CWSEAJ44FA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.507172602 (582h31m19.378255648s) [noise=true , address=77.90.31.11:4553 , piece_id=3Y5HSD73DLZ6ALJ5TFRIVTLOE4ZNZ256YSZ2W3X5LPKL5J5LX2EQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.527548842 (582h31m19.398631887s) [noise=true , address=77.90.31.14:4661 , piece_id=B7IDSCDHBUXUIPSUT3HMQSCMPGBMYASZSQO2EJBXRR2DQHCLWGYA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.531443508 (582h31m19.402526553s) [noise=true , address=45.91.116.5:4297 , piece_id=PLUQ7Q32BTGYFQRGVJFAJWHPZCPQTC5RX5UX27S5HJ5NNJPLDMPQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.552364906 (582h31m19.423447951s) [noise=true , address=45.91.116.14:4190 , piece_id=FIL6AIXGE7DE2M4QBVSIO53KGY43VFPZQ2FTXSWJFPJU4VZCKUXQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.564056769 (582h31m19.435139814s) [noise=true , address=45.91.116.10:4084 , piece_id=5BFUYVGIFOYXC6RCPU7VBUM36LSM32TVBLS3W54OWFNR4SM6GCSA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.564087016 (582h31m19.435170051s) [seg_pos={0 0}, upload=510642]: Segment reached optimal threshold of 13 pieces. Dec 29 22:00:19.564294431 (582h31m19.435377466s) [noise=true , address=145.79.227.3:4150 , piece_id=HS4S5UDYD2VWI2N4MH6EWEYQ5G6KCFCJBDZR7AVWPXRYALFXP4FQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK): protocol: expected piece hash; context canceled; EOF Dec 29 22:00:19.564302366 (582h31m19.435385411s) [noise=true , address=145.79.227.3:4150 , piece_id=HS4S5UDYD2VWI2N4MH6EWEYQ5G6KCFCJBDZR7AVWPXRYALFXP4FQ, seg_pos={0 0}, upload=510642]: Long tail cancellation due to optimal threshold Dec 29 22:00:19.564388932 (582h31m19.435471977s) [noise=true , address=45.91.116.13:4154 , piece_id=FE4NFODQ3TWGFXARFYIL64TKK6X7H6PPLIG6LO44LKNAZOEU7DJA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Dec 29 22:00:19.564401876 (582h31m19.435484911s) [noise=true , address=45.91.116.13:4154 , piece_id=FE4NFODQ3TWGFXARFYIL64TKK6X7H6PPLIG6LO44LKNAZOEU7DJA, seg_pos={0 0}, upload=510642]: Long tail cancellation due to optimal threshold Dec 29 22:00:19.564558945 (582h31m19.435641981s) [noise=true , address=145.79.227.2:4115 , piece_id=P2AS5B3VYO2K2AWSM5W4BXAXBHSXQQWM7JOXFPOV64NTKDFLT62Q, seg_pos={0 0}, upload=510642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: write tcp 188.166.225.170:38246->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:38246->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38246->145.79.227.2:4115: use of closed network connection Dec 29 22:00:19.564570738 (582h31m19.435653783s) [noise=true , address=145.79.227.2:4115 , piece_id=P2AS5B3VYO2K2AWSM5W4BXAXBHSXQQWM7JOXFPOV64NTKDFLT62Q, seg_pos={0 0}, upload=510642]: Long tail cancellation due to optimal threshold Dec 29 22:00:19.564601426 (582h31m19.435684471s) [seg_pos={0 0}, upload=510642]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 22:00:19.564683973 (582h31m19.435767018s) [upload=510642]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 22:00:19.564690004 (582h31m19.435773049s) [upload=510642]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 22:00:19.871860472 (582h31m19.742943527s) [upload=510642]: Done uploading segments... Dec 29 22:00:19.871892413 (582h31m19.742975468s) [upload=510642]: Upload finished. err: <nil> Duration: 6.069139632s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [X........X............................X.....................] duration: 13.72064381s trace: https://jaeger.dev.storj.io/trace/5209b3d035311fd FAIL (2025-12-29 22:16:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :00:17.043136250 (582h31m16.914219305s) [upload=510642]: Starting upload Dec 29 22:00:17.043227904 (582h31m16.91431096s) [upload=510642]: Done starting upload Dec 29 22:00:17.043335189 (582h31m16.914418234s) [upload=510642]: Uploading segments... Dec 29 22:00:17.043587749 (582h31m16.914670824s) [upload=510642]: Got next segment. Inline: false Dec 29 22:00:17.043644478 (582h31m16.914727523s) [upload=510642]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 22:00:17.043686427 (582h31m16.914769503s) [upload=510642]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 22:00:17.294384363 (582h31m17.165467408s) [seg_pos={0 0}, upload=510642]: Begin upload segment... Dec 29 22:00:17.294445219 (582h31m17.165528264s) [seg_pos={0 0}, upload=510642]: Done begin upload segment. Dec 29 22:00:17.294484133 (582h31m17.165567178s) [noise=true , address=145.79.227.31:4797 , piece_id=NI3M5NVVVX5HUXZ2SOMQD4S5ZOSSNPCD2F3Z5X2BA2VXMGT57AYQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.294560979 (582h31m17.165644014s) [noise=true , address=77.90.31.11:4553 , piece_id=3Y5HSD73DLZ6ALJ5TFRIVTLOE4ZNZ256YSZ2W3X5LPKL5J5LX2EQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.295149941 (582h31m17.166232976s) [noise=true , address=45.91.116.5:4297 , piece_id=PLUQ7Q32BTGYFQRGVJFAJWHPZCPQTC5RX5UX27S5HJ5NNJPLDMPQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.295419574 (582h31m17.166502609s) [noise=true , address=145.79.227.2:4115 , piece_id=P2AS5B3VYO2K2AWSM5W4BXAXBHSXQQWM7JOXFPOV64NTKDFLT62Q, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.295483997 (582h31m17.166567042s) [noise=true , address=145.79.227.51:4411 , piece_id=HTZQ6SW6KSACLQF7G4CFGQA572J5JVHMLGEWQYSSGZRYX7FJ6DXA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.295762207 (582h31m17.166845252s) [noise=true , address=145.79.227.5:4221 , piece_id=LD3FENLBM2L5N7QDOAMAYJUVH54QVSJQJFP7E36UAH2ZD7R65CCQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.295928003 (582h31m17.167011048s) [noise=true , address=145.79.227.16:4909 , piece_id=CF7WHNKC4LFJQRVFBPFV62JGEBLD4KVPE6Q5JBWQVADC6ZAFFLVQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.296039615 (582h31m17.16712266s) [noise=true , address=45.91.116.13:4154 , piece_id=FE4NFODQ3TWGFXARFYIL64TKK6X7H6PPLIG6LO44LKNAZOEU7DJA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.296272429 (582h31m17.167355474s) [noise=true , address=45.91.116.10:4084 , piece_id=5BFUYVGIFOYXC6RCPU7VBUM36LSM32TVBLS3W54OWFNR4SM6GCSA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.296403588 (582h31m17.167486633s) [noise=true , address=77.90.31.19:4844 , piece_id=CDOWECXLSRPACV4HMAH57UK4EEIYNOMH6GYV6HFHV7CWSEAJ44FA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.296646560 (582h31m17.167729596s) [noise=true , address=77.90.31.14:4661 , piece_id=B7IDSCDHBUXUIPSUT3HMQSCMPGBMYASZSQO2EJBXRR2DQHCLWGYA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.296906235 (582h31m17.167989291s) [noise=true , address=145.79.227.47:4265 , piece_id=FYZA2LRIFKLKVUCUAWMNWTUH3VYR5VFAA7XKAYIMD2E2VOJTI7PA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.296989804 (582h31m17.168072849s) [noise=true , address=145.79.227.46:4601 , piece_id=YJTMZID24VBM5ZZSUUHOCZSZSSNGTFHOSKK6OI5QP3A4G6ZWVWQA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.297235221 (582h31m17.168318267s) [noise=true , address=145.79.227.3:4150 , piece_id=HS4S5UDYD2VWI2N4MH6EWEYQ5G6KCFCJBDZR7AVWPXRYALFXP4FQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.297424432 (582h31m17.168507467s) [noise=true , address=145.79.227.13:4803 , piece_id=YIEUZEDLSC5C67SCFEPCHHZSTYP6MYLYJZJTGZF3UP6JWDTGUOPQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.297606900 (582h31m17.168689945s) [noise=true , address=45.91.116.14:4190 , piece_id=FIL6AIXGE7DE2M4QBVSIO53KGY43VFPZQ2FTXSWJFPJU4VZCKUXQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:18.995364155 (582h31m18.866447201s) [upload=510642]: Next returned nil segment Dec 29 22:00:18.995383603 (582h31m18.866466648s) [upload=510642]: Waiting for error group managing segments... Dec 29 22:00:19.284009408 (582h31m19.155092453s) [noise=true , address=145.79.227.51:4411 , piece_id=HTZQ6SW6KSACLQF7G4CFGQA572J5JVHMLGEWQYSSGZRYX7FJ6DXA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.295985232 (582h31m19.167068277s) [noise=true , address=145.79.227.13:4803 , piece_id=YIEUZEDLSC5C67SCFEPCHHZSTYP6MYLYJZJTGZF3UP6JWDTGUOPQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.296135458 (582h31m19.167218503s) [noise=true , address=145.79.227.16:4909 , piece_id=CF7WHNKC4LFJQRVFBPFV62JGEBLD4KVPE6Q5JBWQVADC6ZAFFLVQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.310119916 (582h31m19.181202971s) [noise=true , address=145.79.227.46:4601 , piece_id=YJTMZID24VBM5ZZSUUHOCZSZSSNGTFHOSKK6OI5QP3A4G6ZWVWQA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.310998939 (582h31m19.182081984s) [noise=true , address=145.79.227.31:4797 , piece_id=NI3M5NVVVX5HUXZ2SOMQD4S5ZOSSNPCD2F3Z5X2BA2VXMGT57AYQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.318707138 (582h31m19.189790183s) [noise=true , address=145.79.227.47:4265 , piece_id=FYZA2LRIFKLKVUCUAWMNWTUH3VYR5VFAA7XKAYIMD2E2VOJTI7PA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.322059381 (582h31m19.193142426s) [noise=true , address=145.79.227.5:4221 , piece_id=LD3FENLBM2L5N7QDOAMAYJUVH54QVSJQJFP7E36UAH2ZD7R65CCQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.506372108 (582h31m19.377455153s) [noise=true , address=77.90.31.19:4844 , piece_id=CDOWECXLSRPACV4HMAH57UK4EEIYNOMH6GYV6HFHV7CWSEAJ44FA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.507172602 (582h31m19.378255648s) [noise=true , address=77.90.31.11:4553 , piece_id=3Y5HSD73DLZ6ALJ5TFRIVTLOE4ZNZ256YSZ2W3X5LPKL5J5LX2EQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.527548842 (582h31m19.398631887s) [noise=true , address=77.90.31.14:4661 , piece_id=B7IDSCDHBUXUIPSUT3HMQSCMPGBMYASZSQO2EJBXRR2DQHCLWGYA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.531443508 (582h31m19.402526553s) [noise=true , address=45.91.116.5:4297 , piece_id=PLUQ7Q32BTGYFQRGVJFAJWHPZCPQTC5RX5UX27S5HJ5NNJPLDMPQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.552364906 (582h31m19.423447951s) [noise=true , address=45.91.116.14:4190 , piece_id=FIL6AIXGE7DE2M4QBVSIO53KGY43VFPZQ2FTXSWJFPJU4VZCKUXQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.564056769 (582h31m19.435139814s) [noise=true , address=45.91.116.10:4084 , piece_id=5BFUYVGIFOYXC6RCPU7VBUM36LSM32TVBLS3W54OWFNR4SM6GCSA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.564087016 (582h31m19.435170051s) [seg_pos={0 0}, upload=510642]: Segment reached optimal threshold of 13 pieces. Dec 29 22:00:19.564294431 (582h31m19.435377466s) [noise=true , address=145.79.227.3:4150 , piece_id=HS4S5UDYD2VWI2N4MH6EWEYQ5G6KCFCJBDZR7AVWPXRYALFXP4FQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK): protocol: expected piece hash; context canceled; EOF Dec 29 22:00:19.564302366 (582h31m19.435385411s) [noise=true , address=145.79.227.3:4150 , piece_id=HS4S5UDYD2VWI2N4MH6EWEYQ5G6KCFCJBDZR7AVWPXRYALFXP4FQ, seg_pos={0 0}, upload=510642]: Long tail cancellation due to optimal threshold Dec 29 22:00:19.564388932 (582h31m19.435471977s) [noise=true , address=45.91.116.13:4154 , piece_id=FE4NFODQ3TWGFXARFYIL64TKK6X7H6PPLIG6LO44LKNAZOEU7DJA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Dec 29 22:00:19.564401876 (582h31m19.435484911s) [noise=true , address=45.91.116.13:4154 , piece_id=FE4NFODQ3TWGFXARFYIL64TKK6X7H6PPLIG6LO44LKNAZOEU7DJA, seg_pos={0 0}, upload=510642]: Long tail cancellation due to optimal threshold Dec 29 22:00:19.564558945 (582h31m19.435641981s) [noise=true , address=145.79.227.2:4115 , piece_id=P2AS5B3VYO2K2AWSM5W4BXAXBHSXQQWM7JOXFPOV64NTKDFLT62Q, seg_pos={0 0}, upload=510642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: write tcp 188.166.225.170:38246->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:38246->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38246->145.79.227.2:4115: use of closed network connection Dec 29 22:00:19.564570738 (582h31m19.435653783s) [noise=true , address=145.79.227.2:4115 , piece_id=P2AS5B3VYO2K2AWSM5W4BXAXBHSXQQWM7JOXFPOV64NTKDFLT62Q, seg_pos={0 0}, upload=510642]: Long tail cancellation due to optimal threshold Dec 29 22:00:19.564601426 (582h31m19.435684471s) [seg_pos={0 0}, upload=510642]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 22:00:19.564683973 (582h31m19.435767018s) [upload=510642]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 22:00:19.564690004 (582h31m19.435773049s) [upload=510642]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 22:00:19.871860472 (582h31m19.742943527s) [upload=510642]: Done uploading segments... Dec 29 22:00:19.871892413 (582h31m19.742975468s) [upload=510642]: Upload finished. err: <nil> Duration: 13.720672095s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................X............................X...........] duration: 34.970861716s trace: https://jaeger.dev.storj.io/trace/7096862780f29049 FAIL (2025-12-29 22:16:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== oise=true , address=145.79.227.30:4766 , piece_id=YWDMUGNTYVXWOCMSN63VIC6HOTWHPY7VJ7QWT76P63UAJIBS2XLA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.060493636 (582h37m18.931576671s) [noise=true , address=5.231.73.5:4022 , piece_id=5WVAJH3Y6TLATXY4IYB6ZADKKJDXDS52B2ZN6FJAZ7UHFVRYRRAA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.060685291 (582h37m18.931768326s) [noise=true , address=77.90.31.11:4579 , piece_id=DEJPYHGYNAP52CZ45BKGNXJM5MLIA2OQZ4BRKN5PBADNRVE3VLEA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.060749954 (582h37m18.931832989s) [noise=true , address=145.79.227.29:4976 , piece_id=YRCRXSSQOXPOJGVX46L7WCB3TRDZ5QB755SEOTNHSCHN4ZDMCOUQ, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.061033203 (582h37m18.932116248s) [noise=true , address=193.149.164.143:4698 , piece_id=6X5KB4TXFV4ZOF5JTPNRW4Q3GWNCVCXCA4JSB3AZURBHXY2VHTRQ, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.061248855 (582h37m18.9323319s) [noise=true , address=145.79.227.49:4347 , piece_id=75B3DRRHEPDL6EK6ZUXTKASXUKWGAVUWBPSTNRYT4BJMAU5ALXRA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.061382729 (582h37m18.932465764s) [noise=true , address=5.231.73.2:4004 , piece_id=PTHHO3XEPGGCIRCUBIWHN4LVNBLMISJM5LESQGR2ERRESFK3S4RA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.061645560 (582h37m18.932728595s) [noise=true , address=5.231.73.3:4007 , piece_id=6AF3CEPLUMRO4CDW3SRWZSYHWEXP35AVSM5M3LH4KCRARSZ6NFNA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.061769094 (582h37m18.932852139s) [noise=true , address=145.79.227.7:4321 , piece_id=SPWPZ5MR5ZCQNYVYYQQEZC4OURAM5VAH3WN5UP6HMGCCDMCH6ZGA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.061970298 (582h37m18.933053333s) [noise=true , address=145.79.227.35:4507 , piece_id=ZZSVIHOAN2SWXI4NS3TRD4B64NDB4D43J63DAS74TCPEE6MPGRXA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.062237577 (582h37m18.933320612s) [noise=true , address=145.79.227.35:4489 , piece_id=BM7ZQESTHEUIRJWAPORTKDMT5TVLEXMOHULGFVEQKHT4AJV3HT5A, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.062304304 (582h37m18.933387349s) [noise=true , address=145.79.227.34:4931 , piece_id=XJHLE3B5HIF4RC4LV32SBCNMVTOTEXWBBCXBVPQFN6N5HFQ6ZS7A, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.062522640 (582h37m18.933605685s) [noise=true , address=145.79.227.48:4329 , piece_id=YWCVTYLCAGU57SE3ID3WW2JLCJGTJ7CPDRCZENAMO42BVPIAVLIQ, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.062904968 (582h37m18.933988003s) [noise=true , address=77.90.31.17:4801 , piece_id=K3QD7FPVRQHZL5R57EKKAC27RGDJQH6HDCND7FIVXSZBRH42WSIA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.063420419 (582h37m18.934503464s) [noise=true , address=145.79.227.17:4995 , piece_id=JJTVPVBAMCTDGHGWNPUVPYFVKD4SMTDAHF4VWVIW7JFUAMBIPRHQ, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.063943495 (582h37m18.93502654s) [noise=true , address=5.231.73.4:4018 , piece_id=X45H47HER27IM5FEQSEAMXSDX7UQG3AEOY65Q4WCCUOX7SVFZ5KA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:20.386418430 (582h37m20.257501465s) [upload=510734]: Next returned nil segment Dec 29 22:06:20.386428990 (582h37m20.257512025s) [upload=510734]: Waiting for error group managing segments... Dec 29 22:06:20.935543714 (582h37m20.806626749s) [noise=true , address=145.79.227.48:4329 , piece_id=YWCVTYLCAGU57SE3ID3WW2JLCJGTJ7CPDRCZENAMO42BVPIAVLIQ, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:20.978073277 (582h37m20.849156312s) [noise=true , address=145.79.227.39:4022 , piece_id=5IJCSVRVXCICRNRB3TTJAVNRF5FTUQMMY4KL7ZFF4SBKILFGHJWA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.079004013 (582h37m20.950087058s) [noise=true , address=5.231.73.4:4018 , piece_id=X45H47HER27IM5FEQSEAMXSDX7UQG3AEOY65Q4WCCUOX7SVFZ5KA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.101144552 (582h37m20.972227597s) [noise=true , address=77.90.31.11:4579 , piece_id=DEJPYHGYNAP52CZ45BKGNXJM5MLIA2OQZ4BRKN5PBADNRVE3VLEA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.138766369 (582h37m21.009849405s) [noise=true , address=77.90.31.17:4801 , piece_id=K3QD7FPVRQHZL5R57EKKAC27RGDJQH6HDCND7FIVXSZBRH42WSIA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.231047035 (582h37m21.10213007s) [noise=true , address=77.90.31.19:4857 , piece_id=DMJ4O3TWZYWKWVLJFSQBULHDFWZCNLVDBIGYJEOBAVVCU5UGG27Q, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.272062706 (582h37m21.143145741s) [noise=true , address=5.231.73.3:4007 , piece_id=6AF3CEPLUMRO4CDW3SRWZSYHWEXP35AVSM5M3LH4KCRARSZ6NFNA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.303520666 (582h37m21.174603711s) [noise=true , address=145.79.227.29:4976 , piece_id=YRCRXSSQOXPOJGVX46L7WCB3TRDZ5QB755SEOTNHSCHN4ZDMCOUQ, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.340640177 (582h37m21.211723212s) [noise=true , address=5.231.73.5:4022 , piece_id=5WVAJH3Y6TLATXY4IYB6ZADKKJDXDS52B2ZN6FJAZ7UHFVRYRRAA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.481118228 (582h37m21.352201263s) [noise=true , address=5.231.73.2:4004 , piece_id=PTHHO3XEPGGCIRCUBIWHN4LVNBLMISJM5LESQGR2ERRESFK3S4RA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.896953769 (582h37m21.768036804s) [noise=true , address=145.79.227.49:4347 , piece_id=75B3DRRHEPDL6EK6ZUXTKASXUKWGAVUWBPSTNRYT4BJMAU5ALXRA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.911314353 (582h37m21.782397399s) [noise=true , address=145.79.227.34:4931 , piece_id=XJHLE3B5HIF4RC4LV32SBCNMVTOTEXWBBCXBVPQFN6N5HFQ6ZS7A, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.915845371 (582h37m21.786928406s) [noise=true , address=145.79.227.30:4766 , piece_id=YWDMUGNTYVXWOCMSN63VIC6HOTWHPY7VJ7QWT76P63UAJIBS2XLA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.928785972 (582h37m21.799869017s) [noise=true , address=145.79.227.35:4489 , piece_id=BM7ZQESTHEUIRJWAPORTKDMT5TVLEXMOHULGFVEQKHT4AJV3HT5A, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.928810859 (582h37m21.799893904s) [seg_pos={0 0}, upload=510734]: Segment reached optimal threshold of 14 pieces. Dec 29 22:06:21.929016471 (582h37m21.800099516s) [noise=true , address=145.79.227.7:4321 , piece_id=SPWPZ5MR5ZCQNYVYYQQEZC4OURAM5VAH3WN5UP6HMGCCDMCH6ZGA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12edsHqD9iRBDgMU8vYhSM1EWG8Sn57GZ8TPiTytri3rp7W9Eae): protocol: expected piece hash; context canceled; EOF Dec 29 22:06:21.929028644 (582h37m21.800111709s) [noise=true , address=145.79.227.7:4321 , piece_id=SPWPZ5MR5ZCQNYVYYQQEZC4OURAM5VAH3WN5UP6HMGCCDMCH6ZGA, seg_pos={0 0}, upload=510734]: Long tail cancellation due to optimal threshold Dec 29 22:06:21.929093137 (582h37m21.800176192s) [noise=true , address=145.79.227.35:4507 , piece_id=ZZSVIHOAN2SWXI4NS3TRD4B64NDB4D43J63DAS74TCPEE6MPGRXA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14jurWWPpmxwiBS3XmZysEUwugsceYJqVXFj7BbYpx8UFcTR8R): protocol: expected piece hash; context canceled; EOF Dec 29 22:06:21.929106793 (582h37m21.800189838s) [noise=true , address=145.79.227.35:4507 , piece_id=ZZSVIHOAN2SWXI4NS3TRD4B64NDB4D43J63DAS74TCPEE6MPGRXA, seg_pos={0 0}, upload=510734]: Long tail cancellation due to optimal threshold Dec 29 22:06:21.929206612 (582h37m21.800289657s) [noise=true , address=145.79.227.17:4995 , piece_id=JJTVPVBAMCTDGHGWNPUVPYFVKD4SMTDAHF4VWVIW7JFUAMBIPRHQ, seg_pos={0 0}, upload=510734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tvoTD1LoSj3q2rLiDckfT56dXNKKwdVu1ZAsBZExNFVTom9tH): protocol: write tcp 188.166.225.170:47696->145.79.227.17:4995: use of closed network connection; write tcp 188.166.225.170:47696->145.79.227.17:4995: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47696->145.79.227.17:4995: use of closed network connection Dec 29 22:06:21.929215911 (582h37m21.800298976s) [noise=true , address=145.79.227.17:4995 , piece_id=JJTVPVBAMCTDGHGWNPUVPYFVKD4SMTDAHF4VWVIW7JFUAMBIPRHQ, seg_pos={0 0}, upload=510734]: Long tail cancellation due to optimal threshold Dec 29 22:06:21.929387247 (582h37m21.800470292s) [noise=true , address=193.149.164.143:4698 , piece_id=6X5KB4TXFV4ZOF5JTPNRW4Q3GWNCVCXCA4JSB3AZURBHXY2VHTRQ, seg_pos={0 0}, upload=510734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F4fJAGJM2ZPRt7vJH2cZmEbYeujmxcfi7aPZ2g9VPTNYBYRAi): protocol: expected piece hash; context canceled; EOF Dec 29 22:06:21.929396174 (582h37m21.800479219s) [noise=true , address=193.149.164.143:4698 , piece_id=6X5KB4TXFV4ZOF5JTPNRW4Q3GWNCVCXCA4JSB3AZURBHXY2VHTRQ, seg_pos={0 0}, upload=510734]: Long tail cancellation due to optimal threshold Dec 29 22:06:21.929491536 (582h37m21.800574571s) [seg_pos={0 0}, upload=510734]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 22:06:21.929573792 (582h37m21.800656837s) [upload=510734]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 22:06:21.929580555 (582h37m21.8006636s) [upload=510734]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 22:06:22.233045366 (582h37m22.104128401s) [upload=510734]: Done uploading segments... Dec 29 22:06:22.233064963 (582h37m22.104147998s) [upload=510734]: Upload finished. err: <nil> Duration: 34.970893548s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.X........X............................X....................] duration: 5.959630805s trace: https://jaeger.dev.storj.io/trace/5209b3d035311fd FAIL (2025-12-29 22:16:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :00:17.043136250 (582h31m16.914219305s) [upload=510642]: Starting upload Dec 29 22:00:17.043227904 (582h31m16.91431096s) [upload=510642]: Done starting upload Dec 29 22:00:17.043335189 (582h31m16.914418234s) [upload=510642]: Uploading segments... Dec 29 22:00:17.043587749 (582h31m16.914670824s) [upload=510642]: Got next segment. Inline: false Dec 29 22:00:17.043644478 (582h31m16.914727523s) [upload=510642]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 22:00:17.043686427 (582h31m16.914769503s) [upload=510642]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 22:00:17.294384363 (582h31m17.165467408s) [seg_pos={0 0}, upload=510642]: Begin upload segment... Dec 29 22:00:17.294445219 (582h31m17.165528264s) [seg_pos={0 0}, upload=510642]: Done begin upload segment. Dec 29 22:00:17.294484133 (582h31m17.165567178s) [noise=true , address=145.79.227.31:4797 , piece_id=NI3M5NVVVX5HUXZ2SOMQD4S5ZOSSNPCD2F3Z5X2BA2VXMGT57AYQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.294560979 (582h31m17.165644014s) [noise=true , address=77.90.31.11:4553 , piece_id=3Y5HSD73DLZ6ALJ5TFRIVTLOE4ZNZ256YSZ2W3X5LPKL5J5LX2EQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.295149941 (582h31m17.166232976s) [noise=true , address=45.91.116.5:4297 , piece_id=PLUQ7Q32BTGYFQRGVJFAJWHPZCPQTC5RX5UX27S5HJ5NNJPLDMPQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.295419574 (582h31m17.166502609s) [noise=true , address=145.79.227.2:4115 , piece_id=P2AS5B3VYO2K2AWSM5W4BXAXBHSXQQWM7JOXFPOV64NTKDFLT62Q, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.295483997 (582h31m17.166567042s) [noise=true , address=145.79.227.51:4411 , piece_id=HTZQ6SW6KSACLQF7G4CFGQA572J5JVHMLGEWQYSSGZRYX7FJ6DXA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.295762207 (582h31m17.166845252s) [noise=true , address=145.79.227.5:4221 , piece_id=LD3FENLBM2L5N7QDOAMAYJUVH54QVSJQJFP7E36UAH2ZD7R65CCQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.295928003 (582h31m17.167011048s) [noise=true , address=145.79.227.16:4909 , piece_id=CF7WHNKC4LFJQRVFBPFV62JGEBLD4KVPE6Q5JBWQVADC6ZAFFLVQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.296039615 (582h31m17.16712266s) [noise=true , address=45.91.116.13:4154 , piece_id=FE4NFODQ3TWGFXARFYIL64TKK6X7H6PPLIG6LO44LKNAZOEU7DJA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.296272429 (582h31m17.167355474s) [noise=true , address=45.91.116.10:4084 , piece_id=5BFUYVGIFOYXC6RCPU7VBUM36LSM32TVBLS3W54OWFNR4SM6GCSA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.296403588 (582h31m17.167486633s) [noise=true , address=77.90.31.19:4844 , piece_id=CDOWECXLSRPACV4HMAH57UK4EEIYNOMH6GYV6HFHV7CWSEAJ44FA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.296646560 (582h31m17.167729596s) [noise=true , address=77.90.31.14:4661 , piece_id=B7IDSCDHBUXUIPSUT3HMQSCMPGBMYASZSQO2EJBXRR2DQHCLWGYA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.296906235 (582h31m17.167989291s) [noise=true , address=145.79.227.47:4265 , piece_id=FYZA2LRIFKLKVUCUAWMNWTUH3VYR5VFAA7XKAYIMD2E2VOJTI7PA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.296989804 (582h31m17.168072849s) [noise=true , address=145.79.227.46:4601 , piece_id=YJTMZID24VBM5ZZSUUHOCZSZSSNGTFHOSKK6OI5QP3A4G6ZWVWQA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.297235221 (582h31m17.168318267s) [noise=true , address=145.79.227.3:4150 , piece_id=HS4S5UDYD2VWI2N4MH6EWEYQ5G6KCFCJBDZR7AVWPXRYALFXP4FQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.297424432 (582h31m17.168507467s) [noise=true , address=145.79.227.13:4803 , piece_id=YIEUZEDLSC5C67SCFEPCHHZSTYP6MYLYJZJTGZF3UP6JWDTGUOPQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.297606900 (582h31m17.168689945s) [noise=true , address=45.91.116.14:4190 , piece_id=FIL6AIXGE7DE2M4QBVSIO53KGY43VFPZQ2FTXSWJFPJU4VZCKUXQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:18.995364155 (582h31m18.866447201s) [upload=510642]: Next returned nil segment Dec 29 22:00:18.995383603 (582h31m18.866466648s) [upload=510642]: Waiting for error group managing segments... Dec 29 22:00:19.284009408 (582h31m19.155092453s) [noise=true , address=145.79.227.51:4411 , piece_id=HTZQ6SW6KSACLQF7G4CFGQA572J5JVHMLGEWQYSSGZRYX7FJ6DXA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.295985232 (582h31m19.167068277s) [noise=true , address=145.79.227.13:4803 , piece_id=YIEUZEDLSC5C67SCFEPCHHZSTYP6MYLYJZJTGZF3UP6JWDTGUOPQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.296135458 (582h31m19.167218503s) [noise=true , address=145.79.227.16:4909 , piece_id=CF7WHNKC4LFJQRVFBPFV62JGEBLD4KVPE6Q5JBWQVADC6ZAFFLVQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.310119916 (582h31m19.181202971s) [noise=true , address=145.79.227.46:4601 , piece_id=YJTMZID24VBM5ZZSUUHOCZSZSSNGTFHOSKK6OI5QP3A4G6ZWVWQA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.310998939 (582h31m19.182081984s) [noise=true , address=145.79.227.31:4797 , piece_id=NI3M5NVVVX5HUXZ2SOMQD4S5ZOSSNPCD2F3Z5X2BA2VXMGT57AYQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.318707138 (582h31m19.189790183s) [noise=true , address=145.79.227.47:4265 , piece_id=FYZA2LRIFKLKVUCUAWMNWTUH3VYR5VFAA7XKAYIMD2E2VOJTI7PA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.322059381 (582h31m19.193142426s) [noise=true , address=145.79.227.5:4221 , piece_id=LD3FENLBM2L5N7QDOAMAYJUVH54QVSJQJFP7E36UAH2ZD7R65CCQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.506372108 (582h31m19.377455153s) [noise=true , address=77.90.31.19:4844 , piece_id=CDOWECXLSRPACV4HMAH57UK4EEIYNOMH6GYV6HFHV7CWSEAJ44FA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.507172602 (582h31m19.378255648s) [noise=true , address=77.90.31.11:4553 , piece_id=3Y5HSD73DLZ6ALJ5TFRIVTLOE4ZNZ256YSZ2W3X5LPKL5J5LX2EQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.527548842 (582h31m19.398631887s) [noise=true , address=77.90.31.14:4661 , piece_id=B7IDSCDHBUXUIPSUT3HMQSCMPGBMYASZSQO2EJBXRR2DQHCLWGYA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.531443508 (582h31m19.402526553s) [noise=true , address=45.91.116.5:4297 , piece_id=PLUQ7Q32BTGYFQRGVJFAJWHPZCPQTC5RX5UX27S5HJ5NNJPLDMPQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.552364906 (582h31m19.423447951s) [noise=true , address=45.91.116.14:4190 , piece_id=FIL6AIXGE7DE2M4QBVSIO53KGY43VFPZQ2FTXSWJFPJU4VZCKUXQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.564056769 (582h31m19.435139814s) [noise=true , address=45.91.116.10:4084 , piece_id=5BFUYVGIFOYXC6RCPU7VBUM36LSM32TVBLS3W54OWFNR4SM6GCSA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.564087016 (582h31m19.435170051s) [seg_pos={0 0}, upload=510642]: Segment reached optimal threshold of 13 pieces. Dec 29 22:00:19.564294431 (582h31m19.435377466s) [noise=true , address=145.79.227.3:4150 , piece_id=HS4S5UDYD2VWI2N4MH6EWEYQ5G6KCFCJBDZR7AVWPXRYALFXP4FQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK): protocol: expected piece hash; context canceled; EOF Dec 29 22:00:19.564302366 (582h31m19.435385411s) [noise=true , address=145.79.227.3:4150 , piece_id=HS4S5UDYD2VWI2N4MH6EWEYQ5G6KCFCJBDZR7AVWPXRYALFXP4FQ, seg_pos={0 0}, upload=510642]: Long tail cancellation due to optimal threshold Dec 29 22:00:19.564388932 (582h31m19.435471977s) [noise=true , address=45.91.116.13:4154 , piece_id=FE4NFODQ3TWGFXARFYIL64TKK6X7H6PPLIG6LO44LKNAZOEU7DJA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Dec 29 22:00:19.564401876 (582h31m19.435484911s) [noise=true , address=45.91.116.13:4154 , piece_id=FE4NFODQ3TWGFXARFYIL64TKK6X7H6PPLIG6LO44LKNAZOEU7DJA, seg_pos={0 0}, upload=510642]: Long tail cancellation due to optimal threshold Dec 29 22:00:19.564558945 (582h31m19.435641981s) [noise=true , address=145.79.227.2:4115 , piece_id=P2AS5B3VYO2K2AWSM5W4BXAXBHSXQQWM7JOXFPOV64NTKDFLT62Q, seg_pos={0 0}, upload=510642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: write tcp 188.166.225.170:38246->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:38246->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38246->145.79.227.2:4115: use of closed network connection Dec 29 22:00:19.564570738 (582h31m19.435653783s) [noise=true , address=145.79.227.2:4115 , piece_id=P2AS5B3VYO2K2AWSM5W4BXAXBHSXQQWM7JOXFPOV64NTKDFLT62Q, seg_pos={0 0}, upload=510642]: Long tail cancellation due to optimal threshold Dec 29 22:00:19.564601426 (582h31m19.435684471s) [seg_pos={0 0}, upload=510642]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 22:00:19.564683973 (582h31m19.435767018s) [upload=510642]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 22:00:19.564690004 (582h31m19.435773049s) [upload=510642]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 22:00:19.871860472 (582h31m19.742943527s) [upload=510642]: Done uploading segments... Dec 29 22:00:19.871892413 (582h31m19.742975468s) [upload=510642]: Upload finished. err: <nil> Duration: 5.959648327s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..X........X............................X...................] duration: 6.819250534s trace: https://jaeger.dev.storj.io/trace/5209b3d035311fd FAIL (2025-12-29 22:15:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :00:17.043136250 (582h31m16.914219305s) [upload=510642]: Starting upload Dec 29 22:00:17.043227904 (582h31m16.91431096s) [upload=510642]: Done starting upload Dec 29 22:00:17.043335189 (582h31m16.914418234s) [upload=510642]: Uploading segments... Dec 29 22:00:17.043587749 (582h31m16.914670824s) [upload=510642]: Got next segment. Inline: false Dec 29 22:00:17.043644478 (582h31m16.914727523s) [upload=510642]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 22:00:17.043686427 (582h31m16.914769503s) [upload=510642]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 22:00:17.294384363 (582h31m17.165467408s) [seg_pos={0 0}, upload=510642]: Begin upload segment... Dec 29 22:00:17.294445219 (582h31m17.165528264s) [seg_pos={0 0}, upload=510642]: Done begin upload segment. Dec 29 22:00:17.294484133 (582h31m17.165567178s) [noise=true , address=145.79.227.31:4797 , piece_id=NI3M5NVVVX5HUXZ2SOMQD4S5ZOSSNPCD2F3Z5X2BA2VXMGT57AYQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.294560979 (582h31m17.165644014s) [noise=true , address=77.90.31.11:4553 , piece_id=3Y5HSD73DLZ6ALJ5TFRIVTLOE4ZNZ256YSZ2W3X5LPKL5J5LX2EQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.295149941 (582h31m17.166232976s) [noise=true , address=45.91.116.5:4297 , piece_id=PLUQ7Q32BTGYFQRGVJFAJWHPZCPQTC5RX5UX27S5HJ5NNJPLDMPQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.295419574 (582h31m17.166502609s) [noise=true , address=145.79.227.2:4115 , piece_id=P2AS5B3VYO2K2AWSM5W4BXAXBHSXQQWM7JOXFPOV64NTKDFLT62Q, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.295483997 (582h31m17.166567042s) [noise=true , address=145.79.227.51:4411 , piece_id=HTZQ6SW6KSACLQF7G4CFGQA572J5JVHMLGEWQYSSGZRYX7FJ6DXA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.295762207 (582h31m17.166845252s) [noise=true , address=145.79.227.5:4221 , piece_id=LD3FENLBM2L5N7QDOAMAYJUVH54QVSJQJFP7E36UAH2ZD7R65CCQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.295928003 (582h31m17.167011048s) [noise=true , address=145.79.227.16:4909 , piece_id=CF7WHNKC4LFJQRVFBPFV62JGEBLD4KVPE6Q5JBWQVADC6ZAFFLVQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.296039615 (582h31m17.16712266s) [noise=true , address=45.91.116.13:4154 , piece_id=FE4NFODQ3TWGFXARFYIL64TKK6X7H6PPLIG6LO44LKNAZOEU7DJA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.296272429 (582h31m17.167355474s) [noise=true , address=45.91.116.10:4084 , piece_id=5BFUYVGIFOYXC6RCPU7VBUM36LSM32TVBLS3W54OWFNR4SM6GCSA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.296403588 (582h31m17.167486633s) [noise=true , address=77.90.31.19:4844 , piece_id=CDOWECXLSRPACV4HMAH57UK4EEIYNOMH6GYV6HFHV7CWSEAJ44FA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.296646560 (582h31m17.167729596s) [noise=true , address=77.90.31.14:4661 , piece_id=B7IDSCDHBUXUIPSUT3HMQSCMPGBMYASZSQO2EJBXRR2DQHCLWGYA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.296906235 (582h31m17.167989291s) [noise=true , address=145.79.227.47:4265 , piece_id=FYZA2LRIFKLKVUCUAWMNWTUH3VYR5VFAA7XKAYIMD2E2VOJTI7PA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.296989804 (582h31m17.168072849s) [noise=true , address=145.79.227.46:4601 , piece_id=YJTMZID24VBM5ZZSUUHOCZSZSSNGTFHOSKK6OI5QP3A4G6ZWVWQA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.297235221 (582h31m17.168318267s) [noise=true , address=145.79.227.3:4150 , piece_id=HS4S5UDYD2VWI2N4MH6EWEYQ5G6KCFCJBDZR7AVWPXRYALFXP4FQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.297424432 (582h31m17.168507467s) [noise=true , address=145.79.227.13:4803 , piece_id=YIEUZEDLSC5C67SCFEPCHHZSTYP6MYLYJZJTGZF3UP6JWDTGUOPQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.297606900 (582h31m17.168689945s) [noise=true , address=45.91.116.14:4190 , piece_id=FIL6AIXGE7DE2M4QBVSIO53KGY43VFPZQ2FTXSWJFPJU4VZCKUXQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:18.995364155 (582h31m18.866447201s) [upload=510642]: Next returned nil segment Dec 29 22:00:18.995383603 (582h31m18.866466648s) [upload=510642]: Waiting for error group managing segments... Dec 29 22:00:19.284009408 (582h31m19.155092453s) [noise=true , address=145.79.227.51:4411 , piece_id=HTZQ6SW6KSACLQF7G4CFGQA572J5JVHMLGEWQYSSGZRYX7FJ6DXA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.295985232 (582h31m19.167068277s) [noise=true , address=145.79.227.13:4803 , piece_id=YIEUZEDLSC5C67SCFEPCHHZSTYP6MYLYJZJTGZF3UP6JWDTGUOPQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.296135458 (582h31m19.167218503s) [noise=true , address=145.79.227.16:4909 , piece_id=CF7WHNKC4LFJQRVFBPFV62JGEBLD4KVPE6Q5JBWQVADC6ZAFFLVQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.310119916 (582h31m19.181202971s) [noise=true , address=145.79.227.46:4601 , piece_id=YJTMZID24VBM5ZZSUUHOCZSZSSNGTFHOSKK6OI5QP3A4G6ZWVWQA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.310998939 (582h31m19.182081984s) [noise=true , address=145.79.227.31:4797 , piece_id=NI3M5NVVVX5HUXZ2SOMQD4S5ZOSSNPCD2F3Z5X2BA2VXMGT57AYQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.318707138 (582h31m19.189790183s) [noise=true , address=145.79.227.47:4265 , piece_id=FYZA2LRIFKLKVUCUAWMNWTUH3VYR5VFAA7XKAYIMD2E2VOJTI7PA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.322059381 (582h31m19.193142426s) [noise=true , address=145.79.227.5:4221 , piece_id=LD3FENLBM2L5N7QDOAMAYJUVH54QVSJQJFP7E36UAH2ZD7R65CCQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.506372108 (582h31m19.377455153s) [noise=true , address=77.90.31.19:4844 , piece_id=CDOWECXLSRPACV4HMAH57UK4EEIYNOMH6GYV6HFHV7CWSEAJ44FA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.507172602 (582h31m19.378255648s) [noise=true , address=77.90.31.11:4553 , piece_id=3Y5HSD73DLZ6ALJ5TFRIVTLOE4ZNZ256YSZ2W3X5LPKL5J5LX2EQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.527548842 (582h31m19.398631887s) [noise=true , address=77.90.31.14:4661 , piece_id=B7IDSCDHBUXUIPSUT3HMQSCMPGBMYASZSQO2EJBXRR2DQHCLWGYA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.531443508 (582h31m19.402526553s) [noise=true , address=45.91.116.5:4297 , piece_id=PLUQ7Q32BTGYFQRGVJFAJWHPZCPQTC5RX5UX27S5HJ5NNJPLDMPQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.552364906 (582h31m19.423447951s) [noise=true , address=45.91.116.14:4190 , piece_id=FIL6AIXGE7DE2M4QBVSIO53KGY43VFPZQ2FTXSWJFPJU4VZCKUXQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.564056769 (582h31m19.435139814s) [noise=true , address=45.91.116.10:4084 , piece_id=5BFUYVGIFOYXC6RCPU7VBUM36LSM32TVBLS3W54OWFNR4SM6GCSA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.564087016 (582h31m19.435170051s) [seg_pos={0 0}, upload=510642]: Segment reached optimal threshold of 13 pieces. Dec 29 22:00:19.564294431 (582h31m19.435377466s) [noise=true , address=145.79.227.3:4150 , piece_id=HS4S5UDYD2VWI2N4MH6EWEYQ5G6KCFCJBDZR7AVWPXRYALFXP4FQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK): protocol: expected piece hash; context canceled; EOF Dec 29 22:00:19.564302366 (582h31m19.435385411s) [noise=true , address=145.79.227.3:4150 , piece_id=HS4S5UDYD2VWI2N4MH6EWEYQ5G6KCFCJBDZR7AVWPXRYALFXP4FQ, seg_pos={0 0}, upload=510642]: Long tail cancellation due to optimal threshold Dec 29 22:00:19.564388932 (582h31m19.435471977s) [noise=true , address=45.91.116.13:4154 , piece_id=FE4NFODQ3TWGFXARFYIL64TKK6X7H6PPLIG6LO44LKNAZOEU7DJA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Dec 29 22:00:19.564401876 (582h31m19.435484911s) [noise=true , address=45.91.116.13:4154 , piece_id=FE4NFODQ3TWGFXARFYIL64TKK6X7H6PPLIG6LO44LKNAZOEU7DJA, seg_pos={0 0}, upload=510642]: Long tail cancellation due to optimal threshold Dec 29 22:00:19.564558945 (582h31m19.435641981s) [noise=true , address=145.79.227.2:4115 , piece_id=P2AS5B3VYO2K2AWSM5W4BXAXBHSXQQWM7JOXFPOV64NTKDFLT62Q, seg_pos={0 0}, upload=510642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: write tcp 188.166.225.170:38246->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:38246->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38246->145.79.227.2:4115: use of closed network connection Dec 29 22:00:19.564570738 (582h31m19.435653783s) [noise=true , address=145.79.227.2:4115 , piece_id=P2AS5B3VYO2K2AWSM5W4BXAXBHSXQQWM7JOXFPOV64NTKDFLT62Q, seg_pos={0 0}, upload=510642]: Long tail cancellation due to optimal threshold Dec 29 22:00:19.564601426 (582h31m19.435684471s) [seg_pos={0 0}, upload=510642]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 22:00:19.564683973 (582h31m19.435767018s) [upload=510642]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 22:00:19.564690004 (582h31m19.435773049s) [upload=510642]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 22:00:19.871860472 (582h31m19.742943527s) [upload=510642]: Done uploading segments... Dec 29 22:00:19.871892413 (582h31m19.742975468s) [upload=510642]: Upload finished. err: <nil> Duration: 6.819276655s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................X............................X..........] duration: 29.300104931s trace: https://jaeger.dev.storj.io/trace/7096862780f29049 FAIL (2025-12-29 22:15:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== oise=true , address=145.79.227.30:4766 , piece_id=YWDMUGNTYVXWOCMSN63VIC6HOTWHPY7VJ7QWT76P63UAJIBS2XLA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.060493636 (582h37m18.931576671s) [noise=true , address=5.231.73.5:4022 , piece_id=5WVAJH3Y6TLATXY4IYB6ZADKKJDXDS52B2ZN6FJAZ7UHFVRYRRAA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.060685291 (582h37m18.931768326s) [noise=true , address=77.90.31.11:4579 , piece_id=DEJPYHGYNAP52CZ45BKGNXJM5MLIA2OQZ4BRKN5PBADNRVE3VLEA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.060749954 (582h37m18.931832989s) [noise=true , address=145.79.227.29:4976 , piece_id=YRCRXSSQOXPOJGVX46L7WCB3TRDZ5QB755SEOTNHSCHN4ZDMCOUQ, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.061033203 (582h37m18.932116248s) [noise=true , address=193.149.164.143:4698 , piece_id=6X5KB4TXFV4ZOF5JTPNRW4Q3GWNCVCXCA4JSB3AZURBHXY2VHTRQ, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.061248855 (582h37m18.9323319s) [noise=true , address=145.79.227.49:4347 , piece_id=75B3DRRHEPDL6EK6ZUXTKASXUKWGAVUWBPSTNRYT4BJMAU5ALXRA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.061382729 (582h37m18.932465764s) [noise=true , address=5.231.73.2:4004 , piece_id=PTHHO3XEPGGCIRCUBIWHN4LVNBLMISJM5LESQGR2ERRESFK3S4RA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.061645560 (582h37m18.932728595s) [noise=true , address=5.231.73.3:4007 , piece_id=6AF3CEPLUMRO4CDW3SRWZSYHWEXP35AVSM5M3LH4KCRARSZ6NFNA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.061769094 (582h37m18.932852139s) [noise=true , address=145.79.227.7:4321 , piece_id=SPWPZ5MR5ZCQNYVYYQQEZC4OURAM5VAH3WN5UP6HMGCCDMCH6ZGA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.061970298 (582h37m18.933053333s) [noise=true , address=145.79.227.35:4507 , piece_id=ZZSVIHOAN2SWXI4NS3TRD4B64NDB4D43J63DAS74TCPEE6MPGRXA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.062237577 (582h37m18.933320612s) [noise=true , address=145.79.227.35:4489 , piece_id=BM7ZQESTHEUIRJWAPORTKDMT5TVLEXMOHULGFVEQKHT4AJV3HT5A, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.062304304 (582h37m18.933387349s) [noise=true , address=145.79.227.34:4931 , piece_id=XJHLE3B5HIF4RC4LV32SBCNMVTOTEXWBBCXBVPQFN6N5HFQ6ZS7A, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.062522640 (582h37m18.933605685s) [noise=true , address=145.79.227.48:4329 , piece_id=YWCVTYLCAGU57SE3ID3WW2JLCJGTJ7CPDRCZENAMO42BVPIAVLIQ, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.062904968 (582h37m18.933988003s) [noise=true , address=77.90.31.17:4801 , piece_id=K3QD7FPVRQHZL5R57EKKAC27RGDJQH6HDCND7FIVXSZBRH42WSIA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.063420419 (582h37m18.934503464s) [noise=true , address=145.79.227.17:4995 , piece_id=JJTVPVBAMCTDGHGWNPUVPYFVKD4SMTDAHF4VWVIW7JFUAMBIPRHQ, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.063943495 (582h37m18.93502654s) [noise=true , address=5.231.73.4:4018 , piece_id=X45H47HER27IM5FEQSEAMXSDX7UQG3AEOY65Q4WCCUOX7SVFZ5KA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:20.386418430 (582h37m20.257501465s) [upload=510734]: Next returned nil segment Dec 29 22:06:20.386428990 (582h37m20.257512025s) [upload=510734]: Waiting for error group managing segments... Dec 29 22:06:20.935543714 (582h37m20.806626749s) [noise=true , address=145.79.227.48:4329 , piece_id=YWCVTYLCAGU57SE3ID3WW2JLCJGTJ7CPDRCZENAMO42BVPIAVLIQ, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:20.978073277 (582h37m20.849156312s) [noise=true , address=145.79.227.39:4022 , piece_id=5IJCSVRVXCICRNRB3TTJAVNRF5FTUQMMY4KL7ZFF4SBKILFGHJWA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.079004013 (582h37m20.950087058s) [noise=true , address=5.231.73.4:4018 , piece_id=X45H47HER27IM5FEQSEAMXSDX7UQG3AEOY65Q4WCCUOX7SVFZ5KA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.101144552 (582h37m20.972227597s) [noise=true , address=77.90.31.11:4579 , piece_id=DEJPYHGYNAP52CZ45BKGNXJM5MLIA2OQZ4BRKN5PBADNRVE3VLEA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.138766369 (582h37m21.009849405s) [noise=true , address=77.90.31.17:4801 , piece_id=K3QD7FPVRQHZL5R57EKKAC27RGDJQH6HDCND7FIVXSZBRH42WSIA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.231047035 (582h37m21.10213007s) [noise=true , address=77.90.31.19:4857 , piece_id=DMJ4O3TWZYWKWVLJFSQBULHDFWZCNLVDBIGYJEOBAVVCU5UGG27Q, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.272062706 (582h37m21.143145741s) [noise=true , address=5.231.73.3:4007 , piece_id=6AF3CEPLUMRO4CDW3SRWZSYHWEXP35AVSM5M3LH4KCRARSZ6NFNA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.303520666 (582h37m21.174603711s) [noise=true , address=145.79.227.29:4976 , piece_id=YRCRXSSQOXPOJGVX46L7WCB3TRDZ5QB755SEOTNHSCHN4ZDMCOUQ, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.340640177 (582h37m21.211723212s) [noise=true , address=5.231.73.5:4022 , piece_id=5WVAJH3Y6TLATXY4IYB6ZADKKJDXDS52B2ZN6FJAZ7UHFVRYRRAA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.481118228 (582h37m21.352201263s) [noise=true , address=5.231.73.2:4004 , piece_id=PTHHO3XEPGGCIRCUBIWHN4LVNBLMISJM5LESQGR2ERRESFK3S4RA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.896953769 (582h37m21.768036804s) [noise=true , address=145.79.227.49:4347 , piece_id=75B3DRRHEPDL6EK6ZUXTKASXUKWGAVUWBPSTNRYT4BJMAU5ALXRA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.911314353 (582h37m21.782397399s) [noise=true , address=145.79.227.34:4931 , piece_id=XJHLE3B5HIF4RC4LV32SBCNMVTOTEXWBBCXBVPQFN6N5HFQ6ZS7A, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.915845371 (582h37m21.786928406s) [noise=true , address=145.79.227.30:4766 , piece_id=YWDMUGNTYVXWOCMSN63VIC6HOTWHPY7VJ7QWT76P63UAJIBS2XLA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.928785972 (582h37m21.799869017s) [noise=true , address=145.79.227.35:4489 , piece_id=BM7ZQESTHEUIRJWAPORTKDMT5TVLEXMOHULGFVEQKHT4AJV3HT5A, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.928810859 (582h37m21.799893904s) [seg_pos={0 0}, upload=510734]: Segment reached optimal threshold of 14 pieces. Dec 29 22:06:21.929016471 (582h37m21.800099516s) [noise=true , address=145.79.227.7:4321 , piece_id=SPWPZ5MR5ZCQNYVYYQQEZC4OURAM5VAH3WN5UP6HMGCCDMCH6ZGA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12edsHqD9iRBDgMU8vYhSM1EWG8Sn57GZ8TPiTytri3rp7W9Eae): protocol: expected piece hash; context canceled; EOF Dec 29 22:06:21.929028644 (582h37m21.800111709s) [noise=true , address=145.79.227.7:4321 , piece_id=SPWPZ5MR5ZCQNYVYYQQEZC4OURAM5VAH3WN5UP6HMGCCDMCH6ZGA, seg_pos={0 0}, upload=510734]: Long tail cancellation due to optimal threshold Dec 29 22:06:21.929093137 (582h37m21.800176192s) [noise=true , address=145.79.227.35:4507 , piece_id=ZZSVIHOAN2SWXI4NS3TRD4B64NDB4D43J63DAS74TCPEE6MPGRXA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14jurWWPpmxwiBS3XmZysEUwugsceYJqVXFj7BbYpx8UFcTR8R): protocol: expected piece hash; context canceled; EOF Dec 29 22:06:21.929106793 (582h37m21.800189838s) [noise=true , address=145.79.227.35:4507 , piece_id=ZZSVIHOAN2SWXI4NS3TRD4B64NDB4D43J63DAS74TCPEE6MPGRXA, seg_pos={0 0}, upload=510734]: Long tail cancellation due to optimal threshold Dec 29 22:06:21.929206612 (582h37m21.800289657s) [noise=true , address=145.79.227.17:4995 , piece_id=JJTVPVBAMCTDGHGWNPUVPYFVKD4SMTDAHF4VWVIW7JFUAMBIPRHQ, seg_pos={0 0}, upload=510734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tvoTD1LoSj3q2rLiDckfT56dXNKKwdVu1ZAsBZExNFVTom9tH): protocol: write tcp 188.166.225.170:47696->145.79.227.17:4995: use of closed network connection; write tcp 188.166.225.170:47696->145.79.227.17:4995: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47696->145.79.227.17:4995: use of closed network connection Dec 29 22:06:21.929215911 (582h37m21.800298976s) [noise=true , address=145.79.227.17:4995 , piece_id=JJTVPVBAMCTDGHGWNPUVPYFVKD4SMTDAHF4VWVIW7JFUAMBIPRHQ, seg_pos={0 0}, upload=510734]: Long tail cancellation due to optimal threshold Dec 29 22:06:21.929387247 (582h37m21.800470292s) [noise=true , address=193.149.164.143:4698 , piece_id=6X5KB4TXFV4ZOF5JTPNRW4Q3GWNCVCXCA4JSB3AZURBHXY2VHTRQ, seg_pos={0 0}, upload=510734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F4fJAGJM2ZPRt7vJH2cZmEbYeujmxcfi7aPZ2g9VPTNYBYRAi): protocol: expected piece hash; context canceled; EOF Dec 29 22:06:21.929396174 (582h37m21.800479219s) [noise=true , address=193.149.164.143:4698 , piece_id=6X5KB4TXFV4ZOF5JTPNRW4Q3GWNCVCXCA4JSB3AZURBHXY2VHTRQ, seg_pos={0 0}, upload=510734]: Long tail cancellation due to optimal threshold Dec 29 22:06:21.929491536 (582h37m21.800574571s) [seg_pos={0 0}, upload=510734]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 22:06:21.929573792 (582h37m21.800656837s) [upload=510734]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 22:06:21.929580555 (582h37m21.8006636s) [upload=510734]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 22:06:22.233045366 (582h37m22.104128401s) [upload=510734]: Done uploading segments... Dec 29 22:06:22.233064963 (582h37m22.104147998s) [upload=510734]: Upload finished. err: <nil> Duration: 29.30013021s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [X..X........X............................X..................] duration: 5.957575678s trace: https://jaeger.dev.storj.io/trace/5209b3d035311fd FAIL (2025-12-29 22:15:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :00:17.043136250 (582h31m16.914219305s) [upload=510642]: Starting upload Dec 29 22:00:17.043227904 (582h31m16.91431096s) [upload=510642]: Done starting upload Dec 29 22:00:17.043335189 (582h31m16.914418234s) [upload=510642]: Uploading segments... Dec 29 22:00:17.043587749 (582h31m16.914670824s) [upload=510642]: Got next segment. Inline: false Dec 29 22:00:17.043644478 (582h31m16.914727523s) [upload=510642]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 22:00:17.043686427 (582h31m16.914769503s) [upload=510642]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 22:00:17.294384363 (582h31m17.165467408s) [seg_pos={0 0}, upload=510642]: Begin upload segment... Dec 29 22:00:17.294445219 (582h31m17.165528264s) [seg_pos={0 0}, upload=510642]: Done begin upload segment. Dec 29 22:00:17.294484133 (582h31m17.165567178s) [noise=true , address=145.79.227.31:4797 , piece_id=NI3M5NVVVX5HUXZ2SOMQD4S5ZOSSNPCD2F3Z5X2BA2VXMGT57AYQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.294560979 (582h31m17.165644014s) [noise=true , address=77.90.31.11:4553 , piece_id=3Y5HSD73DLZ6ALJ5TFRIVTLOE4ZNZ256YSZ2W3X5LPKL5J5LX2EQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.295149941 (582h31m17.166232976s) [noise=true , address=45.91.116.5:4297 , piece_id=PLUQ7Q32BTGYFQRGVJFAJWHPZCPQTC5RX5UX27S5HJ5NNJPLDMPQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.295419574 (582h31m17.166502609s) [noise=true , address=145.79.227.2:4115 , piece_id=P2AS5B3VYO2K2AWSM5W4BXAXBHSXQQWM7JOXFPOV64NTKDFLT62Q, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.295483997 (582h31m17.166567042s) [noise=true , address=145.79.227.51:4411 , piece_id=HTZQ6SW6KSACLQF7G4CFGQA572J5JVHMLGEWQYSSGZRYX7FJ6DXA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.295762207 (582h31m17.166845252s) [noise=true , address=145.79.227.5:4221 , piece_id=LD3FENLBM2L5N7QDOAMAYJUVH54QVSJQJFP7E36UAH2ZD7R65CCQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.295928003 (582h31m17.167011048s) [noise=true , address=145.79.227.16:4909 , piece_id=CF7WHNKC4LFJQRVFBPFV62JGEBLD4KVPE6Q5JBWQVADC6ZAFFLVQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.296039615 (582h31m17.16712266s) [noise=true , address=45.91.116.13:4154 , piece_id=FE4NFODQ3TWGFXARFYIL64TKK6X7H6PPLIG6LO44LKNAZOEU7DJA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.296272429 (582h31m17.167355474s) [noise=true , address=45.91.116.10:4084 , piece_id=5BFUYVGIFOYXC6RCPU7VBUM36LSM32TVBLS3W54OWFNR4SM6GCSA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.296403588 (582h31m17.167486633s) [noise=true , address=77.90.31.19:4844 , piece_id=CDOWECXLSRPACV4HMAH57UK4EEIYNOMH6GYV6HFHV7CWSEAJ44FA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.296646560 (582h31m17.167729596s) [noise=true , address=77.90.31.14:4661 , piece_id=B7IDSCDHBUXUIPSUT3HMQSCMPGBMYASZSQO2EJBXRR2DQHCLWGYA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.296906235 (582h31m17.167989291s) [noise=true , address=145.79.227.47:4265 , piece_id=FYZA2LRIFKLKVUCUAWMNWTUH3VYR5VFAA7XKAYIMD2E2VOJTI7PA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.296989804 (582h31m17.168072849s) [noise=true , address=145.79.227.46:4601 , piece_id=YJTMZID24VBM5ZZSUUHOCZSZSSNGTFHOSKK6OI5QP3A4G6ZWVWQA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.297235221 (582h31m17.168318267s) [noise=true , address=145.79.227.3:4150 , piece_id=HS4S5UDYD2VWI2N4MH6EWEYQ5G6KCFCJBDZR7AVWPXRYALFXP4FQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.297424432 (582h31m17.168507467s) [noise=true , address=145.79.227.13:4803 , piece_id=YIEUZEDLSC5C67SCFEPCHHZSTYP6MYLYJZJTGZF3UP6JWDTGUOPQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.297606900 (582h31m17.168689945s) [noise=true , address=45.91.116.14:4190 , piece_id=FIL6AIXGE7DE2M4QBVSIO53KGY43VFPZQ2FTXSWJFPJU4VZCKUXQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:18.995364155 (582h31m18.866447201s) [upload=510642]: Next returned nil segment Dec 29 22:00:18.995383603 (582h31m18.866466648s) [upload=510642]: Waiting for error group managing segments... Dec 29 22:00:19.284009408 (582h31m19.155092453s) [noise=true , address=145.79.227.51:4411 , piece_id=HTZQ6SW6KSACLQF7G4CFGQA572J5JVHMLGEWQYSSGZRYX7FJ6DXA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.295985232 (582h31m19.167068277s) [noise=true , address=145.79.227.13:4803 , piece_id=YIEUZEDLSC5C67SCFEPCHHZSTYP6MYLYJZJTGZF3UP6JWDTGUOPQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.296135458 (582h31m19.167218503s) [noise=true , address=145.79.227.16:4909 , piece_id=CF7WHNKC4LFJQRVFBPFV62JGEBLD4KVPE6Q5JBWQVADC6ZAFFLVQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.310119916 (582h31m19.181202971s) [noise=true , address=145.79.227.46:4601 , piece_id=YJTMZID24VBM5ZZSUUHOCZSZSSNGTFHOSKK6OI5QP3A4G6ZWVWQA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.310998939 (582h31m19.182081984s) [noise=true , address=145.79.227.31:4797 , piece_id=NI3M5NVVVX5HUXZ2SOMQD4S5ZOSSNPCD2F3Z5X2BA2VXMGT57AYQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.318707138 (582h31m19.189790183s) [noise=true , address=145.79.227.47:4265 , piece_id=FYZA2LRIFKLKVUCUAWMNWTUH3VYR5VFAA7XKAYIMD2E2VOJTI7PA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.322059381 (582h31m19.193142426s) [noise=true , address=145.79.227.5:4221 , piece_id=LD3FENLBM2L5N7QDOAMAYJUVH54QVSJQJFP7E36UAH2ZD7R65CCQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.506372108 (582h31m19.377455153s) [noise=true , address=77.90.31.19:4844 , piece_id=CDOWECXLSRPACV4HMAH57UK4EEIYNOMH6GYV6HFHV7CWSEAJ44FA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.507172602 (582h31m19.378255648s) [noise=true , address=77.90.31.11:4553 , piece_id=3Y5HSD73DLZ6ALJ5TFRIVTLOE4ZNZ256YSZ2W3X5LPKL5J5LX2EQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.527548842 (582h31m19.398631887s) [noise=true , address=77.90.31.14:4661 , piece_id=B7IDSCDHBUXUIPSUT3HMQSCMPGBMYASZSQO2EJBXRR2DQHCLWGYA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.531443508 (582h31m19.402526553s) [noise=true , address=45.91.116.5:4297 , piece_id=PLUQ7Q32BTGYFQRGVJFAJWHPZCPQTC5RX5UX27S5HJ5NNJPLDMPQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.552364906 (582h31m19.423447951s) [noise=true , address=45.91.116.14:4190 , piece_id=FIL6AIXGE7DE2M4QBVSIO53KGY43VFPZQ2FTXSWJFPJU4VZCKUXQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.564056769 (582h31m19.435139814s) [noise=true , address=45.91.116.10:4084 , piece_id=5BFUYVGIFOYXC6RCPU7VBUM36LSM32TVBLS3W54OWFNR4SM6GCSA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.564087016 (582h31m19.435170051s) [seg_pos={0 0}, upload=510642]: Segment reached optimal threshold of 13 pieces. Dec 29 22:00:19.564294431 (582h31m19.435377466s) [noise=true , address=145.79.227.3:4150 , piece_id=HS4S5UDYD2VWI2N4MH6EWEYQ5G6KCFCJBDZR7AVWPXRYALFXP4FQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK): protocol: expected piece hash; context canceled; EOF Dec 29 22:00:19.564302366 (582h31m19.435385411s) [noise=true , address=145.79.227.3:4150 , piece_id=HS4S5UDYD2VWI2N4MH6EWEYQ5G6KCFCJBDZR7AVWPXRYALFXP4FQ, seg_pos={0 0}, upload=510642]: Long tail cancellation due to optimal threshold Dec 29 22:00:19.564388932 (582h31m19.435471977s) [noise=true , address=45.91.116.13:4154 , piece_id=FE4NFODQ3TWGFXARFYIL64TKK6X7H6PPLIG6LO44LKNAZOEU7DJA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Dec 29 22:00:19.564401876 (582h31m19.435484911s) [noise=true , address=45.91.116.13:4154 , piece_id=FE4NFODQ3TWGFXARFYIL64TKK6X7H6PPLIG6LO44LKNAZOEU7DJA, seg_pos={0 0}, upload=510642]: Long tail cancellation due to optimal threshold Dec 29 22:00:19.564558945 (582h31m19.435641981s) [noise=true , address=145.79.227.2:4115 , piece_id=P2AS5B3VYO2K2AWSM5W4BXAXBHSXQQWM7JOXFPOV64NTKDFLT62Q, seg_pos={0 0}, upload=510642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: write tcp 188.166.225.170:38246->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:38246->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38246->145.79.227.2:4115: use of closed network connection Dec 29 22:00:19.564570738 (582h31m19.435653783s) [noise=true , address=145.79.227.2:4115 , piece_id=P2AS5B3VYO2K2AWSM5W4BXAXBHSXQQWM7JOXFPOV64NTKDFLT62Q, seg_pos={0 0}, upload=510642]: Long tail cancellation due to optimal threshold Dec 29 22:00:19.564601426 (582h31m19.435684471s) [seg_pos={0 0}, upload=510642]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 22:00:19.564683973 (582h31m19.435767018s) [upload=510642]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 22:00:19.564690004 (582h31m19.435773049s) [upload=510642]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 22:00:19.871860472 (582h31m19.742943527s) [upload=510642]: Done uploading segments... Dec 29 22:00:19.871892413 (582h31m19.742975468s) [upload=510642]: Upload finished. err: <nil> Duration: 5.957601757s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.X..X........X............................X.................] duration: 6.12484827s trace: https://jaeger.dev.storj.io/trace/5209b3d035311fd FAIL (2025-12-29 22:13:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :00:17.043136250 (582h31m16.914219305s) [upload=510642]: Starting upload Dec 29 22:00:17.043227904 (582h31m16.91431096s) [upload=510642]: Done starting upload Dec 29 22:00:17.043335189 (582h31m16.914418234s) [upload=510642]: Uploading segments... Dec 29 22:00:17.043587749 (582h31m16.914670824s) [upload=510642]: Got next segment. Inline: false Dec 29 22:00:17.043644478 (582h31m16.914727523s) [upload=510642]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 22:00:17.043686427 (582h31m16.914769503s) [upload=510642]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 22:00:17.294384363 (582h31m17.165467408s) [seg_pos={0 0}, upload=510642]: Begin upload segment... Dec 29 22:00:17.294445219 (582h31m17.165528264s) [seg_pos={0 0}, upload=510642]: Done begin upload segment. Dec 29 22:00:17.294484133 (582h31m17.165567178s) [noise=true , address=145.79.227.31:4797 , piece_id=NI3M5NVVVX5HUXZ2SOMQD4S5ZOSSNPCD2F3Z5X2BA2VXMGT57AYQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.294560979 (582h31m17.165644014s) [noise=true , address=77.90.31.11:4553 , piece_id=3Y5HSD73DLZ6ALJ5TFRIVTLOE4ZNZ256YSZ2W3X5LPKL5J5LX2EQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.295149941 (582h31m17.166232976s) [noise=true , address=45.91.116.5:4297 , piece_id=PLUQ7Q32BTGYFQRGVJFAJWHPZCPQTC5RX5UX27S5HJ5NNJPLDMPQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.295419574 (582h31m17.166502609s) [noise=true , address=145.79.227.2:4115 , piece_id=P2AS5B3VYO2K2AWSM5W4BXAXBHSXQQWM7JOXFPOV64NTKDFLT62Q, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.295483997 (582h31m17.166567042s) [noise=true , address=145.79.227.51:4411 , piece_id=HTZQ6SW6KSACLQF7G4CFGQA572J5JVHMLGEWQYSSGZRYX7FJ6DXA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.295762207 (582h31m17.166845252s) [noise=true , address=145.79.227.5:4221 , piece_id=LD3FENLBM2L5N7QDOAMAYJUVH54QVSJQJFP7E36UAH2ZD7R65CCQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.295928003 (582h31m17.167011048s) [noise=true , address=145.79.227.16:4909 , piece_id=CF7WHNKC4LFJQRVFBPFV62JGEBLD4KVPE6Q5JBWQVADC6ZAFFLVQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.296039615 (582h31m17.16712266s) [noise=true , address=45.91.116.13:4154 , piece_id=FE4NFODQ3TWGFXARFYIL64TKK6X7H6PPLIG6LO44LKNAZOEU7DJA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.296272429 (582h31m17.167355474s) [noise=true , address=45.91.116.10:4084 , piece_id=5BFUYVGIFOYXC6RCPU7VBUM36LSM32TVBLS3W54OWFNR4SM6GCSA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.296403588 (582h31m17.167486633s) [noise=true , address=77.90.31.19:4844 , piece_id=CDOWECXLSRPACV4HMAH57UK4EEIYNOMH6GYV6HFHV7CWSEAJ44FA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.296646560 (582h31m17.167729596s) [noise=true , address=77.90.31.14:4661 , piece_id=B7IDSCDHBUXUIPSUT3HMQSCMPGBMYASZSQO2EJBXRR2DQHCLWGYA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.296906235 (582h31m17.167989291s) [noise=true , address=145.79.227.47:4265 , piece_id=FYZA2LRIFKLKVUCUAWMNWTUH3VYR5VFAA7XKAYIMD2E2VOJTI7PA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.296989804 (582h31m17.168072849s) [noise=true , address=145.79.227.46:4601 , piece_id=YJTMZID24VBM5ZZSUUHOCZSZSSNGTFHOSKK6OI5QP3A4G6ZWVWQA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.297235221 (582h31m17.168318267s) [noise=true , address=145.79.227.3:4150 , piece_id=HS4S5UDYD2VWI2N4MH6EWEYQ5G6KCFCJBDZR7AVWPXRYALFXP4FQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.297424432 (582h31m17.168507467s) [noise=true , address=145.79.227.13:4803 , piece_id=YIEUZEDLSC5C67SCFEPCHHZSTYP6MYLYJZJTGZF3UP6JWDTGUOPQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.297606900 (582h31m17.168689945s) [noise=true , address=45.91.116.14:4190 , piece_id=FIL6AIXGE7DE2M4QBVSIO53KGY43VFPZQ2FTXSWJFPJU4VZCKUXQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:18.995364155 (582h31m18.866447201s) [upload=510642]: Next returned nil segment Dec 29 22:00:18.995383603 (582h31m18.866466648s) [upload=510642]: Waiting for error group managing segments... Dec 29 22:00:19.284009408 (582h31m19.155092453s) [noise=true , address=145.79.227.51:4411 , piece_id=HTZQ6SW6KSACLQF7G4CFGQA572J5JVHMLGEWQYSSGZRYX7FJ6DXA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.295985232 (582h31m19.167068277s) [noise=true , address=145.79.227.13:4803 , piece_id=YIEUZEDLSC5C67SCFEPCHHZSTYP6MYLYJZJTGZF3UP6JWDTGUOPQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.296135458 (582h31m19.167218503s) [noise=true , address=145.79.227.16:4909 , piece_id=CF7WHNKC4LFJQRVFBPFV62JGEBLD4KVPE6Q5JBWQVADC6ZAFFLVQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.310119916 (582h31m19.181202971s) [noise=true , address=145.79.227.46:4601 , piece_id=YJTMZID24VBM5ZZSUUHOCZSZSSNGTFHOSKK6OI5QP3A4G6ZWVWQA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.310998939 (582h31m19.182081984s) [noise=true , address=145.79.227.31:4797 , piece_id=NI3M5NVVVX5HUXZ2SOMQD4S5ZOSSNPCD2F3Z5X2BA2VXMGT57AYQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.318707138 (582h31m19.189790183s) [noise=true , address=145.79.227.47:4265 , piece_id=FYZA2LRIFKLKVUCUAWMNWTUH3VYR5VFAA7XKAYIMD2E2VOJTI7PA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.322059381 (582h31m19.193142426s) [noise=true , address=145.79.227.5:4221 , piece_id=LD3FENLBM2L5N7QDOAMAYJUVH54QVSJQJFP7E36UAH2ZD7R65CCQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.506372108 (582h31m19.377455153s) [noise=true , address=77.90.31.19:4844 , piece_id=CDOWECXLSRPACV4HMAH57UK4EEIYNOMH6GYV6HFHV7CWSEAJ44FA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.507172602 (582h31m19.378255648s) [noise=true , address=77.90.31.11:4553 , piece_id=3Y5HSD73DLZ6ALJ5TFRIVTLOE4ZNZ256YSZ2W3X5LPKL5J5LX2EQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.527548842 (582h31m19.398631887s) [noise=true , address=77.90.31.14:4661 , piece_id=B7IDSCDHBUXUIPSUT3HMQSCMPGBMYASZSQO2EJBXRR2DQHCLWGYA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.531443508 (582h31m19.402526553s) [noise=true , address=45.91.116.5:4297 , piece_id=PLUQ7Q32BTGYFQRGVJFAJWHPZCPQTC5RX5UX27S5HJ5NNJPLDMPQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.552364906 (582h31m19.423447951s) [noise=true , address=45.91.116.14:4190 , piece_id=FIL6AIXGE7DE2M4QBVSIO53KGY43VFPZQ2FTXSWJFPJU4VZCKUXQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.564056769 (582h31m19.435139814s) [noise=true , address=45.91.116.10:4084 , piece_id=5BFUYVGIFOYXC6RCPU7VBUM36LSM32TVBLS3W54OWFNR4SM6GCSA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.564087016 (582h31m19.435170051s) [seg_pos={0 0}, upload=510642]: Segment reached optimal threshold of 13 pieces. Dec 29 22:00:19.564294431 (582h31m19.435377466s) [noise=true , address=145.79.227.3:4150 , piece_id=HS4S5UDYD2VWI2N4MH6EWEYQ5G6KCFCJBDZR7AVWPXRYALFXP4FQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK): protocol: expected piece hash; context canceled; EOF Dec 29 22:00:19.564302366 (582h31m19.435385411s) [noise=true , address=145.79.227.3:4150 , piece_id=HS4S5UDYD2VWI2N4MH6EWEYQ5G6KCFCJBDZR7AVWPXRYALFXP4FQ, seg_pos={0 0}, upload=510642]: Long tail cancellation due to optimal threshold Dec 29 22:00:19.564388932 (582h31m19.435471977s) [noise=true , address=45.91.116.13:4154 , piece_id=FE4NFODQ3TWGFXARFYIL64TKK6X7H6PPLIG6LO44LKNAZOEU7DJA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Dec 29 22:00:19.564401876 (582h31m19.435484911s) [noise=true , address=45.91.116.13:4154 , piece_id=FE4NFODQ3TWGFXARFYIL64TKK6X7H6PPLIG6LO44LKNAZOEU7DJA, seg_pos={0 0}, upload=510642]: Long tail cancellation due to optimal threshold Dec 29 22:00:19.564558945 (582h31m19.435641981s) [noise=true , address=145.79.227.2:4115 , piece_id=P2AS5B3VYO2K2AWSM5W4BXAXBHSXQQWM7JOXFPOV64NTKDFLT62Q, seg_pos={0 0}, upload=510642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: write tcp 188.166.225.170:38246->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:38246->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38246->145.79.227.2:4115: use of closed network connection Dec 29 22:00:19.564570738 (582h31m19.435653783s) [noise=true , address=145.79.227.2:4115 , piece_id=P2AS5B3VYO2K2AWSM5W4BXAXBHSXQQWM7JOXFPOV64NTKDFLT62Q, seg_pos={0 0}, upload=510642]: Long tail cancellation due to optimal threshold Dec 29 22:00:19.564601426 (582h31m19.435684471s) [seg_pos={0 0}, upload=510642]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 22:00:19.564683973 (582h31m19.435767018s) [upload=510642]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 22:00:19.564690004 (582h31m19.435773049s) [upload=510642]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 22:00:19.871860472 (582h31m19.742943527s) [upload=510642]: Done uploading segments... Dec 29 22:00:19.871892413 (582h31m19.742975468s) [upload=510642]: Upload finished. err: <nil> Duration: 6.124875871s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................X............................X.........] duration: 6.764745451s trace: https://jaeger.dev.storj.io/trace/7096862780f29049 FAIL (2025-12-29 22:13:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== oise=true , address=145.79.227.30:4766 , piece_id=YWDMUGNTYVXWOCMSN63VIC6HOTWHPY7VJ7QWT76P63UAJIBS2XLA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.060493636 (582h37m18.931576671s) [noise=true , address=5.231.73.5:4022 , piece_id=5WVAJH3Y6TLATXY4IYB6ZADKKJDXDS52B2ZN6FJAZ7UHFVRYRRAA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.060685291 (582h37m18.931768326s) [noise=true , address=77.90.31.11:4579 , piece_id=DEJPYHGYNAP52CZ45BKGNXJM5MLIA2OQZ4BRKN5PBADNRVE3VLEA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.060749954 (582h37m18.931832989s) [noise=true , address=145.79.227.29:4976 , piece_id=YRCRXSSQOXPOJGVX46L7WCB3TRDZ5QB755SEOTNHSCHN4ZDMCOUQ, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.061033203 (582h37m18.932116248s) [noise=true , address=193.149.164.143:4698 , piece_id=6X5KB4TXFV4ZOF5JTPNRW4Q3GWNCVCXCA4JSB3AZURBHXY2VHTRQ, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.061248855 (582h37m18.9323319s) [noise=true , address=145.79.227.49:4347 , piece_id=75B3DRRHEPDL6EK6ZUXTKASXUKWGAVUWBPSTNRYT4BJMAU5ALXRA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.061382729 (582h37m18.932465764s) [noise=true , address=5.231.73.2:4004 , piece_id=PTHHO3XEPGGCIRCUBIWHN4LVNBLMISJM5LESQGR2ERRESFK3S4RA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.061645560 (582h37m18.932728595s) [noise=true , address=5.231.73.3:4007 , piece_id=6AF3CEPLUMRO4CDW3SRWZSYHWEXP35AVSM5M3LH4KCRARSZ6NFNA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.061769094 (582h37m18.932852139s) [noise=true , address=145.79.227.7:4321 , piece_id=SPWPZ5MR5ZCQNYVYYQQEZC4OURAM5VAH3WN5UP6HMGCCDMCH6ZGA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.061970298 (582h37m18.933053333s) [noise=true , address=145.79.227.35:4507 , piece_id=ZZSVIHOAN2SWXI4NS3TRD4B64NDB4D43J63DAS74TCPEE6MPGRXA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.062237577 (582h37m18.933320612s) [noise=true , address=145.79.227.35:4489 , piece_id=BM7ZQESTHEUIRJWAPORTKDMT5TVLEXMOHULGFVEQKHT4AJV3HT5A, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.062304304 (582h37m18.933387349s) [noise=true , address=145.79.227.34:4931 , piece_id=XJHLE3B5HIF4RC4LV32SBCNMVTOTEXWBBCXBVPQFN6N5HFQ6ZS7A, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.062522640 (582h37m18.933605685s) [noise=true , address=145.79.227.48:4329 , piece_id=YWCVTYLCAGU57SE3ID3WW2JLCJGTJ7CPDRCZENAMO42BVPIAVLIQ, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.062904968 (582h37m18.933988003s) [noise=true , address=77.90.31.17:4801 , piece_id=K3QD7FPVRQHZL5R57EKKAC27RGDJQH6HDCND7FIVXSZBRH42WSIA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.063420419 (582h37m18.934503464s) [noise=true , address=145.79.227.17:4995 , piece_id=JJTVPVBAMCTDGHGWNPUVPYFVKD4SMTDAHF4VWVIW7JFUAMBIPRHQ, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.063943495 (582h37m18.93502654s) [noise=true , address=5.231.73.4:4018 , piece_id=X45H47HER27IM5FEQSEAMXSDX7UQG3AEOY65Q4WCCUOX7SVFZ5KA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:20.386418430 (582h37m20.257501465s) [upload=510734]: Next returned nil segment Dec 29 22:06:20.386428990 (582h37m20.257512025s) [upload=510734]: Waiting for error group managing segments... Dec 29 22:06:20.935543714 (582h37m20.806626749s) [noise=true , address=145.79.227.48:4329 , piece_id=YWCVTYLCAGU57SE3ID3WW2JLCJGTJ7CPDRCZENAMO42BVPIAVLIQ, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:20.978073277 (582h37m20.849156312s) [noise=true , address=145.79.227.39:4022 , piece_id=5IJCSVRVXCICRNRB3TTJAVNRF5FTUQMMY4KL7ZFF4SBKILFGHJWA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.079004013 (582h37m20.950087058s) [noise=true , address=5.231.73.4:4018 , piece_id=X45H47HER27IM5FEQSEAMXSDX7UQG3AEOY65Q4WCCUOX7SVFZ5KA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.101144552 (582h37m20.972227597s) [noise=true , address=77.90.31.11:4579 , piece_id=DEJPYHGYNAP52CZ45BKGNXJM5MLIA2OQZ4BRKN5PBADNRVE3VLEA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.138766369 (582h37m21.009849405s) [noise=true , address=77.90.31.17:4801 , piece_id=K3QD7FPVRQHZL5R57EKKAC27RGDJQH6HDCND7FIVXSZBRH42WSIA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.231047035 (582h37m21.10213007s) [noise=true , address=77.90.31.19:4857 , piece_id=DMJ4O3TWZYWKWVLJFSQBULHDFWZCNLVDBIGYJEOBAVVCU5UGG27Q, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.272062706 (582h37m21.143145741s) [noise=true , address=5.231.73.3:4007 , piece_id=6AF3CEPLUMRO4CDW3SRWZSYHWEXP35AVSM5M3LH4KCRARSZ6NFNA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.303520666 (582h37m21.174603711s) [noise=true , address=145.79.227.29:4976 , piece_id=YRCRXSSQOXPOJGVX46L7WCB3TRDZ5QB755SEOTNHSCHN4ZDMCOUQ, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.340640177 (582h37m21.211723212s) [noise=true , address=5.231.73.5:4022 , piece_id=5WVAJH3Y6TLATXY4IYB6ZADKKJDXDS52B2ZN6FJAZ7UHFVRYRRAA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.481118228 (582h37m21.352201263s) [noise=true , address=5.231.73.2:4004 , piece_id=PTHHO3XEPGGCIRCUBIWHN4LVNBLMISJM5LESQGR2ERRESFK3S4RA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.896953769 (582h37m21.768036804s) [noise=true , address=145.79.227.49:4347 , piece_id=75B3DRRHEPDL6EK6ZUXTKASXUKWGAVUWBPSTNRYT4BJMAU5ALXRA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.911314353 (582h37m21.782397399s) [noise=true , address=145.79.227.34:4931 , piece_id=XJHLE3B5HIF4RC4LV32SBCNMVTOTEXWBBCXBVPQFN6N5HFQ6ZS7A, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.915845371 (582h37m21.786928406s) [noise=true , address=145.79.227.30:4766 , piece_id=YWDMUGNTYVXWOCMSN63VIC6HOTWHPY7VJ7QWT76P63UAJIBS2XLA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.928785972 (582h37m21.799869017s) [noise=true , address=145.79.227.35:4489 , piece_id=BM7ZQESTHEUIRJWAPORTKDMT5TVLEXMOHULGFVEQKHT4AJV3HT5A, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.928810859 (582h37m21.799893904s) [seg_pos={0 0}, upload=510734]: Segment reached optimal threshold of 14 pieces. Dec 29 22:06:21.929016471 (582h37m21.800099516s) [noise=true , address=145.79.227.7:4321 , piece_id=SPWPZ5MR5ZCQNYVYYQQEZC4OURAM5VAH3WN5UP6HMGCCDMCH6ZGA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12edsHqD9iRBDgMU8vYhSM1EWG8Sn57GZ8TPiTytri3rp7W9Eae): protocol: expected piece hash; context canceled; EOF Dec 29 22:06:21.929028644 (582h37m21.800111709s) [noise=true , address=145.79.227.7:4321 , piece_id=SPWPZ5MR5ZCQNYVYYQQEZC4OURAM5VAH3WN5UP6HMGCCDMCH6ZGA, seg_pos={0 0}, upload=510734]: Long tail cancellation due to optimal threshold Dec 29 22:06:21.929093137 (582h37m21.800176192s) [noise=true , address=145.79.227.35:4507 , piece_id=ZZSVIHOAN2SWXI4NS3TRD4B64NDB4D43J63DAS74TCPEE6MPGRXA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14jurWWPpmxwiBS3XmZysEUwugsceYJqVXFj7BbYpx8UFcTR8R): protocol: expected piece hash; context canceled; EOF Dec 29 22:06:21.929106793 (582h37m21.800189838s) [noise=true , address=145.79.227.35:4507 , piece_id=ZZSVIHOAN2SWXI4NS3TRD4B64NDB4D43J63DAS74TCPEE6MPGRXA, seg_pos={0 0}, upload=510734]: Long tail cancellation due to optimal threshold Dec 29 22:06:21.929206612 (582h37m21.800289657s) [noise=true , address=145.79.227.17:4995 , piece_id=JJTVPVBAMCTDGHGWNPUVPYFVKD4SMTDAHF4VWVIW7JFUAMBIPRHQ, seg_pos={0 0}, upload=510734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tvoTD1LoSj3q2rLiDckfT56dXNKKwdVu1ZAsBZExNFVTom9tH): protocol: write tcp 188.166.225.170:47696->145.79.227.17:4995: use of closed network connection; write tcp 188.166.225.170:47696->145.79.227.17:4995: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47696->145.79.227.17:4995: use of closed network connection Dec 29 22:06:21.929215911 (582h37m21.800298976s) [noise=true , address=145.79.227.17:4995 , piece_id=JJTVPVBAMCTDGHGWNPUVPYFVKD4SMTDAHF4VWVIW7JFUAMBIPRHQ, seg_pos={0 0}, upload=510734]: Long tail cancellation due to optimal threshold Dec 29 22:06:21.929387247 (582h37m21.800470292s) [noise=true , address=193.149.164.143:4698 , piece_id=6X5KB4TXFV4ZOF5JTPNRW4Q3GWNCVCXCA4JSB3AZURBHXY2VHTRQ, seg_pos={0 0}, upload=510734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F4fJAGJM2ZPRt7vJH2cZmEbYeujmxcfi7aPZ2g9VPTNYBYRAi): protocol: expected piece hash; context canceled; EOF Dec 29 22:06:21.929396174 (582h37m21.800479219s) [noise=true , address=193.149.164.143:4698 , piece_id=6X5KB4TXFV4ZOF5JTPNRW4Q3GWNCVCXCA4JSB3AZURBHXY2VHTRQ, seg_pos={0 0}, upload=510734]: Long tail cancellation due to optimal threshold Dec 29 22:06:21.929491536 (582h37m21.800574571s) [seg_pos={0 0}, upload=510734]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 22:06:21.929573792 (582h37m21.800656837s) [upload=510734]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 22:06:21.929580555 (582h37m21.8006636s) [upload=510734]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 22:06:22.233045366 (582h37m22.104128401s) [upload=510734]: Done uploading segments... Dec 29 22:06:22.233064963 (582h37m22.104147998s) [upload=510734]: Upload finished. err: <nil> Duration: 6.764763907s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................X............................X........] duration: 15.134454562s trace: https://jaeger.dev.storj.io/trace/7096862780f29049 FAIL (2025-12-29 22:13:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== oise=true , address=145.79.227.30:4766 , piece_id=YWDMUGNTYVXWOCMSN63VIC6HOTWHPY7VJ7QWT76P63UAJIBS2XLA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.060493636 (582h37m18.931576671s) [noise=true , address=5.231.73.5:4022 , piece_id=5WVAJH3Y6TLATXY4IYB6ZADKKJDXDS52B2ZN6FJAZ7UHFVRYRRAA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.060685291 (582h37m18.931768326s) [noise=true , address=77.90.31.11:4579 , piece_id=DEJPYHGYNAP52CZ45BKGNXJM5MLIA2OQZ4BRKN5PBADNRVE3VLEA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.060749954 (582h37m18.931832989s) [noise=true , address=145.79.227.29:4976 , piece_id=YRCRXSSQOXPOJGVX46L7WCB3TRDZ5QB755SEOTNHSCHN4ZDMCOUQ, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.061033203 (582h37m18.932116248s) [noise=true , address=193.149.164.143:4698 , piece_id=6X5KB4TXFV4ZOF5JTPNRW4Q3GWNCVCXCA4JSB3AZURBHXY2VHTRQ, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.061248855 (582h37m18.9323319s) [noise=true , address=145.79.227.49:4347 , piece_id=75B3DRRHEPDL6EK6ZUXTKASXUKWGAVUWBPSTNRYT4BJMAU5ALXRA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.061382729 (582h37m18.932465764s) [noise=true , address=5.231.73.2:4004 , piece_id=PTHHO3XEPGGCIRCUBIWHN4LVNBLMISJM5LESQGR2ERRESFK3S4RA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.061645560 (582h37m18.932728595s) [noise=true , address=5.231.73.3:4007 , piece_id=6AF3CEPLUMRO4CDW3SRWZSYHWEXP35AVSM5M3LH4KCRARSZ6NFNA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.061769094 (582h37m18.932852139s) [noise=true , address=145.79.227.7:4321 , piece_id=SPWPZ5MR5ZCQNYVYYQQEZC4OURAM5VAH3WN5UP6HMGCCDMCH6ZGA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.061970298 (582h37m18.933053333s) [noise=true , address=145.79.227.35:4507 , piece_id=ZZSVIHOAN2SWXI4NS3TRD4B64NDB4D43J63DAS74TCPEE6MPGRXA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.062237577 (582h37m18.933320612s) [noise=true , address=145.79.227.35:4489 , piece_id=BM7ZQESTHEUIRJWAPORTKDMT5TVLEXMOHULGFVEQKHT4AJV3HT5A, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.062304304 (582h37m18.933387349s) [noise=true , address=145.79.227.34:4931 , piece_id=XJHLE3B5HIF4RC4LV32SBCNMVTOTEXWBBCXBVPQFN6N5HFQ6ZS7A, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.062522640 (582h37m18.933605685s) [noise=true , address=145.79.227.48:4329 , piece_id=YWCVTYLCAGU57SE3ID3WW2JLCJGTJ7CPDRCZENAMO42BVPIAVLIQ, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.062904968 (582h37m18.933988003s) [noise=true , address=77.90.31.17:4801 , piece_id=K3QD7FPVRQHZL5R57EKKAC27RGDJQH6HDCND7FIVXSZBRH42WSIA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.063420419 (582h37m18.934503464s) [noise=true , address=145.79.227.17:4995 , piece_id=JJTVPVBAMCTDGHGWNPUVPYFVKD4SMTDAHF4VWVIW7JFUAMBIPRHQ, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.063943495 (582h37m18.93502654s) [noise=true , address=5.231.73.4:4018 , piece_id=X45H47HER27IM5FEQSEAMXSDX7UQG3AEOY65Q4WCCUOX7SVFZ5KA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:20.386418430 (582h37m20.257501465s) [upload=510734]: Next returned nil segment Dec 29 22:06:20.386428990 (582h37m20.257512025s) [upload=510734]: Waiting for error group managing segments... Dec 29 22:06:20.935543714 (582h37m20.806626749s) [noise=true , address=145.79.227.48:4329 , piece_id=YWCVTYLCAGU57SE3ID3WW2JLCJGTJ7CPDRCZENAMO42BVPIAVLIQ, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:20.978073277 (582h37m20.849156312s) [noise=true , address=145.79.227.39:4022 , piece_id=5IJCSVRVXCICRNRB3TTJAVNRF5FTUQMMY4KL7ZFF4SBKILFGHJWA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.079004013 (582h37m20.950087058s) [noise=true , address=5.231.73.4:4018 , piece_id=X45H47HER27IM5FEQSEAMXSDX7UQG3AEOY65Q4WCCUOX7SVFZ5KA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.101144552 (582h37m20.972227597s) [noise=true , address=77.90.31.11:4579 , piece_id=DEJPYHGYNAP52CZ45BKGNXJM5MLIA2OQZ4BRKN5PBADNRVE3VLEA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.138766369 (582h37m21.009849405s) [noise=true , address=77.90.31.17:4801 , piece_id=K3QD7FPVRQHZL5R57EKKAC27RGDJQH6HDCND7FIVXSZBRH42WSIA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.231047035 (582h37m21.10213007s) [noise=true , address=77.90.31.19:4857 , piece_id=DMJ4O3TWZYWKWVLJFSQBULHDFWZCNLVDBIGYJEOBAVVCU5UGG27Q, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.272062706 (582h37m21.143145741s) [noise=true , address=5.231.73.3:4007 , piece_id=6AF3CEPLUMRO4CDW3SRWZSYHWEXP35AVSM5M3LH4KCRARSZ6NFNA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.303520666 (582h37m21.174603711s) [noise=true , address=145.79.227.29:4976 , piece_id=YRCRXSSQOXPOJGVX46L7WCB3TRDZ5QB755SEOTNHSCHN4ZDMCOUQ, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.340640177 (582h37m21.211723212s) [noise=true , address=5.231.73.5:4022 , piece_id=5WVAJH3Y6TLATXY4IYB6ZADKKJDXDS52B2ZN6FJAZ7UHFVRYRRAA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.481118228 (582h37m21.352201263s) [noise=true , address=5.231.73.2:4004 , piece_id=PTHHO3XEPGGCIRCUBIWHN4LVNBLMISJM5LESQGR2ERRESFK3S4RA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.896953769 (582h37m21.768036804s) [noise=true , address=145.79.227.49:4347 , piece_id=75B3DRRHEPDL6EK6ZUXTKASXUKWGAVUWBPSTNRYT4BJMAU5ALXRA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.911314353 (582h37m21.782397399s) [noise=true , address=145.79.227.34:4931 , piece_id=XJHLE3B5HIF4RC4LV32SBCNMVTOTEXWBBCXBVPQFN6N5HFQ6ZS7A, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.915845371 (582h37m21.786928406s) [noise=true , address=145.79.227.30:4766 , piece_id=YWDMUGNTYVXWOCMSN63VIC6HOTWHPY7VJ7QWT76P63UAJIBS2XLA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.928785972 (582h37m21.799869017s) [noise=true , address=145.79.227.35:4489 , piece_id=BM7ZQESTHEUIRJWAPORTKDMT5TVLEXMOHULGFVEQKHT4AJV3HT5A, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.928810859 (582h37m21.799893904s) [seg_pos={0 0}, upload=510734]: Segment reached optimal threshold of 14 pieces. Dec 29 22:06:21.929016471 (582h37m21.800099516s) [noise=true , address=145.79.227.7:4321 , piece_id=SPWPZ5MR5ZCQNYVYYQQEZC4OURAM5VAH3WN5UP6HMGCCDMCH6ZGA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12edsHqD9iRBDgMU8vYhSM1EWG8Sn57GZ8TPiTytri3rp7W9Eae): protocol: expected piece hash; context canceled; EOF Dec 29 22:06:21.929028644 (582h37m21.800111709s) [noise=true , address=145.79.227.7:4321 , piece_id=SPWPZ5MR5ZCQNYVYYQQEZC4OURAM5VAH3WN5UP6HMGCCDMCH6ZGA, seg_pos={0 0}, upload=510734]: Long tail cancellation due to optimal threshold Dec 29 22:06:21.929093137 (582h37m21.800176192s) [noise=true , address=145.79.227.35:4507 , piece_id=ZZSVIHOAN2SWXI4NS3TRD4B64NDB4D43J63DAS74TCPEE6MPGRXA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14jurWWPpmxwiBS3XmZysEUwugsceYJqVXFj7BbYpx8UFcTR8R): protocol: expected piece hash; context canceled; EOF Dec 29 22:06:21.929106793 (582h37m21.800189838s) [noise=true , address=145.79.227.35:4507 , piece_id=ZZSVIHOAN2SWXI4NS3TRD4B64NDB4D43J63DAS74TCPEE6MPGRXA, seg_pos={0 0}, upload=510734]: Long tail cancellation due to optimal threshold Dec 29 22:06:21.929206612 (582h37m21.800289657s) [noise=true , address=145.79.227.17:4995 , piece_id=JJTVPVBAMCTDGHGWNPUVPYFVKD4SMTDAHF4VWVIW7JFUAMBIPRHQ, seg_pos={0 0}, upload=510734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tvoTD1LoSj3q2rLiDckfT56dXNKKwdVu1ZAsBZExNFVTom9tH): protocol: write tcp 188.166.225.170:47696->145.79.227.17:4995: use of closed network connection; write tcp 188.166.225.170:47696->145.79.227.17:4995: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47696->145.79.227.17:4995: use of closed network connection Dec 29 22:06:21.929215911 (582h37m21.800298976s) [noise=true , address=145.79.227.17:4995 , piece_id=JJTVPVBAMCTDGHGWNPUVPYFVKD4SMTDAHF4VWVIW7JFUAMBIPRHQ, seg_pos={0 0}, upload=510734]: Long tail cancellation due to optimal threshold Dec 29 22:06:21.929387247 (582h37m21.800470292s) [noise=true , address=193.149.164.143:4698 , piece_id=6X5KB4TXFV4ZOF5JTPNRW4Q3GWNCVCXCA4JSB3AZURBHXY2VHTRQ, seg_pos={0 0}, upload=510734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F4fJAGJM2ZPRt7vJH2cZmEbYeujmxcfi7aPZ2g9VPTNYBYRAi): protocol: expected piece hash; context canceled; EOF Dec 29 22:06:21.929396174 (582h37m21.800479219s) [noise=true , address=193.149.164.143:4698 , piece_id=6X5KB4TXFV4ZOF5JTPNRW4Q3GWNCVCXCA4JSB3AZURBHXY2VHTRQ, seg_pos={0 0}, upload=510734]: Long tail cancellation due to optimal threshold Dec 29 22:06:21.929491536 (582h37m21.800574571s) [seg_pos={0 0}, upload=510734]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 22:06:21.929573792 (582h37m21.800656837s) [upload=510734]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 22:06:21.929580555 (582h37m21.8006636s) [upload=510734]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 22:06:22.233045366 (582h37m22.104128401s) [upload=510734]: Done uploading segments... Dec 29 22:06:22.233064963 (582h37m22.104147998s) [upload=510734]: Upload finished. err: <nil> Duration: 15.134492626s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..X..X........X............................X................] duration: 6.889062256s trace: https://jaeger.dev.storj.io/trace/5209b3d035311fd FAIL (2025-12-29 22:13:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :00:17.043136250 (582h31m16.914219305s) [upload=510642]: Starting upload Dec 29 22:00:17.043227904 (582h31m16.91431096s) [upload=510642]: Done starting upload Dec 29 22:00:17.043335189 (582h31m16.914418234s) [upload=510642]: Uploading segments... Dec 29 22:00:17.043587749 (582h31m16.914670824s) [upload=510642]: Got next segment. Inline: false Dec 29 22:00:17.043644478 (582h31m16.914727523s) [upload=510642]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 22:00:17.043686427 (582h31m16.914769503s) [upload=510642]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 22:00:17.294384363 (582h31m17.165467408s) [seg_pos={0 0}, upload=510642]: Begin upload segment... Dec 29 22:00:17.294445219 (582h31m17.165528264s) [seg_pos={0 0}, upload=510642]: Done begin upload segment. Dec 29 22:00:17.294484133 (582h31m17.165567178s) [noise=true , address=145.79.227.31:4797 , piece_id=NI3M5NVVVX5HUXZ2SOMQD4S5ZOSSNPCD2F3Z5X2BA2VXMGT57AYQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.294560979 (582h31m17.165644014s) [noise=true , address=77.90.31.11:4553 , piece_id=3Y5HSD73DLZ6ALJ5TFRIVTLOE4ZNZ256YSZ2W3X5LPKL5J5LX2EQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.295149941 (582h31m17.166232976s) [noise=true , address=45.91.116.5:4297 , piece_id=PLUQ7Q32BTGYFQRGVJFAJWHPZCPQTC5RX5UX27S5HJ5NNJPLDMPQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.295419574 (582h31m17.166502609s) [noise=true , address=145.79.227.2:4115 , piece_id=P2AS5B3VYO2K2AWSM5W4BXAXBHSXQQWM7JOXFPOV64NTKDFLT62Q, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.295483997 (582h31m17.166567042s) [noise=true , address=145.79.227.51:4411 , piece_id=HTZQ6SW6KSACLQF7G4CFGQA572J5JVHMLGEWQYSSGZRYX7FJ6DXA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.295762207 (582h31m17.166845252s) [noise=true , address=145.79.227.5:4221 , piece_id=LD3FENLBM2L5N7QDOAMAYJUVH54QVSJQJFP7E36UAH2ZD7R65CCQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.295928003 (582h31m17.167011048s) [noise=true , address=145.79.227.16:4909 , piece_id=CF7WHNKC4LFJQRVFBPFV62JGEBLD4KVPE6Q5JBWQVADC6ZAFFLVQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.296039615 (582h31m17.16712266s) [noise=true , address=45.91.116.13:4154 , piece_id=FE4NFODQ3TWGFXARFYIL64TKK6X7H6PPLIG6LO44LKNAZOEU7DJA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.296272429 (582h31m17.167355474s) [noise=true , address=45.91.116.10:4084 , piece_id=5BFUYVGIFOYXC6RCPU7VBUM36LSM32TVBLS3W54OWFNR4SM6GCSA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.296403588 (582h31m17.167486633s) [noise=true , address=77.90.31.19:4844 , piece_id=CDOWECXLSRPACV4HMAH57UK4EEIYNOMH6GYV6HFHV7CWSEAJ44FA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.296646560 (582h31m17.167729596s) [noise=true , address=77.90.31.14:4661 , piece_id=B7IDSCDHBUXUIPSUT3HMQSCMPGBMYASZSQO2EJBXRR2DQHCLWGYA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.296906235 (582h31m17.167989291s) [noise=true , address=145.79.227.47:4265 , piece_id=FYZA2LRIFKLKVUCUAWMNWTUH3VYR5VFAA7XKAYIMD2E2VOJTI7PA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.296989804 (582h31m17.168072849s) [noise=true , address=145.79.227.46:4601 , piece_id=YJTMZID24VBM5ZZSUUHOCZSZSSNGTFHOSKK6OI5QP3A4G6ZWVWQA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.297235221 (582h31m17.168318267s) [noise=true , address=145.79.227.3:4150 , piece_id=HS4S5UDYD2VWI2N4MH6EWEYQ5G6KCFCJBDZR7AVWPXRYALFXP4FQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.297424432 (582h31m17.168507467s) [noise=true , address=145.79.227.13:4803 , piece_id=YIEUZEDLSC5C67SCFEPCHHZSTYP6MYLYJZJTGZF3UP6JWDTGUOPQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.297606900 (582h31m17.168689945s) [noise=true , address=45.91.116.14:4190 , piece_id=FIL6AIXGE7DE2M4QBVSIO53KGY43VFPZQ2FTXSWJFPJU4VZCKUXQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:18.995364155 (582h31m18.866447201s) [upload=510642]: Next returned nil segment Dec 29 22:00:18.995383603 (582h31m18.866466648s) [upload=510642]: Waiting for error group managing segments... Dec 29 22:00:19.284009408 (582h31m19.155092453s) [noise=true , address=145.79.227.51:4411 , piece_id=HTZQ6SW6KSACLQF7G4CFGQA572J5JVHMLGEWQYSSGZRYX7FJ6DXA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.295985232 (582h31m19.167068277s) [noise=true , address=145.79.227.13:4803 , piece_id=YIEUZEDLSC5C67SCFEPCHHZSTYP6MYLYJZJTGZF3UP6JWDTGUOPQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.296135458 (582h31m19.167218503s) [noise=true , address=145.79.227.16:4909 , piece_id=CF7WHNKC4LFJQRVFBPFV62JGEBLD4KVPE6Q5JBWQVADC6ZAFFLVQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.310119916 (582h31m19.181202971s) [noise=true , address=145.79.227.46:4601 , piece_id=YJTMZID24VBM5ZZSUUHOCZSZSSNGTFHOSKK6OI5QP3A4G6ZWVWQA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.310998939 (582h31m19.182081984s) [noise=true , address=145.79.227.31:4797 , piece_id=NI3M5NVVVX5HUXZ2SOMQD4S5ZOSSNPCD2F3Z5X2BA2VXMGT57AYQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.318707138 (582h31m19.189790183s) [noise=true , address=145.79.227.47:4265 , piece_id=FYZA2LRIFKLKVUCUAWMNWTUH3VYR5VFAA7XKAYIMD2E2VOJTI7PA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.322059381 (582h31m19.193142426s) [noise=true , address=145.79.227.5:4221 , piece_id=LD3FENLBM2L5N7QDOAMAYJUVH54QVSJQJFP7E36UAH2ZD7R65CCQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.506372108 (582h31m19.377455153s) [noise=true , address=77.90.31.19:4844 , piece_id=CDOWECXLSRPACV4HMAH57UK4EEIYNOMH6GYV6HFHV7CWSEAJ44FA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.507172602 (582h31m19.378255648s) [noise=true , address=77.90.31.11:4553 , piece_id=3Y5HSD73DLZ6ALJ5TFRIVTLOE4ZNZ256YSZ2W3X5LPKL5J5LX2EQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.527548842 (582h31m19.398631887s) [noise=true , address=77.90.31.14:4661 , piece_id=B7IDSCDHBUXUIPSUT3HMQSCMPGBMYASZSQO2EJBXRR2DQHCLWGYA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.531443508 (582h31m19.402526553s) [noise=true , address=45.91.116.5:4297 , piece_id=PLUQ7Q32BTGYFQRGVJFAJWHPZCPQTC5RX5UX27S5HJ5NNJPLDMPQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.552364906 (582h31m19.423447951s) [noise=true , address=45.91.116.14:4190 , piece_id=FIL6AIXGE7DE2M4QBVSIO53KGY43VFPZQ2FTXSWJFPJU4VZCKUXQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.564056769 (582h31m19.435139814s) [noise=true , address=45.91.116.10:4084 , piece_id=5BFUYVGIFOYXC6RCPU7VBUM36LSM32TVBLS3W54OWFNR4SM6GCSA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.564087016 (582h31m19.435170051s) [seg_pos={0 0}, upload=510642]: Segment reached optimal threshold of 13 pieces. Dec 29 22:00:19.564294431 (582h31m19.435377466s) [noise=true , address=145.79.227.3:4150 , piece_id=HS4S5UDYD2VWI2N4MH6EWEYQ5G6KCFCJBDZR7AVWPXRYALFXP4FQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK): protocol: expected piece hash; context canceled; EOF Dec 29 22:00:19.564302366 (582h31m19.435385411s) [noise=true , address=145.79.227.3:4150 , piece_id=HS4S5UDYD2VWI2N4MH6EWEYQ5G6KCFCJBDZR7AVWPXRYALFXP4FQ, seg_pos={0 0}, upload=510642]: Long tail cancellation due to optimal threshold Dec 29 22:00:19.564388932 (582h31m19.435471977s) [noise=true , address=45.91.116.13:4154 , piece_id=FE4NFODQ3TWGFXARFYIL64TKK6X7H6PPLIG6LO44LKNAZOEU7DJA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Dec 29 22:00:19.564401876 (582h31m19.435484911s) [noise=true , address=45.91.116.13:4154 , piece_id=FE4NFODQ3TWGFXARFYIL64TKK6X7H6PPLIG6LO44LKNAZOEU7DJA, seg_pos={0 0}, upload=510642]: Long tail cancellation due to optimal threshold Dec 29 22:00:19.564558945 (582h31m19.435641981s) [noise=true , address=145.79.227.2:4115 , piece_id=P2AS5B3VYO2K2AWSM5W4BXAXBHSXQQWM7JOXFPOV64NTKDFLT62Q, seg_pos={0 0}, upload=510642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: write tcp 188.166.225.170:38246->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:38246->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38246->145.79.227.2:4115: use of closed network connection Dec 29 22:00:19.564570738 (582h31m19.435653783s) [noise=true , address=145.79.227.2:4115 , piece_id=P2AS5B3VYO2K2AWSM5W4BXAXBHSXQQWM7JOXFPOV64NTKDFLT62Q, seg_pos={0 0}, upload=510642]: Long tail cancellation due to optimal threshold Dec 29 22:00:19.564601426 (582h31m19.435684471s) [seg_pos={0 0}, upload=510642]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 22:00:19.564683973 (582h31m19.435767018s) [upload=510642]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 22:00:19.564690004 (582h31m19.435773049s) [upload=510642]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 22:00:19.871860472 (582h31m19.742943527s) [upload=510642]: Done uploading segments... Dec 29 22:00:19.871892413 (582h31m19.742975468s) [upload=510642]: Upload finished. err: <nil> Duration: 6.889091301s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................X............................X.......] duration: 31.563402333s trace: https://jaeger.dev.storj.io/trace/7096862780f29049 FAIL (2025-12-29 22:12:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== oise=true , address=145.79.227.30:4766 , piece_id=YWDMUGNTYVXWOCMSN63VIC6HOTWHPY7VJ7QWT76P63UAJIBS2XLA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.060493636 (582h37m18.931576671s) [noise=true , address=5.231.73.5:4022 , piece_id=5WVAJH3Y6TLATXY4IYB6ZADKKJDXDS52B2ZN6FJAZ7UHFVRYRRAA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.060685291 (582h37m18.931768326s) [noise=true , address=77.90.31.11:4579 , piece_id=DEJPYHGYNAP52CZ45BKGNXJM5MLIA2OQZ4BRKN5PBADNRVE3VLEA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.060749954 (582h37m18.931832989s) [noise=true , address=145.79.227.29:4976 , piece_id=YRCRXSSQOXPOJGVX46L7WCB3TRDZ5QB755SEOTNHSCHN4ZDMCOUQ, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.061033203 (582h37m18.932116248s) [noise=true , address=193.149.164.143:4698 , piece_id=6X5KB4TXFV4ZOF5JTPNRW4Q3GWNCVCXCA4JSB3AZURBHXY2VHTRQ, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.061248855 (582h37m18.9323319s) [noise=true , address=145.79.227.49:4347 , piece_id=75B3DRRHEPDL6EK6ZUXTKASXUKWGAVUWBPSTNRYT4BJMAU5ALXRA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.061382729 (582h37m18.932465764s) [noise=true , address=5.231.73.2:4004 , piece_id=PTHHO3XEPGGCIRCUBIWHN4LVNBLMISJM5LESQGR2ERRESFK3S4RA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.061645560 (582h37m18.932728595s) [noise=true , address=5.231.73.3:4007 , piece_id=6AF3CEPLUMRO4CDW3SRWZSYHWEXP35AVSM5M3LH4KCRARSZ6NFNA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.061769094 (582h37m18.932852139s) [noise=true , address=145.79.227.7:4321 , piece_id=SPWPZ5MR5ZCQNYVYYQQEZC4OURAM5VAH3WN5UP6HMGCCDMCH6ZGA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.061970298 (582h37m18.933053333s) [noise=true , address=145.79.227.35:4507 , piece_id=ZZSVIHOAN2SWXI4NS3TRD4B64NDB4D43J63DAS74TCPEE6MPGRXA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.062237577 (582h37m18.933320612s) [noise=true , address=145.79.227.35:4489 , piece_id=BM7ZQESTHEUIRJWAPORTKDMT5TVLEXMOHULGFVEQKHT4AJV3HT5A, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.062304304 (582h37m18.933387349s) [noise=true , address=145.79.227.34:4931 , piece_id=XJHLE3B5HIF4RC4LV32SBCNMVTOTEXWBBCXBVPQFN6N5HFQ6ZS7A, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.062522640 (582h37m18.933605685s) [noise=true , address=145.79.227.48:4329 , piece_id=YWCVTYLCAGU57SE3ID3WW2JLCJGTJ7CPDRCZENAMO42BVPIAVLIQ, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.062904968 (582h37m18.933988003s) [noise=true , address=77.90.31.17:4801 , piece_id=K3QD7FPVRQHZL5R57EKKAC27RGDJQH6HDCND7FIVXSZBRH42WSIA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.063420419 (582h37m18.934503464s) [noise=true , address=145.79.227.17:4995 , piece_id=JJTVPVBAMCTDGHGWNPUVPYFVKD4SMTDAHF4VWVIW7JFUAMBIPRHQ, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.063943495 (582h37m18.93502654s) [noise=true , address=5.231.73.4:4018 , piece_id=X45H47HER27IM5FEQSEAMXSDX7UQG3AEOY65Q4WCCUOX7SVFZ5KA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:20.386418430 (582h37m20.257501465s) [upload=510734]: Next returned nil segment Dec 29 22:06:20.386428990 (582h37m20.257512025s) [upload=510734]: Waiting for error group managing segments... Dec 29 22:06:20.935543714 (582h37m20.806626749s) [noise=true , address=145.79.227.48:4329 , piece_id=YWCVTYLCAGU57SE3ID3WW2JLCJGTJ7CPDRCZENAMO42BVPIAVLIQ, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:20.978073277 (582h37m20.849156312s) [noise=true , address=145.79.227.39:4022 , piece_id=5IJCSVRVXCICRNRB3TTJAVNRF5FTUQMMY4KL7ZFF4SBKILFGHJWA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.079004013 (582h37m20.950087058s) [noise=true , address=5.231.73.4:4018 , piece_id=X45H47HER27IM5FEQSEAMXSDX7UQG3AEOY65Q4WCCUOX7SVFZ5KA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.101144552 (582h37m20.972227597s) [noise=true , address=77.90.31.11:4579 , piece_id=DEJPYHGYNAP52CZ45BKGNXJM5MLIA2OQZ4BRKN5PBADNRVE3VLEA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.138766369 (582h37m21.009849405s) [noise=true , address=77.90.31.17:4801 , piece_id=K3QD7FPVRQHZL5R57EKKAC27RGDJQH6HDCND7FIVXSZBRH42WSIA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.231047035 (582h37m21.10213007s) [noise=true , address=77.90.31.19:4857 , piece_id=DMJ4O3TWZYWKWVLJFSQBULHDFWZCNLVDBIGYJEOBAVVCU5UGG27Q, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.272062706 (582h37m21.143145741s) [noise=true , address=5.231.73.3:4007 , piece_id=6AF3CEPLUMRO4CDW3SRWZSYHWEXP35AVSM5M3LH4KCRARSZ6NFNA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.303520666 (582h37m21.174603711s) [noise=true , address=145.79.227.29:4976 , piece_id=YRCRXSSQOXPOJGVX46L7WCB3TRDZ5QB755SEOTNHSCHN4ZDMCOUQ, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.340640177 (582h37m21.211723212s) [noise=true , address=5.231.73.5:4022 , piece_id=5WVAJH3Y6TLATXY4IYB6ZADKKJDXDS52B2ZN6FJAZ7UHFVRYRRAA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.481118228 (582h37m21.352201263s) [noise=true , address=5.231.73.2:4004 , piece_id=PTHHO3XEPGGCIRCUBIWHN4LVNBLMISJM5LESQGR2ERRESFK3S4RA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.896953769 (582h37m21.768036804s) [noise=true , address=145.79.227.49:4347 , piece_id=75B3DRRHEPDL6EK6ZUXTKASXUKWGAVUWBPSTNRYT4BJMAU5ALXRA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.911314353 (582h37m21.782397399s) [noise=true , address=145.79.227.34:4931 , piece_id=XJHLE3B5HIF4RC4LV32SBCNMVTOTEXWBBCXBVPQFN6N5HFQ6ZS7A, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.915845371 (582h37m21.786928406s) [noise=true , address=145.79.227.30:4766 , piece_id=YWDMUGNTYVXWOCMSN63VIC6HOTWHPY7VJ7QWT76P63UAJIBS2XLA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.928785972 (582h37m21.799869017s) [noise=true , address=145.79.227.35:4489 , piece_id=BM7ZQESTHEUIRJWAPORTKDMT5TVLEXMOHULGFVEQKHT4AJV3HT5A, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.928810859 (582h37m21.799893904s) [seg_pos={0 0}, upload=510734]: Segment reached optimal threshold of 14 pieces. Dec 29 22:06:21.929016471 (582h37m21.800099516s) [noise=true , address=145.79.227.7:4321 , piece_id=SPWPZ5MR5ZCQNYVYYQQEZC4OURAM5VAH3WN5UP6HMGCCDMCH6ZGA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12edsHqD9iRBDgMU8vYhSM1EWG8Sn57GZ8TPiTytri3rp7W9Eae): protocol: expected piece hash; context canceled; EOF Dec 29 22:06:21.929028644 (582h37m21.800111709s) [noise=true , address=145.79.227.7:4321 , piece_id=SPWPZ5MR5ZCQNYVYYQQEZC4OURAM5VAH3WN5UP6HMGCCDMCH6ZGA, seg_pos={0 0}, upload=510734]: Long tail cancellation due to optimal threshold Dec 29 22:06:21.929093137 (582h37m21.800176192s) [noise=true , address=145.79.227.35:4507 , piece_id=ZZSVIHOAN2SWXI4NS3TRD4B64NDB4D43J63DAS74TCPEE6MPGRXA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14jurWWPpmxwiBS3XmZysEUwugsceYJqVXFj7BbYpx8UFcTR8R): protocol: expected piece hash; context canceled; EOF Dec 29 22:06:21.929106793 (582h37m21.800189838s) [noise=true , address=145.79.227.35:4507 , piece_id=ZZSVIHOAN2SWXI4NS3TRD4B64NDB4D43J63DAS74TCPEE6MPGRXA, seg_pos={0 0}, upload=510734]: Long tail cancellation due to optimal threshold Dec 29 22:06:21.929206612 (582h37m21.800289657s) [noise=true , address=145.79.227.17:4995 , piece_id=JJTVPVBAMCTDGHGWNPUVPYFVKD4SMTDAHF4VWVIW7JFUAMBIPRHQ, seg_pos={0 0}, upload=510734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tvoTD1LoSj3q2rLiDckfT56dXNKKwdVu1ZAsBZExNFVTom9tH): protocol: write tcp 188.166.225.170:47696->145.79.227.17:4995: use of closed network connection; write tcp 188.166.225.170:47696->145.79.227.17:4995: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47696->145.79.227.17:4995: use of closed network connection Dec 29 22:06:21.929215911 (582h37m21.800298976s) [noise=true , address=145.79.227.17:4995 , piece_id=JJTVPVBAMCTDGHGWNPUVPYFVKD4SMTDAHF4VWVIW7JFUAMBIPRHQ, seg_pos={0 0}, upload=510734]: Long tail cancellation due to optimal threshold Dec 29 22:06:21.929387247 (582h37m21.800470292s) [noise=true , address=193.149.164.143:4698 , piece_id=6X5KB4TXFV4ZOF5JTPNRW4Q3GWNCVCXCA4JSB3AZURBHXY2VHTRQ, seg_pos={0 0}, upload=510734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F4fJAGJM2ZPRt7vJH2cZmEbYeujmxcfi7aPZ2g9VPTNYBYRAi): protocol: expected piece hash; context canceled; EOF Dec 29 22:06:21.929396174 (582h37m21.800479219s) [noise=true , address=193.149.164.143:4698 , piece_id=6X5KB4TXFV4ZOF5JTPNRW4Q3GWNCVCXCA4JSB3AZURBHXY2VHTRQ, seg_pos={0 0}, upload=510734]: Long tail cancellation due to optimal threshold Dec 29 22:06:21.929491536 (582h37m21.800574571s) [seg_pos={0 0}, upload=510734]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 22:06:21.929573792 (582h37m21.800656837s) [upload=510734]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 22:06:21.929580555 (582h37m21.8006636s) [upload=510734]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 22:06:22.233045366 (582h37m22.104128401s) [upload=510734]: Done uploading segments... Dec 29 22:06:22.233064963 (582h37m22.104147998s) [upload=510734]: Upload finished. err: <nil> Duration: 31.563430678s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [X..X..X........X............................X...............] duration: 5.750586122s trace: https://jaeger.dev.storj.io/trace/5209b3d035311fd FAIL (2025-12-29 22:12:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :00:17.043136250 (582h31m16.914219305s) [upload=510642]: Starting upload Dec 29 22:00:17.043227904 (582h31m16.91431096s) [upload=510642]: Done starting upload Dec 29 22:00:17.043335189 (582h31m16.914418234s) [upload=510642]: Uploading segments... Dec 29 22:00:17.043587749 (582h31m16.914670824s) [upload=510642]: Got next segment. Inline: false Dec 29 22:00:17.043644478 (582h31m16.914727523s) [upload=510642]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 22:00:17.043686427 (582h31m16.914769503s) [upload=510642]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 22:00:17.294384363 (582h31m17.165467408s) [seg_pos={0 0}, upload=510642]: Begin upload segment... Dec 29 22:00:17.294445219 (582h31m17.165528264s) [seg_pos={0 0}, upload=510642]: Done begin upload segment. Dec 29 22:00:17.294484133 (582h31m17.165567178s) [noise=true , address=145.79.227.31:4797 , piece_id=NI3M5NVVVX5HUXZ2SOMQD4S5ZOSSNPCD2F3Z5X2BA2VXMGT57AYQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.294560979 (582h31m17.165644014s) [noise=true , address=77.90.31.11:4553 , piece_id=3Y5HSD73DLZ6ALJ5TFRIVTLOE4ZNZ256YSZ2W3X5LPKL5J5LX2EQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.295149941 (582h31m17.166232976s) [noise=true , address=45.91.116.5:4297 , piece_id=PLUQ7Q32BTGYFQRGVJFAJWHPZCPQTC5RX5UX27S5HJ5NNJPLDMPQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.295419574 (582h31m17.166502609s) [noise=true , address=145.79.227.2:4115 , piece_id=P2AS5B3VYO2K2AWSM5W4BXAXBHSXQQWM7JOXFPOV64NTKDFLT62Q, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.295483997 (582h31m17.166567042s) [noise=true , address=145.79.227.51:4411 , piece_id=HTZQ6SW6KSACLQF7G4CFGQA572J5JVHMLGEWQYSSGZRYX7FJ6DXA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.295762207 (582h31m17.166845252s) [noise=true , address=145.79.227.5:4221 , piece_id=LD3FENLBM2L5N7QDOAMAYJUVH54QVSJQJFP7E36UAH2ZD7R65CCQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.295928003 (582h31m17.167011048s) [noise=true , address=145.79.227.16:4909 , piece_id=CF7WHNKC4LFJQRVFBPFV62JGEBLD4KVPE6Q5JBWQVADC6ZAFFLVQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.296039615 (582h31m17.16712266s) [noise=true , address=45.91.116.13:4154 , piece_id=FE4NFODQ3TWGFXARFYIL64TKK6X7H6PPLIG6LO44LKNAZOEU7DJA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.296272429 (582h31m17.167355474s) [noise=true , address=45.91.116.10:4084 , piece_id=5BFUYVGIFOYXC6RCPU7VBUM36LSM32TVBLS3W54OWFNR4SM6GCSA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.296403588 (582h31m17.167486633s) [noise=true , address=77.90.31.19:4844 , piece_id=CDOWECXLSRPACV4HMAH57UK4EEIYNOMH6GYV6HFHV7CWSEAJ44FA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.296646560 (582h31m17.167729596s) [noise=true , address=77.90.31.14:4661 , piece_id=B7IDSCDHBUXUIPSUT3HMQSCMPGBMYASZSQO2EJBXRR2DQHCLWGYA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.296906235 (582h31m17.167989291s) [noise=true , address=145.79.227.47:4265 , piece_id=FYZA2LRIFKLKVUCUAWMNWTUH3VYR5VFAA7XKAYIMD2E2VOJTI7PA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.296989804 (582h31m17.168072849s) [noise=true , address=145.79.227.46:4601 , piece_id=YJTMZID24VBM5ZZSUUHOCZSZSSNGTFHOSKK6OI5QP3A4G6ZWVWQA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.297235221 (582h31m17.168318267s) [noise=true , address=145.79.227.3:4150 , piece_id=HS4S5UDYD2VWI2N4MH6EWEYQ5G6KCFCJBDZR7AVWPXRYALFXP4FQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.297424432 (582h31m17.168507467s) [noise=true , address=145.79.227.13:4803 , piece_id=YIEUZEDLSC5C67SCFEPCHHZSTYP6MYLYJZJTGZF3UP6JWDTGUOPQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.297606900 (582h31m17.168689945s) [noise=true , address=45.91.116.14:4190 , piece_id=FIL6AIXGE7DE2M4QBVSIO53KGY43VFPZQ2FTXSWJFPJU4VZCKUXQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:18.995364155 (582h31m18.866447201s) [upload=510642]: Next returned nil segment Dec 29 22:00:18.995383603 (582h31m18.866466648s) [upload=510642]: Waiting for error group managing segments... Dec 29 22:00:19.284009408 (582h31m19.155092453s) [noise=true , address=145.79.227.51:4411 , piece_id=HTZQ6SW6KSACLQF7G4CFGQA572J5JVHMLGEWQYSSGZRYX7FJ6DXA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.295985232 (582h31m19.167068277s) [noise=true , address=145.79.227.13:4803 , piece_id=YIEUZEDLSC5C67SCFEPCHHZSTYP6MYLYJZJTGZF3UP6JWDTGUOPQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.296135458 (582h31m19.167218503s) [noise=true , address=145.79.227.16:4909 , piece_id=CF7WHNKC4LFJQRVFBPFV62JGEBLD4KVPE6Q5JBWQVADC6ZAFFLVQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.310119916 (582h31m19.181202971s) [noise=true , address=145.79.227.46:4601 , piece_id=YJTMZID24VBM5ZZSUUHOCZSZSSNGTFHOSKK6OI5QP3A4G6ZWVWQA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.310998939 (582h31m19.182081984s) [noise=true , address=145.79.227.31:4797 , piece_id=NI3M5NVVVX5HUXZ2SOMQD4S5ZOSSNPCD2F3Z5X2BA2VXMGT57AYQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.318707138 (582h31m19.189790183s) [noise=true , address=145.79.227.47:4265 , piece_id=FYZA2LRIFKLKVUCUAWMNWTUH3VYR5VFAA7XKAYIMD2E2VOJTI7PA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.322059381 (582h31m19.193142426s) [noise=true , address=145.79.227.5:4221 , piece_id=LD3FENLBM2L5N7QDOAMAYJUVH54QVSJQJFP7E36UAH2ZD7R65CCQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.506372108 (582h31m19.377455153s) [noise=true , address=77.90.31.19:4844 , piece_id=CDOWECXLSRPACV4HMAH57UK4EEIYNOMH6GYV6HFHV7CWSEAJ44FA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.507172602 (582h31m19.378255648s) [noise=true , address=77.90.31.11:4553 , piece_id=3Y5HSD73DLZ6ALJ5TFRIVTLOE4ZNZ256YSZ2W3X5LPKL5J5LX2EQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.527548842 (582h31m19.398631887s) [noise=true , address=77.90.31.14:4661 , piece_id=B7IDSCDHBUXUIPSUT3HMQSCMPGBMYASZSQO2EJBXRR2DQHCLWGYA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.531443508 (582h31m19.402526553s) [noise=true , address=45.91.116.5:4297 , piece_id=PLUQ7Q32BTGYFQRGVJFAJWHPZCPQTC5RX5UX27S5HJ5NNJPLDMPQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.552364906 (582h31m19.423447951s) [noise=true , address=45.91.116.14:4190 , piece_id=FIL6AIXGE7DE2M4QBVSIO53KGY43VFPZQ2FTXSWJFPJU4VZCKUXQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.564056769 (582h31m19.435139814s) [noise=true , address=45.91.116.10:4084 , piece_id=5BFUYVGIFOYXC6RCPU7VBUM36LSM32TVBLS3W54OWFNR4SM6GCSA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.564087016 (582h31m19.435170051s) [seg_pos={0 0}, upload=510642]: Segment reached optimal threshold of 13 pieces. Dec 29 22:00:19.564294431 (582h31m19.435377466s) [noise=true , address=145.79.227.3:4150 , piece_id=HS4S5UDYD2VWI2N4MH6EWEYQ5G6KCFCJBDZR7AVWPXRYALFXP4FQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK): protocol: expected piece hash; context canceled; EOF Dec 29 22:00:19.564302366 (582h31m19.435385411s) [noise=true , address=145.79.227.3:4150 , piece_id=HS4S5UDYD2VWI2N4MH6EWEYQ5G6KCFCJBDZR7AVWPXRYALFXP4FQ, seg_pos={0 0}, upload=510642]: Long tail cancellation due to optimal threshold Dec 29 22:00:19.564388932 (582h31m19.435471977s) [noise=true , address=45.91.116.13:4154 , piece_id=FE4NFODQ3TWGFXARFYIL64TKK6X7H6PPLIG6LO44LKNAZOEU7DJA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Dec 29 22:00:19.564401876 (582h31m19.435484911s) [noise=true , address=45.91.116.13:4154 , piece_id=FE4NFODQ3TWGFXARFYIL64TKK6X7H6PPLIG6LO44LKNAZOEU7DJA, seg_pos={0 0}, upload=510642]: Long tail cancellation due to optimal threshold Dec 29 22:00:19.564558945 (582h31m19.435641981s) [noise=true , address=145.79.227.2:4115 , piece_id=P2AS5B3VYO2K2AWSM5W4BXAXBHSXQQWM7JOXFPOV64NTKDFLT62Q, seg_pos={0 0}, upload=510642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: write tcp 188.166.225.170:38246->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:38246->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38246->145.79.227.2:4115: use of closed network connection Dec 29 22:00:19.564570738 (582h31m19.435653783s) [noise=true , address=145.79.227.2:4115 , piece_id=P2AS5B3VYO2K2AWSM5W4BXAXBHSXQQWM7JOXFPOV64NTKDFLT62Q, seg_pos={0 0}, upload=510642]: Long tail cancellation due to optimal threshold Dec 29 22:00:19.564601426 (582h31m19.435684471s) [seg_pos={0 0}, upload=510642]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 22:00:19.564683973 (582h31m19.435767018s) [upload=510642]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 22:00:19.564690004 (582h31m19.435773049s) [upload=510642]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 22:00:19.871860472 (582h31m19.742943527s) [upload=510642]: Done uploading segments... Dec 29 22:00:19.871892413 (582h31m19.742975468s) [upload=510642]: Upload finished. err: <nil> Duration: 5.750616571s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................X............................X......] duration: 15.440826826s trace: https://jaeger.dev.storj.io/trace/7096862780f29049 FAIL (2025-12-29 22:11:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== oise=true , address=145.79.227.30:4766 , piece_id=YWDMUGNTYVXWOCMSN63VIC6HOTWHPY7VJ7QWT76P63UAJIBS2XLA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.060493636 (582h37m18.931576671s) [noise=true , address=5.231.73.5:4022 , piece_id=5WVAJH3Y6TLATXY4IYB6ZADKKJDXDS52B2ZN6FJAZ7UHFVRYRRAA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.060685291 (582h37m18.931768326s) [noise=true , address=77.90.31.11:4579 , piece_id=DEJPYHGYNAP52CZ45BKGNXJM5MLIA2OQZ4BRKN5PBADNRVE3VLEA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.060749954 (582h37m18.931832989s) [noise=true , address=145.79.227.29:4976 , piece_id=YRCRXSSQOXPOJGVX46L7WCB3TRDZ5QB755SEOTNHSCHN4ZDMCOUQ, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.061033203 (582h37m18.932116248s) [noise=true , address=193.149.164.143:4698 , piece_id=6X5KB4TXFV4ZOF5JTPNRW4Q3GWNCVCXCA4JSB3AZURBHXY2VHTRQ, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.061248855 (582h37m18.9323319s) [noise=true , address=145.79.227.49:4347 , piece_id=75B3DRRHEPDL6EK6ZUXTKASXUKWGAVUWBPSTNRYT4BJMAU5ALXRA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.061382729 (582h37m18.932465764s) [noise=true , address=5.231.73.2:4004 , piece_id=PTHHO3XEPGGCIRCUBIWHN4LVNBLMISJM5LESQGR2ERRESFK3S4RA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.061645560 (582h37m18.932728595s) [noise=true , address=5.231.73.3:4007 , piece_id=6AF3CEPLUMRO4CDW3SRWZSYHWEXP35AVSM5M3LH4KCRARSZ6NFNA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.061769094 (582h37m18.932852139s) [noise=true , address=145.79.227.7:4321 , piece_id=SPWPZ5MR5ZCQNYVYYQQEZC4OURAM5VAH3WN5UP6HMGCCDMCH6ZGA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.061970298 (582h37m18.933053333s) [noise=true , address=145.79.227.35:4507 , piece_id=ZZSVIHOAN2SWXI4NS3TRD4B64NDB4D43J63DAS74TCPEE6MPGRXA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.062237577 (582h37m18.933320612s) [noise=true , address=145.79.227.35:4489 , piece_id=BM7ZQESTHEUIRJWAPORTKDMT5TVLEXMOHULGFVEQKHT4AJV3HT5A, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.062304304 (582h37m18.933387349s) [noise=true , address=145.79.227.34:4931 , piece_id=XJHLE3B5HIF4RC4LV32SBCNMVTOTEXWBBCXBVPQFN6N5HFQ6ZS7A, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.062522640 (582h37m18.933605685s) [noise=true , address=145.79.227.48:4329 , piece_id=YWCVTYLCAGU57SE3ID3WW2JLCJGTJ7CPDRCZENAMO42BVPIAVLIQ, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.062904968 (582h37m18.933988003s) [noise=true , address=77.90.31.17:4801 , piece_id=K3QD7FPVRQHZL5R57EKKAC27RGDJQH6HDCND7FIVXSZBRH42WSIA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.063420419 (582h37m18.934503464s) [noise=true , address=145.79.227.17:4995 , piece_id=JJTVPVBAMCTDGHGWNPUVPYFVKD4SMTDAHF4VWVIW7JFUAMBIPRHQ, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.063943495 (582h37m18.93502654s) [noise=true , address=5.231.73.4:4018 , piece_id=X45H47HER27IM5FEQSEAMXSDX7UQG3AEOY65Q4WCCUOX7SVFZ5KA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:20.386418430 (582h37m20.257501465s) [upload=510734]: Next returned nil segment Dec 29 22:06:20.386428990 (582h37m20.257512025s) [upload=510734]: Waiting for error group managing segments... Dec 29 22:06:20.935543714 (582h37m20.806626749s) [noise=true , address=145.79.227.48:4329 , piece_id=YWCVTYLCAGU57SE3ID3WW2JLCJGTJ7CPDRCZENAMO42BVPIAVLIQ, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:20.978073277 (582h37m20.849156312s) [noise=true , address=145.79.227.39:4022 , piece_id=5IJCSVRVXCICRNRB3TTJAVNRF5FTUQMMY4KL7ZFF4SBKILFGHJWA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.079004013 (582h37m20.950087058s) [noise=true , address=5.231.73.4:4018 , piece_id=X45H47HER27IM5FEQSEAMXSDX7UQG3AEOY65Q4WCCUOX7SVFZ5KA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.101144552 (582h37m20.972227597s) [noise=true , address=77.90.31.11:4579 , piece_id=DEJPYHGYNAP52CZ45BKGNXJM5MLIA2OQZ4BRKN5PBADNRVE3VLEA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.138766369 (582h37m21.009849405s) [noise=true , address=77.90.31.17:4801 , piece_id=K3QD7FPVRQHZL5R57EKKAC27RGDJQH6HDCND7FIVXSZBRH42WSIA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.231047035 (582h37m21.10213007s) [noise=true , address=77.90.31.19:4857 , piece_id=DMJ4O3TWZYWKWVLJFSQBULHDFWZCNLVDBIGYJEOBAVVCU5UGG27Q, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.272062706 (582h37m21.143145741s) [noise=true , address=5.231.73.3:4007 , piece_id=6AF3CEPLUMRO4CDW3SRWZSYHWEXP35AVSM5M3LH4KCRARSZ6NFNA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.303520666 (582h37m21.174603711s) [noise=true , address=145.79.227.29:4976 , piece_id=YRCRXSSQOXPOJGVX46L7WCB3TRDZ5QB755SEOTNHSCHN4ZDMCOUQ, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.340640177 (582h37m21.211723212s) [noise=true , address=5.231.73.5:4022 , piece_id=5WVAJH3Y6TLATXY4IYB6ZADKKJDXDS52B2ZN6FJAZ7UHFVRYRRAA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.481118228 (582h37m21.352201263s) [noise=true , address=5.231.73.2:4004 , piece_id=PTHHO3XEPGGCIRCUBIWHN4LVNBLMISJM5LESQGR2ERRESFK3S4RA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.896953769 (582h37m21.768036804s) [noise=true , address=145.79.227.49:4347 , piece_id=75B3DRRHEPDL6EK6ZUXTKASXUKWGAVUWBPSTNRYT4BJMAU5ALXRA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.911314353 (582h37m21.782397399s) [noise=true , address=145.79.227.34:4931 , piece_id=XJHLE3B5HIF4RC4LV32SBCNMVTOTEXWBBCXBVPQFN6N5HFQ6ZS7A, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.915845371 (582h37m21.786928406s) [noise=true , address=145.79.227.30:4766 , piece_id=YWDMUGNTYVXWOCMSN63VIC6HOTWHPY7VJ7QWT76P63UAJIBS2XLA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.928785972 (582h37m21.799869017s) [noise=true , address=145.79.227.35:4489 , piece_id=BM7ZQESTHEUIRJWAPORTKDMT5TVLEXMOHULGFVEQKHT4AJV3HT5A, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.928810859 (582h37m21.799893904s) [seg_pos={0 0}, upload=510734]: Segment reached optimal threshold of 14 pieces. Dec 29 22:06:21.929016471 (582h37m21.800099516s) [noise=true , address=145.79.227.7:4321 , piece_id=SPWPZ5MR5ZCQNYVYYQQEZC4OURAM5VAH3WN5UP6HMGCCDMCH6ZGA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12edsHqD9iRBDgMU8vYhSM1EWG8Sn57GZ8TPiTytri3rp7W9Eae): protocol: expected piece hash; context canceled; EOF Dec 29 22:06:21.929028644 (582h37m21.800111709s) [noise=true , address=145.79.227.7:4321 , piece_id=SPWPZ5MR5ZCQNYVYYQQEZC4OURAM5VAH3WN5UP6HMGCCDMCH6ZGA, seg_pos={0 0}, upload=510734]: Long tail cancellation due to optimal threshold Dec 29 22:06:21.929093137 (582h37m21.800176192s) [noise=true , address=145.79.227.35:4507 , piece_id=ZZSVIHOAN2SWXI4NS3TRD4B64NDB4D43J63DAS74TCPEE6MPGRXA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14jurWWPpmxwiBS3XmZysEUwugsceYJqVXFj7BbYpx8UFcTR8R): protocol: expected piece hash; context canceled; EOF Dec 29 22:06:21.929106793 (582h37m21.800189838s) [noise=true , address=145.79.227.35:4507 , piece_id=ZZSVIHOAN2SWXI4NS3TRD4B64NDB4D43J63DAS74TCPEE6MPGRXA, seg_pos={0 0}, upload=510734]: Long tail cancellation due to optimal threshold Dec 29 22:06:21.929206612 (582h37m21.800289657s) [noise=true , address=145.79.227.17:4995 , piece_id=JJTVPVBAMCTDGHGWNPUVPYFVKD4SMTDAHF4VWVIW7JFUAMBIPRHQ, seg_pos={0 0}, upload=510734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tvoTD1LoSj3q2rLiDckfT56dXNKKwdVu1ZAsBZExNFVTom9tH): protocol: write tcp 188.166.225.170:47696->145.79.227.17:4995: use of closed network connection; write tcp 188.166.225.170:47696->145.79.227.17:4995: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47696->145.79.227.17:4995: use of closed network connection Dec 29 22:06:21.929215911 (582h37m21.800298976s) [noise=true , address=145.79.227.17:4995 , piece_id=JJTVPVBAMCTDGHGWNPUVPYFVKD4SMTDAHF4VWVIW7JFUAMBIPRHQ, seg_pos={0 0}, upload=510734]: Long tail cancellation due to optimal threshold Dec 29 22:06:21.929387247 (582h37m21.800470292s) [noise=true , address=193.149.164.143:4698 , piece_id=6X5KB4TXFV4ZOF5JTPNRW4Q3GWNCVCXCA4JSB3AZURBHXY2VHTRQ, seg_pos={0 0}, upload=510734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F4fJAGJM2ZPRt7vJH2cZmEbYeujmxcfi7aPZ2g9VPTNYBYRAi): protocol: expected piece hash; context canceled; EOF Dec 29 22:06:21.929396174 (582h37m21.800479219s) [noise=true , address=193.149.164.143:4698 , piece_id=6X5KB4TXFV4ZOF5JTPNRW4Q3GWNCVCXCA4JSB3AZURBHXY2VHTRQ, seg_pos={0 0}, upload=510734]: Long tail cancellation due to optimal threshold Dec 29 22:06:21.929491536 (582h37m21.800574571s) [seg_pos={0 0}, upload=510734]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 22:06:21.929573792 (582h37m21.800656837s) [upload=510734]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 22:06:21.929580555 (582h37m21.8006636s) [upload=510734]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 22:06:22.233045366 (582h37m22.104128401s) [upload=510734]: Done uploading segments... Dec 29 22:06:22.233064963 (582h37m22.104147998s) [upload=510734]: Upload finished. err: <nil> Duration: 15.440841503s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.X..X..X........X............................X..............] duration: 5.914000666s trace: https://jaeger.dev.storj.io/trace/5209b3d035311fd FAIL (2025-12-29 22:11:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :00:17.043136250 (582h31m16.914219305s) [upload=510642]: Starting upload Dec 29 22:00:17.043227904 (582h31m16.91431096s) [upload=510642]: Done starting upload Dec 29 22:00:17.043335189 (582h31m16.914418234s) [upload=510642]: Uploading segments... Dec 29 22:00:17.043587749 (582h31m16.914670824s) [upload=510642]: Got next segment. Inline: false Dec 29 22:00:17.043644478 (582h31m16.914727523s) [upload=510642]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 22:00:17.043686427 (582h31m16.914769503s) [upload=510642]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 22:00:17.294384363 (582h31m17.165467408s) [seg_pos={0 0}, upload=510642]: Begin upload segment... Dec 29 22:00:17.294445219 (582h31m17.165528264s) [seg_pos={0 0}, upload=510642]: Done begin upload segment. Dec 29 22:00:17.294484133 (582h31m17.165567178s) [noise=true , address=145.79.227.31:4797 , piece_id=NI3M5NVVVX5HUXZ2SOMQD4S5ZOSSNPCD2F3Z5X2BA2VXMGT57AYQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.294560979 (582h31m17.165644014s) [noise=true , address=77.90.31.11:4553 , piece_id=3Y5HSD73DLZ6ALJ5TFRIVTLOE4ZNZ256YSZ2W3X5LPKL5J5LX2EQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.295149941 (582h31m17.166232976s) [noise=true , address=45.91.116.5:4297 , piece_id=PLUQ7Q32BTGYFQRGVJFAJWHPZCPQTC5RX5UX27S5HJ5NNJPLDMPQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.295419574 (582h31m17.166502609s) [noise=true , address=145.79.227.2:4115 , piece_id=P2AS5B3VYO2K2AWSM5W4BXAXBHSXQQWM7JOXFPOV64NTKDFLT62Q, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.295483997 (582h31m17.166567042s) [noise=true , address=145.79.227.51:4411 , piece_id=HTZQ6SW6KSACLQF7G4CFGQA572J5JVHMLGEWQYSSGZRYX7FJ6DXA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.295762207 (582h31m17.166845252s) [noise=true , address=145.79.227.5:4221 , piece_id=LD3FENLBM2L5N7QDOAMAYJUVH54QVSJQJFP7E36UAH2ZD7R65CCQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.295928003 (582h31m17.167011048s) [noise=true , address=145.79.227.16:4909 , piece_id=CF7WHNKC4LFJQRVFBPFV62JGEBLD4KVPE6Q5JBWQVADC6ZAFFLVQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.296039615 (582h31m17.16712266s) [noise=true , address=45.91.116.13:4154 , piece_id=FE4NFODQ3TWGFXARFYIL64TKK6X7H6PPLIG6LO44LKNAZOEU7DJA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.296272429 (582h31m17.167355474s) [noise=true , address=45.91.116.10:4084 , piece_id=5BFUYVGIFOYXC6RCPU7VBUM36LSM32TVBLS3W54OWFNR4SM6GCSA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.296403588 (582h31m17.167486633s) [noise=true , address=77.90.31.19:4844 , piece_id=CDOWECXLSRPACV4HMAH57UK4EEIYNOMH6GYV6HFHV7CWSEAJ44FA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.296646560 (582h31m17.167729596s) [noise=true , address=77.90.31.14:4661 , piece_id=B7IDSCDHBUXUIPSUT3HMQSCMPGBMYASZSQO2EJBXRR2DQHCLWGYA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.296906235 (582h31m17.167989291s) [noise=true , address=145.79.227.47:4265 , piece_id=FYZA2LRIFKLKVUCUAWMNWTUH3VYR5VFAA7XKAYIMD2E2VOJTI7PA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.296989804 (582h31m17.168072849s) [noise=true , address=145.79.227.46:4601 , piece_id=YJTMZID24VBM5ZZSUUHOCZSZSSNGTFHOSKK6OI5QP3A4G6ZWVWQA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.297235221 (582h31m17.168318267s) [noise=true , address=145.79.227.3:4150 , piece_id=HS4S5UDYD2VWI2N4MH6EWEYQ5G6KCFCJBDZR7AVWPXRYALFXP4FQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.297424432 (582h31m17.168507467s) [noise=true , address=145.79.227.13:4803 , piece_id=YIEUZEDLSC5C67SCFEPCHHZSTYP6MYLYJZJTGZF3UP6JWDTGUOPQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.297606900 (582h31m17.168689945s) [noise=true , address=45.91.116.14:4190 , piece_id=FIL6AIXGE7DE2M4QBVSIO53KGY43VFPZQ2FTXSWJFPJU4VZCKUXQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:18.995364155 (582h31m18.866447201s) [upload=510642]: Next returned nil segment Dec 29 22:00:18.995383603 (582h31m18.866466648s) [upload=510642]: Waiting for error group managing segments... Dec 29 22:00:19.284009408 (582h31m19.155092453s) [noise=true , address=145.79.227.51:4411 , piece_id=HTZQ6SW6KSACLQF7G4CFGQA572J5JVHMLGEWQYSSGZRYX7FJ6DXA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.295985232 (582h31m19.167068277s) [noise=true , address=145.79.227.13:4803 , piece_id=YIEUZEDLSC5C67SCFEPCHHZSTYP6MYLYJZJTGZF3UP6JWDTGUOPQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.296135458 (582h31m19.167218503s) [noise=true , address=145.79.227.16:4909 , piece_id=CF7WHNKC4LFJQRVFBPFV62JGEBLD4KVPE6Q5JBWQVADC6ZAFFLVQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.310119916 (582h31m19.181202971s) [noise=true , address=145.79.227.46:4601 , piece_id=YJTMZID24VBM5ZZSUUHOCZSZSSNGTFHOSKK6OI5QP3A4G6ZWVWQA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.310998939 (582h31m19.182081984s) [noise=true , address=145.79.227.31:4797 , piece_id=NI3M5NVVVX5HUXZ2SOMQD4S5ZOSSNPCD2F3Z5X2BA2VXMGT57AYQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.318707138 (582h31m19.189790183s) [noise=true , address=145.79.227.47:4265 , piece_id=FYZA2LRIFKLKVUCUAWMNWTUH3VYR5VFAA7XKAYIMD2E2VOJTI7PA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.322059381 (582h31m19.193142426s) [noise=true , address=145.79.227.5:4221 , piece_id=LD3FENLBM2L5N7QDOAMAYJUVH54QVSJQJFP7E36UAH2ZD7R65CCQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.506372108 (582h31m19.377455153s) [noise=true , address=77.90.31.19:4844 , piece_id=CDOWECXLSRPACV4HMAH57UK4EEIYNOMH6GYV6HFHV7CWSEAJ44FA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.507172602 (582h31m19.378255648s) [noise=true , address=77.90.31.11:4553 , piece_id=3Y5HSD73DLZ6ALJ5TFRIVTLOE4ZNZ256YSZ2W3X5LPKL5J5LX2EQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.527548842 (582h31m19.398631887s) [noise=true , address=77.90.31.14:4661 , piece_id=B7IDSCDHBUXUIPSUT3HMQSCMPGBMYASZSQO2EJBXRR2DQHCLWGYA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.531443508 (582h31m19.402526553s) [noise=true , address=45.91.116.5:4297 , piece_id=PLUQ7Q32BTGYFQRGVJFAJWHPZCPQTC5RX5UX27S5HJ5NNJPLDMPQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.552364906 (582h31m19.423447951s) [noise=true , address=45.91.116.14:4190 , piece_id=FIL6AIXGE7DE2M4QBVSIO53KGY43VFPZQ2FTXSWJFPJU4VZCKUXQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.564056769 (582h31m19.435139814s) [noise=true , address=45.91.116.10:4084 , piece_id=5BFUYVGIFOYXC6RCPU7VBUM36LSM32TVBLS3W54OWFNR4SM6GCSA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.564087016 (582h31m19.435170051s) [seg_pos={0 0}, upload=510642]: Segment reached optimal threshold of 13 pieces. Dec 29 22:00:19.564294431 (582h31m19.435377466s) [noise=true , address=145.79.227.3:4150 , piece_id=HS4S5UDYD2VWI2N4MH6EWEYQ5G6KCFCJBDZR7AVWPXRYALFXP4FQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK): protocol: expected piece hash; context canceled; EOF Dec 29 22:00:19.564302366 (582h31m19.435385411s) [noise=true , address=145.79.227.3:4150 , piece_id=HS4S5UDYD2VWI2N4MH6EWEYQ5G6KCFCJBDZR7AVWPXRYALFXP4FQ, seg_pos={0 0}, upload=510642]: Long tail cancellation due to optimal threshold Dec 29 22:00:19.564388932 (582h31m19.435471977s) [noise=true , address=45.91.116.13:4154 , piece_id=FE4NFODQ3TWGFXARFYIL64TKK6X7H6PPLIG6LO44LKNAZOEU7DJA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Dec 29 22:00:19.564401876 (582h31m19.435484911s) [noise=true , address=45.91.116.13:4154 , piece_id=FE4NFODQ3TWGFXARFYIL64TKK6X7H6PPLIG6LO44LKNAZOEU7DJA, seg_pos={0 0}, upload=510642]: Long tail cancellation due to optimal threshold Dec 29 22:00:19.564558945 (582h31m19.435641981s) [noise=true , address=145.79.227.2:4115 , piece_id=P2AS5B3VYO2K2AWSM5W4BXAXBHSXQQWM7JOXFPOV64NTKDFLT62Q, seg_pos={0 0}, upload=510642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: write tcp 188.166.225.170:38246->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:38246->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38246->145.79.227.2:4115: use of closed network connection Dec 29 22:00:19.564570738 (582h31m19.435653783s) [noise=true , address=145.79.227.2:4115 , piece_id=P2AS5B3VYO2K2AWSM5W4BXAXBHSXQQWM7JOXFPOV64NTKDFLT62Q, seg_pos={0 0}, upload=510642]: Long tail cancellation due to optimal threshold Dec 29 22:00:19.564601426 (582h31m19.435684471s) [seg_pos={0 0}, upload=510642]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 22:00:19.564683973 (582h31m19.435767018s) [upload=510642]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 22:00:19.564690004 (582h31m19.435773049s) [upload=510642]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 22:00:19.871860472 (582h31m19.742943527s) [upload=510642]: Done uploading segments... Dec 29 22:00:19.871892413 (582h31m19.742975468s) [upload=510642]: Upload finished. err: <nil> Duration: 5.914030693s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................X............................X.....] duration: 16.803783906s trace: https://jaeger.dev.storj.io/trace/7096862780f29049 FAIL (2025-12-29 22:11:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== oise=true , address=145.79.227.30:4766 , piece_id=YWDMUGNTYVXWOCMSN63VIC6HOTWHPY7VJ7QWT76P63UAJIBS2XLA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.060493636 (582h37m18.931576671s) [noise=true , address=5.231.73.5:4022 , piece_id=5WVAJH3Y6TLATXY4IYB6ZADKKJDXDS52B2ZN6FJAZ7UHFVRYRRAA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.060685291 (582h37m18.931768326s) [noise=true , address=77.90.31.11:4579 , piece_id=DEJPYHGYNAP52CZ45BKGNXJM5MLIA2OQZ4BRKN5PBADNRVE3VLEA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.060749954 (582h37m18.931832989s) [noise=true , address=145.79.227.29:4976 , piece_id=YRCRXSSQOXPOJGVX46L7WCB3TRDZ5QB755SEOTNHSCHN4ZDMCOUQ, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.061033203 (582h37m18.932116248s) [noise=true , address=193.149.164.143:4698 , piece_id=6X5KB4TXFV4ZOF5JTPNRW4Q3GWNCVCXCA4JSB3AZURBHXY2VHTRQ, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.061248855 (582h37m18.9323319s) [noise=true , address=145.79.227.49:4347 , piece_id=75B3DRRHEPDL6EK6ZUXTKASXUKWGAVUWBPSTNRYT4BJMAU5ALXRA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.061382729 (582h37m18.932465764s) [noise=true , address=5.231.73.2:4004 , piece_id=PTHHO3XEPGGCIRCUBIWHN4LVNBLMISJM5LESQGR2ERRESFK3S4RA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.061645560 (582h37m18.932728595s) [noise=true , address=5.231.73.3:4007 , piece_id=6AF3CEPLUMRO4CDW3SRWZSYHWEXP35AVSM5M3LH4KCRARSZ6NFNA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.061769094 (582h37m18.932852139s) [noise=true , address=145.79.227.7:4321 , piece_id=SPWPZ5MR5ZCQNYVYYQQEZC4OURAM5VAH3WN5UP6HMGCCDMCH6ZGA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.061970298 (582h37m18.933053333s) [noise=true , address=145.79.227.35:4507 , piece_id=ZZSVIHOAN2SWXI4NS3TRD4B64NDB4D43J63DAS74TCPEE6MPGRXA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.062237577 (582h37m18.933320612s) [noise=true , address=145.79.227.35:4489 , piece_id=BM7ZQESTHEUIRJWAPORTKDMT5TVLEXMOHULGFVEQKHT4AJV3HT5A, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.062304304 (582h37m18.933387349s) [noise=true , address=145.79.227.34:4931 , piece_id=XJHLE3B5HIF4RC4LV32SBCNMVTOTEXWBBCXBVPQFN6N5HFQ6ZS7A, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.062522640 (582h37m18.933605685s) [noise=true , address=145.79.227.48:4329 , piece_id=YWCVTYLCAGU57SE3ID3WW2JLCJGTJ7CPDRCZENAMO42BVPIAVLIQ, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.062904968 (582h37m18.933988003s) [noise=true , address=77.90.31.17:4801 , piece_id=K3QD7FPVRQHZL5R57EKKAC27RGDJQH6HDCND7FIVXSZBRH42WSIA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.063420419 (582h37m18.934503464s) [noise=true , address=145.79.227.17:4995 , piece_id=JJTVPVBAMCTDGHGWNPUVPYFVKD4SMTDAHF4VWVIW7JFUAMBIPRHQ, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.063943495 (582h37m18.93502654s) [noise=true , address=5.231.73.4:4018 , piece_id=X45H47HER27IM5FEQSEAMXSDX7UQG3AEOY65Q4WCCUOX7SVFZ5KA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:20.386418430 (582h37m20.257501465s) [upload=510734]: Next returned nil segment Dec 29 22:06:20.386428990 (582h37m20.257512025s) [upload=510734]: Waiting for error group managing segments... Dec 29 22:06:20.935543714 (582h37m20.806626749s) [noise=true , address=145.79.227.48:4329 , piece_id=YWCVTYLCAGU57SE3ID3WW2JLCJGTJ7CPDRCZENAMO42BVPIAVLIQ, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:20.978073277 (582h37m20.849156312s) [noise=true , address=145.79.227.39:4022 , piece_id=5IJCSVRVXCICRNRB3TTJAVNRF5FTUQMMY4KL7ZFF4SBKILFGHJWA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.079004013 (582h37m20.950087058s) [noise=true , address=5.231.73.4:4018 , piece_id=X45H47HER27IM5FEQSEAMXSDX7UQG3AEOY65Q4WCCUOX7SVFZ5KA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.101144552 (582h37m20.972227597s) [noise=true , address=77.90.31.11:4579 , piece_id=DEJPYHGYNAP52CZ45BKGNXJM5MLIA2OQZ4BRKN5PBADNRVE3VLEA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.138766369 (582h37m21.009849405s) [noise=true , address=77.90.31.17:4801 , piece_id=K3QD7FPVRQHZL5R57EKKAC27RGDJQH6HDCND7FIVXSZBRH42WSIA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.231047035 (582h37m21.10213007s) [noise=true , address=77.90.31.19:4857 , piece_id=DMJ4O3TWZYWKWVLJFSQBULHDFWZCNLVDBIGYJEOBAVVCU5UGG27Q, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.272062706 (582h37m21.143145741s) [noise=true , address=5.231.73.3:4007 , piece_id=6AF3CEPLUMRO4CDW3SRWZSYHWEXP35AVSM5M3LH4KCRARSZ6NFNA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.303520666 (582h37m21.174603711s) [noise=true , address=145.79.227.29:4976 , piece_id=YRCRXSSQOXPOJGVX46L7WCB3TRDZ5QB755SEOTNHSCHN4ZDMCOUQ, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.340640177 (582h37m21.211723212s) [noise=true , address=5.231.73.5:4022 , piece_id=5WVAJH3Y6TLATXY4IYB6ZADKKJDXDS52B2ZN6FJAZ7UHFVRYRRAA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.481118228 (582h37m21.352201263s) [noise=true , address=5.231.73.2:4004 , piece_id=PTHHO3XEPGGCIRCUBIWHN4LVNBLMISJM5LESQGR2ERRESFK3S4RA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.896953769 (582h37m21.768036804s) [noise=true , address=145.79.227.49:4347 , piece_id=75B3DRRHEPDL6EK6ZUXTKASXUKWGAVUWBPSTNRYT4BJMAU5ALXRA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.911314353 (582h37m21.782397399s) [noise=true , address=145.79.227.34:4931 , piece_id=XJHLE3B5HIF4RC4LV32SBCNMVTOTEXWBBCXBVPQFN6N5HFQ6ZS7A, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.915845371 (582h37m21.786928406s) [noise=true , address=145.79.227.30:4766 , piece_id=YWDMUGNTYVXWOCMSN63VIC6HOTWHPY7VJ7QWT76P63UAJIBS2XLA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.928785972 (582h37m21.799869017s) [noise=true , address=145.79.227.35:4489 , piece_id=BM7ZQESTHEUIRJWAPORTKDMT5TVLEXMOHULGFVEQKHT4AJV3HT5A, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.928810859 (582h37m21.799893904s) [seg_pos={0 0}, upload=510734]: Segment reached optimal threshold of 14 pieces. Dec 29 22:06:21.929016471 (582h37m21.800099516s) [noise=true , address=145.79.227.7:4321 , piece_id=SPWPZ5MR5ZCQNYVYYQQEZC4OURAM5VAH3WN5UP6HMGCCDMCH6ZGA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12edsHqD9iRBDgMU8vYhSM1EWG8Sn57GZ8TPiTytri3rp7W9Eae): protocol: expected piece hash; context canceled; EOF Dec 29 22:06:21.929028644 (582h37m21.800111709s) [noise=true , address=145.79.227.7:4321 , piece_id=SPWPZ5MR5ZCQNYVYYQQEZC4OURAM5VAH3WN5UP6HMGCCDMCH6ZGA, seg_pos={0 0}, upload=510734]: Long tail cancellation due to optimal threshold Dec 29 22:06:21.929093137 (582h37m21.800176192s) [noise=true , address=145.79.227.35:4507 , piece_id=ZZSVIHOAN2SWXI4NS3TRD4B64NDB4D43J63DAS74TCPEE6MPGRXA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14jurWWPpmxwiBS3XmZysEUwugsceYJqVXFj7BbYpx8UFcTR8R): protocol: expected piece hash; context canceled; EOF Dec 29 22:06:21.929106793 (582h37m21.800189838s) [noise=true , address=145.79.227.35:4507 , piece_id=ZZSVIHOAN2SWXI4NS3TRD4B64NDB4D43J63DAS74TCPEE6MPGRXA, seg_pos={0 0}, upload=510734]: Long tail cancellation due to optimal threshold Dec 29 22:06:21.929206612 (582h37m21.800289657s) [noise=true , address=145.79.227.17:4995 , piece_id=JJTVPVBAMCTDGHGWNPUVPYFVKD4SMTDAHF4VWVIW7JFUAMBIPRHQ, seg_pos={0 0}, upload=510734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tvoTD1LoSj3q2rLiDckfT56dXNKKwdVu1ZAsBZExNFVTom9tH): protocol: write tcp 188.166.225.170:47696->145.79.227.17:4995: use of closed network connection; write tcp 188.166.225.170:47696->145.79.227.17:4995: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47696->145.79.227.17:4995: use of closed network connection Dec 29 22:06:21.929215911 (582h37m21.800298976s) [noise=true , address=145.79.227.17:4995 , piece_id=JJTVPVBAMCTDGHGWNPUVPYFVKD4SMTDAHF4VWVIW7JFUAMBIPRHQ, seg_pos={0 0}, upload=510734]: Long tail cancellation due to optimal threshold Dec 29 22:06:21.929387247 (582h37m21.800470292s) [noise=true , address=193.149.164.143:4698 , piece_id=6X5KB4TXFV4ZOF5JTPNRW4Q3GWNCVCXCA4JSB3AZURBHXY2VHTRQ, seg_pos={0 0}, upload=510734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F4fJAGJM2ZPRt7vJH2cZmEbYeujmxcfi7aPZ2g9VPTNYBYRAi): protocol: expected piece hash; context canceled; EOF Dec 29 22:06:21.929396174 (582h37m21.800479219s) [noise=true , address=193.149.164.143:4698 , piece_id=6X5KB4TXFV4ZOF5JTPNRW4Q3GWNCVCXCA4JSB3AZURBHXY2VHTRQ, seg_pos={0 0}, upload=510734]: Long tail cancellation due to optimal threshold Dec 29 22:06:21.929491536 (582h37m21.800574571s) [seg_pos={0 0}, upload=510734]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 22:06:21.929573792 (582h37m21.800656837s) [upload=510734]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 22:06:21.929580555 (582h37m21.8006636s) [upload=510734]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 22:06:22.233045366 (582h37m22.104128401s) [upload=510734]: Done uploading segments... Dec 29 22:06:22.233064963 (582h37m22.104147998s) [upload=510734]: Upload finished. err: <nil> Duration: 16.803810396s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..X..X..X........X............................X.............] duration: 5.942255427s trace: https://jaeger.dev.storj.io/trace/5209b3d035311fd FAIL (2025-12-29 22:10:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :00:17.043136250 (582h31m16.914219305s) [upload=510642]: Starting upload Dec 29 22:00:17.043227904 (582h31m16.91431096s) [upload=510642]: Done starting upload Dec 29 22:00:17.043335189 (582h31m16.914418234s) [upload=510642]: Uploading segments... Dec 29 22:00:17.043587749 (582h31m16.914670824s) [upload=510642]: Got next segment. Inline: false Dec 29 22:00:17.043644478 (582h31m16.914727523s) [upload=510642]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 22:00:17.043686427 (582h31m16.914769503s) [upload=510642]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 22:00:17.294384363 (582h31m17.165467408s) [seg_pos={0 0}, upload=510642]: Begin upload segment... Dec 29 22:00:17.294445219 (582h31m17.165528264s) [seg_pos={0 0}, upload=510642]: Done begin upload segment. Dec 29 22:00:17.294484133 (582h31m17.165567178s) [noise=true , address=145.79.227.31:4797 , piece_id=NI3M5NVVVX5HUXZ2SOMQD4S5ZOSSNPCD2F3Z5X2BA2VXMGT57AYQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.294560979 (582h31m17.165644014s) [noise=true , address=77.90.31.11:4553 , piece_id=3Y5HSD73DLZ6ALJ5TFRIVTLOE4ZNZ256YSZ2W3X5LPKL5J5LX2EQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.295149941 (582h31m17.166232976s) [noise=true , address=45.91.116.5:4297 , piece_id=PLUQ7Q32BTGYFQRGVJFAJWHPZCPQTC5RX5UX27S5HJ5NNJPLDMPQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.295419574 (582h31m17.166502609s) [noise=true , address=145.79.227.2:4115 , piece_id=P2AS5B3VYO2K2AWSM5W4BXAXBHSXQQWM7JOXFPOV64NTKDFLT62Q, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.295483997 (582h31m17.166567042s) [noise=true , address=145.79.227.51:4411 , piece_id=HTZQ6SW6KSACLQF7G4CFGQA572J5JVHMLGEWQYSSGZRYX7FJ6DXA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.295762207 (582h31m17.166845252s) [noise=true , address=145.79.227.5:4221 , piece_id=LD3FENLBM2L5N7QDOAMAYJUVH54QVSJQJFP7E36UAH2ZD7R65CCQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.295928003 (582h31m17.167011048s) [noise=true , address=145.79.227.16:4909 , piece_id=CF7WHNKC4LFJQRVFBPFV62JGEBLD4KVPE6Q5JBWQVADC6ZAFFLVQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.296039615 (582h31m17.16712266s) [noise=true , address=45.91.116.13:4154 , piece_id=FE4NFODQ3TWGFXARFYIL64TKK6X7H6PPLIG6LO44LKNAZOEU7DJA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.296272429 (582h31m17.167355474s) [noise=true , address=45.91.116.10:4084 , piece_id=5BFUYVGIFOYXC6RCPU7VBUM36LSM32TVBLS3W54OWFNR4SM6GCSA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.296403588 (582h31m17.167486633s) [noise=true , address=77.90.31.19:4844 , piece_id=CDOWECXLSRPACV4HMAH57UK4EEIYNOMH6GYV6HFHV7CWSEAJ44FA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.296646560 (582h31m17.167729596s) [noise=true , address=77.90.31.14:4661 , piece_id=B7IDSCDHBUXUIPSUT3HMQSCMPGBMYASZSQO2EJBXRR2DQHCLWGYA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.296906235 (582h31m17.167989291s) [noise=true , address=145.79.227.47:4265 , piece_id=FYZA2LRIFKLKVUCUAWMNWTUH3VYR5VFAA7XKAYIMD2E2VOJTI7PA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.296989804 (582h31m17.168072849s) [noise=true , address=145.79.227.46:4601 , piece_id=YJTMZID24VBM5ZZSUUHOCZSZSSNGTFHOSKK6OI5QP3A4G6ZWVWQA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.297235221 (582h31m17.168318267s) [noise=true , address=145.79.227.3:4150 , piece_id=HS4S5UDYD2VWI2N4MH6EWEYQ5G6KCFCJBDZR7AVWPXRYALFXP4FQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.297424432 (582h31m17.168507467s) [noise=true , address=145.79.227.13:4803 , piece_id=YIEUZEDLSC5C67SCFEPCHHZSTYP6MYLYJZJTGZF3UP6JWDTGUOPQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.297606900 (582h31m17.168689945s) [noise=true , address=45.91.116.14:4190 , piece_id=FIL6AIXGE7DE2M4QBVSIO53KGY43VFPZQ2FTXSWJFPJU4VZCKUXQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:18.995364155 (582h31m18.866447201s) [upload=510642]: Next returned nil segment Dec 29 22:00:18.995383603 (582h31m18.866466648s) [upload=510642]: Waiting for error group managing segments... Dec 29 22:00:19.284009408 (582h31m19.155092453s) [noise=true , address=145.79.227.51:4411 , piece_id=HTZQ6SW6KSACLQF7G4CFGQA572J5JVHMLGEWQYSSGZRYX7FJ6DXA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.295985232 (582h31m19.167068277s) [noise=true , address=145.79.227.13:4803 , piece_id=YIEUZEDLSC5C67SCFEPCHHZSTYP6MYLYJZJTGZF3UP6JWDTGUOPQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.296135458 (582h31m19.167218503s) [noise=true , address=145.79.227.16:4909 , piece_id=CF7WHNKC4LFJQRVFBPFV62JGEBLD4KVPE6Q5JBWQVADC6ZAFFLVQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.310119916 (582h31m19.181202971s) [noise=true , address=145.79.227.46:4601 , piece_id=YJTMZID24VBM5ZZSUUHOCZSZSSNGTFHOSKK6OI5QP3A4G6ZWVWQA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.310998939 (582h31m19.182081984s) [noise=true , address=145.79.227.31:4797 , piece_id=NI3M5NVVVX5HUXZ2SOMQD4S5ZOSSNPCD2F3Z5X2BA2VXMGT57AYQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.318707138 (582h31m19.189790183s) [noise=true , address=145.79.227.47:4265 , piece_id=FYZA2LRIFKLKVUCUAWMNWTUH3VYR5VFAA7XKAYIMD2E2VOJTI7PA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.322059381 (582h31m19.193142426s) [noise=true , address=145.79.227.5:4221 , piece_id=LD3FENLBM2L5N7QDOAMAYJUVH54QVSJQJFP7E36UAH2ZD7R65CCQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.506372108 (582h31m19.377455153s) [noise=true , address=77.90.31.19:4844 , piece_id=CDOWECXLSRPACV4HMAH57UK4EEIYNOMH6GYV6HFHV7CWSEAJ44FA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.507172602 (582h31m19.378255648s) [noise=true , address=77.90.31.11:4553 , piece_id=3Y5HSD73DLZ6ALJ5TFRIVTLOE4ZNZ256YSZ2W3X5LPKL5J5LX2EQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.527548842 (582h31m19.398631887s) [noise=true , address=77.90.31.14:4661 , piece_id=B7IDSCDHBUXUIPSUT3HMQSCMPGBMYASZSQO2EJBXRR2DQHCLWGYA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.531443508 (582h31m19.402526553s) [noise=true , address=45.91.116.5:4297 , piece_id=PLUQ7Q32BTGYFQRGVJFAJWHPZCPQTC5RX5UX27S5HJ5NNJPLDMPQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.552364906 (582h31m19.423447951s) [noise=true , address=45.91.116.14:4190 , piece_id=FIL6AIXGE7DE2M4QBVSIO53KGY43VFPZQ2FTXSWJFPJU4VZCKUXQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.564056769 (582h31m19.435139814s) [noise=true , address=45.91.116.10:4084 , piece_id=5BFUYVGIFOYXC6RCPU7VBUM36LSM32TVBLS3W54OWFNR4SM6GCSA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.564087016 (582h31m19.435170051s) [seg_pos={0 0}, upload=510642]: Segment reached optimal threshold of 13 pieces. Dec 29 22:00:19.564294431 (582h31m19.435377466s) [noise=true , address=145.79.227.3:4150 , piece_id=HS4S5UDYD2VWI2N4MH6EWEYQ5G6KCFCJBDZR7AVWPXRYALFXP4FQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK): protocol: expected piece hash; context canceled; EOF Dec 29 22:00:19.564302366 (582h31m19.435385411s) [noise=true , address=145.79.227.3:4150 , piece_id=HS4S5UDYD2VWI2N4MH6EWEYQ5G6KCFCJBDZR7AVWPXRYALFXP4FQ, seg_pos={0 0}, upload=510642]: Long tail cancellation due to optimal threshold Dec 29 22:00:19.564388932 (582h31m19.435471977s) [noise=true , address=45.91.116.13:4154 , piece_id=FE4NFODQ3TWGFXARFYIL64TKK6X7H6PPLIG6LO44LKNAZOEU7DJA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Dec 29 22:00:19.564401876 (582h31m19.435484911s) [noise=true , address=45.91.116.13:4154 , piece_id=FE4NFODQ3TWGFXARFYIL64TKK6X7H6PPLIG6LO44LKNAZOEU7DJA, seg_pos={0 0}, upload=510642]: Long tail cancellation due to optimal threshold Dec 29 22:00:19.564558945 (582h31m19.435641981s) [noise=true , address=145.79.227.2:4115 , piece_id=P2AS5B3VYO2K2AWSM5W4BXAXBHSXQQWM7JOXFPOV64NTKDFLT62Q, seg_pos={0 0}, upload=510642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: write tcp 188.166.225.170:38246->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:38246->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38246->145.79.227.2:4115: use of closed network connection Dec 29 22:00:19.564570738 (582h31m19.435653783s) [noise=true , address=145.79.227.2:4115 , piece_id=P2AS5B3VYO2K2AWSM5W4BXAXBHSXQQWM7JOXFPOV64NTKDFLT62Q, seg_pos={0 0}, upload=510642]: Long tail cancellation due to optimal threshold Dec 29 22:00:19.564601426 (582h31m19.435684471s) [seg_pos={0 0}, upload=510642]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 22:00:19.564683973 (582h31m19.435767018s) [upload=510642]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 22:00:19.564690004 (582h31m19.435773049s) [upload=510642]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 22:00:19.871860472 (582h31m19.742943527s) [upload=510642]: Done uploading segments... Dec 29 22:00:19.871892413 (582h31m19.742975468s) [upload=510642]: Upload finished. err: <nil> Duration: 5.942284122s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...X..X..X........X............................X............] duration: 6.675612613s trace: https://jaeger.dev.storj.io/trace/5209b3d035311fd FAIL (2025-12-29 22:10:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :00:17.043136250 (582h31m16.914219305s) [upload=510642]: Starting upload Dec 29 22:00:17.043227904 (582h31m16.91431096s) [upload=510642]: Done starting upload Dec 29 22:00:17.043335189 (582h31m16.914418234s) [upload=510642]: Uploading segments... Dec 29 22:00:17.043587749 (582h31m16.914670824s) [upload=510642]: Got next segment. Inline: false Dec 29 22:00:17.043644478 (582h31m16.914727523s) [upload=510642]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 22:00:17.043686427 (582h31m16.914769503s) [upload=510642]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 22:00:17.294384363 (582h31m17.165467408s) [seg_pos={0 0}, upload=510642]: Begin upload segment... Dec 29 22:00:17.294445219 (582h31m17.165528264s) [seg_pos={0 0}, upload=510642]: Done begin upload segment. Dec 29 22:00:17.294484133 (582h31m17.165567178s) [noise=true , address=145.79.227.31:4797 , piece_id=NI3M5NVVVX5HUXZ2SOMQD4S5ZOSSNPCD2F3Z5X2BA2VXMGT57AYQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.294560979 (582h31m17.165644014s) [noise=true , address=77.90.31.11:4553 , piece_id=3Y5HSD73DLZ6ALJ5TFRIVTLOE4ZNZ256YSZ2W3X5LPKL5J5LX2EQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.295149941 (582h31m17.166232976s) [noise=true , address=45.91.116.5:4297 , piece_id=PLUQ7Q32BTGYFQRGVJFAJWHPZCPQTC5RX5UX27S5HJ5NNJPLDMPQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.295419574 (582h31m17.166502609s) [noise=true , address=145.79.227.2:4115 , piece_id=P2AS5B3VYO2K2AWSM5W4BXAXBHSXQQWM7JOXFPOV64NTKDFLT62Q, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.295483997 (582h31m17.166567042s) [noise=true , address=145.79.227.51:4411 , piece_id=HTZQ6SW6KSACLQF7G4CFGQA572J5JVHMLGEWQYSSGZRYX7FJ6DXA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.295762207 (582h31m17.166845252s) [noise=true , address=145.79.227.5:4221 , piece_id=LD3FENLBM2L5N7QDOAMAYJUVH54QVSJQJFP7E36UAH2ZD7R65CCQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.295928003 (582h31m17.167011048s) [noise=true , address=145.79.227.16:4909 , piece_id=CF7WHNKC4LFJQRVFBPFV62JGEBLD4KVPE6Q5JBWQVADC6ZAFFLVQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.296039615 (582h31m17.16712266s) [noise=true , address=45.91.116.13:4154 , piece_id=FE4NFODQ3TWGFXARFYIL64TKK6X7H6PPLIG6LO44LKNAZOEU7DJA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.296272429 (582h31m17.167355474s) [noise=true , address=45.91.116.10:4084 , piece_id=5BFUYVGIFOYXC6RCPU7VBUM36LSM32TVBLS3W54OWFNR4SM6GCSA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.296403588 (582h31m17.167486633s) [noise=true , address=77.90.31.19:4844 , piece_id=CDOWECXLSRPACV4HMAH57UK4EEIYNOMH6GYV6HFHV7CWSEAJ44FA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.296646560 (582h31m17.167729596s) [noise=true , address=77.90.31.14:4661 , piece_id=B7IDSCDHBUXUIPSUT3HMQSCMPGBMYASZSQO2EJBXRR2DQHCLWGYA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.296906235 (582h31m17.167989291s) [noise=true , address=145.79.227.47:4265 , piece_id=FYZA2LRIFKLKVUCUAWMNWTUH3VYR5VFAA7XKAYIMD2E2VOJTI7PA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.296989804 (582h31m17.168072849s) [noise=true , address=145.79.227.46:4601 , piece_id=YJTMZID24VBM5ZZSUUHOCZSZSSNGTFHOSKK6OI5QP3A4G6ZWVWQA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.297235221 (582h31m17.168318267s) [noise=true , address=145.79.227.3:4150 , piece_id=HS4S5UDYD2VWI2N4MH6EWEYQ5G6KCFCJBDZR7AVWPXRYALFXP4FQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.297424432 (582h31m17.168507467s) [noise=true , address=145.79.227.13:4803 , piece_id=YIEUZEDLSC5C67SCFEPCHHZSTYP6MYLYJZJTGZF3UP6JWDTGUOPQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.297606900 (582h31m17.168689945s) [noise=true , address=45.91.116.14:4190 , piece_id=FIL6AIXGE7DE2M4QBVSIO53KGY43VFPZQ2FTXSWJFPJU4VZCKUXQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:18.995364155 (582h31m18.866447201s) [upload=510642]: Next returned nil segment Dec 29 22:00:18.995383603 (582h31m18.866466648s) [upload=510642]: Waiting for error group managing segments... Dec 29 22:00:19.284009408 (582h31m19.155092453s) [noise=true , address=145.79.227.51:4411 , piece_id=HTZQ6SW6KSACLQF7G4CFGQA572J5JVHMLGEWQYSSGZRYX7FJ6DXA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.295985232 (582h31m19.167068277s) [noise=true , address=145.79.227.13:4803 , piece_id=YIEUZEDLSC5C67SCFEPCHHZSTYP6MYLYJZJTGZF3UP6JWDTGUOPQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.296135458 (582h31m19.167218503s) [noise=true , address=145.79.227.16:4909 , piece_id=CF7WHNKC4LFJQRVFBPFV62JGEBLD4KVPE6Q5JBWQVADC6ZAFFLVQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.310119916 (582h31m19.181202971s) [noise=true , address=145.79.227.46:4601 , piece_id=YJTMZID24VBM5ZZSUUHOCZSZSSNGTFHOSKK6OI5QP3A4G6ZWVWQA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.310998939 (582h31m19.182081984s) [noise=true , address=145.79.227.31:4797 , piece_id=NI3M5NVVVX5HUXZ2SOMQD4S5ZOSSNPCD2F3Z5X2BA2VXMGT57AYQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.318707138 (582h31m19.189790183s) [noise=true , address=145.79.227.47:4265 , piece_id=FYZA2LRIFKLKVUCUAWMNWTUH3VYR5VFAA7XKAYIMD2E2VOJTI7PA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.322059381 (582h31m19.193142426s) [noise=true , address=145.79.227.5:4221 , piece_id=LD3FENLBM2L5N7QDOAMAYJUVH54QVSJQJFP7E36UAH2ZD7R65CCQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.506372108 (582h31m19.377455153s) [noise=true , address=77.90.31.19:4844 , piece_id=CDOWECXLSRPACV4HMAH57UK4EEIYNOMH6GYV6HFHV7CWSEAJ44FA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.507172602 (582h31m19.378255648s) [noise=true , address=77.90.31.11:4553 , piece_id=3Y5HSD73DLZ6ALJ5TFRIVTLOE4ZNZ256YSZ2W3X5LPKL5J5LX2EQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.527548842 (582h31m19.398631887s) [noise=true , address=77.90.31.14:4661 , piece_id=B7IDSCDHBUXUIPSUT3HMQSCMPGBMYASZSQO2EJBXRR2DQHCLWGYA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.531443508 (582h31m19.402526553s) [noise=true , address=45.91.116.5:4297 , piece_id=PLUQ7Q32BTGYFQRGVJFAJWHPZCPQTC5RX5UX27S5HJ5NNJPLDMPQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.552364906 (582h31m19.423447951s) [noise=true , address=45.91.116.14:4190 , piece_id=FIL6AIXGE7DE2M4QBVSIO53KGY43VFPZQ2FTXSWJFPJU4VZCKUXQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.564056769 (582h31m19.435139814s) [noise=true , address=45.91.116.10:4084 , piece_id=5BFUYVGIFOYXC6RCPU7VBUM36LSM32TVBLS3W54OWFNR4SM6GCSA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.564087016 (582h31m19.435170051s) [seg_pos={0 0}, upload=510642]: Segment reached optimal threshold of 13 pieces. Dec 29 22:00:19.564294431 (582h31m19.435377466s) [noise=true , address=145.79.227.3:4150 , piece_id=HS4S5UDYD2VWI2N4MH6EWEYQ5G6KCFCJBDZR7AVWPXRYALFXP4FQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK): protocol: expected piece hash; context canceled; EOF Dec 29 22:00:19.564302366 (582h31m19.435385411s) [noise=true , address=145.79.227.3:4150 , piece_id=HS4S5UDYD2VWI2N4MH6EWEYQ5G6KCFCJBDZR7AVWPXRYALFXP4FQ, seg_pos={0 0}, upload=510642]: Long tail cancellation due to optimal threshold Dec 29 22:00:19.564388932 (582h31m19.435471977s) [noise=true , address=45.91.116.13:4154 , piece_id=FE4NFODQ3TWGFXARFYIL64TKK6X7H6PPLIG6LO44LKNAZOEU7DJA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Dec 29 22:00:19.564401876 (582h31m19.435484911s) [noise=true , address=45.91.116.13:4154 , piece_id=FE4NFODQ3TWGFXARFYIL64TKK6X7H6PPLIG6LO44LKNAZOEU7DJA, seg_pos={0 0}, upload=510642]: Long tail cancellation due to optimal threshold Dec 29 22:00:19.564558945 (582h31m19.435641981s) [noise=true , address=145.79.227.2:4115 , piece_id=P2AS5B3VYO2K2AWSM5W4BXAXBHSXQQWM7JOXFPOV64NTKDFLT62Q, seg_pos={0 0}, upload=510642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: write tcp 188.166.225.170:38246->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:38246->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38246->145.79.227.2:4115: use of closed network connection Dec 29 22:00:19.564570738 (582h31m19.435653783s) [noise=true , address=145.79.227.2:4115 , piece_id=P2AS5B3VYO2K2AWSM5W4BXAXBHSXQQWM7JOXFPOV64NTKDFLT62Q, seg_pos={0 0}, upload=510642]: Long tail cancellation due to optimal threshold Dec 29 22:00:19.564601426 (582h31m19.435684471s) [seg_pos={0 0}, upload=510642]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 22:00:19.564683973 (582h31m19.435767018s) [upload=510642]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 22:00:19.564690004 (582h31m19.435773049s) [upload=510642]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 22:00:19.871860472 (582h31m19.742943527s) [upload=510642]: Done uploading segments... Dec 29 22:00:19.871892413 (582h31m19.742975468s) [upload=510642]: Upload finished. err: <nil> Duration: 6.675639916s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................X............................X....] duration: 15.758422856s trace: https://jaeger.dev.storj.io/trace/7096862780f29049 FAIL (2025-12-29 22:10:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== oise=true , address=145.79.227.30:4766 , piece_id=YWDMUGNTYVXWOCMSN63VIC6HOTWHPY7VJ7QWT76P63UAJIBS2XLA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.060493636 (582h37m18.931576671s) [noise=true , address=5.231.73.5:4022 , piece_id=5WVAJH3Y6TLATXY4IYB6ZADKKJDXDS52B2ZN6FJAZ7UHFVRYRRAA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.060685291 (582h37m18.931768326s) [noise=true , address=77.90.31.11:4579 , piece_id=DEJPYHGYNAP52CZ45BKGNXJM5MLIA2OQZ4BRKN5PBADNRVE3VLEA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.060749954 (582h37m18.931832989s) [noise=true , address=145.79.227.29:4976 , piece_id=YRCRXSSQOXPOJGVX46L7WCB3TRDZ5QB755SEOTNHSCHN4ZDMCOUQ, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.061033203 (582h37m18.932116248s) [noise=true , address=193.149.164.143:4698 , piece_id=6X5KB4TXFV4ZOF5JTPNRW4Q3GWNCVCXCA4JSB3AZURBHXY2VHTRQ, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.061248855 (582h37m18.9323319s) [noise=true , address=145.79.227.49:4347 , piece_id=75B3DRRHEPDL6EK6ZUXTKASXUKWGAVUWBPSTNRYT4BJMAU5ALXRA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.061382729 (582h37m18.932465764s) [noise=true , address=5.231.73.2:4004 , piece_id=PTHHO3XEPGGCIRCUBIWHN4LVNBLMISJM5LESQGR2ERRESFK3S4RA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.061645560 (582h37m18.932728595s) [noise=true , address=5.231.73.3:4007 , piece_id=6AF3CEPLUMRO4CDW3SRWZSYHWEXP35AVSM5M3LH4KCRARSZ6NFNA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.061769094 (582h37m18.932852139s) [noise=true , address=145.79.227.7:4321 , piece_id=SPWPZ5MR5ZCQNYVYYQQEZC4OURAM5VAH3WN5UP6HMGCCDMCH6ZGA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.061970298 (582h37m18.933053333s) [noise=true , address=145.79.227.35:4507 , piece_id=ZZSVIHOAN2SWXI4NS3TRD4B64NDB4D43J63DAS74TCPEE6MPGRXA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.062237577 (582h37m18.933320612s) [noise=true , address=145.79.227.35:4489 , piece_id=BM7ZQESTHEUIRJWAPORTKDMT5TVLEXMOHULGFVEQKHT4AJV3HT5A, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.062304304 (582h37m18.933387349s) [noise=true , address=145.79.227.34:4931 , piece_id=XJHLE3B5HIF4RC4LV32SBCNMVTOTEXWBBCXBVPQFN6N5HFQ6ZS7A, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.062522640 (582h37m18.933605685s) [noise=true , address=145.79.227.48:4329 , piece_id=YWCVTYLCAGU57SE3ID3WW2JLCJGTJ7CPDRCZENAMO42BVPIAVLIQ, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.062904968 (582h37m18.933988003s) [noise=true , address=77.90.31.17:4801 , piece_id=K3QD7FPVRQHZL5R57EKKAC27RGDJQH6HDCND7FIVXSZBRH42WSIA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.063420419 (582h37m18.934503464s) [noise=true , address=145.79.227.17:4995 , piece_id=JJTVPVBAMCTDGHGWNPUVPYFVKD4SMTDAHF4VWVIW7JFUAMBIPRHQ, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.063943495 (582h37m18.93502654s) [noise=true , address=5.231.73.4:4018 , piece_id=X45H47HER27IM5FEQSEAMXSDX7UQG3AEOY65Q4WCCUOX7SVFZ5KA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:20.386418430 (582h37m20.257501465s) [upload=510734]: Next returned nil segment Dec 29 22:06:20.386428990 (582h37m20.257512025s) [upload=510734]: Waiting for error group managing segments... Dec 29 22:06:20.935543714 (582h37m20.806626749s) [noise=true , address=145.79.227.48:4329 , piece_id=YWCVTYLCAGU57SE3ID3WW2JLCJGTJ7CPDRCZENAMO42BVPIAVLIQ, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:20.978073277 (582h37m20.849156312s) [noise=true , address=145.79.227.39:4022 , piece_id=5IJCSVRVXCICRNRB3TTJAVNRF5FTUQMMY4KL7ZFF4SBKILFGHJWA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.079004013 (582h37m20.950087058s) [noise=true , address=5.231.73.4:4018 , piece_id=X45H47HER27IM5FEQSEAMXSDX7UQG3AEOY65Q4WCCUOX7SVFZ5KA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.101144552 (582h37m20.972227597s) [noise=true , address=77.90.31.11:4579 , piece_id=DEJPYHGYNAP52CZ45BKGNXJM5MLIA2OQZ4BRKN5PBADNRVE3VLEA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.138766369 (582h37m21.009849405s) [noise=true , address=77.90.31.17:4801 , piece_id=K3QD7FPVRQHZL5R57EKKAC27RGDJQH6HDCND7FIVXSZBRH42WSIA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.231047035 (582h37m21.10213007s) [noise=true , address=77.90.31.19:4857 , piece_id=DMJ4O3TWZYWKWVLJFSQBULHDFWZCNLVDBIGYJEOBAVVCU5UGG27Q, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.272062706 (582h37m21.143145741s) [noise=true , address=5.231.73.3:4007 , piece_id=6AF3CEPLUMRO4CDW3SRWZSYHWEXP35AVSM5M3LH4KCRARSZ6NFNA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.303520666 (582h37m21.174603711s) [noise=true , address=145.79.227.29:4976 , piece_id=YRCRXSSQOXPOJGVX46L7WCB3TRDZ5QB755SEOTNHSCHN4ZDMCOUQ, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.340640177 (582h37m21.211723212s) [noise=true , address=5.231.73.5:4022 , piece_id=5WVAJH3Y6TLATXY4IYB6ZADKKJDXDS52B2ZN6FJAZ7UHFVRYRRAA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.481118228 (582h37m21.352201263s) [noise=true , address=5.231.73.2:4004 , piece_id=PTHHO3XEPGGCIRCUBIWHN4LVNBLMISJM5LESQGR2ERRESFK3S4RA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.896953769 (582h37m21.768036804s) [noise=true , address=145.79.227.49:4347 , piece_id=75B3DRRHEPDL6EK6ZUXTKASXUKWGAVUWBPSTNRYT4BJMAU5ALXRA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.911314353 (582h37m21.782397399s) [noise=true , address=145.79.227.34:4931 , piece_id=XJHLE3B5HIF4RC4LV32SBCNMVTOTEXWBBCXBVPQFN6N5HFQ6ZS7A, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.915845371 (582h37m21.786928406s) [noise=true , address=145.79.227.30:4766 , piece_id=YWDMUGNTYVXWOCMSN63VIC6HOTWHPY7VJ7QWT76P63UAJIBS2XLA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.928785972 (582h37m21.799869017s) [noise=true , address=145.79.227.35:4489 , piece_id=BM7ZQESTHEUIRJWAPORTKDMT5TVLEXMOHULGFVEQKHT4AJV3HT5A, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.928810859 (582h37m21.799893904s) [seg_pos={0 0}, upload=510734]: Segment reached optimal threshold of 14 pieces. Dec 29 22:06:21.929016471 (582h37m21.800099516s) [noise=true , address=145.79.227.7:4321 , piece_id=SPWPZ5MR5ZCQNYVYYQQEZC4OURAM5VAH3WN5UP6HMGCCDMCH6ZGA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12edsHqD9iRBDgMU8vYhSM1EWG8Sn57GZ8TPiTytri3rp7W9Eae): protocol: expected piece hash; context canceled; EOF Dec 29 22:06:21.929028644 (582h37m21.800111709s) [noise=true , address=145.79.227.7:4321 , piece_id=SPWPZ5MR5ZCQNYVYYQQEZC4OURAM5VAH3WN5UP6HMGCCDMCH6ZGA, seg_pos={0 0}, upload=510734]: Long tail cancellation due to optimal threshold Dec 29 22:06:21.929093137 (582h37m21.800176192s) [noise=true , address=145.79.227.35:4507 , piece_id=ZZSVIHOAN2SWXI4NS3TRD4B64NDB4D43J63DAS74TCPEE6MPGRXA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14jurWWPpmxwiBS3XmZysEUwugsceYJqVXFj7BbYpx8UFcTR8R): protocol: expected piece hash; context canceled; EOF Dec 29 22:06:21.929106793 (582h37m21.800189838s) [noise=true , address=145.79.227.35:4507 , piece_id=ZZSVIHOAN2SWXI4NS3TRD4B64NDB4D43J63DAS74TCPEE6MPGRXA, seg_pos={0 0}, upload=510734]: Long tail cancellation due to optimal threshold Dec 29 22:06:21.929206612 (582h37m21.800289657s) [noise=true , address=145.79.227.17:4995 , piece_id=JJTVPVBAMCTDGHGWNPUVPYFVKD4SMTDAHF4VWVIW7JFUAMBIPRHQ, seg_pos={0 0}, upload=510734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tvoTD1LoSj3q2rLiDckfT56dXNKKwdVu1ZAsBZExNFVTom9tH): protocol: write tcp 188.166.225.170:47696->145.79.227.17:4995: use of closed network connection; write tcp 188.166.225.170:47696->145.79.227.17:4995: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47696->145.79.227.17:4995: use of closed network connection Dec 29 22:06:21.929215911 (582h37m21.800298976s) [noise=true , address=145.79.227.17:4995 , piece_id=JJTVPVBAMCTDGHGWNPUVPYFVKD4SMTDAHF4VWVIW7JFUAMBIPRHQ, seg_pos={0 0}, upload=510734]: Long tail cancellation due to optimal threshold Dec 29 22:06:21.929387247 (582h37m21.800470292s) [noise=true , address=193.149.164.143:4698 , piece_id=6X5KB4TXFV4ZOF5JTPNRW4Q3GWNCVCXCA4JSB3AZURBHXY2VHTRQ, seg_pos={0 0}, upload=510734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F4fJAGJM2ZPRt7vJH2cZmEbYeujmxcfi7aPZ2g9VPTNYBYRAi): protocol: expected piece hash; context canceled; EOF Dec 29 22:06:21.929396174 (582h37m21.800479219s) [noise=true , address=193.149.164.143:4698 , piece_id=6X5KB4TXFV4ZOF5JTPNRW4Q3GWNCVCXCA4JSB3AZURBHXY2VHTRQ, seg_pos={0 0}, upload=510734]: Long tail cancellation due to optimal threshold Dec 29 22:06:21.929491536 (582h37m21.800574571s) [seg_pos={0 0}, upload=510734]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 22:06:21.929573792 (582h37m21.800656837s) [upload=510734]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 22:06:21.929580555 (582h37m21.8006636s) [upload=510734]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 22:06:22.233045366 (582h37m22.104128401s) [upload=510734]: Done uploading segments... Dec 29 22:06:22.233064963 (582h37m22.104147998s) [upload=510734]: Upload finished. err: <nil> Duration: 15.758451571s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [....X..X..X........X............................X...........] duration: 6.037714467s trace: https://jaeger.dev.storj.io/trace/5209b3d035311fd FAIL (2025-12-29 22:09:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :00:17.043136250 (582h31m16.914219305s) [upload=510642]: Starting upload Dec 29 22:00:17.043227904 (582h31m16.91431096s) [upload=510642]: Done starting upload Dec 29 22:00:17.043335189 (582h31m16.914418234s) [upload=510642]: Uploading segments... Dec 29 22:00:17.043587749 (582h31m16.914670824s) [upload=510642]: Got next segment. Inline: false Dec 29 22:00:17.043644478 (582h31m16.914727523s) [upload=510642]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 22:00:17.043686427 (582h31m16.914769503s) [upload=510642]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 22:00:17.294384363 (582h31m17.165467408s) [seg_pos={0 0}, upload=510642]: Begin upload segment... Dec 29 22:00:17.294445219 (582h31m17.165528264s) [seg_pos={0 0}, upload=510642]: Done begin upload segment. Dec 29 22:00:17.294484133 (582h31m17.165567178s) [noise=true , address=145.79.227.31:4797 , piece_id=NI3M5NVVVX5HUXZ2SOMQD4S5ZOSSNPCD2F3Z5X2BA2VXMGT57AYQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.294560979 (582h31m17.165644014s) [noise=true , address=77.90.31.11:4553 , piece_id=3Y5HSD73DLZ6ALJ5TFRIVTLOE4ZNZ256YSZ2W3X5LPKL5J5LX2EQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.295149941 (582h31m17.166232976s) [noise=true , address=45.91.116.5:4297 , piece_id=PLUQ7Q32BTGYFQRGVJFAJWHPZCPQTC5RX5UX27S5HJ5NNJPLDMPQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.295419574 (582h31m17.166502609s) [noise=true , address=145.79.227.2:4115 , piece_id=P2AS5B3VYO2K2AWSM5W4BXAXBHSXQQWM7JOXFPOV64NTKDFLT62Q, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.295483997 (582h31m17.166567042s) [noise=true , address=145.79.227.51:4411 , piece_id=HTZQ6SW6KSACLQF7G4CFGQA572J5JVHMLGEWQYSSGZRYX7FJ6DXA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.295762207 (582h31m17.166845252s) [noise=true , address=145.79.227.5:4221 , piece_id=LD3FENLBM2L5N7QDOAMAYJUVH54QVSJQJFP7E36UAH2ZD7R65CCQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.295928003 (582h31m17.167011048s) [noise=true , address=145.79.227.16:4909 , piece_id=CF7WHNKC4LFJQRVFBPFV62JGEBLD4KVPE6Q5JBWQVADC6ZAFFLVQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.296039615 (582h31m17.16712266s) [noise=true , address=45.91.116.13:4154 , piece_id=FE4NFODQ3TWGFXARFYIL64TKK6X7H6PPLIG6LO44LKNAZOEU7DJA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.296272429 (582h31m17.167355474s) [noise=true , address=45.91.116.10:4084 , piece_id=5BFUYVGIFOYXC6RCPU7VBUM36LSM32TVBLS3W54OWFNR4SM6GCSA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.296403588 (582h31m17.167486633s) [noise=true , address=77.90.31.19:4844 , piece_id=CDOWECXLSRPACV4HMAH57UK4EEIYNOMH6GYV6HFHV7CWSEAJ44FA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.296646560 (582h31m17.167729596s) [noise=true , address=77.90.31.14:4661 , piece_id=B7IDSCDHBUXUIPSUT3HMQSCMPGBMYASZSQO2EJBXRR2DQHCLWGYA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.296906235 (582h31m17.167989291s) [noise=true , address=145.79.227.47:4265 , piece_id=FYZA2LRIFKLKVUCUAWMNWTUH3VYR5VFAA7XKAYIMD2E2VOJTI7PA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.296989804 (582h31m17.168072849s) [noise=true , address=145.79.227.46:4601 , piece_id=YJTMZID24VBM5ZZSUUHOCZSZSSNGTFHOSKK6OI5QP3A4G6ZWVWQA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.297235221 (582h31m17.168318267s) [noise=true , address=145.79.227.3:4150 , piece_id=HS4S5UDYD2VWI2N4MH6EWEYQ5G6KCFCJBDZR7AVWPXRYALFXP4FQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.297424432 (582h31m17.168507467s) [noise=true , address=145.79.227.13:4803 , piece_id=YIEUZEDLSC5C67SCFEPCHHZSTYP6MYLYJZJTGZF3UP6JWDTGUOPQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.297606900 (582h31m17.168689945s) [noise=true , address=45.91.116.14:4190 , piece_id=FIL6AIXGE7DE2M4QBVSIO53KGY43VFPZQ2FTXSWJFPJU4VZCKUXQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:18.995364155 (582h31m18.866447201s) [upload=510642]: Next returned nil segment Dec 29 22:00:18.995383603 (582h31m18.866466648s) [upload=510642]: Waiting for error group managing segments... Dec 29 22:00:19.284009408 (582h31m19.155092453s) [noise=true , address=145.79.227.51:4411 , piece_id=HTZQ6SW6KSACLQF7G4CFGQA572J5JVHMLGEWQYSSGZRYX7FJ6DXA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.295985232 (582h31m19.167068277s) [noise=true , address=145.79.227.13:4803 , piece_id=YIEUZEDLSC5C67SCFEPCHHZSTYP6MYLYJZJTGZF3UP6JWDTGUOPQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.296135458 (582h31m19.167218503s) [noise=true , address=145.79.227.16:4909 , piece_id=CF7WHNKC4LFJQRVFBPFV62JGEBLD4KVPE6Q5JBWQVADC6ZAFFLVQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.310119916 (582h31m19.181202971s) [noise=true , address=145.79.227.46:4601 , piece_id=YJTMZID24VBM5ZZSUUHOCZSZSSNGTFHOSKK6OI5QP3A4G6ZWVWQA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.310998939 (582h31m19.182081984s) [noise=true , address=145.79.227.31:4797 , piece_id=NI3M5NVVVX5HUXZ2SOMQD4S5ZOSSNPCD2F3Z5X2BA2VXMGT57AYQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.318707138 (582h31m19.189790183s) [noise=true , address=145.79.227.47:4265 , piece_id=FYZA2LRIFKLKVUCUAWMNWTUH3VYR5VFAA7XKAYIMD2E2VOJTI7PA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.322059381 (582h31m19.193142426s) [noise=true , address=145.79.227.5:4221 , piece_id=LD3FENLBM2L5N7QDOAMAYJUVH54QVSJQJFP7E36UAH2ZD7R65CCQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.506372108 (582h31m19.377455153s) [noise=true , address=77.90.31.19:4844 , piece_id=CDOWECXLSRPACV4HMAH57UK4EEIYNOMH6GYV6HFHV7CWSEAJ44FA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.507172602 (582h31m19.378255648s) [noise=true , address=77.90.31.11:4553 , piece_id=3Y5HSD73DLZ6ALJ5TFRIVTLOE4ZNZ256YSZ2W3X5LPKL5J5LX2EQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.527548842 (582h31m19.398631887s) [noise=true , address=77.90.31.14:4661 , piece_id=B7IDSCDHBUXUIPSUT3HMQSCMPGBMYASZSQO2EJBXRR2DQHCLWGYA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.531443508 (582h31m19.402526553s) [noise=true , address=45.91.116.5:4297 , piece_id=PLUQ7Q32BTGYFQRGVJFAJWHPZCPQTC5RX5UX27S5HJ5NNJPLDMPQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.552364906 (582h31m19.423447951s) [noise=true , address=45.91.116.14:4190 , piece_id=FIL6AIXGE7DE2M4QBVSIO53KGY43VFPZQ2FTXSWJFPJU4VZCKUXQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.564056769 (582h31m19.435139814s) [noise=true , address=45.91.116.10:4084 , piece_id=5BFUYVGIFOYXC6RCPU7VBUM36LSM32TVBLS3W54OWFNR4SM6GCSA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.564087016 (582h31m19.435170051s) [seg_pos={0 0}, upload=510642]: Segment reached optimal threshold of 13 pieces. Dec 29 22:00:19.564294431 (582h31m19.435377466s) [noise=true , address=145.79.227.3:4150 , piece_id=HS4S5UDYD2VWI2N4MH6EWEYQ5G6KCFCJBDZR7AVWPXRYALFXP4FQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK): protocol: expected piece hash; context canceled; EOF Dec 29 22:00:19.564302366 (582h31m19.435385411s) [noise=true , address=145.79.227.3:4150 , piece_id=HS4S5UDYD2VWI2N4MH6EWEYQ5G6KCFCJBDZR7AVWPXRYALFXP4FQ, seg_pos={0 0}, upload=510642]: Long tail cancellation due to optimal threshold Dec 29 22:00:19.564388932 (582h31m19.435471977s) [noise=true , address=45.91.116.13:4154 , piece_id=FE4NFODQ3TWGFXARFYIL64TKK6X7H6PPLIG6LO44LKNAZOEU7DJA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Dec 29 22:00:19.564401876 (582h31m19.435484911s) [noise=true , address=45.91.116.13:4154 , piece_id=FE4NFODQ3TWGFXARFYIL64TKK6X7H6PPLIG6LO44LKNAZOEU7DJA, seg_pos={0 0}, upload=510642]: Long tail cancellation due to optimal threshold Dec 29 22:00:19.564558945 (582h31m19.435641981s) [noise=true , address=145.79.227.2:4115 , piece_id=P2AS5B3VYO2K2AWSM5W4BXAXBHSXQQWM7JOXFPOV64NTKDFLT62Q, seg_pos={0 0}, upload=510642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: write tcp 188.166.225.170:38246->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:38246->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38246->145.79.227.2:4115: use of closed network connection Dec 29 22:00:19.564570738 (582h31m19.435653783s) [noise=true , address=145.79.227.2:4115 , piece_id=P2AS5B3VYO2K2AWSM5W4BXAXBHSXQQWM7JOXFPOV64NTKDFLT62Q, seg_pos={0 0}, upload=510642]: Long tail cancellation due to optimal threshold Dec 29 22:00:19.564601426 (582h31m19.435684471s) [seg_pos={0 0}, upload=510642]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 22:00:19.564683973 (582h31m19.435767018s) [upload=510642]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 22:00:19.564690004 (582h31m19.435773049s) [upload=510642]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 22:00:19.871860472 (582h31m19.742943527s) [upload=510642]: Done uploading segments... Dec 29 22:00:19.871892413 (582h31m19.742975468s) [upload=510642]: Upload finished. err: <nil> Duration: 6.037742551s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........................X............................X...] duration: 15.313900179s trace: https://jaeger.dev.storj.io/trace/7096862780f29049 FAIL (2025-12-29 22:09:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== oise=true , address=145.79.227.30:4766 , piece_id=YWDMUGNTYVXWOCMSN63VIC6HOTWHPY7VJ7QWT76P63UAJIBS2XLA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.060493636 (582h37m18.931576671s) [noise=true , address=5.231.73.5:4022 , piece_id=5WVAJH3Y6TLATXY4IYB6ZADKKJDXDS52B2ZN6FJAZ7UHFVRYRRAA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.060685291 (582h37m18.931768326s) [noise=true , address=77.90.31.11:4579 , piece_id=DEJPYHGYNAP52CZ45BKGNXJM5MLIA2OQZ4BRKN5PBADNRVE3VLEA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.060749954 (582h37m18.931832989s) [noise=true , address=145.79.227.29:4976 , piece_id=YRCRXSSQOXPOJGVX46L7WCB3TRDZ5QB755SEOTNHSCHN4ZDMCOUQ, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.061033203 (582h37m18.932116248s) [noise=true , address=193.149.164.143:4698 , piece_id=6X5KB4TXFV4ZOF5JTPNRW4Q3GWNCVCXCA4JSB3AZURBHXY2VHTRQ, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.061248855 (582h37m18.9323319s) [noise=true , address=145.79.227.49:4347 , piece_id=75B3DRRHEPDL6EK6ZUXTKASXUKWGAVUWBPSTNRYT4BJMAU5ALXRA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.061382729 (582h37m18.932465764s) [noise=true , address=5.231.73.2:4004 , piece_id=PTHHO3XEPGGCIRCUBIWHN4LVNBLMISJM5LESQGR2ERRESFK3S4RA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.061645560 (582h37m18.932728595s) [noise=true , address=5.231.73.3:4007 , piece_id=6AF3CEPLUMRO4CDW3SRWZSYHWEXP35AVSM5M3LH4KCRARSZ6NFNA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.061769094 (582h37m18.932852139s) [noise=true , address=145.79.227.7:4321 , piece_id=SPWPZ5MR5ZCQNYVYYQQEZC4OURAM5VAH3WN5UP6HMGCCDMCH6ZGA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.061970298 (582h37m18.933053333s) [noise=true , address=145.79.227.35:4507 , piece_id=ZZSVIHOAN2SWXI4NS3TRD4B64NDB4D43J63DAS74TCPEE6MPGRXA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.062237577 (582h37m18.933320612s) [noise=true , address=145.79.227.35:4489 , piece_id=BM7ZQESTHEUIRJWAPORTKDMT5TVLEXMOHULGFVEQKHT4AJV3HT5A, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.062304304 (582h37m18.933387349s) [noise=true , address=145.79.227.34:4931 , piece_id=XJHLE3B5HIF4RC4LV32SBCNMVTOTEXWBBCXBVPQFN6N5HFQ6ZS7A, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.062522640 (582h37m18.933605685s) [noise=true , address=145.79.227.48:4329 , piece_id=YWCVTYLCAGU57SE3ID3WW2JLCJGTJ7CPDRCZENAMO42BVPIAVLIQ, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.062904968 (582h37m18.933988003s) [noise=true , address=77.90.31.17:4801 , piece_id=K3QD7FPVRQHZL5R57EKKAC27RGDJQH6HDCND7FIVXSZBRH42WSIA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.063420419 (582h37m18.934503464s) [noise=true , address=145.79.227.17:4995 , piece_id=JJTVPVBAMCTDGHGWNPUVPYFVKD4SMTDAHF4VWVIW7JFUAMBIPRHQ, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.063943495 (582h37m18.93502654s) [noise=true , address=5.231.73.4:4018 , piece_id=X45H47HER27IM5FEQSEAMXSDX7UQG3AEOY65Q4WCCUOX7SVFZ5KA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:20.386418430 (582h37m20.257501465s) [upload=510734]: Next returned nil segment Dec 29 22:06:20.386428990 (582h37m20.257512025s) [upload=510734]: Waiting for error group managing segments... Dec 29 22:06:20.935543714 (582h37m20.806626749s) [noise=true , address=145.79.227.48:4329 , piece_id=YWCVTYLCAGU57SE3ID3WW2JLCJGTJ7CPDRCZENAMO42BVPIAVLIQ, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:20.978073277 (582h37m20.849156312s) [noise=true , address=145.79.227.39:4022 , piece_id=5IJCSVRVXCICRNRB3TTJAVNRF5FTUQMMY4KL7ZFF4SBKILFGHJWA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.079004013 (582h37m20.950087058s) [noise=true , address=5.231.73.4:4018 , piece_id=X45H47HER27IM5FEQSEAMXSDX7UQG3AEOY65Q4WCCUOX7SVFZ5KA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.101144552 (582h37m20.972227597s) [noise=true , address=77.90.31.11:4579 , piece_id=DEJPYHGYNAP52CZ45BKGNXJM5MLIA2OQZ4BRKN5PBADNRVE3VLEA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.138766369 (582h37m21.009849405s) [noise=true , address=77.90.31.17:4801 , piece_id=K3QD7FPVRQHZL5R57EKKAC27RGDJQH6HDCND7FIVXSZBRH42WSIA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.231047035 (582h37m21.10213007s) [noise=true , address=77.90.31.19:4857 , piece_id=DMJ4O3TWZYWKWVLJFSQBULHDFWZCNLVDBIGYJEOBAVVCU5UGG27Q, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.272062706 (582h37m21.143145741s) [noise=true , address=5.231.73.3:4007 , piece_id=6AF3CEPLUMRO4CDW3SRWZSYHWEXP35AVSM5M3LH4KCRARSZ6NFNA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.303520666 (582h37m21.174603711s) [noise=true , address=145.79.227.29:4976 , piece_id=YRCRXSSQOXPOJGVX46L7WCB3TRDZ5QB755SEOTNHSCHN4ZDMCOUQ, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.340640177 (582h37m21.211723212s) [noise=true , address=5.231.73.5:4022 , piece_id=5WVAJH3Y6TLATXY4IYB6ZADKKJDXDS52B2ZN6FJAZ7UHFVRYRRAA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.481118228 (582h37m21.352201263s) [noise=true , address=5.231.73.2:4004 , piece_id=PTHHO3XEPGGCIRCUBIWHN4LVNBLMISJM5LESQGR2ERRESFK3S4RA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.896953769 (582h37m21.768036804s) [noise=true , address=145.79.227.49:4347 , piece_id=75B3DRRHEPDL6EK6ZUXTKASXUKWGAVUWBPSTNRYT4BJMAU5ALXRA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.911314353 (582h37m21.782397399s) [noise=true , address=145.79.227.34:4931 , piece_id=XJHLE3B5HIF4RC4LV32SBCNMVTOTEXWBBCXBVPQFN6N5HFQ6ZS7A, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.915845371 (582h37m21.786928406s) [noise=true , address=145.79.227.30:4766 , piece_id=YWDMUGNTYVXWOCMSN63VIC6HOTWHPY7VJ7QWT76P63UAJIBS2XLA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.928785972 (582h37m21.799869017s) [noise=true , address=145.79.227.35:4489 , piece_id=BM7ZQESTHEUIRJWAPORTKDMT5TVLEXMOHULGFVEQKHT4AJV3HT5A, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.928810859 (582h37m21.799893904s) [seg_pos={0 0}, upload=510734]: Segment reached optimal threshold of 14 pieces. Dec 29 22:06:21.929016471 (582h37m21.800099516s) [noise=true , address=145.79.227.7:4321 , piece_id=SPWPZ5MR5ZCQNYVYYQQEZC4OURAM5VAH3WN5UP6HMGCCDMCH6ZGA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12edsHqD9iRBDgMU8vYhSM1EWG8Sn57GZ8TPiTytri3rp7W9Eae): protocol: expected piece hash; context canceled; EOF Dec 29 22:06:21.929028644 (582h37m21.800111709s) [noise=true , address=145.79.227.7:4321 , piece_id=SPWPZ5MR5ZCQNYVYYQQEZC4OURAM5VAH3WN5UP6HMGCCDMCH6ZGA, seg_pos={0 0}, upload=510734]: Long tail cancellation due to optimal threshold Dec 29 22:06:21.929093137 (582h37m21.800176192s) [noise=true , address=145.79.227.35:4507 , piece_id=ZZSVIHOAN2SWXI4NS3TRD4B64NDB4D43J63DAS74TCPEE6MPGRXA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14jurWWPpmxwiBS3XmZysEUwugsceYJqVXFj7BbYpx8UFcTR8R): protocol: expected piece hash; context canceled; EOF Dec 29 22:06:21.929106793 (582h37m21.800189838s) [noise=true , address=145.79.227.35:4507 , piece_id=ZZSVIHOAN2SWXI4NS3TRD4B64NDB4D43J63DAS74TCPEE6MPGRXA, seg_pos={0 0}, upload=510734]: Long tail cancellation due to optimal threshold Dec 29 22:06:21.929206612 (582h37m21.800289657s) [noise=true , address=145.79.227.17:4995 , piece_id=JJTVPVBAMCTDGHGWNPUVPYFVKD4SMTDAHF4VWVIW7JFUAMBIPRHQ, seg_pos={0 0}, upload=510734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tvoTD1LoSj3q2rLiDckfT56dXNKKwdVu1ZAsBZExNFVTom9tH): protocol: write tcp 188.166.225.170:47696->145.79.227.17:4995: use of closed network connection; write tcp 188.166.225.170:47696->145.79.227.17:4995: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47696->145.79.227.17:4995: use of closed network connection Dec 29 22:06:21.929215911 (582h37m21.800298976s) [noise=true , address=145.79.227.17:4995 , piece_id=JJTVPVBAMCTDGHGWNPUVPYFVKD4SMTDAHF4VWVIW7JFUAMBIPRHQ, seg_pos={0 0}, upload=510734]: Long tail cancellation due to optimal threshold Dec 29 22:06:21.929387247 (582h37m21.800470292s) [noise=true , address=193.149.164.143:4698 , piece_id=6X5KB4TXFV4ZOF5JTPNRW4Q3GWNCVCXCA4JSB3AZURBHXY2VHTRQ, seg_pos={0 0}, upload=510734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F4fJAGJM2ZPRt7vJH2cZmEbYeujmxcfi7aPZ2g9VPTNYBYRAi): protocol: expected piece hash; context canceled; EOF Dec 29 22:06:21.929396174 (582h37m21.800479219s) [noise=true , address=193.149.164.143:4698 , piece_id=6X5KB4TXFV4ZOF5JTPNRW4Q3GWNCVCXCA4JSB3AZURBHXY2VHTRQ, seg_pos={0 0}, upload=510734]: Long tail cancellation due to optimal threshold Dec 29 22:06:21.929491536 (582h37m21.800574571s) [seg_pos={0 0}, upload=510734]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 22:06:21.929573792 (582h37m21.800656837s) [upload=510734]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 22:06:21.929580555 (582h37m21.8006636s) [upload=510734]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 22:06:22.233045366 (582h37m22.104128401s) [upload=510734]: Done uploading segments... Dec 29 22:06:22.233064963 (582h37m22.104147998s) [upload=510734]: Upload finished. err: <nil> Duration: 15.313931419s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.....X..X..X........X............................X..........] duration: 6.138362342s trace: https://jaeger.dev.storj.io/trace/5209b3d035311fd FAIL (2025-12-29 22:08:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :00:17.043136250 (582h31m16.914219305s) [upload=510642]: Starting upload Dec 29 22:00:17.043227904 (582h31m16.91431096s) [upload=510642]: Done starting upload Dec 29 22:00:17.043335189 (582h31m16.914418234s) [upload=510642]: Uploading segments... Dec 29 22:00:17.043587749 (582h31m16.914670824s) [upload=510642]: Got next segment. Inline: false Dec 29 22:00:17.043644478 (582h31m16.914727523s) [upload=510642]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 22:00:17.043686427 (582h31m16.914769503s) [upload=510642]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 22:00:17.294384363 (582h31m17.165467408s) [seg_pos={0 0}, upload=510642]: Begin upload segment... Dec 29 22:00:17.294445219 (582h31m17.165528264s) [seg_pos={0 0}, upload=510642]: Done begin upload segment. Dec 29 22:00:17.294484133 (582h31m17.165567178s) [noise=true , address=145.79.227.31:4797 , piece_id=NI3M5NVVVX5HUXZ2SOMQD4S5ZOSSNPCD2F3Z5X2BA2VXMGT57AYQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.294560979 (582h31m17.165644014s) [noise=true , address=77.90.31.11:4553 , piece_id=3Y5HSD73DLZ6ALJ5TFRIVTLOE4ZNZ256YSZ2W3X5LPKL5J5LX2EQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.295149941 (582h31m17.166232976s) [noise=true , address=45.91.116.5:4297 , piece_id=PLUQ7Q32BTGYFQRGVJFAJWHPZCPQTC5RX5UX27S5HJ5NNJPLDMPQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.295419574 (582h31m17.166502609s) [noise=true , address=145.79.227.2:4115 , piece_id=P2AS5B3VYO2K2AWSM5W4BXAXBHSXQQWM7JOXFPOV64NTKDFLT62Q, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.295483997 (582h31m17.166567042s) [noise=true , address=145.79.227.51:4411 , piece_id=HTZQ6SW6KSACLQF7G4CFGQA572J5JVHMLGEWQYSSGZRYX7FJ6DXA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.295762207 (582h31m17.166845252s) [noise=true , address=145.79.227.5:4221 , piece_id=LD3FENLBM2L5N7QDOAMAYJUVH54QVSJQJFP7E36UAH2ZD7R65CCQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.295928003 (582h31m17.167011048s) [noise=true , address=145.79.227.16:4909 , piece_id=CF7WHNKC4LFJQRVFBPFV62JGEBLD4KVPE6Q5JBWQVADC6ZAFFLVQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.296039615 (582h31m17.16712266s) [noise=true , address=45.91.116.13:4154 , piece_id=FE4NFODQ3TWGFXARFYIL64TKK6X7H6PPLIG6LO44LKNAZOEU7DJA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.296272429 (582h31m17.167355474s) [noise=true , address=45.91.116.10:4084 , piece_id=5BFUYVGIFOYXC6RCPU7VBUM36LSM32TVBLS3W54OWFNR4SM6GCSA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.296403588 (582h31m17.167486633s) [noise=true , address=77.90.31.19:4844 , piece_id=CDOWECXLSRPACV4HMAH57UK4EEIYNOMH6GYV6HFHV7CWSEAJ44FA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.296646560 (582h31m17.167729596s) [noise=true , address=77.90.31.14:4661 , piece_id=B7IDSCDHBUXUIPSUT3HMQSCMPGBMYASZSQO2EJBXRR2DQHCLWGYA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.296906235 (582h31m17.167989291s) [noise=true , address=145.79.227.47:4265 , piece_id=FYZA2LRIFKLKVUCUAWMNWTUH3VYR5VFAA7XKAYIMD2E2VOJTI7PA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.296989804 (582h31m17.168072849s) [noise=true , address=145.79.227.46:4601 , piece_id=YJTMZID24VBM5ZZSUUHOCZSZSSNGTFHOSKK6OI5QP3A4G6ZWVWQA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.297235221 (582h31m17.168318267s) [noise=true , address=145.79.227.3:4150 , piece_id=HS4S5UDYD2VWI2N4MH6EWEYQ5G6KCFCJBDZR7AVWPXRYALFXP4FQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.297424432 (582h31m17.168507467s) [noise=true , address=145.79.227.13:4803 , piece_id=YIEUZEDLSC5C67SCFEPCHHZSTYP6MYLYJZJTGZF3UP6JWDTGUOPQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.297606900 (582h31m17.168689945s) [noise=true , address=45.91.116.14:4190 , piece_id=FIL6AIXGE7DE2M4QBVSIO53KGY43VFPZQ2FTXSWJFPJU4VZCKUXQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:18.995364155 (582h31m18.866447201s) [upload=510642]: Next returned nil segment Dec 29 22:00:18.995383603 (582h31m18.866466648s) [upload=510642]: Waiting for error group managing segments... Dec 29 22:00:19.284009408 (582h31m19.155092453s) [noise=true , address=145.79.227.51:4411 , piece_id=HTZQ6SW6KSACLQF7G4CFGQA572J5JVHMLGEWQYSSGZRYX7FJ6DXA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.295985232 (582h31m19.167068277s) [noise=true , address=145.79.227.13:4803 , piece_id=YIEUZEDLSC5C67SCFEPCHHZSTYP6MYLYJZJTGZF3UP6JWDTGUOPQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.296135458 (582h31m19.167218503s) [noise=true , address=145.79.227.16:4909 , piece_id=CF7WHNKC4LFJQRVFBPFV62JGEBLD4KVPE6Q5JBWQVADC6ZAFFLVQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.310119916 (582h31m19.181202971s) [noise=true , address=145.79.227.46:4601 , piece_id=YJTMZID24VBM5ZZSUUHOCZSZSSNGTFHOSKK6OI5QP3A4G6ZWVWQA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.310998939 (582h31m19.182081984s) [noise=true , address=145.79.227.31:4797 , piece_id=NI3M5NVVVX5HUXZ2SOMQD4S5ZOSSNPCD2F3Z5X2BA2VXMGT57AYQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.318707138 (582h31m19.189790183s) [noise=true , address=145.79.227.47:4265 , piece_id=FYZA2LRIFKLKVUCUAWMNWTUH3VYR5VFAA7XKAYIMD2E2VOJTI7PA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.322059381 (582h31m19.193142426s) [noise=true , address=145.79.227.5:4221 , piece_id=LD3FENLBM2L5N7QDOAMAYJUVH54QVSJQJFP7E36UAH2ZD7R65CCQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.506372108 (582h31m19.377455153s) [noise=true , address=77.90.31.19:4844 , piece_id=CDOWECXLSRPACV4HMAH57UK4EEIYNOMH6GYV6HFHV7CWSEAJ44FA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.507172602 (582h31m19.378255648s) [noise=true , address=77.90.31.11:4553 , piece_id=3Y5HSD73DLZ6ALJ5TFRIVTLOE4ZNZ256YSZ2W3X5LPKL5J5LX2EQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.527548842 (582h31m19.398631887s) [noise=true , address=77.90.31.14:4661 , piece_id=B7IDSCDHBUXUIPSUT3HMQSCMPGBMYASZSQO2EJBXRR2DQHCLWGYA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.531443508 (582h31m19.402526553s) [noise=true , address=45.91.116.5:4297 , piece_id=PLUQ7Q32BTGYFQRGVJFAJWHPZCPQTC5RX5UX27S5HJ5NNJPLDMPQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.552364906 (582h31m19.423447951s) [noise=true , address=45.91.116.14:4190 , piece_id=FIL6AIXGE7DE2M4QBVSIO53KGY43VFPZQ2FTXSWJFPJU4VZCKUXQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.564056769 (582h31m19.435139814s) [noise=true , address=45.91.116.10:4084 , piece_id=5BFUYVGIFOYXC6RCPU7VBUM36LSM32TVBLS3W54OWFNR4SM6GCSA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.564087016 (582h31m19.435170051s) [seg_pos={0 0}, upload=510642]: Segment reached optimal threshold of 13 pieces. Dec 29 22:00:19.564294431 (582h31m19.435377466s) [noise=true , address=145.79.227.3:4150 , piece_id=HS4S5UDYD2VWI2N4MH6EWEYQ5G6KCFCJBDZR7AVWPXRYALFXP4FQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK): protocol: expected piece hash; context canceled; EOF Dec 29 22:00:19.564302366 (582h31m19.435385411s) [noise=true , address=145.79.227.3:4150 , piece_id=HS4S5UDYD2VWI2N4MH6EWEYQ5G6KCFCJBDZR7AVWPXRYALFXP4FQ, seg_pos={0 0}, upload=510642]: Long tail cancellation due to optimal threshold Dec 29 22:00:19.564388932 (582h31m19.435471977s) [noise=true , address=45.91.116.13:4154 , piece_id=FE4NFODQ3TWGFXARFYIL64TKK6X7H6PPLIG6LO44LKNAZOEU7DJA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Dec 29 22:00:19.564401876 (582h31m19.435484911s) [noise=true , address=45.91.116.13:4154 , piece_id=FE4NFODQ3TWGFXARFYIL64TKK6X7H6PPLIG6LO44LKNAZOEU7DJA, seg_pos={0 0}, upload=510642]: Long tail cancellation due to optimal threshold Dec 29 22:00:19.564558945 (582h31m19.435641981s) [noise=true , address=145.79.227.2:4115 , piece_id=P2AS5B3VYO2K2AWSM5W4BXAXBHSXQQWM7JOXFPOV64NTKDFLT62Q, seg_pos={0 0}, upload=510642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: write tcp 188.166.225.170:38246->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:38246->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38246->145.79.227.2:4115: use of closed network connection Dec 29 22:00:19.564570738 (582h31m19.435653783s) [noise=true , address=145.79.227.2:4115 , piece_id=P2AS5B3VYO2K2AWSM5W4BXAXBHSXQQWM7JOXFPOV64NTKDFLT62Q, seg_pos={0 0}, upload=510642]: Long tail cancellation due to optimal threshold Dec 29 22:00:19.564601426 (582h31m19.435684471s) [seg_pos={0 0}, upload=510642]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 22:00:19.564683973 (582h31m19.435767018s) [upload=510642]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 22:00:19.564690004 (582h31m19.435773049s) [upload=510642]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 22:00:19.871860472 (582h31m19.742943527s) [upload=510642]: Done uploading segments... Dec 29 22:00:19.871892413 (582h31m19.742975468s) [upload=510642]: Upload finished. err: <nil> Duration: 6.138386548s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................X............................X..] duration: 6.683188983s trace: https://jaeger.dev.storj.io/trace/7096862780f29049 FAIL (2025-12-29 22:08:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== oise=true , address=145.79.227.30:4766 , piece_id=YWDMUGNTYVXWOCMSN63VIC6HOTWHPY7VJ7QWT76P63UAJIBS2XLA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.060493636 (582h37m18.931576671s) [noise=true , address=5.231.73.5:4022 , piece_id=5WVAJH3Y6TLATXY4IYB6ZADKKJDXDS52B2ZN6FJAZ7UHFVRYRRAA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.060685291 (582h37m18.931768326s) [noise=true , address=77.90.31.11:4579 , piece_id=DEJPYHGYNAP52CZ45BKGNXJM5MLIA2OQZ4BRKN5PBADNRVE3VLEA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.060749954 (582h37m18.931832989s) [noise=true , address=145.79.227.29:4976 , piece_id=YRCRXSSQOXPOJGVX46L7WCB3TRDZ5QB755SEOTNHSCHN4ZDMCOUQ, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.061033203 (582h37m18.932116248s) [noise=true , address=193.149.164.143:4698 , piece_id=6X5KB4TXFV4ZOF5JTPNRW4Q3GWNCVCXCA4JSB3AZURBHXY2VHTRQ, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.061248855 (582h37m18.9323319s) [noise=true , address=145.79.227.49:4347 , piece_id=75B3DRRHEPDL6EK6ZUXTKASXUKWGAVUWBPSTNRYT4BJMAU5ALXRA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.061382729 (582h37m18.932465764s) [noise=true , address=5.231.73.2:4004 , piece_id=PTHHO3XEPGGCIRCUBIWHN4LVNBLMISJM5LESQGR2ERRESFK3S4RA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.061645560 (582h37m18.932728595s) [noise=true , address=5.231.73.3:4007 , piece_id=6AF3CEPLUMRO4CDW3SRWZSYHWEXP35AVSM5M3LH4KCRARSZ6NFNA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.061769094 (582h37m18.932852139s) [noise=true , address=145.79.227.7:4321 , piece_id=SPWPZ5MR5ZCQNYVYYQQEZC4OURAM5VAH3WN5UP6HMGCCDMCH6ZGA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.061970298 (582h37m18.933053333s) [noise=true , address=145.79.227.35:4507 , piece_id=ZZSVIHOAN2SWXI4NS3TRD4B64NDB4D43J63DAS74TCPEE6MPGRXA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.062237577 (582h37m18.933320612s) [noise=true , address=145.79.227.35:4489 , piece_id=BM7ZQESTHEUIRJWAPORTKDMT5TVLEXMOHULGFVEQKHT4AJV3HT5A, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.062304304 (582h37m18.933387349s) [noise=true , address=145.79.227.34:4931 , piece_id=XJHLE3B5HIF4RC4LV32SBCNMVTOTEXWBBCXBVPQFN6N5HFQ6ZS7A, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.062522640 (582h37m18.933605685s) [noise=true , address=145.79.227.48:4329 , piece_id=YWCVTYLCAGU57SE3ID3WW2JLCJGTJ7CPDRCZENAMO42BVPIAVLIQ, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.062904968 (582h37m18.933988003s) [noise=true , address=77.90.31.17:4801 , piece_id=K3QD7FPVRQHZL5R57EKKAC27RGDJQH6HDCND7FIVXSZBRH42WSIA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.063420419 (582h37m18.934503464s) [noise=true , address=145.79.227.17:4995 , piece_id=JJTVPVBAMCTDGHGWNPUVPYFVKD4SMTDAHF4VWVIW7JFUAMBIPRHQ, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.063943495 (582h37m18.93502654s) [noise=true , address=5.231.73.4:4018 , piece_id=X45H47HER27IM5FEQSEAMXSDX7UQG3AEOY65Q4WCCUOX7SVFZ5KA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:20.386418430 (582h37m20.257501465s) [upload=510734]: Next returned nil segment Dec 29 22:06:20.386428990 (582h37m20.257512025s) [upload=510734]: Waiting for error group managing segments... Dec 29 22:06:20.935543714 (582h37m20.806626749s) [noise=true , address=145.79.227.48:4329 , piece_id=YWCVTYLCAGU57SE3ID3WW2JLCJGTJ7CPDRCZENAMO42BVPIAVLIQ, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:20.978073277 (582h37m20.849156312s) [noise=true , address=145.79.227.39:4022 , piece_id=5IJCSVRVXCICRNRB3TTJAVNRF5FTUQMMY4KL7ZFF4SBKILFGHJWA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.079004013 (582h37m20.950087058s) [noise=true , address=5.231.73.4:4018 , piece_id=X45H47HER27IM5FEQSEAMXSDX7UQG3AEOY65Q4WCCUOX7SVFZ5KA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.101144552 (582h37m20.972227597s) [noise=true , address=77.90.31.11:4579 , piece_id=DEJPYHGYNAP52CZ45BKGNXJM5MLIA2OQZ4BRKN5PBADNRVE3VLEA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.138766369 (582h37m21.009849405s) [noise=true , address=77.90.31.17:4801 , piece_id=K3QD7FPVRQHZL5R57EKKAC27RGDJQH6HDCND7FIVXSZBRH42WSIA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.231047035 (582h37m21.10213007s) [noise=true , address=77.90.31.19:4857 , piece_id=DMJ4O3TWZYWKWVLJFSQBULHDFWZCNLVDBIGYJEOBAVVCU5UGG27Q, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.272062706 (582h37m21.143145741s) [noise=true , address=5.231.73.3:4007 , piece_id=6AF3CEPLUMRO4CDW3SRWZSYHWEXP35AVSM5M3LH4KCRARSZ6NFNA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.303520666 (582h37m21.174603711s) [noise=true , address=145.79.227.29:4976 , piece_id=YRCRXSSQOXPOJGVX46L7WCB3TRDZ5QB755SEOTNHSCHN4ZDMCOUQ, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.340640177 (582h37m21.211723212s) [noise=true , address=5.231.73.5:4022 , piece_id=5WVAJH3Y6TLATXY4IYB6ZADKKJDXDS52B2ZN6FJAZ7UHFVRYRRAA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.481118228 (582h37m21.352201263s) [noise=true , address=5.231.73.2:4004 , piece_id=PTHHO3XEPGGCIRCUBIWHN4LVNBLMISJM5LESQGR2ERRESFK3S4RA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.896953769 (582h37m21.768036804s) [noise=true , address=145.79.227.49:4347 , piece_id=75B3DRRHEPDL6EK6ZUXTKASXUKWGAVUWBPSTNRYT4BJMAU5ALXRA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.911314353 (582h37m21.782397399s) [noise=true , address=145.79.227.34:4931 , piece_id=XJHLE3B5HIF4RC4LV32SBCNMVTOTEXWBBCXBVPQFN6N5HFQ6ZS7A, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.915845371 (582h37m21.786928406s) [noise=true , address=145.79.227.30:4766 , piece_id=YWDMUGNTYVXWOCMSN63VIC6HOTWHPY7VJ7QWT76P63UAJIBS2XLA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.928785972 (582h37m21.799869017s) [noise=true , address=145.79.227.35:4489 , piece_id=BM7ZQESTHEUIRJWAPORTKDMT5TVLEXMOHULGFVEQKHT4AJV3HT5A, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.928810859 (582h37m21.799893904s) [seg_pos={0 0}, upload=510734]: Segment reached optimal threshold of 14 pieces. Dec 29 22:06:21.929016471 (582h37m21.800099516s) [noise=true , address=145.79.227.7:4321 , piece_id=SPWPZ5MR5ZCQNYVYYQQEZC4OURAM5VAH3WN5UP6HMGCCDMCH6ZGA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12edsHqD9iRBDgMU8vYhSM1EWG8Sn57GZ8TPiTytri3rp7W9Eae): protocol: expected piece hash; context canceled; EOF Dec 29 22:06:21.929028644 (582h37m21.800111709s) [noise=true , address=145.79.227.7:4321 , piece_id=SPWPZ5MR5ZCQNYVYYQQEZC4OURAM5VAH3WN5UP6HMGCCDMCH6ZGA, seg_pos={0 0}, upload=510734]: Long tail cancellation due to optimal threshold Dec 29 22:06:21.929093137 (582h37m21.800176192s) [noise=true , address=145.79.227.35:4507 , piece_id=ZZSVIHOAN2SWXI4NS3TRD4B64NDB4D43J63DAS74TCPEE6MPGRXA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14jurWWPpmxwiBS3XmZysEUwugsceYJqVXFj7BbYpx8UFcTR8R): protocol: expected piece hash; context canceled; EOF Dec 29 22:06:21.929106793 (582h37m21.800189838s) [noise=true , address=145.79.227.35:4507 , piece_id=ZZSVIHOAN2SWXI4NS3TRD4B64NDB4D43J63DAS74TCPEE6MPGRXA, seg_pos={0 0}, upload=510734]: Long tail cancellation due to optimal threshold Dec 29 22:06:21.929206612 (582h37m21.800289657s) [noise=true , address=145.79.227.17:4995 , piece_id=JJTVPVBAMCTDGHGWNPUVPYFVKD4SMTDAHF4VWVIW7JFUAMBIPRHQ, seg_pos={0 0}, upload=510734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tvoTD1LoSj3q2rLiDckfT56dXNKKwdVu1ZAsBZExNFVTom9tH): protocol: write tcp 188.166.225.170:47696->145.79.227.17:4995: use of closed network connection; write tcp 188.166.225.170:47696->145.79.227.17:4995: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47696->145.79.227.17:4995: use of closed network connection Dec 29 22:06:21.929215911 (582h37m21.800298976s) [noise=true , address=145.79.227.17:4995 , piece_id=JJTVPVBAMCTDGHGWNPUVPYFVKD4SMTDAHF4VWVIW7JFUAMBIPRHQ, seg_pos={0 0}, upload=510734]: Long tail cancellation due to optimal threshold Dec 29 22:06:21.929387247 (582h37m21.800470292s) [noise=true , address=193.149.164.143:4698 , piece_id=6X5KB4TXFV4ZOF5JTPNRW4Q3GWNCVCXCA4JSB3AZURBHXY2VHTRQ, seg_pos={0 0}, upload=510734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F4fJAGJM2ZPRt7vJH2cZmEbYeujmxcfi7aPZ2g9VPTNYBYRAi): protocol: expected piece hash; context canceled; EOF Dec 29 22:06:21.929396174 (582h37m21.800479219s) [noise=true , address=193.149.164.143:4698 , piece_id=6X5KB4TXFV4ZOF5JTPNRW4Q3GWNCVCXCA4JSB3AZURBHXY2VHTRQ, seg_pos={0 0}, upload=510734]: Long tail cancellation due to optimal threshold Dec 29 22:06:21.929491536 (582h37m21.800574571s) [seg_pos={0 0}, upload=510734]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 22:06:21.929573792 (582h37m21.800656837s) [upload=510734]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 22:06:21.929580555 (582h37m21.8006636s) [upload=510734]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 22:06:22.233045366 (582h37m22.104128401s) [upload=510734]: Done uploading segments... Dec 29 22:06:22.233064963 (582h37m22.104147998s) [upload=510734]: Upload finished. err: <nil> Duration: 6.683237083s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [......X..X..X........X............................X.........] duration: 6.831820311s trace: https://jaeger.dev.storj.io/trace/5209b3d035311fd FAIL (2025-12-29 22:07:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :00:17.043136250 (582h31m16.914219305s) [upload=510642]: Starting upload Dec 29 22:00:17.043227904 (582h31m16.91431096s) [upload=510642]: Done starting upload Dec 29 22:00:17.043335189 (582h31m16.914418234s) [upload=510642]: Uploading segments... Dec 29 22:00:17.043587749 (582h31m16.914670824s) [upload=510642]: Got next segment. Inline: false Dec 29 22:00:17.043644478 (582h31m16.914727523s) [upload=510642]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 22:00:17.043686427 (582h31m16.914769503s) [upload=510642]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 22:00:17.294384363 (582h31m17.165467408s) [seg_pos={0 0}, upload=510642]: Begin upload segment... Dec 29 22:00:17.294445219 (582h31m17.165528264s) [seg_pos={0 0}, upload=510642]: Done begin upload segment. Dec 29 22:00:17.294484133 (582h31m17.165567178s) [noise=true , address=145.79.227.31:4797 , piece_id=NI3M5NVVVX5HUXZ2SOMQD4S5ZOSSNPCD2F3Z5X2BA2VXMGT57AYQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.294560979 (582h31m17.165644014s) [noise=true , address=77.90.31.11:4553 , piece_id=3Y5HSD73DLZ6ALJ5TFRIVTLOE4ZNZ256YSZ2W3X5LPKL5J5LX2EQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.295149941 (582h31m17.166232976s) [noise=true , address=45.91.116.5:4297 , piece_id=PLUQ7Q32BTGYFQRGVJFAJWHPZCPQTC5RX5UX27S5HJ5NNJPLDMPQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.295419574 (582h31m17.166502609s) [noise=true , address=145.79.227.2:4115 , piece_id=P2AS5B3VYO2K2AWSM5W4BXAXBHSXQQWM7JOXFPOV64NTKDFLT62Q, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.295483997 (582h31m17.166567042s) [noise=true , address=145.79.227.51:4411 , piece_id=HTZQ6SW6KSACLQF7G4CFGQA572J5JVHMLGEWQYSSGZRYX7FJ6DXA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.295762207 (582h31m17.166845252s) [noise=true , address=145.79.227.5:4221 , piece_id=LD3FENLBM2L5N7QDOAMAYJUVH54QVSJQJFP7E36UAH2ZD7R65CCQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.295928003 (582h31m17.167011048s) [noise=true , address=145.79.227.16:4909 , piece_id=CF7WHNKC4LFJQRVFBPFV62JGEBLD4KVPE6Q5JBWQVADC6ZAFFLVQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.296039615 (582h31m17.16712266s) [noise=true , address=45.91.116.13:4154 , piece_id=FE4NFODQ3TWGFXARFYIL64TKK6X7H6PPLIG6LO44LKNAZOEU7DJA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.296272429 (582h31m17.167355474s) [noise=true , address=45.91.116.10:4084 , piece_id=5BFUYVGIFOYXC6RCPU7VBUM36LSM32TVBLS3W54OWFNR4SM6GCSA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.296403588 (582h31m17.167486633s) [noise=true , address=77.90.31.19:4844 , piece_id=CDOWECXLSRPACV4HMAH57UK4EEIYNOMH6GYV6HFHV7CWSEAJ44FA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.296646560 (582h31m17.167729596s) [noise=true , address=77.90.31.14:4661 , piece_id=B7IDSCDHBUXUIPSUT3HMQSCMPGBMYASZSQO2EJBXRR2DQHCLWGYA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.296906235 (582h31m17.167989291s) [noise=true , address=145.79.227.47:4265 , piece_id=FYZA2LRIFKLKVUCUAWMNWTUH3VYR5VFAA7XKAYIMD2E2VOJTI7PA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.296989804 (582h31m17.168072849s) [noise=true , address=145.79.227.46:4601 , piece_id=YJTMZID24VBM5ZZSUUHOCZSZSSNGTFHOSKK6OI5QP3A4G6ZWVWQA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.297235221 (582h31m17.168318267s) [noise=true , address=145.79.227.3:4150 , piece_id=HS4S5UDYD2VWI2N4MH6EWEYQ5G6KCFCJBDZR7AVWPXRYALFXP4FQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.297424432 (582h31m17.168507467s) [noise=true , address=145.79.227.13:4803 , piece_id=YIEUZEDLSC5C67SCFEPCHHZSTYP6MYLYJZJTGZF3UP6JWDTGUOPQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.297606900 (582h31m17.168689945s) [noise=true , address=45.91.116.14:4190 , piece_id=FIL6AIXGE7DE2M4QBVSIO53KGY43VFPZQ2FTXSWJFPJU4VZCKUXQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:18.995364155 (582h31m18.866447201s) [upload=510642]: Next returned nil segment Dec 29 22:00:18.995383603 (582h31m18.866466648s) [upload=510642]: Waiting for error group managing segments... Dec 29 22:00:19.284009408 (582h31m19.155092453s) [noise=true , address=145.79.227.51:4411 , piece_id=HTZQ6SW6KSACLQF7G4CFGQA572J5JVHMLGEWQYSSGZRYX7FJ6DXA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.295985232 (582h31m19.167068277s) [noise=true , address=145.79.227.13:4803 , piece_id=YIEUZEDLSC5C67SCFEPCHHZSTYP6MYLYJZJTGZF3UP6JWDTGUOPQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.296135458 (582h31m19.167218503s) [noise=true , address=145.79.227.16:4909 , piece_id=CF7WHNKC4LFJQRVFBPFV62JGEBLD4KVPE6Q5JBWQVADC6ZAFFLVQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.310119916 (582h31m19.181202971s) [noise=true , address=145.79.227.46:4601 , piece_id=YJTMZID24VBM5ZZSUUHOCZSZSSNGTFHOSKK6OI5QP3A4G6ZWVWQA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.310998939 (582h31m19.182081984s) [noise=true , address=145.79.227.31:4797 , piece_id=NI3M5NVVVX5HUXZ2SOMQD4S5ZOSSNPCD2F3Z5X2BA2VXMGT57AYQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.318707138 (582h31m19.189790183s) [noise=true , address=145.79.227.47:4265 , piece_id=FYZA2LRIFKLKVUCUAWMNWTUH3VYR5VFAA7XKAYIMD2E2VOJTI7PA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.322059381 (582h31m19.193142426s) [noise=true , address=145.79.227.5:4221 , piece_id=LD3FENLBM2L5N7QDOAMAYJUVH54QVSJQJFP7E36UAH2ZD7R65CCQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.506372108 (582h31m19.377455153s) [noise=true , address=77.90.31.19:4844 , piece_id=CDOWECXLSRPACV4HMAH57UK4EEIYNOMH6GYV6HFHV7CWSEAJ44FA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.507172602 (582h31m19.378255648s) [noise=true , address=77.90.31.11:4553 , piece_id=3Y5HSD73DLZ6ALJ5TFRIVTLOE4ZNZ256YSZ2W3X5LPKL5J5LX2EQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.527548842 (582h31m19.398631887s) [noise=true , address=77.90.31.14:4661 , piece_id=B7IDSCDHBUXUIPSUT3HMQSCMPGBMYASZSQO2EJBXRR2DQHCLWGYA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.531443508 (582h31m19.402526553s) [noise=true , address=45.91.116.5:4297 , piece_id=PLUQ7Q32BTGYFQRGVJFAJWHPZCPQTC5RX5UX27S5HJ5NNJPLDMPQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.552364906 (582h31m19.423447951s) [noise=true , address=45.91.116.14:4190 , piece_id=FIL6AIXGE7DE2M4QBVSIO53KGY43VFPZQ2FTXSWJFPJU4VZCKUXQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.564056769 (582h31m19.435139814s) [noise=true , address=45.91.116.10:4084 , piece_id=5BFUYVGIFOYXC6RCPU7VBUM36LSM32TVBLS3W54OWFNR4SM6GCSA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.564087016 (582h31m19.435170051s) [seg_pos={0 0}, upload=510642]: Segment reached optimal threshold of 13 pieces. Dec 29 22:00:19.564294431 (582h31m19.435377466s) [noise=true , address=145.79.227.3:4150 , piece_id=HS4S5UDYD2VWI2N4MH6EWEYQ5G6KCFCJBDZR7AVWPXRYALFXP4FQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK): protocol: expected piece hash; context canceled; EOF Dec 29 22:00:19.564302366 (582h31m19.435385411s) [noise=true , address=145.79.227.3:4150 , piece_id=HS4S5UDYD2VWI2N4MH6EWEYQ5G6KCFCJBDZR7AVWPXRYALFXP4FQ, seg_pos={0 0}, upload=510642]: Long tail cancellation due to optimal threshold Dec 29 22:00:19.564388932 (582h31m19.435471977s) [noise=true , address=45.91.116.13:4154 , piece_id=FE4NFODQ3TWGFXARFYIL64TKK6X7H6PPLIG6LO44LKNAZOEU7DJA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Dec 29 22:00:19.564401876 (582h31m19.435484911s) [noise=true , address=45.91.116.13:4154 , piece_id=FE4NFODQ3TWGFXARFYIL64TKK6X7H6PPLIG6LO44LKNAZOEU7DJA, seg_pos={0 0}, upload=510642]: Long tail cancellation due to optimal threshold Dec 29 22:00:19.564558945 (582h31m19.435641981s) [noise=true , address=145.79.227.2:4115 , piece_id=P2AS5B3VYO2K2AWSM5W4BXAXBHSXQQWM7JOXFPOV64NTKDFLT62Q, seg_pos={0 0}, upload=510642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: write tcp 188.166.225.170:38246->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:38246->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38246->145.79.227.2:4115: use of closed network connection Dec 29 22:00:19.564570738 (582h31m19.435653783s) [noise=true , address=145.79.227.2:4115 , piece_id=P2AS5B3VYO2K2AWSM5W4BXAXBHSXQQWM7JOXFPOV64NTKDFLT62Q, seg_pos={0 0}, upload=510642]: Long tail cancellation due to optimal threshold Dec 29 22:00:19.564601426 (582h31m19.435684471s) [seg_pos={0 0}, upload=510642]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 22:00:19.564683973 (582h31m19.435767018s) [upload=510642]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 22:00:19.564690004 (582h31m19.435773049s) [upload=510642]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 22:00:19.871860472 (582h31m19.742943527s) [upload=510642]: Done uploading segments... Dec 29 22:00:19.871892413 (582h31m19.742975468s) [upload=510642]: Upload finished. err: <nil> Duration: 6.831846822s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.......X..X..X........X............................X........] duration: 26.446162886s trace: https://jaeger.dev.storj.io/trace/5209b3d035311fd FAIL (2025-12-29 22:07:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :00:17.043136250 (582h31m16.914219305s) [upload=510642]: Starting upload Dec 29 22:00:17.043227904 (582h31m16.91431096s) [upload=510642]: Done starting upload Dec 29 22:00:17.043335189 (582h31m16.914418234s) [upload=510642]: Uploading segments... Dec 29 22:00:17.043587749 (582h31m16.914670824s) [upload=510642]: Got next segment. Inline: false Dec 29 22:00:17.043644478 (582h31m16.914727523s) [upload=510642]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 22:00:17.043686427 (582h31m16.914769503s) [upload=510642]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 22:00:17.294384363 (582h31m17.165467408s) [seg_pos={0 0}, upload=510642]: Begin upload segment... Dec 29 22:00:17.294445219 (582h31m17.165528264s) [seg_pos={0 0}, upload=510642]: Done begin upload segment. Dec 29 22:00:17.294484133 (582h31m17.165567178s) [noise=true , address=145.79.227.31:4797 , piece_id=NI3M5NVVVX5HUXZ2SOMQD4S5ZOSSNPCD2F3Z5X2BA2VXMGT57AYQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.294560979 (582h31m17.165644014s) [noise=true , address=77.90.31.11:4553 , piece_id=3Y5HSD73DLZ6ALJ5TFRIVTLOE4ZNZ256YSZ2W3X5LPKL5J5LX2EQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.295149941 (582h31m17.166232976s) [noise=true , address=45.91.116.5:4297 , piece_id=PLUQ7Q32BTGYFQRGVJFAJWHPZCPQTC5RX5UX27S5HJ5NNJPLDMPQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.295419574 (582h31m17.166502609s) [noise=true , address=145.79.227.2:4115 , piece_id=P2AS5B3VYO2K2AWSM5W4BXAXBHSXQQWM7JOXFPOV64NTKDFLT62Q, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.295483997 (582h31m17.166567042s) [noise=true , address=145.79.227.51:4411 , piece_id=HTZQ6SW6KSACLQF7G4CFGQA572J5JVHMLGEWQYSSGZRYX7FJ6DXA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.295762207 (582h31m17.166845252s) [noise=true , address=145.79.227.5:4221 , piece_id=LD3FENLBM2L5N7QDOAMAYJUVH54QVSJQJFP7E36UAH2ZD7R65CCQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.295928003 (582h31m17.167011048s) [noise=true , address=145.79.227.16:4909 , piece_id=CF7WHNKC4LFJQRVFBPFV62JGEBLD4KVPE6Q5JBWQVADC6ZAFFLVQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.296039615 (582h31m17.16712266s) [noise=true , address=45.91.116.13:4154 , piece_id=FE4NFODQ3TWGFXARFYIL64TKK6X7H6PPLIG6LO44LKNAZOEU7DJA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.296272429 (582h31m17.167355474s) [noise=true , address=45.91.116.10:4084 , piece_id=5BFUYVGIFOYXC6RCPU7VBUM36LSM32TVBLS3W54OWFNR4SM6GCSA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.296403588 (582h31m17.167486633s) [noise=true , address=77.90.31.19:4844 , piece_id=CDOWECXLSRPACV4HMAH57UK4EEIYNOMH6GYV6HFHV7CWSEAJ44FA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.296646560 (582h31m17.167729596s) [noise=true , address=77.90.31.14:4661 , piece_id=B7IDSCDHBUXUIPSUT3HMQSCMPGBMYASZSQO2EJBXRR2DQHCLWGYA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.296906235 (582h31m17.167989291s) [noise=true , address=145.79.227.47:4265 , piece_id=FYZA2LRIFKLKVUCUAWMNWTUH3VYR5VFAA7XKAYIMD2E2VOJTI7PA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.296989804 (582h31m17.168072849s) [noise=true , address=145.79.227.46:4601 , piece_id=YJTMZID24VBM5ZZSUUHOCZSZSSNGTFHOSKK6OI5QP3A4G6ZWVWQA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.297235221 (582h31m17.168318267s) [noise=true , address=145.79.227.3:4150 , piece_id=HS4S5UDYD2VWI2N4MH6EWEYQ5G6KCFCJBDZR7AVWPXRYALFXP4FQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.297424432 (582h31m17.168507467s) [noise=true , address=145.79.227.13:4803 , piece_id=YIEUZEDLSC5C67SCFEPCHHZSTYP6MYLYJZJTGZF3UP6JWDTGUOPQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.297606900 (582h31m17.168689945s) [noise=true , address=45.91.116.14:4190 , piece_id=FIL6AIXGE7DE2M4QBVSIO53KGY43VFPZQ2FTXSWJFPJU4VZCKUXQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:18.995364155 (582h31m18.866447201s) [upload=510642]: Next returned nil segment Dec 29 22:00:18.995383603 (582h31m18.866466648s) [upload=510642]: Waiting for error group managing segments... Dec 29 22:00:19.284009408 (582h31m19.155092453s) [noise=true , address=145.79.227.51:4411 , piece_id=HTZQ6SW6KSACLQF7G4CFGQA572J5JVHMLGEWQYSSGZRYX7FJ6DXA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.295985232 (582h31m19.167068277s) [noise=true , address=145.79.227.13:4803 , piece_id=YIEUZEDLSC5C67SCFEPCHHZSTYP6MYLYJZJTGZF3UP6JWDTGUOPQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.296135458 (582h31m19.167218503s) [noise=true , address=145.79.227.16:4909 , piece_id=CF7WHNKC4LFJQRVFBPFV62JGEBLD4KVPE6Q5JBWQVADC6ZAFFLVQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.310119916 (582h31m19.181202971s) [noise=true , address=145.79.227.46:4601 , piece_id=YJTMZID24VBM5ZZSUUHOCZSZSSNGTFHOSKK6OI5QP3A4G6ZWVWQA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.310998939 (582h31m19.182081984s) [noise=true , address=145.79.227.31:4797 , piece_id=NI3M5NVVVX5HUXZ2SOMQD4S5ZOSSNPCD2F3Z5X2BA2VXMGT57AYQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.318707138 (582h31m19.189790183s) [noise=true , address=145.79.227.47:4265 , piece_id=FYZA2LRIFKLKVUCUAWMNWTUH3VYR5VFAA7XKAYIMD2E2VOJTI7PA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.322059381 (582h31m19.193142426s) [noise=true , address=145.79.227.5:4221 , piece_id=LD3FENLBM2L5N7QDOAMAYJUVH54QVSJQJFP7E36UAH2ZD7R65CCQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.506372108 (582h31m19.377455153s) [noise=true , address=77.90.31.19:4844 , piece_id=CDOWECXLSRPACV4HMAH57UK4EEIYNOMH6GYV6HFHV7CWSEAJ44FA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.507172602 (582h31m19.378255648s) [noise=true , address=77.90.31.11:4553 , piece_id=3Y5HSD73DLZ6ALJ5TFRIVTLOE4ZNZ256YSZ2W3X5LPKL5J5LX2EQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.527548842 (582h31m19.398631887s) [noise=true , address=77.90.31.14:4661 , piece_id=B7IDSCDHBUXUIPSUT3HMQSCMPGBMYASZSQO2EJBXRR2DQHCLWGYA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.531443508 (582h31m19.402526553s) [noise=true , address=45.91.116.5:4297 , piece_id=PLUQ7Q32BTGYFQRGVJFAJWHPZCPQTC5RX5UX27S5HJ5NNJPLDMPQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.552364906 (582h31m19.423447951s) [noise=true , address=45.91.116.14:4190 , piece_id=FIL6AIXGE7DE2M4QBVSIO53KGY43VFPZQ2FTXSWJFPJU4VZCKUXQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.564056769 (582h31m19.435139814s) [noise=true , address=45.91.116.10:4084 , piece_id=5BFUYVGIFOYXC6RCPU7VBUM36LSM32TVBLS3W54OWFNR4SM6GCSA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.564087016 (582h31m19.435170051s) [seg_pos={0 0}, upload=510642]: Segment reached optimal threshold of 13 pieces. Dec 29 22:00:19.564294431 (582h31m19.435377466s) [noise=true , address=145.79.227.3:4150 , piece_id=HS4S5UDYD2VWI2N4MH6EWEYQ5G6KCFCJBDZR7AVWPXRYALFXP4FQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK): protocol: expected piece hash; context canceled; EOF Dec 29 22:00:19.564302366 (582h31m19.435385411s) [noise=true , address=145.79.227.3:4150 , piece_id=HS4S5UDYD2VWI2N4MH6EWEYQ5G6KCFCJBDZR7AVWPXRYALFXP4FQ, seg_pos={0 0}, upload=510642]: Long tail cancellation due to optimal threshold Dec 29 22:00:19.564388932 (582h31m19.435471977s) [noise=true , address=45.91.116.13:4154 , piece_id=FE4NFODQ3TWGFXARFYIL64TKK6X7H6PPLIG6LO44LKNAZOEU7DJA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Dec 29 22:00:19.564401876 (582h31m19.435484911s) [noise=true , address=45.91.116.13:4154 , piece_id=FE4NFODQ3TWGFXARFYIL64TKK6X7H6PPLIG6LO44LKNAZOEU7DJA, seg_pos={0 0}, upload=510642]: Long tail cancellation due to optimal threshold Dec 29 22:00:19.564558945 (582h31m19.435641981s) [noise=true , address=145.79.227.2:4115 , piece_id=P2AS5B3VYO2K2AWSM5W4BXAXBHSXQQWM7JOXFPOV64NTKDFLT62Q, seg_pos={0 0}, upload=510642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: write tcp 188.166.225.170:38246->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:38246->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38246->145.79.227.2:4115: use of closed network connection Dec 29 22:00:19.564570738 (582h31m19.435653783s) [noise=true , address=145.79.227.2:4115 , piece_id=P2AS5B3VYO2K2AWSM5W4BXAXBHSXQQWM7JOXFPOV64NTKDFLT62Q, seg_pos={0 0}, upload=510642]: Long tail cancellation due to optimal threshold Dec 29 22:00:19.564601426 (582h31m19.435684471s) [seg_pos={0 0}, upload=510642]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 22:00:19.564683973 (582h31m19.435767018s) [upload=510642]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 22:00:19.564690004 (582h31m19.435773049s) [upload=510642]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 22:00:19.871860472 (582h31m19.742943527s) [upload=510642]: Done uploading segments... Dec 29 22:00:19.871892413 (582h31m19.742975468s) [upload=510642]: Upload finished. err: <nil> Duration: 26.446187824s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................X............................X.] duration: 15.258183085s trace: https://jaeger.dev.storj.io/trace/7096862780f29049 FAIL (2025-12-29 22:07:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== oise=true , address=145.79.227.30:4766 , piece_id=YWDMUGNTYVXWOCMSN63VIC6HOTWHPY7VJ7QWT76P63UAJIBS2XLA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.060493636 (582h37m18.931576671s) [noise=true , address=5.231.73.5:4022 , piece_id=5WVAJH3Y6TLATXY4IYB6ZADKKJDXDS52B2ZN6FJAZ7UHFVRYRRAA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.060685291 (582h37m18.931768326s) [noise=true , address=77.90.31.11:4579 , piece_id=DEJPYHGYNAP52CZ45BKGNXJM5MLIA2OQZ4BRKN5PBADNRVE3VLEA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.060749954 (582h37m18.931832989s) [noise=true , address=145.79.227.29:4976 , piece_id=YRCRXSSQOXPOJGVX46L7WCB3TRDZ5QB755SEOTNHSCHN4ZDMCOUQ, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.061033203 (582h37m18.932116248s) [noise=true , address=193.149.164.143:4698 , piece_id=6X5KB4TXFV4ZOF5JTPNRW4Q3GWNCVCXCA4JSB3AZURBHXY2VHTRQ, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.061248855 (582h37m18.9323319s) [noise=true , address=145.79.227.49:4347 , piece_id=75B3DRRHEPDL6EK6ZUXTKASXUKWGAVUWBPSTNRYT4BJMAU5ALXRA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.061382729 (582h37m18.932465764s) [noise=true , address=5.231.73.2:4004 , piece_id=PTHHO3XEPGGCIRCUBIWHN4LVNBLMISJM5LESQGR2ERRESFK3S4RA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.061645560 (582h37m18.932728595s) [noise=true , address=5.231.73.3:4007 , piece_id=6AF3CEPLUMRO4CDW3SRWZSYHWEXP35AVSM5M3LH4KCRARSZ6NFNA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.061769094 (582h37m18.932852139s) [noise=true , address=145.79.227.7:4321 , piece_id=SPWPZ5MR5ZCQNYVYYQQEZC4OURAM5VAH3WN5UP6HMGCCDMCH6ZGA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.061970298 (582h37m18.933053333s) [noise=true , address=145.79.227.35:4507 , piece_id=ZZSVIHOAN2SWXI4NS3TRD4B64NDB4D43J63DAS74TCPEE6MPGRXA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.062237577 (582h37m18.933320612s) [noise=true , address=145.79.227.35:4489 , piece_id=BM7ZQESTHEUIRJWAPORTKDMT5TVLEXMOHULGFVEQKHT4AJV3HT5A, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.062304304 (582h37m18.933387349s) [noise=true , address=145.79.227.34:4931 , piece_id=XJHLE3B5HIF4RC4LV32SBCNMVTOTEXWBBCXBVPQFN6N5HFQ6ZS7A, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.062522640 (582h37m18.933605685s) [noise=true , address=145.79.227.48:4329 , piece_id=YWCVTYLCAGU57SE3ID3WW2JLCJGTJ7CPDRCZENAMO42BVPIAVLIQ, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.062904968 (582h37m18.933988003s) [noise=true , address=77.90.31.17:4801 , piece_id=K3QD7FPVRQHZL5R57EKKAC27RGDJQH6HDCND7FIVXSZBRH42WSIA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.063420419 (582h37m18.934503464s) [noise=true , address=145.79.227.17:4995 , piece_id=JJTVPVBAMCTDGHGWNPUVPYFVKD4SMTDAHF4VWVIW7JFUAMBIPRHQ, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.063943495 (582h37m18.93502654s) [noise=true , address=5.231.73.4:4018 , piece_id=X45H47HER27IM5FEQSEAMXSDX7UQG3AEOY65Q4WCCUOX7SVFZ5KA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:20.386418430 (582h37m20.257501465s) [upload=510734]: Next returned nil segment Dec 29 22:06:20.386428990 (582h37m20.257512025s) [upload=510734]: Waiting for error group managing segments... Dec 29 22:06:20.935543714 (582h37m20.806626749s) [noise=true , address=145.79.227.48:4329 , piece_id=YWCVTYLCAGU57SE3ID3WW2JLCJGTJ7CPDRCZENAMO42BVPIAVLIQ, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:20.978073277 (582h37m20.849156312s) [noise=true , address=145.79.227.39:4022 , piece_id=5IJCSVRVXCICRNRB3TTJAVNRF5FTUQMMY4KL7ZFF4SBKILFGHJWA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.079004013 (582h37m20.950087058s) [noise=true , address=5.231.73.4:4018 , piece_id=X45H47HER27IM5FEQSEAMXSDX7UQG3AEOY65Q4WCCUOX7SVFZ5KA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.101144552 (582h37m20.972227597s) [noise=true , address=77.90.31.11:4579 , piece_id=DEJPYHGYNAP52CZ45BKGNXJM5MLIA2OQZ4BRKN5PBADNRVE3VLEA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.138766369 (582h37m21.009849405s) [noise=true , address=77.90.31.17:4801 , piece_id=K3QD7FPVRQHZL5R57EKKAC27RGDJQH6HDCND7FIVXSZBRH42WSIA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.231047035 (582h37m21.10213007s) [noise=true , address=77.90.31.19:4857 , piece_id=DMJ4O3TWZYWKWVLJFSQBULHDFWZCNLVDBIGYJEOBAVVCU5UGG27Q, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.272062706 (582h37m21.143145741s) [noise=true , address=5.231.73.3:4007 , piece_id=6AF3CEPLUMRO4CDW3SRWZSYHWEXP35AVSM5M3LH4KCRARSZ6NFNA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.303520666 (582h37m21.174603711s) [noise=true , address=145.79.227.29:4976 , piece_id=YRCRXSSQOXPOJGVX46L7WCB3TRDZ5QB755SEOTNHSCHN4ZDMCOUQ, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.340640177 (582h37m21.211723212s) [noise=true , address=5.231.73.5:4022 , piece_id=5WVAJH3Y6TLATXY4IYB6ZADKKJDXDS52B2ZN6FJAZ7UHFVRYRRAA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.481118228 (582h37m21.352201263s) [noise=true , address=5.231.73.2:4004 , piece_id=PTHHO3XEPGGCIRCUBIWHN4LVNBLMISJM5LESQGR2ERRESFK3S4RA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.896953769 (582h37m21.768036804s) [noise=true , address=145.79.227.49:4347 , piece_id=75B3DRRHEPDL6EK6ZUXTKASXUKWGAVUWBPSTNRYT4BJMAU5ALXRA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.911314353 (582h37m21.782397399s) [noise=true , address=145.79.227.34:4931 , piece_id=XJHLE3B5HIF4RC4LV32SBCNMVTOTEXWBBCXBVPQFN6N5HFQ6ZS7A, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.915845371 (582h37m21.786928406s) [noise=true , address=145.79.227.30:4766 , piece_id=YWDMUGNTYVXWOCMSN63VIC6HOTWHPY7VJ7QWT76P63UAJIBS2XLA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.928785972 (582h37m21.799869017s) [noise=true , address=145.79.227.35:4489 , piece_id=BM7ZQESTHEUIRJWAPORTKDMT5TVLEXMOHULGFVEQKHT4AJV3HT5A, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.928810859 (582h37m21.799893904s) [seg_pos={0 0}, upload=510734]: Segment reached optimal threshold of 14 pieces. Dec 29 22:06:21.929016471 (582h37m21.800099516s) [noise=true , address=145.79.227.7:4321 , piece_id=SPWPZ5MR5ZCQNYVYYQQEZC4OURAM5VAH3WN5UP6HMGCCDMCH6ZGA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12edsHqD9iRBDgMU8vYhSM1EWG8Sn57GZ8TPiTytri3rp7W9Eae): protocol: expected piece hash; context canceled; EOF Dec 29 22:06:21.929028644 (582h37m21.800111709s) [noise=true , address=145.79.227.7:4321 , piece_id=SPWPZ5MR5ZCQNYVYYQQEZC4OURAM5VAH3WN5UP6HMGCCDMCH6ZGA, seg_pos={0 0}, upload=510734]: Long tail cancellation due to optimal threshold Dec 29 22:06:21.929093137 (582h37m21.800176192s) [noise=true , address=145.79.227.35:4507 , piece_id=ZZSVIHOAN2SWXI4NS3TRD4B64NDB4D43J63DAS74TCPEE6MPGRXA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14jurWWPpmxwiBS3XmZysEUwugsceYJqVXFj7BbYpx8UFcTR8R): protocol: expected piece hash; context canceled; EOF Dec 29 22:06:21.929106793 (582h37m21.800189838s) [noise=true , address=145.79.227.35:4507 , piece_id=ZZSVIHOAN2SWXI4NS3TRD4B64NDB4D43J63DAS74TCPEE6MPGRXA, seg_pos={0 0}, upload=510734]: Long tail cancellation due to optimal threshold Dec 29 22:06:21.929206612 (582h37m21.800289657s) [noise=true , address=145.79.227.17:4995 , piece_id=JJTVPVBAMCTDGHGWNPUVPYFVKD4SMTDAHF4VWVIW7JFUAMBIPRHQ, seg_pos={0 0}, upload=510734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tvoTD1LoSj3q2rLiDckfT56dXNKKwdVu1ZAsBZExNFVTom9tH): protocol: write tcp 188.166.225.170:47696->145.79.227.17:4995: use of closed network connection; write tcp 188.166.225.170:47696->145.79.227.17:4995: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47696->145.79.227.17:4995: use of closed network connection Dec 29 22:06:21.929215911 (582h37m21.800298976s) [noise=true , address=145.79.227.17:4995 , piece_id=JJTVPVBAMCTDGHGWNPUVPYFVKD4SMTDAHF4VWVIW7JFUAMBIPRHQ, seg_pos={0 0}, upload=510734]: Long tail cancellation due to optimal threshold Dec 29 22:06:21.929387247 (582h37m21.800470292s) [noise=true , address=193.149.164.143:4698 , piece_id=6X5KB4TXFV4ZOF5JTPNRW4Q3GWNCVCXCA4JSB3AZURBHXY2VHTRQ, seg_pos={0 0}, upload=510734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F4fJAGJM2ZPRt7vJH2cZmEbYeujmxcfi7aPZ2g9VPTNYBYRAi): protocol: expected piece hash; context canceled; EOF Dec 29 22:06:21.929396174 (582h37m21.800479219s) [noise=true , address=193.149.164.143:4698 , piece_id=6X5KB4TXFV4ZOF5JTPNRW4Q3GWNCVCXCA4JSB3AZURBHXY2VHTRQ, seg_pos={0 0}, upload=510734]: Long tail cancellation due to optimal threshold Dec 29 22:06:21.929491536 (582h37m21.800574571s) [seg_pos={0 0}, upload=510734]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 22:06:21.929573792 (582h37m21.800656837s) [upload=510734]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 22:06:21.929580555 (582h37m21.8006636s) [upload=510734]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 22:06:22.233045366 (582h37m22.104128401s) [upload=510734]: Done uploading segments... Dec 29 22:06:22.233064963 (582h37m22.104147998s) [upload=510734]: Upload finished. err: <nil> Duration: 15.258198294s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................X............................X] duration: 1m0.005336448s trace: https://jaeger.dev.storj.io/trace/7096862780f29049 FAIL (2025-12-29 22:07:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== oise=true , address=145.79.227.30:4766 , piece_id=YWDMUGNTYVXWOCMSN63VIC6HOTWHPY7VJ7QWT76P63UAJIBS2XLA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.060493636 (582h37m18.931576671s) [noise=true , address=5.231.73.5:4022 , piece_id=5WVAJH3Y6TLATXY4IYB6ZADKKJDXDS52B2ZN6FJAZ7UHFVRYRRAA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.060685291 (582h37m18.931768326s) [noise=true , address=77.90.31.11:4579 , piece_id=DEJPYHGYNAP52CZ45BKGNXJM5MLIA2OQZ4BRKN5PBADNRVE3VLEA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.060749954 (582h37m18.931832989s) [noise=true , address=145.79.227.29:4976 , piece_id=YRCRXSSQOXPOJGVX46L7WCB3TRDZ5QB755SEOTNHSCHN4ZDMCOUQ, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.061033203 (582h37m18.932116248s) [noise=true , address=193.149.164.143:4698 , piece_id=6X5KB4TXFV4ZOF5JTPNRW4Q3GWNCVCXCA4JSB3AZURBHXY2VHTRQ, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.061248855 (582h37m18.9323319s) [noise=true , address=145.79.227.49:4347 , piece_id=75B3DRRHEPDL6EK6ZUXTKASXUKWGAVUWBPSTNRYT4BJMAU5ALXRA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.061382729 (582h37m18.932465764s) [noise=true , address=5.231.73.2:4004 , piece_id=PTHHO3XEPGGCIRCUBIWHN4LVNBLMISJM5LESQGR2ERRESFK3S4RA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.061645560 (582h37m18.932728595s) [noise=true , address=5.231.73.3:4007 , piece_id=6AF3CEPLUMRO4CDW3SRWZSYHWEXP35AVSM5M3LH4KCRARSZ6NFNA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.061769094 (582h37m18.932852139s) [noise=true , address=145.79.227.7:4321 , piece_id=SPWPZ5MR5ZCQNYVYYQQEZC4OURAM5VAH3WN5UP6HMGCCDMCH6ZGA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.061970298 (582h37m18.933053333s) [noise=true , address=145.79.227.35:4507 , piece_id=ZZSVIHOAN2SWXI4NS3TRD4B64NDB4D43J63DAS74TCPEE6MPGRXA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.062237577 (582h37m18.933320612s) [noise=true , address=145.79.227.35:4489 , piece_id=BM7ZQESTHEUIRJWAPORTKDMT5TVLEXMOHULGFVEQKHT4AJV3HT5A, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.062304304 (582h37m18.933387349s) [noise=true , address=145.79.227.34:4931 , piece_id=XJHLE3B5HIF4RC4LV32SBCNMVTOTEXWBBCXBVPQFN6N5HFQ6ZS7A, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.062522640 (582h37m18.933605685s) [noise=true , address=145.79.227.48:4329 , piece_id=YWCVTYLCAGU57SE3ID3WW2JLCJGTJ7CPDRCZENAMO42BVPIAVLIQ, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.062904968 (582h37m18.933988003s) [noise=true , address=77.90.31.17:4801 , piece_id=K3QD7FPVRQHZL5R57EKKAC27RGDJQH6HDCND7FIVXSZBRH42WSIA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.063420419 (582h37m18.934503464s) [noise=true , address=145.79.227.17:4995 , piece_id=JJTVPVBAMCTDGHGWNPUVPYFVKD4SMTDAHF4VWVIW7JFUAMBIPRHQ, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:19.063943495 (582h37m18.93502654s) [noise=true , address=5.231.73.4:4018 , piece_id=X45H47HER27IM5FEQSEAMXSDX7UQG3AEOY65Q4WCCUOX7SVFZ5KA, seg_pos={0 0}, upload=510734]: Uploading piece... Dec 29 22:06:20.386418430 (582h37m20.257501465s) [upload=510734]: Next returned nil segment Dec 29 22:06:20.386428990 (582h37m20.257512025s) [upload=510734]: Waiting for error group managing segments... Dec 29 22:06:20.935543714 (582h37m20.806626749s) [noise=true , address=145.79.227.48:4329 , piece_id=YWCVTYLCAGU57SE3ID3WW2JLCJGTJ7CPDRCZENAMO42BVPIAVLIQ, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:20.978073277 (582h37m20.849156312s) [noise=true , address=145.79.227.39:4022 , piece_id=5IJCSVRVXCICRNRB3TTJAVNRF5FTUQMMY4KL7ZFF4SBKILFGHJWA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.079004013 (582h37m20.950087058s) [noise=true , address=5.231.73.4:4018 , piece_id=X45H47HER27IM5FEQSEAMXSDX7UQG3AEOY65Q4WCCUOX7SVFZ5KA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.101144552 (582h37m20.972227597s) [noise=true , address=77.90.31.11:4579 , piece_id=DEJPYHGYNAP52CZ45BKGNXJM5MLIA2OQZ4BRKN5PBADNRVE3VLEA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.138766369 (582h37m21.009849405s) [noise=true , address=77.90.31.17:4801 , piece_id=K3QD7FPVRQHZL5R57EKKAC27RGDJQH6HDCND7FIVXSZBRH42WSIA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.231047035 (582h37m21.10213007s) [noise=true , address=77.90.31.19:4857 , piece_id=DMJ4O3TWZYWKWVLJFSQBULHDFWZCNLVDBIGYJEOBAVVCU5UGG27Q, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.272062706 (582h37m21.143145741s) [noise=true , address=5.231.73.3:4007 , piece_id=6AF3CEPLUMRO4CDW3SRWZSYHWEXP35AVSM5M3LH4KCRARSZ6NFNA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.303520666 (582h37m21.174603711s) [noise=true , address=145.79.227.29:4976 , piece_id=YRCRXSSQOXPOJGVX46L7WCB3TRDZ5QB755SEOTNHSCHN4ZDMCOUQ, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.340640177 (582h37m21.211723212s) [noise=true , address=5.231.73.5:4022 , piece_id=5WVAJH3Y6TLATXY4IYB6ZADKKJDXDS52B2ZN6FJAZ7UHFVRYRRAA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.481118228 (582h37m21.352201263s) [noise=true , address=5.231.73.2:4004 , piece_id=PTHHO3XEPGGCIRCUBIWHN4LVNBLMISJM5LESQGR2ERRESFK3S4RA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.896953769 (582h37m21.768036804s) [noise=true , address=145.79.227.49:4347 , piece_id=75B3DRRHEPDL6EK6ZUXTKASXUKWGAVUWBPSTNRYT4BJMAU5ALXRA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.911314353 (582h37m21.782397399s) [noise=true , address=145.79.227.34:4931 , piece_id=XJHLE3B5HIF4RC4LV32SBCNMVTOTEXWBBCXBVPQFN6N5HFQ6ZS7A, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.915845371 (582h37m21.786928406s) [noise=true , address=145.79.227.30:4766 , piece_id=YWDMUGNTYVXWOCMSN63VIC6HOTWHPY7VJ7QWT76P63UAJIBS2XLA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.928785972 (582h37m21.799869017s) [noise=true , address=145.79.227.35:4489 , piece_id=BM7ZQESTHEUIRJWAPORTKDMT5TVLEXMOHULGFVEQKHT4AJV3HT5A, seg_pos={0 0}, upload=510734]: Done uploading piece. err: <nil> Dec 29 22:06:21.928810859 (582h37m21.799893904s) [seg_pos={0 0}, upload=510734]: Segment reached optimal threshold of 14 pieces. Dec 29 22:06:21.929016471 (582h37m21.800099516s) [noise=true , address=145.79.227.7:4321 , piece_id=SPWPZ5MR5ZCQNYVYYQQEZC4OURAM5VAH3WN5UP6HMGCCDMCH6ZGA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12edsHqD9iRBDgMU8vYhSM1EWG8Sn57GZ8TPiTytri3rp7W9Eae): protocol: expected piece hash; context canceled; EOF Dec 29 22:06:21.929028644 (582h37m21.800111709s) [noise=true , address=145.79.227.7:4321 , piece_id=SPWPZ5MR5ZCQNYVYYQQEZC4OURAM5VAH3WN5UP6HMGCCDMCH6ZGA, seg_pos={0 0}, upload=510734]: Long tail cancellation due to optimal threshold Dec 29 22:06:21.929093137 (582h37m21.800176192s) [noise=true , address=145.79.227.35:4507 , piece_id=ZZSVIHOAN2SWXI4NS3TRD4B64NDB4D43J63DAS74TCPEE6MPGRXA, seg_pos={0 0}, upload=510734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14jurWWPpmxwiBS3XmZysEUwugsceYJqVXFj7BbYpx8UFcTR8R): protocol: expected piece hash; context canceled; EOF Dec 29 22:06:21.929106793 (582h37m21.800189838s) [noise=true , address=145.79.227.35:4507 , piece_id=ZZSVIHOAN2SWXI4NS3TRD4B64NDB4D43J63DAS74TCPEE6MPGRXA, seg_pos={0 0}, upload=510734]: Long tail cancellation due to optimal threshold Dec 29 22:06:21.929206612 (582h37m21.800289657s) [noise=true , address=145.79.227.17:4995 , piece_id=JJTVPVBAMCTDGHGWNPUVPYFVKD4SMTDAHF4VWVIW7JFUAMBIPRHQ, seg_pos={0 0}, upload=510734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tvoTD1LoSj3q2rLiDckfT56dXNKKwdVu1ZAsBZExNFVTom9tH): protocol: write tcp 188.166.225.170:47696->145.79.227.17:4995: use of closed network connection; write tcp 188.166.225.170:47696->145.79.227.17:4995: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47696->145.79.227.17:4995: use of closed network connection Dec 29 22:06:21.929215911 (582h37m21.800298976s) [noise=true , address=145.79.227.17:4995 , piece_id=JJTVPVBAMCTDGHGWNPUVPYFVKD4SMTDAHF4VWVIW7JFUAMBIPRHQ, seg_pos={0 0}, upload=510734]: Long tail cancellation due to optimal threshold Dec 29 22:06:21.929387247 (582h37m21.800470292s) [noise=true , address=193.149.164.143:4698 , piece_id=6X5KB4TXFV4ZOF5JTPNRW4Q3GWNCVCXCA4JSB3AZURBHXY2VHTRQ, seg_pos={0 0}, upload=510734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F4fJAGJM2ZPRt7vJH2cZmEbYeujmxcfi7aPZ2g9VPTNYBYRAi): protocol: expected piece hash; context canceled; EOF Dec 29 22:06:21.929396174 (582h37m21.800479219s) [noise=true , address=193.149.164.143:4698 , piece_id=6X5KB4TXFV4ZOF5JTPNRW4Q3GWNCVCXCA4JSB3AZURBHXY2VHTRQ, seg_pos={0 0}, upload=510734]: Long tail cancellation due to optimal threshold Dec 29 22:06:21.929491536 (582h37m21.800574571s) [seg_pos={0 0}, upload=510734]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 22:06:21.929573792 (582h37m21.800656837s) [upload=510734]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 22:06:21.929580555 (582h37m21.8006636s) [upload=510734]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 22:06:22.233045366 (582h37m22.104128401s) [upload=510734]: Done uploading segments... Dec 29 22:06:22.233064963 (582h37m22.104147998s) [upload=510734]: Upload finished. err: <nil> Duration: 1m0.005361947s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [........X..X..X........X............................X.......] duration: 6.126975362s trace: https://jaeger.dev.storj.io/trace/5209b3d035311fd FAIL (2025-12-29 22:06:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :00:17.043136250 (582h31m16.914219305s) [upload=510642]: Starting upload Dec 29 22:00:17.043227904 (582h31m16.91431096s) [upload=510642]: Done starting upload Dec 29 22:00:17.043335189 (582h31m16.914418234s) [upload=510642]: Uploading segments... Dec 29 22:00:17.043587749 (582h31m16.914670824s) [upload=510642]: Got next segment. Inline: false Dec 29 22:00:17.043644478 (582h31m16.914727523s) [upload=510642]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 22:00:17.043686427 (582h31m16.914769503s) [upload=510642]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 22:00:17.294384363 (582h31m17.165467408s) [seg_pos={0 0}, upload=510642]: Begin upload segment... Dec 29 22:00:17.294445219 (582h31m17.165528264s) [seg_pos={0 0}, upload=510642]: Done begin upload segment. Dec 29 22:00:17.294484133 (582h31m17.165567178s) [noise=true , address=145.79.227.31:4797 , piece_id=NI3M5NVVVX5HUXZ2SOMQD4S5ZOSSNPCD2F3Z5X2BA2VXMGT57AYQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.294560979 (582h31m17.165644014s) [noise=true , address=77.90.31.11:4553 , piece_id=3Y5HSD73DLZ6ALJ5TFRIVTLOE4ZNZ256YSZ2W3X5LPKL5J5LX2EQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.295149941 (582h31m17.166232976s) [noise=true , address=45.91.116.5:4297 , piece_id=PLUQ7Q32BTGYFQRGVJFAJWHPZCPQTC5RX5UX27S5HJ5NNJPLDMPQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.295419574 (582h31m17.166502609s) [noise=true , address=145.79.227.2:4115 , piece_id=P2AS5B3VYO2K2AWSM5W4BXAXBHSXQQWM7JOXFPOV64NTKDFLT62Q, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.295483997 (582h31m17.166567042s) [noise=true , address=145.79.227.51:4411 , piece_id=HTZQ6SW6KSACLQF7G4CFGQA572J5JVHMLGEWQYSSGZRYX7FJ6DXA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.295762207 (582h31m17.166845252s) [noise=true , address=145.79.227.5:4221 , piece_id=LD3FENLBM2L5N7QDOAMAYJUVH54QVSJQJFP7E36UAH2ZD7R65CCQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.295928003 (582h31m17.167011048s) [noise=true , address=145.79.227.16:4909 , piece_id=CF7WHNKC4LFJQRVFBPFV62JGEBLD4KVPE6Q5JBWQVADC6ZAFFLVQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.296039615 (582h31m17.16712266s) [noise=true , address=45.91.116.13:4154 , piece_id=FE4NFODQ3TWGFXARFYIL64TKK6X7H6PPLIG6LO44LKNAZOEU7DJA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.296272429 (582h31m17.167355474s) [noise=true , address=45.91.116.10:4084 , piece_id=5BFUYVGIFOYXC6RCPU7VBUM36LSM32TVBLS3W54OWFNR4SM6GCSA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.296403588 (582h31m17.167486633s) [noise=true , address=77.90.31.19:4844 , piece_id=CDOWECXLSRPACV4HMAH57UK4EEIYNOMH6GYV6HFHV7CWSEAJ44FA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.296646560 (582h31m17.167729596s) [noise=true , address=77.90.31.14:4661 , piece_id=B7IDSCDHBUXUIPSUT3HMQSCMPGBMYASZSQO2EJBXRR2DQHCLWGYA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.296906235 (582h31m17.167989291s) [noise=true , address=145.79.227.47:4265 , piece_id=FYZA2LRIFKLKVUCUAWMNWTUH3VYR5VFAA7XKAYIMD2E2VOJTI7PA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.296989804 (582h31m17.168072849s) [noise=true , address=145.79.227.46:4601 , piece_id=YJTMZID24VBM5ZZSUUHOCZSZSSNGTFHOSKK6OI5QP3A4G6ZWVWQA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.297235221 (582h31m17.168318267s) [noise=true , address=145.79.227.3:4150 , piece_id=HS4S5UDYD2VWI2N4MH6EWEYQ5G6KCFCJBDZR7AVWPXRYALFXP4FQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.297424432 (582h31m17.168507467s) [noise=true , address=145.79.227.13:4803 , piece_id=YIEUZEDLSC5C67SCFEPCHHZSTYP6MYLYJZJTGZF3UP6JWDTGUOPQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.297606900 (582h31m17.168689945s) [noise=true , address=45.91.116.14:4190 , piece_id=FIL6AIXGE7DE2M4QBVSIO53KGY43VFPZQ2FTXSWJFPJU4VZCKUXQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:18.995364155 (582h31m18.866447201s) [upload=510642]: Next returned nil segment Dec 29 22:00:18.995383603 (582h31m18.866466648s) [upload=510642]: Waiting for error group managing segments... Dec 29 22:00:19.284009408 (582h31m19.155092453s) [noise=true , address=145.79.227.51:4411 , piece_id=HTZQ6SW6KSACLQF7G4CFGQA572J5JVHMLGEWQYSSGZRYX7FJ6DXA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.295985232 (582h31m19.167068277s) [noise=true , address=145.79.227.13:4803 , piece_id=YIEUZEDLSC5C67SCFEPCHHZSTYP6MYLYJZJTGZF3UP6JWDTGUOPQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.296135458 (582h31m19.167218503s) [noise=true , address=145.79.227.16:4909 , piece_id=CF7WHNKC4LFJQRVFBPFV62JGEBLD4KVPE6Q5JBWQVADC6ZAFFLVQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.310119916 (582h31m19.181202971s) [noise=true , address=145.79.227.46:4601 , piece_id=YJTMZID24VBM5ZZSUUHOCZSZSSNGTFHOSKK6OI5QP3A4G6ZWVWQA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.310998939 (582h31m19.182081984s) [noise=true , address=145.79.227.31:4797 , piece_id=NI3M5NVVVX5HUXZ2SOMQD4S5ZOSSNPCD2F3Z5X2BA2VXMGT57AYQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.318707138 (582h31m19.189790183s) [noise=true , address=145.79.227.47:4265 , piece_id=FYZA2LRIFKLKVUCUAWMNWTUH3VYR5VFAA7XKAYIMD2E2VOJTI7PA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.322059381 (582h31m19.193142426s) [noise=true , address=145.79.227.5:4221 , piece_id=LD3FENLBM2L5N7QDOAMAYJUVH54QVSJQJFP7E36UAH2ZD7R65CCQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.506372108 (582h31m19.377455153s) [noise=true , address=77.90.31.19:4844 , piece_id=CDOWECXLSRPACV4HMAH57UK4EEIYNOMH6GYV6HFHV7CWSEAJ44FA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.507172602 (582h31m19.378255648s) [noise=true , address=77.90.31.11:4553 , piece_id=3Y5HSD73DLZ6ALJ5TFRIVTLOE4ZNZ256YSZ2W3X5LPKL5J5LX2EQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.527548842 (582h31m19.398631887s) [noise=true , address=77.90.31.14:4661 , piece_id=B7IDSCDHBUXUIPSUT3HMQSCMPGBMYASZSQO2EJBXRR2DQHCLWGYA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.531443508 (582h31m19.402526553s) [noise=true , address=45.91.116.5:4297 , piece_id=PLUQ7Q32BTGYFQRGVJFAJWHPZCPQTC5RX5UX27S5HJ5NNJPLDMPQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.552364906 (582h31m19.423447951s) [noise=true , address=45.91.116.14:4190 , piece_id=FIL6AIXGE7DE2M4QBVSIO53KGY43VFPZQ2FTXSWJFPJU4VZCKUXQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.564056769 (582h31m19.435139814s) [noise=true , address=45.91.116.10:4084 , piece_id=5BFUYVGIFOYXC6RCPU7VBUM36LSM32TVBLS3W54OWFNR4SM6GCSA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.564087016 (582h31m19.435170051s) [seg_pos={0 0}, upload=510642]: Segment reached optimal threshold of 13 pieces. Dec 29 22:00:19.564294431 (582h31m19.435377466s) [noise=true , address=145.79.227.3:4150 , piece_id=HS4S5UDYD2VWI2N4MH6EWEYQ5G6KCFCJBDZR7AVWPXRYALFXP4FQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK): protocol: expected piece hash; context canceled; EOF Dec 29 22:00:19.564302366 (582h31m19.435385411s) [noise=true , address=145.79.227.3:4150 , piece_id=HS4S5UDYD2VWI2N4MH6EWEYQ5G6KCFCJBDZR7AVWPXRYALFXP4FQ, seg_pos={0 0}, upload=510642]: Long tail cancellation due to optimal threshold Dec 29 22:00:19.564388932 (582h31m19.435471977s) [noise=true , address=45.91.116.13:4154 , piece_id=FE4NFODQ3TWGFXARFYIL64TKK6X7H6PPLIG6LO44LKNAZOEU7DJA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Dec 29 22:00:19.564401876 (582h31m19.435484911s) [noise=true , address=45.91.116.13:4154 , piece_id=FE4NFODQ3TWGFXARFYIL64TKK6X7H6PPLIG6LO44LKNAZOEU7DJA, seg_pos={0 0}, upload=510642]: Long tail cancellation due to optimal threshold Dec 29 22:00:19.564558945 (582h31m19.435641981s) [noise=true , address=145.79.227.2:4115 , piece_id=P2AS5B3VYO2K2AWSM5W4BXAXBHSXQQWM7JOXFPOV64NTKDFLT62Q, seg_pos={0 0}, upload=510642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: write tcp 188.166.225.170:38246->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:38246->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38246->145.79.227.2:4115: use of closed network connection Dec 29 22:00:19.564570738 (582h31m19.435653783s) [noise=true , address=145.79.227.2:4115 , piece_id=P2AS5B3VYO2K2AWSM5W4BXAXBHSXQQWM7JOXFPOV64NTKDFLT62Q, seg_pos={0 0}, upload=510642]: Long tail cancellation due to optimal threshold Dec 29 22:00:19.564601426 (582h31m19.435684471s) [seg_pos={0 0}, upload=510642]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 22:00:19.564683973 (582h31m19.435767018s) [upload=510642]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 22:00:19.564690004 (582h31m19.435773049s) [upload=510642]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 22:00:19.871860472 (582h31m19.742943527s) [upload=510642]: Done uploading segments... Dec 29 22:00:19.871892413 (582h31m19.742975468s) [upload=510642]: Upload finished. err: <nil> Duration: 6.12702208s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 15.170452739s trace: https://jaeger.dev.storj.io/trace/54393681021e4741 FAIL (2025-12-29 22:06:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== upload=510381]: Uploading piece... Dec 29 21:42:00.490334342 (582h13m0.361417387s) [noise=true , address=77.90.31.13:4637 , piece_id=KZRLS4PBFYGGHYJRQLMBEOCNBYCQIMF3F32I3HZFTIUIAT6AE67A, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.490753791 (582h13m0.361836826s) [noise=true , address=77.90.31.19:4861 , piece_id=5ODRE74A5PTEPPYWCQS25K6HXMRPZKQIXMX6EOTQMB5XSGNN6TJA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.491447442 (582h13m0.362530487s) [noise=true , address=145.79.227.49:4357 , piece_id=TP57SIC35AAYKPEF2WIQMJJE76C4ERVDRQGKF7462KW66AMCEFSA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.491940861 (582h13m0.363023896s) [noise=true , address=145.79.227.33:4891 , piece_id=MRIXCIGX6WVNEVCXPLWCWZ67JVIA5JDZLHCT6RE4NCMAFANMVVEQ, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.492346664 (582h13m0.363429699s) [noise=true , address=77.90.31.15:4713 , piece_id=SPWJMKD6RJ555CL5PZ2Z7HNW2K67MC2U7QHG6NELHNRFQEB4OFKA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.492751746 (582h13m0.363834791s) [noise=true , address=77.90.31.13:4656 , piece_id=T3NU3UMWI7VTETXC2G5VCPH5YASFGCQMHOHB6UGQT2PB5ECA272A, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.493112152 (582h13m0.364195187s) [noise=true , address=64.49.13.140:4510 , piece_id=XFDDU2JYEYIWKN7EZWJZEP34QDUWEPL3OW2TR3JSDE2QY6CV3VDQ, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.493462459 (582h13m0.364545504s) [noise=true , address=145.79.227.29:4946 , piece_id=DGZAZPHNELKHOGDMC6GFKSO4QVCIFGQHXSH6VJJZUJRHPXLWSUPA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.493859384 (582h13m0.36494242s) [noise=true , address=145.79.227.44:4226 , piece_id=JSXESMXNV3MERJUKWNFKJDOWCTXIG6IQ7JMIXXW2JL2TMYBNBDLQ, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.494272812 (582h13m0.365355847s) [noise=true , address=77.90.31.16:4766 , piece_id=DBPOMKHGXGZQLUHK2NL5M26VRBOWAGZXOOBB2U3XUGHWDGLHEV6Q, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.495004285 (582h13m0.36608733s) [noise=true , address=145.79.227.4:4219 , piece_id=YVMXT3TQ5DFBGRF45A2FT4QVVVWJNLTUTFGRHRUXB2LDBLGCU7XA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.495625678 (582h13m0.366708723s) [noise=true , address=145.79.227.43:4184 , piece_id=ZYUY4IH7NV42CRLTWLA56ASUA3RBKE3HRNG33VEVR6YBSFRM5QSA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.496138595 (582h13m0.36722163s) [noise=true , address=145.79.227.40:4053 , piece_id=SBBX53GUBMB57OJBNXJMCVS3KYD4ZQ5LUCV572LKJUNYDF6MIJ7A, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.496524049 (582h13m0.367607094s) [noise=true , address=145.79.227.17:4974 , piece_id=ECY3XG222XZAVEEY3W4RFOXMUO7IMAXGMKA7F2MSM6WQJYQT6YZA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.496920243 (582h13m0.368003288s) [noise=true , address=145.79.227.34:4936 , piece_id=WPHQ4WM222WNPSNQHJRHGWWOY3453GNWFU2VOWGME2NUVKGTTCLA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.497333270 (582h13m0.368416305s) [noise=true , address=145.79.227.21:4492 , piece_id=SYF3MNTT4KCEKJIWUVVMK4WS2PYZNYJNLG2HE6UW2F4DAHFHLTAQ, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.497852078 (582h13m0.368935113s) [noise=true , address=145.79.227.39:4038 , piece_id=WK7OI2D5XAKGOM25ISG4IN6FPXIS2F6YDEOQFIQZPYWVSTCZC2CA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:01.777921879 (582h13m1.649004924s) [upload=510381]: Next returned nil segment Dec 29 21:42:01.777940274 (582h13m1.649023349s) [upload=510381]: Waiting for error group managing segments... Dec 29 21:42:02.141106696 (582h13m2.012189741s) [noise=true , address=64.49.13.140:4510 , piece_id=XFDDU2JYEYIWKN7EZWJZEP34QDUWEPL3OW2TR3JSDE2QY6CV3VDQ, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.376774281 (582h13m2.247857327s) [noise=true , address=145.79.227.39:4038 , piece_id=WK7OI2D5XAKGOM25ISG4IN6FPXIS2F6YDEOQFIQZPYWVSTCZC2CA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.379986568 (582h13m2.251069613s) [noise=true , address=145.79.227.21:4492 , piece_id=SYF3MNTT4KCEKJIWUVVMK4WS2PYZNYJNLG2HE6UW2F4DAHFHLTAQ, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.386102975 (582h13m2.25718602s) [noise=true , address=145.79.227.17:4974 , piece_id=ECY3XG222XZAVEEY3W4RFOXMUO7IMAXGMKA7F2MSM6WQJYQT6YZA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.390565783 (582h13m2.261648818s) [noise=true , address=145.79.227.4:4219 , piece_id=YVMXT3TQ5DFBGRF45A2FT4QVVVWJNLTUTFGRHRUXB2LDBLGCU7XA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.393984613 (582h13m2.265067658s) [noise=true , address=145.79.227.6:4266 , piece_id=QXMR4B6HMIOHPRFH3ABT3ZHSOF2ZE23SAGXIWUAEM5LO67NFNEFA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.524875288 (582h13m2.395958333s) [noise=true , address=77.90.31.13:4637 , piece_id=KZRLS4PBFYGGHYJRQLMBEOCNBYCQIMF3F32I3HZFTIUIAT6AE67A, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.540524625 (582h13m2.41160767s) [noise=true , address=77.90.31.13:4656 , piece_id=T3NU3UMWI7VTETXC2G5VCPH5YASFGCQMHOHB6UGQT2PB5ECA272A, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.557053169 (582h13m2.428136214s) [noise=true , address=77.90.31.16:4766 , piece_id=DBPOMKHGXGZQLUHK2NL5M26VRBOWAGZXOOBB2U3XUGHWDGLHEV6Q, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.641331988 (582h13m2.512415043s) [noise=true , address=77.90.31.19:4861 , piece_id=5ODRE74A5PTEPPYWCQS25K6HXMRPZKQIXMX6EOTQMB5XSGNN6TJA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:03.276156729 (582h13m3.147239814s) [noise=true , address=145.79.227.34:4936 , piece_id=WPHQ4WM222WNPSNQHJRHGWWOY3453GNWFU2VOWGME2NUVKGTTCLA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:03.285451126 (582h13m3.156534171s) [noise=true , address=77.90.31.15:4713 , piece_id=SPWJMKD6RJ555CL5PZ2Z7HNW2K67MC2U7QHG6NELHNRFQEB4OFKA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:03.293385374 (582h13m3.164468419s) [noise=true , address=145.79.227.44:4226 , piece_id=JSXESMXNV3MERJUKWNFKJDOWCTXIG6IQ7JMIXXW2JL2TMYBNBDLQ, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:03.293494552 (582h13m3.164577597s) [noise=true , address=145.79.227.49:4357 , piece_id=TP57SIC35AAYKPEF2WIQMJJE76C4ERVDRQGKF7462KW66AMCEFSA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:03.293514761 (582h13m3.164597796s) [seg_pos={0 0}, upload=510381]: Segment reached optimal threshold of 14 pieces. Dec 29 21:42:03.293900185 (582h13m3.16498322s) [noise=true , address=145.79.227.43:4184 , piece_id=ZYUY4IH7NV42CRLTWLA56ASUA3RBKE3HRNG33VEVR6YBSFRM5QSA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VYcRgiFADDQ1rLsiMufThN1K51cuoqKRXNjbp2w9Exu1d3gQG): protocol: expected piece hash; context canceled; EOF Dec 29 21:42:03.293912828 (582h13m3.164995873s) [noise=true , address=145.79.227.43:4184 , piece_id=ZYUY4IH7NV42CRLTWLA56ASUA3RBKE3HRNG33VEVR6YBSFRM5QSA, seg_pos={0 0}, upload=510381]: Long tail cancellation due to optimal threshold Dec 29 21:42:03.293954789 (582h13m3.165037834s) [noise=true , address=145.79.227.29:4946 , piece_id=DGZAZPHNELKHOGDMC6GFKSO4QVCIFGQHXSH6VJJZUJRHPXLWSUPA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12v9Q7aigjWPNXyshHmQt9YKrDu5af1vd1TxkDeHeKixSrWgrLX): protocol: expected piece hash; context canceled; EOF Dec 29 21:42:03.293966481 (582h13m3.165049546s) [noise=true , address=145.79.227.29:4946 , piece_id=DGZAZPHNELKHOGDMC6GFKSO4QVCIFGQHXSH6VJJZUJRHPXLWSUPA, seg_pos={0 0}, upload=510381]: Long tail cancellation due to optimal threshold Dec 29 21:42:03.294146864 (582h13m3.165229899s) [noise=true , address=145.79.227.33:4891 , piece_id=MRIXCIGX6WVNEVCXPLWCWZ67JVIA5JDZLHCT6RE4NCMAFANMVVEQ, seg_pos={0 0}, upload=510381]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RHLM4MJmtKoHsUnsqccy1ehyD7zVSn6gnJNp7yJJmZhTJqPkU): protocol: expected piece hash; context canceled; EOF Dec 29 21:42:03.294155210 (582h13m3.165238245s) [noise=true , address=145.79.227.33:4891 , piece_id=MRIXCIGX6WVNEVCXPLWCWZ67JVIA5JDZLHCT6RE4NCMAFANMVVEQ, seg_pos={0 0}, upload=510381]: Long tail cancellation due to optimal threshold Dec 29 21:42:03.294191519 (582h13m3.165274574s) [noise=true , address=145.79.227.40:4053 , piece_id=SBBX53GUBMB57OJBNXJMCVS3KYD4ZQ5LUCV572LKJUNYDF6MIJ7A, seg_pos={0 0}, upload=510381]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DqyTc4MHvxutFXs2WWRCk8MSuxF3Y1HayyQY4Y5sy8cqDLWuG): protocol: expected piece hash; context canceled; EOF Dec 29 21:42:03.294202380 (582h13m3.165285425s) [noise=true , address=145.79.227.40:4053 , piece_id=SBBX53GUBMB57OJBNXJMCVS3KYD4ZQ5LUCV572LKJUNYDF6MIJ7A, seg_pos={0 0}, upload=510381]: Long tail cancellation due to optimal threshold Dec 29 21:42:03.294227728 (582h13m3.165310763s) [seg_pos={0 0}, upload=510381]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 21:42:03.294328840 (582h13m3.165411886s) [upload=510381]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 21:42:03.294336927 (582h13m3.165419972s) [upload=510381]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 21:42:03.627673008 (582h13m3.498756053s) [upload=510381]: Done uploading segments... Dec 29 21:42:03.627701772 (582h13m3.498784807s) [upload=510381]: Upload finished. err: <nil> Duration: 15.170482635s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.........X..X..X........X............................X......] duration: 7.186763809s trace: https://jaeger.dev.storj.io/trace/5209b3d035311fd FAIL (2025-12-29 22:05:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :00:17.043136250 (582h31m16.914219305s) [upload=510642]: Starting upload Dec 29 22:00:17.043227904 (582h31m16.91431096s) [upload=510642]: Done starting upload Dec 29 22:00:17.043335189 (582h31m16.914418234s) [upload=510642]: Uploading segments... Dec 29 22:00:17.043587749 (582h31m16.914670824s) [upload=510642]: Got next segment. Inline: false Dec 29 22:00:17.043644478 (582h31m16.914727523s) [upload=510642]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 22:00:17.043686427 (582h31m16.914769503s) [upload=510642]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 22:00:17.294384363 (582h31m17.165467408s) [seg_pos={0 0}, upload=510642]: Begin upload segment... Dec 29 22:00:17.294445219 (582h31m17.165528264s) [seg_pos={0 0}, upload=510642]: Done begin upload segment. Dec 29 22:00:17.294484133 (582h31m17.165567178s) [noise=true , address=145.79.227.31:4797 , piece_id=NI3M5NVVVX5HUXZ2SOMQD4S5ZOSSNPCD2F3Z5X2BA2VXMGT57AYQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.294560979 (582h31m17.165644014s) [noise=true , address=77.90.31.11:4553 , piece_id=3Y5HSD73DLZ6ALJ5TFRIVTLOE4ZNZ256YSZ2W3X5LPKL5J5LX2EQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.295149941 (582h31m17.166232976s) [noise=true , address=45.91.116.5:4297 , piece_id=PLUQ7Q32BTGYFQRGVJFAJWHPZCPQTC5RX5UX27S5HJ5NNJPLDMPQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.295419574 (582h31m17.166502609s) [noise=true , address=145.79.227.2:4115 , piece_id=P2AS5B3VYO2K2AWSM5W4BXAXBHSXQQWM7JOXFPOV64NTKDFLT62Q, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.295483997 (582h31m17.166567042s) [noise=true , address=145.79.227.51:4411 , piece_id=HTZQ6SW6KSACLQF7G4CFGQA572J5JVHMLGEWQYSSGZRYX7FJ6DXA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.295762207 (582h31m17.166845252s) [noise=true , address=145.79.227.5:4221 , piece_id=LD3FENLBM2L5N7QDOAMAYJUVH54QVSJQJFP7E36UAH2ZD7R65CCQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.295928003 (582h31m17.167011048s) [noise=true , address=145.79.227.16:4909 , piece_id=CF7WHNKC4LFJQRVFBPFV62JGEBLD4KVPE6Q5JBWQVADC6ZAFFLVQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.296039615 (582h31m17.16712266s) [noise=true , address=45.91.116.13:4154 , piece_id=FE4NFODQ3TWGFXARFYIL64TKK6X7H6PPLIG6LO44LKNAZOEU7DJA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.296272429 (582h31m17.167355474s) [noise=true , address=45.91.116.10:4084 , piece_id=5BFUYVGIFOYXC6RCPU7VBUM36LSM32TVBLS3W54OWFNR4SM6GCSA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.296403588 (582h31m17.167486633s) [noise=true , address=77.90.31.19:4844 , piece_id=CDOWECXLSRPACV4HMAH57UK4EEIYNOMH6GYV6HFHV7CWSEAJ44FA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.296646560 (582h31m17.167729596s) [noise=true , address=77.90.31.14:4661 , piece_id=B7IDSCDHBUXUIPSUT3HMQSCMPGBMYASZSQO2EJBXRR2DQHCLWGYA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.296906235 (582h31m17.167989291s) [noise=true , address=145.79.227.47:4265 , piece_id=FYZA2LRIFKLKVUCUAWMNWTUH3VYR5VFAA7XKAYIMD2E2VOJTI7PA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.296989804 (582h31m17.168072849s) [noise=true , address=145.79.227.46:4601 , piece_id=YJTMZID24VBM5ZZSUUHOCZSZSSNGTFHOSKK6OI5QP3A4G6ZWVWQA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.297235221 (582h31m17.168318267s) [noise=true , address=145.79.227.3:4150 , piece_id=HS4S5UDYD2VWI2N4MH6EWEYQ5G6KCFCJBDZR7AVWPXRYALFXP4FQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.297424432 (582h31m17.168507467s) [noise=true , address=145.79.227.13:4803 , piece_id=YIEUZEDLSC5C67SCFEPCHHZSTYP6MYLYJZJTGZF3UP6JWDTGUOPQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.297606900 (582h31m17.168689945s) [noise=true , address=45.91.116.14:4190 , piece_id=FIL6AIXGE7DE2M4QBVSIO53KGY43VFPZQ2FTXSWJFPJU4VZCKUXQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:18.995364155 (582h31m18.866447201s) [upload=510642]: Next returned nil segment Dec 29 22:00:18.995383603 (582h31m18.866466648s) [upload=510642]: Waiting for error group managing segments... Dec 29 22:00:19.284009408 (582h31m19.155092453s) [noise=true , address=145.79.227.51:4411 , piece_id=HTZQ6SW6KSACLQF7G4CFGQA572J5JVHMLGEWQYSSGZRYX7FJ6DXA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.295985232 (582h31m19.167068277s) [noise=true , address=145.79.227.13:4803 , piece_id=YIEUZEDLSC5C67SCFEPCHHZSTYP6MYLYJZJTGZF3UP6JWDTGUOPQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.296135458 (582h31m19.167218503s) [noise=true , address=145.79.227.16:4909 , piece_id=CF7WHNKC4LFJQRVFBPFV62JGEBLD4KVPE6Q5JBWQVADC6ZAFFLVQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.310119916 (582h31m19.181202971s) [noise=true , address=145.79.227.46:4601 , piece_id=YJTMZID24VBM5ZZSUUHOCZSZSSNGTFHOSKK6OI5QP3A4G6ZWVWQA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.310998939 (582h31m19.182081984s) [noise=true , address=145.79.227.31:4797 , piece_id=NI3M5NVVVX5HUXZ2SOMQD4S5ZOSSNPCD2F3Z5X2BA2VXMGT57AYQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.318707138 (582h31m19.189790183s) [noise=true , address=145.79.227.47:4265 , piece_id=FYZA2LRIFKLKVUCUAWMNWTUH3VYR5VFAA7XKAYIMD2E2VOJTI7PA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.322059381 (582h31m19.193142426s) [noise=true , address=145.79.227.5:4221 , piece_id=LD3FENLBM2L5N7QDOAMAYJUVH54QVSJQJFP7E36UAH2ZD7R65CCQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.506372108 (582h31m19.377455153s) [noise=true , address=77.90.31.19:4844 , piece_id=CDOWECXLSRPACV4HMAH57UK4EEIYNOMH6GYV6HFHV7CWSEAJ44FA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.507172602 (582h31m19.378255648s) [noise=true , address=77.90.31.11:4553 , piece_id=3Y5HSD73DLZ6ALJ5TFRIVTLOE4ZNZ256YSZ2W3X5LPKL5J5LX2EQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.527548842 (582h31m19.398631887s) [noise=true , address=77.90.31.14:4661 , piece_id=B7IDSCDHBUXUIPSUT3HMQSCMPGBMYASZSQO2EJBXRR2DQHCLWGYA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.531443508 (582h31m19.402526553s) [noise=true , address=45.91.116.5:4297 , piece_id=PLUQ7Q32BTGYFQRGVJFAJWHPZCPQTC5RX5UX27S5HJ5NNJPLDMPQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.552364906 (582h31m19.423447951s) [noise=true , address=45.91.116.14:4190 , piece_id=FIL6AIXGE7DE2M4QBVSIO53KGY43VFPZQ2FTXSWJFPJU4VZCKUXQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.564056769 (582h31m19.435139814s) [noise=true , address=45.91.116.10:4084 , piece_id=5BFUYVGIFOYXC6RCPU7VBUM36LSM32TVBLS3W54OWFNR4SM6GCSA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.564087016 (582h31m19.435170051s) [seg_pos={0 0}, upload=510642]: Segment reached optimal threshold of 13 pieces. Dec 29 22:00:19.564294431 (582h31m19.435377466s) [noise=true , address=145.79.227.3:4150 , piece_id=HS4S5UDYD2VWI2N4MH6EWEYQ5G6KCFCJBDZR7AVWPXRYALFXP4FQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK): protocol: expected piece hash; context canceled; EOF Dec 29 22:00:19.564302366 (582h31m19.435385411s) [noise=true , address=145.79.227.3:4150 , piece_id=HS4S5UDYD2VWI2N4MH6EWEYQ5G6KCFCJBDZR7AVWPXRYALFXP4FQ, seg_pos={0 0}, upload=510642]: Long tail cancellation due to optimal threshold Dec 29 22:00:19.564388932 (582h31m19.435471977s) [noise=true , address=45.91.116.13:4154 , piece_id=FE4NFODQ3TWGFXARFYIL64TKK6X7H6PPLIG6LO44LKNAZOEU7DJA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Dec 29 22:00:19.564401876 (582h31m19.435484911s) [noise=true , address=45.91.116.13:4154 , piece_id=FE4NFODQ3TWGFXARFYIL64TKK6X7H6PPLIG6LO44LKNAZOEU7DJA, seg_pos={0 0}, upload=510642]: Long tail cancellation due to optimal threshold Dec 29 22:00:19.564558945 (582h31m19.435641981s) [noise=true , address=145.79.227.2:4115 , piece_id=P2AS5B3VYO2K2AWSM5W4BXAXBHSXQQWM7JOXFPOV64NTKDFLT62Q, seg_pos={0 0}, upload=510642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: write tcp 188.166.225.170:38246->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:38246->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38246->145.79.227.2:4115: use of closed network connection Dec 29 22:00:19.564570738 (582h31m19.435653783s) [noise=true , address=145.79.227.2:4115 , piece_id=P2AS5B3VYO2K2AWSM5W4BXAXBHSXQQWM7JOXFPOV64NTKDFLT62Q, seg_pos={0 0}, upload=510642]: Long tail cancellation due to optimal threshold Dec 29 22:00:19.564601426 (582h31m19.435684471s) [seg_pos={0 0}, upload=510642]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 22:00:19.564683973 (582h31m19.435767018s) [upload=510642]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 22:00:19.564690004 (582h31m19.435773049s) [upload=510642]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 22:00:19.871860472 (582h31m19.742943527s) [upload=510642]: Done uploading segments... Dec 29 22:00:19.871892413 (582h31m19.742975468s) [upload=510642]: Upload finished. err: <nil> Duration: 7.186794959s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 6.538828634s trace: https://jaeger.dev.storj.io/trace/54393681021e4741 FAIL (2025-12-29 22:05:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== upload=510381]: Uploading piece... Dec 29 21:42:00.490334342 (582h13m0.361417387s) [noise=true , address=77.90.31.13:4637 , piece_id=KZRLS4PBFYGGHYJRQLMBEOCNBYCQIMF3F32I3HZFTIUIAT6AE67A, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.490753791 (582h13m0.361836826s) [noise=true , address=77.90.31.19:4861 , piece_id=5ODRE74A5PTEPPYWCQS25K6HXMRPZKQIXMX6EOTQMB5XSGNN6TJA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.491447442 (582h13m0.362530487s) [noise=true , address=145.79.227.49:4357 , piece_id=TP57SIC35AAYKPEF2WIQMJJE76C4ERVDRQGKF7462KW66AMCEFSA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.491940861 (582h13m0.363023896s) [noise=true , address=145.79.227.33:4891 , piece_id=MRIXCIGX6WVNEVCXPLWCWZ67JVIA5JDZLHCT6RE4NCMAFANMVVEQ, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.492346664 (582h13m0.363429699s) [noise=true , address=77.90.31.15:4713 , piece_id=SPWJMKD6RJ555CL5PZ2Z7HNW2K67MC2U7QHG6NELHNRFQEB4OFKA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.492751746 (582h13m0.363834791s) [noise=true , address=77.90.31.13:4656 , piece_id=T3NU3UMWI7VTETXC2G5VCPH5YASFGCQMHOHB6UGQT2PB5ECA272A, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.493112152 (582h13m0.364195187s) [noise=true , address=64.49.13.140:4510 , piece_id=XFDDU2JYEYIWKN7EZWJZEP34QDUWEPL3OW2TR3JSDE2QY6CV3VDQ, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.493462459 (582h13m0.364545504s) [noise=true , address=145.79.227.29:4946 , piece_id=DGZAZPHNELKHOGDMC6GFKSO4QVCIFGQHXSH6VJJZUJRHPXLWSUPA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.493859384 (582h13m0.36494242s) [noise=true , address=145.79.227.44:4226 , piece_id=JSXESMXNV3MERJUKWNFKJDOWCTXIG6IQ7JMIXXW2JL2TMYBNBDLQ, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.494272812 (582h13m0.365355847s) [noise=true , address=77.90.31.16:4766 , piece_id=DBPOMKHGXGZQLUHK2NL5M26VRBOWAGZXOOBB2U3XUGHWDGLHEV6Q, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.495004285 (582h13m0.36608733s) [noise=true , address=145.79.227.4:4219 , piece_id=YVMXT3TQ5DFBGRF45A2FT4QVVVWJNLTUTFGRHRUXB2LDBLGCU7XA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.495625678 (582h13m0.366708723s) [noise=true , address=145.79.227.43:4184 , piece_id=ZYUY4IH7NV42CRLTWLA56ASUA3RBKE3HRNG33VEVR6YBSFRM5QSA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.496138595 (582h13m0.36722163s) [noise=true , address=145.79.227.40:4053 , piece_id=SBBX53GUBMB57OJBNXJMCVS3KYD4ZQ5LUCV572LKJUNYDF6MIJ7A, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.496524049 (582h13m0.367607094s) [noise=true , address=145.79.227.17:4974 , piece_id=ECY3XG222XZAVEEY3W4RFOXMUO7IMAXGMKA7F2MSM6WQJYQT6YZA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.496920243 (582h13m0.368003288s) [noise=true , address=145.79.227.34:4936 , piece_id=WPHQ4WM222WNPSNQHJRHGWWOY3453GNWFU2VOWGME2NUVKGTTCLA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.497333270 (582h13m0.368416305s) [noise=true , address=145.79.227.21:4492 , piece_id=SYF3MNTT4KCEKJIWUVVMK4WS2PYZNYJNLG2HE6UW2F4DAHFHLTAQ, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.497852078 (582h13m0.368935113s) [noise=true , address=145.79.227.39:4038 , piece_id=WK7OI2D5XAKGOM25ISG4IN6FPXIS2F6YDEOQFIQZPYWVSTCZC2CA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:01.777921879 (582h13m1.649004924s) [upload=510381]: Next returned nil segment Dec 29 21:42:01.777940274 (582h13m1.649023349s) [upload=510381]: Waiting for error group managing segments... Dec 29 21:42:02.141106696 (582h13m2.012189741s) [noise=true , address=64.49.13.140:4510 , piece_id=XFDDU2JYEYIWKN7EZWJZEP34QDUWEPL3OW2TR3JSDE2QY6CV3VDQ, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.376774281 (582h13m2.247857327s) [noise=true , address=145.79.227.39:4038 , piece_id=WK7OI2D5XAKGOM25ISG4IN6FPXIS2F6YDEOQFIQZPYWVSTCZC2CA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.379986568 (582h13m2.251069613s) [noise=true , address=145.79.227.21:4492 , piece_id=SYF3MNTT4KCEKJIWUVVMK4WS2PYZNYJNLG2HE6UW2F4DAHFHLTAQ, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.386102975 (582h13m2.25718602s) [noise=true , address=145.79.227.17:4974 , piece_id=ECY3XG222XZAVEEY3W4RFOXMUO7IMAXGMKA7F2MSM6WQJYQT6YZA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.390565783 (582h13m2.261648818s) [noise=true , address=145.79.227.4:4219 , piece_id=YVMXT3TQ5DFBGRF45A2FT4QVVVWJNLTUTFGRHRUXB2LDBLGCU7XA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.393984613 (582h13m2.265067658s) [noise=true , address=145.79.227.6:4266 , piece_id=QXMR4B6HMIOHPRFH3ABT3ZHSOF2ZE23SAGXIWUAEM5LO67NFNEFA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.524875288 (582h13m2.395958333s) [noise=true , address=77.90.31.13:4637 , piece_id=KZRLS4PBFYGGHYJRQLMBEOCNBYCQIMF3F32I3HZFTIUIAT6AE67A, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.540524625 (582h13m2.41160767s) [noise=true , address=77.90.31.13:4656 , piece_id=T3NU3UMWI7VTETXC2G5VCPH5YASFGCQMHOHB6UGQT2PB5ECA272A, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.557053169 (582h13m2.428136214s) [noise=true , address=77.90.31.16:4766 , piece_id=DBPOMKHGXGZQLUHK2NL5M26VRBOWAGZXOOBB2U3XUGHWDGLHEV6Q, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.641331988 (582h13m2.512415043s) [noise=true , address=77.90.31.19:4861 , piece_id=5ODRE74A5PTEPPYWCQS25K6HXMRPZKQIXMX6EOTQMB5XSGNN6TJA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:03.276156729 (582h13m3.147239814s) [noise=true , address=145.79.227.34:4936 , piece_id=WPHQ4WM222WNPSNQHJRHGWWOY3453GNWFU2VOWGME2NUVKGTTCLA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:03.285451126 (582h13m3.156534171s) [noise=true , address=77.90.31.15:4713 , piece_id=SPWJMKD6RJ555CL5PZ2Z7HNW2K67MC2U7QHG6NELHNRFQEB4OFKA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:03.293385374 (582h13m3.164468419s) [noise=true , address=145.79.227.44:4226 , piece_id=JSXESMXNV3MERJUKWNFKJDOWCTXIG6IQ7JMIXXW2JL2TMYBNBDLQ, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:03.293494552 (582h13m3.164577597s) [noise=true , address=145.79.227.49:4357 , piece_id=TP57SIC35AAYKPEF2WIQMJJE76C4ERVDRQGKF7462KW66AMCEFSA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:03.293514761 (582h13m3.164597796s) [seg_pos={0 0}, upload=510381]: Segment reached optimal threshold of 14 pieces. Dec 29 21:42:03.293900185 (582h13m3.16498322s) [noise=true , address=145.79.227.43:4184 , piece_id=ZYUY4IH7NV42CRLTWLA56ASUA3RBKE3HRNG33VEVR6YBSFRM5QSA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VYcRgiFADDQ1rLsiMufThN1K51cuoqKRXNjbp2w9Exu1d3gQG): protocol: expected piece hash; context canceled; EOF Dec 29 21:42:03.293912828 (582h13m3.164995873s) [noise=true , address=145.79.227.43:4184 , piece_id=ZYUY4IH7NV42CRLTWLA56ASUA3RBKE3HRNG33VEVR6YBSFRM5QSA, seg_pos={0 0}, upload=510381]: Long tail cancellation due to optimal threshold Dec 29 21:42:03.293954789 (582h13m3.165037834s) [noise=true , address=145.79.227.29:4946 , piece_id=DGZAZPHNELKHOGDMC6GFKSO4QVCIFGQHXSH6VJJZUJRHPXLWSUPA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12v9Q7aigjWPNXyshHmQt9YKrDu5af1vd1TxkDeHeKixSrWgrLX): protocol: expected piece hash; context canceled; EOF Dec 29 21:42:03.293966481 (582h13m3.165049546s) [noise=true , address=145.79.227.29:4946 , piece_id=DGZAZPHNELKHOGDMC6GFKSO4QVCIFGQHXSH6VJJZUJRHPXLWSUPA, seg_pos={0 0}, upload=510381]: Long tail cancellation due to optimal threshold Dec 29 21:42:03.294146864 (582h13m3.165229899s) [noise=true , address=145.79.227.33:4891 , piece_id=MRIXCIGX6WVNEVCXPLWCWZ67JVIA5JDZLHCT6RE4NCMAFANMVVEQ, seg_pos={0 0}, upload=510381]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RHLM4MJmtKoHsUnsqccy1ehyD7zVSn6gnJNp7yJJmZhTJqPkU): protocol: expected piece hash; context canceled; EOF Dec 29 21:42:03.294155210 (582h13m3.165238245s) [noise=true , address=145.79.227.33:4891 , piece_id=MRIXCIGX6WVNEVCXPLWCWZ67JVIA5JDZLHCT6RE4NCMAFANMVVEQ, seg_pos={0 0}, upload=510381]: Long tail cancellation due to optimal threshold Dec 29 21:42:03.294191519 (582h13m3.165274574s) [noise=true , address=145.79.227.40:4053 , piece_id=SBBX53GUBMB57OJBNXJMCVS3KYD4ZQ5LUCV572LKJUNYDF6MIJ7A, seg_pos={0 0}, upload=510381]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DqyTc4MHvxutFXs2WWRCk8MSuxF3Y1HayyQY4Y5sy8cqDLWuG): protocol: expected piece hash; context canceled; EOF Dec 29 21:42:03.294202380 (582h13m3.165285425s) [noise=true , address=145.79.227.40:4053 , piece_id=SBBX53GUBMB57OJBNXJMCVS3KYD4ZQ5LUCV572LKJUNYDF6MIJ7A, seg_pos={0 0}, upload=510381]: Long tail cancellation due to optimal threshold Dec 29 21:42:03.294227728 (582h13m3.165310763s) [seg_pos={0 0}, upload=510381]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 21:42:03.294328840 (582h13m3.165411886s) [upload=510381]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 21:42:03.294336927 (582h13m3.165419972s) [upload=510381]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 21:42:03.627673008 (582h13m3.498756053s) [upload=510381]: Done uploading segments... Dec 29 21:42:03.627701772 (582h13m3.498784807s) [upload=510381]: Upload finished. err: <nil> Duration: 6.538966607s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 15.337310519s trace: https://jaeger.dev.storj.io/trace/54393681021e4741 FAIL (2025-12-29 22:04:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== upload=510381]: Uploading piece... Dec 29 21:42:00.490334342 (582h13m0.361417387s) [noise=true , address=77.90.31.13:4637 , piece_id=KZRLS4PBFYGGHYJRQLMBEOCNBYCQIMF3F32I3HZFTIUIAT6AE67A, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.490753791 (582h13m0.361836826s) [noise=true , address=77.90.31.19:4861 , piece_id=5ODRE74A5PTEPPYWCQS25K6HXMRPZKQIXMX6EOTQMB5XSGNN6TJA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.491447442 (582h13m0.362530487s) [noise=true , address=145.79.227.49:4357 , piece_id=TP57SIC35AAYKPEF2WIQMJJE76C4ERVDRQGKF7462KW66AMCEFSA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.491940861 (582h13m0.363023896s) [noise=true , address=145.79.227.33:4891 , piece_id=MRIXCIGX6WVNEVCXPLWCWZ67JVIA5JDZLHCT6RE4NCMAFANMVVEQ, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.492346664 (582h13m0.363429699s) [noise=true , address=77.90.31.15:4713 , piece_id=SPWJMKD6RJ555CL5PZ2Z7HNW2K67MC2U7QHG6NELHNRFQEB4OFKA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.492751746 (582h13m0.363834791s) [noise=true , address=77.90.31.13:4656 , piece_id=T3NU3UMWI7VTETXC2G5VCPH5YASFGCQMHOHB6UGQT2PB5ECA272A, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.493112152 (582h13m0.364195187s) [noise=true , address=64.49.13.140:4510 , piece_id=XFDDU2JYEYIWKN7EZWJZEP34QDUWEPL3OW2TR3JSDE2QY6CV3VDQ, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.493462459 (582h13m0.364545504s) [noise=true , address=145.79.227.29:4946 , piece_id=DGZAZPHNELKHOGDMC6GFKSO4QVCIFGQHXSH6VJJZUJRHPXLWSUPA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.493859384 (582h13m0.36494242s) [noise=true , address=145.79.227.44:4226 , piece_id=JSXESMXNV3MERJUKWNFKJDOWCTXIG6IQ7JMIXXW2JL2TMYBNBDLQ, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.494272812 (582h13m0.365355847s) [noise=true , address=77.90.31.16:4766 , piece_id=DBPOMKHGXGZQLUHK2NL5M26VRBOWAGZXOOBB2U3XUGHWDGLHEV6Q, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.495004285 (582h13m0.36608733s) [noise=true , address=145.79.227.4:4219 , piece_id=YVMXT3TQ5DFBGRF45A2FT4QVVVWJNLTUTFGRHRUXB2LDBLGCU7XA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.495625678 (582h13m0.366708723s) [noise=true , address=145.79.227.43:4184 , piece_id=ZYUY4IH7NV42CRLTWLA56ASUA3RBKE3HRNG33VEVR6YBSFRM5QSA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.496138595 (582h13m0.36722163s) [noise=true , address=145.79.227.40:4053 , piece_id=SBBX53GUBMB57OJBNXJMCVS3KYD4ZQ5LUCV572LKJUNYDF6MIJ7A, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.496524049 (582h13m0.367607094s) [noise=true , address=145.79.227.17:4974 , piece_id=ECY3XG222XZAVEEY3W4RFOXMUO7IMAXGMKA7F2MSM6WQJYQT6YZA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.496920243 (582h13m0.368003288s) [noise=true , address=145.79.227.34:4936 , piece_id=WPHQ4WM222WNPSNQHJRHGWWOY3453GNWFU2VOWGME2NUVKGTTCLA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.497333270 (582h13m0.368416305s) [noise=true , address=145.79.227.21:4492 , piece_id=SYF3MNTT4KCEKJIWUVVMK4WS2PYZNYJNLG2HE6UW2F4DAHFHLTAQ, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.497852078 (582h13m0.368935113s) [noise=true , address=145.79.227.39:4038 , piece_id=WK7OI2D5XAKGOM25ISG4IN6FPXIS2F6YDEOQFIQZPYWVSTCZC2CA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:01.777921879 (582h13m1.649004924s) [upload=510381]: Next returned nil segment Dec 29 21:42:01.777940274 (582h13m1.649023349s) [upload=510381]: Waiting for error group managing segments... Dec 29 21:42:02.141106696 (582h13m2.012189741s) [noise=true , address=64.49.13.140:4510 , piece_id=XFDDU2JYEYIWKN7EZWJZEP34QDUWEPL3OW2TR3JSDE2QY6CV3VDQ, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.376774281 (582h13m2.247857327s) [noise=true , address=145.79.227.39:4038 , piece_id=WK7OI2D5XAKGOM25ISG4IN6FPXIS2F6YDEOQFIQZPYWVSTCZC2CA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.379986568 (582h13m2.251069613s) [noise=true , address=145.79.227.21:4492 , piece_id=SYF3MNTT4KCEKJIWUVVMK4WS2PYZNYJNLG2HE6UW2F4DAHFHLTAQ, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.386102975 (582h13m2.25718602s) [noise=true , address=145.79.227.17:4974 , piece_id=ECY3XG222XZAVEEY3W4RFOXMUO7IMAXGMKA7F2MSM6WQJYQT6YZA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.390565783 (582h13m2.261648818s) [noise=true , address=145.79.227.4:4219 , piece_id=YVMXT3TQ5DFBGRF45A2FT4QVVVWJNLTUTFGRHRUXB2LDBLGCU7XA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.393984613 (582h13m2.265067658s) [noise=true , address=145.79.227.6:4266 , piece_id=QXMR4B6HMIOHPRFH3ABT3ZHSOF2ZE23SAGXIWUAEM5LO67NFNEFA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.524875288 (582h13m2.395958333s) [noise=true , address=77.90.31.13:4637 , piece_id=KZRLS4PBFYGGHYJRQLMBEOCNBYCQIMF3F32I3HZFTIUIAT6AE67A, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.540524625 (582h13m2.41160767s) [noise=true , address=77.90.31.13:4656 , piece_id=T3NU3UMWI7VTETXC2G5VCPH5YASFGCQMHOHB6UGQT2PB5ECA272A, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.557053169 (582h13m2.428136214s) [noise=true , address=77.90.31.16:4766 , piece_id=DBPOMKHGXGZQLUHK2NL5M26VRBOWAGZXOOBB2U3XUGHWDGLHEV6Q, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.641331988 (582h13m2.512415043s) [noise=true , address=77.90.31.19:4861 , piece_id=5ODRE74A5PTEPPYWCQS25K6HXMRPZKQIXMX6EOTQMB5XSGNN6TJA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:03.276156729 (582h13m3.147239814s) [noise=true , address=145.79.227.34:4936 , piece_id=WPHQ4WM222WNPSNQHJRHGWWOY3453GNWFU2VOWGME2NUVKGTTCLA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:03.285451126 (582h13m3.156534171s) [noise=true , address=77.90.31.15:4713 , piece_id=SPWJMKD6RJ555CL5PZ2Z7HNW2K67MC2U7QHG6NELHNRFQEB4OFKA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:03.293385374 (582h13m3.164468419s) [noise=true , address=145.79.227.44:4226 , piece_id=JSXESMXNV3MERJUKWNFKJDOWCTXIG6IQ7JMIXXW2JL2TMYBNBDLQ, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:03.293494552 (582h13m3.164577597s) [noise=true , address=145.79.227.49:4357 , piece_id=TP57SIC35AAYKPEF2WIQMJJE76C4ERVDRQGKF7462KW66AMCEFSA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:03.293514761 (582h13m3.164597796s) [seg_pos={0 0}, upload=510381]: Segment reached optimal threshold of 14 pieces. Dec 29 21:42:03.293900185 (582h13m3.16498322s) [noise=true , address=145.79.227.43:4184 , piece_id=ZYUY4IH7NV42CRLTWLA56ASUA3RBKE3HRNG33VEVR6YBSFRM5QSA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VYcRgiFADDQ1rLsiMufThN1K51cuoqKRXNjbp2w9Exu1d3gQG): protocol: expected piece hash; context canceled; EOF Dec 29 21:42:03.293912828 (582h13m3.164995873s) [noise=true , address=145.79.227.43:4184 , piece_id=ZYUY4IH7NV42CRLTWLA56ASUA3RBKE3HRNG33VEVR6YBSFRM5QSA, seg_pos={0 0}, upload=510381]: Long tail cancellation due to optimal threshold Dec 29 21:42:03.293954789 (582h13m3.165037834s) [noise=true , address=145.79.227.29:4946 , piece_id=DGZAZPHNELKHOGDMC6GFKSO4QVCIFGQHXSH6VJJZUJRHPXLWSUPA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12v9Q7aigjWPNXyshHmQt9YKrDu5af1vd1TxkDeHeKixSrWgrLX): protocol: expected piece hash; context canceled; EOF Dec 29 21:42:03.293966481 (582h13m3.165049546s) [noise=true , address=145.79.227.29:4946 , piece_id=DGZAZPHNELKHOGDMC6GFKSO4QVCIFGQHXSH6VJJZUJRHPXLWSUPA, seg_pos={0 0}, upload=510381]: Long tail cancellation due to optimal threshold Dec 29 21:42:03.294146864 (582h13m3.165229899s) [noise=true , address=145.79.227.33:4891 , piece_id=MRIXCIGX6WVNEVCXPLWCWZ67JVIA5JDZLHCT6RE4NCMAFANMVVEQ, seg_pos={0 0}, upload=510381]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RHLM4MJmtKoHsUnsqccy1ehyD7zVSn6gnJNp7yJJmZhTJqPkU): protocol: expected piece hash; context canceled; EOF Dec 29 21:42:03.294155210 (582h13m3.165238245s) [noise=true , address=145.79.227.33:4891 , piece_id=MRIXCIGX6WVNEVCXPLWCWZ67JVIA5JDZLHCT6RE4NCMAFANMVVEQ, seg_pos={0 0}, upload=510381]: Long tail cancellation due to optimal threshold Dec 29 21:42:03.294191519 (582h13m3.165274574s) [noise=true , address=145.79.227.40:4053 , piece_id=SBBX53GUBMB57OJBNXJMCVS3KYD4ZQ5LUCV572LKJUNYDF6MIJ7A, seg_pos={0 0}, upload=510381]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DqyTc4MHvxutFXs2WWRCk8MSuxF3Y1HayyQY4Y5sy8cqDLWuG): protocol: expected piece hash; context canceled; EOF Dec 29 21:42:03.294202380 (582h13m3.165285425s) [noise=true , address=145.79.227.40:4053 , piece_id=SBBX53GUBMB57OJBNXJMCVS3KYD4ZQ5LUCV572LKJUNYDF6MIJ7A, seg_pos={0 0}, upload=510381]: Long tail cancellation due to optimal threshold Dec 29 21:42:03.294227728 (582h13m3.165310763s) [seg_pos={0 0}, upload=510381]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 21:42:03.294328840 (582h13m3.165411886s) [upload=510381]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 21:42:03.294336927 (582h13m3.165419972s) [upload=510381]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 21:42:03.627673008 (582h13m3.498756053s) [upload=510381]: Done uploading segments... Dec 29 21:42:03.627701772 (582h13m3.498784807s) [upload=510381]: Upload finished. err: <nil> Duration: 15.337333574s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..........X..X..X........X............................X.....] duration: 5.999060585s trace: https://jaeger.dev.storj.io/trace/5209b3d035311fd FAIL (2025-12-29 22:04:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :00:17.043136250 (582h31m16.914219305s) [upload=510642]: Starting upload Dec 29 22:00:17.043227904 (582h31m16.91431096s) [upload=510642]: Done starting upload Dec 29 22:00:17.043335189 (582h31m16.914418234s) [upload=510642]: Uploading segments... Dec 29 22:00:17.043587749 (582h31m16.914670824s) [upload=510642]: Got next segment. Inline: false Dec 29 22:00:17.043644478 (582h31m16.914727523s) [upload=510642]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 22:00:17.043686427 (582h31m16.914769503s) [upload=510642]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 22:00:17.294384363 (582h31m17.165467408s) [seg_pos={0 0}, upload=510642]: Begin upload segment... Dec 29 22:00:17.294445219 (582h31m17.165528264s) [seg_pos={0 0}, upload=510642]: Done begin upload segment. Dec 29 22:00:17.294484133 (582h31m17.165567178s) [noise=true , address=145.79.227.31:4797 , piece_id=NI3M5NVVVX5HUXZ2SOMQD4S5ZOSSNPCD2F3Z5X2BA2VXMGT57AYQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.294560979 (582h31m17.165644014s) [noise=true , address=77.90.31.11:4553 , piece_id=3Y5HSD73DLZ6ALJ5TFRIVTLOE4ZNZ256YSZ2W3X5LPKL5J5LX2EQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.295149941 (582h31m17.166232976s) [noise=true , address=45.91.116.5:4297 , piece_id=PLUQ7Q32BTGYFQRGVJFAJWHPZCPQTC5RX5UX27S5HJ5NNJPLDMPQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.295419574 (582h31m17.166502609s) [noise=true , address=145.79.227.2:4115 , piece_id=P2AS5B3VYO2K2AWSM5W4BXAXBHSXQQWM7JOXFPOV64NTKDFLT62Q, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.295483997 (582h31m17.166567042s) [noise=true , address=145.79.227.51:4411 , piece_id=HTZQ6SW6KSACLQF7G4CFGQA572J5JVHMLGEWQYSSGZRYX7FJ6DXA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.295762207 (582h31m17.166845252s) [noise=true , address=145.79.227.5:4221 , piece_id=LD3FENLBM2L5N7QDOAMAYJUVH54QVSJQJFP7E36UAH2ZD7R65CCQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.295928003 (582h31m17.167011048s) [noise=true , address=145.79.227.16:4909 , piece_id=CF7WHNKC4LFJQRVFBPFV62JGEBLD4KVPE6Q5JBWQVADC6ZAFFLVQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.296039615 (582h31m17.16712266s) [noise=true , address=45.91.116.13:4154 , piece_id=FE4NFODQ3TWGFXARFYIL64TKK6X7H6PPLIG6LO44LKNAZOEU7DJA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.296272429 (582h31m17.167355474s) [noise=true , address=45.91.116.10:4084 , piece_id=5BFUYVGIFOYXC6RCPU7VBUM36LSM32TVBLS3W54OWFNR4SM6GCSA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.296403588 (582h31m17.167486633s) [noise=true , address=77.90.31.19:4844 , piece_id=CDOWECXLSRPACV4HMAH57UK4EEIYNOMH6GYV6HFHV7CWSEAJ44FA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.296646560 (582h31m17.167729596s) [noise=true , address=77.90.31.14:4661 , piece_id=B7IDSCDHBUXUIPSUT3HMQSCMPGBMYASZSQO2EJBXRR2DQHCLWGYA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.296906235 (582h31m17.167989291s) [noise=true , address=145.79.227.47:4265 , piece_id=FYZA2LRIFKLKVUCUAWMNWTUH3VYR5VFAA7XKAYIMD2E2VOJTI7PA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.296989804 (582h31m17.168072849s) [noise=true , address=145.79.227.46:4601 , piece_id=YJTMZID24VBM5ZZSUUHOCZSZSSNGTFHOSKK6OI5QP3A4G6ZWVWQA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.297235221 (582h31m17.168318267s) [noise=true , address=145.79.227.3:4150 , piece_id=HS4S5UDYD2VWI2N4MH6EWEYQ5G6KCFCJBDZR7AVWPXRYALFXP4FQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.297424432 (582h31m17.168507467s) [noise=true , address=145.79.227.13:4803 , piece_id=YIEUZEDLSC5C67SCFEPCHHZSTYP6MYLYJZJTGZF3UP6JWDTGUOPQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.297606900 (582h31m17.168689945s) [noise=true , address=45.91.116.14:4190 , piece_id=FIL6AIXGE7DE2M4QBVSIO53KGY43VFPZQ2FTXSWJFPJU4VZCKUXQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:18.995364155 (582h31m18.866447201s) [upload=510642]: Next returned nil segment Dec 29 22:00:18.995383603 (582h31m18.866466648s) [upload=510642]: Waiting for error group managing segments... Dec 29 22:00:19.284009408 (582h31m19.155092453s) [noise=true , address=145.79.227.51:4411 , piece_id=HTZQ6SW6KSACLQF7G4CFGQA572J5JVHMLGEWQYSSGZRYX7FJ6DXA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.295985232 (582h31m19.167068277s) [noise=true , address=145.79.227.13:4803 , piece_id=YIEUZEDLSC5C67SCFEPCHHZSTYP6MYLYJZJTGZF3UP6JWDTGUOPQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.296135458 (582h31m19.167218503s) [noise=true , address=145.79.227.16:4909 , piece_id=CF7WHNKC4LFJQRVFBPFV62JGEBLD4KVPE6Q5JBWQVADC6ZAFFLVQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.310119916 (582h31m19.181202971s) [noise=true , address=145.79.227.46:4601 , piece_id=YJTMZID24VBM5ZZSUUHOCZSZSSNGTFHOSKK6OI5QP3A4G6ZWVWQA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.310998939 (582h31m19.182081984s) [noise=true , address=145.79.227.31:4797 , piece_id=NI3M5NVVVX5HUXZ2SOMQD4S5ZOSSNPCD2F3Z5X2BA2VXMGT57AYQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.318707138 (582h31m19.189790183s) [noise=true , address=145.79.227.47:4265 , piece_id=FYZA2LRIFKLKVUCUAWMNWTUH3VYR5VFAA7XKAYIMD2E2VOJTI7PA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.322059381 (582h31m19.193142426s) [noise=true , address=145.79.227.5:4221 , piece_id=LD3FENLBM2L5N7QDOAMAYJUVH54QVSJQJFP7E36UAH2ZD7R65CCQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.506372108 (582h31m19.377455153s) [noise=true , address=77.90.31.19:4844 , piece_id=CDOWECXLSRPACV4HMAH57UK4EEIYNOMH6GYV6HFHV7CWSEAJ44FA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.507172602 (582h31m19.378255648s) [noise=true , address=77.90.31.11:4553 , piece_id=3Y5HSD73DLZ6ALJ5TFRIVTLOE4ZNZ256YSZ2W3X5LPKL5J5LX2EQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.527548842 (582h31m19.398631887s) [noise=true , address=77.90.31.14:4661 , piece_id=B7IDSCDHBUXUIPSUT3HMQSCMPGBMYASZSQO2EJBXRR2DQHCLWGYA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.531443508 (582h31m19.402526553s) [noise=true , address=45.91.116.5:4297 , piece_id=PLUQ7Q32BTGYFQRGVJFAJWHPZCPQTC5RX5UX27S5HJ5NNJPLDMPQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.552364906 (582h31m19.423447951s) [noise=true , address=45.91.116.14:4190 , piece_id=FIL6AIXGE7DE2M4QBVSIO53KGY43VFPZQ2FTXSWJFPJU4VZCKUXQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.564056769 (582h31m19.435139814s) [noise=true , address=45.91.116.10:4084 , piece_id=5BFUYVGIFOYXC6RCPU7VBUM36LSM32TVBLS3W54OWFNR4SM6GCSA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.564087016 (582h31m19.435170051s) [seg_pos={0 0}, upload=510642]: Segment reached optimal threshold of 13 pieces. Dec 29 22:00:19.564294431 (582h31m19.435377466s) [noise=true , address=145.79.227.3:4150 , piece_id=HS4S5UDYD2VWI2N4MH6EWEYQ5G6KCFCJBDZR7AVWPXRYALFXP4FQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK): protocol: expected piece hash; context canceled; EOF Dec 29 22:00:19.564302366 (582h31m19.435385411s) [noise=true , address=145.79.227.3:4150 , piece_id=HS4S5UDYD2VWI2N4MH6EWEYQ5G6KCFCJBDZR7AVWPXRYALFXP4FQ, seg_pos={0 0}, upload=510642]: Long tail cancellation due to optimal threshold Dec 29 22:00:19.564388932 (582h31m19.435471977s) [noise=true , address=45.91.116.13:4154 , piece_id=FE4NFODQ3TWGFXARFYIL64TKK6X7H6PPLIG6LO44LKNAZOEU7DJA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Dec 29 22:00:19.564401876 (582h31m19.435484911s) [noise=true , address=45.91.116.13:4154 , piece_id=FE4NFODQ3TWGFXARFYIL64TKK6X7H6PPLIG6LO44LKNAZOEU7DJA, seg_pos={0 0}, upload=510642]: Long tail cancellation due to optimal threshold Dec 29 22:00:19.564558945 (582h31m19.435641981s) [noise=true , address=145.79.227.2:4115 , piece_id=P2AS5B3VYO2K2AWSM5W4BXAXBHSXQQWM7JOXFPOV64NTKDFLT62Q, seg_pos={0 0}, upload=510642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: write tcp 188.166.225.170:38246->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:38246->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38246->145.79.227.2:4115: use of closed network connection Dec 29 22:00:19.564570738 (582h31m19.435653783s) [noise=true , address=145.79.227.2:4115 , piece_id=P2AS5B3VYO2K2AWSM5W4BXAXBHSXQQWM7JOXFPOV64NTKDFLT62Q, seg_pos={0 0}, upload=510642]: Long tail cancellation due to optimal threshold Dec 29 22:00:19.564601426 (582h31m19.435684471s) [seg_pos={0 0}, upload=510642]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 22:00:19.564683973 (582h31m19.435767018s) [upload=510642]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 22:00:19.564690004 (582h31m19.435773049s) [upload=510642]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 22:00:19.871860472 (582h31m19.742943527s) [upload=510642]: Done uploading segments... Dec 29 22:00:19.871892413 (582h31m19.742975468s) [upload=510642]: Upload finished. err: <nil> Duration: 5.999087847s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...........X..X..X........X............................X....] duration: 16.123705135s trace: https://jaeger.dev.storj.io/trace/5209b3d035311fd FAIL (2025-12-29 22:04:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :00:17.043136250 (582h31m16.914219305s) [upload=510642]: Starting upload Dec 29 22:00:17.043227904 (582h31m16.91431096s) [upload=510642]: Done starting upload Dec 29 22:00:17.043335189 (582h31m16.914418234s) [upload=510642]: Uploading segments... Dec 29 22:00:17.043587749 (582h31m16.914670824s) [upload=510642]: Got next segment. Inline: false Dec 29 22:00:17.043644478 (582h31m16.914727523s) [upload=510642]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 22:00:17.043686427 (582h31m16.914769503s) [upload=510642]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 22:00:17.294384363 (582h31m17.165467408s) [seg_pos={0 0}, upload=510642]: Begin upload segment... Dec 29 22:00:17.294445219 (582h31m17.165528264s) [seg_pos={0 0}, upload=510642]: Done begin upload segment. Dec 29 22:00:17.294484133 (582h31m17.165567178s) [noise=true , address=145.79.227.31:4797 , piece_id=NI3M5NVVVX5HUXZ2SOMQD4S5ZOSSNPCD2F3Z5X2BA2VXMGT57AYQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.294560979 (582h31m17.165644014s) [noise=true , address=77.90.31.11:4553 , piece_id=3Y5HSD73DLZ6ALJ5TFRIVTLOE4ZNZ256YSZ2W3X5LPKL5J5LX2EQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.295149941 (582h31m17.166232976s) [noise=true , address=45.91.116.5:4297 , piece_id=PLUQ7Q32BTGYFQRGVJFAJWHPZCPQTC5RX5UX27S5HJ5NNJPLDMPQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.295419574 (582h31m17.166502609s) [noise=true , address=145.79.227.2:4115 , piece_id=P2AS5B3VYO2K2AWSM5W4BXAXBHSXQQWM7JOXFPOV64NTKDFLT62Q, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.295483997 (582h31m17.166567042s) [noise=true , address=145.79.227.51:4411 , piece_id=HTZQ6SW6KSACLQF7G4CFGQA572J5JVHMLGEWQYSSGZRYX7FJ6DXA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.295762207 (582h31m17.166845252s) [noise=true , address=145.79.227.5:4221 , piece_id=LD3FENLBM2L5N7QDOAMAYJUVH54QVSJQJFP7E36UAH2ZD7R65CCQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.295928003 (582h31m17.167011048s) [noise=true , address=145.79.227.16:4909 , piece_id=CF7WHNKC4LFJQRVFBPFV62JGEBLD4KVPE6Q5JBWQVADC6ZAFFLVQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.296039615 (582h31m17.16712266s) [noise=true , address=45.91.116.13:4154 , piece_id=FE4NFODQ3TWGFXARFYIL64TKK6X7H6PPLIG6LO44LKNAZOEU7DJA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.296272429 (582h31m17.167355474s) [noise=true , address=45.91.116.10:4084 , piece_id=5BFUYVGIFOYXC6RCPU7VBUM36LSM32TVBLS3W54OWFNR4SM6GCSA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.296403588 (582h31m17.167486633s) [noise=true , address=77.90.31.19:4844 , piece_id=CDOWECXLSRPACV4HMAH57UK4EEIYNOMH6GYV6HFHV7CWSEAJ44FA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.296646560 (582h31m17.167729596s) [noise=true , address=77.90.31.14:4661 , piece_id=B7IDSCDHBUXUIPSUT3HMQSCMPGBMYASZSQO2EJBXRR2DQHCLWGYA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.296906235 (582h31m17.167989291s) [noise=true , address=145.79.227.47:4265 , piece_id=FYZA2LRIFKLKVUCUAWMNWTUH3VYR5VFAA7XKAYIMD2E2VOJTI7PA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.296989804 (582h31m17.168072849s) [noise=true , address=145.79.227.46:4601 , piece_id=YJTMZID24VBM5ZZSUUHOCZSZSSNGTFHOSKK6OI5QP3A4G6ZWVWQA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.297235221 (582h31m17.168318267s) [noise=true , address=145.79.227.3:4150 , piece_id=HS4S5UDYD2VWI2N4MH6EWEYQ5G6KCFCJBDZR7AVWPXRYALFXP4FQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.297424432 (582h31m17.168507467s) [noise=true , address=145.79.227.13:4803 , piece_id=YIEUZEDLSC5C67SCFEPCHHZSTYP6MYLYJZJTGZF3UP6JWDTGUOPQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.297606900 (582h31m17.168689945s) [noise=true , address=45.91.116.14:4190 , piece_id=FIL6AIXGE7DE2M4QBVSIO53KGY43VFPZQ2FTXSWJFPJU4VZCKUXQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:18.995364155 (582h31m18.866447201s) [upload=510642]: Next returned nil segment Dec 29 22:00:18.995383603 (582h31m18.866466648s) [upload=510642]: Waiting for error group managing segments... Dec 29 22:00:19.284009408 (582h31m19.155092453s) [noise=true , address=145.79.227.51:4411 , piece_id=HTZQ6SW6KSACLQF7G4CFGQA572J5JVHMLGEWQYSSGZRYX7FJ6DXA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.295985232 (582h31m19.167068277s) [noise=true , address=145.79.227.13:4803 , piece_id=YIEUZEDLSC5C67SCFEPCHHZSTYP6MYLYJZJTGZF3UP6JWDTGUOPQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.296135458 (582h31m19.167218503s) [noise=true , address=145.79.227.16:4909 , piece_id=CF7WHNKC4LFJQRVFBPFV62JGEBLD4KVPE6Q5JBWQVADC6ZAFFLVQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.310119916 (582h31m19.181202971s) [noise=true , address=145.79.227.46:4601 , piece_id=YJTMZID24VBM5ZZSUUHOCZSZSSNGTFHOSKK6OI5QP3A4G6ZWVWQA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.310998939 (582h31m19.182081984s) [noise=true , address=145.79.227.31:4797 , piece_id=NI3M5NVVVX5HUXZ2SOMQD4S5ZOSSNPCD2F3Z5X2BA2VXMGT57AYQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.318707138 (582h31m19.189790183s) [noise=true , address=145.79.227.47:4265 , piece_id=FYZA2LRIFKLKVUCUAWMNWTUH3VYR5VFAA7XKAYIMD2E2VOJTI7PA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.322059381 (582h31m19.193142426s) [noise=true , address=145.79.227.5:4221 , piece_id=LD3FENLBM2L5N7QDOAMAYJUVH54QVSJQJFP7E36UAH2ZD7R65CCQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.506372108 (582h31m19.377455153s) [noise=true , address=77.90.31.19:4844 , piece_id=CDOWECXLSRPACV4HMAH57UK4EEIYNOMH6GYV6HFHV7CWSEAJ44FA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.507172602 (582h31m19.378255648s) [noise=true , address=77.90.31.11:4553 , piece_id=3Y5HSD73DLZ6ALJ5TFRIVTLOE4ZNZ256YSZ2W3X5LPKL5J5LX2EQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.527548842 (582h31m19.398631887s) [noise=true , address=77.90.31.14:4661 , piece_id=B7IDSCDHBUXUIPSUT3HMQSCMPGBMYASZSQO2EJBXRR2DQHCLWGYA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.531443508 (582h31m19.402526553s) [noise=true , address=45.91.116.5:4297 , piece_id=PLUQ7Q32BTGYFQRGVJFAJWHPZCPQTC5RX5UX27S5HJ5NNJPLDMPQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.552364906 (582h31m19.423447951s) [noise=true , address=45.91.116.14:4190 , piece_id=FIL6AIXGE7DE2M4QBVSIO53KGY43VFPZQ2FTXSWJFPJU4VZCKUXQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.564056769 (582h31m19.435139814s) [noise=true , address=45.91.116.10:4084 , piece_id=5BFUYVGIFOYXC6RCPU7VBUM36LSM32TVBLS3W54OWFNR4SM6GCSA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.564087016 (582h31m19.435170051s) [seg_pos={0 0}, upload=510642]: Segment reached optimal threshold of 13 pieces. Dec 29 22:00:19.564294431 (582h31m19.435377466s) [noise=true , address=145.79.227.3:4150 , piece_id=HS4S5UDYD2VWI2N4MH6EWEYQ5G6KCFCJBDZR7AVWPXRYALFXP4FQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK): protocol: expected piece hash; context canceled; EOF Dec 29 22:00:19.564302366 (582h31m19.435385411s) [noise=true , address=145.79.227.3:4150 , piece_id=HS4S5UDYD2VWI2N4MH6EWEYQ5G6KCFCJBDZR7AVWPXRYALFXP4FQ, seg_pos={0 0}, upload=510642]: Long tail cancellation due to optimal threshold Dec 29 22:00:19.564388932 (582h31m19.435471977s) [noise=true , address=45.91.116.13:4154 , piece_id=FE4NFODQ3TWGFXARFYIL64TKK6X7H6PPLIG6LO44LKNAZOEU7DJA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Dec 29 22:00:19.564401876 (582h31m19.435484911s) [noise=true , address=45.91.116.13:4154 , piece_id=FE4NFODQ3TWGFXARFYIL64TKK6X7H6PPLIG6LO44LKNAZOEU7DJA, seg_pos={0 0}, upload=510642]: Long tail cancellation due to optimal threshold Dec 29 22:00:19.564558945 (582h31m19.435641981s) [noise=true , address=145.79.227.2:4115 , piece_id=P2AS5B3VYO2K2AWSM5W4BXAXBHSXQQWM7JOXFPOV64NTKDFLT62Q, seg_pos={0 0}, upload=510642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: write tcp 188.166.225.170:38246->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:38246->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38246->145.79.227.2:4115: use of closed network connection Dec 29 22:00:19.564570738 (582h31m19.435653783s) [noise=true , address=145.79.227.2:4115 , piece_id=P2AS5B3VYO2K2AWSM5W4BXAXBHSXQQWM7JOXFPOV64NTKDFLT62Q, seg_pos={0 0}, upload=510642]: Long tail cancellation due to optimal threshold Dec 29 22:00:19.564601426 (582h31m19.435684471s) [seg_pos={0 0}, upload=510642]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 22:00:19.564683973 (582h31m19.435767018s) [upload=510642]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 22:00:19.564690004 (582h31m19.435773049s) [upload=510642]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 22:00:19.871860472 (582h31m19.742943527s) [upload=510642]: Done uploading segments... Dec 29 22:00:19.871892413 (582h31m19.742975468s) [upload=510642]: Upload finished. err: <nil> Duration: 16.123726387s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 18.548169694s trace: https://jaeger.dev.storj.io/trace/54393681021e4741 FAIL (2025-12-29 22:03:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== upload=510381]: Uploading piece... Dec 29 21:42:00.490334342 (582h13m0.361417387s) [noise=true , address=77.90.31.13:4637 , piece_id=KZRLS4PBFYGGHYJRQLMBEOCNBYCQIMF3F32I3HZFTIUIAT6AE67A, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.490753791 (582h13m0.361836826s) [noise=true , address=77.90.31.19:4861 , piece_id=5ODRE74A5PTEPPYWCQS25K6HXMRPZKQIXMX6EOTQMB5XSGNN6TJA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.491447442 (582h13m0.362530487s) [noise=true , address=145.79.227.49:4357 , piece_id=TP57SIC35AAYKPEF2WIQMJJE76C4ERVDRQGKF7462KW66AMCEFSA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.491940861 (582h13m0.363023896s) [noise=true , address=145.79.227.33:4891 , piece_id=MRIXCIGX6WVNEVCXPLWCWZ67JVIA5JDZLHCT6RE4NCMAFANMVVEQ, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.492346664 (582h13m0.363429699s) [noise=true , address=77.90.31.15:4713 , piece_id=SPWJMKD6RJ555CL5PZ2Z7HNW2K67MC2U7QHG6NELHNRFQEB4OFKA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.492751746 (582h13m0.363834791s) [noise=true , address=77.90.31.13:4656 , piece_id=T3NU3UMWI7VTETXC2G5VCPH5YASFGCQMHOHB6UGQT2PB5ECA272A, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.493112152 (582h13m0.364195187s) [noise=true , address=64.49.13.140:4510 , piece_id=XFDDU2JYEYIWKN7EZWJZEP34QDUWEPL3OW2TR3JSDE2QY6CV3VDQ, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.493462459 (582h13m0.364545504s) [noise=true , address=145.79.227.29:4946 , piece_id=DGZAZPHNELKHOGDMC6GFKSO4QVCIFGQHXSH6VJJZUJRHPXLWSUPA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.493859384 (582h13m0.36494242s) [noise=true , address=145.79.227.44:4226 , piece_id=JSXESMXNV3MERJUKWNFKJDOWCTXIG6IQ7JMIXXW2JL2TMYBNBDLQ, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.494272812 (582h13m0.365355847s) [noise=true , address=77.90.31.16:4766 , piece_id=DBPOMKHGXGZQLUHK2NL5M26VRBOWAGZXOOBB2U3XUGHWDGLHEV6Q, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.495004285 (582h13m0.36608733s) [noise=true , address=145.79.227.4:4219 , piece_id=YVMXT3TQ5DFBGRF45A2FT4QVVVWJNLTUTFGRHRUXB2LDBLGCU7XA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.495625678 (582h13m0.366708723s) [noise=true , address=145.79.227.43:4184 , piece_id=ZYUY4IH7NV42CRLTWLA56ASUA3RBKE3HRNG33VEVR6YBSFRM5QSA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.496138595 (582h13m0.36722163s) [noise=true , address=145.79.227.40:4053 , piece_id=SBBX53GUBMB57OJBNXJMCVS3KYD4ZQ5LUCV572LKJUNYDF6MIJ7A, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.496524049 (582h13m0.367607094s) [noise=true , address=145.79.227.17:4974 , piece_id=ECY3XG222XZAVEEY3W4RFOXMUO7IMAXGMKA7F2MSM6WQJYQT6YZA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.496920243 (582h13m0.368003288s) [noise=true , address=145.79.227.34:4936 , piece_id=WPHQ4WM222WNPSNQHJRHGWWOY3453GNWFU2VOWGME2NUVKGTTCLA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.497333270 (582h13m0.368416305s) [noise=true , address=145.79.227.21:4492 , piece_id=SYF3MNTT4KCEKJIWUVVMK4WS2PYZNYJNLG2HE6UW2F4DAHFHLTAQ, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.497852078 (582h13m0.368935113s) [noise=true , address=145.79.227.39:4038 , piece_id=WK7OI2D5XAKGOM25ISG4IN6FPXIS2F6YDEOQFIQZPYWVSTCZC2CA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:01.777921879 (582h13m1.649004924s) [upload=510381]: Next returned nil segment Dec 29 21:42:01.777940274 (582h13m1.649023349s) [upload=510381]: Waiting for error group managing segments... Dec 29 21:42:02.141106696 (582h13m2.012189741s) [noise=true , address=64.49.13.140:4510 , piece_id=XFDDU2JYEYIWKN7EZWJZEP34QDUWEPL3OW2TR3JSDE2QY6CV3VDQ, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.376774281 (582h13m2.247857327s) [noise=true , address=145.79.227.39:4038 , piece_id=WK7OI2D5XAKGOM25ISG4IN6FPXIS2F6YDEOQFIQZPYWVSTCZC2CA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.379986568 (582h13m2.251069613s) [noise=true , address=145.79.227.21:4492 , piece_id=SYF3MNTT4KCEKJIWUVVMK4WS2PYZNYJNLG2HE6UW2F4DAHFHLTAQ, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.386102975 (582h13m2.25718602s) [noise=true , address=145.79.227.17:4974 , piece_id=ECY3XG222XZAVEEY3W4RFOXMUO7IMAXGMKA7F2MSM6WQJYQT6YZA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.390565783 (582h13m2.261648818s) [noise=true , address=145.79.227.4:4219 , piece_id=YVMXT3TQ5DFBGRF45A2FT4QVVVWJNLTUTFGRHRUXB2LDBLGCU7XA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.393984613 (582h13m2.265067658s) [noise=true , address=145.79.227.6:4266 , piece_id=QXMR4B6HMIOHPRFH3ABT3ZHSOF2ZE23SAGXIWUAEM5LO67NFNEFA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.524875288 (582h13m2.395958333s) [noise=true , address=77.90.31.13:4637 , piece_id=KZRLS4PBFYGGHYJRQLMBEOCNBYCQIMF3F32I3HZFTIUIAT6AE67A, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.540524625 (582h13m2.41160767s) [noise=true , address=77.90.31.13:4656 , piece_id=T3NU3UMWI7VTETXC2G5VCPH5YASFGCQMHOHB6UGQT2PB5ECA272A, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.557053169 (582h13m2.428136214s) [noise=true , address=77.90.31.16:4766 , piece_id=DBPOMKHGXGZQLUHK2NL5M26VRBOWAGZXOOBB2U3XUGHWDGLHEV6Q, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.641331988 (582h13m2.512415043s) [noise=true , address=77.90.31.19:4861 , piece_id=5ODRE74A5PTEPPYWCQS25K6HXMRPZKQIXMX6EOTQMB5XSGNN6TJA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:03.276156729 (582h13m3.147239814s) [noise=true , address=145.79.227.34:4936 , piece_id=WPHQ4WM222WNPSNQHJRHGWWOY3453GNWFU2VOWGME2NUVKGTTCLA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:03.285451126 (582h13m3.156534171s) [noise=true , address=77.90.31.15:4713 , piece_id=SPWJMKD6RJ555CL5PZ2Z7HNW2K67MC2U7QHG6NELHNRFQEB4OFKA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:03.293385374 (582h13m3.164468419s) [noise=true , address=145.79.227.44:4226 , piece_id=JSXESMXNV3MERJUKWNFKJDOWCTXIG6IQ7JMIXXW2JL2TMYBNBDLQ, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:03.293494552 (582h13m3.164577597s) [noise=true , address=145.79.227.49:4357 , piece_id=TP57SIC35AAYKPEF2WIQMJJE76C4ERVDRQGKF7462KW66AMCEFSA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:03.293514761 (582h13m3.164597796s) [seg_pos={0 0}, upload=510381]: Segment reached optimal threshold of 14 pieces. Dec 29 21:42:03.293900185 (582h13m3.16498322s) [noise=true , address=145.79.227.43:4184 , piece_id=ZYUY4IH7NV42CRLTWLA56ASUA3RBKE3HRNG33VEVR6YBSFRM5QSA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VYcRgiFADDQ1rLsiMufThN1K51cuoqKRXNjbp2w9Exu1d3gQG): protocol: expected piece hash; context canceled; EOF Dec 29 21:42:03.293912828 (582h13m3.164995873s) [noise=true , address=145.79.227.43:4184 , piece_id=ZYUY4IH7NV42CRLTWLA56ASUA3RBKE3HRNG33VEVR6YBSFRM5QSA, seg_pos={0 0}, upload=510381]: Long tail cancellation due to optimal threshold Dec 29 21:42:03.293954789 (582h13m3.165037834s) [noise=true , address=145.79.227.29:4946 , piece_id=DGZAZPHNELKHOGDMC6GFKSO4QVCIFGQHXSH6VJJZUJRHPXLWSUPA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12v9Q7aigjWPNXyshHmQt9YKrDu5af1vd1TxkDeHeKixSrWgrLX): protocol: expected piece hash; context canceled; EOF Dec 29 21:42:03.293966481 (582h13m3.165049546s) [noise=true , address=145.79.227.29:4946 , piece_id=DGZAZPHNELKHOGDMC6GFKSO4QVCIFGQHXSH6VJJZUJRHPXLWSUPA, seg_pos={0 0}, upload=510381]: Long tail cancellation due to optimal threshold Dec 29 21:42:03.294146864 (582h13m3.165229899s) [noise=true , address=145.79.227.33:4891 , piece_id=MRIXCIGX6WVNEVCXPLWCWZ67JVIA5JDZLHCT6RE4NCMAFANMVVEQ, seg_pos={0 0}, upload=510381]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RHLM4MJmtKoHsUnsqccy1ehyD7zVSn6gnJNp7yJJmZhTJqPkU): protocol: expected piece hash; context canceled; EOF Dec 29 21:42:03.294155210 (582h13m3.165238245s) [noise=true , address=145.79.227.33:4891 , piece_id=MRIXCIGX6WVNEVCXPLWCWZ67JVIA5JDZLHCT6RE4NCMAFANMVVEQ, seg_pos={0 0}, upload=510381]: Long tail cancellation due to optimal threshold Dec 29 21:42:03.294191519 (582h13m3.165274574s) [noise=true , address=145.79.227.40:4053 , piece_id=SBBX53GUBMB57OJBNXJMCVS3KYD4ZQ5LUCV572LKJUNYDF6MIJ7A, seg_pos={0 0}, upload=510381]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DqyTc4MHvxutFXs2WWRCk8MSuxF3Y1HayyQY4Y5sy8cqDLWuG): protocol: expected piece hash; context canceled; EOF Dec 29 21:42:03.294202380 (582h13m3.165285425s) [noise=true , address=145.79.227.40:4053 , piece_id=SBBX53GUBMB57OJBNXJMCVS3KYD4ZQ5LUCV572LKJUNYDF6MIJ7A, seg_pos={0 0}, upload=510381]: Long tail cancellation due to optimal threshold Dec 29 21:42:03.294227728 (582h13m3.165310763s) [seg_pos={0 0}, upload=510381]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 21:42:03.294328840 (582h13m3.165411886s) [upload=510381]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 21:42:03.294336927 (582h13m3.165419972s) [upload=510381]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 21:42:03.627673008 (582h13m3.498756053s) [upload=510381]: Done uploading segments... Dec 29 21:42:03.627701772 (582h13m3.498784807s) [upload=510381]: Upload finished. err: <nil> Duration: 18.548204973s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 15.468220046s trace: https://jaeger.dev.storj.io/trace/54393681021e4741 FAIL (2025-12-29 22:03:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== upload=510381]: Uploading piece... Dec 29 21:42:00.490334342 (582h13m0.361417387s) [noise=true , address=77.90.31.13:4637 , piece_id=KZRLS4PBFYGGHYJRQLMBEOCNBYCQIMF3F32I3HZFTIUIAT6AE67A, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.490753791 (582h13m0.361836826s) [noise=true , address=77.90.31.19:4861 , piece_id=5ODRE74A5PTEPPYWCQS25K6HXMRPZKQIXMX6EOTQMB5XSGNN6TJA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.491447442 (582h13m0.362530487s) [noise=true , address=145.79.227.49:4357 , piece_id=TP57SIC35AAYKPEF2WIQMJJE76C4ERVDRQGKF7462KW66AMCEFSA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.491940861 (582h13m0.363023896s) [noise=true , address=145.79.227.33:4891 , piece_id=MRIXCIGX6WVNEVCXPLWCWZ67JVIA5JDZLHCT6RE4NCMAFANMVVEQ, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.492346664 (582h13m0.363429699s) [noise=true , address=77.90.31.15:4713 , piece_id=SPWJMKD6RJ555CL5PZ2Z7HNW2K67MC2U7QHG6NELHNRFQEB4OFKA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.492751746 (582h13m0.363834791s) [noise=true , address=77.90.31.13:4656 , piece_id=T3NU3UMWI7VTETXC2G5VCPH5YASFGCQMHOHB6UGQT2PB5ECA272A, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.493112152 (582h13m0.364195187s) [noise=true , address=64.49.13.140:4510 , piece_id=XFDDU2JYEYIWKN7EZWJZEP34QDUWEPL3OW2TR3JSDE2QY6CV3VDQ, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.493462459 (582h13m0.364545504s) [noise=true , address=145.79.227.29:4946 , piece_id=DGZAZPHNELKHOGDMC6GFKSO4QVCIFGQHXSH6VJJZUJRHPXLWSUPA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.493859384 (582h13m0.36494242s) [noise=true , address=145.79.227.44:4226 , piece_id=JSXESMXNV3MERJUKWNFKJDOWCTXIG6IQ7JMIXXW2JL2TMYBNBDLQ, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.494272812 (582h13m0.365355847s) [noise=true , address=77.90.31.16:4766 , piece_id=DBPOMKHGXGZQLUHK2NL5M26VRBOWAGZXOOBB2U3XUGHWDGLHEV6Q, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.495004285 (582h13m0.36608733s) [noise=true , address=145.79.227.4:4219 , piece_id=YVMXT3TQ5DFBGRF45A2FT4QVVVWJNLTUTFGRHRUXB2LDBLGCU7XA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.495625678 (582h13m0.366708723s) [noise=true , address=145.79.227.43:4184 , piece_id=ZYUY4IH7NV42CRLTWLA56ASUA3RBKE3HRNG33VEVR6YBSFRM5QSA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.496138595 (582h13m0.36722163s) [noise=true , address=145.79.227.40:4053 , piece_id=SBBX53GUBMB57OJBNXJMCVS3KYD4ZQ5LUCV572LKJUNYDF6MIJ7A, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.496524049 (582h13m0.367607094s) [noise=true , address=145.79.227.17:4974 , piece_id=ECY3XG222XZAVEEY3W4RFOXMUO7IMAXGMKA7F2MSM6WQJYQT6YZA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.496920243 (582h13m0.368003288s) [noise=true , address=145.79.227.34:4936 , piece_id=WPHQ4WM222WNPSNQHJRHGWWOY3453GNWFU2VOWGME2NUVKGTTCLA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.497333270 (582h13m0.368416305s) [noise=true , address=145.79.227.21:4492 , piece_id=SYF3MNTT4KCEKJIWUVVMK4WS2PYZNYJNLG2HE6UW2F4DAHFHLTAQ, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.497852078 (582h13m0.368935113s) [noise=true , address=145.79.227.39:4038 , piece_id=WK7OI2D5XAKGOM25ISG4IN6FPXIS2F6YDEOQFIQZPYWVSTCZC2CA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:01.777921879 (582h13m1.649004924s) [upload=510381]: Next returned nil segment Dec 29 21:42:01.777940274 (582h13m1.649023349s) [upload=510381]: Waiting for error group managing segments... Dec 29 21:42:02.141106696 (582h13m2.012189741s) [noise=true , address=64.49.13.140:4510 , piece_id=XFDDU2JYEYIWKN7EZWJZEP34QDUWEPL3OW2TR3JSDE2QY6CV3VDQ, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.376774281 (582h13m2.247857327s) [noise=true , address=145.79.227.39:4038 , piece_id=WK7OI2D5XAKGOM25ISG4IN6FPXIS2F6YDEOQFIQZPYWVSTCZC2CA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.379986568 (582h13m2.251069613s) [noise=true , address=145.79.227.21:4492 , piece_id=SYF3MNTT4KCEKJIWUVVMK4WS2PYZNYJNLG2HE6UW2F4DAHFHLTAQ, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.386102975 (582h13m2.25718602s) [noise=true , address=145.79.227.17:4974 , piece_id=ECY3XG222XZAVEEY3W4RFOXMUO7IMAXGMKA7F2MSM6WQJYQT6YZA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.390565783 (582h13m2.261648818s) [noise=true , address=145.79.227.4:4219 , piece_id=YVMXT3TQ5DFBGRF45A2FT4QVVVWJNLTUTFGRHRUXB2LDBLGCU7XA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.393984613 (582h13m2.265067658s) [noise=true , address=145.79.227.6:4266 , piece_id=QXMR4B6HMIOHPRFH3ABT3ZHSOF2ZE23SAGXIWUAEM5LO67NFNEFA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.524875288 (582h13m2.395958333s) [noise=true , address=77.90.31.13:4637 , piece_id=KZRLS4PBFYGGHYJRQLMBEOCNBYCQIMF3F32I3HZFTIUIAT6AE67A, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.540524625 (582h13m2.41160767s) [noise=true , address=77.90.31.13:4656 , piece_id=T3NU3UMWI7VTETXC2G5VCPH5YASFGCQMHOHB6UGQT2PB5ECA272A, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.557053169 (582h13m2.428136214s) [noise=true , address=77.90.31.16:4766 , piece_id=DBPOMKHGXGZQLUHK2NL5M26VRBOWAGZXOOBB2U3XUGHWDGLHEV6Q, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.641331988 (582h13m2.512415043s) [noise=true , address=77.90.31.19:4861 , piece_id=5ODRE74A5PTEPPYWCQS25K6HXMRPZKQIXMX6EOTQMB5XSGNN6TJA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:03.276156729 (582h13m3.147239814s) [noise=true , address=145.79.227.34:4936 , piece_id=WPHQ4WM222WNPSNQHJRHGWWOY3453GNWFU2VOWGME2NUVKGTTCLA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:03.285451126 (582h13m3.156534171s) [noise=true , address=77.90.31.15:4713 , piece_id=SPWJMKD6RJ555CL5PZ2Z7HNW2K67MC2U7QHG6NELHNRFQEB4OFKA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:03.293385374 (582h13m3.164468419s) [noise=true , address=145.79.227.44:4226 , piece_id=JSXESMXNV3MERJUKWNFKJDOWCTXIG6IQ7JMIXXW2JL2TMYBNBDLQ, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:03.293494552 (582h13m3.164577597s) [noise=true , address=145.79.227.49:4357 , piece_id=TP57SIC35AAYKPEF2WIQMJJE76C4ERVDRQGKF7462KW66AMCEFSA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:03.293514761 (582h13m3.164597796s) [seg_pos={0 0}, upload=510381]: Segment reached optimal threshold of 14 pieces. Dec 29 21:42:03.293900185 (582h13m3.16498322s) [noise=true , address=145.79.227.43:4184 , piece_id=ZYUY4IH7NV42CRLTWLA56ASUA3RBKE3HRNG33VEVR6YBSFRM5QSA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VYcRgiFADDQ1rLsiMufThN1K51cuoqKRXNjbp2w9Exu1d3gQG): protocol: expected piece hash; context canceled; EOF Dec 29 21:42:03.293912828 (582h13m3.164995873s) [noise=true , address=145.79.227.43:4184 , piece_id=ZYUY4IH7NV42CRLTWLA56ASUA3RBKE3HRNG33VEVR6YBSFRM5QSA, seg_pos={0 0}, upload=510381]: Long tail cancellation due to optimal threshold Dec 29 21:42:03.293954789 (582h13m3.165037834s) [noise=true , address=145.79.227.29:4946 , piece_id=DGZAZPHNELKHOGDMC6GFKSO4QVCIFGQHXSH6VJJZUJRHPXLWSUPA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12v9Q7aigjWPNXyshHmQt9YKrDu5af1vd1TxkDeHeKixSrWgrLX): protocol: expected piece hash; context canceled; EOF Dec 29 21:42:03.293966481 (582h13m3.165049546s) [noise=true , address=145.79.227.29:4946 , piece_id=DGZAZPHNELKHOGDMC6GFKSO4QVCIFGQHXSH6VJJZUJRHPXLWSUPA, seg_pos={0 0}, upload=510381]: Long tail cancellation due to optimal threshold Dec 29 21:42:03.294146864 (582h13m3.165229899s) [noise=true , address=145.79.227.33:4891 , piece_id=MRIXCIGX6WVNEVCXPLWCWZ67JVIA5JDZLHCT6RE4NCMAFANMVVEQ, seg_pos={0 0}, upload=510381]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RHLM4MJmtKoHsUnsqccy1ehyD7zVSn6gnJNp7yJJmZhTJqPkU): protocol: expected piece hash; context canceled; EOF Dec 29 21:42:03.294155210 (582h13m3.165238245s) [noise=true , address=145.79.227.33:4891 , piece_id=MRIXCIGX6WVNEVCXPLWCWZ67JVIA5JDZLHCT6RE4NCMAFANMVVEQ, seg_pos={0 0}, upload=510381]: Long tail cancellation due to optimal threshold Dec 29 21:42:03.294191519 (582h13m3.165274574s) [noise=true , address=145.79.227.40:4053 , piece_id=SBBX53GUBMB57OJBNXJMCVS3KYD4ZQ5LUCV572LKJUNYDF6MIJ7A, seg_pos={0 0}, upload=510381]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DqyTc4MHvxutFXs2WWRCk8MSuxF3Y1HayyQY4Y5sy8cqDLWuG): protocol: expected piece hash; context canceled; EOF Dec 29 21:42:03.294202380 (582h13m3.165285425s) [noise=true , address=145.79.227.40:4053 , piece_id=SBBX53GUBMB57OJBNXJMCVS3KYD4ZQ5LUCV572LKJUNYDF6MIJ7A, seg_pos={0 0}, upload=510381]: Long tail cancellation due to optimal threshold Dec 29 21:42:03.294227728 (582h13m3.165310763s) [seg_pos={0 0}, upload=510381]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 21:42:03.294328840 (582h13m3.165411886s) [upload=510381]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 21:42:03.294336927 (582h13m3.165419972s) [upload=510381]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 21:42:03.627673008 (582h13m3.498756053s) [upload=510381]: Done uploading segments... Dec 29 21:42:03.627701772 (582h13m3.498784807s) [upload=510381]: Upload finished. err: <nil> Duration: 15.468251146s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [............X..X..X........X............................X...] duration: 7.390512524s trace: https://jaeger.dev.storj.io/trace/5209b3d035311fd FAIL (2025-12-29 22:03:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :00:17.043136250 (582h31m16.914219305s) [upload=510642]: Starting upload Dec 29 22:00:17.043227904 (582h31m16.91431096s) [upload=510642]: Done starting upload Dec 29 22:00:17.043335189 (582h31m16.914418234s) [upload=510642]: Uploading segments... Dec 29 22:00:17.043587749 (582h31m16.914670824s) [upload=510642]: Got next segment. Inline: false Dec 29 22:00:17.043644478 (582h31m16.914727523s) [upload=510642]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 22:00:17.043686427 (582h31m16.914769503s) [upload=510642]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 22:00:17.294384363 (582h31m17.165467408s) [seg_pos={0 0}, upload=510642]: Begin upload segment... Dec 29 22:00:17.294445219 (582h31m17.165528264s) [seg_pos={0 0}, upload=510642]: Done begin upload segment. Dec 29 22:00:17.294484133 (582h31m17.165567178s) [noise=true , address=145.79.227.31:4797 , piece_id=NI3M5NVVVX5HUXZ2SOMQD4S5ZOSSNPCD2F3Z5X2BA2VXMGT57AYQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.294560979 (582h31m17.165644014s) [noise=true , address=77.90.31.11:4553 , piece_id=3Y5HSD73DLZ6ALJ5TFRIVTLOE4ZNZ256YSZ2W3X5LPKL5J5LX2EQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.295149941 (582h31m17.166232976s) [noise=true , address=45.91.116.5:4297 , piece_id=PLUQ7Q32BTGYFQRGVJFAJWHPZCPQTC5RX5UX27S5HJ5NNJPLDMPQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.295419574 (582h31m17.166502609s) [noise=true , address=145.79.227.2:4115 , piece_id=P2AS5B3VYO2K2AWSM5W4BXAXBHSXQQWM7JOXFPOV64NTKDFLT62Q, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.295483997 (582h31m17.166567042s) [noise=true , address=145.79.227.51:4411 , piece_id=HTZQ6SW6KSACLQF7G4CFGQA572J5JVHMLGEWQYSSGZRYX7FJ6DXA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.295762207 (582h31m17.166845252s) [noise=true , address=145.79.227.5:4221 , piece_id=LD3FENLBM2L5N7QDOAMAYJUVH54QVSJQJFP7E36UAH2ZD7R65CCQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.295928003 (582h31m17.167011048s) [noise=true , address=145.79.227.16:4909 , piece_id=CF7WHNKC4LFJQRVFBPFV62JGEBLD4KVPE6Q5JBWQVADC6ZAFFLVQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.296039615 (582h31m17.16712266s) [noise=true , address=45.91.116.13:4154 , piece_id=FE4NFODQ3TWGFXARFYIL64TKK6X7H6PPLIG6LO44LKNAZOEU7DJA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.296272429 (582h31m17.167355474s) [noise=true , address=45.91.116.10:4084 , piece_id=5BFUYVGIFOYXC6RCPU7VBUM36LSM32TVBLS3W54OWFNR4SM6GCSA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.296403588 (582h31m17.167486633s) [noise=true , address=77.90.31.19:4844 , piece_id=CDOWECXLSRPACV4HMAH57UK4EEIYNOMH6GYV6HFHV7CWSEAJ44FA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.296646560 (582h31m17.167729596s) [noise=true , address=77.90.31.14:4661 , piece_id=B7IDSCDHBUXUIPSUT3HMQSCMPGBMYASZSQO2EJBXRR2DQHCLWGYA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.296906235 (582h31m17.167989291s) [noise=true , address=145.79.227.47:4265 , piece_id=FYZA2LRIFKLKVUCUAWMNWTUH3VYR5VFAA7XKAYIMD2E2VOJTI7PA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.296989804 (582h31m17.168072849s) [noise=true , address=145.79.227.46:4601 , piece_id=YJTMZID24VBM5ZZSUUHOCZSZSSNGTFHOSKK6OI5QP3A4G6ZWVWQA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.297235221 (582h31m17.168318267s) [noise=true , address=145.79.227.3:4150 , piece_id=HS4S5UDYD2VWI2N4MH6EWEYQ5G6KCFCJBDZR7AVWPXRYALFXP4FQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.297424432 (582h31m17.168507467s) [noise=true , address=145.79.227.13:4803 , piece_id=YIEUZEDLSC5C67SCFEPCHHZSTYP6MYLYJZJTGZF3UP6JWDTGUOPQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.297606900 (582h31m17.168689945s) [noise=true , address=45.91.116.14:4190 , piece_id=FIL6AIXGE7DE2M4QBVSIO53KGY43VFPZQ2FTXSWJFPJU4VZCKUXQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:18.995364155 (582h31m18.866447201s) [upload=510642]: Next returned nil segment Dec 29 22:00:18.995383603 (582h31m18.866466648s) [upload=510642]: Waiting for error group managing segments... Dec 29 22:00:19.284009408 (582h31m19.155092453s) [noise=true , address=145.79.227.51:4411 , piece_id=HTZQ6SW6KSACLQF7G4CFGQA572J5JVHMLGEWQYSSGZRYX7FJ6DXA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.295985232 (582h31m19.167068277s) [noise=true , address=145.79.227.13:4803 , piece_id=YIEUZEDLSC5C67SCFEPCHHZSTYP6MYLYJZJTGZF3UP6JWDTGUOPQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.296135458 (582h31m19.167218503s) [noise=true , address=145.79.227.16:4909 , piece_id=CF7WHNKC4LFJQRVFBPFV62JGEBLD4KVPE6Q5JBWQVADC6ZAFFLVQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.310119916 (582h31m19.181202971s) [noise=true , address=145.79.227.46:4601 , piece_id=YJTMZID24VBM5ZZSUUHOCZSZSSNGTFHOSKK6OI5QP3A4G6ZWVWQA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.310998939 (582h31m19.182081984s) [noise=true , address=145.79.227.31:4797 , piece_id=NI3M5NVVVX5HUXZ2SOMQD4S5ZOSSNPCD2F3Z5X2BA2VXMGT57AYQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.318707138 (582h31m19.189790183s) [noise=true , address=145.79.227.47:4265 , piece_id=FYZA2LRIFKLKVUCUAWMNWTUH3VYR5VFAA7XKAYIMD2E2VOJTI7PA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.322059381 (582h31m19.193142426s) [noise=true , address=145.79.227.5:4221 , piece_id=LD3FENLBM2L5N7QDOAMAYJUVH54QVSJQJFP7E36UAH2ZD7R65CCQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.506372108 (582h31m19.377455153s) [noise=true , address=77.90.31.19:4844 , piece_id=CDOWECXLSRPACV4HMAH57UK4EEIYNOMH6GYV6HFHV7CWSEAJ44FA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.507172602 (582h31m19.378255648s) [noise=true , address=77.90.31.11:4553 , piece_id=3Y5HSD73DLZ6ALJ5TFRIVTLOE4ZNZ256YSZ2W3X5LPKL5J5LX2EQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.527548842 (582h31m19.398631887s) [noise=true , address=77.90.31.14:4661 , piece_id=B7IDSCDHBUXUIPSUT3HMQSCMPGBMYASZSQO2EJBXRR2DQHCLWGYA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.531443508 (582h31m19.402526553s) [noise=true , address=45.91.116.5:4297 , piece_id=PLUQ7Q32BTGYFQRGVJFAJWHPZCPQTC5RX5UX27S5HJ5NNJPLDMPQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.552364906 (582h31m19.423447951s) [noise=true , address=45.91.116.14:4190 , piece_id=FIL6AIXGE7DE2M4QBVSIO53KGY43VFPZQ2FTXSWJFPJU4VZCKUXQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.564056769 (582h31m19.435139814s) [noise=true , address=45.91.116.10:4084 , piece_id=5BFUYVGIFOYXC6RCPU7VBUM36LSM32TVBLS3W54OWFNR4SM6GCSA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.564087016 (582h31m19.435170051s) [seg_pos={0 0}, upload=510642]: Segment reached optimal threshold of 13 pieces. Dec 29 22:00:19.564294431 (582h31m19.435377466s) [noise=true , address=145.79.227.3:4150 , piece_id=HS4S5UDYD2VWI2N4MH6EWEYQ5G6KCFCJBDZR7AVWPXRYALFXP4FQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK): protocol: expected piece hash; context canceled; EOF Dec 29 22:00:19.564302366 (582h31m19.435385411s) [noise=true , address=145.79.227.3:4150 , piece_id=HS4S5UDYD2VWI2N4MH6EWEYQ5G6KCFCJBDZR7AVWPXRYALFXP4FQ, seg_pos={0 0}, upload=510642]: Long tail cancellation due to optimal threshold Dec 29 22:00:19.564388932 (582h31m19.435471977s) [noise=true , address=45.91.116.13:4154 , piece_id=FE4NFODQ3TWGFXARFYIL64TKK6X7H6PPLIG6LO44LKNAZOEU7DJA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Dec 29 22:00:19.564401876 (582h31m19.435484911s) [noise=true , address=45.91.116.13:4154 , piece_id=FE4NFODQ3TWGFXARFYIL64TKK6X7H6PPLIG6LO44LKNAZOEU7DJA, seg_pos={0 0}, upload=510642]: Long tail cancellation due to optimal threshold Dec 29 22:00:19.564558945 (582h31m19.435641981s) [noise=true , address=145.79.227.2:4115 , piece_id=P2AS5B3VYO2K2AWSM5W4BXAXBHSXQQWM7JOXFPOV64NTKDFLT62Q, seg_pos={0 0}, upload=510642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: write tcp 188.166.225.170:38246->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:38246->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38246->145.79.227.2:4115: use of closed network connection Dec 29 22:00:19.564570738 (582h31m19.435653783s) [noise=true , address=145.79.227.2:4115 , piece_id=P2AS5B3VYO2K2AWSM5W4BXAXBHSXQQWM7JOXFPOV64NTKDFLT62Q, seg_pos={0 0}, upload=510642]: Long tail cancellation due to optimal threshold Dec 29 22:00:19.564601426 (582h31m19.435684471s) [seg_pos={0 0}, upload=510642]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 22:00:19.564683973 (582h31m19.435767018s) [upload=510642]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 22:00:19.564690004 (582h31m19.435773049s) [upload=510642]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 22:00:19.871860472 (582h31m19.742943527s) [upload=510642]: Done uploading segments... Dec 29 22:00:19.871892413 (582h31m19.742975468s) [upload=510642]: Upload finished. err: <nil> Duration: 7.390533595s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 25.557515498s trace: https://jaeger.dev.storj.io/trace/54393681021e4741 FAIL (2025-12-29 22:02:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== upload=510381]: Uploading piece... Dec 29 21:42:00.490334342 (582h13m0.361417387s) [noise=true , address=77.90.31.13:4637 , piece_id=KZRLS4PBFYGGHYJRQLMBEOCNBYCQIMF3F32I3HZFTIUIAT6AE67A, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.490753791 (582h13m0.361836826s) [noise=true , address=77.90.31.19:4861 , piece_id=5ODRE74A5PTEPPYWCQS25K6HXMRPZKQIXMX6EOTQMB5XSGNN6TJA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.491447442 (582h13m0.362530487s) [noise=true , address=145.79.227.49:4357 , piece_id=TP57SIC35AAYKPEF2WIQMJJE76C4ERVDRQGKF7462KW66AMCEFSA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.491940861 (582h13m0.363023896s) [noise=true , address=145.79.227.33:4891 , piece_id=MRIXCIGX6WVNEVCXPLWCWZ67JVIA5JDZLHCT6RE4NCMAFANMVVEQ, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.492346664 (582h13m0.363429699s) [noise=true , address=77.90.31.15:4713 , piece_id=SPWJMKD6RJ555CL5PZ2Z7HNW2K67MC2U7QHG6NELHNRFQEB4OFKA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.492751746 (582h13m0.363834791s) [noise=true , address=77.90.31.13:4656 , piece_id=T3NU3UMWI7VTETXC2G5VCPH5YASFGCQMHOHB6UGQT2PB5ECA272A, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.493112152 (582h13m0.364195187s) [noise=true , address=64.49.13.140:4510 , piece_id=XFDDU2JYEYIWKN7EZWJZEP34QDUWEPL3OW2TR3JSDE2QY6CV3VDQ, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.493462459 (582h13m0.364545504s) [noise=true , address=145.79.227.29:4946 , piece_id=DGZAZPHNELKHOGDMC6GFKSO4QVCIFGQHXSH6VJJZUJRHPXLWSUPA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.493859384 (582h13m0.36494242s) [noise=true , address=145.79.227.44:4226 , piece_id=JSXESMXNV3MERJUKWNFKJDOWCTXIG6IQ7JMIXXW2JL2TMYBNBDLQ, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.494272812 (582h13m0.365355847s) [noise=true , address=77.90.31.16:4766 , piece_id=DBPOMKHGXGZQLUHK2NL5M26VRBOWAGZXOOBB2U3XUGHWDGLHEV6Q, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.495004285 (582h13m0.36608733s) [noise=true , address=145.79.227.4:4219 , piece_id=YVMXT3TQ5DFBGRF45A2FT4QVVVWJNLTUTFGRHRUXB2LDBLGCU7XA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.495625678 (582h13m0.366708723s) [noise=true , address=145.79.227.43:4184 , piece_id=ZYUY4IH7NV42CRLTWLA56ASUA3RBKE3HRNG33VEVR6YBSFRM5QSA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.496138595 (582h13m0.36722163s) [noise=true , address=145.79.227.40:4053 , piece_id=SBBX53GUBMB57OJBNXJMCVS3KYD4ZQ5LUCV572LKJUNYDF6MIJ7A, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.496524049 (582h13m0.367607094s) [noise=true , address=145.79.227.17:4974 , piece_id=ECY3XG222XZAVEEY3W4RFOXMUO7IMAXGMKA7F2MSM6WQJYQT6YZA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.496920243 (582h13m0.368003288s) [noise=true , address=145.79.227.34:4936 , piece_id=WPHQ4WM222WNPSNQHJRHGWWOY3453GNWFU2VOWGME2NUVKGTTCLA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.497333270 (582h13m0.368416305s) [noise=true , address=145.79.227.21:4492 , piece_id=SYF3MNTT4KCEKJIWUVVMK4WS2PYZNYJNLG2HE6UW2F4DAHFHLTAQ, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.497852078 (582h13m0.368935113s) [noise=true , address=145.79.227.39:4038 , piece_id=WK7OI2D5XAKGOM25ISG4IN6FPXIS2F6YDEOQFIQZPYWVSTCZC2CA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:01.777921879 (582h13m1.649004924s) [upload=510381]: Next returned nil segment Dec 29 21:42:01.777940274 (582h13m1.649023349s) [upload=510381]: Waiting for error group managing segments... Dec 29 21:42:02.141106696 (582h13m2.012189741s) [noise=true , address=64.49.13.140:4510 , piece_id=XFDDU2JYEYIWKN7EZWJZEP34QDUWEPL3OW2TR3JSDE2QY6CV3VDQ, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.376774281 (582h13m2.247857327s) [noise=true , address=145.79.227.39:4038 , piece_id=WK7OI2D5XAKGOM25ISG4IN6FPXIS2F6YDEOQFIQZPYWVSTCZC2CA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.379986568 (582h13m2.251069613s) [noise=true , address=145.79.227.21:4492 , piece_id=SYF3MNTT4KCEKJIWUVVMK4WS2PYZNYJNLG2HE6UW2F4DAHFHLTAQ, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.386102975 (582h13m2.25718602s) [noise=true , address=145.79.227.17:4974 , piece_id=ECY3XG222XZAVEEY3W4RFOXMUO7IMAXGMKA7F2MSM6WQJYQT6YZA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.390565783 (582h13m2.261648818s) [noise=true , address=145.79.227.4:4219 , piece_id=YVMXT3TQ5DFBGRF45A2FT4QVVVWJNLTUTFGRHRUXB2LDBLGCU7XA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.393984613 (582h13m2.265067658s) [noise=true , address=145.79.227.6:4266 , piece_id=QXMR4B6HMIOHPRFH3ABT3ZHSOF2ZE23SAGXIWUAEM5LO67NFNEFA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.524875288 (582h13m2.395958333s) [noise=true , address=77.90.31.13:4637 , piece_id=KZRLS4PBFYGGHYJRQLMBEOCNBYCQIMF3F32I3HZFTIUIAT6AE67A, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.540524625 (582h13m2.41160767s) [noise=true , address=77.90.31.13:4656 , piece_id=T3NU3UMWI7VTETXC2G5VCPH5YASFGCQMHOHB6UGQT2PB5ECA272A, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.557053169 (582h13m2.428136214s) [noise=true , address=77.90.31.16:4766 , piece_id=DBPOMKHGXGZQLUHK2NL5M26VRBOWAGZXOOBB2U3XUGHWDGLHEV6Q, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.641331988 (582h13m2.512415043s) [noise=true , address=77.90.31.19:4861 , piece_id=5ODRE74A5PTEPPYWCQS25K6HXMRPZKQIXMX6EOTQMB5XSGNN6TJA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:03.276156729 (582h13m3.147239814s) [noise=true , address=145.79.227.34:4936 , piece_id=WPHQ4WM222WNPSNQHJRHGWWOY3453GNWFU2VOWGME2NUVKGTTCLA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:03.285451126 (582h13m3.156534171s) [noise=true , address=77.90.31.15:4713 , piece_id=SPWJMKD6RJ555CL5PZ2Z7HNW2K67MC2U7QHG6NELHNRFQEB4OFKA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:03.293385374 (582h13m3.164468419s) [noise=true , address=145.79.227.44:4226 , piece_id=JSXESMXNV3MERJUKWNFKJDOWCTXIG6IQ7JMIXXW2JL2TMYBNBDLQ, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:03.293494552 (582h13m3.164577597s) [noise=true , address=145.79.227.49:4357 , piece_id=TP57SIC35AAYKPEF2WIQMJJE76C4ERVDRQGKF7462KW66AMCEFSA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:03.293514761 (582h13m3.164597796s) [seg_pos={0 0}, upload=510381]: Segment reached optimal threshold of 14 pieces. Dec 29 21:42:03.293900185 (582h13m3.16498322s) [noise=true , address=145.79.227.43:4184 , piece_id=ZYUY4IH7NV42CRLTWLA56ASUA3RBKE3HRNG33VEVR6YBSFRM5QSA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VYcRgiFADDQ1rLsiMufThN1K51cuoqKRXNjbp2w9Exu1d3gQG): protocol: expected piece hash; context canceled; EOF Dec 29 21:42:03.293912828 (582h13m3.164995873s) [noise=true , address=145.79.227.43:4184 , piece_id=ZYUY4IH7NV42CRLTWLA56ASUA3RBKE3HRNG33VEVR6YBSFRM5QSA, seg_pos={0 0}, upload=510381]: Long tail cancellation due to optimal threshold Dec 29 21:42:03.293954789 (582h13m3.165037834s) [noise=true , address=145.79.227.29:4946 , piece_id=DGZAZPHNELKHOGDMC6GFKSO4QVCIFGQHXSH6VJJZUJRHPXLWSUPA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12v9Q7aigjWPNXyshHmQt9YKrDu5af1vd1TxkDeHeKixSrWgrLX): protocol: expected piece hash; context canceled; EOF Dec 29 21:42:03.293966481 (582h13m3.165049546s) [noise=true , address=145.79.227.29:4946 , piece_id=DGZAZPHNELKHOGDMC6GFKSO4QVCIFGQHXSH6VJJZUJRHPXLWSUPA, seg_pos={0 0}, upload=510381]: Long tail cancellation due to optimal threshold Dec 29 21:42:03.294146864 (582h13m3.165229899s) [noise=true , address=145.79.227.33:4891 , piece_id=MRIXCIGX6WVNEVCXPLWCWZ67JVIA5JDZLHCT6RE4NCMAFANMVVEQ, seg_pos={0 0}, upload=510381]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RHLM4MJmtKoHsUnsqccy1ehyD7zVSn6gnJNp7yJJmZhTJqPkU): protocol: expected piece hash; context canceled; EOF Dec 29 21:42:03.294155210 (582h13m3.165238245s) [noise=true , address=145.79.227.33:4891 , piece_id=MRIXCIGX6WVNEVCXPLWCWZ67JVIA5JDZLHCT6RE4NCMAFANMVVEQ, seg_pos={0 0}, upload=510381]: Long tail cancellation due to optimal threshold Dec 29 21:42:03.294191519 (582h13m3.165274574s) [noise=true , address=145.79.227.40:4053 , piece_id=SBBX53GUBMB57OJBNXJMCVS3KYD4ZQ5LUCV572LKJUNYDF6MIJ7A, seg_pos={0 0}, upload=510381]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DqyTc4MHvxutFXs2WWRCk8MSuxF3Y1HayyQY4Y5sy8cqDLWuG): protocol: expected piece hash; context canceled; EOF Dec 29 21:42:03.294202380 (582h13m3.165285425s) [noise=true , address=145.79.227.40:4053 , piece_id=SBBX53GUBMB57OJBNXJMCVS3KYD4ZQ5LUCV572LKJUNYDF6MIJ7A, seg_pos={0 0}, upload=510381]: Long tail cancellation due to optimal threshold Dec 29 21:42:03.294227728 (582h13m3.165310763s) [seg_pos={0 0}, upload=510381]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 21:42:03.294328840 (582h13m3.165411886s) [upload=510381]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 21:42:03.294336927 (582h13m3.165419972s) [upload=510381]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 21:42:03.627673008 (582h13m3.498756053s) [upload=510381]: Done uploading segments... Dec 29 21:42:03.627701772 (582h13m3.498784807s) [upload=510381]: Upload finished. err: <nil> Duration: 25.557545166s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 21.354002669s trace: https://jaeger.dev.storj.io/trace/54393681021e4741 FAIL (2025-12-29 22:01:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== upload=510381]: Uploading piece... Dec 29 21:42:00.490334342 (582h13m0.361417387s) [noise=true , address=77.90.31.13:4637 , piece_id=KZRLS4PBFYGGHYJRQLMBEOCNBYCQIMF3F32I3HZFTIUIAT6AE67A, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.490753791 (582h13m0.361836826s) [noise=true , address=77.90.31.19:4861 , piece_id=5ODRE74A5PTEPPYWCQS25K6HXMRPZKQIXMX6EOTQMB5XSGNN6TJA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.491447442 (582h13m0.362530487s) [noise=true , address=145.79.227.49:4357 , piece_id=TP57SIC35AAYKPEF2WIQMJJE76C4ERVDRQGKF7462KW66AMCEFSA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.491940861 (582h13m0.363023896s) [noise=true , address=145.79.227.33:4891 , piece_id=MRIXCIGX6WVNEVCXPLWCWZ67JVIA5JDZLHCT6RE4NCMAFANMVVEQ, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.492346664 (582h13m0.363429699s) [noise=true , address=77.90.31.15:4713 , piece_id=SPWJMKD6RJ555CL5PZ2Z7HNW2K67MC2U7QHG6NELHNRFQEB4OFKA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.492751746 (582h13m0.363834791s) [noise=true , address=77.90.31.13:4656 , piece_id=T3NU3UMWI7VTETXC2G5VCPH5YASFGCQMHOHB6UGQT2PB5ECA272A, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.493112152 (582h13m0.364195187s) [noise=true , address=64.49.13.140:4510 , piece_id=XFDDU2JYEYIWKN7EZWJZEP34QDUWEPL3OW2TR3JSDE2QY6CV3VDQ, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.493462459 (582h13m0.364545504s) [noise=true , address=145.79.227.29:4946 , piece_id=DGZAZPHNELKHOGDMC6GFKSO4QVCIFGQHXSH6VJJZUJRHPXLWSUPA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.493859384 (582h13m0.36494242s) [noise=true , address=145.79.227.44:4226 , piece_id=JSXESMXNV3MERJUKWNFKJDOWCTXIG6IQ7JMIXXW2JL2TMYBNBDLQ, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.494272812 (582h13m0.365355847s) [noise=true , address=77.90.31.16:4766 , piece_id=DBPOMKHGXGZQLUHK2NL5M26VRBOWAGZXOOBB2U3XUGHWDGLHEV6Q, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.495004285 (582h13m0.36608733s) [noise=true , address=145.79.227.4:4219 , piece_id=YVMXT3TQ5DFBGRF45A2FT4QVVVWJNLTUTFGRHRUXB2LDBLGCU7XA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.495625678 (582h13m0.366708723s) [noise=true , address=145.79.227.43:4184 , piece_id=ZYUY4IH7NV42CRLTWLA56ASUA3RBKE3HRNG33VEVR6YBSFRM5QSA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.496138595 (582h13m0.36722163s) [noise=true , address=145.79.227.40:4053 , piece_id=SBBX53GUBMB57OJBNXJMCVS3KYD4ZQ5LUCV572LKJUNYDF6MIJ7A, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.496524049 (582h13m0.367607094s) [noise=true , address=145.79.227.17:4974 , piece_id=ECY3XG222XZAVEEY3W4RFOXMUO7IMAXGMKA7F2MSM6WQJYQT6YZA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.496920243 (582h13m0.368003288s) [noise=true , address=145.79.227.34:4936 , piece_id=WPHQ4WM222WNPSNQHJRHGWWOY3453GNWFU2VOWGME2NUVKGTTCLA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.497333270 (582h13m0.368416305s) [noise=true , address=145.79.227.21:4492 , piece_id=SYF3MNTT4KCEKJIWUVVMK4WS2PYZNYJNLG2HE6UW2F4DAHFHLTAQ, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:00.497852078 (582h13m0.368935113s) [noise=true , address=145.79.227.39:4038 , piece_id=WK7OI2D5XAKGOM25ISG4IN6FPXIS2F6YDEOQFIQZPYWVSTCZC2CA, seg_pos={0 0}, upload=510381]: Uploading piece... Dec 29 21:42:01.777921879 (582h13m1.649004924s) [upload=510381]: Next returned nil segment Dec 29 21:42:01.777940274 (582h13m1.649023349s) [upload=510381]: Waiting for error group managing segments... Dec 29 21:42:02.141106696 (582h13m2.012189741s) [noise=true , address=64.49.13.140:4510 , piece_id=XFDDU2JYEYIWKN7EZWJZEP34QDUWEPL3OW2TR3JSDE2QY6CV3VDQ, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.376774281 (582h13m2.247857327s) [noise=true , address=145.79.227.39:4038 , piece_id=WK7OI2D5XAKGOM25ISG4IN6FPXIS2F6YDEOQFIQZPYWVSTCZC2CA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.379986568 (582h13m2.251069613s) [noise=true , address=145.79.227.21:4492 , piece_id=SYF3MNTT4KCEKJIWUVVMK4WS2PYZNYJNLG2HE6UW2F4DAHFHLTAQ, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.386102975 (582h13m2.25718602s) [noise=true , address=145.79.227.17:4974 , piece_id=ECY3XG222XZAVEEY3W4RFOXMUO7IMAXGMKA7F2MSM6WQJYQT6YZA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.390565783 (582h13m2.261648818s) [noise=true , address=145.79.227.4:4219 , piece_id=YVMXT3TQ5DFBGRF45A2FT4QVVVWJNLTUTFGRHRUXB2LDBLGCU7XA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.393984613 (582h13m2.265067658s) [noise=true , address=145.79.227.6:4266 , piece_id=QXMR4B6HMIOHPRFH3ABT3ZHSOF2ZE23SAGXIWUAEM5LO67NFNEFA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.524875288 (582h13m2.395958333s) [noise=true , address=77.90.31.13:4637 , piece_id=KZRLS4PBFYGGHYJRQLMBEOCNBYCQIMF3F32I3HZFTIUIAT6AE67A, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.540524625 (582h13m2.41160767s) [noise=true , address=77.90.31.13:4656 , piece_id=T3NU3UMWI7VTETXC2G5VCPH5YASFGCQMHOHB6UGQT2PB5ECA272A, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.557053169 (582h13m2.428136214s) [noise=true , address=77.90.31.16:4766 , piece_id=DBPOMKHGXGZQLUHK2NL5M26VRBOWAGZXOOBB2U3XUGHWDGLHEV6Q, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:02.641331988 (582h13m2.512415043s) [noise=true , address=77.90.31.19:4861 , piece_id=5ODRE74A5PTEPPYWCQS25K6HXMRPZKQIXMX6EOTQMB5XSGNN6TJA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:03.276156729 (582h13m3.147239814s) [noise=true , address=145.79.227.34:4936 , piece_id=WPHQ4WM222WNPSNQHJRHGWWOY3453GNWFU2VOWGME2NUVKGTTCLA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:03.285451126 (582h13m3.156534171s) [noise=true , address=77.90.31.15:4713 , piece_id=SPWJMKD6RJ555CL5PZ2Z7HNW2K67MC2U7QHG6NELHNRFQEB4OFKA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:03.293385374 (582h13m3.164468419s) [noise=true , address=145.79.227.44:4226 , piece_id=JSXESMXNV3MERJUKWNFKJDOWCTXIG6IQ7JMIXXW2JL2TMYBNBDLQ, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:03.293494552 (582h13m3.164577597s) [noise=true , address=145.79.227.49:4357 , piece_id=TP57SIC35AAYKPEF2WIQMJJE76C4ERVDRQGKF7462KW66AMCEFSA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: <nil> Dec 29 21:42:03.293514761 (582h13m3.164597796s) [seg_pos={0 0}, upload=510381]: Segment reached optimal threshold of 14 pieces. Dec 29 21:42:03.293900185 (582h13m3.16498322s) [noise=true , address=145.79.227.43:4184 , piece_id=ZYUY4IH7NV42CRLTWLA56ASUA3RBKE3HRNG33VEVR6YBSFRM5QSA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VYcRgiFADDQ1rLsiMufThN1K51cuoqKRXNjbp2w9Exu1d3gQG): protocol: expected piece hash; context canceled; EOF Dec 29 21:42:03.293912828 (582h13m3.164995873s) [noise=true , address=145.79.227.43:4184 , piece_id=ZYUY4IH7NV42CRLTWLA56ASUA3RBKE3HRNG33VEVR6YBSFRM5QSA, seg_pos={0 0}, upload=510381]: Long tail cancellation due to optimal threshold Dec 29 21:42:03.293954789 (582h13m3.165037834s) [noise=true , address=145.79.227.29:4946 , piece_id=DGZAZPHNELKHOGDMC6GFKSO4QVCIFGQHXSH6VJJZUJRHPXLWSUPA, seg_pos={0 0}, upload=510381]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12v9Q7aigjWPNXyshHmQt9YKrDu5af1vd1TxkDeHeKixSrWgrLX): protocol: expected piece hash; context canceled; EOF Dec 29 21:42:03.293966481 (582h13m3.165049546s) [noise=true , address=145.79.227.29:4946 , piece_id=DGZAZPHNELKHOGDMC6GFKSO4QVCIFGQHXSH6VJJZUJRHPXLWSUPA, seg_pos={0 0}, upload=510381]: Long tail cancellation due to optimal threshold Dec 29 21:42:03.294146864 (582h13m3.165229899s) [noise=true , address=145.79.227.33:4891 , piece_id=MRIXCIGX6WVNEVCXPLWCWZ67JVIA5JDZLHCT6RE4NCMAFANMVVEQ, seg_pos={0 0}, upload=510381]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RHLM4MJmtKoHsUnsqccy1ehyD7zVSn6gnJNp7yJJmZhTJqPkU): protocol: expected piece hash; context canceled; EOF Dec 29 21:42:03.294155210 (582h13m3.165238245s) [noise=true , address=145.79.227.33:4891 , piece_id=MRIXCIGX6WVNEVCXPLWCWZ67JVIA5JDZLHCT6RE4NCMAFANMVVEQ, seg_pos={0 0}, upload=510381]: Long tail cancellation due to optimal threshold Dec 29 21:42:03.294191519 (582h13m3.165274574s) [noise=true , address=145.79.227.40:4053 , piece_id=SBBX53GUBMB57OJBNXJMCVS3KYD4ZQ5LUCV572LKJUNYDF6MIJ7A, seg_pos={0 0}, upload=510381]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DqyTc4MHvxutFXs2WWRCk8MSuxF3Y1HayyQY4Y5sy8cqDLWuG): protocol: expected piece hash; context canceled; EOF Dec 29 21:42:03.294202380 (582h13m3.165285425s) [noise=true , address=145.79.227.40:4053 , piece_id=SBBX53GUBMB57OJBNXJMCVS3KYD4ZQ5LUCV572LKJUNYDF6MIJ7A, seg_pos={0 0}, upload=510381]: Long tail cancellation due to optimal threshold Dec 29 21:42:03.294227728 (582h13m3.165310763s) [seg_pos={0 0}, upload=510381]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 21:42:03.294328840 (582h13m3.165411886s) [upload=510381]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 21:42:03.294336927 (582h13m3.165419972s) [upload=510381]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 21:42:03.627673008 (582h13m3.498756053s) [upload=510381]: Done uploading segments... Dec 29 21:42:03.627701772 (582h13m3.498784807s) [upload=510381]: Upload finished. err: <nil> Duration: 21.354031574s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.............X..X..X........X............................X..] duration: 6.129666582s trace: https://jaeger.dev.storj.io/trace/5209b3d035311fd FAIL (2025-12-29 22:01:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :00:17.043136250 (582h31m16.914219305s) [upload=510642]: Starting upload Dec 29 22:00:17.043227904 (582h31m16.91431096s) [upload=510642]: Done starting upload Dec 29 22:00:17.043335189 (582h31m16.914418234s) [upload=510642]: Uploading segments... Dec 29 22:00:17.043587749 (582h31m16.914670824s) [upload=510642]: Got next segment. Inline: false Dec 29 22:00:17.043644478 (582h31m16.914727523s) [upload=510642]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 22:00:17.043686427 (582h31m16.914769503s) [upload=510642]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 22:00:17.294384363 (582h31m17.165467408s) [seg_pos={0 0}, upload=510642]: Begin upload segment... Dec 29 22:00:17.294445219 (582h31m17.165528264s) [seg_pos={0 0}, upload=510642]: Done begin upload segment. Dec 29 22:00:17.294484133 (582h31m17.165567178s) [noise=true , address=145.79.227.31:4797 , piece_id=NI3M5NVVVX5HUXZ2SOMQD4S5ZOSSNPCD2F3Z5X2BA2VXMGT57AYQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.294560979 (582h31m17.165644014s) [noise=true , address=77.90.31.11:4553 , piece_id=3Y5HSD73DLZ6ALJ5TFRIVTLOE4ZNZ256YSZ2W3X5LPKL5J5LX2EQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.295149941 (582h31m17.166232976s) [noise=true , address=45.91.116.5:4297 , piece_id=PLUQ7Q32BTGYFQRGVJFAJWHPZCPQTC5RX5UX27S5HJ5NNJPLDMPQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.295419574 (582h31m17.166502609s) [noise=true , address=145.79.227.2:4115 , piece_id=P2AS5B3VYO2K2AWSM5W4BXAXBHSXQQWM7JOXFPOV64NTKDFLT62Q, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.295483997 (582h31m17.166567042s) [noise=true , address=145.79.227.51:4411 , piece_id=HTZQ6SW6KSACLQF7G4CFGQA572J5JVHMLGEWQYSSGZRYX7FJ6DXA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.295762207 (582h31m17.166845252s) [noise=true , address=145.79.227.5:4221 , piece_id=LD3FENLBM2L5N7QDOAMAYJUVH54QVSJQJFP7E36UAH2ZD7R65CCQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.295928003 (582h31m17.167011048s) [noise=true , address=145.79.227.16:4909 , piece_id=CF7WHNKC4LFJQRVFBPFV62JGEBLD4KVPE6Q5JBWQVADC6ZAFFLVQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.296039615 (582h31m17.16712266s) [noise=true , address=45.91.116.13:4154 , piece_id=FE4NFODQ3TWGFXARFYIL64TKK6X7H6PPLIG6LO44LKNAZOEU7DJA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.296272429 (582h31m17.167355474s) [noise=true , address=45.91.116.10:4084 , piece_id=5BFUYVGIFOYXC6RCPU7VBUM36LSM32TVBLS3W54OWFNR4SM6GCSA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.296403588 (582h31m17.167486633s) [noise=true , address=77.90.31.19:4844 , piece_id=CDOWECXLSRPACV4HMAH57UK4EEIYNOMH6GYV6HFHV7CWSEAJ44FA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.296646560 (582h31m17.167729596s) [noise=true , address=77.90.31.14:4661 , piece_id=B7IDSCDHBUXUIPSUT3HMQSCMPGBMYASZSQO2EJBXRR2DQHCLWGYA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.296906235 (582h31m17.167989291s) [noise=true , address=145.79.227.47:4265 , piece_id=FYZA2LRIFKLKVUCUAWMNWTUH3VYR5VFAA7XKAYIMD2E2VOJTI7PA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.296989804 (582h31m17.168072849s) [noise=true , address=145.79.227.46:4601 , piece_id=YJTMZID24VBM5ZZSUUHOCZSZSSNGTFHOSKK6OI5QP3A4G6ZWVWQA, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.297235221 (582h31m17.168318267s) [noise=true , address=145.79.227.3:4150 , piece_id=HS4S5UDYD2VWI2N4MH6EWEYQ5G6KCFCJBDZR7AVWPXRYALFXP4FQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.297424432 (582h31m17.168507467s) [noise=true , address=145.79.227.13:4803 , piece_id=YIEUZEDLSC5C67SCFEPCHHZSTYP6MYLYJZJTGZF3UP6JWDTGUOPQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:17.297606900 (582h31m17.168689945s) [noise=true , address=45.91.116.14:4190 , piece_id=FIL6AIXGE7DE2M4QBVSIO53KGY43VFPZQ2FTXSWJFPJU4VZCKUXQ, seg_pos={0 0}, upload=510642]: Uploading piece... Dec 29 22:00:18.995364155 (582h31m18.866447201s) [upload=510642]: Next returned nil segment Dec 29 22:00:18.995383603 (582h31m18.866466648s) [upload=510642]: Waiting for error group managing segments... Dec 29 22:00:19.284009408 (582h31m19.155092453s) [noise=true , address=145.79.227.51:4411 , piece_id=HTZQ6SW6KSACLQF7G4CFGQA572J5JVHMLGEWQYSSGZRYX7FJ6DXA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.295985232 (582h31m19.167068277s) [noise=true , address=145.79.227.13:4803 , piece_id=YIEUZEDLSC5C67SCFEPCHHZSTYP6MYLYJZJTGZF3UP6JWDTGUOPQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.296135458 (582h31m19.167218503s) [noise=true , address=145.79.227.16:4909 , piece_id=CF7WHNKC4LFJQRVFBPFV62JGEBLD4KVPE6Q5JBWQVADC6ZAFFLVQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.310119916 (582h31m19.181202971s) [noise=true , address=145.79.227.46:4601 , piece_id=YJTMZID24VBM5ZZSUUHOCZSZSSNGTFHOSKK6OI5QP3A4G6ZWVWQA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.310998939 (582h31m19.182081984s) [noise=true , address=145.79.227.31:4797 , piece_id=NI3M5NVVVX5HUXZ2SOMQD4S5ZOSSNPCD2F3Z5X2BA2VXMGT57AYQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.318707138 (582h31m19.189790183s) [noise=true , address=145.79.227.47:4265 , piece_id=FYZA2LRIFKLKVUCUAWMNWTUH3VYR5VFAA7XKAYIMD2E2VOJTI7PA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.322059381 (582h31m19.193142426s) [noise=true , address=145.79.227.5:4221 , piece_id=LD3FENLBM2L5N7QDOAMAYJUVH54QVSJQJFP7E36UAH2ZD7R65CCQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.506372108 (582h31m19.377455153s) [noise=true , address=77.90.31.19:4844 , piece_id=CDOWECXLSRPACV4HMAH57UK4EEIYNOMH6GYV6HFHV7CWSEAJ44FA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.507172602 (582h31m19.378255648s) [noise=true , address=77.90.31.11:4553 , piece_id=3Y5HSD73DLZ6ALJ5TFRIVTLOE4ZNZ256YSZ2W3X5LPKL5J5LX2EQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.527548842 (582h31m19.398631887s) [noise=true , address=77.90.31.14:4661 , piece_id=B7IDSCDHBUXUIPSUT3HMQSCMPGBMYASZSQO2EJBXRR2DQHCLWGYA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.531443508 (582h31m19.402526553s) [noise=true , address=45.91.116.5:4297 , piece_id=PLUQ7Q32BTGYFQRGVJFAJWHPZCPQTC5RX5UX27S5HJ5NNJPLDMPQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.552364906 (582h31m19.423447951s) [noise=true , address=45.91.116.14:4190 , piece_id=FIL6AIXGE7DE2M4QBVSIO53KGY43VFPZQ2FTXSWJFPJU4VZCKUXQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.564056769 (582h31m19.435139814s) [noise=true , address=45.91.116.10:4084 , piece_id=5BFUYVGIFOYXC6RCPU7VBUM36LSM32TVBLS3W54OWFNR4SM6GCSA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: <nil> Dec 29 22:00:19.564087016 (582h31m19.435170051s) [seg_pos={0 0}, upload=510642]: Segment reached optimal threshold of 13 pieces. Dec 29 22:00:19.564294431 (582h31m19.435377466s) [noise=true , address=145.79.227.3:4150 , piece_id=HS4S5UDYD2VWI2N4MH6EWEYQ5G6KCFCJBDZR7AVWPXRYALFXP4FQ, seg_pos={0 0}, upload=510642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK): protocol: expected piece hash; context canceled; EOF Dec 29 22:00:19.564302366 (582h31m19.435385411s) [noise=true , address=145.79.227.3:4150 , piece_id=HS4S5UDYD2VWI2N4MH6EWEYQ5G6KCFCJBDZR7AVWPXRYALFXP4FQ, seg_pos={0 0}, upload=510642]: Long tail cancellation due to optimal threshold Dec 29 22:00:19.564388932 (582h31m19.435471977s) [noise=true , address=45.91.116.13:4154 , piece_id=FE4NFODQ3TWGFXARFYIL64TKK6X7H6PPLIG6LO44LKNAZOEU7DJA, seg_pos={0 0}, upload=510642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Dec 29 22:00:19.564401876 (582h31m19.435484911s) [noise=true , address=45.91.116.13:4154 , piece_id=FE4NFODQ3TWGFXARFYIL64TKK6X7H6PPLIG6LO44LKNAZOEU7DJA, seg_pos={0 0}, upload=510642]: Long tail cancellation due to optimal threshold Dec 29 22:00:19.564558945 (582h31m19.435641981s) [noise=true , address=145.79.227.2:4115 , piece_id=P2AS5B3VYO2K2AWSM5W4BXAXBHSXQQWM7JOXFPOV64NTKDFLT62Q, seg_pos={0 0}, upload=510642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: write tcp 188.166.225.170:38246->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:38246->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38246->145.79.227.2:4115: use of closed network connection Dec 29 22:00:19.564570738 (582h31m19.435653783s) [noise=true , address=145.79.227.2:4115 , piece_id=P2AS5B3VYO2K2AWSM5W4BXAXBHSXQQWM7JOXFPOV64NTKDFLT62Q, seg_pos={0 0}, upload=510642]: Long tail cancellation due to optimal threshold Dec 29 22:00:19.564601426 (582h31m19.435684471s) [seg_pos={0 0}, upload=510642]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 22:00:19.564683973 (582h31m19.435767018s) [upload=510642]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 22:00:19.564690004 (582h31m19.435773049s) [upload=510642]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 22:00:19.871860472 (582h31m19.742943527s) [upload=510642]: Done uploading segments... Dec 29 22:00:19.871892413 (582h31m19.742975468s) [upload=510642]: Upload finished. err: <nil> Duration: 6.129695689s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)