Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [X....XX........X...X.......X.....X...XX.X...........X..X...X] duration: 30.003691943s trace: https://jaeger.dev.storj.io/trace/6abc868c332118e8 FAIL (2025-12-31 21:30:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: UAQCQRG3R4FGJWJFMGMP54D24V6IRQAIZNZKOMSQWTVP6PK3E37A) context deadline exceeded; piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: FLWRDI6JSQ67VYWCQ46U2MUIMTHTZCYHM5ATAJQBTV3S2ZQKRYSA) context deadline exceeded ======LOGS====== Dec 31 21:30:11.077196959 (630h1m10.948279994s) [upload=552074]: Starting upload Dec 31 21:30:11.077255921 (630h1m10.948338966s) [upload=552074]: Done starting upload Dec 31 21:30:11.077446544 (630h1m10.948529599s) [upload=552074]: Uploading segments... Dec 31 21:30:11.077555401 (630h1m10.948638436s) [upload=552074]: Got next segment. Inline: false Dec 31 21:30:11.077606569 (630h1m10.948689654s) [upload=552074]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 21:30:11.077613141 (630h1m10.948696186s) [upload=552074]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 21:30:11.329286120 (630h1m11.200369165s) [seg_pos={0 0}, upload=552074]: Begin upload segment... Dec 31 21:30:11.329340124 (630h1m11.200423169s) [seg_pos={0 0}, upload=552074]: Done begin upload segment. Dec 31 21:30:11.329371021 (630h1m11.200454067s) [noise=true , address=145.79.227.38:4978 , piece_id=QGKK3EFSUPE3NV746UP6B44SZPWNTI52VRY2XX4YZS5YQ4S7KOHA, seg_pos={0 0}, upload=552074]: Uploading piece... Dec 31 21:30:11.329449391 (630h1m11.200532437s) [noise=true , address=145.79.227.17:4972 , piece_id=2UTWXJD4OBJFWCTF2TXHDLIOWXFXTW2MA5XO3WZ4ERXHRWEGPTQQ, seg_pos={0 0}, upload=552074]: Uploading piece... Dec 31 21:30:11.329993477 (630h1m11.201076512s) [noise=true , address=145.79.227.9:4366 , piece_id=KF6CHRTJZSGDM4NSIUIGYRQL25MQA4V75E2WNLQBJWNXNFLQFDBA, seg_pos={0 0}, upload=552074]: Uploading piece... Dec 31 21:30:11.330379692 (630h1m11.201462737s) [noise=true , address=145.79.227.4:4185 , piece_id=LSKVIQQR5PGLZOIW6DXNNGIDL4TB3AV2WTUNPSOPJZVSTLDJ636Q, seg_pos={0 0}, upload=552074]: Uploading piece... Dec 31 21:30:11.330850489 (630h1m11.201933534s) [noise=true , address=145.79.227.49:4337 , piece_id=N4R65AT7KP4CPB3R4FM3DZ4PGP2FSMOY3L5M7HOBNRASL2M3P7RA, seg_pos={0 0}, upload=552074]: Uploading piece... Dec 31 21:30:11.331261181 (630h1m11.202344226s) [noise=true , address=145.79.227.51:4409 , piece_id=ZNL7DVBGIATO2HNXOZQ43WIPVBKP7RSIORWF4OGT4TOTHSLF6O7A, seg_pos={0 0}, upload=552074]: Uploading piece... Dec 31 21:30:11.331693935 (630h1m11.20277697s) [noise=true , address=145.79.227.19:4077 , piece_id=KXWDKGATYO6T4QMBCUJG2TMQ62723ADU4PEEZ4WAVT2GYGYB46IQ, seg_pos={0 0}, upload=552074]: Uploading piece... Dec 31 21:30:11.332166105 (630h1m11.20324915s) [noise=true , address=145.79.227.39:4015 , piece_id=CL67SRJPBAPRU73AETYDLARBNEMO5I75QJCPLTRXWGOKYNMVKQUA, seg_pos={0 0}, upload=552074]: Uploading piece... Dec 31 21:30:11.332811763 (630h1m11.203894808s) [noise=true , address=145.79.227.29:4942 , piece_id=4JIRKT7BWU3JXMBVCRWLLWRFZSOJETO3TRA4I5IIJBXNL5TVDYFA, seg_pos={0 0}, upload=552074]: Uploading piece... Dec 31 21:30:11.334341076 (630h1m11.205424111s) [noise=true , address=145.79.227.36:4518 , piece_id=XIUEYTTWGCJSSAW5G2NA7XGQ6D3Q7WSIQXTCTJP5UDXODTBTK5EQ, seg_pos={0 0}, upload=552074]: Uploading piece... Dec 31 21:30:11.335805614 (630h1m11.206888659s) [noise=true , address=145.79.227.7:4294 , piece_id=WW3NQKXXJI2JZKLCREJTAQP6UWMQUEAU4RT4KPVHQ6KEHQ24WFRA, seg_pos={0 0}, upload=552074]: Uploading piece... Dec 31 21:30:11.337395412 (630h1m11.208478457s) [noise=true , address=45.91.116.5:4300 , piece_id=UAQCQRG3R4FGJWJFMGMP54D24V6IRQAIZNZKOMSQWTVP6PK3E37A, seg_pos={0 0}, upload=552074]: Uploading piece... Dec 31 21:30:11.339189538 (630h1m11.210272573s) [noise=true , address=77.90.31.11:4554 , piece_id=FLWRDI6JSQ67VYWCQ46U2MUIMTHTZCYHM5ATAJQBTV3S2ZQKRYSA, seg_pos={0 0}, upload=552074]: Uploading piece... Dec 31 21:30:11.341569820 (630h1m11.212652865s) [noise=true , address=77.90.31.12:4590 , piece_id=7E2KORSASWJYSAAB6YLZ4BSM4QMBRSBC37KZBMY3LNZ546M2WJ7Q, seg_pos={0 0}, upload=552074]: Uploading piece... Dec 31 21:30:11.343738560 (630h1m11.214821605s) [noise=true , address=145.79.227.41:4085 , piece_id=AJSQ5A2DHZODYEUECBQGGMYOUCEB3MZCGOJHMHOO6GCQY2C6KU5A, seg_pos={0 0}, upload=552074]: Uploading piece... Dec 31 21:30:11.345700265 (630h1m11.21678331s) [noise=true , address=145.79.227.50:4373 , piece_id=JS4ETDYFJR3A5K2FNLKUXNSWCNDXY2JZBTIHESQWYWIF55WAVKAA, seg_pos={0 0}, upload=552074]: Uploading piece... Dec 31 21:30:13.011641963 (630h1m12.882724998s) [upload=552074]: Next returned nil segment Dec 31 21:30:13.011662241 (630h1m12.882745276s) [upload=552074]: Waiting for error group managing segments... Dec 31 21:30:13.313766207 (630h1m13.184849242s) [noise=true , address=145.79.227.19:4077 , piece_id=KXWDKGATYO6T4QMBCUJG2TMQ62723ADU4PEEZ4WAVT2GYGYB46IQ, seg_pos={0 0}, upload=552074]: Done uploading piece. err: <nil> Dec 31 21:30:13.316510132 (630h1m13.187593168s) [noise=true , address=145.79.227.36:4518 , piece_id=XIUEYTTWGCJSSAW5G2NA7XGQ6D3Q7WSIQXTCTJP5UDXODTBTK5EQ, seg_pos={0 0}, upload=552074]: Done uploading piece. err: <nil> Dec 31 21:30:13.322272715 (630h1m13.19335575s) [noise=true , address=145.79.227.17:4972 , piece_id=2UTWXJD4OBJFWCTF2TXHDLIOWXFXTW2MA5XO3WZ4ERXHRWEGPTQQ, seg_pos={0 0}, upload=552074]: Done uploading piece. err: <nil> Dec 31 21:30:13.325383579 (630h1m13.196466614s) [noise=true , address=145.79.227.41:4085 , piece_id=AJSQ5A2DHZODYEUECBQGGMYOUCEB3MZCGOJHMHOO6GCQY2C6KU5A, seg_pos={0 0}, upload=552074]: Done uploading piece. err: <nil> Dec 31 21:30:13.333010972 (630h1m13.204094007s) [noise=true , address=145.79.227.7:4294 , piece_id=WW3NQKXXJI2JZKLCREJTAQP6UWMQUEAU4RT4KPVHQ6KEHQ24WFRA, seg_pos={0 0}, upload=552074]: Done uploading piece. err: <nil> Dec 31 21:30:13.333935162 (630h1m13.205018197s) [noise=true , address=145.79.227.50:4373 , piece_id=JS4ETDYFJR3A5K2FNLKUXNSWCNDXY2JZBTIHESQWYWIF55WAVKAA, seg_pos={0 0}, upload=552074]: Done uploading piece. err: <nil> Dec 31 21:30:13.334144200 (630h1m13.205227235s) [noise=true , address=145.79.227.51:4409 , piece_id=ZNL7DVBGIATO2HNXOZQ43WIPVBKP7RSIORWF4OGT4TOTHSLF6O7A, seg_pos={0 0}, upload=552074]: Done uploading piece. err: <nil> Dec 31 21:30:13.346856747 (630h1m13.217939792s) [noise=true , address=145.79.227.38:4978 , piece_id=QGKK3EFSUPE3NV746UP6B44SZPWNTI52VRY2XX4YZS5YQ4S7KOHA, seg_pos={0 0}, upload=552074]: Done uploading piece. err: <nil> Dec 31 21:30:13.368650054 (630h1m13.239733089s) [noise=true , address=145.79.227.39:4015 , piece_id=CL67SRJPBAPRU73AETYDLARBNEMO5I75QJCPLTRXWGOKYNMVKQUA, seg_pos={0 0}, upload=552074]: Done uploading piece. err: <nil> Dec 31 21:30:13.389861604 (630h1m13.260944639s) [noise=true , address=145.79.227.9:4366 , piece_id=KF6CHRTJZSGDM4NSIUIGYRQL25MQA4V75E2WNLQBJWNXNFLQFDBA, seg_pos={0 0}, upload=552074]: Done uploading piece. err: <nil> Dec 31 21:30:13.400650377 (630h1m13.271733412s) [noise=true , address=145.79.227.4:4185 , piece_id=LSKVIQQR5PGLZOIW6DXNNGIDL4TB3AV2WTUNPSOPJZVSTLDJ636Q, seg_pos={0 0}, upload=552074]: Done uploading piece. err: <nil> Dec 31 21:30:13.523553451 (630h1m13.394636486s) [noise=true , address=77.90.31.11:4554 , piece_id=FLWRDI6JSQ67VYWCQ46U2MUIMTHTZCYHM5ATAJQBTV3S2ZQKRYSA, seg_pos={0 0}, upload=552074]: Done uploading piece. err: <nil> Dec 31 21:30:13.628950400 (630h1m13.500033435s) [noise=true , address=45.91.116.5:4300 , piece_id=UAQCQRG3R4FGJWJFMGMP54D24V6IRQAIZNZKOMSQWTVP6PK3E37A, seg_pos={0 0}, upload=552074]: Done uploading piece. err: <nil> Dec 31 21:30:13.628970038 (630h1m13.500053073s) [seg_pos={0 0}, upload=552074]: Segment reached optimal threshold of 13 pieces. Dec 31 21:30:13.629174737 (630h1m13.500257782s) [noise=true , address=145.79.227.29:4942 , piece_id=4JIRKT7BWU3JXMBVCRWLLWRFZSOJETO3TRA4I5IIJBXNL5TVDYFA, seg_pos={0 0}, upload=552074]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7): protocol: expected piece hash; context canceled; EOF Dec 31 21:30:13.629181932 (630h1m13.500264987s) [noise=true , address=145.79.227.29:4942 , piece_id=4JIRKT7BWU3JXMBVCRWLLWRFZSOJETO3TRA4I5IIJBXNL5TVDYFA, seg_pos={0 0}, upload=552074]: Long tail cancellation due to optimal threshold Dec 31 21:30:13.629327909 (630h1m13.500410954s) [noise=true , address=145.79.227.49:4337 , piece_id=N4R65AT7KP4CPB3R4FM3DZ4PGP2FSMOY3L5M7HOBNRASL2M3P7RA, seg_pos={0 0}, upload=552074]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nTZFfzNCcVAjut7NXf9seifysJDfvXkS8aZSnXKfw6gh1TrjE): protocol: expected piece hash; context canceled; EOF Dec 31 21:30:13.629349510 (630h1m13.500432545s) [noise=true , address=145.79.227.49:4337 , piece_id=N4R65AT7KP4CPB3R4FM3DZ4PGP2FSMOY3L5M7HOBNRASL2M3P7RA, seg_pos={0 0}, upload=552074]: Long tail cancellation due to optimal threshold Dec 31 21:30:13.629400808 (630h1m13.500483843s) [noise=true , address=77.90.31.12:4590 , piece_id=7E2KORSASWJYSAAB6YLZ4BSM4QMBRSBC37KZBMY3LNZ546M2WJ7Q, seg_pos={0 0}, upload=552074]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 31 21:30:13.629409755 (630h1m13.50049279s) [noise=true , address=77.90.31.12:4590 , piece_id=7E2KORSASWJYSAAB6YLZ4BSM4QMBRSBC37KZBMY3LNZ546M2WJ7Q, seg_pos={0 0}, upload=552074]: Long tail cancellation due to optimal threshold Dec 31 21:30:13.629454830 (630h1m13.500537865s) [seg_pos={0 0}, upload=552074]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 21:30:13.629522750 (630h1m13.500605785s) [upload=552074]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 21:30:13.629528522 (630h1m13.500611567s) [upload=552074]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 21:30:13.923238034 (630h1m13.794321089s) [upload=552074]: Done uploading segments... Dec 31 21:30:13.923263412 (630h1m13.794346448s) [upload=552074]: Upload finished. err: <nil> Duration: 30.003718924s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.X....XX........X...X.......X.....X...XX.X...........X..X...] duration: 6.21274191s trace: https://jaeger.dev.storj.io/trace/5d999dac32a7be3e FAIL (2025-12-31 21:29:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ts... Dec 31 21:26:19.065719105 (629h57m18.93680214s) [upload=552020]: Got next segment. Inline: false Dec 31 21:26:19.065750896 (629h57m18.936833951s) [upload=552020]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 21:26:19.065755996 (629h57m18.936839031s) [upload=552020]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 21:26:19.317558331 (629h57m19.188641376s) [seg_pos={0 0}, upload=552020]: Begin upload segment... Dec 31 21:26:19.317624367 (629h57m19.188707402s) [seg_pos={0 0}, upload=552020]: Done begin upload segment. Dec 31 21:26:19.317666858 (629h57m19.188749893s) [noise=true , address=145.79.227.31:4797 , piece_id=EY2CW27J47HOBSSWTXFZK3XGYGBZH3W7275DTW7IOGFZSCUENORA, seg_pos={0 0}, upload=552020]: Uploading piece... Dec 31 21:26:19.317697767 (629h57m19.188780812s) [noise=true , address=45.91.116.6:4975 , piece_id=DUMPZKKIOIZ35UNUV4SFISNWZSLN6NIO5ZWKURWMQFX5PUGTJSIQ, seg_pos={0 0}, upload=552020]: Uploading piece... Dec 31 21:26:19.318149457 (629h57m19.189232492s) [noise=true , address=145.79.227.7:4294 , piece_id=H6QSIINGNE3T6F7H6MLRN7CIX3X5WR4UIZKZPOFAZVLZUG3WICGQ, seg_pos={0 0}, upload=552020]: Uploading piece... Dec 31 21:26:19.318482261 (629h57m19.189565296s) [noise=true , address=45.91.116.7:4010 , piece_id=PZWO3Q2S6ZXWGDGAMSSYQBTCQCGBV5ZXIZB5OAOH3XAGCWJLYUQQ, seg_pos={0 0}, upload=552020]: Uploading piece... Dec 31 21:26:19.318885508 (629h57m19.189968553s) [noise=true , address=145.79.227.4:4187 , piece_id=MQET2M3D4QPHG4IKPHSMSYMWR3ZUW5D2RYCHEOABF5OC2TRBS7MA, seg_pos={0 0}, upload=552020]: Uploading piece... Dec 31 21:26:19.319188986 (629h57m19.190272021s) [noise=true , address=145.79.227.36:4518 , piece_id=PUHWY66CUZ5ONPFNC2LRZ52BFVSS3SX6ZXLWON2UQYQJD3TJ5MEQ, seg_pos={0 0}, upload=552020]: Uploading piece... Dec 31 21:26:19.319494178 (629h57m19.190577213s) [noise=true , address=145.79.227.48:4303 , piece_id=W27HEVJ2ZHGTEASHBO6CSOMRRSFHA5P6LDIQHWIQZWLEQYJCWM2Q, seg_pos={0 0}, upload=552020]: Uploading piece... Dec 31 21:26:19.319796994 (629h57m19.190880029s) [noise=true , address=145.79.227.39:4014 , piece_id=FALGJMKNGX2YR2JU4CEEVWSQSLR4HHZMCEIJS2XVXFK5MUIABZ6Q, seg_pos={0 0}, upload=552020]: Uploading piece... Dec 31 21:26:19.320970308 (629h57m19.192053343s) [noise=true , address=145.79.227.37:4565 , piece_id=DBX2EW7P6THCWP3X7WO4J7XXZ4MRLMY7CBQGX5XNA2K3XSMRHBMQ, seg_pos={0 0}, upload=552020]: Uploading piece... Dec 31 21:26:19.321397602 (629h57m19.192480647s) [noise=true , address=45.91.116.9:4263 , piece_id=JFEDL7EWCAWO24MXEFZTEJKPV2SQV35PYE43NDXSVY5SBY24NQHA, seg_pos={0 0}, upload=552020]: Uploading piece... Dec 31 21:26:19.321770171 (629h57m19.192853206s) [noise=true , address=45.91.116.14:4192 , piece_id=Q7ZLWISNLI7KN6IGWPDTHVRZ7L3QSQGJHXBIDUOYTOSLSNOG4P7Q, seg_pos={0 0}, upload=552020]: Uploading piece... Dec 31 21:26:19.322144805 (629h57m19.19322784s) [noise=true , address=145.79.227.12:4767 , piece_id=PV7LS5KCC4FZXJSVA6ZMDDATGLZPVA2VQMMBBEMMMWNHGQ6X2ROA, seg_pos={0 0}, upload=552020]: Uploading piece... Dec 31 21:26:19.322185412 (629h57m19.193268447s) [noise=true , address=145.79.227.45:4229 , piece_id=VJWIBL6ZETNPIVRNRUMM7PV56QW4RU52JYCZI2AB35RKA7JR3DWA, seg_pos={0 0}, upload=552020]: Uploading piece... Dec 31 21:26:19.322577158 (629h57m19.193660193s) [noise=true , address=45.91.116.12:4225 , piece_id=45UVA5PF3F62CFP74BQM42NEMB6M64TJ5CXRSG5QFJIYIKXQVX4Q, seg_pos={0 0}, upload=552020]: Uploading piece... Dec 31 21:26:19.323043807 (629h57m19.194126852s) [noise=true , address=145.79.227.32:4836 , piece_id=NVAEEGAKTDIXC3PX6XQCQLZ6Q7MIUVITKMQMBCHXRX2ZJQ6OKS6Q, seg_pos={0 0}, upload=552020]: Uploading piece... Dec 31 21:26:19.323453035 (629h57m19.19453607s) [noise=true , address=145.79.227.15:4874 , piece_id=EGNFR25OH7YC3XFGUUYLUT4HKGFUCVZMCMEFLECNMHADKZCKNBDQ, seg_pos={0 0}, upload=552020]: Uploading piece... Dec 31 21:26:20.943196324 (629h57m20.814279359s) [upload=552020]: Next returned nil segment Dec 31 21:26:20.943261358 (629h57m20.814344403s) [upload=552020]: Waiting for error group managing segments... Dec 31 21:26:21.311565930 (629h57m21.182648965s) [noise=true , address=145.79.227.7:4294 , piece_id=H6QSIINGNE3T6F7H6MLRN7CIX3X5WR4UIZKZPOFAZVLZUG3WICGQ, seg_pos={0 0}, upload=552020]: Done uploading piece. err: <nil> Dec 31 21:26:21.316669898 (629h57m21.187752933s) [noise=true , address=145.79.227.12:4767 , piece_id=PV7LS5KCC4FZXJSVA6ZMDDATGLZPVA2VQMMBBEMMMWNHGQ6X2ROA, seg_pos={0 0}, upload=552020]: Done uploading piece. err: <nil> Dec 31 21:26:21.320222032 (629h57m21.191305087s) [noise=true , address=145.79.227.39:4014 , piece_id=FALGJMKNGX2YR2JU4CEEVWSQSLR4HHZMCEIJS2XVXFK5MUIABZ6Q, seg_pos={0 0}, upload=552020]: Done uploading piece. err: <nil> Dec 31 21:26:21.325999112 (629h57m21.197082147s) [noise=true , address=145.79.227.37:4565 , piece_id=DBX2EW7P6THCWP3X7WO4J7XXZ4MRLMY7CBQGX5XNA2K3XSMRHBMQ, seg_pos={0 0}, upload=552020]: Done uploading piece. err: <nil> Dec 31 21:26:21.329606841 (629h57m21.200689876s) [noise=true , address=145.79.227.15:4874 , piece_id=EGNFR25OH7YC3XFGUUYLUT4HKGFUCVZMCMEFLECNMHADKZCKNBDQ, seg_pos={0 0}, upload=552020]: Done uploading piece. err: <nil> Dec 31 21:26:21.329954333 (629h57m21.201037368s) [noise=true , address=145.79.227.36:4518 , piece_id=PUHWY66CUZ5ONPFNC2LRZ52BFVSS3SX6ZXLWON2UQYQJD3TJ5MEQ, seg_pos={0 0}, upload=552020]: Done uploading piece. err: <nil> Dec 31 21:26:21.332051476 (629h57m21.203134511s) [noise=true , address=145.79.227.31:4797 , piece_id=EY2CW27J47HOBSSWTXFZK3XGYGBZH3W7275DTW7IOGFZSCUENORA, seg_pos={0 0}, upload=552020]: Done uploading piece. err: <nil> Dec 31 21:26:21.393201484 (629h57m21.264284529s) [noise=true , address=145.79.227.45:4229 , piece_id=VJWIBL6ZETNPIVRNRUMM7PV56QW4RU52JYCZI2AB35RKA7JR3DWA, seg_pos={0 0}, upload=552020]: Done uploading piece. err: <nil> Dec 31 21:26:21.396482262 (629h57m21.267565307s) [noise=true , address=145.79.227.4:4187 , piece_id=MQET2M3D4QPHG4IKPHSMSYMWR3ZUW5D2RYCHEOABF5OC2TRBS7MA, seg_pos={0 0}, upload=552020]: Done uploading piece. err: <nil> Dec 31 21:26:21.406883036 (629h57m21.277966071s) [noise=true , address=145.79.227.32:4836 , piece_id=NVAEEGAKTDIXC3PX6XQCQLZ6Q7MIUVITKMQMBCHXRX2ZJQ6OKS6Q, seg_pos={0 0}, upload=552020]: Done uploading piece. err: <nil> Dec 31 21:26:21.607595369 (629h57m21.478678404s) [noise=true , address=45.91.116.7:4010 , piece_id=PZWO3Q2S6ZXWGDGAMSSYQBTCQCGBV5ZXIZB5OAOH3XAGCWJLYUQQ, seg_pos={0 0}, upload=552020]: Done uploading piece. err: <nil> Dec 31 21:26:21.619681403 (629h57m21.490764438s) [noise=true , address=45.91.116.12:4225 , piece_id=45UVA5PF3F62CFP74BQM42NEMB6M64TJ5CXRSG5QFJIYIKXQVX4Q, seg_pos={0 0}, upload=552020]: Done uploading piece. err: <nil> Dec 31 21:26:21.632491554 (629h57m21.503574589s) [noise=true , address=45.91.116.14:4192 , piece_id=Q7ZLWISNLI7KN6IGWPDTHVRZ7L3QSQGJHXBIDUOYTOSLSNOG4P7Q, seg_pos={0 0}, upload=552020]: Done uploading piece. err: <nil> Dec 31 21:26:21.632510571 (629h57m21.503593606s) [seg_pos={0 0}, upload=552020]: Segment reached optimal threshold of 13 pieces. Dec 31 21:26:21.632676006 (629h57m21.503759051s) [noise=true , address=45.91.116.6:4975 , piece_id=DUMPZKKIOIZ35UNUV4SFISNWZSLN6NIO5ZWKURWMQFX5PUGTJSIQ, seg_pos={0 0}, upload=552020]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: write tcp 188.166.225.170:49332->45.91.116.6:4975: use of closed network connection; write tcp 188.166.225.170:49332->45.91.116.6:4975: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49332->45.91.116.6:4975: use of closed network connection Dec 31 21:26:21.632681406 (629h57m21.503764441s) [noise=true , address=45.91.116.6:4975 , piece_id=DUMPZKKIOIZ35UNUV4SFISNWZSLN6NIO5ZWKURWMQFX5PUGTJSIQ, seg_pos={0 0}, upload=552020]: Long tail cancellation due to optimal threshold Dec 31 21:26:21.632687738 (629h57m21.503770783s) [noise=true , address=145.79.227.48:4303 , piece_id=W27HEVJ2ZHGTEASHBO6CSOMRRSFHA5P6LDIQHWIQZWLEQYJCWM2Q, seg_pos={0 0}, upload=552020]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gJKrCBLJWN1SBKibfeqBtcqqiv4veKTGmtnycjZyb3VN9kwoT): protocol: write tcp 188.166.225.170:54402->145.79.227.48:4303: use of closed network connection; write tcp 188.166.225.170:54402->145.79.227.48:4303: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54402->145.79.227.48:4303: use of closed network connection Dec 31 21:26:21.632698188 (629h57m21.503781223s) [noise=true , address=145.79.227.48:4303 , piece_id=W27HEVJ2ZHGTEASHBO6CSOMRRSFHA5P6LDIQHWIQZWLEQYJCWM2Q, seg_pos={0 0}, upload=552020]: Long tail cancellation due to optimal threshold Dec 31 21:26:21.632864845 (629h57m21.50394788s) [noise=true , address=45.91.116.9:4263 , piece_id=JFEDL7EWCAWO24MXEFZTEJKPV2SQV35PYE43NDXSVY5SBY24NQHA, seg_pos={0 0}, upload=552020]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 31 21:26:21.632870346 (629h57m21.503953381s) [noise=true , address=45.91.116.9:4263 , piece_id=JFEDL7EWCAWO24MXEFZTEJKPV2SQV35PYE43NDXSVY5SBY24NQHA, seg_pos={0 0}, upload=552020]: Long tail cancellation due to optimal threshold Dec 31 21:26:21.632884203 (629h57m21.503967238s) [seg_pos={0 0}, upload=552020]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 21:26:21.632945068 (629h57m21.504028103s) [upload=552020]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 21:26:21.632949466 (629h57m21.504032501s) [upload=552020]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 21:26:21.929926801 (629h57m21.801009846s) [upload=552020]: Done uploading segments... Dec 31 21:26:21.929948742 (629h57m21.801031777s) [upload=552020]: Upload finished. err: <nil> Duration: 6.212766016s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [..X....XX........X...X.......X.....X...XX.X...........X..X..] duration: 5.833605739s trace: https://jaeger.dev.storj.io/trace/5d999dac32a7be3e FAIL (2025-12-31 21:28:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ts... Dec 31 21:26:19.065719105 (629h57m18.93680214s) [upload=552020]: Got next segment. Inline: false Dec 31 21:26:19.065750896 (629h57m18.936833951s) [upload=552020]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 21:26:19.065755996 (629h57m18.936839031s) [upload=552020]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 21:26:19.317558331 (629h57m19.188641376s) [seg_pos={0 0}, upload=552020]: Begin upload segment... Dec 31 21:26:19.317624367 (629h57m19.188707402s) [seg_pos={0 0}, upload=552020]: Done begin upload segment. Dec 31 21:26:19.317666858 (629h57m19.188749893s) [noise=true , address=145.79.227.31:4797 , piece_id=EY2CW27J47HOBSSWTXFZK3XGYGBZH3W7275DTW7IOGFZSCUENORA, seg_pos={0 0}, upload=552020]: Uploading piece... Dec 31 21:26:19.317697767 (629h57m19.188780812s) [noise=true , address=45.91.116.6:4975 , piece_id=DUMPZKKIOIZ35UNUV4SFISNWZSLN6NIO5ZWKURWMQFX5PUGTJSIQ, seg_pos={0 0}, upload=552020]: Uploading piece... Dec 31 21:26:19.318149457 (629h57m19.189232492s) [noise=true , address=145.79.227.7:4294 , piece_id=H6QSIINGNE3T6F7H6MLRN7CIX3X5WR4UIZKZPOFAZVLZUG3WICGQ, seg_pos={0 0}, upload=552020]: Uploading piece... Dec 31 21:26:19.318482261 (629h57m19.189565296s) [noise=true , address=45.91.116.7:4010 , piece_id=PZWO3Q2S6ZXWGDGAMSSYQBTCQCGBV5ZXIZB5OAOH3XAGCWJLYUQQ, seg_pos={0 0}, upload=552020]: Uploading piece... Dec 31 21:26:19.318885508 (629h57m19.189968553s) [noise=true , address=145.79.227.4:4187 , piece_id=MQET2M3D4QPHG4IKPHSMSYMWR3ZUW5D2RYCHEOABF5OC2TRBS7MA, seg_pos={0 0}, upload=552020]: Uploading piece... Dec 31 21:26:19.319188986 (629h57m19.190272021s) [noise=true , address=145.79.227.36:4518 , piece_id=PUHWY66CUZ5ONPFNC2LRZ52BFVSS3SX6ZXLWON2UQYQJD3TJ5MEQ, seg_pos={0 0}, upload=552020]: Uploading piece... Dec 31 21:26:19.319494178 (629h57m19.190577213s) [noise=true , address=145.79.227.48:4303 , piece_id=W27HEVJ2ZHGTEASHBO6CSOMRRSFHA5P6LDIQHWIQZWLEQYJCWM2Q, seg_pos={0 0}, upload=552020]: Uploading piece... Dec 31 21:26:19.319796994 (629h57m19.190880029s) [noise=true , address=145.79.227.39:4014 , piece_id=FALGJMKNGX2YR2JU4CEEVWSQSLR4HHZMCEIJS2XVXFK5MUIABZ6Q, seg_pos={0 0}, upload=552020]: Uploading piece... Dec 31 21:26:19.320970308 (629h57m19.192053343s) [noise=true , address=145.79.227.37:4565 , piece_id=DBX2EW7P6THCWP3X7WO4J7XXZ4MRLMY7CBQGX5XNA2K3XSMRHBMQ, seg_pos={0 0}, upload=552020]: Uploading piece... Dec 31 21:26:19.321397602 (629h57m19.192480647s) [noise=true , address=45.91.116.9:4263 , piece_id=JFEDL7EWCAWO24MXEFZTEJKPV2SQV35PYE43NDXSVY5SBY24NQHA, seg_pos={0 0}, upload=552020]: Uploading piece... Dec 31 21:26:19.321770171 (629h57m19.192853206s) [noise=true , address=45.91.116.14:4192 , piece_id=Q7ZLWISNLI7KN6IGWPDTHVRZ7L3QSQGJHXBIDUOYTOSLSNOG4P7Q, seg_pos={0 0}, upload=552020]: Uploading piece... Dec 31 21:26:19.322144805 (629h57m19.19322784s) [noise=true , address=145.79.227.12:4767 , piece_id=PV7LS5KCC4FZXJSVA6ZMDDATGLZPVA2VQMMBBEMMMWNHGQ6X2ROA, seg_pos={0 0}, upload=552020]: Uploading piece... Dec 31 21:26:19.322185412 (629h57m19.193268447s) [noise=true , address=145.79.227.45:4229 , piece_id=VJWIBL6ZETNPIVRNRUMM7PV56QW4RU52JYCZI2AB35RKA7JR3DWA, seg_pos={0 0}, upload=552020]: Uploading piece... Dec 31 21:26:19.322577158 (629h57m19.193660193s) [noise=true , address=45.91.116.12:4225 , piece_id=45UVA5PF3F62CFP74BQM42NEMB6M64TJ5CXRSG5QFJIYIKXQVX4Q, seg_pos={0 0}, upload=552020]: Uploading piece... Dec 31 21:26:19.323043807 (629h57m19.194126852s) [noise=true , address=145.79.227.32:4836 , piece_id=NVAEEGAKTDIXC3PX6XQCQLZ6Q7MIUVITKMQMBCHXRX2ZJQ6OKS6Q, seg_pos={0 0}, upload=552020]: Uploading piece... Dec 31 21:26:19.323453035 (629h57m19.19453607s) [noise=true , address=145.79.227.15:4874 , piece_id=EGNFR25OH7YC3XFGUUYLUT4HKGFUCVZMCMEFLECNMHADKZCKNBDQ, seg_pos={0 0}, upload=552020]: Uploading piece... Dec 31 21:26:20.943196324 (629h57m20.814279359s) [upload=552020]: Next returned nil segment Dec 31 21:26:20.943261358 (629h57m20.814344403s) [upload=552020]: Waiting for error group managing segments... Dec 31 21:26:21.311565930 (629h57m21.182648965s) [noise=true , address=145.79.227.7:4294 , piece_id=H6QSIINGNE3T6F7H6MLRN7CIX3X5WR4UIZKZPOFAZVLZUG3WICGQ, seg_pos={0 0}, upload=552020]: Done uploading piece. err: <nil> Dec 31 21:26:21.316669898 (629h57m21.187752933s) [noise=true , address=145.79.227.12:4767 , piece_id=PV7LS5KCC4FZXJSVA6ZMDDATGLZPVA2VQMMBBEMMMWNHGQ6X2ROA, seg_pos={0 0}, upload=552020]: Done uploading piece. err: <nil> Dec 31 21:26:21.320222032 (629h57m21.191305087s) [noise=true , address=145.79.227.39:4014 , piece_id=FALGJMKNGX2YR2JU4CEEVWSQSLR4HHZMCEIJS2XVXFK5MUIABZ6Q, seg_pos={0 0}, upload=552020]: Done uploading piece. err: <nil> Dec 31 21:26:21.325999112 (629h57m21.197082147s) [noise=true , address=145.79.227.37:4565 , piece_id=DBX2EW7P6THCWP3X7WO4J7XXZ4MRLMY7CBQGX5XNA2K3XSMRHBMQ, seg_pos={0 0}, upload=552020]: Done uploading piece. err: <nil> Dec 31 21:26:21.329606841 (629h57m21.200689876s) [noise=true , address=145.79.227.15:4874 , piece_id=EGNFR25OH7YC3XFGUUYLUT4HKGFUCVZMCMEFLECNMHADKZCKNBDQ, seg_pos={0 0}, upload=552020]: Done uploading piece. err: <nil> Dec 31 21:26:21.329954333 (629h57m21.201037368s) [noise=true , address=145.79.227.36:4518 , piece_id=PUHWY66CUZ5ONPFNC2LRZ52BFVSS3SX6ZXLWON2UQYQJD3TJ5MEQ, seg_pos={0 0}, upload=552020]: Done uploading piece. err: <nil> Dec 31 21:26:21.332051476 (629h57m21.203134511s) [noise=true , address=145.79.227.31:4797 , piece_id=EY2CW27J47HOBSSWTXFZK3XGYGBZH3W7275DTW7IOGFZSCUENORA, seg_pos={0 0}, upload=552020]: Done uploading piece. err: <nil> Dec 31 21:26:21.393201484 (629h57m21.264284529s) [noise=true , address=145.79.227.45:4229 , piece_id=VJWIBL6ZETNPIVRNRUMM7PV56QW4RU52JYCZI2AB35RKA7JR3DWA, seg_pos={0 0}, upload=552020]: Done uploading piece. err: <nil> Dec 31 21:26:21.396482262 (629h57m21.267565307s) [noise=true , address=145.79.227.4:4187 , piece_id=MQET2M3D4QPHG4IKPHSMSYMWR3ZUW5D2RYCHEOABF5OC2TRBS7MA, seg_pos={0 0}, upload=552020]: Done uploading piece. err: <nil> Dec 31 21:26:21.406883036 (629h57m21.277966071s) [noise=true , address=145.79.227.32:4836 , piece_id=NVAEEGAKTDIXC3PX6XQCQLZ6Q7MIUVITKMQMBCHXRX2ZJQ6OKS6Q, seg_pos={0 0}, upload=552020]: Done uploading piece. err: <nil> Dec 31 21:26:21.607595369 (629h57m21.478678404s) [noise=true , address=45.91.116.7:4010 , piece_id=PZWO3Q2S6ZXWGDGAMSSYQBTCQCGBV5ZXIZB5OAOH3XAGCWJLYUQQ, seg_pos={0 0}, upload=552020]: Done uploading piece. err: <nil> Dec 31 21:26:21.619681403 (629h57m21.490764438s) [noise=true , address=45.91.116.12:4225 , piece_id=45UVA5PF3F62CFP74BQM42NEMB6M64TJ5CXRSG5QFJIYIKXQVX4Q, seg_pos={0 0}, upload=552020]: Done uploading piece. err: <nil> Dec 31 21:26:21.632491554 (629h57m21.503574589s) [noise=true , address=45.91.116.14:4192 , piece_id=Q7ZLWISNLI7KN6IGWPDTHVRZ7L3QSQGJHXBIDUOYTOSLSNOG4P7Q, seg_pos={0 0}, upload=552020]: Done uploading piece. err: <nil> Dec 31 21:26:21.632510571 (629h57m21.503593606s) [seg_pos={0 0}, upload=552020]: Segment reached optimal threshold of 13 pieces. Dec 31 21:26:21.632676006 (629h57m21.503759051s) [noise=true , address=45.91.116.6:4975 , piece_id=DUMPZKKIOIZ35UNUV4SFISNWZSLN6NIO5ZWKURWMQFX5PUGTJSIQ, seg_pos={0 0}, upload=552020]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: write tcp 188.166.225.170:49332->45.91.116.6:4975: use of closed network connection; write tcp 188.166.225.170:49332->45.91.116.6:4975: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49332->45.91.116.6:4975: use of closed network connection Dec 31 21:26:21.632681406 (629h57m21.503764441s) [noise=true , address=45.91.116.6:4975 , piece_id=DUMPZKKIOIZ35UNUV4SFISNWZSLN6NIO5ZWKURWMQFX5PUGTJSIQ, seg_pos={0 0}, upload=552020]: Long tail cancellation due to optimal threshold Dec 31 21:26:21.632687738 (629h57m21.503770783s) [noise=true , address=145.79.227.48:4303 , piece_id=W27HEVJ2ZHGTEASHBO6CSOMRRSFHA5P6LDIQHWIQZWLEQYJCWM2Q, seg_pos={0 0}, upload=552020]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gJKrCBLJWN1SBKibfeqBtcqqiv4veKTGmtnycjZyb3VN9kwoT): protocol: write tcp 188.166.225.170:54402->145.79.227.48:4303: use of closed network connection; write tcp 188.166.225.170:54402->145.79.227.48:4303: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54402->145.79.227.48:4303: use of closed network connection Dec 31 21:26:21.632698188 (629h57m21.503781223s) [noise=true , address=145.79.227.48:4303 , piece_id=W27HEVJ2ZHGTEASHBO6CSOMRRSFHA5P6LDIQHWIQZWLEQYJCWM2Q, seg_pos={0 0}, upload=552020]: Long tail cancellation due to optimal threshold Dec 31 21:26:21.632864845 (629h57m21.50394788s) [noise=true , address=45.91.116.9:4263 , piece_id=JFEDL7EWCAWO24MXEFZTEJKPV2SQV35PYE43NDXSVY5SBY24NQHA, seg_pos={0 0}, upload=552020]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 31 21:26:21.632870346 (629h57m21.503953381s) [noise=true , address=45.91.116.9:4263 , piece_id=JFEDL7EWCAWO24MXEFZTEJKPV2SQV35PYE43NDXSVY5SBY24NQHA, seg_pos={0 0}, upload=552020]: Long tail cancellation due to optimal threshold Dec 31 21:26:21.632884203 (629h57m21.503967238s) [seg_pos={0 0}, upload=552020]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 21:26:21.632945068 (629h57m21.504028103s) [upload=552020]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 21:26:21.632949466 (629h57m21.504032501s) [upload=552020]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 21:26:21.929926801 (629h57m21.801009846s) [upload=552020]: Done uploading segments... Dec 31 21:26:21.929948742 (629h57m21.801031777s) [upload=552020]: Upload finished. err: <nil> Duration: 5.833636999s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [...X....XX........X...X.......X.....X...XX.X...........X..X.] duration: 7.528604747s trace: https://jaeger.dev.storj.io/trace/5d999dac32a7be3e FAIL (2025-12-31 21:27:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ts... Dec 31 21:26:19.065719105 (629h57m18.93680214s) [upload=552020]: Got next segment. Inline: false Dec 31 21:26:19.065750896 (629h57m18.936833951s) [upload=552020]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 21:26:19.065755996 (629h57m18.936839031s) [upload=552020]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 21:26:19.317558331 (629h57m19.188641376s) [seg_pos={0 0}, upload=552020]: Begin upload segment... Dec 31 21:26:19.317624367 (629h57m19.188707402s) [seg_pos={0 0}, upload=552020]: Done begin upload segment. Dec 31 21:26:19.317666858 (629h57m19.188749893s) [noise=true , address=145.79.227.31:4797 , piece_id=EY2CW27J47HOBSSWTXFZK3XGYGBZH3W7275DTW7IOGFZSCUENORA, seg_pos={0 0}, upload=552020]: Uploading piece... Dec 31 21:26:19.317697767 (629h57m19.188780812s) [noise=true , address=45.91.116.6:4975 , piece_id=DUMPZKKIOIZ35UNUV4SFISNWZSLN6NIO5ZWKURWMQFX5PUGTJSIQ, seg_pos={0 0}, upload=552020]: Uploading piece... Dec 31 21:26:19.318149457 (629h57m19.189232492s) [noise=true , address=145.79.227.7:4294 , piece_id=H6QSIINGNE3T6F7H6MLRN7CIX3X5WR4UIZKZPOFAZVLZUG3WICGQ, seg_pos={0 0}, upload=552020]: Uploading piece... Dec 31 21:26:19.318482261 (629h57m19.189565296s) [noise=true , address=45.91.116.7:4010 , piece_id=PZWO3Q2S6ZXWGDGAMSSYQBTCQCGBV5ZXIZB5OAOH3XAGCWJLYUQQ, seg_pos={0 0}, upload=552020]: Uploading piece... Dec 31 21:26:19.318885508 (629h57m19.189968553s) [noise=true , address=145.79.227.4:4187 , piece_id=MQET2M3D4QPHG4IKPHSMSYMWR3ZUW5D2RYCHEOABF5OC2TRBS7MA, seg_pos={0 0}, upload=552020]: Uploading piece... Dec 31 21:26:19.319188986 (629h57m19.190272021s) [noise=true , address=145.79.227.36:4518 , piece_id=PUHWY66CUZ5ONPFNC2LRZ52BFVSS3SX6ZXLWON2UQYQJD3TJ5MEQ, seg_pos={0 0}, upload=552020]: Uploading piece... Dec 31 21:26:19.319494178 (629h57m19.190577213s) [noise=true , address=145.79.227.48:4303 , piece_id=W27HEVJ2ZHGTEASHBO6CSOMRRSFHA5P6LDIQHWIQZWLEQYJCWM2Q, seg_pos={0 0}, upload=552020]: Uploading piece... Dec 31 21:26:19.319796994 (629h57m19.190880029s) [noise=true , address=145.79.227.39:4014 , piece_id=FALGJMKNGX2YR2JU4CEEVWSQSLR4HHZMCEIJS2XVXFK5MUIABZ6Q, seg_pos={0 0}, upload=552020]: Uploading piece... Dec 31 21:26:19.320970308 (629h57m19.192053343s) [noise=true , address=145.79.227.37:4565 , piece_id=DBX2EW7P6THCWP3X7WO4J7XXZ4MRLMY7CBQGX5XNA2K3XSMRHBMQ, seg_pos={0 0}, upload=552020]: Uploading piece... Dec 31 21:26:19.321397602 (629h57m19.192480647s) [noise=true , address=45.91.116.9:4263 , piece_id=JFEDL7EWCAWO24MXEFZTEJKPV2SQV35PYE43NDXSVY5SBY24NQHA, seg_pos={0 0}, upload=552020]: Uploading piece... Dec 31 21:26:19.321770171 (629h57m19.192853206s) [noise=true , address=45.91.116.14:4192 , piece_id=Q7ZLWISNLI7KN6IGWPDTHVRZ7L3QSQGJHXBIDUOYTOSLSNOG4P7Q, seg_pos={0 0}, upload=552020]: Uploading piece... Dec 31 21:26:19.322144805 (629h57m19.19322784s) [noise=true , address=145.79.227.12:4767 , piece_id=PV7LS5KCC4FZXJSVA6ZMDDATGLZPVA2VQMMBBEMMMWNHGQ6X2ROA, seg_pos={0 0}, upload=552020]: Uploading piece... Dec 31 21:26:19.322185412 (629h57m19.193268447s) [noise=true , address=145.79.227.45:4229 , piece_id=VJWIBL6ZETNPIVRNRUMM7PV56QW4RU52JYCZI2AB35RKA7JR3DWA, seg_pos={0 0}, upload=552020]: Uploading piece... Dec 31 21:26:19.322577158 (629h57m19.193660193s) [noise=true , address=45.91.116.12:4225 , piece_id=45UVA5PF3F62CFP74BQM42NEMB6M64TJ5CXRSG5QFJIYIKXQVX4Q, seg_pos={0 0}, upload=552020]: Uploading piece... Dec 31 21:26:19.323043807 (629h57m19.194126852s) [noise=true , address=145.79.227.32:4836 , piece_id=NVAEEGAKTDIXC3PX6XQCQLZ6Q7MIUVITKMQMBCHXRX2ZJQ6OKS6Q, seg_pos={0 0}, upload=552020]: Uploading piece... Dec 31 21:26:19.323453035 (629h57m19.19453607s) [noise=true , address=145.79.227.15:4874 , piece_id=EGNFR25OH7YC3XFGUUYLUT4HKGFUCVZMCMEFLECNMHADKZCKNBDQ, seg_pos={0 0}, upload=552020]: Uploading piece... Dec 31 21:26:20.943196324 (629h57m20.814279359s) [upload=552020]: Next returned nil segment Dec 31 21:26:20.943261358 (629h57m20.814344403s) [upload=552020]: Waiting for error group managing segments... Dec 31 21:26:21.311565930 (629h57m21.182648965s) [noise=true , address=145.79.227.7:4294 , piece_id=H6QSIINGNE3T6F7H6MLRN7CIX3X5WR4UIZKZPOFAZVLZUG3WICGQ, seg_pos={0 0}, upload=552020]: Done uploading piece. err: <nil> Dec 31 21:26:21.316669898 (629h57m21.187752933s) [noise=true , address=145.79.227.12:4767 , piece_id=PV7LS5KCC4FZXJSVA6ZMDDATGLZPVA2VQMMBBEMMMWNHGQ6X2ROA, seg_pos={0 0}, upload=552020]: Done uploading piece. err: <nil> Dec 31 21:26:21.320222032 (629h57m21.191305087s) [noise=true , address=145.79.227.39:4014 , piece_id=FALGJMKNGX2YR2JU4CEEVWSQSLR4HHZMCEIJS2XVXFK5MUIABZ6Q, seg_pos={0 0}, upload=552020]: Done uploading piece. err: <nil> Dec 31 21:26:21.325999112 (629h57m21.197082147s) [noise=true , address=145.79.227.37:4565 , piece_id=DBX2EW7P6THCWP3X7WO4J7XXZ4MRLMY7CBQGX5XNA2K3XSMRHBMQ, seg_pos={0 0}, upload=552020]: Done uploading piece. err: <nil> Dec 31 21:26:21.329606841 (629h57m21.200689876s) [noise=true , address=145.79.227.15:4874 , piece_id=EGNFR25OH7YC3XFGUUYLUT4HKGFUCVZMCMEFLECNMHADKZCKNBDQ, seg_pos={0 0}, upload=552020]: Done uploading piece. err: <nil> Dec 31 21:26:21.329954333 (629h57m21.201037368s) [noise=true , address=145.79.227.36:4518 , piece_id=PUHWY66CUZ5ONPFNC2LRZ52BFVSS3SX6ZXLWON2UQYQJD3TJ5MEQ, seg_pos={0 0}, upload=552020]: Done uploading piece. err: <nil> Dec 31 21:26:21.332051476 (629h57m21.203134511s) [noise=true , address=145.79.227.31:4797 , piece_id=EY2CW27J47HOBSSWTXFZK3XGYGBZH3W7275DTW7IOGFZSCUENORA, seg_pos={0 0}, upload=552020]: Done uploading piece. err: <nil> Dec 31 21:26:21.393201484 (629h57m21.264284529s) [noise=true , address=145.79.227.45:4229 , piece_id=VJWIBL6ZETNPIVRNRUMM7PV56QW4RU52JYCZI2AB35RKA7JR3DWA, seg_pos={0 0}, upload=552020]: Done uploading piece. err: <nil> Dec 31 21:26:21.396482262 (629h57m21.267565307s) [noise=true , address=145.79.227.4:4187 , piece_id=MQET2M3D4QPHG4IKPHSMSYMWR3ZUW5D2RYCHEOABF5OC2TRBS7MA, seg_pos={0 0}, upload=552020]: Done uploading piece. err: <nil> Dec 31 21:26:21.406883036 (629h57m21.277966071s) [noise=true , address=145.79.227.32:4836 , piece_id=NVAEEGAKTDIXC3PX6XQCQLZ6Q7MIUVITKMQMBCHXRX2ZJQ6OKS6Q, seg_pos={0 0}, upload=552020]: Done uploading piece. err: <nil> Dec 31 21:26:21.607595369 (629h57m21.478678404s) [noise=true , address=45.91.116.7:4010 , piece_id=PZWO3Q2S6ZXWGDGAMSSYQBTCQCGBV5ZXIZB5OAOH3XAGCWJLYUQQ, seg_pos={0 0}, upload=552020]: Done uploading piece. err: <nil> Dec 31 21:26:21.619681403 (629h57m21.490764438s) [noise=true , address=45.91.116.12:4225 , piece_id=45UVA5PF3F62CFP74BQM42NEMB6M64TJ5CXRSG5QFJIYIKXQVX4Q, seg_pos={0 0}, upload=552020]: Done uploading piece. err: <nil> Dec 31 21:26:21.632491554 (629h57m21.503574589s) [noise=true , address=45.91.116.14:4192 , piece_id=Q7ZLWISNLI7KN6IGWPDTHVRZ7L3QSQGJHXBIDUOYTOSLSNOG4P7Q, seg_pos={0 0}, upload=552020]: Done uploading piece. err: <nil> Dec 31 21:26:21.632510571 (629h57m21.503593606s) [seg_pos={0 0}, upload=552020]: Segment reached optimal threshold of 13 pieces. Dec 31 21:26:21.632676006 (629h57m21.503759051s) [noise=true , address=45.91.116.6:4975 , piece_id=DUMPZKKIOIZ35UNUV4SFISNWZSLN6NIO5ZWKURWMQFX5PUGTJSIQ, seg_pos={0 0}, upload=552020]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: write tcp 188.166.225.170:49332->45.91.116.6:4975: use of closed network connection; write tcp 188.166.225.170:49332->45.91.116.6:4975: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49332->45.91.116.6:4975: use of closed network connection Dec 31 21:26:21.632681406 (629h57m21.503764441s) [noise=true , address=45.91.116.6:4975 , piece_id=DUMPZKKIOIZ35UNUV4SFISNWZSLN6NIO5ZWKURWMQFX5PUGTJSIQ, seg_pos={0 0}, upload=552020]: Long tail cancellation due to optimal threshold Dec 31 21:26:21.632687738 (629h57m21.503770783s) [noise=true , address=145.79.227.48:4303 , piece_id=W27HEVJ2ZHGTEASHBO6CSOMRRSFHA5P6LDIQHWIQZWLEQYJCWM2Q, seg_pos={0 0}, upload=552020]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gJKrCBLJWN1SBKibfeqBtcqqiv4veKTGmtnycjZyb3VN9kwoT): protocol: write tcp 188.166.225.170:54402->145.79.227.48:4303: use of closed network connection; write tcp 188.166.225.170:54402->145.79.227.48:4303: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54402->145.79.227.48:4303: use of closed network connection Dec 31 21:26:21.632698188 (629h57m21.503781223s) [noise=true , address=145.79.227.48:4303 , piece_id=W27HEVJ2ZHGTEASHBO6CSOMRRSFHA5P6LDIQHWIQZWLEQYJCWM2Q, seg_pos={0 0}, upload=552020]: Long tail cancellation due to optimal threshold Dec 31 21:26:21.632864845 (629h57m21.50394788s) [noise=true , address=45.91.116.9:4263 , piece_id=JFEDL7EWCAWO24MXEFZTEJKPV2SQV35PYE43NDXSVY5SBY24NQHA, seg_pos={0 0}, upload=552020]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 31 21:26:21.632870346 (629h57m21.503953381s) [noise=true , address=45.91.116.9:4263 , piece_id=JFEDL7EWCAWO24MXEFZTEJKPV2SQV35PYE43NDXSVY5SBY24NQHA, seg_pos={0 0}, upload=552020]: Long tail cancellation due to optimal threshold Dec 31 21:26:21.632884203 (629h57m21.503967238s) [seg_pos={0 0}, upload=552020]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 21:26:21.632945068 (629h57m21.504028103s) [upload=552020]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 21:26:21.632949466 (629h57m21.504032501s) [upload=552020]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 21:26:21.929926801 (629h57m21.801009846s) [upload=552020]: Done uploading segments... Dec 31 21:26:21.929948742 (629h57m21.801031777s) [upload=552020]: Upload finished. err: <nil> Duration: 7.52863288s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [....X....XX........X...X.......X.....X...XX.X...........X..X] duration: 30.003253106s trace: https://jaeger.dev.storj.io/trace/5d999dac32a7be3e FAIL (2025-12-31 21:26:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ts... Dec 31 21:26:19.065719105 (629h57m18.93680214s) [upload=552020]: Got next segment. Inline: false Dec 31 21:26:19.065750896 (629h57m18.936833951s) [upload=552020]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 21:26:19.065755996 (629h57m18.936839031s) [upload=552020]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 21:26:19.317558331 (629h57m19.188641376s) [seg_pos={0 0}, upload=552020]: Begin upload segment... Dec 31 21:26:19.317624367 (629h57m19.188707402s) [seg_pos={0 0}, upload=552020]: Done begin upload segment. Dec 31 21:26:19.317666858 (629h57m19.188749893s) [noise=true , address=145.79.227.31:4797 , piece_id=EY2CW27J47HOBSSWTXFZK3XGYGBZH3W7275DTW7IOGFZSCUENORA, seg_pos={0 0}, upload=552020]: Uploading piece... Dec 31 21:26:19.317697767 (629h57m19.188780812s) [noise=true , address=45.91.116.6:4975 , piece_id=DUMPZKKIOIZ35UNUV4SFISNWZSLN6NIO5ZWKURWMQFX5PUGTJSIQ, seg_pos={0 0}, upload=552020]: Uploading piece... Dec 31 21:26:19.318149457 (629h57m19.189232492s) [noise=true , address=145.79.227.7:4294 , piece_id=H6QSIINGNE3T6F7H6MLRN7CIX3X5WR4UIZKZPOFAZVLZUG3WICGQ, seg_pos={0 0}, upload=552020]: Uploading piece... Dec 31 21:26:19.318482261 (629h57m19.189565296s) [noise=true , address=45.91.116.7:4010 , piece_id=PZWO3Q2S6ZXWGDGAMSSYQBTCQCGBV5ZXIZB5OAOH3XAGCWJLYUQQ, seg_pos={0 0}, upload=552020]: Uploading piece... Dec 31 21:26:19.318885508 (629h57m19.189968553s) [noise=true , address=145.79.227.4:4187 , piece_id=MQET2M3D4QPHG4IKPHSMSYMWR3ZUW5D2RYCHEOABF5OC2TRBS7MA, seg_pos={0 0}, upload=552020]: Uploading piece... Dec 31 21:26:19.319188986 (629h57m19.190272021s) [noise=true , address=145.79.227.36:4518 , piece_id=PUHWY66CUZ5ONPFNC2LRZ52BFVSS3SX6ZXLWON2UQYQJD3TJ5MEQ, seg_pos={0 0}, upload=552020]: Uploading piece... Dec 31 21:26:19.319494178 (629h57m19.190577213s) [noise=true , address=145.79.227.48:4303 , piece_id=W27HEVJ2ZHGTEASHBO6CSOMRRSFHA5P6LDIQHWIQZWLEQYJCWM2Q, seg_pos={0 0}, upload=552020]: Uploading piece... Dec 31 21:26:19.319796994 (629h57m19.190880029s) [noise=true , address=145.79.227.39:4014 , piece_id=FALGJMKNGX2YR2JU4CEEVWSQSLR4HHZMCEIJS2XVXFK5MUIABZ6Q, seg_pos={0 0}, upload=552020]: Uploading piece... Dec 31 21:26:19.320970308 (629h57m19.192053343s) [noise=true , address=145.79.227.37:4565 , piece_id=DBX2EW7P6THCWP3X7WO4J7XXZ4MRLMY7CBQGX5XNA2K3XSMRHBMQ, seg_pos={0 0}, upload=552020]: Uploading piece... Dec 31 21:26:19.321397602 (629h57m19.192480647s) [noise=true , address=45.91.116.9:4263 , piece_id=JFEDL7EWCAWO24MXEFZTEJKPV2SQV35PYE43NDXSVY5SBY24NQHA, seg_pos={0 0}, upload=552020]: Uploading piece... Dec 31 21:26:19.321770171 (629h57m19.192853206s) [noise=true , address=45.91.116.14:4192 , piece_id=Q7ZLWISNLI7KN6IGWPDTHVRZ7L3QSQGJHXBIDUOYTOSLSNOG4P7Q, seg_pos={0 0}, upload=552020]: Uploading piece... Dec 31 21:26:19.322144805 (629h57m19.19322784s) [noise=true , address=145.79.227.12:4767 , piece_id=PV7LS5KCC4FZXJSVA6ZMDDATGLZPVA2VQMMBBEMMMWNHGQ6X2ROA, seg_pos={0 0}, upload=552020]: Uploading piece... Dec 31 21:26:19.322185412 (629h57m19.193268447s) [noise=true , address=145.79.227.45:4229 , piece_id=VJWIBL6ZETNPIVRNRUMM7PV56QW4RU52JYCZI2AB35RKA7JR3DWA, seg_pos={0 0}, upload=552020]: Uploading piece... Dec 31 21:26:19.322577158 (629h57m19.193660193s) [noise=true , address=45.91.116.12:4225 , piece_id=45UVA5PF3F62CFP74BQM42NEMB6M64TJ5CXRSG5QFJIYIKXQVX4Q, seg_pos={0 0}, upload=552020]: Uploading piece... Dec 31 21:26:19.323043807 (629h57m19.194126852s) [noise=true , address=145.79.227.32:4836 , piece_id=NVAEEGAKTDIXC3PX6XQCQLZ6Q7MIUVITKMQMBCHXRX2ZJQ6OKS6Q, seg_pos={0 0}, upload=552020]: Uploading piece... Dec 31 21:26:19.323453035 (629h57m19.19453607s) [noise=true , address=145.79.227.15:4874 , piece_id=EGNFR25OH7YC3XFGUUYLUT4HKGFUCVZMCMEFLECNMHADKZCKNBDQ, seg_pos={0 0}, upload=552020]: Uploading piece... Dec 31 21:26:20.943196324 (629h57m20.814279359s) [upload=552020]: Next returned nil segment Dec 31 21:26:20.943261358 (629h57m20.814344403s) [upload=552020]: Waiting for error group managing segments... Dec 31 21:26:21.311565930 (629h57m21.182648965s) [noise=true , address=145.79.227.7:4294 , piece_id=H6QSIINGNE3T6F7H6MLRN7CIX3X5WR4UIZKZPOFAZVLZUG3WICGQ, seg_pos={0 0}, upload=552020]: Done uploading piece. err: <nil> Dec 31 21:26:21.316669898 (629h57m21.187752933s) [noise=true , address=145.79.227.12:4767 , piece_id=PV7LS5KCC4FZXJSVA6ZMDDATGLZPVA2VQMMBBEMMMWNHGQ6X2ROA, seg_pos={0 0}, upload=552020]: Done uploading piece. err: <nil> Dec 31 21:26:21.320222032 (629h57m21.191305087s) [noise=true , address=145.79.227.39:4014 , piece_id=FALGJMKNGX2YR2JU4CEEVWSQSLR4HHZMCEIJS2XVXFK5MUIABZ6Q, seg_pos={0 0}, upload=552020]: Done uploading piece. err: <nil> Dec 31 21:26:21.325999112 (629h57m21.197082147s) [noise=true , address=145.79.227.37:4565 , piece_id=DBX2EW7P6THCWP3X7WO4J7XXZ4MRLMY7CBQGX5XNA2K3XSMRHBMQ, seg_pos={0 0}, upload=552020]: Done uploading piece. err: <nil> Dec 31 21:26:21.329606841 (629h57m21.200689876s) [noise=true , address=145.79.227.15:4874 , piece_id=EGNFR25OH7YC3XFGUUYLUT4HKGFUCVZMCMEFLECNMHADKZCKNBDQ, seg_pos={0 0}, upload=552020]: Done uploading piece. err: <nil> Dec 31 21:26:21.329954333 (629h57m21.201037368s) [noise=true , address=145.79.227.36:4518 , piece_id=PUHWY66CUZ5ONPFNC2LRZ52BFVSS3SX6ZXLWON2UQYQJD3TJ5MEQ, seg_pos={0 0}, upload=552020]: Done uploading piece. err: <nil> Dec 31 21:26:21.332051476 (629h57m21.203134511s) [noise=true , address=145.79.227.31:4797 , piece_id=EY2CW27J47HOBSSWTXFZK3XGYGBZH3W7275DTW7IOGFZSCUENORA, seg_pos={0 0}, upload=552020]: Done uploading piece. err: <nil> Dec 31 21:26:21.393201484 (629h57m21.264284529s) [noise=true , address=145.79.227.45:4229 , piece_id=VJWIBL6ZETNPIVRNRUMM7PV56QW4RU52JYCZI2AB35RKA7JR3DWA, seg_pos={0 0}, upload=552020]: Done uploading piece. err: <nil> Dec 31 21:26:21.396482262 (629h57m21.267565307s) [noise=true , address=145.79.227.4:4187 , piece_id=MQET2M3D4QPHG4IKPHSMSYMWR3ZUW5D2RYCHEOABF5OC2TRBS7MA, seg_pos={0 0}, upload=552020]: Done uploading piece. err: <nil> Dec 31 21:26:21.406883036 (629h57m21.277966071s) [noise=true , address=145.79.227.32:4836 , piece_id=NVAEEGAKTDIXC3PX6XQCQLZ6Q7MIUVITKMQMBCHXRX2ZJQ6OKS6Q, seg_pos={0 0}, upload=552020]: Done uploading piece. err: <nil> Dec 31 21:26:21.607595369 (629h57m21.478678404s) [noise=true , address=45.91.116.7:4010 , piece_id=PZWO3Q2S6ZXWGDGAMSSYQBTCQCGBV5ZXIZB5OAOH3XAGCWJLYUQQ, seg_pos={0 0}, upload=552020]: Done uploading piece. err: <nil> Dec 31 21:26:21.619681403 (629h57m21.490764438s) [noise=true , address=45.91.116.12:4225 , piece_id=45UVA5PF3F62CFP74BQM42NEMB6M64TJ5CXRSG5QFJIYIKXQVX4Q, seg_pos={0 0}, upload=552020]: Done uploading piece. err: <nil> Dec 31 21:26:21.632491554 (629h57m21.503574589s) [noise=true , address=45.91.116.14:4192 , piece_id=Q7ZLWISNLI7KN6IGWPDTHVRZ7L3QSQGJHXBIDUOYTOSLSNOG4P7Q, seg_pos={0 0}, upload=552020]: Done uploading piece. err: <nil> Dec 31 21:26:21.632510571 (629h57m21.503593606s) [seg_pos={0 0}, upload=552020]: Segment reached optimal threshold of 13 pieces. Dec 31 21:26:21.632676006 (629h57m21.503759051s) [noise=true , address=45.91.116.6:4975 , piece_id=DUMPZKKIOIZ35UNUV4SFISNWZSLN6NIO5ZWKURWMQFX5PUGTJSIQ, seg_pos={0 0}, upload=552020]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: write tcp 188.166.225.170:49332->45.91.116.6:4975: use of closed network connection; write tcp 188.166.225.170:49332->45.91.116.6:4975: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49332->45.91.116.6:4975: use of closed network connection Dec 31 21:26:21.632681406 (629h57m21.503764441s) [noise=true , address=45.91.116.6:4975 , piece_id=DUMPZKKIOIZ35UNUV4SFISNWZSLN6NIO5ZWKURWMQFX5PUGTJSIQ, seg_pos={0 0}, upload=552020]: Long tail cancellation due to optimal threshold Dec 31 21:26:21.632687738 (629h57m21.503770783s) [noise=true , address=145.79.227.48:4303 , piece_id=W27HEVJ2ZHGTEASHBO6CSOMRRSFHA5P6LDIQHWIQZWLEQYJCWM2Q, seg_pos={0 0}, upload=552020]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gJKrCBLJWN1SBKibfeqBtcqqiv4veKTGmtnycjZyb3VN9kwoT): protocol: write tcp 188.166.225.170:54402->145.79.227.48:4303: use of closed network connection; write tcp 188.166.225.170:54402->145.79.227.48:4303: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54402->145.79.227.48:4303: use of closed network connection Dec 31 21:26:21.632698188 (629h57m21.503781223s) [noise=true , address=145.79.227.48:4303 , piece_id=W27HEVJ2ZHGTEASHBO6CSOMRRSFHA5P6LDIQHWIQZWLEQYJCWM2Q, seg_pos={0 0}, upload=552020]: Long tail cancellation due to optimal threshold Dec 31 21:26:21.632864845 (629h57m21.50394788s) [noise=true , address=45.91.116.9:4263 , piece_id=JFEDL7EWCAWO24MXEFZTEJKPV2SQV35PYE43NDXSVY5SBY24NQHA, seg_pos={0 0}, upload=552020]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 31 21:26:21.632870346 (629h57m21.503953381s) [noise=true , address=45.91.116.9:4263 , piece_id=JFEDL7EWCAWO24MXEFZTEJKPV2SQV35PYE43NDXSVY5SBY24NQHA, seg_pos={0 0}, upload=552020]: Long tail cancellation due to optimal threshold Dec 31 21:26:21.632884203 (629h57m21.503967238s) [seg_pos={0 0}, upload=552020]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 21:26:21.632945068 (629h57m21.504028103s) [upload=552020]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 21:26:21.632949466 (629h57m21.504032501s) [upload=552020]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 21:26:21.929926801 (629h57m21.801009846s) [upload=552020]: Done uploading segments... Dec 31 21:26:21.929948742 (629h57m21.801031777s) [upload=552020]: Upload finished. err: <nil> Duration: 30.003273223s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [.....X....XX........X...X.......X.....X...XX.X...........X..] duration: 6.273941543s trace: https://jaeger.dev.storj.io/trace/34133ac248c4bfc2 FAIL (2025-12-31 21:25:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: CUIA44NBZRB3WCN34G3YU3RIWM4QUQR4Q3EFZAYOLTMVVXQEFK7A) context deadline exceeded; piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: TNUFF3LHYJTTKHC4KJICYAQ7HBJGZOYD2F54L7WZ3A4OANY3W2EQ) context deadline exceeded ======LOGS====== Dec 31 21:23:47.503570100 (629h54m47.374653135s) [upload=551981]: Starting upload Dec 31 21:23:47.503654712 (629h54m47.374737757s) [upload=551981]: Done starting upload Dec 31 21:23:47.503739112 (629h54m47.374822147s) [upload=551981]: Uploading segments... Dec 31 21:23:47.503920067 (629h54m47.375003102s) [upload=551981]: Got next segment. Inline: false Dec 31 21:23:47.503963520 (629h54m47.375046565s) [upload=551981]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 21:23:47.503969541 (629h54m47.375052576s) [upload=551981]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 21:23:47.752135212 (629h54m47.623218247s) [seg_pos={0 0}, upload=551981]: Begin upload segment... Dec 31 21:23:47.752181980 (629h54m47.623265015s) [seg_pos={0 0}, upload=551981]: Done begin upload segment. Dec 31 21:23:47.752251813 (629h54m47.623334848s) [noise=true , address=45.91.116.9:4264 , piece_id=OJ6L5G5PRAASB7524BECUEWW3HKDALEPSE5QFDRW33BNK4UPOBAA, seg_pos={0 0}, upload=551981]: Uploading piece... Dec 31 21:23:47.752268125 (629h54m47.62335117s) [noise=true , address=145.79.227.46:4601 , piece_id=OEF3RAV7QNQE6UX6GBJ5DRMOWACTPQGYZQ57OHVBJW44OP655YMQ, seg_pos={0 0}, upload=551981]: Uploading piece... Dec 31 21:23:47.752770551 (629h54m47.623853586s) [noise=true , address=77.90.31.13:4629 , piece_id=CUIA44NBZRB3WCN34G3YU3RIWM4QUQR4Q3EFZAYOLTMVVXQEFK7A, seg_pos={0 0}, upload=551981]: Uploading piece... Dec 31 21:23:47.753060584 (629h54m47.624143619s) [noise=true , address=77.90.31.10:4518 , piece_id=TNUFF3LHYJTTKHC4KJICYAQ7HBJGZOYD2F54L7WZ3A4OANY3W2EQ, seg_pos={0 0}, upload=551981]: Uploading piece... Dec 31 21:23:47.753361977 (629h54m47.624445013s) [noise=true , address=45.91.116.6:4973 , piece_id=3MPPLOHQ4TJHOTQ6D3JAOQ6TJM6HXWXZNPZ2H2D2MQ6TIHJWMKWA, seg_pos={0 0}, upload=551981]: Uploading piece... Dec 31 21:23:47.754451572 (629h54m47.625534607s) [noise=true , address=45.91.116.13:4153 , piece_id=DDBTFO2D2ITOJC32PIPAOM3JPKASKKIOBRDLY23WZTW4WPX3RUDA, seg_pos={0 0}, upload=551981]: Uploading piece... Dec 31 21:23:47.754859469 (629h54m47.625942514s) [noise=true , address=145.79.227.38:4978 , piece_id=3TW4LC6I6EHEKVTXJ4INFCYIPCBBCAJI4YGPIIADVQDKEIOZSZNQ, seg_pos={0 0}, upload=551981]: Uploading piece... Dec 31 21:23:47.755921792 (629h54m47.627004827s) [noise=true , address=145.79.227.10:4402 , piece_id=5DOB3NEHDKIL6ZV72FKDRPTUFI4MAFA3GNHNVS7CQ5ICIM7VCJCA, seg_pos={0 0}, upload=551981]: Uploading piece... Dec 31 21:23:47.757109444 (629h54m47.628192479s) [noise=true , address=145.79.227.42:4122 , piece_id=SYNPHH7MVXODGYT67UZBPY2JUDYB77R6VOWDSM4QMOYMDGPPIOMA, seg_pos={0 0}, upload=551981]: Uploading piece... Dec 31 21:23:47.758327874 (629h54m47.629410909s) [noise=true , address=145.79.227.11:4731 , piece_id=XW4ZV66PTNQQVUX6JMH5H3ALIMSO5ASE5OLWLSGXSDR4MD3TNA7Q, seg_pos={0 0}, upload=551981]: Uploading piece... Dec 31 21:23:47.759352755 (629h54m47.6304358s) [noise=true , address=145.79.227.27:4690 , piece_id=MUT3VNSQPPFHL6WCO2LEUG42J2ZGXSZ6YBSJQ3COSR3WLPTKSKZQ, seg_pos={0 0}, upload=551981]: Uploading piece... Dec 31 21:23:47.760478740 (629h54m47.631561775s) [noise=true , address=145.79.227.17:4971 , piece_id=EJDVGX2A3XP5R77SAN4BWVMIGFP7CLPPZPD43WPH7UNEUWLCCHAQ, seg_pos={0 0}, upload=551981]: Uploading piece... Dec 31 21:23:47.760737782 (629h54m47.631820817s) [noise=true , address=145.79.227.2:4114 , piece_id=TUNO34RBHOFUJAYC67XPI5KM2TU5Y4GPJSBAOGGOMTZVHLNGLCWA, seg_pos={0 0}, upload=551981]: Uploading piece... Dec 31 21:23:47.762291451 (629h54m47.633374486s) [noise=true , address=145.79.227.19:4077 , piece_id=YRO3FA2LQEEOGU7ITNQWWYJX3POUDYYXH6ETPRVG6HJQOLBGD6AQ, seg_pos={0 0}, upload=551981]: Uploading piece... Dec 31 21:23:47.763316833 (629h54m47.634399868s) [noise=true , address=145.79.227.7:4295 , piece_id=6HZVPIQV2GK3SFWNZMV4Q4KJCYSD5L72YMJSHQKVBIAFEAQBWL6A, seg_pos={0 0}, upload=551981]: Uploading piece... Dec 31 21:23:47.763769024 (629h54m47.634852059s) [noise=true , address=45.91.116.4:4939 , piece_id=FEBS55TIIOWMJ24TNBGF5ZVTKUPKASEJNT5YI4AZIG5Q6DFRYENQ, seg_pos={0 0}, upload=551981]: Uploading piece... Dec 31 21:23:49.463919640 (629h54m49.335002675s) [upload=551981]: Next returned nil segment Dec 31 21:23:49.463930370 (629h54m49.335013405s) [upload=551981]: Waiting for error group managing segments... Dec 31 21:23:49.754494513 (629h54m49.625577548s) [noise=true , address=145.79.227.2:4114 , piece_id=TUNO34RBHOFUJAYC67XPI5KM2TU5Y4GPJSBAOGGOMTZVHLNGLCWA, seg_pos={0 0}, upload=551981]: Done uploading piece. err: <nil> Dec 31 21:23:49.757678846 (629h54m49.628761891s) [noise=true , address=145.79.227.27:4690 , piece_id=MUT3VNSQPPFHL6WCO2LEUG42J2ZGXSZ6YBSJQ3COSR3WLPTKSKZQ, seg_pos={0 0}, upload=551981]: Done uploading piece. err: <nil> Dec 31 21:23:49.772870554 (629h54m49.643953589s) [noise=true , address=145.79.227.17:4971 , piece_id=EJDVGX2A3XP5R77SAN4BWVMIGFP7CLPPZPD43WPH7UNEUWLCCHAQ, seg_pos={0 0}, upload=551981]: Done uploading piece. err: <nil> Dec 31 21:23:49.774695429 (629h54m49.645778464s) [noise=true , address=145.79.227.19:4077 , piece_id=YRO3FA2LQEEOGU7ITNQWWYJX3POUDYYXH6ETPRVG6HJQOLBGD6AQ, seg_pos={0 0}, upload=551981]: Done uploading piece. err: <nil> Dec 31 21:23:49.775122051 (629h54m49.646205086s) [noise=true , address=145.79.227.38:4978 , piece_id=3TW4LC6I6EHEKVTXJ4INFCYIPCBBCAJI4YGPIIADVQDKEIOZSZNQ, seg_pos={0 0}, upload=551981]: Done uploading piece. err: <nil> Dec 31 21:23:49.775570936 (629h54m49.646653971s) [noise=true , address=145.79.227.7:4295 , piece_id=6HZVPIQV2GK3SFWNZMV4Q4KJCYSD5L72YMJSHQKVBIAFEAQBWL6A, seg_pos={0 0}, upload=551981]: Done uploading piece. err: <nil> Dec 31 21:23:49.829347603 (629h54m49.700430638s) [noise=true , address=145.79.227.42:4122 , piece_id=SYNPHH7MVXODGYT67UZBPY2JUDYB77R6VOWDSM4QMOYMDGPPIOMA, seg_pos={0 0}, upload=551981]: Done uploading piece. err: <nil> Dec 31 21:23:49.831377820 (629h54m49.702460855s) [noise=true , address=145.79.227.10:4402 , piece_id=5DOB3NEHDKIL6ZV72FKDRPTUFI4MAFA3GNHNVS7CQ5ICIM7VCJCA, seg_pos={0 0}, upload=551981]: Done uploading piece. err: <nil> Dec 31 21:23:49.843008305 (629h54m49.71409135s) [noise=true , address=145.79.227.11:4731 , piece_id=XW4ZV66PTNQQVUX6JMH5H3ALIMSO5ASE5OLWLSGXSDR4MD3TNA7Q, seg_pos={0 0}, upload=551981]: Done uploading piece. err: <nil> Dec 31 21:23:49.848922437 (629h54m49.720005472s) [noise=true , address=145.79.227.46:4601 , piece_id=OEF3RAV7QNQE6UX6GBJ5DRMOWACTPQGYZQ57OHVBJW44OP655YMQ, seg_pos={0 0}, upload=551981]: Done uploading piece. err: <nil> Dec 31 21:23:49.920062277 (629h54m49.791145312s) [noise=true , address=77.90.31.13:4629 , piece_id=CUIA44NBZRB3WCN34G3YU3RIWM4QUQR4Q3EFZAYOLTMVVXQEFK7A, seg_pos={0 0}, upload=551981]: Done uploading piece. err: <nil> Dec 31 21:23:50.016284045 (629h54m49.887367091s) [noise=true , address=77.90.31.10:4518 , piece_id=TNUFF3LHYJTTKHC4KJICYAQ7HBJGZOYD2F54L7WZ3A4OANY3W2EQ, seg_pos={0 0}, upload=551981]: Done uploading piece. err: <nil> Dec 31 21:23:50.044797088 (629h54m49.915880133s) [noise=true , address=45.91.116.6:4973 , piece_id=3MPPLOHQ4TJHOTQ6D3JAOQ6TJM6HXWXZNPZ2H2D2MQ6TIHJWMKWA, seg_pos={0 0}, upload=551981]: Done uploading piece. err: <nil> Dec 31 21:23:50.044816606 (629h54m49.915899641s) [seg_pos={0 0}, upload=551981]: Segment reached optimal threshold of 13 pieces. Dec 31 21:23:50.044996067 (629h54m49.916079112s) [noise=true , address=45.91.116.13:4153 , piece_id=DDBTFO2D2ITOJC32PIPAOM3JPKASKKIOBRDLY23WZTW4WPX3RUDA, seg_pos={0 0}, upload=551981]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Dec 31 21:23:50.045001578 (629h54m49.916084613s) [noise=true , address=45.91.116.13:4153 , piece_id=DDBTFO2D2ITOJC32PIPAOM3JPKASKKIOBRDLY23WZTW4WPX3RUDA, seg_pos={0 0}, upload=551981]: Long tail cancellation due to optimal threshold Dec 31 21:23:50.045188524 (629h54m49.916271559s) [noise=true , address=45.91.116.4:4939 , piece_id=FEBS55TIIOWMJ24TNBGF5ZVTKUPKASEJNT5YI4AZIG5Q6DFRYENQ, seg_pos={0 0}, upload=551981]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 31 21:23:50.045209514 (629h54m49.916292549s) [noise=true , address=45.91.116.4:4939 , piece_id=FEBS55TIIOWMJ24TNBGF5ZVTKUPKASEJNT5YI4AZIG5Q6DFRYENQ, seg_pos={0 0}, upload=551981]: Long tail cancellation due to optimal threshold Dec 31 21:23:50.045227719 (629h54m49.916310754s) [noise=true , address=45.91.116.9:4264 , piece_id=OJ6L5G5PRAASB7524BECUEWW3HKDALEPSE5QFDRW33BNK4UPOBAA, seg_pos={0 0}, upload=551981]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 31 21:23:50.045237056 (629h54m49.916320091s) [noise=true , address=45.91.116.9:4264 , piece_id=OJ6L5G5PRAASB7524BECUEWW3HKDALEPSE5QFDRW33BNK4UPOBAA, seg_pos={0 0}, upload=551981]: Long tail cancellation due to optimal threshold Dec 31 21:23:50.045261082 (629h54m49.916344127s) [seg_pos={0 0}, upload=551981]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 21:23:50.045354771 (629h54m49.916437806s) [upload=551981]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 21:23:50.045360832 (629h54m49.916443877s) [upload=551981]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 21:23:50.341102093 (629h54m50.212185128s) [upload=551981]: Done uploading segments... Dec 31 21:23:50.341120539 (629h54m50.212203574s) [upload=551981]: Upload finished. err: <nil> Duration: 6.273971539s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [......X....XX........X...X.......X.....X...XX.X...........X.] duration: 5.82791192s trace: https://jaeger.dev.storj.io/trace/34133ac248c4bfc2 FAIL (2025-12-31 21:24:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: CUIA44NBZRB3WCN34G3YU3RIWM4QUQR4Q3EFZAYOLTMVVXQEFK7A) context deadline exceeded; piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: TNUFF3LHYJTTKHC4KJICYAQ7HBJGZOYD2F54L7WZ3A4OANY3W2EQ) context deadline exceeded ======LOGS====== Dec 31 21:23:47.503570100 (629h54m47.374653135s) [upload=551981]: Starting upload Dec 31 21:23:47.503654712 (629h54m47.374737757s) [upload=551981]: Done starting upload Dec 31 21:23:47.503739112 (629h54m47.374822147s) [upload=551981]: Uploading segments... Dec 31 21:23:47.503920067 (629h54m47.375003102s) [upload=551981]: Got next segment. Inline: false Dec 31 21:23:47.503963520 (629h54m47.375046565s) [upload=551981]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 21:23:47.503969541 (629h54m47.375052576s) [upload=551981]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 21:23:47.752135212 (629h54m47.623218247s) [seg_pos={0 0}, upload=551981]: Begin upload segment... Dec 31 21:23:47.752181980 (629h54m47.623265015s) [seg_pos={0 0}, upload=551981]: Done begin upload segment. Dec 31 21:23:47.752251813 (629h54m47.623334848s) [noise=true , address=45.91.116.9:4264 , piece_id=OJ6L5G5PRAASB7524BECUEWW3HKDALEPSE5QFDRW33BNK4UPOBAA, seg_pos={0 0}, upload=551981]: Uploading piece... Dec 31 21:23:47.752268125 (629h54m47.62335117s) [noise=true , address=145.79.227.46:4601 , piece_id=OEF3RAV7QNQE6UX6GBJ5DRMOWACTPQGYZQ57OHVBJW44OP655YMQ, seg_pos={0 0}, upload=551981]: Uploading piece... Dec 31 21:23:47.752770551 (629h54m47.623853586s) [noise=true , address=77.90.31.13:4629 , piece_id=CUIA44NBZRB3WCN34G3YU3RIWM4QUQR4Q3EFZAYOLTMVVXQEFK7A, seg_pos={0 0}, upload=551981]: Uploading piece... Dec 31 21:23:47.753060584 (629h54m47.624143619s) [noise=true , address=77.90.31.10:4518 , piece_id=TNUFF3LHYJTTKHC4KJICYAQ7HBJGZOYD2F54L7WZ3A4OANY3W2EQ, seg_pos={0 0}, upload=551981]: Uploading piece... Dec 31 21:23:47.753361977 (629h54m47.624445013s) [noise=true , address=45.91.116.6:4973 , piece_id=3MPPLOHQ4TJHOTQ6D3JAOQ6TJM6HXWXZNPZ2H2D2MQ6TIHJWMKWA, seg_pos={0 0}, upload=551981]: Uploading piece... Dec 31 21:23:47.754451572 (629h54m47.625534607s) [noise=true , address=45.91.116.13:4153 , piece_id=DDBTFO2D2ITOJC32PIPAOM3JPKASKKIOBRDLY23WZTW4WPX3RUDA, seg_pos={0 0}, upload=551981]: Uploading piece... Dec 31 21:23:47.754859469 (629h54m47.625942514s) [noise=true , address=145.79.227.38:4978 , piece_id=3TW4LC6I6EHEKVTXJ4INFCYIPCBBCAJI4YGPIIADVQDKEIOZSZNQ, seg_pos={0 0}, upload=551981]: Uploading piece... Dec 31 21:23:47.755921792 (629h54m47.627004827s) [noise=true , address=145.79.227.10:4402 , piece_id=5DOB3NEHDKIL6ZV72FKDRPTUFI4MAFA3GNHNVS7CQ5ICIM7VCJCA, seg_pos={0 0}, upload=551981]: Uploading piece... Dec 31 21:23:47.757109444 (629h54m47.628192479s) [noise=true , address=145.79.227.42:4122 , piece_id=SYNPHH7MVXODGYT67UZBPY2JUDYB77R6VOWDSM4QMOYMDGPPIOMA, seg_pos={0 0}, upload=551981]: Uploading piece... Dec 31 21:23:47.758327874 (629h54m47.629410909s) [noise=true , address=145.79.227.11:4731 , piece_id=XW4ZV66PTNQQVUX6JMH5H3ALIMSO5ASE5OLWLSGXSDR4MD3TNA7Q, seg_pos={0 0}, upload=551981]: Uploading piece... Dec 31 21:23:47.759352755 (629h54m47.6304358s) [noise=true , address=145.79.227.27:4690 , piece_id=MUT3VNSQPPFHL6WCO2LEUG42J2ZGXSZ6YBSJQ3COSR3WLPTKSKZQ, seg_pos={0 0}, upload=551981]: Uploading piece... Dec 31 21:23:47.760478740 (629h54m47.631561775s) [noise=true , address=145.79.227.17:4971 , piece_id=EJDVGX2A3XP5R77SAN4BWVMIGFP7CLPPZPD43WPH7UNEUWLCCHAQ, seg_pos={0 0}, upload=551981]: Uploading piece... Dec 31 21:23:47.760737782 (629h54m47.631820817s) [noise=true , address=145.79.227.2:4114 , piece_id=TUNO34RBHOFUJAYC67XPI5KM2TU5Y4GPJSBAOGGOMTZVHLNGLCWA, seg_pos={0 0}, upload=551981]: Uploading piece... Dec 31 21:23:47.762291451 (629h54m47.633374486s) [noise=true , address=145.79.227.19:4077 , piece_id=YRO3FA2LQEEOGU7ITNQWWYJX3POUDYYXH6ETPRVG6HJQOLBGD6AQ, seg_pos={0 0}, upload=551981]: Uploading piece... Dec 31 21:23:47.763316833 (629h54m47.634399868s) [noise=true , address=145.79.227.7:4295 , piece_id=6HZVPIQV2GK3SFWNZMV4Q4KJCYSD5L72YMJSHQKVBIAFEAQBWL6A, seg_pos={0 0}, upload=551981]: Uploading piece... Dec 31 21:23:47.763769024 (629h54m47.634852059s) [noise=true , address=45.91.116.4:4939 , piece_id=FEBS55TIIOWMJ24TNBGF5ZVTKUPKASEJNT5YI4AZIG5Q6DFRYENQ, seg_pos={0 0}, upload=551981]: Uploading piece... Dec 31 21:23:49.463919640 (629h54m49.335002675s) [upload=551981]: Next returned nil segment Dec 31 21:23:49.463930370 (629h54m49.335013405s) [upload=551981]: Waiting for error group managing segments... Dec 31 21:23:49.754494513 (629h54m49.625577548s) [noise=true , address=145.79.227.2:4114 , piece_id=TUNO34RBHOFUJAYC67XPI5KM2TU5Y4GPJSBAOGGOMTZVHLNGLCWA, seg_pos={0 0}, upload=551981]: Done uploading piece. err: <nil> Dec 31 21:23:49.757678846 (629h54m49.628761891s) [noise=true , address=145.79.227.27:4690 , piece_id=MUT3VNSQPPFHL6WCO2LEUG42J2ZGXSZ6YBSJQ3COSR3WLPTKSKZQ, seg_pos={0 0}, upload=551981]: Done uploading piece. err: <nil> Dec 31 21:23:49.772870554 (629h54m49.643953589s) [noise=true , address=145.79.227.17:4971 , piece_id=EJDVGX2A3XP5R77SAN4BWVMIGFP7CLPPZPD43WPH7UNEUWLCCHAQ, seg_pos={0 0}, upload=551981]: Done uploading piece. err: <nil> Dec 31 21:23:49.774695429 (629h54m49.645778464s) [noise=true , address=145.79.227.19:4077 , piece_id=YRO3FA2LQEEOGU7ITNQWWYJX3POUDYYXH6ETPRVG6HJQOLBGD6AQ, seg_pos={0 0}, upload=551981]: Done uploading piece. err: <nil> Dec 31 21:23:49.775122051 (629h54m49.646205086s) [noise=true , address=145.79.227.38:4978 , piece_id=3TW4LC6I6EHEKVTXJ4INFCYIPCBBCAJI4YGPIIADVQDKEIOZSZNQ, seg_pos={0 0}, upload=551981]: Done uploading piece. err: <nil> Dec 31 21:23:49.775570936 (629h54m49.646653971s) [noise=true , address=145.79.227.7:4295 , piece_id=6HZVPIQV2GK3SFWNZMV4Q4KJCYSD5L72YMJSHQKVBIAFEAQBWL6A, seg_pos={0 0}, upload=551981]: Done uploading piece. err: <nil> Dec 31 21:23:49.829347603 (629h54m49.700430638s) [noise=true , address=145.79.227.42:4122 , piece_id=SYNPHH7MVXODGYT67UZBPY2JUDYB77R6VOWDSM4QMOYMDGPPIOMA, seg_pos={0 0}, upload=551981]: Done uploading piece. err: <nil> Dec 31 21:23:49.831377820 (629h54m49.702460855s) [noise=true , address=145.79.227.10:4402 , piece_id=5DOB3NEHDKIL6ZV72FKDRPTUFI4MAFA3GNHNVS7CQ5ICIM7VCJCA, seg_pos={0 0}, upload=551981]: Done uploading piece. err: <nil> Dec 31 21:23:49.843008305 (629h54m49.71409135s) [noise=true , address=145.79.227.11:4731 , piece_id=XW4ZV66PTNQQVUX6JMH5H3ALIMSO5ASE5OLWLSGXSDR4MD3TNA7Q, seg_pos={0 0}, upload=551981]: Done uploading piece. err: <nil> Dec 31 21:23:49.848922437 (629h54m49.720005472s) [noise=true , address=145.79.227.46:4601 , piece_id=OEF3RAV7QNQE6UX6GBJ5DRMOWACTPQGYZQ57OHVBJW44OP655YMQ, seg_pos={0 0}, upload=551981]: Done uploading piece. err: <nil> Dec 31 21:23:49.920062277 (629h54m49.791145312s) [noise=true , address=77.90.31.13:4629 , piece_id=CUIA44NBZRB3WCN34G3YU3RIWM4QUQR4Q3EFZAYOLTMVVXQEFK7A, seg_pos={0 0}, upload=551981]: Done uploading piece. err: <nil> Dec 31 21:23:50.016284045 (629h54m49.887367091s) [noise=true , address=77.90.31.10:4518 , piece_id=TNUFF3LHYJTTKHC4KJICYAQ7HBJGZOYD2F54L7WZ3A4OANY3W2EQ, seg_pos={0 0}, upload=551981]: Done uploading piece. err: <nil> Dec 31 21:23:50.044797088 (629h54m49.915880133s) [noise=true , address=45.91.116.6:4973 , piece_id=3MPPLOHQ4TJHOTQ6D3JAOQ6TJM6HXWXZNPZ2H2D2MQ6TIHJWMKWA, seg_pos={0 0}, upload=551981]: Done uploading piece. err: <nil> Dec 31 21:23:50.044816606 (629h54m49.915899641s) [seg_pos={0 0}, upload=551981]: Segment reached optimal threshold of 13 pieces. Dec 31 21:23:50.044996067 (629h54m49.916079112s) [noise=true , address=45.91.116.13:4153 , piece_id=DDBTFO2D2ITOJC32PIPAOM3JPKASKKIOBRDLY23WZTW4WPX3RUDA, seg_pos={0 0}, upload=551981]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Dec 31 21:23:50.045001578 (629h54m49.916084613s) [noise=true , address=45.91.116.13:4153 , piece_id=DDBTFO2D2ITOJC32PIPAOM3JPKASKKIOBRDLY23WZTW4WPX3RUDA, seg_pos={0 0}, upload=551981]: Long tail cancellation due to optimal threshold Dec 31 21:23:50.045188524 (629h54m49.916271559s) [noise=true , address=45.91.116.4:4939 , piece_id=FEBS55TIIOWMJ24TNBGF5ZVTKUPKASEJNT5YI4AZIG5Q6DFRYENQ, seg_pos={0 0}, upload=551981]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 31 21:23:50.045209514 (629h54m49.916292549s) [noise=true , address=45.91.116.4:4939 , piece_id=FEBS55TIIOWMJ24TNBGF5ZVTKUPKASEJNT5YI4AZIG5Q6DFRYENQ, seg_pos={0 0}, upload=551981]: Long tail cancellation due to optimal threshold Dec 31 21:23:50.045227719 (629h54m49.916310754s) [noise=true , address=45.91.116.9:4264 , piece_id=OJ6L5G5PRAASB7524BECUEWW3HKDALEPSE5QFDRW33BNK4UPOBAA, seg_pos={0 0}, upload=551981]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 31 21:23:50.045237056 (629h54m49.916320091s) [noise=true , address=45.91.116.9:4264 , piece_id=OJ6L5G5PRAASB7524BECUEWW3HKDALEPSE5QFDRW33BNK4UPOBAA, seg_pos={0 0}, upload=551981]: Long tail cancellation due to optimal threshold Dec 31 21:23:50.045261082 (629h54m49.916344127s) [seg_pos={0 0}, upload=551981]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 21:23:50.045354771 (629h54m49.916437806s) [upload=551981]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 21:23:50.045360832 (629h54m49.916443877s) [upload=551981]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 21:23:50.341102093 (629h54m50.212185128s) [upload=551981]: Done uploading segments... Dec 31 21:23:50.341120539 (629h54m50.212203574s) [upload=551981]: Upload finished. err: <nil> Duration: 5.827936907s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [X......X....XX........X...X.......X.....X...XX.X...........X] duration: 30.002998072s trace: https://jaeger.dev.storj.io/trace/34133ac248c4bfc2 FAIL (2025-12-31 21:24:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: CUIA44NBZRB3WCN34G3YU3RIWM4QUQR4Q3EFZAYOLTMVVXQEFK7A) context deadline exceeded; piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: TNUFF3LHYJTTKHC4KJICYAQ7HBJGZOYD2F54L7WZ3A4OANY3W2EQ) context deadline exceeded ======LOGS====== Dec 31 21:23:47.503570100 (629h54m47.374653135s) [upload=551981]: Starting upload Dec 31 21:23:47.503654712 (629h54m47.374737757s) [upload=551981]: Done starting upload Dec 31 21:23:47.503739112 (629h54m47.374822147s) [upload=551981]: Uploading segments... Dec 31 21:23:47.503920067 (629h54m47.375003102s) [upload=551981]: Got next segment. Inline: false Dec 31 21:23:47.503963520 (629h54m47.375046565s) [upload=551981]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 21:23:47.503969541 (629h54m47.375052576s) [upload=551981]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 21:23:47.752135212 (629h54m47.623218247s) [seg_pos={0 0}, upload=551981]: Begin upload segment... Dec 31 21:23:47.752181980 (629h54m47.623265015s) [seg_pos={0 0}, upload=551981]: Done begin upload segment. Dec 31 21:23:47.752251813 (629h54m47.623334848s) [noise=true , address=45.91.116.9:4264 , piece_id=OJ6L5G5PRAASB7524BECUEWW3HKDALEPSE5QFDRW33BNK4UPOBAA, seg_pos={0 0}, upload=551981]: Uploading piece... Dec 31 21:23:47.752268125 (629h54m47.62335117s) [noise=true , address=145.79.227.46:4601 , piece_id=OEF3RAV7QNQE6UX6GBJ5DRMOWACTPQGYZQ57OHVBJW44OP655YMQ, seg_pos={0 0}, upload=551981]: Uploading piece... Dec 31 21:23:47.752770551 (629h54m47.623853586s) [noise=true , address=77.90.31.13:4629 , piece_id=CUIA44NBZRB3WCN34G3YU3RIWM4QUQR4Q3EFZAYOLTMVVXQEFK7A, seg_pos={0 0}, upload=551981]: Uploading piece... Dec 31 21:23:47.753060584 (629h54m47.624143619s) [noise=true , address=77.90.31.10:4518 , piece_id=TNUFF3LHYJTTKHC4KJICYAQ7HBJGZOYD2F54L7WZ3A4OANY3W2EQ, seg_pos={0 0}, upload=551981]: Uploading piece... Dec 31 21:23:47.753361977 (629h54m47.624445013s) [noise=true , address=45.91.116.6:4973 , piece_id=3MPPLOHQ4TJHOTQ6D3JAOQ6TJM6HXWXZNPZ2H2D2MQ6TIHJWMKWA, seg_pos={0 0}, upload=551981]: Uploading piece... Dec 31 21:23:47.754451572 (629h54m47.625534607s) [noise=true , address=45.91.116.13:4153 , piece_id=DDBTFO2D2ITOJC32PIPAOM3JPKASKKIOBRDLY23WZTW4WPX3RUDA, seg_pos={0 0}, upload=551981]: Uploading piece... Dec 31 21:23:47.754859469 (629h54m47.625942514s) [noise=true , address=145.79.227.38:4978 , piece_id=3TW4LC6I6EHEKVTXJ4INFCYIPCBBCAJI4YGPIIADVQDKEIOZSZNQ, seg_pos={0 0}, upload=551981]: Uploading piece... Dec 31 21:23:47.755921792 (629h54m47.627004827s) [noise=true , address=145.79.227.10:4402 , piece_id=5DOB3NEHDKIL6ZV72FKDRPTUFI4MAFA3GNHNVS7CQ5ICIM7VCJCA, seg_pos={0 0}, upload=551981]: Uploading piece... Dec 31 21:23:47.757109444 (629h54m47.628192479s) [noise=true , address=145.79.227.42:4122 , piece_id=SYNPHH7MVXODGYT67UZBPY2JUDYB77R6VOWDSM4QMOYMDGPPIOMA, seg_pos={0 0}, upload=551981]: Uploading piece... Dec 31 21:23:47.758327874 (629h54m47.629410909s) [noise=true , address=145.79.227.11:4731 , piece_id=XW4ZV66PTNQQVUX6JMH5H3ALIMSO5ASE5OLWLSGXSDR4MD3TNA7Q, seg_pos={0 0}, upload=551981]: Uploading piece... Dec 31 21:23:47.759352755 (629h54m47.6304358s) [noise=true , address=145.79.227.27:4690 , piece_id=MUT3VNSQPPFHL6WCO2LEUG42J2ZGXSZ6YBSJQ3COSR3WLPTKSKZQ, seg_pos={0 0}, upload=551981]: Uploading piece... Dec 31 21:23:47.760478740 (629h54m47.631561775s) [noise=true , address=145.79.227.17:4971 , piece_id=EJDVGX2A3XP5R77SAN4BWVMIGFP7CLPPZPD43WPH7UNEUWLCCHAQ, seg_pos={0 0}, upload=551981]: Uploading piece... Dec 31 21:23:47.760737782 (629h54m47.631820817s) [noise=true , address=145.79.227.2:4114 , piece_id=TUNO34RBHOFUJAYC67XPI5KM2TU5Y4GPJSBAOGGOMTZVHLNGLCWA, seg_pos={0 0}, upload=551981]: Uploading piece... Dec 31 21:23:47.762291451 (629h54m47.633374486s) [noise=true , address=145.79.227.19:4077 , piece_id=YRO3FA2LQEEOGU7ITNQWWYJX3POUDYYXH6ETPRVG6HJQOLBGD6AQ, seg_pos={0 0}, upload=551981]: Uploading piece... Dec 31 21:23:47.763316833 (629h54m47.634399868s) [noise=true , address=145.79.227.7:4295 , piece_id=6HZVPIQV2GK3SFWNZMV4Q4KJCYSD5L72YMJSHQKVBIAFEAQBWL6A, seg_pos={0 0}, upload=551981]: Uploading piece... Dec 31 21:23:47.763769024 (629h54m47.634852059s) [noise=true , address=45.91.116.4:4939 , piece_id=FEBS55TIIOWMJ24TNBGF5ZVTKUPKASEJNT5YI4AZIG5Q6DFRYENQ, seg_pos={0 0}, upload=551981]: Uploading piece... Dec 31 21:23:49.463919640 (629h54m49.335002675s) [upload=551981]: Next returned nil segment Dec 31 21:23:49.463930370 (629h54m49.335013405s) [upload=551981]: Waiting for error group managing segments... Dec 31 21:23:49.754494513 (629h54m49.625577548s) [noise=true , address=145.79.227.2:4114 , piece_id=TUNO34RBHOFUJAYC67XPI5KM2TU5Y4GPJSBAOGGOMTZVHLNGLCWA, seg_pos={0 0}, upload=551981]: Done uploading piece. err: <nil> Dec 31 21:23:49.757678846 (629h54m49.628761891s) [noise=true , address=145.79.227.27:4690 , piece_id=MUT3VNSQPPFHL6WCO2LEUG42J2ZGXSZ6YBSJQ3COSR3WLPTKSKZQ, seg_pos={0 0}, upload=551981]: Done uploading piece. err: <nil> Dec 31 21:23:49.772870554 (629h54m49.643953589s) [noise=true , address=145.79.227.17:4971 , piece_id=EJDVGX2A3XP5R77SAN4BWVMIGFP7CLPPZPD43WPH7UNEUWLCCHAQ, seg_pos={0 0}, upload=551981]: Done uploading piece. err: <nil> Dec 31 21:23:49.774695429 (629h54m49.645778464s) [noise=true , address=145.79.227.19:4077 , piece_id=YRO3FA2LQEEOGU7ITNQWWYJX3POUDYYXH6ETPRVG6HJQOLBGD6AQ, seg_pos={0 0}, upload=551981]: Done uploading piece. err: <nil> Dec 31 21:23:49.775122051 (629h54m49.646205086s) [noise=true , address=145.79.227.38:4978 , piece_id=3TW4LC6I6EHEKVTXJ4INFCYIPCBBCAJI4YGPIIADVQDKEIOZSZNQ, seg_pos={0 0}, upload=551981]: Done uploading piece. err: <nil> Dec 31 21:23:49.775570936 (629h54m49.646653971s) [noise=true , address=145.79.227.7:4295 , piece_id=6HZVPIQV2GK3SFWNZMV4Q4KJCYSD5L72YMJSHQKVBIAFEAQBWL6A, seg_pos={0 0}, upload=551981]: Done uploading piece. err: <nil> Dec 31 21:23:49.829347603 (629h54m49.700430638s) [noise=true , address=145.79.227.42:4122 , piece_id=SYNPHH7MVXODGYT67UZBPY2JUDYB77R6VOWDSM4QMOYMDGPPIOMA, seg_pos={0 0}, upload=551981]: Done uploading piece. err: <nil> Dec 31 21:23:49.831377820 (629h54m49.702460855s) [noise=true , address=145.79.227.10:4402 , piece_id=5DOB3NEHDKIL6ZV72FKDRPTUFI4MAFA3GNHNVS7CQ5ICIM7VCJCA, seg_pos={0 0}, upload=551981]: Done uploading piece. err: <nil> Dec 31 21:23:49.843008305 (629h54m49.71409135s) [noise=true , address=145.79.227.11:4731 , piece_id=XW4ZV66PTNQQVUX6JMH5H3ALIMSO5ASE5OLWLSGXSDR4MD3TNA7Q, seg_pos={0 0}, upload=551981]: Done uploading piece. err: <nil> Dec 31 21:23:49.848922437 (629h54m49.720005472s) [noise=true , address=145.79.227.46:4601 , piece_id=OEF3RAV7QNQE6UX6GBJ5DRMOWACTPQGYZQ57OHVBJW44OP655YMQ, seg_pos={0 0}, upload=551981]: Done uploading piece. err: <nil> Dec 31 21:23:49.920062277 (629h54m49.791145312s) [noise=true , address=77.90.31.13:4629 , piece_id=CUIA44NBZRB3WCN34G3YU3RIWM4QUQR4Q3EFZAYOLTMVVXQEFK7A, seg_pos={0 0}, upload=551981]: Done uploading piece. err: <nil> Dec 31 21:23:50.016284045 (629h54m49.887367091s) [noise=true , address=77.90.31.10:4518 , piece_id=TNUFF3LHYJTTKHC4KJICYAQ7HBJGZOYD2F54L7WZ3A4OANY3W2EQ, seg_pos={0 0}, upload=551981]: Done uploading piece. err: <nil> Dec 31 21:23:50.044797088 (629h54m49.915880133s) [noise=true , address=45.91.116.6:4973 , piece_id=3MPPLOHQ4TJHOTQ6D3JAOQ6TJM6HXWXZNPZ2H2D2MQ6TIHJWMKWA, seg_pos={0 0}, upload=551981]: Done uploading piece. err: <nil> Dec 31 21:23:50.044816606 (629h54m49.915899641s) [seg_pos={0 0}, upload=551981]: Segment reached optimal threshold of 13 pieces. Dec 31 21:23:50.044996067 (629h54m49.916079112s) [noise=true , address=45.91.116.13:4153 , piece_id=DDBTFO2D2ITOJC32PIPAOM3JPKASKKIOBRDLY23WZTW4WPX3RUDA, seg_pos={0 0}, upload=551981]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Dec 31 21:23:50.045001578 (629h54m49.916084613s) [noise=true , address=45.91.116.13:4153 , piece_id=DDBTFO2D2ITOJC32PIPAOM3JPKASKKIOBRDLY23WZTW4WPX3RUDA, seg_pos={0 0}, upload=551981]: Long tail cancellation due to optimal threshold Dec 31 21:23:50.045188524 (629h54m49.916271559s) [noise=true , address=45.91.116.4:4939 , piece_id=FEBS55TIIOWMJ24TNBGF5ZVTKUPKASEJNT5YI4AZIG5Q6DFRYENQ, seg_pos={0 0}, upload=551981]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 31 21:23:50.045209514 (629h54m49.916292549s) [noise=true , address=45.91.116.4:4939 , piece_id=FEBS55TIIOWMJ24TNBGF5ZVTKUPKASEJNT5YI4AZIG5Q6DFRYENQ, seg_pos={0 0}, upload=551981]: Long tail cancellation due to optimal threshold Dec 31 21:23:50.045227719 (629h54m49.916310754s) [noise=true , address=45.91.116.9:4264 , piece_id=OJ6L5G5PRAASB7524BECUEWW3HKDALEPSE5QFDRW33BNK4UPOBAA, seg_pos={0 0}, upload=551981]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 31 21:23:50.045237056 (629h54m49.916320091s) [noise=true , address=45.91.116.9:4264 , piece_id=OJ6L5G5PRAASB7524BECUEWW3HKDALEPSE5QFDRW33BNK4UPOBAA, seg_pos={0 0}, upload=551981]: Long tail cancellation due to optimal threshold Dec 31 21:23:50.045261082 (629h54m49.916344127s) [seg_pos={0 0}, upload=551981]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 21:23:50.045354771 (629h54m49.916437806s) [upload=551981]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 21:23:50.045360832 (629h54m49.916443877s) [upload=551981]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 21:23:50.341102093 (629h54m50.212185128s) [upload=551981]: Done uploading segments... Dec 31 21:23:50.341120539 (629h54m50.212203574s) [upload=551981]: Upload finished. err: <nil> Duration: 30.003015004s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [.X......X....XX........X...X.......X.....X...XX.X...........] duration: 5.960701586s trace: https://jaeger.dev.storj.io/trace/5f2b7d22d21cf0f8 FAIL (2025-12-31 21:23:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: U4ZYIZQIZLHUQDHRK43NC2XEXOR3LDSS32WLEHYA5QGNIW4R5ZPQ) context deadline exceeded; piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: UEJYYPH4WYBOEV2ZMZDDHNGIHT2JH57AIQY3XZ2HLBV6F5QOYCTQ) context deadline exceeded; piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: CC55YZD4CWFZBRQASU6VB33RMJZ6RYGEZ6Y7SNLYILCRVRDBMMUQ) context deadline exceeded ======LOGS====== Dec 31 21:15:44.386938901 (629h46m44.258021936s) [upload=551868]: Starting upload Dec 31 21:15:44.386994888 (629h46m44.258077923s) [upload=551868]: Done starting upload Dec 31 21:15:44.387183197 (629h46m44.258266242s) [upload=551868]: Uploading segments... Dec 31 21:15:44.387281093 (629h46m44.258364158s) [upload=551868]: Got next segment. Inline: false Dec 31 21:15:44.387329566 (629h46m44.258412621s) [upload=551868]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 21:15:44.387337030 (629h46m44.258420075s) [upload=551868]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 21:15:44.637074656 (629h46m44.508157691s) [seg_pos={0 0}, upload=551868]: Begin upload segment... Dec 31 21:15:44.637126884 (629h46m44.50820992s) [seg_pos={0 0}, upload=551868]: Done begin upload segment. Dec 31 21:15:44.637137575 (629h46m44.50822061s) [noise=true , address=45.91.116.13:4156 , piece_id=CVX3XD5GRJVGPMTIO3GQGMPL3V76CREYNE4VOS22CNW3KNLH5VXA, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.637158976 (629h46m44.508242021s) [noise=true , address=45.91.116.12:4225 , piece_id=QF3QQSR5EF5T42IFPEWOCIAQBPAQPNVGBY5LLRO2IUJ5VHC4QNZA, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.637344248 (629h46m44.508427283s) [noise=true , address=145.79.227.48:4303 , piece_id=5CQNY7DPO4HVD4JXS4WBJEN3I7ZUPUNSY45HCEC25VZSW7R4Y5BQ, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.637642296 (629h46m44.508725331s) [noise=true , address=145.79.227.10:4403 , piece_id=CHUKGI35OWLDMZG2GGZHMC7RL6RDGNGIXT3KZZHBWKRXPLSJ3PMA, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.637935754 (629h46m44.5090188s) [noise=true , address=145.79.227.52:4446 , piece_id=7QK754CTPSPTRHM6LCLEBKRKSYLHJMAM5FZOJNDJKXZFJITSHWRQ, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.638274770 (629h46m44.509357815s) [noise=true , address=145.79.227.13:4803 , piece_id=TZJI24W2EQRLGP5NOL3KVN7R5HDJMEEWPH5WCXZG5WTDMA4FUDXQ, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.638548111 (629h46m44.509631146s) [noise=true , address=145.79.227.11:4730 , piece_id=BQLBPYXQPS7NTNKWT3NE4Z4VZPCKRAMBFBEAY3BVF7Q34FHZD6AQ, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.638806723 (629h46m44.509889768s) [noise=true , address=145.79.227.15:4875 , piece_id=KQMUIZMYY5TKHKLOVE2HYQEU5WYCZOOB5AJVROAE5WVSGZJDXMOQ, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.639124738 (629h46m44.510207783s) [noise=true , address=145.79.227.51:4410 , piece_id=QH57IGTPJZ57O6EHENLY2JU7VKVOZSYVGBVODRC2I5LL2AWZHV6Q, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.639522837 (629h46m44.510605882s) [noise=true , address=77.90.31.13:4627 , piece_id=CC55YZD4CWFZBRQASU6VB33RMJZ6RYGEZ6Y7SNLYILCRVRDBMMUQ, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.641025558 (629h46m44.512108593s) [noise=true , address=45.91.116.11:4118 , piece_id=U4ZYIZQIZLHUQDHRK43NC2XEXOR3LDSS32WLEHYA5QGNIW4R5ZPQ, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.642391289 (629h46m44.513474334s) [noise=true , address=145.79.227.46:4603 , piece_id=NNYCFUISSSOYPBTF2YGIMM7BNAW3EIYZTFKBGMQVEA5MNEHU6GQA, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.643600902 (629h46m44.514683937s) [noise=true , address=45.91.116.10:4083 , piece_id=EKQLXTCN4KSWG4GTBSB6FVVSBEP4L5ONBKXEMEQIOBYDYA5ZOUIA, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.644749149 (629h46m44.515832184s) [noise=true , address=145.79.227.31:4799 , piece_id=54IF73MWOTIOJ4WXV3WGUJMK7JMLGIVZIEOSI3PTYHZE5M263JFQ, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.646078710 (629h46m44.517161745s) [noise=true , address=77.90.31.21:4880 , piece_id=UEJYYPH4WYBOEV2ZMZDDHNGIHT2JH57AIQY3XZ2HLBV6F5QOYCTQ, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.647522739 (629h46m44.518605784s) [noise=true , address=45.91.116.6:4973 , piece_id=NSEDZOAZWJEHL2YJPRPT3Y37HLGYQSLCSIRP62IHYLFTDVFEIUUQ, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:46.344821972 (629h46m46.215905007s) [upload=551868]: Next returned nil segment Dec 31 21:15:46.344867048 (629h46m46.215950093s) [upload=551868]: Waiting for error group managing segments... Dec 31 21:15:46.624836599 (629h46m46.495919634s) [noise=true , address=145.79.227.48:4303 , piece_id=5CQNY7DPO4HVD4JXS4WBJEN3I7ZUPUNSY45HCEC25VZSW7R4Y5BQ, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.624940787 (629h46m46.496023832s) [noise=true , address=145.79.227.15:4875 , piece_id=KQMUIZMYY5TKHKLOVE2HYQEU5WYCZOOB5AJVROAE5WVSGZJDXMOQ, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.636984079 (629h46m46.508067114s) [noise=true , address=145.79.227.51:4410 , piece_id=QH57IGTPJZ57O6EHENLY2JU7VKVOZSYVGBVODRC2I5LL2AWZHV6Q, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.641160311 (629h46m46.512243356s) [noise=true , address=145.79.227.13:4803 , piece_id=TZJI24W2EQRLGP5NOL3KVN7R5HDJMEEWPH5WCXZG5WTDMA4FUDXQ, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.643189464 (629h46m46.514272509s) [noise=true , address=145.79.227.52:4446 , piece_id=7QK754CTPSPTRHM6LCLEBKRKSYLHJMAM5FZOJNDJKXZFJITSHWRQ, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.673375523 (629h46m46.544458568s) [noise=true , address=145.79.227.31:4799 , piece_id=54IF73MWOTIOJ4WXV3WGUJMK7JMLGIVZIEOSI3PTYHZE5M263JFQ, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.677399746 (629h46m46.548482801s) [noise=true , address=145.79.227.11:4730 , piece_id=BQLBPYXQPS7NTNKWT3NE4Z4VZPCKRAMBFBEAY3BVF7Q34FHZD6AQ, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.710091896 (629h46m46.581174931s) [noise=true , address=145.79.227.10:4403 , piece_id=CHUKGI35OWLDMZG2GGZHMC7RL6RDGNGIXT3KZZHBWKRXPLSJ3PMA, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.716974232 (629h46m46.588057267s) [noise=true , address=145.79.227.46:4603 , piece_id=NNYCFUISSSOYPBTF2YGIMM7BNAW3EIYZTFKBGMQVEA5MNEHU6GQA, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.799312184 (629h46m46.670395219s) [noise=true , address=77.90.31.13:4627 , piece_id=CC55YZD4CWFZBRQASU6VB33RMJZ6RYGEZ6Y7SNLYILCRVRDBMMUQ, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.929310008 (629h46m46.800393053s) [noise=true , address=45.91.116.12:4225 , piece_id=QF3QQSR5EF5T42IFPEWOCIAQBPAQPNVGBY5LLRO2IUJ5VHC4QNZA, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.930088710 (629h46m46.801171755s) [noise=true , address=77.90.31.21:4880 , piece_id=UEJYYPH4WYBOEV2ZMZDDHNGIHT2JH57AIQY3XZ2HLBV6F5QOYCTQ, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.943538120 (629h46m46.814621155s) [noise=true , address=45.91.116.11:4118 , piece_id=U4ZYIZQIZLHUQDHRK43NC2XEXOR3LDSS32WLEHYA5QGNIW4R5ZPQ, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.943559901 (629h46m46.814642946s) [seg_pos={0 0}, upload=551868]: Segment reached optimal threshold of 13 pieces. Dec 31 21:15:46.943767576 (629h46m46.814850621s) [noise=true , address=45.91.116.13:4156 , piece_id=CVX3XD5GRJVGPMTIO3GQGMPL3V76CREYNE4VOS22CNW3KNLH5VXA, seg_pos={0 0}, upload=551868]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 31 21:15:46.943777847 (629h46m46.814860892s) [noise=true , address=45.91.116.13:4156 , piece_id=CVX3XD5GRJVGPMTIO3GQGMPL3V76CREYNE4VOS22CNW3KNLH5VXA, seg_pos={0 0}, upload=551868]: Long tail cancellation due to optimal threshold Dec 31 21:15:46.944026911 (629h46m46.815109956s) [noise=true , address=45.91.116.6:4973 , piece_id=NSEDZOAZWJEHL2YJPRPT3Y37HLGYQSLCSIRP62IHYLFTDVFEIUUQ, seg_pos={0 0}, upload=551868]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF Dec 31 21:15:46.944037180 (629h46m46.815120215s) [noise=true , address=45.91.116.6:4973 , piece_id=NSEDZOAZWJEHL2YJPRPT3Y37HLGYQSLCSIRP62IHYLFTDVFEIUUQ, seg_pos={0 0}, upload=551868]: Long tail cancellation due to optimal threshold Dec 31 21:15:46.944059453 (629h46m46.815142498s) [noise=true , address=45.91.116.10:4083 , piece_id=EKQLXTCN4KSWG4GTBSB6FVVSBEP4L5ONBKXEMEQIOBYDYA5ZOUIA, seg_pos={0 0}, upload=551868]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 31 21:15:46.944068169 (629h46m46.815151204s) [noise=true , address=45.91.116.10:4083 , piece_id=EKQLXTCN4KSWG4GTBSB6FVVSBEP4L5ONBKXEMEQIOBYDYA5ZOUIA, seg_pos={0 0}, upload=551868]: Long tail cancellation due to optimal threshold Dec 31 21:15:46.944090411 (629h46m46.815173446s) [seg_pos={0 0}, upload=551868]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 21:15:46.944174672 (629h46m46.815257717s) [upload=551868]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 21:15:46.944181034 (629h46m46.815264069s) [upload=551868]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 21:15:47.234878081 (629h46m47.105961136s) [upload=551868]: Done uploading segments... Dec 31 21:15:47.234901195 (629h46m47.10598424s) [upload=551868]: Upload finished. err: <nil> Duration: 5.960725262s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [..X......X....XX........X...X.......X.....X...XX.X..........] duration: 5.816814077s trace: https://jaeger.dev.storj.io/trace/5f2b7d22d21cf0f8 FAIL (2025-12-31 21:22:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: U4ZYIZQIZLHUQDHRK43NC2XEXOR3LDSS32WLEHYA5QGNIW4R5ZPQ) context deadline exceeded; piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: UEJYYPH4WYBOEV2ZMZDDHNGIHT2JH57AIQY3XZ2HLBV6F5QOYCTQ) context deadline exceeded; piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: CC55YZD4CWFZBRQASU6VB33RMJZ6RYGEZ6Y7SNLYILCRVRDBMMUQ) context deadline exceeded ======LOGS====== Dec 31 21:15:44.386938901 (629h46m44.258021936s) [upload=551868]: Starting upload Dec 31 21:15:44.386994888 (629h46m44.258077923s) [upload=551868]: Done starting upload Dec 31 21:15:44.387183197 (629h46m44.258266242s) [upload=551868]: Uploading segments... Dec 31 21:15:44.387281093 (629h46m44.258364158s) [upload=551868]: Got next segment. Inline: false Dec 31 21:15:44.387329566 (629h46m44.258412621s) [upload=551868]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 21:15:44.387337030 (629h46m44.258420075s) [upload=551868]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 21:15:44.637074656 (629h46m44.508157691s) [seg_pos={0 0}, upload=551868]: Begin upload segment... Dec 31 21:15:44.637126884 (629h46m44.50820992s) [seg_pos={0 0}, upload=551868]: Done begin upload segment. Dec 31 21:15:44.637137575 (629h46m44.50822061s) [noise=true , address=45.91.116.13:4156 , piece_id=CVX3XD5GRJVGPMTIO3GQGMPL3V76CREYNE4VOS22CNW3KNLH5VXA, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.637158976 (629h46m44.508242021s) [noise=true , address=45.91.116.12:4225 , piece_id=QF3QQSR5EF5T42IFPEWOCIAQBPAQPNVGBY5LLRO2IUJ5VHC4QNZA, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.637344248 (629h46m44.508427283s) [noise=true , address=145.79.227.48:4303 , piece_id=5CQNY7DPO4HVD4JXS4WBJEN3I7ZUPUNSY45HCEC25VZSW7R4Y5BQ, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.637642296 (629h46m44.508725331s) [noise=true , address=145.79.227.10:4403 , piece_id=CHUKGI35OWLDMZG2GGZHMC7RL6RDGNGIXT3KZZHBWKRXPLSJ3PMA, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.637935754 (629h46m44.5090188s) [noise=true , address=145.79.227.52:4446 , piece_id=7QK754CTPSPTRHM6LCLEBKRKSYLHJMAM5FZOJNDJKXZFJITSHWRQ, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.638274770 (629h46m44.509357815s) [noise=true , address=145.79.227.13:4803 , piece_id=TZJI24W2EQRLGP5NOL3KVN7R5HDJMEEWPH5WCXZG5WTDMA4FUDXQ, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.638548111 (629h46m44.509631146s) [noise=true , address=145.79.227.11:4730 , piece_id=BQLBPYXQPS7NTNKWT3NE4Z4VZPCKRAMBFBEAY3BVF7Q34FHZD6AQ, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.638806723 (629h46m44.509889768s) [noise=true , address=145.79.227.15:4875 , piece_id=KQMUIZMYY5TKHKLOVE2HYQEU5WYCZOOB5AJVROAE5WVSGZJDXMOQ, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.639124738 (629h46m44.510207783s) [noise=true , address=145.79.227.51:4410 , piece_id=QH57IGTPJZ57O6EHENLY2JU7VKVOZSYVGBVODRC2I5LL2AWZHV6Q, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.639522837 (629h46m44.510605882s) [noise=true , address=77.90.31.13:4627 , piece_id=CC55YZD4CWFZBRQASU6VB33RMJZ6RYGEZ6Y7SNLYILCRVRDBMMUQ, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.641025558 (629h46m44.512108593s) [noise=true , address=45.91.116.11:4118 , piece_id=U4ZYIZQIZLHUQDHRK43NC2XEXOR3LDSS32WLEHYA5QGNIW4R5ZPQ, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.642391289 (629h46m44.513474334s) [noise=true , address=145.79.227.46:4603 , piece_id=NNYCFUISSSOYPBTF2YGIMM7BNAW3EIYZTFKBGMQVEA5MNEHU6GQA, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.643600902 (629h46m44.514683937s) [noise=true , address=45.91.116.10:4083 , piece_id=EKQLXTCN4KSWG4GTBSB6FVVSBEP4L5ONBKXEMEQIOBYDYA5ZOUIA, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.644749149 (629h46m44.515832184s) [noise=true , address=145.79.227.31:4799 , piece_id=54IF73MWOTIOJ4WXV3WGUJMK7JMLGIVZIEOSI3PTYHZE5M263JFQ, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.646078710 (629h46m44.517161745s) [noise=true , address=77.90.31.21:4880 , piece_id=UEJYYPH4WYBOEV2ZMZDDHNGIHT2JH57AIQY3XZ2HLBV6F5QOYCTQ, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.647522739 (629h46m44.518605784s) [noise=true , address=45.91.116.6:4973 , piece_id=NSEDZOAZWJEHL2YJPRPT3Y37HLGYQSLCSIRP62IHYLFTDVFEIUUQ, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:46.344821972 (629h46m46.215905007s) [upload=551868]: Next returned nil segment Dec 31 21:15:46.344867048 (629h46m46.215950093s) [upload=551868]: Waiting for error group managing segments... Dec 31 21:15:46.624836599 (629h46m46.495919634s) [noise=true , address=145.79.227.48:4303 , piece_id=5CQNY7DPO4HVD4JXS4WBJEN3I7ZUPUNSY45HCEC25VZSW7R4Y5BQ, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.624940787 (629h46m46.496023832s) [noise=true , address=145.79.227.15:4875 , piece_id=KQMUIZMYY5TKHKLOVE2HYQEU5WYCZOOB5AJVROAE5WVSGZJDXMOQ, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.636984079 (629h46m46.508067114s) [noise=true , address=145.79.227.51:4410 , piece_id=QH57IGTPJZ57O6EHENLY2JU7VKVOZSYVGBVODRC2I5LL2AWZHV6Q, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.641160311 (629h46m46.512243356s) [noise=true , address=145.79.227.13:4803 , piece_id=TZJI24W2EQRLGP5NOL3KVN7R5HDJMEEWPH5WCXZG5WTDMA4FUDXQ, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.643189464 (629h46m46.514272509s) [noise=true , address=145.79.227.52:4446 , piece_id=7QK754CTPSPTRHM6LCLEBKRKSYLHJMAM5FZOJNDJKXZFJITSHWRQ, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.673375523 (629h46m46.544458568s) [noise=true , address=145.79.227.31:4799 , piece_id=54IF73MWOTIOJ4WXV3WGUJMK7JMLGIVZIEOSI3PTYHZE5M263JFQ, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.677399746 (629h46m46.548482801s) [noise=true , address=145.79.227.11:4730 , piece_id=BQLBPYXQPS7NTNKWT3NE4Z4VZPCKRAMBFBEAY3BVF7Q34FHZD6AQ, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.710091896 (629h46m46.581174931s) [noise=true , address=145.79.227.10:4403 , piece_id=CHUKGI35OWLDMZG2GGZHMC7RL6RDGNGIXT3KZZHBWKRXPLSJ3PMA, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.716974232 (629h46m46.588057267s) [noise=true , address=145.79.227.46:4603 , piece_id=NNYCFUISSSOYPBTF2YGIMM7BNAW3EIYZTFKBGMQVEA5MNEHU6GQA, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.799312184 (629h46m46.670395219s) [noise=true , address=77.90.31.13:4627 , piece_id=CC55YZD4CWFZBRQASU6VB33RMJZ6RYGEZ6Y7SNLYILCRVRDBMMUQ, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.929310008 (629h46m46.800393053s) [noise=true , address=45.91.116.12:4225 , piece_id=QF3QQSR5EF5T42IFPEWOCIAQBPAQPNVGBY5LLRO2IUJ5VHC4QNZA, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.930088710 (629h46m46.801171755s) [noise=true , address=77.90.31.21:4880 , piece_id=UEJYYPH4WYBOEV2ZMZDDHNGIHT2JH57AIQY3XZ2HLBV6F5QOYCTQ, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.943538120 (629h46m46.814621155s) [noise=true , address=45.91.116.11:4118 , piece_id=U4ZYIZQIZLHUQDHRK43NC2XEXOR3LDSS32WLEHYA5QGNIW4R5ZPQ, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.943559901 (629h46m46.814642946s) [seg_pos={0 0}, upload=551868]: Segment reached optimal threshold of 13 pieces. Dec 31 21:15:46.943767576 (629h46m46.814850621s) [noise=true , address=45.91.116.13:4156 , piece_id=CVX3XD5GRJVGPMTIO3GQGMPL3V76CREYNE4VOS22CNW3KNLH5VXA, seg_pos={0 0}, upload=551868]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 31 21:15:46.943777847 (629h46m46.814860892s) [noise=true , address=45.91.116.13:4156 , piece_id=CVX3XD5GRJVGPMTIO3GQGMPL3V76CREYNE4VOS22CNW3KNLH5VXA, seg_pos={0 0}, upload=551868]: Long tail cancellation due to optimal threshold Dec 31 21:15:46.944026911 (629h46m46.815109956s) [noise=true , address=45.91.116.6:4973 , piece_id=NSEDZOAZWJEHL2YJPRPT3Y37HLGYQSLCSIRP62IHYLFTDVFEIUUQ, seg_pos={0 0}, upload=551868]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF Dec 31 21:15:46.944037180 (629h46m46.815120215s) [noise=true , address=45.91.116.6:4973 , piece_id=NSEDZOAZWJEHL2YJPRPT3Y37HLGYQSLCSIRP62IHYLFTDVFEIUUQ, seg_pos={0 0}, upload=551868]: Long tail cancellation due to optimal threshold Dec 31 21:15:46.944059453 (629h46m46.815142498s) [noise=true , address=45.91.116.10:4083 , piece_id=EKQLXTCN4KSWG4GTBSB6FVVSBEP4L5ONBKXEMEQIOBYDYA5ZOUIA, seg_pos={0 0}, upload=551868]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 31 21:15:46.944068169 (629h46m46.815151204s) [noise=true , address=45.91.116.10:4083 , piece_id=EKQLXTCN4KSWG4GTBSB6FVVSBEP4L5ONBKXEMEQIOBYDYA5ZOUIA, seg_pos={0 0}, upload=551868]: Long tail cancellation due to optimal threshold Dec 31 21:15:46.944090411 (629h46m46.815173446s) [seg_pos={0 0}, upload=551868]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 21:15:46.944174672 (629h46m46.815257717s) [upload=551868]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 21:15:46.944181034 (629h46m46.815264069s) [upload=551868]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 21:15:47.234878081 (629h46m47.105961136s) [upload=551868]: Done uploading segments... Dec 31 21:15:47.234901195 (629h46m47.10598424s) [upload=551868]: Upload finished. err: <nil> Duration: 5.81683748s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [...X......X....XX........X...X.......X.....X...XX.X.........] duration: 6.009041222s trace: https://jaeger.dev.storj.io/trace/5f2b7d22d21cf0f8 FAIL (2025-12-31 21:22:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: U4ZYIZQIZLHUQDHRK43NC2XEXOR3LDSS32WLEHYA5QGNIW4R5ZPQ) context deadline exceeded; piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: UEJYYPH4WYBOEV2ZMZDDHNGIHT2JH57AIQY3XZ2HLBV6F5QOYCTQ) context deadline exceeded; piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: CC55YZD4CWFZBRQASU6VB33RMJZ6RYGEZ6Y7SNLYILCRVRDBMMUQ) context deadline exceeded ======LOGS====== Dec 31 21:15:44.386938901 (629h46m44.258021936s) [upload=551868]: Starting upload Dec 31 21:15:44.386994888 (629h46m44.258077923s) [upload=551868]: Done starting upload Dec 31 21:15:44.387183197 (629h46m44.258266242s) [upload=551868]: Uploading segments... Dec 31 21:15:44.387281093 (629h46m44.258364158s) [upload=551868]: Got next segment. Inline: false Dec 31 21:15:44.387329566 (629h46m44.258412621s) [upload=551868]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 21:15:44.387337030 (629h46m44.258420075s) [upload=551868]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 21:15:44.637074656 (629h46m44.508157691s) [seg_pos={0 0}, upload=551868]: Begin upload segment... Dec 31 21:15:44.637126884 (629h46m44.50820992s) [seg_pos={0 0}, upload=551868]: Done begin upload segment. Dec 31 21:15:44.637137575 (629h46m44.50822061s) [noise=true , address=45.91.116.13:4156 , piece_id=CVX3XD5GRJVGPMTIO3GQGMPL3V76CREYNE4VOS22CNW3KNLH5VXA, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.637158976 (629h46m44.508242021s) [noise=true , address=45.91.116.12:4225 , piece_id=QF3QQSR5EF5T42IFPEWOCIAQBPAQPNVGBY5LLRO2IUJ5VHC4QNZA, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.637344248 (629h46m44.508427283s) [noise=true , address=145.79.227.48:4303 , piece_id=5CQNY7DPO4HVD4JXS4WBJEN3I7ZUPUNSY45HCEC25VZSW7R4Y5BQ, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.637642296 (629h46m44.508725331s) [noise=true , address=145.79.227.10:4403 , piece_id=CHUKGI35OWLDMZG2GGZHMC7RL6RDGNGIXT3KZZHBWKRXPLSJ3PMA, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.637935754 (629h46m44.5090188s) [noise=true , address=145.79.227.52:4446 , piece_id=7QK754CTPSPTRHM6LCLEBKRKSYLHJMAM5FZOJNDJKXZFJITSHWRQ, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.638274770 (629h46m44.509357815s) [noise=true , address=145.79.227.13:4803 , piece_id=TZJI24W2EQRLGP5NOL3KVN7R5HDJMEEWPH5WCXZG5WTDMA4FUDXQ, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.638548111 (629h46m44.509631146s) [noise=true , address=145.79.227.11:4730 , piece_id=BQLBPYXQPS7NTNKWT3NE4Z4VZPCKRAMBFBEAY3BVF7Q34FHZD6AQ, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.638806723 (629h46m44.509889768s) [noise=true , address=145.79.227.15:4875 , piece_id=KQMUIZMYY5TKHKLOVE2HYQEU5WYCZOOB5AJVROAE5WVSGZJDXMOQ, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.639124738 (629h46m44.510207783s) [noise=true , address=145.79.227.51:4410 , piece_id=QH57IGTPJZ57O6EHENLY2JU7VKVOZSYVGBVODRC2I5LL2AWZHV6Q, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.639522837 (629h46m44.510605882s) [noise=true , address=77.90.31.13:4627 , piece_id=CC55YZD4CWFZBRQASU6VB33RMJZ6RYGEZ6Y7SNLYILCRVRDBMMUQ, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.641025558 (629h46m44.512108593s) [noise=true , address=45.91.116.11:4118 , piece_id=U4ZYIZQIZLHUQDHRK43NC2XEXOR3LDSS32WLEHYA5QGNIW4R5ZPQ, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.642391289 (629h46m44.513474334s) [noise=true , address=145.79.227.46:4603 , piece_id=NNYCFUISSSOYPBTF2YGIMM7BNAW3EIYZTFKBGMQVEA5MNEHU6GQA, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.643600902 (629h46m44.514683937s) [noise=true , address=45.91.116.10:4083 , piece_id=EKQLXTCN4KSWG4GTBSB6FVVSBEP4L5ONBKXEMEQIOBYDYA5ZOUIA, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.644749149 (629h46m44.515832184s) [noise=true , address=145.79.227.31:4799 , piece_id=54IF73MWOTIOJ4WXV3WGUJMK7JMLGIVZIEOSI3PTYHZE5M263JFQ, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.646078710 (629h46m44.517161745s) [noise=true , address=77.90.31.21:4880 , piece_id=UEJYYPH4WYBOEV2ZMZDDHNGIHT2JH57AIQY3XZ2HLBV6F5QOYCTQ, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.647522739 (629h46m44.518605784s) [noise=true , address=45.91.116.6:4973 , piece_id=NSEDZOAZWJEHL2YJPRPT3Y37HLGYQSLCSIRP62IHYLFTDVFEIUUQ, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:46.344821972 (629h46m46.215905007s) [upload=551868]: Next returned nil segment Dec 31 21:15:46.344867048 (629h46m46.215950093s) [upload=551868]: Waiting for error group managing segments... Dec 31 21:15:46.624836599 (629h46m46.495919634s) [noise=true , address=145.79.227.48:4303 , piece_id=5CQNY7DPO4HVD4JXS4WBJEN3I7ZUPUNSY45HCEC25VZSW7R4Y5BQ, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.624940787 (629h46m46.496023832s) [noise=true , address=145.79.227.15:4875 , piece_id=KQMUIZMYY5TKHKLOVE2HYQEU5WYCZOOB5AJVROAE5WVSGZJDXMOQ, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.636984079 (629h46m46.508067114s) [noise=true , address=145.79.227.51:4410 , piece_id=QH57IGTPJZ57O6EHENLY2JU7VKVOZSYVGBVODRC2I5LL2AWZHV6Q, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.641160311 (629h46m46.512243356s) [noise=true , address=145.79.227.13:4803 , piece_id=TZJI24W2EQRLGP5NOL3KVN7R5HDJMEEWPH5WCXZG5WTDMA4FUDXQ, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.643189464 (629h46m46.514272509s) [noise=true , address=145.79.227.52:4446 , piece_id=7QK754CTPSPTRHM6LCLEBKRKSYLHJMAM5FZOJNDJKXZFJITSHWRQ, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.673375523 (629h46m46.544458568s) [noise=true , address=145.79.227.31:4799 , piece_id=54IF73MWOTIOJ4WXV3WGUJMK7JMLGIVZIEOSI3PTYHZE5M263JFQ, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.677399746 (629h46m46.548482801s) [noise=true , address=145.79.227.11:4730 , piece_id=BQLBPYXQPS7NTNKWT3NE4Z4VZPCKRAMBFBEAY3BVF7Q34FHZD6AQ, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.710091896 (629h46m46.581174931s) [noise=true , address=145.79.227.10:4403 , piece_id=CHUKGI35OWLDMZG2GGZHMC7RL6RDGNGIXT3KZZHBWKRXPLSJ3PMA, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.716974232 (629h46m46.588057267s) [noise=true , address=145.79.227.46:4603 , piece_id=NNYCFUISSSOYPBTF2YGIMM7BNAW3EIYZTFKBGMQVEA5MNEHU6GQA, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.799312184 (629h46m46.670395219s) [noise=true , address=77.90.31.13:4627 , piece_id=CC55YZD4CWFZBRQASU6VB33RMJZ6RYGEZ6Y7SNLYILCRVRDBMMUQ, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.929310008 (629h46m46.800393053s) [noise=true , address=45.91.116.12:4225 , piece_id=QF3QQSR5EF5T42IFPEWOCIAQBPAQPNVGBY5LLRO2IUJ5VHC4QNZA, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.930088710 (629h46m46.801171755s) [noise=true , address=77.90.31.21:4880 , piece_id=UEJYYPH4WYBOEV2ZMZDDHNGIHT2JH57AIQY3XZ2HLBV6F5QOYCTQ, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.943538120 (629h46m46.814621155s) [noise=true , address=45.91.116.11:4118 , piece_id=U4ZYIZQIZLHUQDHRK43NC2XEXOR3LDSS32WLEHYA5QGNIW4R5ZPQ, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.943559901 (629h46m46.814642946s) [seg_pos={0 0}, upload=551868]: Segment reached optimal threshold of 13 pieces. Dec 31 21:15:46.943767576 (629h46m46.814850621s) [noise=true , address=45.91.116.13:4156 , piece_id=CVX3XD5GRJVGPMTIO3GQGMPL3V76CREYNE4VOS22CNW3KNLH5VXA, seg_pos={0 0}, upload=551868]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 31 21:15:46.943777847 (629h46m46.814860892s) [noise=true , address=45.91.116.13:4156 , piece_id=CVX3XD5GRJVGPMTIO3GQGMPL3V76CREYNE4VOS22CNW3KNLH5VXA, seg_pos={0 0}, upload=551868]: Long tail cancellation due to optimal threshold Dec 31 21:15:46.944026911 (629h46m46.815109956s) [noise=true , address=45.91.116.6:4973 , piece_id=NSEDZOAZWJEHL2YJPRPT3Y37HLGYQSLCSIRP62IHYLFTDVFEIUUQ, seg_pos={0 0}, upload=551868]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF Dec 31 21:15:46.944037180 (629h46m46.815120215s) [noise=true , address=45.91.116.6:4973 , piece_id=NSEDZOAZWJEHL2YJPRPT3Y37HLGYQSLCSIRP62IHYLFTDVFEIUUQ, seg_pos={0 0}, upload=551868]: Long tail cancellation due to optimal threshold Dec 31 21:15:46.944059453 (629h46m46.815142498s) [noise=true , address=45.91.116.10:4083 , piece_id=EKQLXTCN4KSWG4GTBSB6FVVSBEP4L5ONBKXEMEQIOBYDYA5ZOUIA, seg_pos={0 0}, upload=551868]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 31 21:15:46.944068169 (629h46m46.815151204s) [noise=true , address=45.91.116.10:4083 , piece_id=EKQLXTCN4KSWG4GTBSB6FVVSBEP4L5ONBKXEMEQIOBYDYA5ZOUIA, seg_pos={0 0}, upload=551868]: Long tail cancellation due to optimal threshold Dec 31 21:15:46.944090411 (629h46m46.815173446s) [seg_pos={0 0}, upload=551868]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 21:15:46.944174672 (629h46m46.815257717s) [upload=551868]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 21:15:46.944181034 (629h46m46.815264069s) [upload=551868]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 21:15:47.234878081 (629h46m47.105961136s) [upload=551868]: Done uploading segments... Dec 31 21:15:47.234901195 (629h46m47.10598424s) [upload=551868]: Upload finished. err: <nil> Duration: 6.0090719s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [....X......X....XX........X...X.......X.....X...XX.X........] duration: 6.067763017s trace: https://jaeger.dev.storj.io/trace/5f2b7d22d21cf0f8 FAIL (2025-12-31 21:21:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: U4ZYIZQIZLHUQDHRK43NC2XEXOR3LDSS32WLEHYA5QGNIW4R5ZPQ) context deadline exceeded; piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: UEJYYPH4WYBOEV2ZMZDDHNGIHT2JH57AIQY3XZ2HLBV6F5QOYCTQ) context deadline exceeded; piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: CC55YZD4CWFZBRQASU6VB33RMJZ6RYGEZ6Y7SNLYILCRVRDBMMUQ) context deadline exceeded ======LOGS====== Dec 31 21:15:44.386938901 (629h46m44.258021936s) [upload=551868]: Starting upload Dec 31 21:15:44.386994888 (629h46m44.258077923s) [upload=551868]: Done starting upload Dec 31 21:15:44.387183197 (629h46m44.258266242s) [upload=551868]: Uploading segments... Dec 31 21:15:44.387281093 (629h46m44.258364158s) [upload=551868]: Got next segment. Inline: false Dec 31 21:15:44.387329566 (629h46m44.258412621s) [upload=551868]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 21:15:44.387337030 (629h46m44.258420075s) [upload=551868]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 21:15:44.637074656 (629h46m44.508157691s) [seg_pos={0 0}, upload=551868]: Begin upload segment... Dec 31 21:15:44.637126884 (629h46m44.50820992s) [seg_pos={0 0}, upload=551868]: Done begin upload segment. Dec 31 21:15:44.637137575 (629h46m44.50822061s) [noise=true , address=45.91.116.13:4156 , piece_id=CVX3XD5GRJVGPMTIO3GQGMPL3V76CREYNE4VOS22CNW3KNLH5VXA, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.637158976 (629h46m44.508242021s) [noise=true , address=45.91.116.12:4225 , piece_id=QF3QQSR5EF5T42IFPEWOCIAQBPAQPNVGBY5LLRO2IUJ5VHC4QNZA, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.637344248 (629h46m44.508427283s) [noise=true , address=145.79.227.48:4303 , piece_id=5CQNY7DPO4HVD4JXS4WBJEN3I7ZUPUNSY45HCEC25VZSW7R4Y5BQ, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.637642296 (629h46m44.508725331s) [noise=true , address=145.79.227.10:4403 , piece_id=CHUKGI35OWLDMZG2GGZHMC7RL6RDGNGIXT3KZZHBWKRXPLSJ3PMA, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.637935754 (629h46m44.5090188s) [noise=true , address=145.79.227.52:4446 , piece_id=7QK754CTPSPTRHM6LCLEBKRKSYLHJMAM5FZOJNDJKXZFJITSHWRQ, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.638274770 (629h46m44.509357815s) [noise=true , address=145.79.227.13:4803 , piece_id=TZJI24W2EQRLGP5NOL3KVN7R5HDJMEEWPH5WCXZG5WTDMA4FUDXQ, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.638548111 (629h46m44.509631146s) [noise=true , address=145.79.227.11:4730 , piece_id=BQLBPYXQPS7NTNKWT3NE4Z4VZPCKRAMBFBEAY3BVF7Q34FHZD6AQ, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.638806723 (629h46m44.509889768s) [noise=true , address=145.79.227.15:4875 , piece_id=KQMUIZMYY5TKHKLOVE2HYQEU5WYCZOOB5AJVROAE5WVSGZJDXMOQ, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.639124738 (629h46m44.510207783s) [noise=true , address=145.79.227.51:4410 , piece_id=QH57IGTPJZ57O6EHENLY2JU7VKVOZSYVGBVODRC2I5LL2AWZHV6Q, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.639522837 (629h46m44.510605882s) [noise=true , address=77.90.31.13:4627 , piece_id=CC55YZD4CWFZBRQASU6VB33RMJZ6RYGEZ6Y7SNLYILCRVRDBMMUQ, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.641025558 (629h46m44.512108593s) [noise=true , address=45.91.116.11:4118 , piece_id=U4ZYIZQIZLHUQDHRK43NC2XEXOR3LDSS32WLEHYA5QGNIW4R5ZPQ, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.642391289 (629h46m44.513474334s) [noise=true , address=145.79.227.46:4603 , piece_id=NNYCFUISSSOYPBTF2YGIMM7BNAW3EIYZTFKBGMQVEA5MNEHU6GQA, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.643600902 (629h46m44.514683937s) [noise=true , address=45.91.116.10:4083 , piece_id=EKQLXTCN4KSWG4GTBSB6FVVSBEP4L5ONBKXEMEQIOBYDYA5ZOUIA, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.644749149 (629h46m44.515832184s) [noise=true , address=145.79.227.31:4799 , piece_id=54IF73MWOTIOJ4WXV3WGUJMK7JMLGIVZIEOSI3PTYHZE5M263JFQ, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.646078710 (629h46m44.517161745s) [noise=true , address=77.90.31.21:4880 , piece_id=UEJYYPH4WYBOEV2ZMZDDHNGIHT2JH57AIQY3XZ2HLBV6F5QOYCTQ, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.647522739 (629h46m44.518605784s) [noise=true , address=45.91.116.6:4973 , piece_id=NSEDZOAZWJEHL2YJPRPT3Y37HLGYQSLCSIRP62IHYLFTDVFEIUUQ, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:46.344821972 (629h46m46.215905007s) [upload=551868]: Next returned nil segment Dec 31 21:15:46.344867048 (629h46m46.215950093s) [upload=551868]: Waiting for error group managing segments... Dec 31 21:15:46.624836599 (629h46m46.495919634s) [noise=true , address=145.79.227.48:4303 , piece_id=5CQNY7DPO4HVD4JXS4WBJEN3I7ZUPUNSY45HCEC25VZSW7R4Y5BQ, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.624940787 (629h46m46.496023832s) [noise=true , address=145.79.227.15:4875 , piece_id=KQMUIZMYY5TKHKLOVE2HYQEU5WYCZOOB5AJVROAE5WVSGZJDXMOQ, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.636984079 (629h46m46.508067114s) [noise=true , address=145.79.227.51:4410 , piece_id=QH57IGTPJZ57O6EHENLY2JU7VKVOZSYVGBVODRC2I5LL2AWZHV6Q, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.641160311 (629h46m46.512243356s) [noise=true , address=145.79.227.13:4803 , piece_id=TZJI24W2EQRLGP5NOL3KVN7R5HDJMEEWPH5WCXZG5WTDMA4FUDXQ, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.643189464 (629h46m46.514272509s) [noise=true , address=145.79.227.52:4446 , piece_id=7QK754CTPSPTRHM6LCLEBKRKSYLHJMAM5FZOJNDJKXZFJITSHWRQ, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.673375523 (629h46m46.544458568s) [noise=true , address=145.79.227.31:4799 , piece_id=54IF73MWOTIOJ4WXV3WGUJMK7JMLGIVZIEOSI3PTYHZE5M263JFQ, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.677399746 (629h46m46.548482801s) [noise=true , address=145.79.227.11:4730 , piece_id=BQLBPYXQPS7NTNKWT3NE4Z4VZPCKRAMBFBEAY3BVF7Q34FHZD6AQ, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.710091896 (629h46m46.581174931s) [noise=true , address=145.79.227.10:4403 , piece_id=CHUKGI35OWLDMZG2GGZHMC7RL6RDGNGIXT3KZZHBWKRXPLSJ3PMA, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.716974232 (629h46m46.588057267s) [noise=true , address=145.79.227.46:4603 , piece_id=NNYCFUISSSOYPBTF2YGIMM7BNAW3EIYZTFKBGMQVEA5MNEHU6GQA, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.799312184 (629h46m46.670395219s) [noise=true , address=77.90.31.13:4627 , piece_id=CC55YZD4CWFZBRQASU6VB33RMJZ6RYGEZ6Y7SNLYILCRVRDBMMUQ, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.929310008 (629h46m46.800393053s) [noise=true , address=45.91.116.12:4225 , piece_id=QF3QQSR5EF5T42IFPEWOCIAQBPAQPNVGBY5LLRO2IUJ5VHC4QNZA, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.930088710 (629h46m46.801171755s) [noise=true , address=77.90.31.21:4880 , piece_id=UEJYYPH4WYBOEV2ZMZDDHNGIHT2JH57AIQY3XZ2HLBV6F5QOYCTQ, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.943538120 (629h46m46.814621155s) [noise=true , address=45.91.116.11:4118 , piece_id=U4ZYIZQIZLHUQDHRK43NC2XEXOR3LDSS32WLEHYA5QGNIW4R5ZPQ, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.943559901 (629h46m46.814642946s) [seg_pos={0 0}, upload=551868]: Segment reached optimal threshold of 13 pieces. Dec 31 21:15:46.943767576 (629h46m46.814850621s) [noise=true , address=45.91.116.13:4156 , piece_id=CVX3XD5GRJVGPMTIO3GQGMPL3V76CREYNE4VOS22CNW3KNLH5VXA, seg_pos={0 0}, upload=551868]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 31 21:15:46.943777847 (629h46m46.814860892s) [noise=true , address=45.91.116.13:4156 , piece_id=CVX3XD5GRJVGPMTIO3GQGMPL3V76CREYNE4VOS22CNW3KNLH5VXA, seg_pos={0 0}, upload=551868]: Long tail cancellation due to optimal threshold Dec 31 21:15:46.944026911 (629h46m46.815109956s) [noise=true , address=45.91.116.6:4973 , piece_id=NSEDZOAZWJEHL2YJPRPT3Y37HLGYQSLCSIRP62IHYLFTDVFEIUUQ, seg_pos={0 0}, upload=551868]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF Dec 31 21:15:46.944037180 (629h46m46.815120215s) [noise=true , address=45.91.116.6:4973 , piece_id=NSEDZOAZWJEHL2YJPRPT3Y37HLGYQSLCSIRP62IHYLFTDVFEIUUQ, seg_pos={0 0}, upload=551868]: Long tail cancellation due to optimal threshold Dec 31 21:15:46.944059453 (629h46m46.815142498s) [noise=true , address=45.91.116.10:4083 , piece_id=EKQLXTCN4KSWG4GTBSB6FVVSBEP4L5ONBKXEMEQIOBYDYA5ZOUIA, seg_pos={0 0}, upload=551868]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 31 21:15:46.944068169 (629h46m46.815151204s) [noise=true , address=45.91.116.10:4083 , piece_id=EKQLXTCN4KSWG4GTBSB6FVVSBEP4L5ONBKXEMEQIOBYDYA5ZOUIA, seg_pos={0 0}, upload=551868]: Long tail cancellation due to optimal threshold Dec 31 21:15:46.944090411 (629h46m46.815173446s) [seg_pos={0 0}, upload=551868]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 21:15:46.944174672 (629h46m46.815257717s) [upload=551868]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 21:15:46.944181034 (629h46m46.815264069s) [upload=551868]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 21:15:47.234878081 (629h46m47.105961136s) [upload=551868]: Done uploading segments... Dec 31 21:15:47.234901195 (629h46m47.10598424s) [upload=551868]: Upload finished. err: <nil> Duration: 6.067792143s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [X....X......X....XX........X...X.......X.....X...XX.X.......] duration: 5.862326852s trace: https://jaeger.dev.storj.io/trace/5f2b7d22d21cf0f8 FAIL (2025-12-31 21:20:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: U4ZYIZQIZLHUQDHRK43NC2XEXOR3LDSS32WLEHYA5QGNIW4R5ZPQ) context deadline exceeded; piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: UEJYYPH4WYBOEV2ZMZDDHNGIHT2JH57AIQY3XZ2HLBV6F5QOYCTQ) context deadline exceeded; piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: CC55YZD4CWFZBRQASU6VB33RMJZ6RYGEZ6Y7SNLYILCRVRDBMMUQ) context deadline exceeded ======LOGS====== Dec 31 21:15:44.386938901 (629h46m44.258021936s) [upload=551868]: Starting upload Dec 31 21:15:44.386994888 (629h46m44.258077923s) [upload=551868]: Done starting upload Dec 31 21:15:44.387183197 (629h46m44.258266242s) [upload=551868]: Uploading segments... Dec 31 21:15:44.387281093 (629h46m44.258364158s) [upload=551868]: Got next segment. Inline: false Dec 31 21:15:44.387329566 (629h46m44.258412621s) [upload=551868]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 21:15:44.387337030 (629h46m44.258420075s) [upload=551868]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 21:15:44.637074656 (629h46m44.508157691s) [seg_pos={0 0}, upload=551868]: Begin upload segment... Dec 31 21:15:44.637126884 (629h46m44.50820992s) [seg_pos={0 0}, upload=551868]: Done begin upload segment. Dec 31 21:15:44.637137575 (629h46m44.50822061s) [noise=true , address=45.91.116.13:4156 , piece_id=CVX3XD5GRJVGPMTIO3GQGMPL3V76CREYNE4VOS22CNW3KNLH5VXA, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.637158976 (629h46m44.508242021s) [noise=true , address=45.91.116.12:4225 , piece_id=QF3QQSR5EF5T42IFPEWOCIAQBPAQPNVGBY5LLRO2IUJ5VHC4QNZA, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.637344248 (629h46m44.508427283s) [noise=true , address=145.79.227.48:4303 , piece_id=5CQNY7DPO4HVD4JXS4WBJEN3I7ZUPUNSY45HCEC25VZSW7R4Y5BQ, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.637642296 (629h46m44.508725331s) [noise=true , address=145.79.227.10:4403 , piece_id=CHUKGI35OWLDMZG2GGZHMC7RL6RDGNGIXT3KZZHBWKRXPLSJ3PMA, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.637935754 (629h46m44.5090188s) [noise=true , address=145.79.227.52:4446 , piece_id=7QK754CTPSPTRHM6LCLEBKRKSYLHJMAM5FZOJNDJKXZFJITSHWRQ, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.638274770 (629h46m44.509357815s) [noise=true , address=145.79.227.13:4803 , piece_id=TZJI24W2EQRLGP5NOL3KVN7R5HDJMEEWPH5WCXZG5WTDMA4FUDXQ, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.638548111 (629h46m44.509631146s) [noise=true , address=145.79.227.11:4730 , piece_id=BQLBPYXQPS7NTNKWT3NE4Z4VZPCKRAMBFBEAY3BVF7Q34FHZD6AQ, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.638806723 (629h46m44.509889768s) [noise=true , address=145.79.227.15:4875 , piece_id=KQMUIZMYY5TKHKLOVE2HYQEU5WYCZOOB5AJVROAE5WVSGZJDXMOQ, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.639124738 (629h46m44.510207783s) [noise=true , address=145.79.227.51:4410 , piece_id=QH57IGTPJZ57O6EHENLY2JU7VKVOZSYVGBVODRC2I5LL2AWZHV6Q, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.639522837 (629h46m44.510605882s) [noise=true , address=77.90.31.13:4627 , piece_id=CC55YZD4CWFZBRQASU6VB33RMJZ6RYGEZ6Y7SNLYILCRVRDBMMUQ, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.641025558 (629h46m44.512108593s) [noise=true , address=45.91.116.11:4118 , piece_id=U4ZYIZQIZLHUQDHRK43NC2XEXOR3LDSS32WLEHYA5QGNIW4R5ZPQ, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.642391289 (629h46m44.513474334s) [noise=true , address=145.79.227.46:4603 , piece_id=NNYCFUISSSOYPBTF2YGIMM7BNAW3EIYZTFKBGMQVEA5MNEHU6GQA, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.643600902 (629h46m44.514683937s) [noise=true , address=45.91.116.10:4083 , piece_id=EKQLXTCN4KSWG4GTBSB6FVVSBEP4L5ONBKXEMEQIOBYDYA5ZOUIA, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.644749149 (629h46m44.515832184s) [noise=true , address=145.79.227.31:4799 , piece_id=54IF73MWOTIOJ4WXV3WGUJMK7JMLGIVZIEOSI3PTYHZE5M263JFQ, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.646078710 (629h46m44.517161745s) [noise=true , address=77.90.31.21:4880 , piece_id=UEJYYPH4WYBOEV2ZMZDDHNGIHT2JH57AIQY3XZ2HLBV6F5QOYCTQ, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.647522739 (629h46m44.518605784s) [noise=true , address=45.91.116.6:4973 , piece_id=NSEDZOAZWJEHL2YJPRPT3Y37HLGYQSLCSIRP62IHYLFTDVFEIUUQ, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:46.344821972 (629h46m46.215905007s) [upload=551868]: Next returned nil segment Dec 31 21:15:46.344867048 (629h46m46.215950093s) [upload=551868]: Waiting for error group managing segments... Dec 31 21:15:46.624836599 (629h46m46.495919634s) [noise=true , address=145.79.227.48:4303 , piece_id=5CQNY7DPO4HVD4JXS4WBJEN3I7ZUPUNSY45HCEC25VZSW7R4Y5BQ, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.624940787 (629h46m46.496023832s) [noise=true , address=145.79.227.15:4875 , piece_id=KQMUIZMYY5TKHKLOVE2HYQEU5WYCZOOB5AJVROAE5WVSGZJDXMOQ, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.636984079 (629h46m46.508067114s) [noise=true , address=145.79.227.51:4410 , piece_id=QH57IGTPJZ57O6EHENLY2JU7VKVOZSYVGBVODRC2I5LL2AWZHV6Q, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.641160311 (629h46m46.512243356s) [noise=true , address=145.79.227.13:4803 , piece_id=TZJI24W2EQRLGP5NOL3KVN7R5HDJMEEWPH5WCXZG5WTDMA4FUDXQ, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.643189464 (629h46m46.514272509s) [noise=true , address=145.79.227.52:4446 , piece_id=7QK754CTPSPTRHM6LCLEBKRKSYLHJMAM5FZOJNDJKXZFJITSHWRQ, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.673375523 (629h46m46.544458568s) [noise=true , address=145.79.227.31:4799 , piece_id=54IF73MWOTIOJ4WXV3WGUJMK7JMLGIVZIEOSI3PTYHZE5M263JFQ, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.677399746 (629h46m46.548482801s) [noise=true , address=145.79.227.11:4730 , piece_id=BQLBPYXQPS7NTNKWT3NE4Z4VZPCKRAMBFBEAY3BVF7Q34FHZD6AQ, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.710091896 (629h46m46.581174931s) [noise=true , address=145.79.227.10:4403 , piece_id=CHUKGI35OWLDMZG2GGZHMC7RL6RDGNGIXT3KZZHBWKRXPLSJ3PMA, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.716974232 (629h46m46.588057267s) [noise=true , address=145.79.227.46:4603 , piece_id=NNYCFUISSSOYPBTF2YGIMM7BNAW3EIYZTFKBGMQVEA5MNEHU6GQA, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.799312184 (629h46m46.670395219s) [noise=true , address=77.90.31.13:4627 , piece_id=CC55YZD4CWFZBRQASU6VB33RMJZ6RYGEZ6Y7SNLYILCRVRDBMMUQ, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.929310008 (629h46m46.800393053s) [noise=true , address=45.91.116.12:4225 , piece_id=QF3QQSR5EF5T42IFPEWOCIAQBPAQPNVGBY5LLRO2IUJ5VHC4QNZA, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.930088710 (629h46m46.801171755s) [noise=true , address=77.90.31.21:4880 , piece_id=UEJYYPH4WYBOEV2ZMZDDHNGIHT2JH57AIQY3XZ2HLBV6F5QOYCTQ, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.943538120 (629h46m46.814621155s) [noise=true , address=45.91.116.11:4118 , piece_id=U4ZYIZQIZLHUQDHRK43NC2XEXOR3LDSS32WLEHYA5QGNIW4R5ZPQ, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.943559901 (629h46m46.814642946s) [seg_pos={0 0}, upload=551868]: Segment reached optimal threshold of 13 pieces. Dec 31 21:15:46.943767576 (629h46m46.814850621s) [noise=true , address=45.91.116.13:4156 , piece_id=CVX3XD5GRJVGPMTIO3GQGMPL3V76CREYNE4VOS22CNW3KNLH5VXA, seg_pos={0 0}, upload=551868]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 31 21:15:46.943777847 (629h46m46.814860892s) [noise=true , address=45.91.116.13:4156 , piece_id=CVX3XD5GRJVGPMTIO3GQGMPL3V76CREYNE4VOS22CNW3KNLH5VXA, seg_pos={0 0}, upload=551868]: Long tail cancellation due to optimal threshold Dec 31 21:15:46.944026911 (629h46m46.815109956s) [noise=true , address=45.91.116.6:4973 , piece_id=NSEDZOAZWJEHL2YJPRPT3Y37HLGYQSLCSIRP62IHYLFTDVFEIUUQ, seg_pos={0 0}, upload=551868]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF Dec 31 21:15:46.944037180 (629h46m46.815120215s) [noise=true , address=45.91.116.6:4973 , piece_id=NSEDZOAZWJEHL2YJPRPT3Y37HLGYQSLCSIRP62IHYLFTDVFEIUUQ, seg_pos={0 0}, upload=551868]: Long tail cancellation due to optimal threshold Dec 31 21:15:46.944059453 (629h46m46.815142498s) [noise=true , address=45.91.116.10:4083 , piece_id=EKQLXTCN4KSWG4GTBSB6FVVSBEP4L5ONBKXEMEQIOBYDYA5ZOUIA, seg_pos={0 0}, upload=551868]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 31 21:15:46.944068169 (629h46m46.815151204s) [noise=true , address=45.91.116.10:4083 , piece_id=EKQLXTCN4KSWG4GTBSB6FVVSBEP4L5ONBKXEMEQIOBYDYA5ZOUIA, seg_pos={0 0}, upload=551868]: Long tail cancellation due to optimal threshold Dec 31 21:15:46.944090411 (629h46m46.815173446s) [seg_pos={0 0}, upload=551868]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 21:15:46.944174672 (629h46m46.815257717s) [upload=551868]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 21:15:46.944181034 (629h46m46.815264069s) [upload=551868]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 21:15:47.234878081 (629h46m47.105961136s) [upload=551868]: Done uploading segments... Dec 31 21:15:47.234901195 (629h46m47.10598424s) [upload=551868]: Upload finished. err: <nil> Duration: 5.86235732s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [XX....X......X....XX........X...X.......X.....X...XX.X......] duration: 6.655395453s trace: https://jaeger.dev.storj.io/trace/5f2b7d22d21cf0f8 FAIL (2025-12-31 21:19:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: U4ZYIZQIZLHUQDHRK43NC2XEXOR3LDSS32WLEHYA5QGNIW4R5ZPQ) context deadline exceeded; piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: UEJYYPH4WYBOEV2ZMZDDHNGIHT2JH57AIQY3XZ2HLBV6F5QOYCTQ) context deadline exceeded; piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: CC55YZD4CWFZBRQASU6VB33RMJZ6RYGEZ6Y7SNLYILCRVRDBMMUQ) context deadline exceeded ======LOGS====== Dec 31 21:15:44.386938901 (629h46m44.258021936s) [upload=551868]: Starting upload Dec 31 21:15:44.386994888 (629h46m44.258077923s) [upload=551868]: Done starting upload Dec 31 21:15:44.387183197 (629h46m44.258266242s) [upload=551868]: Uploading segments... Dec 31 21:15:44.387281093 (629h46m44.258364158s) [upload=551868]: Got next segment. Inline: false Dec 31 21:15:44.387329566 (629h46m44.258412621s) [upload=551868]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 21:15:44.387337030 (629h46m44.258420075s) [upload=551868]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 21:15:44.637074656 (629h46m44.508157691s) [seg_pos={0 0}, upload=551868]: Begin upload segment... Dec 31 21:15:44.637126884 (629h46m44.50820992s) [seg_pos={0 0}, upload=551868]: Done begin upload segment. Dec 31 21:15:44.637137575 (629h46m44.50822061s) [noise=true , address=45.91.116.13:4156 , piece_id=CVX3XD5GRJVGPMTIO3GQGMPL3V76CREYNE4VOS22CNW3KNLH5VXA, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.637158976 (629h46m44.508242021s) [noise=true , address=45.91.116.12:4225 , piece_id=QF3QQSR5EF5T42IFPEWOCIAQBPAQPNVGBY5LLRO2IUJ5VHC4QNZA, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.637344248 (629h46m44.508427283s) [noise=true , address=145.79.227.48:4303 , piece_id=5CQNY7DPO4HVD4JXS4WBJEN3I7ZUPUNSY45HCEC25VZSW7R4Y5BQ, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.637642296 (629h46m44.508725331s) [noise=true , address=145.79.227.10:4403 , piece_id=CHUKGI35OWLDMZG2GGZHMC7RL6RDGNGIXT3KZZHBWKRXPLSJ3PMA, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.637935754 (629h46m44.5090188s) [noise=true , address=145.79.227.52:4446 , piece_id=7QK754CTPSPTRHM6LCLEBKRKSYLHJMAM5FZOJNDJKXZFJITSHWRQ, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.638274770 (629h46m44.509357815s) [noise=true , address=145.79.227.13:4803 , piece_id=TZJI24W2EQRLGP5NOL3KVN7R5HDJMEEWPH5WCXZG5WTDMA4FUDXQ, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.638548111 (629h46m44.509631146s) [noise=true , address=145.79.227.11:4730 , piece_id=BQLBPYXQPS7NTNKWT3NE4Z4VZPCKRAMBFBEAY3BVF7Q34FHZD6AQ, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.638806723 (629h46m44.509889768s) [noise=true , address=145.79.227.15:4875 , piece_id=KQMUIZMYY5TKHKLOVE2HYQEU5WYCZOOB5AJVROAE5WVSGZJDXMOQ, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.639124738 (629h46m44.510207783s) [noise=true , address=145.79.227.51:4410 , piece_id=QH57IGTPJZ57O6EHENLY2JU7VKVOZSYVGBVODRC2I5LL2AWZHV6Q, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.639522837 (629h46m44.510605882s) [noise=true , address=77.90.31.13:4627 , piece_id=CC55YZD4CWFZBRQASU6VB33RMJZ6RYGEZ6Y7SNLYILCRVRDBMMUQ, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.641025558 (629h46m44.512108593s) [noise=true , address=45.91.116.11:4118 , piece_id=U4ZYIZQIZLHUQDHRK43NC2XEXOR3LDSS32WLEHYA5QGNIW4R5ZPQ, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.642391289 (629h46m44.513474334s) [noise=true , address=145.79.227.46:4603 , piece_id=NNYCFUISSSOYPBTF2YGIMM7BNAW3EIYZTFKBGMQVEA5MNEHU6GQA, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.643600902 (629h46m44.514683937s) [noise=true , address=45.91.116.10:4083 , piece_id=EKQLXTCN4KSWG4GTBSB6FVVSBEP4L5ONBKXEMEQIOBYDYA5ZOUIA, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.644749149 (629h46m44.515832184s) [noise=true , address=145.79.227.31:4799 , piece_id=54IF73MWOTIOJ4WXV3WGUJMK7JMLGIVZIEOSI3PTYHZE5M263JFQ, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.646078710 (629h46m44.517161745s) [noise=true , address=77.90.31.21:4880 , piece_id=UEJYYPH4WYBOEV2ZMZDDHNGIHT2JH57AIQY3XZ2HLBV6F5QOYCTQ, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.647522739 (629h46m44.518605784s) [noise=true , address=45.91.116.6:4973 , piece_id=NSEDZOAZWJEHL2YJPRPT3Y37HLGYQSLCSIRP62IHYLFTDVFEIUUQ, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:46.344821972 (629h46m46.215905007s) [upload=551868]: Next returned nil segment Dec 31 21:15:46.344867048 (629h46m46.215950093s) [upload=551868]: Waiting for error group managing segments... Dec 31 21:15:46.624836599 (629h46m46.495919634s) [noise=true , address=145.79.227.48:4303 , piece_id=5CQNY7DPO4HVD4JXS4WBJEN3I7ZUPUNSY45HCEC25VZSW7R4Y5BQ, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.624940787 (629h46m46.496023832s) [noise=true , address=145.79.227.15:4875 , piece_id=KQMUIZMYY5TKHKLOVE2HYQEU5WYCZOOB5AJVROAE5WVSGZJDXMOQ, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.636984079 (629h46m46.508067114s) [noise=true , address=145.79.227.51:4410 , piece_id=QH57IGTPJZ57O6EHENLY2JU7VKVOZSYVGBVODRC2I5LL2AWZHV6Q, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.641160311 (629h46m46.512243356s) [noise=true , address=145.79.227.13:4803 , piece_id=TZJI24W2EQRLGP5NOL3KVN7R5HDJMEEWPH5WCXZG5WTDMA4FUDXQ, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.643189464 (629h46m46.514272509s) [noise=true , address=145.79.227.52:4446 , piece_id=7QK754CTPSPTRHM6LCLEBKRKSYLHJMAM5FZOJNDJKXZFJITSHWRQ, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.673375523 (629h46m46.544458568s) [noise=true , address=145.79.227.31:4799 , piece_id=54IF73MWOTIOJ4WXV3WGUJMK7JMLGIVZIEOSI3PTYHZE5M263JFQ, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.677399746 (629h46m46.548482801s) [noise=true , address=145.79.227.11:4730 , piece_id=BQLBPYXQPS7NTNKWT3NE4Z4VZPCKRAMBFBEAY3BVF7Q34FHZD6AQ, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.710091896 (629h46m46.581174931s) [noise=true , address=145.79.227.10:4403 , piece_id=CHUKGI35OWLDMZG2GGZHMC7RL6RDGNGIXT3KZZHBWKRXPLSJ3PMA, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.716974232 (629h46m46.588057267s) [noise=true , address=145.79.227.46:4603 , piece_id=NNYCFUISSSOYPBTF2YGIMM7BNAW3EIYZTFKBGMQVEA5MNEHU6GQA, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.799312184 (629h46m46.670395219s) [noise=true , address=77.90.31.13:4627 , piece_id=CC55YZD4CWFZBRQASU6VB33RMJZ6RYGEZ6Y7SNLYILCRVRDBMMUQ, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.929310008 (629h46m46.800393053s) [noise=true , address=45.91.116.12:4225 , piece_id=QF3QQSR5EF5T42IFPEWOCIAQBPAQPNVGBY5LLRO2IUJ5VHC4QNZA, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.930088710 (629h46m46.801171755s) [noise=true , address=77.90.31.21:4880 , piece_id=UEJYYPH4WYBOEV2ZMZDDHNGIHT2JH57AIQY3XZ2HLBV6F5QOYCTQ, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.943538120 (629h46m46.814621155s) [noise=true , address=45.91.116.11:4118 , piece_id=U4ZYIZQIZLHUQDHRK43NC2XEXOR3LDSS32WLEHYA5QGNIW4R5ZPQ, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.943559901 (629h46m46.814642946s) [seg_pos={0 0}, upload=551868]: Segment reached optimal threshold of 13 pieces. Dec 31 21:15:46.943767576 (629h46m46.814850621s) [noise=true , address=45.91.116.13:4156 , piece_id=CVX3XD5GRJVGPMTIO3GQGMPL3V76CREYNE4VOS22CNW3KNLH5VXA, seg_pos={0 0}, upload=551868]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 31 21:15:46.943777847 (629h46m46.814860892s) [noise=true , address=45.91.116.13:4156 , piece_id=CVX3XD5GRJVGPMTIO3GQGMPL3V76CREYNE4VOS22CNW3KNLH5VXA, seg_pos={0 0}, upload=551868]: Long tail cancellation due to optimal threshold Dec 31 21:15:46.944026911 (629h46m46.815109956s) [noise=true , address=45.91.116.6:4973 , piece_id=NSEDZOAZWJEHL2YJPRPT3Y37HLGYQSLCSIRP62IHYLFTDVFEIUUQ, seg_pos={0 0}, upload=551868]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF Dec 31 21:15:46.944037180 (629h46m46.815120215s) [noise=true , address=45.91.116.6:4973 , piece_id=NSEDZOAZWJEHL2YJPRPT3Y37HLGYQSLCSIRP62IHYLFTDVFEIUUQ, seg_pos={0 0}, upload=551868]: Long tail cancellation due to optimal threshold Dec 31 21:15:46.944059453 (629h46m46.815142498s) [noise=true , address=45.91.116.10:4083 , piece_id=EKQLXTCN4KSWG4GTBSB6FVVSBEP4L5ONBKXEMEQIOBYDYA5ZOUIA, seg_pos={0 0}, upload=551868]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 31 21:15:46.944068169 (629h46m46.815151204s) [noise=true , address=45.91.116.10:4083 , piece_id=EKQLXTCN4KSWG4GTBSB6FVVSBEP4L5ONBKXEMEQIOBYDYA5ZOUIA, seg_pos={0 0}, upload=551868]: Long tail cancellation due to optimal threshold Dec 31 21:15:46.944090411 (629h46m46.815173446s) [seg_pos={0 0}, upload=551868]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 21:15:46.944174672 (629h46m46.815257717s) [upload=551868]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 21:15:46.944181034 (629h46m46.815264069s) [upload=551868]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 21:15:47.234878081 (629h46m47.105961136s) [upload=551868]: Done uploading segments... Dec 31 21:15:47.234901195 (629h46m47.10598424s) [upload=551868]: Upload finished. err: <nil> Duration: 6.655411142s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.XX....X......X....XX........X...X.......X.....X...XX.X.....] duration: 5.795939503s trace: https://jaeger.dev.storj.io/trace/5f2b7d22d21cf0f8 FAIL (2025-12-31 21:18:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: U4ZYIZQIZLHUQDHRK43NC2XEXOR3LDSS32WLEHYA5QGNIW4R5ZPQ) context deadline exceeded; piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: UEJYYPH4WYBOEV2ZMZDDHNGIHT2JH57AIQY3XZ2HLBV6F5QOYCTQ) context deadline exceeded; piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: CC55YZD4CWFZBRQASU6VB33RMJZ6RYGEZ6Y7SNLYILCRVRDBMMUQ) context deadline exceeded ======LOGS====== Dec 31 21:15:44.386938901 (629h46m44.258021936s) [upload=551868]: Starting upload Dec 31 21:15:44.386994888 (629h46m44.258077923s) [upload=551868]: Done starting upload Dec 31 21:15:44.387183197 (629h46m44.258266242s) [upload=551868]: Uploading segments... Dec 31 21:15:44.387281093 (629h46m44.258364158s) [upload=551868]: Got next segment. Inline: false Dec 31 21:15:44.387329566 (629h46m44.258412621s) [upload=551868]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 21:15:44.387337030 (629h46m44.258420075s) [upload=551868]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 21:15:44.637074656 (629h46m44.508157691s) [seg_pos={0 0}, upload=551868]: Begin upload segment... Dec 31 21:15:44.637126884 (629h46m44.50820992s) [seg_pos={0 0}, upload=551868]: Done begin upload segment. Dec 31 21:15:44.637137575 (629h46m44.50822061s) [noise=true , address=45.91.116.13:4156 , piece_id=CVX3XD5GRJVGPMTIO3GQGMPL3V76CREYNE4VOS22CNW3KNLH5VXA, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.637158976 (629h46m44.508242021s) [noise=true , address=45.91.116.12:4225 , piece_id=QF3QQSR5EF5T42IFPEWOCIAQBPAQPNVGBY5LLRO2IUJ5VHC4QNZA, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.637344248 (629h46m44.508427283s) [noise=true , address=145.79.227.48:4303 , piece_id=5CQNY7DPO4HVD4JXS4WBJEN3I7ZUPUNSY45HCEC25VZSW7R4Y5BQ, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.637642296 (629h46m44.508725331s) [noise=true , address=145.79.227.10:4403 , piece_id=CHUKGI35OWLDMZG2GGZHMC7RL6RDGNGIXT3KZZHBWKRXPLSJ3PMA, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.637935754 (629h46m44.5090188s) [noise=true , address=145.79.227.52:4446 , piece_id=7QK754CTPSPTRHM6LCLEBKRKSYLHJMAM5FZOJNDJKXZFJITSHWRQ, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.638274770 (629h46m44.509357815s) [noise=true , address=145.79.227.13:4803 , piece_id=TZJI24W2EQRLGP5NOL3KVN7R5HDJMEEWPH5WCXZG5WTDMA4FUDXQ, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.638548111 (629h46m44.509631146s) [noise=true , address=145.79.227.11:4730 , piece_id=BQLBPYXQPS7NTNKWT3NE4Z4VZPCKRAMBFBEAY3BVF7Q34FHZD6AQ, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.638806723 (629h46m44.509889768s) [noise=true , address=145.79.227.15:4875 , piece_id=KQMUIZMYY5TKHKLOVE2HYQEU5WYCZOOB5AJVROAE5WVSGZJDXMOQ, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.639124738 (629h46m44.510207783s) [noise=true , address=145.79.227.51:4410 , piece_id=QH57IGTPJZ57O6EHENLY2JU7VKVOZSYVGBVODRC2I5LL2AWZHV6Q, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.639522837 (629h46m44.510605882s) [noise=true , address=77.90.31.13:4627 , piece_id=CC55YZD4CWFZBRQASU6VB33RMJZ6RYGEZ6Y7SNLYILCRVRDBMMUQ, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.641025558 (629h46m44.512108593s) [noise=true , address=45.91.116.11:4118 , piece_id=U4ZYIZQIZLHUQDHRK43NC2XEXOR3LDSS32WLEHYA5QGNIW4R5ZPQ, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.642391289 (629h46m44.513474334s) [noise=true , address=145.79.227.46:4603 , piece_id=NNYCFUISSSOYPBTF2YGIMM7BNAW3EIYZTFKBGMQVEA5MNEHU6GQA, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.643600902 (629h46m44.514683937s) [noise=true , address=45.91.116.10:4083 , piece_id=EKQLXTCN4KSWG4GTBSB6FVVSBEP4L5ONBKXEMEQIOBYDYA5ZOUIA, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.644749149 (629h46m44.515832184s) [noise=true , address=145.79.227.31:4799 , piece_id=54IF73MWOTIOJ4WXV3WGUJMK7JMLGIVZIEOSI3PTYHZE5M263JFQ, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.646078710 (629h46m44.517161745s) [noise=true , address=77.90.31.21:4880 , piece_id=UEJYYPH4WYBOEV2ZMZDDHNGIHT2JH57AIQY3XZ2HLBV6F5QOYCTQ, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.647522739 (629h46m44.518605784s) [noise=true , address=45.91.116.6:4973 , piece_id=NSEDZOAZWJEHL2YJPRPT3Y37HLGYQSLCSIRP62IHYLFTDVFEIUUQ, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:46.344821972 (629h46m46.215905007s) [upload=551868]: Next returned nil segment Dec 31 21:15:46.344867048 (629h46m46.215950093s) [upload=551868]: Waiting for error group managing segments... Dec 31 21:15:46.624836599 (629h46m46.495919634s) [noise=true , address=145.79.227.48:4303 , piece_id=5CQNY7DPO4HVD4JXS4WBJEN3I7ZUPUNSY45HCEC25VZSW7R4Y5BQ, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.624940787 (629h46m46.496023832s) [noise=true , address=145.79.227.15:4875 , piece_id=KQMUIZMYY5TKHKLOVE2HYQEU5WYCZOOB5AJVROAE5WVSGZJDXMOQ, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.636984079 (629h46m46.508067114s) [noise=true , address=145.79.227.51:4410 , piece_id=QH57IGTPJZ57O6EHENLY2JU7VKVOZSYVGBVODRC2I5LL2AWZHV6Q, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.641160311 (629h46m46.512243356s) [noise=true , address=145.79.227.13:4803 , piece_id=TZJI24W2EQRLGP5NOL3KVN7R5HDJMEEWPH5WCXZG5WTDMA4FUDXQ, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.643189464 (629h46m46.514272509s) [noise=true , address=145.79.227.52:4446 , piece_id=7QK754CTPSPTRHM6LCLEBKRKSYLHJMAM5FZOJNDJKXZFJITSHWRQ, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.673375523 (629h46m46.544458568s) [noise=true , address=145.79.227.31:4799 , piece_id=54IF73MWOTIOJ4WXV3WGUJMK7JMLGIVZIEOSI3PTYHZE5M263JFQ, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.677399746 (629h46m46.548482801s) [noise=true , address=145.79.227.11:4730 , piece_id=BQLBPYXQPS7NTNKWT3NE4Z4VZPCKRAMBFBEAY3BVF7Q34FHZD6AQ, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.710091896 (629h46m46.581174931s) [noise=true , address=145.79.227.10:4403 , piece_id=CHUKGI35OWLDMZG2GGZHMC7RL6RDGNGIXT3KZZHBWKRXPLSJ3PMA, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.716974232 (629h46m46.588057267s) [noise=true , address=145.79.227.46:4603 , piece_id=NNYCFUISSSOYPBTF2YGIMM7BNAW3EIYZTFKBGMQVEA5MNEHU6GQA, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.799312184 (629h46m46.670395219s) [noise=true , address=77.90.31.13:4627 , piece_id=CC55YZD4CWFZBRQASU6VB33RMJZ6RYGEZ6Y7SNLYILCRVRDBMMUQ, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.929310008 (629h46m46.800393053s) [noise=true , address=45.91.116.12:4225 , piece_id=QF3QQSR5EF5T42IFPEWOCIAQBPAQPNVGBY5LLRO2IUJ5VHC4QNZA, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.930088710 (629h46m46.801171755s) [noise=true , address=77.90.31.21:4880 , piece_id=UEJYYPH4WYBOEV2ZMZDDHNGIHT2JH57AIQY3XZ2HLBV6F5QOYCTQ, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.943538120 (629h46m46.814621155s) [noise=true , address=45.91.116.11:4118 , piece_id=U4ZYIZQIZLHUQDHRK43NC2XEXOR3LDSS32WLEHYA5QGNIW4R5ZPQ, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.943559901 (629h46m46.814642946s) [seg_pos={0 0}, upload=551868]: Segment reached optimal threshold of 13 pieces. Dec 31 21:15:46.943767576 (629h46m46.814850621s) [noise=true , address=45.91.116.13:4156 , piece_id=CVX3XD5GRJVGPMTIO3GQGMPL3V76CREYNE4VOS22CNW3KNLH5VXA, seg_pos={0 0}, upload=551868]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 31 21:15:46.943777847 (629h46m46.814860892s) [noise=true , address=45.91.116.13:4156 , piece_id=CVX3XD5GRJVGPMTIO3GQGMPL3V76CREYNE4VOS22CNW3KNLH5VXA, seg_pos={0 0}, upload=551868]: Long tail cancellation due to optimal threshold Dec 31 21:15:46.944026911 (629h46m46.815109956s) [noise=true , address=45.91.116.6:4973 , piece_id=NSEDZOAZWJEHL2YJPRPT3Y37HLGYQSLCSIRP62IHYLFTDVFEIUUQ, seg_pos={0 0}, upload=551868]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF Dec 31 21:15:46.944037180 (629h46m46.815120215s) [noise=true , address=45.91.116.6:4973 , piece_id=NSEDZOAZWJEHL2YJPRPT3Y37HLGYQSLCSIRP62IHYLFTDVFEIUUQ, seg_pos={0 0}, upload=551868]: Long tail cancellation due to optimal threshold Dec 31 21:15:46.944059453 (629h46m46.815142498s) [noise=true , address=45.91.116.10:4083 , piece_id=EKQLXTCN4KSWG4GTBSB6FVVSBEP4L5ONBKXEMEQIOBYDYA5ZOUIA, seg_pos={0 0}, upload=551868]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 31 21:15:46.944068169 (629h46m46.815151204s) [noise=true , address=45.91.116.10:4083 , piece_id=EKQLXTCN4KSWG4GTBSB6FVVSBEP4L5ONBKXEMEQIOBYDYA5ZOUIA, seg_pos={0 0}, upload=551868]: Long tail cancellation due to optimal threshold Dec 31 21:15:46.944090411 (629h46m46.815173446s) [seg_pos={0 0}, upload=551868]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 21:15:46.944174672 (629h46m46.815257717s) [upload=551868]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 21:15:46.944181034 (629h46m46.815264069s) [upload=551868]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 21:15:47.234878081 (629h46m47.105961136s) [upload=551868]: Done uploading segments... Dec 31 21:15:47.234901195 (629h46m47.10598424s) [upload=551868]: Upload finished. err: <nil> Duration: 5.795967076s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [..XX....X......X....XX........X...X.......X.....X...XX.X....] duration: 23.312257685s trace: https://jaeger.dev.storj.io/trace/5f2b7d22d21cf0f8 FAIL (2025-12-31 21:18:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: U4ZYIZQIZLHUQDHRK43NC2XEXOR3LDSS32WLEHYA5QGNIW4R5ZPQ) context deadline exceeded; piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: UEJYYPH4WYBOEV2ZMZDDHNGIHT2JH57AIQY3XZ2HLBV6F5QOYCTQ) context deadline exceeded; piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: CC55YZD4CWFZBRQASU6VB33RMJZ6RYGEZ6Y7SNLYILCRVRDBMMUQ) context deadline exceeded ======LOGS====== Dec 31 21:15:44.386938901 (629h46m44.258021936s) [upload=551868]: Starting upload Dec 31 21:15:44.386994888 (629h46m44.258077923s) [upload=551868]: Done starting upload Dec 31 21:15:44.387183197 (629h46m44.258266242s) [upload=551868]: Uploading segments... Dec 31 21:15:44.387281093 (629h46m44.258364158s) [upload=551868]: Got next segment. Inline: false Dec 31 21:15:44.387329566 (629h46m44.258412621s) [upload=551868]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 21:15:44.387337030 (629h46m44.258420075s) [upload=551868]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 21:15:44.637074656 (629h46m44.508157691s) [seg_pos={0 0}, upload=551868]: Begin upload segment... Dec 31 21:15:44.637126884 (629h46m44.50820992s) [seg_pos={0 0}, upload=551868]: Done begin upload segment. Dec 31 21:15:44.637137575 (629h46m44.50822061s) [noise=true , address=45.91.116.13:4156 , piece_id=CVX3XD5GRJVGPMTIO3GQGMPL3V76CREYNE4VOS22CNW3KNLH5VXA, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.637158976 (629h46m44.508242021s) [noise=true , address=45.91.116.12:4225 , piece_id=QF3QQSR5EF5T42IFPEWOCIAQBPAQPNVGBY5LLRO2IUJ5VHC4QNZA, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.637344248 (629h46m44.508427283s) [noise=true , address=145.79.227.48:4303 , piece_id=5CQNY7DPO4HVD4JXS4WBJEN3I7ZUPUNSY45HCEC25VZSW7R4Y5BQ, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.637642296 (629h46m44.508725331s) [noise=true , address=145.79.227.10:4403 , piece_id=CHUKGI35OWLDMZG2GGZHMC7RL6RDGNGIXT3KZZHBWKRXPLSJ3PMA, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.637935754 (629h46m44.5090188s) [noise=true , address=145.79.227.52:4446 , piece_id=7QK754CTPSPTRHM6LCLEBKRKSYLHJMAM5FZOJNDJKXZFJITSHWRQ, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.638274770 (629h46m44.509357815s) [noise=true , address=145.79.227.13:4803 , piece_id=TZJI24W2EQRLGP5NOL3KVN7R5HDJMEEWPH5WCXZG5WTDMA4FUDXQ, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.638548111 (629h46m44.509631146s) [noise=true , address=145.79.227.11:4730 , piece_id=BQLBPYXQPS7NTNKWT3NE4Z4VZPCKRAMBFBEAY3BVF7Q34FHZD6AQ, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.638806723 (629h46m44.509889768s) [noise=true , address=145.79.227.15:4875 , piece_id=KQMUIZMYY5TKHKLOVE2HYQEU5WYCZOOB5AJVROAE5WVSGZJDXMOQ, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.639124738 (629h46m44.510207783s) [noise=true , address=145.79.227.51:4410 , piece_id=QH57IGTPJZ57O6EHENLY2JU7VKVOZSYVGBVODRC2I5LL2AWZHV6Q, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.639522837 (629h46m44.510605882s) [noise=true , address=77.90.31.13:4627 , piece_id=CC55YZD4CWFZBRQASU6VB33RMJZ6RYGEZ6Y7SNLYILCRVRDBMMUQ, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.641025558 (629h46m44.512108593s) [noise=true , address=45.91.116.11:4118 , piece_id=U4ZYIZQIZLHUQDHRK43NC2XEXOR3LDSS32WLEHYA5QGNIW4R5ZPQ, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.642391289 (629h46m44.513474334s) [noise=true , address=145.79.227.46:4603 , piece_id=NNYCFUISSSOYPBTF2YGIMM7BNAW3EIYZTFKBGMQVEA5MNEHU6GQA, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.643600902 (629h46m44.514683937s) [noise=true , address=45.91.116.10:4083 , piece_id=EKQLXTCN4KSWG4GTBSB6FVVSBEP4L5ONBKXEMEQIOBYDYA5ZOUIA, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.644749149 (629h46m44.515832184s) [noise=true , address=145.79.227.31:4799 , piece_id=54IF73MWOTIOJ4WXV3WGUJMK7JMLGIVZIEOSI3PTYHZE5M263JFQ, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.646078710 (629h46m44.517161745s) [noise=true , address=77.90.31.21:4880 , piece_id=UEJYYPH4WYBOEV2ZMZDDHNGIHT2JH57AIQY3XZ2HLBV6F5QOYCTQ, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.647522739 (629h46m44.518605784s) [noise=true , address=45.91.116.6:4973 , piece_id=NSEDZOAZWJEHL2YJPRPT3Y37HLGYQSLCSIRP62IHYLFTDVFEIUUQ, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:46.344821972 (629h46m46.215905007s) [upload=551868]: Next returned nil segment Dec 31 21:15:46.344867048 (629h46m46.215950093s) [upload=551868]: Waiting for error group managing segments... Dec 31 21:15:46.624836599 (629h46m46.495919634s) [noise=true , address=145.79.227.48:4303 , piece_id=5CQNY7DPO4HVD4JXS4WBJEN3I7ZUPUNSY45HCEC25VZSW7R4Y5BQ, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.624940787 (629h46m46.496023832s) [noise=true , address=145.79.227.15:4875 , piece_id=KQMUIZMYY5TKHKLOVE2HYQEU5WYCZOOB5AJVROAE5WVSGZJDXMOQ, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.636984079 (629h46m46.508067114s) [noise=true , address=145.79.227.51:4410 , piece_id=QH57IGTPJZ57O6EHENLY2JU7VKVOZSYVGBVODRC2I5LL2AWZHV6Q, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.641160311 (629h46m46.512243356s) [noise=true , address=145.79.227.13:4803 , piece_id=TZJI24W2EQRLGP5NOL3KVN7R5HDJMEEWPH5WCXZG5WTDMA4FUDXQ, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.643189464 (629h46m46.514272509s) [noise=true , address=145.79.227.52:4446 , piece_id=7QK754CTPSPTRHM6LCLEBKRKSYLHJMAM5FZOJNDJKXZFJITSHWRQ, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.673375523 (629h46m46.544458568s) [noise=true , address=145.79.227.31:4799 , piece_id=54IF73MWOTIOJ4WXV3WGUJMK7JMLGIVZIEOSI3PTYHZE5M263JFQ, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.677399746 (629h46m46.548482801s) [noise=true , address=145.79.227.11:4730 , piece_id=BQLBPYXQPS7NTNKWT3NE4Z4VZPCKRAMBFBEAY3BVF7Q34FHZD6AQ, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.710091896 (629h46m46.581174931s) [noise=true , address=145.79.227.10:4403 , piece_id=CHUKGI35OWLDMZG2GGZHMC7RL6RDGNGIXT3KZZHBWKRXPLSJ3PMA, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.716974232 (629h46m46.588057267s) [noise=true , address=145.79.227.46:4603 , piece_id=NNYCFUISSSOYPBTF2YGIMM7BNAW3EIYZTFKBGMQVEA5MNEHU6GQA, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.799312184 (629h46m46.670395219s) [noise=true , address=77.90.31.13:4627 , piece_id=CC55YZD4CWFZBRQASU6VB33RMJZ6RYGEZ6Y7SNLYILCRVRDBMMUQ, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.929310008 (629h46m46.800393053s) [noise=true , address=45.91.116.12:4225 , piece_id=QF3QQSR5EF5T42IFPEWOCIAQBPAQPNVGBY5LLRO2IUJ5VHC4QNZA, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.930088710 (629h46m46.801171755s) [noise=true , address=77.90.31.21:4880 , piece_id=UEJYYPH4WYBOEV2ZMZDDHNGIHT2JH57AIQY3XZ2HLBV6F5QOYCTQ, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.943538120 (629h46m46.814621155s) [noise=true , address=45.91.116.11:4118 , piece_id=U4ZYIZQIZLHUQDHRK43NC2XEXOR3LDSS32WLEHYA5QGNIW4R5ZPQ, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.943559901 (629h46m46.814642946s) [seg_pos={0 0}, upload=551868]: Segment reached optimal threshold of 13 pieces. Dec 31 21:15:46.943767576 (629h46m46.814850621s) [noise=true , address=45.91.116.13:4156 , piece_id=CVX3XD5GRJVGPMTIO3GQGMPL3V76CREYNE4VOS22CNW3KNLH5VXA, seg_pos={0 0}, upload=551868]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 31 21:15:46.943777847 (629h46m46.814860892s) [noise=true , address=45.91.116.13:4156 , piece_id=CVX3XD5GRJVGPMTIO3GQGMPL3V76CREYNE4VOS22CNW3KNLH5VXA, seg_pos={0 0}, upload=551868]: Long tail cancellation due to optimal threshold Dec 31 21:15:46.944026911 (629h46m46.815109956s) [noise=true , address=45.91.116.6:4973 , piece_id=NSEDZOAZWJEHL2YJPRPT3Y37HLGYQSLCSIRP62IHYLFTDVFEIUUQ, seg_pos={0 0}, upload=551868]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF Dec 31 21:15:46.944037180 (629h46m46.815120215s) [noise=true , address=45.91.116.6:4973 , piece_id=NSEDZOAZWJEHL2YJPRPT3Y37HLGYQSLCSIRP62IHYLFTDVFEIUUQ, seg_pos={0 0}, upload=551868]: Long tail cancellation due to optimal threshold Dec 31 21:15:46.944059453 (629h46m46.815142498s) [noise=true , address=45.91.116.10:4083 , piece_id=EKQLXTCN4KSWG4GTBSB6FVVSBEP4L5ONBKXEMEQIOBYDYA5ZOUIA, seg_pos={0 0}, upload=551868]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 31 21:15:46.944068169 (629h46m46.815151204s) [noise=true , address=45.91.116.10:4083 , piece_id=EKQLXTCN4KSWG4GTBSB6FVVSBEP4L5ONBKXEMEQIOBYDYA5ZOUIA, seg_pos={0 0}, upload=551868]: Long tail cancellation due to optimal threshold Dec 31 21:15:46.944090411 (629h46m46.815173446s) [seg_pos={0 0}, upload=551868]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 21:15:46.944174672 (629h46m46.815257717s) [upload=551868]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 21:15:46.944181034 (629h46m46.815264069s) [upload=551868]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 21:15:47.234878081 (629h46m47.105961136s) [upload=551868]: Done uploading segments... Dec 31 21:15:47.234901195 (629h46m47.10598424s) [upload=551868]: Upload finished. err: <nil> Duration: 23.312279708s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [...XX....X......X....XX........X...X.......X.....X...XX.X...] duration: 5.825197482s trace: https://jaeger.dev.storj.io/trace/5f2b7d22d21cf0f8 FAIL (2025-12-31 21:17:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: U4ZYIZQIZLHUQDHRK43NC2XEXOR3LDSS32WLEHYA5QGNIW4R5ZPQ) context deadline exceeded; piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: UEJYYPH4WYBOEV2ZMZDDHNGIHT2JH57AIQY3XZ2HLBV6F5QOYCTQ) context deadline exceeded; piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: CC55YZD4CWFZBRQASU6VB33RMJZ6RYGEZ6Y7SNLYILCRVRDBMMUQ) context deadline exceeded ======LOGS====== Dec 31 21:15:44.386938901 (629h46m44.258021936s) [upload=551868]: Starting upload Dec 31 21:15:44.386994888 (629h46m44.258077923s) [upload=551868]: Done starting upload Dec 31 21:15:44.387183197 (629h46m44.258266242s) [upload=551868]: Uploading segments... Dec 31 21:15:44.387281093 (629h46m44.258364158s) [upload=551868]: Got next segment. Inline: false Dec 31 21:15:44.387329566 (629h46m44.258412621s) [upload=551868]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 21:15:44.387337030 (629h46m44.258420075s) [upload=551868]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 21:15:44.637074656 (629h46m44.508157691s) [seg_pos={0 0}, upload=551868]: Begin upload segment... Dec 31 21:15:44.637126884 (629h46m44.50820992s) [seg_pos={0 0}, upload=551868]: Done begin upload segment. Dec 31 21:15:44.637137575 (629h46m44.50822061s) [noise=true , address=45.91.116.13:4156 , piece_id=CVX3XD5GRJVGPMTIO3GQGMPL3V76CREYNE4VOS22CNW3KNLH5VXA, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.637158976 (629h46m44.508242021s) [noise=true , address=45.91.116.12:4225 , piece_id=QF3QQSR5EF5T42IFPEWOCIAQBPAQPNVGBY5LLRO2IUJ5VHC4QNZA, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.637344248 (629h46m44.508427283s) [noise=true , address=145.79.227.48:4303 , piece_id=5CQNY7DPO4HVD4JXS4WBJEN3I7ZUPUNSY45HCEC25VZSW7R4Y5BQ, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.637642296 (629h46m44.508725331s) [noise=true , address=145.79.227.10:4403 , piece_id=CHUKGI35OWLDMZG2GGZHMC7RL6RDGNGIXT3KZZHBWKRXPLSJ3PMA, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.637935754 (629h46m44.5090188s) [noise=true , address=145.79.227.52:4446 , piece_id=7QK754CTPSPTRHM6LCLEBKRKSYLHJMAM5FZOJNDJKXZFJITSHWRQ, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.638274770 (629h46m44.509357815s) [noise=true , address=145.79.227.13:4803 , piece_id=TZJI24W2EQRLGP5NOL3KVN7R5HDJMEEWPH5WCXZG5WTDMA4FUDXQ, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.638548111 (629h46m44.509631146s) [noise=true , address=145.79.227.11:4730 , piece_id=BQLBPYXQPS7NTNKWT3NE4Z4VZPCKRAMBFBEAY3BVF7Q34FHZD6AQ, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.638806723 (629h46m44.509889768s) [noise=true , address=145.79.227.15:4875 , piece_id=KQMUIZMYY5TKHKLOVE2HYQEU5WYCZOOB5AJVROAE5WVSGZJDXMOQ, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.639124738 (629h46m44.510207783s) [noise=true , address=145.79.227.51:4410 , piece_id=QH57IGTPJZ57O6EHENLY2JU7VKVOZSYVGBVODRC2I5LL2AWZHV6Q, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.639522837 (629h46m44.510605882s) [noise=true , address=77.90.31.13:4627 , piece_id=CC55YZD4CWFZBRQASU6VB33RMJZ6RYGEZ6Y7SNLYILCRVRDBMMUQ, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.641025558 (629h46m44.512108593s) [noise=true , address=45.91.116.11:4118 , piece_id=U4ZYIZQIZLHUQDHRK43NC2XEXOR3LDSS32WLEHYA5QGNIW4R5ZPQ, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.642391289 (629h46m44.513474334s) [noise=true , address=145.79.227.46:4603 , piece_id=NNYCFUISSSOYPBTF2YGIMM7BNAW3EIYZTFKBGMQVEA5MNEHU6GQA, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.643600902 (629h46m44.514683937s) [noise=true , address=45.91.116.10:4083 , piece_id=EKQLXTCN4KSWG4GTBSB6FVVSBEP4L5ONBKXEMEQIOBYDYA5ZOUIA, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.644749149 (629h46m44.515832184s) [noise=true , address=145.79.227.31:4799 , piece_id=54IF73MWOTIOJ4WXV3WGUJMK7JMLGIVZIEOSI3PTYHZE5M263JFQ, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.646078710 (629h46m44.517161745s) [noise=true , address=77.90.31.21:4880 , piece_id=UEJYYPH4WYBOEV2ZMZDDHNGIHT2JH57AIQY3XZ2HLBV6F5QOYCTQ, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.647522739 (629h46m44.518605784s) [noise=true , address=45.91.116.6:4973 , piece_id=NSEDZOAZWJEHL2YJPRPT3Y37HLGYQSLCSIRP62IHYLFTDVFEIUUQ, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:46.344821972 (629h46m46.215905007s) [upload=551868]: Next returned nil segment Dec 31 21:15:46.344867048 (629h46m46.215950093s) [upload=551868]: Waiting for error group managing segments... Dec 31 21:15:46.624836599 (629h46m46.495919634s) [noise=true , address=145.79.227.48:4303 , piece_id=5CQNY7DPO4HVD4JXS4WBJEN3I7ZUPUNSY45HCEC25VZSW7R4Y5BQ, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.624940787 (629h46m46.496023832s) [noise=true , address=145.79.227.15:4875 , piece_id=KQMUIZMYY5TKHKLOVE2HYQEU5WYCZOOB5AJVROAE5WVSGZJDXMOQ, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.636984079 (629h46m46.508067114s) [noise=true , address=145.79.227.51:4410 , piece_id=QH57IGTPJZ57O6EHENLY2JU7VKVOZSYVGBVODRC2I5LL2AWZHV6Q, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.641160311 (629h46m46.512243356s) [noise=true , address=145.79.227.13:4803 , piece_id=TZJI24W2EQRLGP5NOL3KVN7R5HDJMEEWPH5WCXZG5WTDMA4FUDXQ, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.643189464 (629h46m46.514272509s) [noise=true , address=145.79.227.52:4446 , piece_id=7QK754CTPSPTRHM6LCLEBKRKSYLHJMAM5FZOJNDJKXZFJITSHWRQ, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.673375523 (629h46m46.544458568s) [noise=true , address=145.79.227.31:4799 , piece_id=54IF73MWOTIOJ4WXV3WGUJMK7JMLGIVZIEOSI3PTYHZE5M263JFQ, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.677399746 (629h46m46.548482801s) [noise=true , address=145.79.227.11:4730 , piece_id=BQLBPYXQPS7NTNKWT3NE4Z4VZPCKRAMBFBEAY3BVF7Q34FHZD6AQ, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.710091896 (629h46m46.581174931s) [noise=true , address=145.79.227.10:4403 , piece_id=CHUKGI35OWLDMZG2GGZHMC7RL6RDGNGIXT3KZZHBWKRXPLSJ3PMA, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.716974232 (629h46m46.588057267s) [noise=true , address=145.79.227.46:4603 , piece_id=NNYCFUISSSOYPBTF2YGIMM7BNAW3EIYZTFKBGMQVEA5MNEHU6GQA, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.799312184 (629h46m46.670395219s) [noise=true , address=77.90.31.13:4627 , piece_id=CC55YZD4CWFZBRQASU6VB33RMJZ6RYGEZ6Y7SNLYILCRVRDBMMUQ, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.929310008 (629h46m46.800393053s) [noise=true , address=45.91.116.12:4225 , piece_id=QF3QQSR5EF5T42IFPEWOCIAQBPAQPNVGBY5LLRO2IUJ5VHC4QNZA, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.930088710 (629h46m46.801171755s) [noise=true , address=77.90.31.21:4880 , piece_id=UEJYYPH4WYBOEV2ZMZDDHNGIHT2JH57AIQY3XZ2HLBV6F5QOYCTQ, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.943538120 (629h46m46.814621155s) [noise=true , address=45.91.116.11:4118 , piece_id=U4ZYIZQIZLHUQDHRK43NC2XEXOR3LDSS32WLEHYA5QGNIW4R5ZPQ, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.943559901 (629h46m46.814642946s) [seg_pos={0 0}, upload=551868]: Segment reached optimal threshold of 13 pieces. Dec 31 21:15:46.943767576 (629h46m46.814850621s) [noise=true , address=45.91.116.13:4156 , piece_id=CVX3XD5GRJVGPMTIO3GQGMPL3V76CREYNE4VOS22CNW3KNLH5VXA, seg_pos={0 0}, upload=551868]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 31 21:15:46.943777847 (629h46m46.814860892s) [noise=true , address=45.91.116.13:4156 , piece_id=CVX3XD5GRJVGPMTIO3GQGMPL3V76CREYNE4VOS22CNW3KNLH5VXA, seg_pos={0 0}, upload=551868]: Long tail cancellation due to optimal threshold Dec 31 21:15:46.944026911 (629h46m46.815109956s) [noise=true , address=45.91.116.6:4973 , piece_id=NSEDZOAZWJEHL2YJPRPT3Y37HLGYQSLCSIRP62IHYLFTDVFEIUUQ, seg_pos={0 0}, upload=551868]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF Dec 31 21:15:46.944037180 (629h46m46.815120215s) [noise=true , address=45.91.116.6:4973 , piece_id=NSEDZOAZWJEHL2YJPRPT3Y37HLGYQSLCSIRP62IHYLFTDVFEIUUQ, seg_pos={0 0}, upload=551868]: Long tail cancellation due to optimal threshold Dec 31 21:15:46.944059453 (629h46m46.815142498s) [noise=true , address=45.91.116.10:4083 , piece_id=EKQLXTCN4KSWG4GTBSB6FVVSBEP4L5ONBKXEMEQIOBYDYA5ZOUIA, seg_pos={0 0}, upload=551868]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 31 21:15:46.944068169 (629h46m46.815151204s) [noise=true , address=45.91.116.10:4083 , piece_id=EKQLXTCN4KSWG4GTBSB6FVVSBEP4L5ONBKXEMEQIOBYDYA5ZOUIA, seg_pos={0 0}, upload=551868]: Long tail cancellation due to optimal threshold Dec 31 21:15:46.944090411 (629h46m46.815173446s) [seg_pos={0 0}, upload=551868]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 21:15:46.944174672 (629h46m46.815257717s) [upload=551868]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 21:15:46.944181034 (629h46m46.815264069s) [upload=551868]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 21:15:47.234878081 (629h46m47.105961136s) [upload=551868]: Done uploading segments... Dec 31 21:15:47.234901195 (629h46m47.10598424s) [upload=551868]: Upload finished. err: <nil> Duration: 5.825223531s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [X...XX....X......X....XX........X...X.......X.....X...XX.X..] duration: 8.993792224s trace: https://jaeger.dev.storj.io/trace/5f2b7d22d21cf0f8 FAIL (2025-12-31 21:16:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: U4ZYIZQIZLHUQDHRK43NC2XEXOR3LDSS32WLEHYA5QGNIW4R5ZPQ) context deadline exceeded; piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: UEJYYPH4WYBOEV2ZMZDDHNGIHT2JH57AIQY3XZ2HLBV6F5QOYCTQ) context deadline exceeded; piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: CC55YZD4CWFZBRQASU6VB33RMJZ6RYGEZ6Y7SNLYILCRVRDBMMUQ) context deadline exceeded ======LOGS====== Dec 31 21:15:44.386938901 (629h46m44.258021936s) [upload=551868]: Starting upload Dec 31 21:15:44.386994888 (629h46m44.258077923s) [upload=551868]: Done starting upload Dec 31 21:15:44.387183197 (629h46m44.258266242s) [upload=551868]: Uploading segments... Dec 31 21:15:44.387281093 (629h46m44.258364158s) [upload=551868]: Got next segment. Inline: false Dec 31 21:15:44.387329566 (629h46m44.258412621s) [upload=551868]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 21:15:44.387337030 (629h46m44.258420075s) [upload=551868]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 21:15:44.637074656 (629h46m44.508157691s) [seg_pos={0 0}, upload=551868]: Begin upload segment... Dec 31 21:15:44.637126884 (629h46m44.50820992s) [seg_pos={0 0}, upload=551868]: Done begin upload segment. Dec 31 21:15:44.637137575 (629h46m44.50822061s) [noise=true , address=45.91.116.13:4156 , piece_id=CVX3XD5GRJVGPMTIO3GQGMPL3V76CREYNE4VOS22CNW3KNLH5VXA, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.637158976 (629h46m44.508242021s) [noise=true , address=45.91.116.12:4225 , piece_id=QF3QQSR5EF5T42IFPEWOCIAQBPAQPNVGBY5LLRO2IUJ5VHC4QNZA, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.637344248 (629h46m44.508427283s) [noise=true , address=145.79.227.48:4303 , piece_id=5CQNY7DPO4HVD4JXS4WBJEN3I7ZUPUNSY45HCEC25VZSW7R4Y5BQ, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.637642296 (629h46m44.508725331s) [noise=true , address=145.79.227.10:4403 , piece_id=CHUKGI35OWLDMZG2GGZHMC7RL6RDGNGIXT3KZZHBWKRXPLSJ3PMA, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.637935754 (629h46m44.5090188s) [noise=true , address=145.79.227.52:4446 , piece_id=7QK754CTPSPTRHM6LCLEBKRKSYLHJMAM5FZOJNDJKXZFJITSHWRQ, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.638274770 (629h46m44.509357815s) [noise=true , address=145.79.227.13:4803 , piece_id=TZJI24W2EQRLGP5NOL3KVN7R5HDJMEEWPH5WCXZG5WTDMA4FUDXQ, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.638548111 (629h46m44.509631146s) [noise=true , address=145.79.227.11:4730 , piece_id=BQLBPYXQPS7NTNKWT3NE4Z4VZPCKRAMBFBEAY3BVF7Q34FHZD6AQ, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.638806723 (629h46m44.509889768s) [noise=true , address=145.79.227.15:4875 , piece_id=KQMUIZMYY5TKHKLOVE2HYQEU5WYCZOOB5AJVROAE5WVSGZJDXMOQ, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.639124738 (629h46m44.510207783s) [noise=true , address=145.79.227.51:4410 , piece_id=QH57IGTPJZ57O6EHENLY2JU7VKVOZSYVGBVODRC2I5LL2AWZHV6Q, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.639522837 (629h46m44.510605882s) [noise=true , address=77.90.31.13:4627 , piece_id=CC55YZD4CWFZBRQASU6VB33RMJZ6RYGEZ6Y7SNLYILCRVRDBMMUQ, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.641025558 (629h46m44.512108593s) [noise=true , address=45.91.116.11:4118 , piece_id=U4ZYIZQIZLHUQDHRK43NC2XEXOR3LDSS32WLEHYA5QGNIW4R5ZPQ, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.642391289 (629h46m44.513474334s) [noise=true , address=145.79.227.46:4603 , piece_id=NNYCFUISSSOYPBTF2YGIMM7BNAW3EIYZTFKBGMQVEA5MNEHU6GQA, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.643600902 (629h46m44.514683937s) [noise=true , address=45.91.116.10:4083 , piece_id=EKQLXTCN4KSWG4GTBSB6FVVSBEP4L5ONBKXEMEQIOBYDYA5ZOUIA, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.644749149 (629h46m44.515832184s) [noise=true , address=145.79.227.31:4799 , piece_id=54IF73MWOTIOJ4WXV3WGUJMK7JMLGIVZIEOSI3PTYHZE5M263JFQ, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.646078710 (629h46m44.517161745s) [noise=true , address=77.90.31.21:4880 , piece_id=UEJYYPH4WYBOEV2ZMZDDHNGIHT2JH57AIQY3XZ2HLBV6F5QOYCTQ, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.647522739 (629h46m44.518605784s) [noise=true , address=45.91.116.6:4973 , piece_id=NSEDZOAZWJEHL2YJPRPT3Y37HLGYQSLCSIRP62IHYLFTDVFEIUUQ, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:46.344821972 (629h46m46.215905007s) [upload=551868]: Next returned nil segment Dec 31 21:15:46.344867048 (629h46m46.215950093s) [upload=551868]: Waiting for error group managing segments... Dec 31 21:15:46.624836599 (629h46m46.495919634s) [noise=true , address=145.79.227.48:4303 , piece_id=5CQNY7DPO4HVD4JXS4WBJEN3I7ZUPUNSY45HCEC25VZSW7R4Y5BQ, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.624940787 (629h46m46.496023832s) [noise=true , address=145.79.227.15:4875 , piece_id=KQMUIZMYY5TKHKLOVE2HYQEU5WYCZOOB5AJVROAE5WVSGZJDXMOQ, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.636984079 (629h46m46.508067114s) [noise=true , address=145.79.227.51:4410 , piece_id=QH57IGTPJZ57O6EHENLY2JU7VKVOZSYVGBVODRC2I5LL2AWZHV6Q, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.641160311 (629h46m46.512243356s) [noise=true , address=145.79.227.13:4803 , piece_id=TZJI24W2EQRLGP5NOL3KVN7R5HDJMEEWPH5WCXZG5WTDMA4FUDXQ, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.643189464 (629h46m46.514272509s) [noise=true , address=145.79.227.52:4446 , piece_id=7QK754CTPSPTRHM6LCLEBKRKSYLHJMAM5FZOJNDJKXZFJITSHWRQ, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.673375523 (629h46m46.544458568s) [noise=true , address=145.79.227.31:4799 , piece_id=54IF73MWOTIOJ4WXV3WGUJMK7JMLGIVZIEOSI3PTYHZE5M263JFQ, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.677399746 (629h46m46.548482801s) [noise=true , address=145.79.227.11:4730 , piece_id=BQLBPYXQPS7NTNKWT3NE4Z4VZPCKRAMBFBEAY3BVF7Q34FHZD6AQ, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.710091896 (629h46m46.581174931s) [noise=true , address=145.79.227.10:4403 , piece_id=CHUKGI35OWLDMZG2GGZHMC7RL6RDGNGIXT3KZZHBWKRXPLSJ3PMA, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.716974232 (629h46m46.588057267s) [noise=true , address=145.79.227.46:4603 , piece_id=NNYCFUISSSOYPBTF2YGIMM7BNAW3EIYZTFKBGMQVEA5MNEHU6GQA, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.799312184 (629h46m46.670395219s) [noise=true , address=77.90.31.13:4627 , piece_id=CC55YZD4CWFZBRQASU6VB33RMJZ6RYGEZ6Y7SNLYILCRVRDBMMUQ, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.929310008 (629h46m46.800393053s) [noise=true , address=45.91.116.12:4225 , piece_id=QF3QQSR5EF5T42IFPEWOCIAQBPAQPNVGBY5LLRO2IUJ5VHC4QNZA, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.930088710 (629h46m46.801171755s) [noise=true , address=77.90.31.21:4880 , piece_id=UEJYYPH4WYBOEV2ZMZDDHNGIHT2JH57AIQY3XZ2HLBV6F5QOYCTQ, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.943538120 (629h46m46.814621155s) [noise=true , address=45.91.116.11:4118 , piece_id=U4ZYIZQIZLHUQDHRK43NC2XEXOR3LDSS32WLEHYA5QGNIW4R5ZPQ, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.943559901 (629h46m46.814642946s) [seg_pos={0 0}, upload=551868]: Segment reached optimal threshold of 13 pieces. Dec 31 21:15:46.943767576 (629h46m46.814850621s) [noise=true , address=45.91.116.13:4156 , piece_id=CVX3XD5GRJVGPMTIO3GQGMPL3V76CREYNE4VOS22CNW3KNLH5VXA, seg_pos={0 0}, upload=551868]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 31 21:15:46.943777847 (629h46m46.814860892s) [noise=true , address=45.91.116.13:4156 , piece_id=CVX3XD5GRJVGPMTIO3GQGMPL3V76CREYNE4VOS22CNW3KNLH5VXA, seg_pos={0 0}, upload=551868]: Long tail cancellation due to optimal threshold Dec 31 21:15:46.944026911 (629h46m46.815109956s) [noise=true , address=45.91.116.6:4973 , piece_id=NSEDZOAZWJEHL2YJPRPT3Y37HLGYQSLCSIRP62IHYLFTDVFEIUUQ, seg_pos={0 0}, upload=551868]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF Dec 31 21:15:46.944037180 (629h46m46.815120215s) [noise=true , address=45.91.116.6:4973 , piece_id=NSEDZOAZWJEHL2YJPRPT3Y37HLGYQSLCSIRP62IHYLFTDVFEIUUQ, seg_pos={0 0}, upload=551868]: Long tail cancellation due to optimal threshold Dec 31 21:15:46.944059453 (629h46m46.815142498s) [noise=true , address=45.91.116.10:4083 , piece_id=EKQLXTCN4KSWG4GTBSB6FVVSBEP4L5ONBKXEMEQIOBYDYA5ZOUIA, seg_pos={0 0}, upload=551868]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 31 21:15:46.944068169 (629h46m46.815151204s) [noise=true , address=45.91.116.10:4083 , piece_id=EKQLXTCN4KSWG4GTBSB6FVVSBEP4L5ONBKXEMEQIOBYDYA5ZOUIA, seg_pos={0 0}, upload=551868]: Long tail cancellation due to optimal threshold Dec 31 21:15:46.944090411 (629h46m46.815173446s) [seg_pos={0 0}, upload=551868]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 21:15:46.944174672 (629h46m46.815257717s) [upload=551868]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 21:15:46.944181034 (629h46m46.815264069s) [upload=551868]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 21:15:47.234878081 (629h46m47.105961136s) [upload=551868]: Done uploading segments... Dec 31 21:15:47.234901195 (629h46m47.10598424s) [upload=551868]: Upload finished. err: <nil> Duration: 8.993819966s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [.X...XX....X......X....XX........X...X.......X.....X...XX.X.] duration: 29.689829544s trace: https://jaeger.dev.storj.io/trace/5f2b7d22d21cf0f8 FAIL (2025-12-31 21:16:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: U4ZYIZQIZLHUQDHRK43NC2XEXOR3LDSS32WLEHYA5QGNIW4R5ZPQ) context deadline exceeded; piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: UEJYYPH4WYBOEV2ZMZDDHNGIHT2JH57AIQY3XZ2HLBV6F5QOYCTQ) context deadline exceeded; piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: CC55YZD4CWFZBRQASU6VB33RMJZ6RYGEZ6Y7SNLYILCRVRDBMMUQ) context deadline exceeded ======LOGS====== Dec 31 21:15:44.386938901 (629h46m44.258021936s) [upload=551868]: Starting upload Dec 31 21:15:44.386994888 (629h46m44.258077923s) [upload=551868]: Done starting upload Dec 31 21:15:44.387183197 (629h46m44.258266242s) [upload=551868]: Uploading segments... Dec 31 21:15:44.387281093 (629h46m44.258364158s) [upload=551868]: Got next segment. Inline: false Dec 31 21:15:44.387329566 (629h46m44.258412621s) [upload=551868]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 21:15:44.387337030 (629h46m44.258420075s) [upload=551868]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 21:15:44.637074656 (629h46m44.508157691s) [seg_pos={0 0}, upload=551868]: Begin upload segment... Dec 31 21:15:44.637126884 (629h46m44.50820992s) [seg_pos={0 0}, upload=551868]: Done begin upload segment. Dec 31 21:15:44.637137575 (629h46m44.50822061s) [noise=true , address=45.91.116.13:4156 , piece_id=CVX3XD5GRJVGPMTIO3GQGMPL3V76CREYNE4VOS22CNW3KNLH5VXA, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.637158976 (629h46m44.508242021s) [noise=true , address=45.91.116.12:4225 , piece_id=QF3QQSR5EF5T42IFPEWOCIAQBPAQPNVGBY5LLRO2IUJ5VHC4QNZA, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.637344248 (629h46m44.508427283s) [noise=true , address=145.79.227.48:4303 , piece_id=5CQNY7DPO4HVD4JXS4WBJEN3I7ZUPUNSY45HCEC25VZSW7R4Y5BQ, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.637642296 (629h46m44.508725331s) [noise=true , address=145.79.227.10:4403 , piece_id=CHUKGI35OWLDMZG2GGZHMC7RL6RDGNGIXT3KZZHBWKRXPLSJ3PMA, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.637935754 (629h46m44.5090188s) [noise=true , address=145.79.227.52:4446 , piece_id=7QK754CTPSPTRHM6LCLEBKRKSYLHJMAM5FZOJNDJKXZFJITSHWRQ, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.638274770 (629h46m44.509357815s) [noise=true , address=145.79.227.13:4803 , piece_id=TZJI24W2EQRLGP5NOL3KVN7R5HDJMEEWPH5WCXZG5WTDMA4FUDXQ, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.638548111 (629h46m44.509631146s) [noise=true , address=145.79.227.11:4730 , piece_id=BQLBPYXQPS7NTNKWT3NE4Z4VZPCKRAMBFBEAY3BVF7Q34FHZD6AQ, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.638806723 (629h46m44.509889768s) [noise=true , address=145.79.227.15:4875 , piece_id=KQMUIZMYY5TKHKLOVE2HYQEU5WYCZOOB5AJVROAE5WVSGZJDXMOQ, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.639124738 (629h46m44.510207783s) [noise=true , address=145.79.227.51:4410 , piece_id=QH57IGTPJZ57O6EHENLY2JU7VKVOZSYVGBVODRC2I5LL2AWZHV6Q, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.639522837 (629h46m44.510605882s) [noise=true , address=77.90.31.13:4627 , piece_id=CC55YZD4CWFZBRQASU6VB33RMJZ6RYGEZ6Y7SNLYILCRVRDBMMUQ, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.641025558 (629h46m44.512108593s) [noise=true , address=45.91.116.11:4118 , piece_id=U4ZYIZQIZLHUQDHRK43NC2XEXOR3LDSS32WLEHYA5QGNIW4R5ZPQ, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.642391289 (629h46m44.513474334s) [noise=true , address=145.79.227.46:4603 , piece_id=NNYCFUISSSOYPBTF2YGIMM7BNAW3EIYZTFKBGMQVEA5MNEHU6GQA, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.643600902 (629h46m44.514683937s) [noise=true , address=45.91.116.10:4083 , piece_id=EKQLXTCN4KSWG4GTBSB6FVVSBEP4L5ONBKXEMEQIOBYDYA5ZOUIA, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.644749149 (629h46m44.515832184s) [noise=true , address=145.79.227.31:4799 , piece_id=54IF73MWOTIOJ4WXV3WGUJMK7JMLGIVZIEOSI3PTYHZE5M263JFQ, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.646078710 (629h46m44.517161745s) [noise=true , address=77.90.31.21:4880 , piece_id=UEJYYPH4WYBOEV2ZMZDDHNGIHT2JH57AIQY3XZ2HLBV6F5QOYCTQ, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.647522739 (629h46m44.518605784s) [noise=true , address=45.91.116.6:4973 , piece_id=NSEDZOAZWJEHL2YJPRPT3Y37HLGYQSLCSIRP62IHYLFTDVFEIUUQ, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:46.344821972 (629h46m46.215905007s) [upload=551868]: Next returned nil segment Dec 31 21:15:46.344867048 (629h46m46.215950093s) [upload=551868]: Waiting for error group managing segments... Dec 31 21:15:46.624836599 (629h46m46.495919634s) [noise=true , address=145.79.227.48:4303 , piece_id=5CQNY7DPO4HVD4JXS4WBJEN3I7ZUPUNSY45HCEC25VZSW7R4Y5BQ, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.624940787 (629h46m46.496023832s) [noise=true , address=145.79.227.15:4875 , piece_id=KQMUIZMYY5TKHKLOVE2HYQEU5WYCZOOB5AJVROAE5WVSGZJDXMOQ, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.636984079 (629h46m46.508067114s) [noise=true , address=145.79.227.51:4410 , piece_id=QH57IGTPJZ57O6EHENLY2JU7VKVOZSYVGBVODRC2I5LL2AWZHV6Q, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.641160311 (629h46m46.512243356s) [noise=true , address=145.79.227.13:4803 , piece_id=TZJI24W2EQRLGP5NOL3KVN7R5HDJMEEWPH5WCXZG5WTDMA4FUDXQ, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.643189464 (629h46m46.514272509s) [noise=true , address=145.79.227.52:4446 , piece_id=7QK754CTPSPTRHM6LCLEBKRKSYLHJMAM5FZOJNDJKXZFJITSHWRQ, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.673375523 (629h46m46.544458568s) [noise=true , address=145.79.227.31:4799 , piece_id=54IF73MWOTIOJ4WXV3WGUJMK7JMLGIVZIEOSI3PTYHZE5M263JFQ, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.677399746 (629h46m46.548482801s) [noise=true , address=145.79.227.11:4730 , piece_id=BQLBPYXQPS7NTNKWT3NE4Z4VZPCKRAMBFBEAY3BVF7Q34FHZD6AQ, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.710091896 (629h46m46.581174931s) [noise=true , address=145.79.227.10:4403 , piece_id=CHUKGI35OWLDMZG2GGZHMC7RL6RDGNGIXT3KZZHBWKRXPLSJ3PMA, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.716974232 (629h46m46.588057267s) [noise=true , address=145.79.227.46:4603 , piece_id=NNYCFUISSSOYPBTF2YGIMM7BNAW3EIYZTFKBGMQVEA5MNEHU6GQA, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.799312184 (629h46m46.670395219s) [noise=true , address=77.90.31.13:4627 , piece_id=CC55YZD4CWFZBRQASU6VB33RMJZ6RYGEZ6Y7SNLYILCRVRDBMMUQ, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.929310008 (629h46m46.800393053s) [noise=true , address=45.91.116.12:4225 , piece_id=QF3QQSR5EF5T42IFPEWOCIAQBPAQPNVGBY5LLRO2IUJ5VHC4QNZA, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.930088710 (629h46m46.801171755s) [noise=true , address=77.90.31.21:4880 , piece_id=UEJYYPH4WYBOEV2ZMZDDHNGIHT2JH57AIQY3XZ2HLBV6F5QOYCTQ, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.943538120 (629h46m46.814621155s) [noise=true , address=45.91.116.11:4118 , piece_id=U4ZYIZQIZLHUQDHRK43NC2XEXOR3LDSS32WLEHYA5QGNIW4R5ZPQ, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.943559901 (629h46m46.814642946s) [seg_pos={0 0}, upload=551868]: Segment reached optimal threshold of 13 pieces. Dec 31 21:15:46.943767576 (629h46m46.814850621s) [noise=true , address=45.91.116.13:4156 , piece_id=CVX3XD5GRJVGPMTIO3GQGMPL3V76CREYNE4VOS22CNW3KNLH5VXA, seg_pos={0 0}, upload=551868]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 31 21:15:46.943777847 (629h46m46.814860892s) [noise=true , address=45.91.116.13:4156 , piece_id=CVX3XD5GRJVGPMTIO3GQGMPL3V76CREYNE4VOS22CNW3KNLH5VXA, seg_pos={0 0}, upload=551868]: Long tail cancellation due to optimal threshold Dec 31 21:15:46.944026911 (629h46m46.815109956s) [noise=true , address=45.91.116.6:4973 , piece_id=NSEDZOAZWJEHL2YJPRPT3Y37HLGYQSLCSIRP62IHYLFTDVFEIUUQ, seg_pos={0 0}, upload=551868]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF Dec 31 21:15:46.944037180 (629h46m46.815120215s) [noise=true , address=45.91.116.6:4973 , piece_id=NSEDZOAZWJEHL2YJPRPT3Y37HLGYQSLCSIRP62IHYLFTDVFEIUUQ, seg_pos={0 0}, upload=551868]: Long tail cancellation due to optimal threshold Dec 31 21:15:46.944059453 (629h46m46.815142498s) [noise=true , address=45.91.116.10:4083 , piece_id=EKQLXTCN4KSWG4GTBSB6FVVSBEP4L5ONBKXEMEQIOBYDYA5ZOUIA, seg_pos={0 0}, upload=551868]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 31 21:15:46.944068169 (629h46m46.815151204s) [noise=true , address=45.91.116.10:4083 , piece_id=EKQLXTCN4KSWG4GTBSB6FVVSBEP4L5ONBKXEMEQIOBYDYA5ZOUIA, seg_pos={0 0}, upload=551868]: Long tail cancellation due to optimal threshold Dec 31 21:15:46.944090411 (629h46m46.815173446s) [seg_pos={0 0}, upload=551868]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 21:15:46.944174672 (629h46m46.815257717s) [upload=551868]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 21:15:46.944181034 (629h46m46.815264069s) [upload=551868]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 21:15:47.234878081 (629h46m47.105961136s) [upload=551868]: Done uploading segments... Dec 31 21:15:47.234901195 (629h46m47.10598424s) [upload=551868]: Upload finished. err: <nil> Duration: 29.689847688s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [..X...XX....X......X....XX........X...X.......X.....X...XX.X] duration: 30.003454648s trace: https://jaeger.dev.storj.io/trace/5f2b7d22d21cf0f8 FAIL (2025-12-31 21:16:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: U4ZYIZQIZLHUQDHRK43NC2XEXOR3LDSS32WLEHYA5QGNIW4R5ZPQ) context deadline exceeded; piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: UEJYYPH4WYBOEV2ZMZDDHNGIHT2JH57AIQY3XZ2HLBV6F5QOYCTQ) context deadline exceeded; piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: CC55YZD4CWFZBRQASU6VB33RMJZ6RYGEZ6Y7SNLYILCRVRDBMMUQ) context deadline exceeded ======LOGS====== Dec 31 21:15:44.386938901 (629h46m44.258021936s) [upload=551868]: Starting upload Dec 31 21:15:44.386994888 (629h46m44.258077923s) [upload=551868]: Done starting upload Dec 31 21:15:44.387183197 (629h46m44.258266242s) [upload=551868]: Uploading segments... Dec 31 21:15:44.387281093 (629h46m44.258364158s) [upload=551868]: Got next segment. Inline: false Dec 31 21:15:44.387329566 (629h46m44.258412621s) [upload=551868]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 21:15:44.387337030 (629h46m44.258420075s) [upload=551868]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 21:15:44.637074656 (629h46m44.508157691s) [seg_pos={0 0}, upload=551868]: Begin upload segment... Dec 31 21:15:44.637126884 (629h46m44.50820992s) [seg_pos={0 0}, upload=551868]: Done begin upload segment. Dec 31 21:15:44.637137575 (629h46m44.50822061s) [noise=true , address=45.91.116.13:4156 , piece_id=CVX3XD5GRJVGPMTIO3GQGMPL3V76CREYNE4VOS22CNW3KNLH5VXA, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.637158976 (629h46m44.508242021s) [noise=true , address=45.91.116.12:4225 , piece_id=QF3QQSR5EF5T42IFPEWOCIAQBPAQPNVGBY5LLRO2IUJ5VHC4QNZA, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.637344248 (629h46m44.508427283s) [noise=true , address=145.79.227.48:4303 , piece_id=5CQNY7DPO4HVD4JXS4WBJEN3I7ZUPUNSY45HCEC25VZSW7R4Y5BQ, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.637642296 (629h46m44.508725331s) [noise=true , address=145.79.227.10:4403 , piece_id=CHUKGI35OWLDMZG2GGZHMC7RL6RDGNGIXT3KZZHBWKRXPLSJ3PMA, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.637935754 (629h46m44.5090188s) [noise=true , address=145.79.227.52:4446 , piece_id=7QK754CTPSPTRHM6LCLEBKRKSYLHJMAM5FZOJNDJKXZFJITSHWRQ, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.638274770 (629h46m44.509357815s) [noise=true , address=145.79.227.13:4803 , piece_id=TZJI24W2EQRLGP5NOL3KVN7R5HDJMEEWPH5WCXZG5WTDMA4FUDXQ, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.638548111 (629h46m44.509631146s) [noise=true , address=145.79.227.11:4730 , piece_id=BQLBPYXQPS7NTNKWT3NE4Z4VZPCKRAMBFBEAY3BVF7Q34FHZD6AQ, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.638806723 (629h46m44.509889768s) [noise=true , address=145.79.227.15:4875 , piece_id=KQMUIZMYY5TKHKLOVE2HYQEU5WYCZOOB5AJVROAE5WVSGZJDXMOQ, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.639124738 (629h46m44.510207783s) [noise=true , address=145.79.227.51:4410 , piece_id=QH57IGTPJZ57O6EHENLY2JU7VKVOZSYVGBVODRC2I5LL2AWZHV6Q, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.639522837 (629h46m44.510605882s) [noise=true , address=77.90.31.13:4627 , piece_id=CC55YZD4CWFZBRQASU6VB33RMJZ6RYGEZ6Y7SNLYILCRVRDBMMUQ, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.641025558 (629h46m44.512108593s) [noise=true , address=45.91.116.11:4118 , piece_id=U4ZYIZQIZLHUQDHRK43NC2XEXOR3LDSS32WLEHYA5QGNIW4R5ZPQ, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.642391289 (629h46m44.513474334s) [noise=true , address=145.79.227.46:4603 , piece_id=NNYCFUISSSOYPBTF2YGIMM7BNAW3EIYZTFKBGMQVEA5MNEHU6GQA, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.643600902 (629h46m44.514683937s) [noise=true , address=45.91.116.10:4083 , piece_id=EKQLXTCN4KSWG4GTBSB6FVVSBEP4L5ONBKXEMEQIOBYDYA5ZOUIA, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.644749149 (629h46m44.515832184s) [noise=true , address=145.79.227.31:4799 , piece_id=54IF73MWOTIOJ4WXV3WGUJMK7JMLGIVZIEOSI3PTYHZE5M263JFQ, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.646078710 (629h46m44.517161745s) [noise=true , address=77.90.31.21:4880 , piece_id=UEJYYPH4WYBOEV2ZMZDDHNGIHT2JH57AIQY3XZ2HLBV6F5QOYCTQ, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:44.647522739 (629h46m44.518605784s) [noise=true , address=45.91.116.6:4973 , piece_id=NSEDZOAZWJEHL2YJPRPT3Y37HLGYQSLCSIRP62IHYLFTDVFEIUUQ, seg_pos={0 0}, upload=551868]: Uploading piece... Dec 31 21:15:46.344821972 (629h46m46.215905007s) [upload=551868]: Next returned nil segment Dec 31 21:15:46.344867048 (629h46m46.215950093s) [upload=551868]: Waiting for error group managing segments... Dec 31 21:15:46.624836599 (629h46m46.495919634s) [noise=true , address=145.79.227.48:4303 , piece_id=5CQNY7DPO4HVD4JXS4WBJEN3I7ZUPUNSY45HCEC25VZSW7R4Y5BQ, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.624940787 (629h46m46.496023832s) [noise=true , address=145.79.227.15:4875 , piece_id=KQMUIZMYY5TKHKLOVE2HYQEU5WYCZOOB5AJVROAE5WVSGZJDXMOQ, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.636984079 (629h46m46.508067114s) [noise=true , address=145.79.227.51:4410 , piece_id=QH57IGTPJZ57O6EHENLY2JU7VKVOZSYVGBVODRC2I5LL2AWZHV6Q, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.641160311 (629h46m46.512243356s) [noise=true , address=145.79.227.13:4803 , piece_id=TZJI24W2EQRLGP5NOL3KVN7R5HDJMEEWPH5WCXZG5WTDMA4FUDXQ, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.643189464 (629h46m46.514272509s) [noise=true , address=145.79.227.52:4446 , piece_id=7QK754CTPSPTRHM6LCLEBKRKSYLHJMAM5FZOJNDJKXZFJITSHWRQ, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.673375523 (629h46m46.544458568s) [noise=true , address=145.79.227.31:4799 , piece_id=54IF73MWOTIOJ4WXV3WGUJMK7JMLGIVZIEOSI3PTYHZE5M263JFQ, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.677399746 (629h46m46.548482801s) [noise=true , address=145.79.227.11:4730 , piece_id=BQLBPYXQPS7NTNKWT3NE4Z4VZPCKRAMBFBEAY3BVF7Q34FHZD6AQ, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.710091896 (629h46m46.581174931s) [noise=true , address=145.79.227.10:4403 , piece_id=CHUKGI35OWLDMZG2GGZHMC7RL6RDGNGIXT3KZZHBWKRXPLSJ3PMA, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.716974232 (629h46m46.588057267s) [noise=true , address=145.79.227.46:4603 , piece_id=NNYCFUISSSOYPBTF2YGIMM7BNAW3EIYZTFKBGMQVEA5MNEHU6GQA, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.799312184 (629h46m46.670395219s) [noise=true , address=77.90.31.13:4627 , piece_id=CC55YZD4CWFZBRQASU6VB33RMJZ6RYGEZ6Y7SNLYILCRVRDBMMUQ, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.929310008 (629h46m46.800393053s) [noise=true , address=45.91.116.12:4225 , piece_id=QF3QQSR5EF5T42IFPEWOCIAQBPAQPNVGBY5LLRO2IUJ5VHC4QNZA, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.930088710 (629h46m46.801171755s) [noise=true , address=77.90.31.21:4880 , piece_id=UEJYYPH4WYBOEV2ZMZDDHNGIHT2JH57AIQY3XZ2HLBV6F5QOYCTQ, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.943538120 (629h46m46.814621155s) [noise=true , address=45.91.116.11:4118 , piece_id=U4ZYIZQIZLHUQDHRK43NC2XEXOR3LDSS32WLEHYA5QGNIW4R5ZPQ, seg_pos={0 0}, upload=551868]: Done uploading piece. err: <nil> Dec 31 21:15:46.943559901 (629h46m46.814642946s) [seg_pos={0 0}, upload=551868]: Segment reached optimal threshold of 13 pieces. Dec 31 21:15:46.943767576 (629h46m46.814850621s) [noise=true , address=45.91.116.13:4156 , piece_id=CVX3XD5GRJVGPMTIO3GQGMPL3V76CREYNE4VOS22CNW3KNLH5VXA, seg_pos={0 0}, upload=551868]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 31 21:15:46.943777847 (629h46m46.814860892s) [noise=true , address=45.91.116.13:4156 , piece_id=CVX3XD5GRJVGPMTIO3GQGMPL3V76CREYNE4VOS22CNW3KNLH5VXA, seg_pos={0 0}, upload=551868]: Long tail cancellation due to optimal threshold Dec 31 21:15:46.944026911 (629h46m46.815109956s) [noise=true , address=45.91.116.6:4973 , piece_id=NSEDZOAZWJEHL2YJPRPT3Y37HLGYQSLCSIRP62IHYLFTDVFEIUUQ, seg_pos={0 0}, upload=551868]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF Dec 31 21:15:46.944037180 (629h46m46.815120215s) [noise=true , address=45.91.116.6:4973 , piece_id=NSEDZOAZWJEHL2YJPRPT3Y37HLGYQSLCSIRP62IHYLFTDVFEIUUQ, seg_pos={0 0}, upload=551868]: Long tail cancellation due to optimal threshold Dec 31 21:15:46.944059453 (629h46m46.815142498s) [noise=true , address=45.91.116.10:4083 , piece_id=EKQLXTCN4KSWG4GTBSB6FVVSBEP4L5ONBKXEMEQIOBYDYA5ZOUIA, seg_pos={0 0}, upload=551868]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 31 21:15:46.944068169 (629h46m46.815151204s) [noise=true , address=45.91.116.10:4083 , piece_id=EKQLXTCN4KSWG4GTBSB6FVVSBEP4L5ONBKXEMEQIOBYDYA5ZOUIA, seg_pos={0 0}, upload=551868]: Long tail cancellation due to optimal threshold Dec 31 21:15:46.944090411 (629h46m46.815173446s) [seg_pos={0 0}, upload=551868]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 21:15:46.944174672 (629h46m46.815257717s) [upload=551868]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 21:15:46.944181034 (629h46m46.815264069s) [upload=551868]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 21:15:47.234878081 (629h46m47.105961136s) [upload=551868]: Done uploading segments... Dec 31 21:15:47.234901195 (629h46m47.10598424s) [upload=551868]: Upload finished. err: <nil> Duration: 30.003479747s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [...X...XX....X......X....XX........X...X.......X.....X...XX.] duration: 6.172506016s trace: https://jaeger.dev.storj.io/trace/327abda4e1f5077a FAIL (2025-12-31 21:15:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 21:14:37.878679957 (629h45m37.749763002s) [upload=551850]: Starting upload Dec 31 21:14:37.878792962 (629h45m37.749875997s) [upload=551850]: Done starting upload Dec 31 21:14:37.878869748 (629h45m37.749952783s) [upload=551850]: Uploading segments... Dec 31 21:14:37.879145713 (629h45m37.750228748s) [upload=551850]: Got next segment. Inline: false Dec 31 21:14:37.879257636 (629h45m37.750340681s) [upload=551850]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 21:14:37.879264580 (629h45m37.750347625s) [upload=551850]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 21:14:38.130526779 (629h45m38.001609824s) [seg_pos={0 0}, upload=551850]: Begin upload segment... Dec 31 21:14:38.130588716 (629h45m38.001671761s) [seg_pos={0 0}, upload=551850]: Done begin upload segment. Dec 31 21:14:38.130641958 (629h45m38.001725003s) [noise=true , address=145.79.227.35:4481 , piece_id=FW5KOVEAH7C6OSHWY5WTNXZA6ZFQUZW7MLF2QCQHTPHS6IT4KWYQ, seg_pos={0 0}, upload=551850]: Uploading piece... Dec 31 21:14:38.130688246 (629h45m38.001771311s) [noise=true , address=45.91.116.4:4940 , piece_id=AYPGEB6DE4ASXTRRYIBUVQ6SELL6XKTU5TU5HEHIWONAQOKRQV7A, seg_pos={0 0}, upload=551850]: Uploading piece... Dec 31 21:14:38.131267910 (629h45m38.002350945s) [noise=true , address=145.79.227.10:4403 , piece_id=4V6UX343EGRGSXDXSNBXOVVFLVI4PCHG5AVAJXD2G3Q6KH4QJZUQ, seg_pos={0 0}, upload=551850]: Uploading piece... Dec 31 21:14:38.131733105 (629h45m38.00281614s) [noise=true , address=145.79.227.31:4797 , piece_id=RB72W6OIIKLEYX3SQL6UDWX4UWPZG2VM737PLBSGAVLJLARQUUOA, seg_pos={0 0}, upload=551850]: Uploading piece... Dec 31 21:14:38.132027897 (629h45m38.003110932s) [noise=true , address=145.79.227.4:4185 , piece_id=GYZ6Y66DQJ3KXMKMIG7UL5PUWXULWQUF5ZBCVV3XWRN2TTMORO2Q, seg_pos={0 0}, upload=551850]: Uploading piece... Dec 31 21:14:38.132416236 (629h45m38.003499271s) [noise=true , address=145.79.227.6:4259 , piece_id=LHRKVVGMVV6JEK2N2IBPCWMDEJBUP6JXM43SIK3A6B6YBP4WQD6A, seg_pos={0 0}, upload=551850]: Uploading piece... Dec 31 21:14:38.132819694 (629h45m38.003902739s) [noise=true , address=145.79.227.3:4150 , piece_id=PXZFPQMFBLDF4C5MWGTAVEKJ3VJTVBELXKBTON3ORGYGLU3HSRPA, seg_pos={0 0}, upload=551850]: Uploading piece... Dec 31 21:14:38.133711292 (629h45m38.004794327s) [noise=true , address=145.79.227.42:4121 , piece_id=GW5MIKYMWQXWNUQUYJQVETS4CMT6SURPZP65L7ZQVHPEZE7JBQPA, seg_pos={0 0}, upload=551850]: Uploading piece... Dec 31 21:14:38.134268243 (629h45m38.005351288s) [noise=true , address=45.91.116.5:4298 , piece_id=L7RM6HIQSBECVWUCJ7FJOUTYVRH2W3D2QDKH4BBK65SGJBM5WKRQ, seg_pos={0 0}, upload=551850]: Uploading piece... Dec 31 21:14:38.134681510 (629h45m38.005764545s) [noise=true , address=145.79.227.19:4078 , piece_id=MJKOSKIVDUNCV3VP3Y2TXU2QJ7KZHSNHEXVVRQ33I3CYZ3MFCN5Q, seg_pos={0 0}, upload=551850]: Uploading piece... Dec 31 21:14:38.135473608 (629h45m38.006556643s) [noise=true , address=145.79.227.17:4972 , piece_id=3JJVCXVXCQYL5DIZBSMJYWRABIJTR55WNDN7K4BSSVAM4LO3QYIA, seg_pos={0 0}, upload=551850]: Uploading piece... Dec 31 21:14:38.136172549 (629h45m38.007255584s) [noise=true , address=145.79.227.43:4158 , piece_id=ZGXPGLZ3IALYSUQXQ76Z5AMUMKER5F34QVXQQYY7UTJVR2JM2SQQ, seg_pos={0 0}, upload=551850]: Uploading piece... Dec 31 21:14:38.137090777 (629h45m38.008173812s) [noise=true , address=145.79.227.37:4567 , piece_id=XVLBULSM5YLXD565X7YLF36BQGMQGSDV5GYQ7IMAAGZMVYJN4K4A, seg_pos={0 0}, upload=551850]: Uploading piece... Dec 31 21:14:38.137540704 (629h45m38.008623739s) [noise=true , address=45.91.116.6:4973 , piece_id=FX64BIWNBLRI5TKUJFMP5FFJAJHSJIQ34RKD7IRXDAWOMNWRBXDA, seg_pos={0 0}, upload=551850]: Uploading piece... Dec 31 21:14:38.138547772 (629h45m38.009630807s) [noise=true , address=45.91.116.7:4010 , piece_id=5W7QIZHMICDRAAY7VCLADKUJVK7UJIBYKXYCVAD4W5GHMUJRBLGQ, seg_pos={0 0}, upload=551850]: Uploading piece... Dec 31 21:14:38.139019900 (629h45m38.010102945s) [noise=true , address=145.79.227.2:4114 , piece_id=3CSU6K7ODOKVHR7FF3CEABXWE2L7N7WZBUX337P2CQJP4BQY7RRQ, seg_pos={0 0}, upload=551850]: Uploading piece... Dec 31 21:14:39.821201908 (629h45m39.692284943s) [upload=551850]: Next returned nil segment Dec 31 21:14:39.821237906 (629h45m39.692320951s) [upload=551850]: Waiting for error group managing segments... Dec 31 21:14:40.122858382 (629h45m39.993941417s) [noise=true , address=145.79.227.4:4185 , piece_id=GYZ6Y66DQJ3KXMKMIG7UL5PUWXULWQUF5ZBCVV3XWRN2TTMORO2Q, seg_pos={0 0}, upload=551850]: Done uploading piece. err: <nil> Dec 31 21:14:40.129533122 (629h45m40.000616167s) [noise=true , address=145.79.227.2:4114 , piece_id=3CSU6K7ODOKVHR7FF3CEABXWE2L7N7WZBUX337P2CQJP4BQY7RRQ, seg_pos={0 0}, upload=551850]: Done uploading piece. err: <nil> Dec 31 21:14:40.129878820 (629h45m40.000961855s) [noise=true , address=145.79.227.31:4797 , piece_id=RB72W6OIIKLEYX3SQL6UDWX4UWPZG2VM737PLBSGAVLJLARQUUOA, seg_pos={0 0}, upload=551850]: Done uploading piece. err: <nil> Dec 31 21:14:40.132780135 (629h45m40.00386317s) [noise=true , address=145.79.227.42:4121 , piece_id=GW5MIKYMWQXWNUQUYJQVETS4CMT6SURPZP65L7ZQVHPEZE7JBQPA, seg_pos={0 0}, upload=551850]: Done uploading piece. err: <nil> Dec 31 21:14:40.134835318 (629h45m40.005918363s) [noise=true , address=145.79.227.10:4403 , piece_id=4V6UX343EGRGSXDXSNBXOVVFLVI4PCHG5AVAJXD2G3Q6KH4QJZUQ, seg_pos={0 0}, upload=551850]: Done uploading piece. err: <nil> Dec 31 21:14:40.144302936 (629h45m40.015385971s) [noise=true , address=145.79.227.6:4259 , piece_id=LHRKVVGMVV6JEK2N2IBPCWMDEJBUP6JXM43SIK3A6B6YBP4WQD6A, seg_pos={0 0}, upload=551850]: Done uploading piece. err: <nil> Dec 31 21:14:40.149179442 (629h45m40.020262477s) [noise=true , address=145.79.227.3:4150 , piece_id=PXZFPQMFBLDF4C5MWGTAVEKJ3VJTVBELXKBTON3ORGYGLU3HSRPA, seg_pos={0 0}, upload=551850]: Done uploading piece. err: <nil> Dec 31 21:14:40.166909511 (629h45m40.037992557s) [noise=true , address=145.79.227.37:4567 , piece_id=XVLBULSM5YLXD565X7YLF36BQGMQGSDV5GYQ7IMAAGZMVYJN4K4A, seg_pos={0 0}, upload=551850]: Done uploading piece. err: <nil> Dec 31 21:14:40.282637284 (629h45m40.153720329s) [noise=true , address=145.79.227.17:4972 , piece_id=3JJVCXVXCQYL5DIZBSMJYWRABIJTR55WNDN7K4BSSVAM4LO3QYIA, seg_pos={0 0}, upload=551850]: Done uploading piece. err: <nil> Dec 31 21:14:40.452166871 (629h45m40.323249906s) [noise=true , address=45.91.116.7:4010 , piece_id=5W7QIZHMICDRAAY7VCLADKUJVK7UJIBYKXYCVAD4W5GHMUJRBLGQ, seg_pos={0 0}, upload=551850]: Done uploading piece. err: <nil> Dec 31 21:14:40.452845433 (629h45m40.323928508s) [noise=true , address=145.79.227.19:4078 , piece_id=MJKOSKIVDUNCV3VP3Y2TXU2QJ7KZHSNHEXVVRQ33I3CYZ3MFCN5Q, seg_pos={0 0}, upload=551850]: Done uploading piece. err: <nil> Dec 31 21:14:40.534638859 (629h45m40.405721894s) [noise=true , address=45.91.116.4:4940 , piece_id=AYPGEB6DE4ASXTRRYIBUVQ6SELL6XKTU5TU5HEHIWONAQOKRQV7A, seg_pos={0 0}, upload=551850]: Done uploading piece. err: <nil> Dec 31 21:14:41.313362154 (629h45m41.184445189s) [noise=true , address=45.91.116.6:4973 , piece_id=FX64BIWNBLRI5TKUJFMP5FFJAJHSJIQ34RKD7IRXDAWOMNWRBXDA, seg_pos={0 0}, upload=551850]: Done uploading piece. err: <nil> Dec 31 21:14:41.313381652 (629h45m41.184464697s) [seg_pos={0 0}, upload=551850]: Segment reached optimal threshold of 13 pieces. Dec 31 21:14:41.313560312 (629h45m41.184643347s) [noise=true , address=145.79.227.35:4481 , piece_id=FW5KOVEAH7C6OSHWY5WTNXZA6ZFQUZW7MLF2QCQHTPHS6IT4KWYQ, seg_pos={0 0}, upload=551850]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 31 21:14:41.313568167 (629h45m41.184651202s) [noise=true , address=145.79.227.35:4481 , piece_id=FW5KOVEAH7C6OSHWY5WTNXZA6ZFQUZW7MLF2QCQHTPHS6IT4KWYQ, seg_pos={0 0}, upload=551850]: Long tail cancellation due to optimal threshold Dec 31 21:14:41.313712041 (629h45m41.184795086s) [noise=true , address=45.91.116.5:4298 , piece_id=L7RM6HIQSBECVWUCJ7FJOUTYVRH2W3D2QDKH4BBK65SGJBM5WKRQ, seg_pos={0 0}, upload=551850]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: write tcp 188.166.225.170:47426->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:47426->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47426->45.91.116.5:4298: use of closed network connection Dec 31 21:14:41.313717492 (629h45m41.184800537s) [noise=true , address=45.91.116.5:4298 , piece_id=L7RM6HIQSBECVWUCJ7FJOUTYVRH2W3D2QDKH4BBK65SGJBM5WKRQ, seg_pos={0 0}, upload=551850]: Long tail cancellation due to optimal threshold Dec 31 21:14:41.313870343 (629h45m41.184953388s) [noise=true , address=145.79.227.43:4158 , piece_id=ZGXPGLZ3IALYSUQXQ76Z5AMUMKER5F34QVXQQYY7UTJVR2JM2SQQ, seg_pos={0 0}, upload=551850]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2): protocol: expected piece hash; context canceled; EOF Dec 31 21:14:41.313909527 (629h45m41.184992562s) [noise=true , address=145.79.227.43:4158 , piece_id=ZGXPGLZ3IALYSUQXQ76Z5AMUMKER5F34QVXQQYY7UTJVR2JM2SQQ, seg_pos={0 0}, upload=551850]: Long tail cancellation due to optimal threshold Dec 31 21:14:41.313940516 (629h45m41.185023551s) [seg_pos={0 0}, upload=551850]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 21:14:41.314094719 (629h45m41.185177764s) [upload=551850]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 21:14:41.314102986 (629h45m41.185186021s) [upload=551850]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 21:14:41.607320391 (629h45m41.478403426s) [upload=551850]: Done uploading segments... Dec 31 21:14:41.607338737 (629h45m41.478421782s) [upload=551850]: Upload finished. err: <nil> Duration: 6.172530563s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [X...X...XX....X......X....XX........X...X.......X.....X...XX] duration: 30.003226485s trace: https://jaeger.dev.storj.io/trace/327abda4e1f5077a FAIL (2025-12-31 21:15:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 21:14:37.878679957 (629h45m37.749763002s) [upload=551850]: Starting upload Dec 31 21:14:37.878792962 (629h45m37.749875997s) [upload=551850]: Done starting upload Dec 31 21:14:37.878869748 (629h45m37.749952783s) [upload=551850]: Uploading segments... Dec 31 21:14:37.879145713 (629h45m37.750228748s) [upload=551850]: Got next segment. Inline: false Dec 31 21:14:37.879257636 (629h45m37.750340681s) [upload=551850]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 21:14:37.879264580 (629h45m37.750347625s) [upload=551850]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 21:14:38.130526779 (629h45m38.001609824s) [seg_pos={0 0}, upload=551850]: Begin upload segment... Dec 31 21:14:38.130588716 (629h45m38.001671761s) [seg_pos={0 0}, upload=551850]: Done begin upload segment. Dec 31 21:14:38.130641958 (629h45m38.001725003s) [noise=true , address=145.79.227.35:4481 , piece_id=FW5KOVEAH7C6OSHWY5WTNXZA6ZFQUZW7MLF2QCQHTPHS6IT4KWYQ, seg_pos={0 0}, upload=551850]: Uploading piece... Dec 31 21:14:38.130688246 (629h45m38.001771311s) [noise=true , address=45.91.116.4:4940 , piece_id=AYPGEB6DE4ASXTRRYIBUVQ6SELL6XKTU5TU5HEHIWONAQOKRQV7A, seg_pos={0 0}, upload=551850]: Uploading piece... Dec 31 21:14:38.131267910 (629h45m38.002350945s) [noise=true , address=145.79.227.10:4403 , piece_id=4V6UX343EGRGSXDXSNBXOVVFLVI4PCHG5AVAJXD2G3Q6KH4QJZUQ, seg_pos={0 0}, upload=551850]: Uploading piece... Dec 31 21:14:38.131733105 (629h45m38.00281614s) [noise=true , address=145.79.227.31:4797 , piece_id=RB72W6OIIKLEYX3SQL6UDWX4UWPZG2VM737PLBSGAVLJLARQUUOA, seg_pos={0 0}, upload=551850]: Uploading piece... Dec 31 21:14:38.132027897 (629h45m38.003110932s) [noise=true , address=145.79.227.4:4185 , piece_id=GYZ6Y66DQJ3KXMKMIG7UL5PUWXULWQUF5ZBCVV3XWRN2TTMORO2Q, seg_pos={0 0}, upload=551850]: Uploading piece... Dec 31 21:14:38.132416236 (629h45m38.003499271s) [noise=true , address=145.79.227.6:4259 , piece_id=LHRKVVGMVV6JEK2N2IBPCWMDEJBUP6JXM43SIK3A6B6YBP4WQD6A, seg_pos={0 0}, upload=551850]: Uploading piece... Dec 31 21:14:38.132819694 (629h45m38.003902739s) [noise=true , address=145.79.227.3:4150 , piece_id=PXZFPQMFBLDF4C5MWGTAVEKJ3VJTVBELXKBTON3ORGYGLU3HSRPA, seg_pos={0 0}, upload=551850]: Uploading piece... Dec 31 21:14:38.133711292 (629h45m38.004794327s) [noise=true , address=145.79.227.42:4121 , piece_id=GW5MIKYMWQXWNUQUYJQVETS4CMT6SURPZP65L7ZQVHPEZE7JBQPA, seg_pos={0 0}, upload=551850]: Uploading piece... Dec 31 21:14:38.134268243 (629h45m38.005351288s) [noise=true , address=45.91.116.5:4298 , piece_id=L7RM6HIQSBECVWUCJ7FJOUTYVRH2W3D2QDKH4BBK65SGJBM5WKRQ, seg_pos={0 0}, upload=551850]: Uploading piece... Dec 31 21:14:38.134681510 (629h45m38.005764545s) [noise=true , address=145.79.227.19:4078 , piece_id=MJKOSKIVDUNCV3VP3Y2TXU2QJ7KZHSNHEXVVRQ33I3CYZ3MFCN5Q, seg_pos={0 0}, upload=551850]: Uploading piece... Dec 31 21:14:38.135473608 (629h45m38.006556643s) [noise=true , address=145.79.227.17:4972 , piece_id=3JJVCXVXCQYL5DIZBSMJYWRABIJTR55WNDN7K4BSSVAM4LO3QYIA, seg_pos={0 0}, upload=551850]: Uploading piece... Dec 31 21:14:38.136172549 (629h45m38.007255584s) [noise=true , address=145.79.227.43:4158 , piece_id=ZGXPGLZ3IALYSUQXQ76Z5AMUMKER5F34QVXQQYY7UTJVR2JM2SQQ, seg_pos={0 0}, upload=551850]: Uploading piece... Dec 31 21:14:38.137090777 (629h45m38.008173812s) [noise=true , address=145.79.227.37:4567 , piece_id=XVLBULSM5YLXD565X7YLF36BQGMQGSDV5GYQ7IMAAGZMVYJN4K4A, seg_pos={0 0}, upload=551850]: Uploading piece... Dec 31 21:14:38.137540704 (629h45m38.008623739s) [noise=true , address=45.91.116.6:4973 , piece_id=FX64BIWNBLRI5TKUJFMP5FFJAJHSJIQ34RKD7IRXDAWOMNWRBXDA, seg_pos={0 0}, upload=551850]: Uploading piece... Dec 31 21:14:38.138547772 (629h45m38.009630807s) [noise=true , address=45.91.116.7:4010 , piece_id=5W7QIZHMICDRAAY7VCLADKUJVK7UJIBYKXYCVAD4W5GHMUJRBLGQ, seg_pos={0 0}, upload=551850]: Uploading piece... Dec 31 21:14:38.139019900 (629h45m38.010102945s) [noise=true , address=145.79.227.2:4114 , piece_id=3CSU6K7ODOKVHR7FF3CEABXWE2L7N7WZBUX337P2CQJP4BQY7RRQ, seg_pos={0 0}, upload=551850]: Uploading piece... Dec 31 21:14:39.821201908 (629h45m39.692284943s) [upload=551850]: Next returned nil segment Dec 31 21:14:39.821237906 (629h45m39.692320951s) [upload=551850]: Waiting for error group managing segments... Dec 31 21:14:40.122858382 (629h45m39.993941417s) [noise=true , address=145.79.227.4:4185 , piece_id=GYZ6Y66DQJ3KXMKMIG7UL5PUWXULWQUF5ZBCVV3XWRN2TTMORO2Q, seg_pos={0 0}, upload=551850]: Done uploading piece. err: <nil> Dec 31 21:14:40.129533122 (629h45m40.000616167s) [noise=true , address=145.79.227.2:4114 , piece_id=3CSU6K7ODOKVHR7FF3CEABXWE2L7N7WZBUX337P2CQJP4BQY7RRQ, seg_pos={0 0}, upload=551850]: Done uploading piece. err: <nil> Dec 31 21:14:40.129878820 (629h45m40.000961855s) [noise=true , address=145.79.227.31:4797 , piece_id=RB72W6OIIKLEYX3SQL6UDWX4UWPZG2VM737PLBSGAVLJLARQUUOA, seg_pos={0 0}, upload=551850]: Done uploading piece. err: <nil> Dec 31 21:14:40.132780135 (629h45m40.00386317s) [noise=true , address=145.79.227.42:4121 , piece_id=GW5MIKYMWQXWNUQUYJQVETS4CMT6SURPZP65L7ZQVHPEZE7JBQPA, seg_pos={0 0}, upload=551850]: Done uploading piece. err: <nil> Dec 31 21:14:40.134835318 (629h45m40.005918363s) [noise=true , address=145.79.227.10:4403 , piece_id=4V6UX343EGRGSXDXSNBXOVVFLVI4PCHG5AVAJXD2G3Q6KH4QJZUQ, seg_pos={0 0}, upload=551850]: Done uploading piece. err: <nil> Dec 31 21:14:40.144302936 (629h45m40.015385971s) [noise=true , address=145.79.227.6:4259 , piece_id=LHRKVVGMVV6JEK2N2IBPCWMDEJBUP6JXM43SIK3A6B6YBP4WQD6A, seg_pos={0 0}, upload=551850]: Done uploading piece. err: <nil> Dec 31 21:14:40.149179442 (629h45m40.020262477s) [noise=true , address=145.79.227.3:4150 , piece_id=PXZFPQMFBLDF4C5MWGTAVEKJ3VJTVBELXKBTON3ORGYGLU3HSRPA, seg_pos={0 0}, upload=551850]: Done uploading piece. err: <nil> Dec 31 21:14:40.166909511 (629h45m40.037992557s) [noise=true , address=145.79.227.37:4567 , piece_id=XVLBULSM5YLXD565X7YLF36BQGMQGSDV5GYQ7IMAAGZMVYJN4K4A, seg_pos={0 0}, upload=551850]: Done uploading piece. err: <nil> Dec 31 21:14:40.282637284 (629h45m40.153720329s) [noise=true , address=145.79.227.17:4972 , piece_id=3JJVCXVXCQYL5DIZBSMJYWRABIJTR55WNDN7K4BSSVAM4LO3QYIA, seg_pos={0 0}, upload=551850]: Done uploading piece. err: <nil> Dec 31 21:14:40.452166871 (629h45m40.323249906s) [noise=true , address=45.91.116.7:4010 , piece_id=5W7QIZHMICDRAAY7VCLADKUJVK7UJIBYKXYCVAD4W5GHMUJRBLGQ, seg_pos={0 0}, upload=551850]: Done uploading piece. err: <nil> Dec 31 21:14:40.452845433 (629h45m40.323928508s) [noise=true , address=145.79.227.19:4078 , piece_id=MJKOSKIVDUNCV3VP3Y2TXU2QJ7KZHSNHEXVVRQ33I3CYZ3MFCN5Q, seg_pos={0 0}, upload=551850]: Done uploading piece. err: <nil> Dec 31 21:14:40.534638859 (629h45m40.405721894s) [noise=true , address=45.91.116.4:4940 , piece_id=AYPGEB6DE4ASXTRRYIBUVQ6SELL6XKTU5TU5HEHIWONAQOKRQV7A, seg_pos={0 0}, upload=551850]: Done uploading piece. err: <nil> Dec 31 21:14:41.313362154 (629h45m41.184445189s) [noise=true , address=45.91.116.6:4973 , piece_id=FX64BIWNBLRI5TKUJFMP5FFJAJHSJIQ34RKD7IRXDAWOMNWRBXDA, seg_pos={0 0}, upload=551850]: Done uploading piece. err: <nil> Dec 31 21:14:41.313381652 (629h45m41.184464697s) [seg_pos={0 0}, upload=551850]: Segment reached optimal threshold of 13 pieces. Dec 31 21:14:41.313560312 (629h45m41.184643347s) [noise=true , address=145.79.227.35:4481 , piece_id=FW5KOVEAH7C6OSHWY5WTNXZA6ZFQUZW7MLF2QCQHTPHS6IT4KWYQ, seg_pos={0 0}, upload=551850]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 31 21:14:41.313568167 (629h45m41.184651202s) [noise=true , address=145.79.227.35:4481 , piece_id=FW5KOVEAH7C6OSHWY5WTNXZA6ZFQUZW7MLF2QCQHTPHS6IT4KWYQ, seg_pos={0 0}, upload=551850]: Long tail cancellation due to optimal threshold Dec 31 21:14:41.313712041 (629h45m41.184795086s) [noise=true , address=45.91.116.5:4298 , piece_id=L7RM6HIQSBECVWUCJ7FJOUTYVRH2W3D2QDKH4BBK65SGJBM5WKRQ, seg_pos={0 0}, upload=551850]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: write tcp 188.166.225.170:47426->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:47426->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47426->45.91.116.5:4298: use of closed network connection Dec 31 21:14:41.313717492 (629h45m41.184800537s) [noise=true , address=45.91.116.5:4298 , piece_id=L7RM6HIQSBECVWUCJ7FJOUTYVRH2W3D2QDKH4BBK65SGJBM5WKRQ, seg_pos={0 0}, upload=551850]: Long tail cancellation due to optimal threshold Dec 31 21:14:41.313870343 (629h45m41.184953388s) [noise=true , address=145.79.227.43:4158 , piece_id=ZGXPGLZ3IALYSUQXQ76Z5AMUMKER5F34QVXQQYY7UTJVR2JM2SQQ, seg_pos={0 0}, upload=551850]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2): protocol: expected piece hash; context canceled; EOF Dec 31 21:14:41.313909527 (629h45m41.184992562s) [noise=true , address=145.79.227.43:4158 , piece_id=ZGXPGLZ3IALYSUQXQ76Z5AMUMKER5F34QVXQQYY7UTJVR2JM2SQQ, seg_pos={0 0}, upload=551850]: Long tail cancellation due to optimal threshold Dec 31 21:14:41.313940516 (629h45m41.185023551s) [seg_pos={0 0}, upload=551850]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 21:14:41.314094719 (629h45m41.185177764s) [upload=551850]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 21:14:41.314102986 (629h45m41.185186021s) [upload=551850]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 21:14:41.607320391 (629h45m41.478403426s) [upload=551850]: Done uploading segments... Dec 31 21:14:41.607338737 (629h45m41.478421782s) [upload=551850]: Upload finished. err: <nil> Duration: 30.003251282s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [XX...X...XX....X......X....XX........X...X.......X.....X...X] duration: 30.003120996s trace: https://jaeger.dev.storj.io/trace/53564573626f28bc FAIL (2025-12-31 21:14:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: PO6C5FNRD3REVNRFQ7OOAASCMF6PLPPGIEFOHMTD3ESBW4EY5AUA) context deadline exceeded; piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: KYX5KNEYLXZ4USMI2GM6T4HSJG7IVXPWKNVDFJQG5KYQUBYJIC7A) context deadline exceeded; piecestore: (Node ID: 12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, Piece ID: AANNLXCCEIC2Q22EWUMOUY2Y7BKIPD6EQSEUP7THZH2D2MSBGZZQ) context deadline exceeded ======LOGS====== pb.BatchRequestItem_ObjectBegin Dec 31 21:14:01.808117542 (629h45m1.679200577s) [upload=551842]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 21:14:02.058076919 (629h45m1.929159964s) [seg_pos={0 0}, upload=551842]: Begin upload segment... Dec 31 21:14:02.058130833 (629h45m1.929213878s) [seg_pos={0 0}, upload=551842]: Done begin upload segment. Dec 31 21:14:02.058206496 (629h45m1.929289541s) [noise=true , address=77.90.31.21:4878 , piece_id=STRBDLLC3JEACN2P7FJ5IF2JC7YPHJ4XYR6H7UPRHRAGHEJHHT3A, seg_pos={0 0}, upload=551842]: Uploading piece... Dec 31 21:14:02.058272362 (629h45m1.929355407s) [noise=true , address=77.90.31.11:4555 , piece_id=K4R5PFMB7IK5ZLS4IWQYTONI7V7UK3E4ZNIWTYP3POZVSXEQREWQ, seg_pos={0 0}, upload=551842]: Uploading piece... Dec 31 21:14:02.058982814 (629h45m1.930065859s) [noise=true , address=45.91.116.13:4153 , piece_id=KYX5KNEYLXZ4USMI2GM6T4HSJG7IVXPWKNVDFJQG5KYQUBYJIC7A, seg_pos={0 0}, upload=551842]: Uploading piece... Dec 31 21:14:02.059559803 (629h45m1.930642849s) [noise=true , address=77.90.31.14:4661 , piece_id=AANNLXCCEIC2Q22EWUMOUY2Y7BKIPD6EQSEUP7THZH2D2MSBGZZQ, seg_pos={0 0}, upload=551842]: Uploading piece... Dec 31 21:14:02.060200192 (629h45m1.931283228s) [noise=true , address=45.91.116.9:4261 , piece_id=Z534MHID3Y2PKONRGIXNA66BAPCTTBT6X5YGHE2YAWYGYNJFTGXA, seg_pos={0 0}, upload=551842]: Uploading piece... Dec 31 21:14:02.060848236 (629h45m1.931931281s) [noise=true , address=77.90.31.17:4773 , piece_id=SP465HOXFJW5MXTASK3BVQCEBHPVLYX6RRYXX3CIW6JELMFYD42Q, seg_pos={0 0}, upload=551842]: Uploading piece... Dec 31 21:14:02.061308603 (629h45m1.932391648s) [noise=true , address=145.79.227.41:4086 , piece_id=XBHQFFVWHGFHDUEKONQ7WDJHHBACWEZU4DUARXXGA55MAGSKTJRQ, seg_pos={0 0}, upload=551842]: Uploading piece... Dec 31 21:14:02.061556245 (629h45m1.9326393s) [noise=true , address=45.91.116.8:4046 , piece_id=MXSEWQQEIOBZ5VLTXIKPWR4CXP4AUHZV6X4WKJWHP3ATVWZ5MV4Q, seg_pos={0 0}, upload=551842]: Uploading piece... Dec 31 21:14:02.062708328 (629h45m1.933791373s) [noise=true , address=45.91.116.14:4190 , piece_id=F3QZBEXXYYYCXRM2RT4YVXNFPH3NZ7FGR42Y2O7RWGUWWEDLBUYQ, seg_pos={0 0}, upload=551842]: Uploading piece... Dec 31 21:14:02.063501679 (629h45m1.934584724s) [noise=true , address=145.79.227.3:4150 , piece_id=3QNAMBKPYOLGMXJKY4VVUNZDBLL4KKSW2LLGH5HXBYC63NQU4JCA, seg_pos={0 0}, upload=551842]: Uploading piece... Dec 31 21:14:02.064533934 (629h45m1.935616979s) [noise=true , address=77.90.31.13:4626 , piece_id=PPH2FIMGOJJR2MQJ4L64MFZSTQQEDQUTKKAEZRFDMWOEAHETOSYQ, seg_pos={0 0}, upload=551842]: Uploading piece... Dec 31 21:14:02.065009941 (629h45m1.936092986s) [noise=true , address=145.79.227.24:4583 , piece_id=2PXRJWY5WVKYB4X3LSNLDY6CMPNI7ZCRHKUN553KHGCBPN4Y26WQ, seg_pos={0 0}, upload=551842]: Uploading piece... Dec 31 21:14:02.065492480 (629h45m1.936575525s) [noise=true , address=45.91.116.7:4012 , piece_id=XRUKC5SLK5WY7ZBVYJTG2QNO2672S2IBGKQT5C23TU3KBQ3HKMSA, seg_pos={0 0}, upload=551842]: Uploading piece... Dec 31 21:14:02.065963406 (629h45m1.937046451s) [noise=true , address=45.91.116.10:4083 , piece_id=PO6C5FNRD3REVNRFQ7OOAASCMF6PLPPGIEFOHMTD3ESBW4EY5AUA, seg_pos={0 0}, upload=551842]: Uploading piece... Dec 31 21:14:02.067008256 (629h45m1.938091301s) [noise=true , address=45.91.116.4:4937 , piece_id=FRC7B4V2A6K5RRAKH6ZOR7OHU2WK7Z77US5UXXZGRPEYQGJQF6UA, seg_pos={0 0}, upload=551842]: Uploading piece... Dec 31 21:14:02.068310946 (629h45m1.939393981s) [noise=true , address=145.79.227.39:4015 , piece_id=FEJDLPICS7RU4SMRGMNJC6M5GXEIZWJPJAEHYHYN7CV7E4IE63MA, seg_pos={0 0}, upload=551842]: Uploading piece... Dec 31 21:14:03.778467322 (629h45m3.649550377s) [upload=551842]: Next returned nil segment Dec 31 21:14:03.778484995 (629h45m3.64956806s) [upload=551842]: Waiting for error group managing segments... Dec 31 21:14:04.050910271 (629h45m3.921993316s) [noise=true , address=145.79.227.41:4086 , piece_id=XBHQFFVWHGFHDUEKONQ7WDJHHBACWEZU4DUARXXGA55MAGSKTJRQ, seg_pos={0 0}, upload=551842]: Done uploading piece. err: <nil> Dec 31 21:14:04.061947116 (629h45m3.933030161s) [noise=true , address=145.79.227.39:4015 , piece_id=FEJDLPICS7RU4SMRGMNJC6M5GXEIZWJPJAEHYHYN7CV7E4IE63MA, seg_pos={0 0}, upload=551842]: Done uploading piece. err: <nil> Dec 31 21:14:04.068309751 (629h45m3.939392806s) [noise=true , address=145.79.227.3:4150 , piece_id=3QNAMBKPYOLGMXJKY4VVUNZDBLL4KKSW2LLGH5HXBYC63NQU4JCA, seg_pos={0 0}, upload=551842]: Done uploading piece. err: <nil> Dec 31 21:14:04.070704612 (629h45m3.941787657s) [noise=true , address=145.79.227.24:4583 , piece_id=2PXRJWY5WVKYB4X3LSNLDY6CMPNI7ZCRHKUN553KHGCBPN4Y26WQ, seg_pos={0 0}, upload=551842]: Done uploading piece. err: <nil> Dec 31 21:14:04.346492149 (629h45m4.217575184s) [noise=true , address=77.90.31.14:4661 , piece_id=AANNLXCCEIC2Q22EWUMOUY2Y7BKIPD6EQSEUP7THZH2D2MSBGZZQ, seg_pos={0 0}, upload=551842]: Done uploading piece. err: <nil> Dec 31 21:14:04.349013510 (629h45m4.220096555s) [noise=true , address=77.90.31.13:4626 , piece_id=PPH2FIMGOJJR2MQJ4L64MFZSTQQEDQUTKKAEZRFDMWOEAHETOSYQ, seg_pos={0 0}, upload=551842]: Done uploading piece. err: <nil> Dec 31 21:14:04.381309456 (629h45m4.252392491s) [noise=true , address=45.91.116.13:4153 , piece_id=KYX5KNEYLXZ4USMI2GM6T4HSJG7IVXPWKNVDFJQG5KYQUBYJIC7A, seg_pos={0 0}, upload=551842]: Done uploading piece. err: <nil> Dec 31 21:14:04.384497146 (629h45m4.255580181s) [noise=true , address=45.91.116.8:4046 , piece_id=MXSEWQQEIOBZ5VLTXIKPWR4CXP4AUHZV6X4WKJWHP3ATVWZ5MV4Q, seg_pos={0 0}, upload=551842]: Done uploading piece. err: <nil> Dec 31 21:14:04.416441873 (629h45m4.287524918s) [noise=true , address=77.90.31.17:4773 , piece_id=SP465HOXFJW5MXTASK3BVQCEBHPVLYX6RRYXX3CIW6JELMFYD42Q, seg_pos={0 0}, upload=551842]: Done uploading piece. err: <nil> Dec 31 21:14:04.440523408 (629h45m4.311606453s) [noise=true , address=45.91.116.10:4083 , piece_id=PO6C5FNRD3REVNRFQ7OOAASCMF6PLPPGIEFOHMTD3ESBW4EY5AUA, seg_pos={0 0}, upload=551842]: Done uploading piece. err: <nil> Dec 31 21:14:04.463283927 (629h45m4.334366972s) [noise=true , address=45.91.116.14:4190 , piece_id=F3QZBEXXYYYCXRM2RT4YVXNFPH3NZ7FGR42Y2O7RWGUWWEDLBUYQ, seg_pos={0 0}, upload=551842]: Done uploading piece. err: <nil> Dec 31 21:14:04.481880818 (629h45m4.352963863s) [noise=true , address=45.91.116.4:4937 , piece_id=FRC7B4V2A6K5RRAKH6ZOR7OHU2WK7Z77US5UXXZGRPEYQGJQF6UA, seg_pos={0 0}, upload=551842]: Done uploading piece. err: <nil> Dec 31 21:14:04.527135850 (629h45m4.398218885s) [noise=true , address=45.91.116.7:4012 , piece_id=XRUKC5SLK5WY7ZBVYJTG2QNO2672S2IBGKQT5C23TU3KBQ3HKMSA, seg_pos={0 0}, upload=551842]: Done uploading piece. err: <nil> Dec 31 21:14:04.527155096 (629h45m4.398238141s) [seg_pos={0 0}, upload=551842]: Segment reached optimal threshold of 13 pieces. Dec 31 21:14:04.527363563 (629h45m4.398446598s) [noise=true , address=45.91.116.9:4261 , piece_id=Z534MHID3Y2PKONRGIXNA66BAPCTTBT6X5YGHE2YAWYGYNJFTGXA, seg_pos={0 0}, upload=551842]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: write tcp 188.166.225.170:59616->45.91.116.9:4261: use of closed network connection; write tcp 188.166.225.170:59616->45.91.116.9:4261: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59616->45.91.116.9:4261: use of closed network connection Dec 31 21:14:04.527374344 (629h45m4.398457379s) [noise=true , address=45.91.116.9:4261 , piece_id=Z534MHID3Y2PKONRGIXNA66BAPCTTBT6X5YGHE2YAWYGYNJFTGXA, seg_pos={0 0}, upload=551842]: Long tail cancellation due to optimal threshold Dec 31 21:14:04.527490946 (629h45m4.398573981s) [noise=true , address=77.90.31.11:4555 , piece_id=K4R5PFMB7IK5ZLS4IWQYTONI7V7UK3E4ZNIWTYP3POZVSXEQREWQ, seg_pos={0 0}, upload=551842]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: write tcp 188.166.225.170:38210->77.90.31.11:4555: use of closed network connection; write tcp 188.166.225.170:38210->77.90.31.11:4555: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38210->77.90.31.11:4555: use of closed network connection Dec 31 21:14:04.527540299 (629h45m4.398623345s) [noise=true , address=77.90.31.11:4555 , piece_id=K4R5PFMB7IK5ZLS4IWQYTONI7V7UK3E4ZNIWTYP3POZVSXEQREWQ, seg_pos={0 0}, upload=551842]: Long tail cancellation due to optimal threshold Dec 31 21:14:04.527552794 (629h45m4.398635829s) [noise=true , address=77.90.31.21:4878 , piece_id=STRBDLLC3JEACN2P7FJ5IF2JC7YPHJ4XYR6H7UPRHRAGHEJHHT3A, seg_pos={0 0}, upload=551842]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: write tcp 188.166.225.170:37860->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:37860->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37860->77.90.31.21:4878: use of closed network connection Dec 31 21:14:04.527560538 (629h45m4.398643573s) [noise=true , address=77.90.31.21:4878 , piece_id=STRBDLLC3JEACN2P7FJ5IF2JC7YPHJ4XYR6H7UPRHRAGHEJHHT3A, seg_pos={0 0}, upload=551842]: Long tail cancellation due to optimal threshold Dec 31 21:14:04.527699423 (629h45m4.398782458s) [seg_pos={0 0}, upload=551842]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 21:14:04.527796237 (629h45m4.398879272s) [upload=551842]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 21:14:04.527801778 (629h45m4.398884823s) [upload=551842]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 21:14:04.818222439 (629h45m4.689305484s) [upload=551842]: Done uploading segments... Dec 31 21:14:04.818247256 (629h45m4.689330291s) [upload=551842]: Upload finished. err: <nil> Duration:* 30.003218744s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.XX...X...XX....X......X....XX........X...X.......X.....X...] duration: 15.950077896s trace: https://jaeger.dev.storj.io/trace/2be50406c0e1a31 FAIL (2025-12-31 21:13:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 31 21:10:38.539111048 (629h41m38.410194083s) [upload=551793]: Starting upload Dec 31 21:10:38.539165561 (629h41m38.410248596s) [upload=551793]: Done starting upload Dec 31 21:10:38.539271082 (629h41m38.410354127s) [upload=551793]: Uploading segments... Dec 31 21:10:38.539381132 (629h41m38.410464167s) [upload=551793]: Got next segment. Inline: false Dec 31 21:10:38.539416439 (629h41m38.410499474s) [upload=551793]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 21:10:38.539421699 (629h41m38.410504734s) [upload=551793]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 21:10:38.782158354 (629h41m38.653241389s) [seg_pos={0 0}, upload=551793]: Begin upload segment... Dec 31 21:10:38.782219509 (629h41m38.653302554s) [seg_pos={0 0}, upload=551793]: Done begin upload segment. Dec 31 21:10:38.782255688 (629h41m38.653338723s) [noise=true , address=45.91.116.6:4974 , piece_id=MXDWBMTNXSN452GGJUARKRIXVFZCQPN4VDUPY5XGFTJIHSC5O5UA, seg_pos={0 0}, upload=551793]: Uploading piece... Dec 31 21:10:38.784440348 (629h41m38.655523393s) [noise=true , address=77.90.31.18:4805 , piece_id=AYGAKHIZPCFX4Q3WI2DPGVHGMQZ2SFBEIKX5P6YIP33TO3SH4QPQ, seg_pos={0 0}, upload=551793]: Uploading piece... Dec 31 21:10:38.785023359 (629h41m38.656106404s) [noise=true , address=77.90.31.10:4519 , piece_id=OIQHC5GAYIFZ6N5VSXVJOUR2OMHV64K4TEQTQTHLDWD42AHHVQQQ, seg_pos={0 0}, upload=551793]: Uploading piece... Dec 31 21:10:38.786556769 (629h41m38.657639814s) [noise=true , address=145.79.227.41:4087 , piece_id=UAPNDU4WQVPUTYB7GTIYQB34LN2GNXOI5AGLSR2XIOXOAKIAQYXQ, seg_pos={0 0}, upload=551793]: Uploading piece... Dec 31 21:10:38.788471094 (629h41m38.659554139s) [noise=true , address=45.91.116.14:4191 , piece_id=QDNXTHD3X4V726V5S37XTRKDCAQAUNRTCKJCBSQQO5LVEXU5MJOA, seg_pos={0 0}, upload=551793]: Uploading piece... Dec 31 21:10:38.790197210 (629h41m38.661280255s) [noise=true , address=145.79.227.48:4301 , piece_id=BOKQXXHCDMYAGDZ4N77EGTJKO45OUZKZ5TPXGRUXA2WKCXJEH5FA, seg_pos={0 0}, upload=551793]: Uploading piece... Dec 31 21:10:38.791443213 (629h41m38.662526258s) [noise=true , address=145.79.227.12:4765 , piece_id=PY2RWJA7UREBPZT4537X3B4U6JBJ7DTVHE6MTNMBRIJDUNB3D54A, seg_pos={0 0}, upload=551793]: Uploading piece... Dec 31 21:10:38.793751678 (629h41m38.664834723s) [noise=true , address=77.90.31.11:4556 , piece_id=WI2NNNMVJE7HVX452VGGX6IGF4VLL62T3A3OKYKRPK5Q7II32WDA, seg_pos={0 0}, upload=551793]: Uploading piece... Dec 31 21:10:38.795161944 (629h41m38.666244979s) [noise=true , address=145.79.227.10:4403 , piece_id=VZYQ7IIJYRZAG54MNYCNX7VKBCQKLL2VHCFZOX64ZN7AUREEPFHA, seg_pos={0 0}, upload=551793]: Uploading piece... Dec 31 21:10:38.796002505 (629h41m38.66708555s) [noise=true , address=145.79.227.21:4474 , piece_id=JRJHQWP4CVKNPRW7BSID6KBTCCFP4FJE5C6C2GXQ4MKBU7UWW66A, seg_pos={0 0}, upload=551793]: Uploading piece... Dec 31 21:10:38.797697161 (629h41m38.668780206s) [noise=true , address=45.91.116.9:4263 , piece_id=7AHUNLUOKI6HN7JEOPQ2M7VZO27LZVFWO6F4HKERWVVMVWPRJI6A, seg_pos={0 0}, upload=551793]: Uploading piece... Dec 31 21:10:38.798649816 (629h41m38.669732861s) [noise=true , address=145.79.227.34:4907 , piece_id=3NXLA3QDZOYKDUNNSO5NY3ZPDHAGQCQ6B6DRMZD4VZPY2T2Q5M7Q, seg_pos={0 0}, upload=551793]: Uploading piece... Dec 31 21:10:38.799914894 (629h41m38.670997939s) [noise=true , address=145.79.227.16:4911 , piece_id=F5LNTF3FOKVZDP5GZ3CBCQK242IMDVJRNKFA74SGOPN3DK36PUGQ, seg_pos={0 0}, upload=551793]: Uploading piece... Dec 31 21:10:38.802521698 (629h41m38.673604753s) [noise=true , address=145.79.227.8:4330 , piece_id=S6HTSKRP3XMPCAKDPLUD36Y5SGSV5QVQGMS4P573QBB2DBCKAOOA, seg_pos={0 0}, upload=551793]: Uploading piece... Dec 31 21:10:38.803720360 (629h41m38.674803395s) [noise=true , address=145.79.227.27:4691 , piece_id=CN3EKK7X63UVZ5U2WODEAUESRCGW7OZLLTI4MB77NZ5K6JQ4EHXQ, seg_pos={0 0}, upload=551793]: Uploading piece... Dec 31 21:10:38.804226144 (629h41m38.675309189s) [noise=true , address=145.79.227.46:4602 , piece_id=ZQBEU4DSSSNO2O5YBYOEI4QGINLSXL2ADEC3JPIGI3ZBGIPWY3YA, seg_pos={0 0}, upload=551793]: Uploading piece... Dec 31 21:10:40.497627021 (629h41m40.368710066s) [upload=551793]: Next returned nil segment Dec 31 21:10:40.497728604 (629h41m40.368811659s) [upload=551793]: Waiting for error group managing segments... Dec 31 21:10:40.780919309 (629h41m40.652002344s) [noise=true , address=145.79.227.12:4765 , piece_id=PY2RWJA7UREBPZT4537X3B4U6JBJ7DTVHE6MTNMBRIJDUNB3D54A, seg_pos={0 0}, upload=551793]: Done uploading piece. err: <nil> Dec 31 21:10:40.781103850 (629h41m40.652186895s) [noise=true , address=145.79.227.48:4301 , piece_id=BOKQXXHCDMYAGDZ4N77EGTJKO45OUZKZ5TPXGRUXA2WKCXJEH5FA, seg_pos={0 0}, upload=551793]: Done uploading piece. err: <nil> Dec 31 21:10:40.797111310 (629h41m40.668194345s) [noise=true , address=145.79.227.46:4602 , piece_id=ZQBEU4DSSSNO2O5YBYOEI4QGINLSXL2ADEC3JPIGI3ZBGIPWY3YA, seg_pos={0 0}, upload=551793]: Done uploading piece. err: <nil> Dec 31 21:10:40.798896910 (629h41m40.669979945s) [noise=true , address=145.79.227.41:4087 , piece_id=UAPNDU4WQVPUTYB7GTIYQB34LN2GNXOI5AGLSR2XIOXOAKIAQYXQ, seg_pos={0 0}, upload=551793]: Done uploading piece. err: <nil> Dec 31 21:10:40.813723702 (629h41m40.684806747s) [noise=true , address=145.79.227.8:4330 , piece_id=S6HTSKRP3XMPCAKDPLUD36Y5SGSV5QVQGMS4P573QBB2DBCKAOOA, seg_pos={0 0}, upload=551793]: Done uploading piece. err: <nil> Dec 31 21:10:40.864901692 (629h41m40.735984737s) [noise=true , address=145.79.227.10:4403 , piece_id=VZYQ7IIJYRZAG54MNYCNX7VKBCQKLL2VHCFZOX64ZN7AUREEPFHA, seg_pos={0 0}, upload=551793]: Done uploading piece. err: <nil> Dec 31 21:10:40.873151520 (629h41m40.744234565s) [noise=true , address=145.79.227.21:4474 , piece_id=JRJHQWP4CVKNPRW7BSID6KBTCCFP4FJE5C6C2GXQ4MKBU7UWW66A, seg_pos={0 0}, upload=551793]: Done uploading piece. err: <nil> Dec 31 21:10:40.883535743 (629h41m40.754618788s) [noise=true , address=145.79.227.27:4691 , piece_id=CN3EKK7X63UVZ5U2WODEAUESRCGW7OZLLTI4MB77NZ5K6JQ4EHXQ, seg_pos={0 0}, upload=551793]: Done uploading piece. err: <nil> Dec 31 21:10:40.946778376 (629h41m40.817861422s) [noise=true , address=77.90.31.10:4519 , piece_id=OIQHC5GAYIFZ6N5VSXVJOUR2OMHV64K4TEQTQTHLDWD42AHHVQQQ, seg_pos={0 0}, upload=551793]: Done uploading piece. err: <nil> Dec 31 21:10:41.005113936 (629h41m40.876196991s) [noise=true , address=145.79.227.16:4911 , piece_id=F5LNTF3FOKVZDP5GZ3CBCQK242IMDVJRNKFA74SGOPN3DK36PUGQ, seg_pos={0 0}, upload=551793]: Done uploading piece. err: <nil> Dec 31 21:10:41.024270231 (629h41m40.895353266s) [noise=true , address=77.90.31.18:4805 , piece_id=AYGAKHIZPCFX4Q3WI2DPGVHGMQZ2SFBEIKX5P6YIP33TO3SH4QPQ, seg_pos={0 0}, upload=551793]: Done uploading piece. err: <nil> Dec 31 21:10:41.068603429 (629h41m40.939686464s) [noise=true , address=45.91.116.14:4191 , piece_id=QDNXTHD3X4V726V5S37XTRKDCAQAUNRTCKJCBSQQO5LVEXU5MJOA, seg_pos={0 0}, upload=551793]: Done uploading piece. err: <nil> Dec 31 21:10:41.138866960 (629h41m41.009950005s) [noise=true , address=45.91.116.9:4263 , piece_id=7AHUNLUOKI6HN7JEOPQ2M7VZO27LZVFWO6F4HKERWVVMVWPRJI6A, seg_pos={0 0}, upload=551793]: Done uploading piece. err: <nil> Dec 31 21:10:41.138894823 (629h41m41.009977869s) [seg_pos={0 0}, upload=551793]: Segment reached optimal threshold of 13 pieces. Dec 31 21:10:41.139141343 (629h41m41.010224388s) [noise=true , address=77.90.31.11:4556 , piece_id=WI2NNNMVJE7HVX452VGGX6IGF4VLL62T3A3OKYKRPK5Q7II32WDA, seg_pos={0 0}, upload=551793]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF Dec 31 21:10:41.139153656 (629h41m41.010236721s) [noise=true , address=77.90.31.11:4556 , piece_id=WI2NNNMVJE7HVX452VGGX6IGF4VLL62T3A3OKYKRPK5Q7II32WDA, seg_pos={0 0}, upload=551793]: Long tail cancellation due to optimal threshold Dec 31 21:10:41.139317548 (629h41m41.010400583s) [noise=true , address=145.79.227.34:4907 , piece_id=3NXLA3QDZOYKDUNNSO5NY3ZPDHAGQCQ6B6DRMZD4VZPY2T2Q5M7Q, seg_pos={0 0}, upload=551793]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 31 21:10:41.139340442 (629h41m41.010423487s) [noise=true , address=145.79.227.34:4907 , piece_id=3NXLA3QDZOYKDUNNSO5NY3ZPDHAGQCQ6B6DRMZD4VZPY2T2Q5M7Q, seg_pos={0 0}, upload=551793]: Long tail cancellation due to optimal threshold Dec 31 21:10:41.139482453 (629h41m41.010565498s) [noise=true , address=45.91.116.6:4974 , piece_id=MXDWBMTNXSN452GGJUARKRIXVFZCQPN4VDUPY5XGFTJIHSC5O5UA, seg_pos={0 0}, upload=551793]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Dec 31 21:10:41.139492521 (629h41m41.010575567s) [noise=true , address=45.91.116.6:4974 , piece_id=MXDWBMTNXSN452GGJUARKRIXVFZCQPN4VDUPY5XGFTJIHSC5O5UA, seg_pos={0 0}, upload=551793]: Long tail cancellation due to optimal threshold Dec 31 21:10:41.139516197 (629h41m41.010599232s) [seg_pos={0 0}, upload=551793]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 21:10:41.139601168 (629h41m41.010684213s) [upload=551793]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 21:10:41.139607801 (629h41m41.010690846s) [upload=551793]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 21:10:41.427824646 (629h41m41.298907681s) [upload=551793]: Done uploading segments... Dec 31 21:10:41.427872106 (629h41m41.298955141s) [upload=551793]: Upload finished. err: <nil> Duration: 15.950111089s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [..XX...X...XX....X......X....XX........X...X.......X.....X..] duration: 6.112547117s trace: https://jaeger.dev.storj.io/trace/2be50406c0e1a31 FAIL (2025-12-31 21:12:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 31 21:10:38.539111048 (629h41m38.410194083s) [upload=551793]: Starting upload Dec 31 21:10:38.539165561 (629h41m38.410248596s) [upload=551793]: Done starting upload Dec 31 21:10:38.539271082 (629h41m38.410354127s) [upload=551793]: Uploading segments... Dec 31 21:10:38.539381132 (629h41m38.410464167s) [upload=551793]: Got next segment. Inline: false Dec 31 21:10:38.539416439 (629h41m38.410499474s) [upload=551793]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 21:10:38.539421699 (629h41m38.410504734s) [upload=551793]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 21:10:38.782158354 (629h41m38.653241389s) [seg_pos={0 0}, upload=551793]: Begin upload segment... Dec 31 21:10:38.782219509 (629h41m38.653302554s) [seg_pos={0 0}, upload=551793]: Done begin upload segment. Dec 31 21:10:38.782255688 (629h41m38.653338723s) [noise=true , address=45.91.116.6:4974 , piece_id=MXDWBMTNXSN452GGJUARKRIXVFZCQPN4VDUPY5XGFTJIHSC5O5UA, seg_pos={0 0}, upload=551793]: Uploading piece... Dec 31 21:10:38.784440348 (629h41m38.655523393s) [noise=true , address=77.90.31.18:4805 , piece_id=AYGAKHIZPCFX4Q3WI2DPGVHGMQZ2SFBEIKX5P6YIP33TO3SH4QPQ, seg_pos={0 0}, upload=551793]: Uploading piece... Dec 31 21:10:38.785023359 (629h41m38.656106404s) [noise=true , address=77.90.31.10:4519 , piece_id=OIQHC5GAYIFZ6N5VSXVJOUR2OMHV64K4TEQTQTHLDWD42AHHVQQQ, seg_pos={0 0}, upload=551793]: Uploading piece... Dec 31 21:10:38.786556769 (629h41m38.657639814s) [noise=true , address=145.79.227.41:4087 , piece_id=UAPNDU4WQVPUTYB7GTIYQB34LN2GNXOI5AGLSR2XIOXOAKIAQYXQ, seg_pos={0 0}, upload=551793]: Uploading piece... Dec 31 21:10:38.788471094 (629h41m38.659554139s) [noise=true , address=45.91.116.14:4191 , piece_id=QDNXTHD3X4V726V5S37XTRKDCAQAUNRTCKJCBSQQO5LVEXU5MJOA, seg_pos={0 0}, upload=551793]: Uploading piece... Dec 31 21:10:38.790197210 (629h41m38.661280255s) [noise=true , address=145.79.227.48:4301 , piece_id=BOKQXXHCDMYAGDZ4N77EGTJKO45OUZKZ5TPXGRUXA2WKCXJEH5FA, seg_pos={0 0}, upload=551793]: Uploading piece... Dec 31 21:10:38.791443213 (629h41m38.662526258s) [noise=true , address=145.79.227.12:4765 , piece_id=PY2RWJA7UREBPZT4537X3B4U6JBJ7DTVHE6MTNMBRIJDUNB3D54A, seg_pos={0 0}, upload=551793]: Uploading piece... Dec 31 21:10:38.793751678 (629h41m38.664834723s) [noise=true , address=77.90.31.11:4556 , piece_id=WI2NNNMVJE7HVX452VGGX6IGF4VLL62T3A3OKYKRPK5Q7II32WDA, seg_pos={0 0}, upload=551793]: Uploading piece... Dec 31 21:10:38.795161944 (629h41m38.666244979s) [noise=true , address=145.79.227.10:4403 , piece_id=VZYQ7IIJYRZAG54MNYCNX7VKBCQKLL2VHCFZOX64ZN7AUREEPFHA, seg_pos={0 0}, upload=551793]: Uploading piece... Dec 31 21:10:38.796002505 (629h41m38.66708555s) [noise=true , address=145.79.227.21:4474 , piece_id=JRJHQWP4CVKNPRW7BSID6KBTCCFP4FJE5C6C2GXQ4MKBU7UWW66A, seg_pos={0 0}, upload=551793]: Uploading piece... Dec 31 21:10:38.797697161 (629h41m38.668780206s) [noise=true , address=45.91.116.9:4263 , piece_id=7AHUNLUOKI6HN7JEOPQ2M7VZO27LZVFWO6F4HKERWVVMVWPRJI6A, seg_pos={0 0}, upload=551793]: Uploading piece... Dec 31 21:10:38.798649816 (629h41m38.669732861s) [noise=true , address=145.79.227.34:4907 , piece_id=3NXLA3QDZOYKDUNNSO5NY3ZPDHAGQCQ6B6DRMZD4VZPY2T2Q5M7Q, seg_pos={0 0}, upload=551793]: Uploading piece... Dec 31 21:10:38.799914894 (629h41m38.670997939s) [noise=true , address=145.79.227.16:4911 , piece_id=F5LNTF3FOKVZDP5GZ3CBCQK242IMDVJRNKFA74SGOPN3DK36PUGQ, seg_pos={0 0}, upload=551793]: Uploading piece... Dec 31 21:10:38.802521698 (629h41m38.673604753s) [noise=true , address=145.79.227.8:4330 , piece_id=S6HTSKRP3XMPCAKDPLUD36Y5SGSV5QVQGMS4P573QBB2DBCKAOOA, seg_pos={0 0}, upload=551793]: Uploading piece... Dec 31 21:10:38.803720360 (629h41m38.674803395s) [noise=true , address=145.79.227.27:4691 , piece_id=CN3EKK7X63UVZ5U2WODEAUESRCGW7OZLLTI4MB77NZ5K6JQ4EHXQ, seg_pos={0 0}, upload=551793]: Uploading piece... Dec 31 21:10:38.804226144 (629h41m38.675309189s) [noise=true , address=145.79.227.46:4602 , piece_id=ZQBEU4DSSSNO2O5YBYOEI4QGINLSXL2ADEC3JPIGI3ZBGIPWY3YA, seg_pos={0 0}, upload=551793]: Uploading piece... Dec 31 21:10:40.497627021 (629h41m40.368710066s) [upload=551793]: Next returned nil segment Dec 31 21:10:40.497728604 (629h41m40.368811659s) [upload=551793]: Waiting for error group managing segments... Dec 31 21:10:40.780919309 (629h41m40.652002344s) [noise=true , address=145.79.227.12:4765 , piece_id=PY2RWJA7UREBPZT4537X3B4U6JBJ7DTVHE6MTNMBRIJDUNB3D54A, seg_pos={0 0}, upload=551793]: Done uploading piece. err: <nil> Dec 31 21:10:40.781103850 (629h41m40.652186895s) [noise=true , address=145.79.227.48:4301 , piece_id=BOKQXXHCDMYAGDZ4N77EGTJKO45OUZKZ5TPXGRUXA2WKCXJEH5FA, seg_pos={0 0}, upload=551793]: Done uploading piece. err: <nil> Dec 31 21:10:40.797111310 (629h41m40.668194345s) [noise=true , address=145.79.227.46:4602 , piece_id=ZQBEU4DSSSNO2O5YBYOEI4QGINLSXL2ADEC3JPIGI3ZBGIPWY3YA, seg_pos={0 0}, upload=551793]: Done uploading piece. err: <nil> Dec 31 21:10:40.798896910 (629h41m40.669979945s) [noise=true , address=145.79.227.41:4087 , piece_id=UAPNDU4WQVPUTYB7GTIYQB34LN2GNXOI5AGLSR2XIOXOAKIAQYXQ, seg_pos={0 0}, upload=551793]: Done uploading piece. err: <nil> Dec 31 21:10:40.813723702 (629h41m40.684806747s) [noise=true , address=145.79.227.8:4330 , piece_id=S6HTSKRP3XMPCAKDPLUD36Y5SGSV5QVQGMS4P573QBB2DBCKAOOA, seg_pos={0 0}, upload=551793]: Done uploading piece. err: <nil> Dec 31 21:10:40.864901692 (629h41m40.735984737s) [noise=true , address=145.79.227.10:4403 , piece_id=VZYQ7IIJYRZAG54MNYCNX7VKBCQKLL2VHCFZOX64ZN7AUREEPFHA, seg_pos={0 0}, upload=551793]: Done uploading piece. err: <nil> Dec 31 21:10:40.873151520 (629h41m40.744234565s) [noise=true , address=145.79.227.21:4474 , piece_id=JRJHQWP4CVKNPRW7BSID6KBTCCFP4FJE5C6C2GXQ4MKBU7UWW66A, seg_pos={0 0}, upload=551793]: Done uploading piece. err: <nil> Dec 31 21:10:40.883535743 (629h41m40.754618788s) [noise=true , address=145.79.227.27:4691 , piece_id=CN3EKK7X63UVZ5U2WODEAUESRCGW7OZLLTI4MB77NZ5K6JQ4EHXQ, seg_pos={0 0}, upload=551793]: Done uploading piece. err: <nil> Dec 31 21:10:40.946778376 (629h41m40.817861422s) [noise=true , address=77.90.31.10:4519 , piece_id=OIQHC5GAYIFZ6N5VSXVJOUR2OMHV64K4TEQTQTHLDWD42AHHVQQQ, seg_pos={0 0}, upload=551793]: Done uploading piece. err: <nil> Dec 31 21:10:41.005113936 (629h41m40.876196991s) [noise=true , address=145.79.227.16:4911 , piece_id=F5LNTF3FOKVZDP5GZ3CBCQK242IMDVJRNKFA74SGOPN3DK36PUGQ, seg_pos={0 0}, upload=551793]: Done uploading piece. err: <nil> Dec 31 21:10:41.024270231 (629h41m40.895353266s) [noise=true , address=77.90.31.18:4805 , piece_id=AYGAKHIZPCFX4Q3WI2DPGVHGMQZ2SFBEIKX5P6YIP33TO3SH4QPQ, seg_pos={0 0}, upload=551793]: Done uploading piece. err: <nil> Dec 31 21:10:41.068603429 (629h41m40.939686464s) [noise=true , address=45.91.116.14:4191 , piece_id=QDNXTHD3X4V726V5S37XTRKDCAQAUNRTCKJCBSQQO5LVEXU5MJOA, seg_pos={0 0}, upload=551793]: Done uploading piece. err: <nil> Dec 31 21:10:41.138866960 (629h41m41.009950005s) [noise=true , address=45.91.116.9:4263 , piece_id=7AHUNLUOKI6HN7JEOPQ2M7VZO27LZVFWO6F4HKERWVVMVWPRJI6A, seg_pos={0 0}, upload=551793]: Done uploading piece. err: <nil> Dec 31 21:10:41.138894823 (629h41m41.009977869s) [seg_pos={0 0}, upload=551793]: Segment reached optimal threshold of 13 pieces. Dec 31 21:10:41.139141343 (629h41m41.010224388s) [noise=true , address=77.90.31.11:4556 , piece_id=WI2NNNMVJE7HVX452VGGX6IGF4VLL62T3A3OKYKRPK5Q7II32WDA, seg_pos={0 0}, upload=551793]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF Dec 31 21:10:41.139153656 (629h41m41.010236721s) [noise=true , address=77.90.31.11:4556 , piece_id=WI2NNNMVJE7HVX452VGGX6IGF4VLL62T3A3OKYKRPK5Q7II32WDA, seg_pos={0 0}, upload=551793]: Long tail cancellation due to optimal threshold Dec 31 21:10:41.139317548 (629h41m41.010400583s) [noise=true , address=145.79.227.34:4907 , piece_id=3NXLA3QDZOYKDUNNSO5NY3ZPDHAGQCQ6B6DRMZD4VZPY2T2Q5M7Q, seg_pos={0 0}, upload=551793]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 31 21:10:41.139340442 (629h41m41.010423487s) [noise=true , address=145.79.227.34:4907 , piece_id=3NXLA3QDZOYKDUNNSO5NY3ZPDHAGQCQ6B6DRMZD4VZPY2T2Q5M7Q, seg_pos={0 0}, upload=551793]: Long tail cancellation due to optimal threshold Dec 31 21:10:41.139482453 (629h41m41.010565498s) [noise=true , address=45.91.116.6:4974 , piece_id=MXDWBMTNXSN452GGJUARKRIXVFZCQPN4VDUPY5XGFTJIHSC5O5UA, seg_pos={0 0}, upload=551793]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Dec 31 21:10:41.139492521 (629h41m41.010575567s) [noise=true , address=45.91.116.6:4974 , piece_id=MXDWBMTNXSN452GGJUARKRIXVFZCQPN4VDUPY5XGFTJIHSC5O5UA, seg_pos={0 0}, upload=551793]: Long tail cancellation due to optimal threshold Dec 31 21:10:41.139516197 (629h41m41.010599232s) [seg_pos={0 0}, upload=551793]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 21:10:41.139601168 (629h41m41.010684213s) [upload=551793]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 21:10:41.139607801 (629h41m41.010690846s) [upload=551793]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 21:10:41.427824646 (629h41m41.298907681s) [upload=551793]: Done uploading segments... Dec 31 21:10:41.427872106 (629h41m41.298955141s) [upload=551793]: Upload finished. err: <nil> Duration: 6.11259622s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [X..XX...X...XX....X......X....XX........X...X.......X.....X.] duration: 5.777472916s trace: https://jaeger.dev.storj.io/trace/2be50406c0e1a31 FAIL (2025-12-31 21:11:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 31 21:10:38.539111048 (629h41m38.410194083s) [upload=551793]: Starting upload Dec 31 21:10:38.539165561 (629h41m38.410248596s) [upload=551793]: Done starting upload Dec 31 21:10:38.539271082 (629h41m38.410354127s) [upload=551793]: Uploading segments... Dec 31 21:10:38.539381132 (629h41m38.410464167s) [upload=551793]: Got next segment. Inline: false Dec 31 21:10:38.539416439 (629h41m38.410499474s) [upload=551793]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 21:10:38.539421699 (629h41m38.410504734s) [upload=551793]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 21:10:38.782158354 (629h41m38.653241389s) [seg_pos={0 0}, upload=551793]: Begin upload segment... Dec 31 21:10:38.782219509 (629h41m38.653302554s) [seg_pos={0 0}, upload=551793]: Done begin upload segment. Dec 31 21:10:38.782255688 (629h41m38.653338723s) [noise=true , address=45.91.116.6:4974 , piece_id=MXDWBMTNXSN452GGJUARKRIXVFZCQPN4VDUPY5XGFTJIHSC5O5UA, seg_pos={0 0}, upload=551793]: Uploading piece... Dec 31 21:10:38.784440348 (629h41m38.655523393s) [noise=true , address=77.90.31.18:4805 , piece_id=AYGAKHIZPCFX4Q3WI2DPGVHGMQZ2SFBEIKX5P6YIP33TO3SH4QPQ, seg_pos={0 0}, upload=551793]: Uploading piece... Dec 31 21:10:38.785023359 (629h41m38.656106404s) [noise=true , address=77.90.31.10:4519 , piece_id=OIQHC5GAYIFZ6N5VSXVJOUR2OMHV64K4TEQTQTHLDWD42AHHVQQQ, seg_pos={0 0}, upload=551793]: Uploading piece... Dec 31 21:10:38.786556769 (629h41m38.657639814s) [noise=true , address=145.79.227.41:4087 , piece_id=UAPNDU4WQVPUTYB7GTIYQB34LN2GNXOI5AGLSR2XIOXOAKIAQYXQ, seg_pos={0 0}, upload=551793]: Uploading piece... Dec 31 21:10:38.788471094 (629h41m38.659554139s) [noise=true , address=45.91.116.14:4191 , piece_id=QDNXTHD3X4V726V5S37XTRKDCAQAUNRTCKJCBSQQO5LVEXU5MJOA, seg_pos={0 0}, upload=551793]: Uploading piece... Dec 31 21:10:38.790197210 (629h41m38.661280255s) [noise=true , address=145.79.227.48:4301 , piece_id=BOKQXXHCDMYAGDZ4N77EGTJKO45OUZKZ5TPXGRUXA2WKCXJEH5FA, seg_pos={0 0}, upload=551793]: Uploading piece... Dec 31 21:10:38.791443213 (629h41m38.662526258s) [noise=true , address=145.79.227.12:4765 , piece_id=PY2RWJA7UREBPZT4537X3B4U6JBJ7DTVHE6MTNMBRIJDUNB3D54A, seg_pos={0 0}, upload=551793]: Uploading piece... Dec 31 21:10:38.793751678 (629h41m38.664834723s) [noise=true , address=77.90.31.11:4556 , piece_id=WI2NNNMVJE7HVX452VGGX6IGF4VLL62T3A3OKYKRPK5Q7II32WDA, seg_pos={0 0}, upload=551793]: Uploading piece... Dec 31 21:10:38.795161944 (629h41m38.666244979s) [noise=true , address=145.79.227.10:4403 , piece_id=VZYQ7IIJYRZAG54MNYCNX7VKBCQKLL2VHCFZOX64ZN7AUREEPFHA, seg_pos={0 0}, upload=551793]: Uploading piece... Dec 31 21:10:38.796002505 (629h41m38.66708555s) [noise=true , address=145.79.227.21:4474 , piece_id=JRJHQWP4CVKNPRW7BSID6KBTCCFP4FJE5C6C2GXQ4MKBU7UWW66A, seg_pos={0 0}, upload=551793]: Uploading piece... Dec 31 21:10:38.797697161 (629h41m38.668780206s) [noise=true , address=45.91.116.9:4263 , piece_id=7AHUNLUOKI6HN7JEOPQ2M7VZO27LZVFWO6F4HKERWVVMVWPRJI6A, seg_pos={0 0}, upload=551793]: Uploading piece... Dec 31 21:10:38.798649816 (629h41m38.669732861s) [noise=true , address=145.79.227.34:4907 , piece_id=3NXLA3QDZOYKDUNNSO5NY3ZPDHAGQCQ6B6DRMZD4VZPY2T2Q5M7Q, seg_pos={0 0}, upload=551793]: Uploading piece... Dec 31 21:10:38.799914894 (629h41m38.670997939s) [noise=true , address=145.79.227.16:4911 , piece_id=F5LNTF3FOKVZDP5GZ3CBCQK242IMDVJRNKFA74SGOPN3DK36PUGQ, seg_pos={0 0}, upload=551793]: Uploading piece... Dec 31 21:10:38.802521698 (629h41m38.673604753s) [noise=true , address=145.79.227.8:4330 , piece_id=S6HTSKRP3XMPCAKDPLUD36Y5SGSV5QVQGMS4P573QBB2DBCKAOOA, seg_pos={0 0}, upload=551793]: Uploading piece... Dec 31 21:10:38.803720360 (629h41m38.674803395s) [noise=true , address=145.79.227.27:4691 , piece_id=CN3EKK7X63UVZ5U2WODEAUESRCGW7OZLLTI4MB77NZ5K6JQ4EHXQ, seg_pos={0 0}, upload=551793]: Uploading piece... Dec 31 21:10:38.804226144 (629h41m38.675309189s) [noise=true , address=145.79.227.46:4602 , piece_id=ZQBEU4DSSSNO2O5YBYOEI4QGINLSXL2ADEC3JPIGI3ZBGIPWY3YA, seg_pos={0 0}, upload=551793]: Uploading piece... Dec 31 21:10:40.497627021 (629h41m40.368710066s) [upload=551793]: Next returned nil segment Dec 31 21:10:40.497728604 (629h41m40.368811659s) [upload=551793]: Waiting for error group managing segments... Dec 31 21:10:40.780919309 (629h41m40.652002344s) [noise=true , address=145.79.227.12:4765 , piece_id=PY2RWJA7UREBPZT4537X3B4U6JBJ7DTVHE6MTNMBRIJDUNB3D54A, seg_pos={0 0}, upload=551793]: Done uploading piece. err: <nil> Dec 31 21:10:40.781103850 (629h41m40.652186895s) [noise=true , address=145.79.227.48:4301 , piece_id=BOKQXXHCDMYAGDZ4N77EGTJKO45OUZKZ5TPXGRUXA2WKCXJEH5FA, seg_pos={0 0}, upload=551793]: Done uploading piece. err: <nil> Dec 31 21:10:40.797111310 (629h41m40.668194345s) [noise=true , address=145.79.227.46:4602 , piece_id=ZQBEU4DSSSNO2O5YBYOEI4QGINLSXL2ADEC3JPIGI3ZBGIPWY3YA, seg_pos={0 0}, upload=551793]: Done uploading piece. err: <nil> Dec 31 21:10:40.798896910 (629h41m40.669979945s) [noise=true , address=145.79.227.41:4087 , piece_id=UAPNDU4WQVPUTYB7GTIYQB34LN2GNXOI5AGLSR2XIOXOAKIAQYXQ, seg_pos={0 0}, upload=551793]: Done uploading piece. err: <nil> Dec 31 21:10:40.813723702 (629h41m40.684806747s) [noise=true , address=145.79.227.8:4330 , piece_id=S6HTSKRP3XMPCAKDPLUD36Y5SGSV5QVQGMS4P573QBB2DBCKAOOA, seg_pos={0 0}, upload=551793]: Done uploading piece. err: <nil> Dec 31 21:10:40.864901692 (629h41m40.735984737s) [noise=true , address=145.79.227.10:4403 , piece_id=VZYQ7IIJYRZAG54MNYCNX7VKBCQKLL2VHCFZOX64ZN7AUREEPFHA, seg_pos={0 0}, upload=551793]: Done uploading piece. err: <nil> Dec 31 21:10:40.873151520 (629h41m40.744234565s) [noise=true , address=145.79.227.21:4474 , piece_id=JRJHQWP4CVKNPRW7BSID6KBTCCFP4FJE5C6C2GXQ4MKBU7UWW66A, seg_pos={0 0}, upload=551793]: Done uploading piece. err: <nil> Dec 31 21:10:40.883535743 (629h41m40.754618788s) [noise=true , address=145.79.227.27:4691 , piece_id=CN3EKK7X63UVZ5U2WODEAUESRCGW7OZLLTI4MB77NZ5K6JQ4EHXQ, seg_pos={0 0}, upload=551793]: Done uploading piece. err: <nil> Dec 31 21:10:40.946778376 (629h41m40.817861422s) [noise=true , address=77.90.31.10:4519 , piece_id=OIQHC5GAYIFZ6N5VSXVJOUR2OMHV64K4TEQTQTHLDWD42AHHVQQQ, seg_pos={0 0}, upload=551793]: Done uploading piece. err: <nil> Dec 31 21:10:41.005113936 (629h41m40.876196991s) [noise=true , address=145.79.227.16:4911 , piece_id=F5LNTF3FOKVZDP5GZ3CBCQK242IMDVJRNKFA74SGOPN3DK36PUGQ, seg_pos={0 0}, upload=551793]: Done uploading piece. err: <nil> Dec 31 21:10:41.024270231 (629h41m40.895353266s) [noise=true , address=77.90.31.18:4805 , piece_id=AYGAKHIZPCFX4Q3WI2DPGVHGMQZ2SFBEIKX5P6YIP33TO3SH4QPQ, seg_pos={0 0}, upload=551793]: Done uploading piece. err: <nil> Dec 31 21:10:41.068603429 (629h41m40.939686464s) [noise=true , address=45.91.116.14:4191 , piece_id=QDNXTHD3X4V726V5S37XTRKDCAQAUNRTCKJCBSQQO5LVEXU5MJOA, seg_pos={0 0}, upload=551793]: Done uploading piece. err: <nil> Dec 31 21:10:41.138866960 (629h41m41.009950005s) [noise=true , address=45.91.116.9:4263 , piece_id=7AHUNLUOKI6HN7JEOPQ2M7VZO27LZVFWO6F4HKERWVVMVWPRJI6A, seg_pos={0 0}, upload=551793]: Done uploading piece. err: <nil> Dec 31 21:10:41.138894823 (629h41m41.009977869s) [seg_pos={0 0}, upload=551793]: Segment reached optimal threshold of 13 pieces. Dec 31 21:10:41.139141343 (629h41m41.010224388s) [noise=true , address=77.90.31.11:4556 , piece_id=WI2NNNMVJE7HVX452VGGX6IGF4VLL62T3A3OKYKRPK5Q7II32WDA, seg_pos={0 0}, upload=551793]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF Dec 31 21:10:41.139153656 (629h41m41.010236721s) [noise=true , address=77.90.31.11:4556 , piece_id=WI2NNNMVJE7HVX452VGGX6IGF4VLL62T3A3OKYKRPK5Q7II32WDA, seg_pos={0 0}, upload=551793]: Long tail cancellation due to optimal threshold Dec 31 21:10:41.139317548 (629h41m41.010400583s) [noise=true , address=145.79.227.34:4907 , piece_id=3NXLA3QDZOYKDUNNSO5NY3ZPDHAGQCQ6B6DRMZD4VZPY2T2Q5M7Q, seg_pos={0 0}, upload=551793]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 31 21:10:41.139340442 (629h41m41.010423487s) [noise=true , address=145.79.227.34:4907 , piece_id=3NXLA3QDZOYKDUNNSO5NY3ZPDHAGQCQ6B6DRMZD4VZPY2T2Q5M7Q, seg_pos={0 0}, upload=551793]: Long tail cancellation due to optimal threshold Dec 31 21:10:41.139482453 (629h41m41.010565498s) [noise=true , address=45.91.116.6:4974 , piece_id=MXDWBMTNXSN452GGJUARKRIXVFZCQPN4VDUPY5XGFTJIHSC5O5UA, seg_pos={0 0}, upload=551793]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Dec 31 21:10:41.139492521 (629h41m41.010575567s) [noise=true , address=45.91.116.6:4974 , piece_id=MXDWBMTNXSN452GGJUARKRIXVFZCQPN4VDUPY5XGFTJIHSC5O5UA, seg_pos={0 0}, upload=551793]: Long tail cancellation due to optimal threshold Dec 31 21:10:41.139516197 (629h41m41.010599232s) [seg_pos={0 0}, upload=551793]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 21:10:41.139601168 (629h41m41.010684213s) [upload=551793]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 21:10:41.139607801 (629h41m41.010690846s) [upload=551793]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 21:10:41.427824646 (629h41m41.298907681s) [upload=551793]: Done uploading segments... Dec 31 21:10:41.427872106 (629h41m41.298955141s) [upload=551793]: Upload finished. err: <nil> Duration: 5.777492724s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [.X..XX...X...XX....X......X....XX........X...X.......X.....X] duration: 30.002886701s trace: https://jaeger.dev.storj.io/trace/2be50406c0e1a31 FAIL (2025-12-31 21:11:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 31 21:10:38.539111048 (629h41m38.410194083s) [upload=551793]: Starting upload Dec 31 21:10:38.539165561 (629h41m38.410248596s) [upload=551793]: Done starting upload Dec 31 21:10:38.539271082 (629h41m38.410354127s) [upload=551793]: Uploading segments... Dec 31 21:10:38.539381132 (629h41m38.410464167s) [upload=551793]: Got next segment. Inline: false Dec 31 21:10:38.539416439 (629h41m38.410499474s) [upload=551793]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 21:10:38.539421699 (629h41m38.410504734s) [upload=551793]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 21:10:38.782158354 (629h41m38.653241389s) [seg_pos={0 0}, upload=551793]: Begin upload segment... Dec 31 21:10:38.782219509 (629h41m38.653302554s) [seg_pos={0 0}, upload=551793]: Done begin upload segment. Dec 31 21:10:38.782255688 (629h41m38.653338723s) [noise=true , address=45.91.116.6:4974 , piece_id=MXDWBMTNXSN452GGJUARKRIXVFZCQPN4VDUPY5XGFTJIHSC5O5UA, seg_pos={0 0}, upload=551793]: Uploading piece... Dec 31 21:10:38.784440348 (629h41m38.655523393s) [noise=true , address=77.90.31.18:4805 , piece_id=AYGAKHIZPCFX4Q3WI2DPGVHGMQZ2SFBEIKX5P6YIP33TO3SH4QPQ, seg_pos={0 0}, upload=551793]: Uploading piece... Dec 31 21:10:38.785023359 (629h41m38.656106404s) [noise=true , address=77.90.31.10:4519 , piece_id=OIQHC5GAYIFZ6N5VSXVJOUR2OMHV64K4TEQTQTHLDWD42AHHVQQQ, seg_pos={0 0}, upload=551793]: Uploading piece... Dec 31 21:10:38.786556769 (629h41m38.657639814s) [noise=true , address=145.79.227.41:4087 , piece_id=UAPNDU4WQVPUTYB7GTIYQB34LN2GNXOI5AGLSR2XIOXOAKIAQYXQ, seg_pos={0 0}, upload=551793]: Uploading piece... Dec 31 21:10:38.788471094 (629h41m38.659554139s) [noise=true , address=45.91.116.14:4191 , piece_id=QDNXTHD3X4V726V5S37XTRKDCAQAUNRTCKJCBSQQO5LVEXU5MJOA, seg_pos={0 0}, upload=551793]: Uploading piece... Dec 31 21:10:38.790197210 (629h41m38.661280255s) [noise=true , address=145.79.227.48:4301 , piece_id=BOKQXXHCDMYAGDZ4N77EGTJKO45OUZKZ5TPXGRUXA2WKCXJEH5FA, seg_pos={0 0}, upload=551793]: Uploading piece... Dec 31 21:10:38.791443213 (629h41m38.662526258s) [noise=true , address=145.79.227.12:4765 , piece_id=PY2RWJA7UREBPZT4537X3B4U6JBJ7DTVHE6MTNMBRIJDUNB3D54A, seg_pos={0 0}, upload=551793]: Uploading piece... Dec 31 21:10:38.793751678 (629h41m38.664834723s) [noise=true , address=77.90.31.11:4556 , piece_id=WI2NNNMVJE7HVX452VGGX6IGF4VLL62T3A3OKYKRPK5Q7II32WDA, seg_pos={0 0}, upload=551793]: Uploading piece... Dec 31 21:10:38.795161944 (629h41m38.666244979s) [noise=true , address=145.79.227.10:4403 , piece_id=VZYQ7IIJYRZAG54MNYCNX7VKBCQKLL2VHCFZOX64ZN7AUREEPFHA, seg_pos={0 0}, upload=551793]: Uploading piece... Dec 31 21:10:38.796002505 (629h41m38.66708555s) [noise=true , address=145.79.227.21:4474 , piece_id=JRJHQWP4CVKNPRW7BSID6KBTCCFP4FJE5C6C2GXQ4MKBU7UWW66A, seg_pos={0 0}, upload=551793]: Uploading piece... Dec 31 21:10:38.797697161 (629h41m38.668780206s) [noise=true , address=45.91.116.9:4263 , piece_id=7AHUNLUOKI6HN7JEOPQ2M7VZO27LZVFWO6F4HKERWVVMVWPRJI6A, seg_pos={0 0}, upload=551793]: Uploading piece... Dec 31 21:10:38.798649816 (629h41m38.669732861s) [noise=true , address=145.79.227.34:4907 , piece_id=3NXLA3QDZOYKDUNNSO5NY3ZPDHAGQCQ6B6DRMZD4VZPY2T2Q5M7Q, seg_pos={0 0}, upload=551793]: Uploading piece... Dec 31 21:10:38.799914894 (629h41m38.670997939s) [noise=true , address=145.79.227.16:4911 , piece_id=F5LNTF3FOKVZDP5GZ3CBCQK242IMDVJRNKFA74SGOPN3DK36PUGQ, seg_pos={0 0}, upload=551793]: Uploading piece... Dec 31 21:10:38.802521698 (629h41m38.673604753s) [noise=true , address=145.79.227.8:4330 , piece_id=S6HTSKRP3XMPCAKDPLUD36Y5SGSV5QVQGMS4P573QBB2DBCKAOOA, seg_pos={0 0}, upload=551793]: Uploading piece... Dec 31 21:10:38.803720360 (629h41m38.674803395s) [noise=true , address=145.79.227.27:4691 , piece_id=CN3EKK7X63UVZ5U2WODEAUESRCGW7OZLLTI4MB77NZ5K6JQ4EHXQ, seg_pos={0 0}, upload=551793]: Uploading piece... Dec 31 21:10:38.804226144 (629h41m38.675309189s) [noise=true , address=145.79.227.46:4602 , piece_id=ZQBEU4DSSSNO2O5YBYOEI4QGINLSXL2ADEC3JPIGI3ZBGIPWY3YA, seg_pos={0 0}, upload=551793]: Uploading piece... Dec 31 21:10:40.497627021 (629h41m40.368710066s) [upload=551793]: Next returned nil segment Dec 31 21:10:40.497728604 (629h41m40.368811659s) [upload=551793]: Waiting for error group managing segments... Dec 31 21:10:40.780919309 (629h41m40.652002344s) [noise=true , address=145.79.227.12:4765 , piece_id=PY2RWJA7UREBPZT4537X3B4U6JBJ7DTVHE6MTNMBRIJDUNB3D54A, seg_pos={0 0}, upload=551793]: Done uploading piece. err: <nil> Dec 31 21:10:40.781103850 (629h41m40.652186895s) [noise=true , address=145.79.227.48:4301 , piece_id=BOKQXXHCDMYAGDZ4N77EGTJKO45OUZKZ5TPXGRUXA2WKCXJEH5FA, seg_pos={0 0}, upload=551793]: Done uploading piece. err: <nil> Dec 31 21:10:40.797111310 (629h41m40.668194345s) [noise=true , address=145.79.227.46:4602 , piece_id=ZQBEU4DSSSNO2O5YBYOEI4QGINLSXL2ADEC3JPIGI3ZBGIPWY3YA, seg_pos={0 0}, upload=551793]: Done uploading piece. err: <nil> Dec 31 21:10:40.798896910 (629h41m40.669979945s) [noise=true , address=145.79.227.41:4087 , piece_id=UAPNDU4WQVPUTYB7GTIYQB34LN2GNXOI5AGLSR2XIOXOAKIAQYXQ, seg_pos={0 0}, upload=551793]: Done uploading piece. err: <nil> Dec 31 21:10:40.813723702 (629h41m40.684806747s) [noise=true , address=145.79.227.8:4330 , piece_id=S6HTSKRP3XMPCAKDPLUD36Y5SGSV5QVQGMS4P573QBB2DBCKAOOA, seg_pos={0 0}, upload=551793]: Done uploading piece. err: <nil> Dec 31 21:10:40.864901692 (629h41m40.735984737s) [noise=true , address=145.79.227.10:4403 , piece_id=VZYQ7IIJYRZAG54MNYCNX7VKBCQKLL2VHCFZOX64ZN7AUREEPFHA, seg_pos={0 0}, upload=551793]: Done uploading piece. err: <nil> Dec 31 21:10:40.873151520 (629h41m40.744234565s) [noise=true , address=145.79.227.21:4474 , piece_id=JRJHQWP4CVKNPRW7BSID6KBTCCFP4FJE5C6C2GXQ4MKBU7UWW66A, seg_pos={0 0}, upload=551793]: Done uploading piece. err: <nil> Dec 31 21:10:40.883535743 (629h41m40.754618788s) [noise=true , address=145.79.227.27:4691 , piece_id=CN3EKK7X63UVZ5U2WODEAUESRCGW7OZLLTI4MB77NZ5K6JQ4EHXQ, seg_pos={0 0}, upload=551793]: Done uploading piece. err: <nil> Dec 31 21:10:40.946778376 (629h41m40.817861422s) [noise=true , address=77.90.31.10:4519 , piece_id=OIQHC5GAYIFZ6N5VSXVJOUR2OMHV64K4TEQTQTHLDWD42AHHVQQQ, seg_pos={0 0}, upload=551793]: Done uploading piece. err: <nil> Dec 31 21:10:41.005113936 (629h41m40.876196991s) [noise=true , address=145.79.227.16:4911 , piece_id=F5LNTF3FOKVZDP5GZ3CBCQK242IMDVJRNKFA74SGOPN3DK36PUGQ, seg_pos={0 0}, upload=551793]: Done uploading piece. err: <nil> Dec 31 21:10:41.024270231 (629h41m40.895353266s) [noise=true , address=77.90.31.18:4805 , piece_id=AYGAKHIZPCFX4Q3WI2DPGVHGMQZ2SFBEIKX5P6YIP33TO3SH4QPQ, seg_pos={0 0}, upload=551793]: Done uploading piece. err: <nil> Dec 31 21:10:41.068603429 (629h41m40.939686464s) [noise=true , address=45.91.116.14:4191 , piece_id=QDNXTHD3X4V726V5S37XTRKDCAQAUNRTCKJCBSQQO5LVEXU5MJOA, seg_pos={0 0}, upload=551793]: Done uploading piece. err: <nil> Dec 31 21:10:41.138866960 (629h41m41.009950005s) [noise=true , address=45.91.116.9:4263 , piece_id=7AHUNLUOKI6HN7JEOPQ2M7VZO27LZVFWO6F4HKERWVVMVWPRJI6A, seg_pos={0 0}, upload=551793]: Done uploading piece. err: <nil> Dec 31 21:10:41.138894823 (629h41m41.009977869s) [seg_pos={0 0}, upload=551793]: Segment reached optimal threshold of 13 pieces. Dec 31 21:10:41.139141343 (629h41m41.010224388s) [noise=true , address=77.90.31.11:4556 , piece_id=WI2NNNMVJE7HVX452VGGX6IGF4VLL62T3A3OKYKRPK5Q7II32WDA, seg_pos={0 0}, upload=551793]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF Dec 31 21:10:41.139153656 (629h41m41.010236721s) [noise=true , address=77.90.31.11:4556 , piece_id=WI2NNNMVJE7HVX452VGGX6IGF4VLL62T3A3OKYKRPK5Q7II32WDA, seg_pos={0 0}, upload=551793]: Long tail cancellation due to optimal threshold Dec 31 21:10:41.139317548 (629h41m41.010400583s) [noise=true , address=145.79.227.34:4907 , piece_id=3NXLA3QDZOYKDUNNSO5NY3ZPDHAGQCQ6B6DRMZD4VZPY2T2Q5M7Q, seg_pos={0 0}, upload=551793]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 31 21:10:41.139340442 (629h41m41.010423487s) [noise=true , address=145.79.227.34:4907 , piece_id=3NXLA3QDZOYKDUNNSO5NY3ZPDHAGQCQ6B6DRMZD4VZPY2T2Q5M7Q, seg_pos={0 0}, upload=551793]: Long tail cancellation due to optimal threshold Dec 31 21:10:41.139482453 (629h41m41.010565498s) [noise=true , address=45.91.116.6:4974 , piece_id=MXDWBMTNXSN452GGJUARKRIXVFZCQPN4VDUPY5XGFTJIHSC5O5UA, seg_pos={0 0}, upload=551793]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Dec 31 21:10:41.139492521 (629h41m41.010575567s) [noise=true , address=45.91.116.6:4974 , piece_id=MXDWBMTNXSN452GGJUARKRIXVFZCQPN4VDUPY5XGFTJIHSC5O5UA, seg_pos={0 0}, upload=551793]: Long tail cancellation due to optimal threshold Dec 31 21:10:41.139516197 (629h41m41.010599232s) [seg_pos={0 0}, upload=551793]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 21:10:41.139601168 (629h41m41.010684213s) [upload=551793]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 21:10:41.139607801 (629h41m41.010690846s) [upload=551793]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 21:10:41.427824646 (629h41m41.298907681s) [upload=551793]: Done uploading segments... Dec 31 21:10:41.427872106 (629h41m41.298955141s) [upload=551793]: Upload finished. err: <nil> Duration: 30.002906878s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [..X..XX...X...XX....X......X....XX........X...X.......X.....] duration: 5.953663115s trace: https://jaeger.dev.storj.io/trace/6b4a006b8b7aa49b FAIL (2025-12-31 21:10:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: VVVRZMWTKJ334XCSMYKQECFKFSSMFBRIKTCUJHQZUIFZNHA3ZT5Q) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: H4EO22GSLMHTDAOC4HXKEJ2ZRATC2GHH7YDLJ2PHRJ4FPYP2Q2FQ) context deadline exceeded ======LOGS====== Dec 31 21:06:19.130592534 (629h37m19.001675569s) [upload=551730]: Starting upload Dec 31 21:06:19.130672245 (629h37m19.00175529s) [upload=551730]: Done starting upload Dec 31 21:06:19.130755443 (629h37m19.001838488s) [upload=551730]: Uploading segments... Dec 31 21:06:19.130901432 (629h37m19.001984467s) [upload=551730]: Got next segment. Inline: false Dec 31 21:06:19.130950555 (629h37m19.0020336s) [upload=551730]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 21:06:19.130955284 (629h37m19.002038319s) [upload=551730]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 21:06:19.381665200 (629h37m19.252748265s) [seg_pos={0 0}, upload=551730]: Begin upload segment... Dec 31 21:06:19.381770831 (629h37m19.252853876s) [seg_pos={0 0}, upload=551730]: Done begin upload segment. Dec 31 21:06:19.381851134 (629h37m19.252934179s) [noise=true , address=145.79.227.21:4474 , piece_id=VVRXFDYZR2HSEYIHQ4KRR2TJ7MA45GLXXUAPRCU2QE47YXWP63RA, seg_pos={0 0}, upload=551730]: Uploading piece... Dec 31 21:06:19.385528566 (629h37m19.256611601s) [noise=true , address=45.91.116.8:4045 , piece_id=H4EO22GSLMHTDAOC4HXKEJ2ZRATC2GHH7YDLJ2PHRJ4FPYP2Q2FQ, seg_pos={0 0}, upload=551730]: Uploading piece... Dec 31 21:06:19.386657856 (629h37m19.257740891s) [noise=true , address=145.79.227.8:4331 , piece_id=VEIWQMSN2RVO3HWYKVPO2GRUQBDKOSCVGSLU3TGHCHC6YRXL4HAQ, seg_pos={0 0}, upload=551730]: Uploading piece... Dec 31 21:06:19.387654513 (629h37m19.258737549s) [noise=true , address=145.79.227.36:4519 , piece_id=OCJNGU7QE7TW3VBDQ53TLZXIUEQGBZFYQNDPCNAXX4HLFX2WQYZA, seg_pos={0 0}, upload=551730]: Uploading piece... Dec 31 21:06:19.388655089 (629h37m19.259738134s) [noise=true , address=145.79.227.27:4690 , piece_id=POOEPKVNRGSZ6QHOEVYUCOOUGQZCCPVJFZ5G5GVSZSE6JVC2QGWQ, seg_pos={0 0}, upload=551730]: Uploading piece... Dec 31 21:06:19.390073680 (629h37m19.261156716s) [noise=true , address=145.79.227.2:4115 , piece_id=ZH3P4BLJCINMEOAXB27FGCQHFJXX2AQK5GICCBCVXX4GFXYYTFVQ, seg_pos={0 0}, upload=551730]: Uploading piece... Dec 31 21:06:19.391092029 (629h37m19.262175074s) [noise=true , address=45.91.116.7:4011 , piece_id=OODB666DEWAW2CHBCHNETCEHMRLGNHPNGSZMKKWOA7K4TDRT6ZWA, seg_pos={0 0}, upload=551730]: Uploading piece... Dec 31 21:06:19.392162568 (629h37m19.263245603s) [noise=true , address=45.91.116.14:4191 , piece_id=XCCCHRXHATZGD2VUXYUMTFZTQ5FACPDX22UG3YSZUW4SJBCBXV5A, seg_pos={0 0}, upload=551730]: Uploading piece... Dec 31 21:06:19.393340190 (629h37m19.264423225s) [noise=true , address=145.79.227.17:4970 , piece_id=37X56AFL2F7EY77MOIAMIT7ZWFXHWNZVBK2WHQ5J55UFEQCQIFJQ, seg_pos={0 0}, upload=551730]: Uploading piece... Dec 31 21:06:19.395205221 (629h37m19.266288256s) [noise=true , address=45.91.116.4:4937 , piece_id=C6IM64R7L5IYDPISRRHSDUXD6TFMD6CTH3BNULTVQEFCBKJOGP3A, seg_pos={0 0}, upload=551730]: Uploading piece... Dec 31 21:06:19.396253277 (629h37m19.267336312s) [noise=true , address=45.91.116.10:4082 , piece_id=VVVRZMWTKJ334XCSMYKQECFKFSSMFBRIKTCUJHQZUIFZNHA3ZT5Q, seg_pos={0 0}, upload=551730]: Uploading piece... Dec 31 21:06:19.397275324 (629h37m19.268358359s) [noise=true , address=145.79.227.44:4193 , piece_id=3JQX6CG4J4QRQYNLVXWOCQXB4WZRR5OPJ5OBIT4JI3JAXJYK733Q, seg_pos={0 0}, upload=551730]: Uploading piece... Dec 31 21:06:19.398334380 (629h37m19.269417415s) [noise=true , address=145.79.227.22:4509 , piece_id=XLPZDZSJCBIRKX5LGIKLLP2CVHUM2X5SRBGNA6U2FGCDN2UXIPDQ, seg_pos={0 0}, upload=551730]: Uploading piece... Dec 31 21:06:19.400004671 (629h37m19.271087716s) [noise=true , address=145.79.227.6:4259 , piece_id=2LGQDOPBK4WUOOKTSFTZKHHLFZ3IMRBC2HCHFMCWOKETMZTFEAFA, seg_pos={0 0}, upload=551730]: Uploading piece... Dec 31 21:06:19.401527831 (629h37m19.272610876s) [noise=true , address=77.90.31.18:4808 , piece_id=4FAKZB3YUM4Y7PATZAWHD7SJOCTM4K7G75E7ARWKRRJUYNS3TZAQ, seg_pos={0 0}, upload=551730]: Uploading piece... Dec 31 21:06:19.402400904 (629h37m19.273483949s) [noise=true , address=145.79.227.37:4565 , piece_id=IAPFHAF5L2YZS2KYURPMYX3UT7FVXI7EKYX4QEJ7N7G7VOU3KTNA, seg_pos={0 0}, upload=551730]: Uploading piece... Dec 31 21:06:21.101007053 (629h37m20.972090098s) [upload=551730]: Next returned nil segment Dec 31 21:06:21.101022401 (629h37m20.972105436s) [upload=551730]: Waiting for error group managing segments... Dec 31 21:06:21.385360411 (629h37m21.256443446s) [noise=true , address=145.79.227.8:4331 , piece_id=VEIWQMSN2RVO3HWYKVPO2GRUQBDKOSCVGSLU3TGHCHC6YRXL4HAQ, seg_pos={0 0}, upload=551730]: Done uploading piece. err: <nil> Dec 31 21:06:21.386079980 (629h37m21.257163015s) [noise=true , address=145.79.227.21:4474 , piece_id=VVRXFDYZR2HSEYIHQ4KRR2TJ7MA45GLXXUAPRCU2QE47YXWP63RA, seg_pos={0 0}, upload=551730]: Done uploading piece. err: <nil> Dec 31 21:06:21.392677413 (629h37m21.263760478s) [noise=true , address=145.79.227.27:4690 , piece_id=POOEPKVNRGSZ6QHOEVYUCOOUGQZCCPVJFZ5G5GVSZSE6JVC2QGWQ, seg_pos={0 0}, upload=551730]: Done uploading piece. err: <nil> Dec 31 21:06:21.393084087 (629h37m21.264167122s) [noise=true , address=145.79.227.17:4970 , piece_id=37X56AFL2F7EY77MOIAMIT7ZWFXHWNZVBK2WHQ5J55UFEQCQIFJQ, seg_pos={0 0}, upload=551730]: Done uploading piece. err: <nil> Dec 31 21:06:21.393326629 (629h37m21.264409674s) [noise=true , address=145.79.227.6:4259 , piece_id=2LGQDOPBK4WUOOKTSFTZKHHLFZ3IMRBC2HCHFMCWOKETMZTFEAFA, seg_pos={0 0}, upload=551730]: Done uploading piece. err: <nil> Dec 31 21:06:21.404225282 (629h37m21.275308317s) [noise=true , address=145.79.227.37:4565 , piece_id=IAPFHAF5L2YZS2KYURPMYX3UT7FVXI7EKYX4QEJ7N7G7VOU3KTNA, seg_pos={0 0}, upload=551730]: Done uploading piece. err: <nil> Dec 31 21:06:21.407043318 (629h37m21.278126353s) [noise=true , address=145.79.227.36:4519 , piece_id=OCJNGU7QE7TW3VBDQ53TLZXIUEQGBZFYQNDPCNAXX4HLFX2WQYZA, seg_pos={0 0}, upload=551730]: Done uploading piece. err: <nil> Dec 31 21:06:21.411401707 (629h37m21.282484742s) [noise=true , address=145.79.227.2:4115 , piece_id=ZH3P4BLJCINMEOAXB27FGCQHFJXX2AQK5GICCBCVXX4GFXYYTFVQ, seg_pos={0 0}, upload=551730]: Done uploading piece. err: <nil> Dec 31 21:06:21.415461036 (629h37m21.286544081s) [noise=true , address=145.79.227.22:4509 , piece_id=XLPZDZSJCBIRKX5LGIKLLP2CVHUM2X5SRBGNA6U2FGCDN2UXIPDQ, seg_pos={0 0}, upload=551730]: Done uploading piece. err: <nil> Dec 31 21:06:21.556582891 (629h37m21.427665936s) [noise=true , address=77.90.31.18:4808 , piece_id=4FAKZB3YUM4Y7PATZAWHD7SJOCTM4K7G75E7ARWKRRJUYNS3TZAQ, seg_pos={0 0}, upload=551730]: Done uploading piece. err: <nil> Dec 31 21:06:21.721535795 (629h37m21.59261884s) [noise=true , address=45.91.116.8:4045 , piece_id=H4EO22GSLMHTDAOC4HXKEJ2ZRATC2GHH7YDLJ2PHRJ4FPYP2Q2FQ, seg_pos={0 0}, upload=551730]: Done uploading piece. err: <nil> Dec 31 21:06:21.775331980 (629h37m21.646415016s) [noise=true , address=45.91.116.10:4082 , piece_id=VVVRZMWTKJ334XCSMYKQECFKFSSMFBRIKTCUJHQZUIFZNHA3ZT5Q, seg_pos={0 0}, upload=551730]: Done uploading piece. err: <nil> Dec 31 21:06:21.807810725 (629h37m21.67889377s) [noise=true , address=45.91.116.4:4937 , piece_id=C6IM64R7L5IYDPISRRHSDUXD6TFMD6CTH3BNULTVQEFCBKJOGP3A, seg_pos={0 0}, upload=551730]: Done uploading piece. err: <nil> Dec 31 21:06:21.807850861 (629h37m21.678933916s) [seg_pos={0 0}, upload=551730]: Segment reached optimal threshold of 13 pieces. Dec 31 21:06:21.808089556 (629h37m21.679172601s) [noise=true , address=45.91.116.14:4191 , piece_id=XCCCHRXHATZGD2VUXYUMTFZTQ5FACPDX22UG3YSZUW4SJBCBXV5A, seg_pos={0 0}, upload=551730]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Dec 31 21:06:21.808097911 (629h37m21.679180966s) [noise=true , address=45.91.116.14:4191 , piece_id=XCCCHRXHATZGD2VUXYUMTFZTQ5FACPDX22UG3YSZUW4SJBCBXV5A, seg_pos={0 0}, upload=551730]: Long tail cancellation due to optimal threshold Dec 31 21:06:21.808156182 (629h37m21.679239228s) [noise=true , address=45.91.116.7:4011 , piece_id=OODB666DEWAW2CHBCHNETCEHMRLGNHPNGSZMKKWOA7K4TDRT6ZWA, seg_pos={0 0}, upload=551730]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 31 21:06:21.808172042 (629h37m21.679255087s) [noise=true , address=45.91.116.7:4011 , piece_id=OODB666DEWAW2CHBCHNETCEHMRLGNHPNGSZMKKWOA7K4TDRT6ZWA, seg_pos={0 0}, upload=551730]: Long tail cancellation due to optimal threshold Dec 31 21:06:21.808323540 (629h37m21.679406575s) [noise=true , address=145.79.227.44:4193 , piece_id=3JQX6CG4J4QRQYNLVXWOCQXB4WZRR5OPJ5OBIT4JI3JAXJYK733Q, seg_pos={0 0}, upload=551730]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 31 21:06:21.808330124 (629h37m21.679413159s) [noise=true , address=145.79.227.44:4193 , piece_id=3JQX6CG4J4QRQYNLVXWOCQXB4WZRR5OPJ5OBIT4JI3JAXJYK733Q, seg_pos={0 0}, upload=551730]: Long tail cancellation due to optimal threshold Dec 31 21:06:21.808540735 (629h37m21.67962377s) [seg_pos={0 0}, upload=551730]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 21:06:21.808632328 (629h37m21.679715364s) [upload=551730]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 21:06:21.808637007 (629h37m21.679720052s) [upload=551730]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 21:06:22.105013640 (629h37m21.976096675s) [upload=551730]: Done uploading segments... Dec 31 21:06:22.105031735 (629h37m21.97611477s) [upload=551730]: Upload finished. err: <nil> Duration: 5.953709203s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [...X..XX...X...XX....X......X....XX........X...X.......X....] duration: 5.971036305s trace: https://jaeger.dev.storj.io/trace/6b4a006b8b7aa49b FAIL (2025-12-31 21:08:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: VVVRZMWTKJ334XCSMYKQECFKFSSMFBRIKTCUJHQZUIFZNHA3ZT5Q) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: H4EO22GSLMHTDAOC4HXKEJ2ZRATC2GHH7YDLJ2PHRJ4FPYP2Q2FQ) context deadline exceeded ======LOGS====== Dec 31 21:06:19.130592534 (629h37m19.001675569s) [upload=551730]: Starting upload Dec 31 21:06:19.130672245 (629h37m19.00175529s) [upload=551730]: Done starting upload Dec 31 21:06:19.130755443 (629h37m19.001838488s) [upload=551730]: Uploading segments... Dec 31 21:06:19.130901432 (629h37m19.001984467s) [upload=551730]: Got next segment. Inline: false Dec 31 21:06:19.130950555 (629h37m19.0020336s) [upload=551730]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 21:06:19.130955284 (629h37m19.002038319s) [upload=551730]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 21:06:19.381665200 (629h37m19.252748265s) [seg_pos={0 0}, upload=551730]: Begin upload segment... Dec 31 21:06:19.381770831 (629h37m19.252853876s) [seg_pos={0 0}, upload=551730]: Done begin upload segment. Dec 31 21:06:19.381851134 (629h37m19.252934179s) [noise=true , address=145.79.227.21:4474 , piece_id=VVRXFDYZR2HSEYIHQ4KRR2TJ7MA45GLXXUAPRCU2QE47YXWP63RA, seg_pos={0 0}, upload=551730]: Uploading piece... Dec 31 21:06:19.385528566 (629h37m19.256611601s) [noise=true , address=45.91.116.8:4045 , piece_id=H4EO22GSLMHTDAOC4HXKEJ2ZRATC2GHH7YDLJ2PHRJ4FPYP2Q2FQ, seg_pos={0 0}, upload=551730]: Uploading piece... Dec 31 21:06:19.386657856 (629h37m19.257740891s) [noise=true , address=145.79.227.8:4331 , piece_id=VEIWQMSN2RVO3HWYKVPO2GRUQBDKOSCVGSLU3TGHCHC6YRXL4HAQ, seg_pos={0 0}, upload=551730]: Uploading piece... Dec 31 21:06:19.387654513 (629h37m19.258737549s) [noise=true , address=145.79.227.36:4519 , piece_id=OCJNGU7QE7TW3VBDQ53TLZXIUEQGBZFYQNDPCNAXX4HLFX2WQYZA, seg_pos={0 0}, upload=551730]: Uploading piece... Dec 31 21:06:19.388655089 (629h37m19.259738134s) [noise=true , address=145.79.227.27:4690 , piece_id=POOEPKVNRGSZ6QHOEVYUCOOUGQZCCPVJFZ5G5GVSZSE6JVC2QGWQ, seg_pos={0 0}, upload=551730]: Uploading piece... Dec 31 21:06:19.390073680 (629h37m19.261156716s) [noise=true , address=145.79.227.2:4115 , piece_id=ZH3P4BLJCINMEOAXB27FGCQHFJXX2AQK5GICCBCVXX4GFXYYTFVQ, seg_pos={0 0}, upload=551730]: Uploading piece... Dec 31 21:06:19.391092029 (629h37m19.262175074s) [noise=true , address=45.91.116.7:4011 , piece_id=OODB666DEWAW2CHBCHNETCEHMRLGNHPNGSZMKKWOA7K4TDRT6ZWA, seg_pos={0 0}, upload=551730]: Uploading piece... Dec 31 21:06:19.392162568 (629h37m19.263245603s) [noise=true , address=45.91.116.14:4191 , piece_id=XCCCHRXHATZGD2VUXYUMTFZTQ5FACPDX22UG3YSZUW4SJBCBXV5A, seg_pos={0 0}, upload=551730]: Uploading piece... Dec 31 21:06:19.393340190 (629h37m19.264423225s) [noise=true , address=145.79.227.17:4970 , piece_id=37X56AFL2F7EY77MOIAMIT7ZWFXHWNZVBK2WHQ5J55UFEQCQIFJQ, seg_pos={0 0}, upload=551730]: Uploading piece... Dec 31 21:06:19.395205221 (629h37m19.266288256s) [noise=true , address=45.91.116.4:4937 , piece_id=C6IM64R7L5IYDPISRRHSDUXD6TFMD6CTH3BNULTVQEFCBKJOGP3A, seg_pos={0 0}, upload=551730]: Uploading piece... Dec 31 21:06:19.396253277 (629h37m19.267336312s) [noise=true , address=45.91.116.10:4082 , piece_id=VVVRZMWTKJ334XCSMYKQECFKFSSMFBRIKTCUJHQZUIFZNHA3ZT5Q, seg_pos={0 0}, upload=551730]: Uploading piece... Dec 31 21:06:19.397275324 (629h37m19.268358359s) [noise=true , address=145.79.227.44:4193 , piece_id=3JQX6CG4J4QRQYNLVXWOCQXB4WZRR5OPJ5OBIT4JI3JAXJYK733Q, seg_pos={0 0}, upload=551730]: Uploading piece... Dec 31 21:06:19.398334380 (629h37m19.269417415s) [noise=true , address=145.79.227.22:4509 , piece_id=XLPZDZSJCBIRKX5LGIKLLP2CVHUM2X5SRBGNA6U2FGCDN2UXIPDQ, seg_pos={0 0}, upload=551730]: Uploading piece... Dec 31 21:06:19.400004671 (629h37m19.271087716s) [noise=true , address=145.79.227.6:4259 , piece_id=2LGQDOPBK4WUOOKTSFTZKHHLFZ3IMRBC2HCHFMCWOKETMZTFEAFA, seg_pos={0 0}, upload=551730]: Uploading piece... Dec 31 21:06:19.401527831 (629h37m19.272610876s) [noise=true , address=77.90.31.18:4808 , piece_id=4FAKZB3YUM4Y7PATZAWHD7SJOCTM4K7G75E7ARWKRRJUYNS3TZAQ, seg_pos={0 0}, upload=551730]: Uploading piece... Dec 31 21:06:19.402400904 (629h37m19.273483949s) [noise=true , address=145.79.227.37:4565 , piece_id=IAPFHAF5L2YZS2KYURPMYX3UT7FVXI7EKYX4QEJ7N7G7VOU3KTNA, seg_pos={0 0}, upload=551730]: Uploading piece... Dec 31 21:06:21.101007053 (629h37m20.972090098s) [upload=551730]: Next returned nil segment Dec 31 21:06:21.101022401 (629h37m20.972105436s) [upload=551730]: Waiting for error group managing segments... Dec 31 21:06:21.385360411 (629h37m21.256443446s) [noise=true , address=145.79.227.8:4331 , piece_id=VEIWQMSN2RVO3HWYKVPO2GRUQBDKOSCVGSLU3TGHCHC6YRXL4HAQ, seg_pos={0 0}, upload=551730]: Done uploading piece. err: <nil> Dec 31 21:06:21.386079980 (629h37m21.257163015s) [noise=true , address=145.79.227.21:4474 , piece_id=VVRXFDYZR2HSEYIHQ4KRR2TJ7MA45GLXXUAPRCU2QE47YXWP63RA, seg_pos={0 0}, upload=551730]: Done uploading piece. err: <nil> Dec 31 21:06:21.392677413 (629h37m21.263760478s) [noise=true , address=145.79.227.27:4690 , piece_id=POOEPKVNRGSZ6QHOEVYUCOOUGQZCCPVJFZ5G5GVSZSE6JVC2QGWQ, seg_pos={0 0}, upload=551730]: Done uploading piece. err: <nil> Dec 31 21:06:21.393084087 (629h37m21.264167122s) [noise=true , address=145.79.227.17:4970 , piece_id=37X56AFL2F7EY77MOIAMIT7ZWFXHWNZVBK2WHQ5J55UFEQCQIFJQ, seg_pos={0 0}, upload=551730]: Done uploading piece. err: <nil> Dec 31 21:06:21.393326629 (629h37m21.264409674s) [noise=true , address=145.79.227.6:4259 , piece_id=2LGQDOPBK4WUOOKTSFTZKHHLFZ3IMRBC2HCHFMCWOKETMZTFEAFA, seg_pos={0 0}, upload=551730]: Done uploading piece. err: <nil> Dec 31 21:06:21.404225282 (629h37m21.275308317s) [noise=true , address=145.79.227.37:4565 , piece_id=IAPFHAF5L2YZS2KYURPMYX3UT7FVXI7EKYX4QEJ7N7G7VOU3KTNA, seg_pos={0 0}, upload=551730]: Done uploading piece. err: <nil> Dec 31 21:06:21.407043318 (629h37m21.278126353s) [noise=true , address=145.79.227.36:4519 , piece_id=OCJNGU7QE7TW3VBDQ53TLZXIUEQGBZFYQNDPCNAXX4HLFX2WQYZA, seg_pos={0 0}, upload=551730]: Done uploading piece. err: <nil> Dec 31 21:06:21.411401707 (629h37m21.282484742s) [noise=true , address=145.79.227.2:4115 , piece_id=ZH3P4BLJCINMEOAXB27FGCQHFJXX2AQK5GICCBCVXX4GFXYYTFVQ, seg_pos={0 0}, upload=551730]: Done uploading piece. err: <nil> Dec 31 21:06:21.415461036 (629h37m21.286544081s) [noise=true , address=145.79.227.22:4509 , piece_id=XLPZDZSJCBIRKX5LGIKLLP2CVHUM2X5SRBGNA6U2FGCDN2UXIPDQ, seg_pos={0 0}, upload=551730]: Done uploading piece. err: <nil> Dec 31 21:06:21.556582891 (629h37m21.427665936s) [noise=true , address=77.90.31.18:4808 , piece_id=4FAKZB3YUM4Y7PATZAWHD7SJOCTM4K7G75E7ARWKRRJUYNS3TZAQ, seg_pos={0 0}, upload=551730]: Done uploading piece. err: <nil> Dec 31 21:06:21.721535795 (629h37m21.59261884s) [noise=true , address=45.91.116.8:4045 , piece_id=H4EO22GSLMHTDAOC4HXKEJ2ZRATC2GHH7YDLJ2PHRJ4FPYP2Q2FQ, seg_pos={0 0}, upload=551730]: Done uploading piece. err: <nil> Dec 31 21:06:21.775331980 (629h37m21.646415016s) [noise=true , address=45.91.116.10:4082 , piece_id=VVVRZMWTKJ334XCSMYKQECFKFSSMFBRIKTCUJHQZUIFZNHA3ZT5Q, seg_pos={0 0}, upload=551730]: Done uploading piece. err: <nil> Dec 31 21:06:21.807810725 (629h37m21.67889377s) [noise=true , address=45.91.116.4:4937 , piece_id=C6IM64R7L5IYDPISRRHSDUXD6TFMD6CTH3BNULTVQEFCBKJOGP3A, seg_pos={0 0}, upload=551730]: Done uploading piece. err: <nil> Dec 31 21:06:21.807850861 (629h37m21.678933916s) [seg_pos={0 0}, upload=551730]: Segment reached optimal threshold of 13 pieces. Dec 31 21:06:21.808089556 (629h37m21.679172601s) [noise=true , address=45.91.116.14:4191 , piece_id=XCCCHRXHATZGD2VUXYUMTFZTQ5FACPDX22UG3YSZUW4SJBCBXV5A, seg_pos={0 0}, upload=551730]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Dec 31 21:06:21.808097911 (629h37m21.679180966s) [noise=true , address=45.91.116.14:4191 , piece_id=XCCCHRXHATZGD2VUXYUMTFZTQ5FACPDX22UG3YSZUW4SJBCBXV5A, seg_pos={0 0}, upload=551730]: Long tail cancellation due to optimal threshold Dec 31 21:06:21.808156182 (629h37m21.679239228s) [noise=true , address=45.91.116.7:4011 , piece_id=OODB666DEWAW2CHBCHNETCEHMRLGNHPNGSZMKKWOA7K4TDRT6ZWA, seg_pos={0 0}, upload=551730]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 31 21:06:21.808172042 (629h37m21.679255087s) [noise=true , address=45.91.116.7:4011 , piece_id=OODB666DEWAW2CHBCHNETCEHMRLGNHPNGSZMKKWOA7K4TDRT6ZWA, seg_pos={0 0}, upload=551730]: Long tail cancellation due to optimal threshold Dec 31 21:06:21.808323540 (629h37m21.679406575s) [noise=true , address=145.79.227.44:4193 , piece_id=3JQX6CG4J4QRQYNLVXWOCQXB4WZRR5OPJ5OBIT4JI3JAXJYK733Q, seg_pos={0 0}, upload=551730]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 31 21:06:21.808330124 (629h37m21.679413159s) [noise=true , address=145.79.227.44:4193 , piece_id=3JQX6CG4J4QRQYNLVXWOCQXB4WZRR5OPJ5OBIT4JI3JAXJYK733Q, seg_pos={0 0}, upload=551730]: Long tail cancellation due to optimal threshold Dec 31 21:06:21.808540735 (629h37m21.67962377s) [seg_pos={0 0}, upload=551730]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 21:06:21.808632328 (629h37m21.679715364s) [upload=551730]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 21:06:21.808637007 (629h37m21.679720052s) [upload=551730]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 21:06:22.105013640 (629h37m21.976096675s) [upload=551730]: Done uploading segments... Dec 31 21:06:22.105031735 (629h37m21.97611477s) [upload=551730]: Upload finished. err: <nil> Duration: 5.971067505s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [X...X..XX...X...XX....X......X....XX........X...X.......X...] duration: 5.944173428s trace: https://jaeger.dev.storj.io/trace/6b4a006b8b7aa49b FAIL (2025-12-31 21:08:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: VVVRZMWTKJ334XCSMYKQECFKFSSMFBRIKTCUJHQZUIFZNHA3ZT5Q) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: H4EO22GSLMHTDAOC4HXKEJ2ZRATC2GHH7YDLJ2PHRJ4FPYP2Q2FQ) context deadline exceeded ======LOGS====== Dec 31 21:06:19.130592534 (629h37m19.001675569s) [upload=551730]: Starting upload Dec 31 21:06:19.130672245 (629h37m19.00175529s) [upload=551730]: Done starting upload Dec 31 21:06:19.130755443 (629h37m19.001838488s) [upload=551730]: Uploading segments... Dec 31 21:06:19.130901432 (629h37m19.001984467s) [upload=551730]: Got next segment. Inline: false Dec 31 21:06:19.130950555 (629h37m19.0020336s) [upload=551730]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 21:06:19.130955284 (629h37m19.002038319s) [upload=551730]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 21:06:19.381665200 (629h37m19.252748265s) [seg_pos={0 0}, upload=551730]: Begin upload segment... Dec 31 21:06:19.381770831 (629h37m19.252853876s) [seg_pos={0 0}, upload=551730]: Done begin upload segment. Dec 31 21:06:19.381851134 (629h37m19.252934179s) [noise=true , address=145.79.227.21:4474 , piece_id=VVRXFDYZR2HSEYIHQ4KRR2TJ7MA45GLXXUAPRCU2QE47YXWP63RA, seg_pos={0 0}, upload=551730]: Uploading piece... Dec 31 21:06:19.385528566 (629h37m19.256611601s) [noise=true , address=45.91.116.8:4045 , piece_id=H4EO22GSLMHTDAOC4HXKEJ2ZRATC2GHH7YDLJ2PHRJ4FPYP2Q2FQ, seg_pos={0 0}, upload=551730]: Uploading piece... Dec 31 21:06:19.386657856 (629h37m19.257740891s) [noise=true , address=145.79.227.8:4331 , piece_id=VEIWQMSN2RVO3HWYKVPO2GRUQBDKOSCVGSLU3TGHCHC6YRXL4HAQ, seg_pos={0 0}, upload=551730]: Uploading piece... Dec 31 21:06:19.387654513 (629h37m19.258737549s) [noise=true , address=145.79.227.36:4519 , piece_id=OCJNGU7QE7TW3VBDQ53TLZXIUEQGBZFYQNDPCNAXX4HLFX2WQYZA, seg_pos={0 0}, upload=551730]: Uploading piece... Dec 31 21:06:19.388655089 (629h37m19.259738134s) [noise=true , address=145.79.227.27:4690 , piece_id=POOEPKVNRGSZ6QHOEVYUCOOUGQZCCPVJFZ5G5GVSZSE6JVC2QGWQ, seg_pos={0 0}, upload=551730]: Uploading piece... Dec 31 21:06:19.390073680 (629h37m19.261156716s) [noise=true , address=145.79.227.2:4115 , piece_id=ZH3P4BLJCINMEOAXB27FGCQHFJXX2AQK5GICCBCVXX4GFXYYTFVQ, seg_pos={0 0}, upload=551730]: Uploading piece... Dec 31 21:06:19.391092029 (629h37m19.262175074s) [noise=true , address=45.91.116.7:4011 , piece_id=OODB666DEWAW2CHBCHNETCEHMRLGNHPNGSZMKKWOA7K4TDRT6ZWA, seg_pos={0 0}, upload=551730]: Uploading piece... Dec 31 21:06:19.392162568 (629h37m19.263245603s) [noise=true , address=45.91.116.14:4191 , piece_id=XCCCHRXHATZGD2VUXYUMTFZTQ5FACPDX22UG3YSZUW4SJBCBXV5A, seg_pos={0 0}, upload=551730]: Uploading piece... Dec 31 21:06:19.393340190 (629h37m19.264423225s) [noise=true , address=145.79.227.17:4970 , piece_id=37X56AFL2F7EY77MOIAMIT7ZWFXHWNZVBK2WHQ5J55UFEQCQIFJQ, seg_pos={0 0}, upload=551730]: Uploading piece... Dec 31 21:06:19.395205221 (629h37m19.266288256s) [noise=true , address=45.91.116.4:4937 , piece_id=C6IM64R7L5IYDPISRRHSDUXD6TFMD6CTH3BNULTVQEFCBKJOGP3A, seg_pos={0 0}, upload=551730]: Uploading piece... Dec 31 21:06:19.396253277 (629h37m19.267336312s) [noise=true , address=45.91.116.10:4082 , piece_id=VVVRZMWTKJ334XCSMYKQECFKFSSMFBRIKTCUJHQZUIFZNHA3ZT5Q, seg_pos={0 0}, upload=551730]: Uploading piece... Dec 31 21:06:19.397275324 (629h37m19.268358359s) [noise=true , address=145.79.227.44:4193 , piece_id=3JQX6CG4J4QRQYNLVXWOCQXB4WZRR5OPJ5OBIT4JI3JAXJYK733Q, seg_pos={0 0}, upload=551730]: Uploading piece... Dec 31 21:06:19.398334380 (629h37m19.269417415s) [noise=true , address=145.79.227.22:4509 , piece_id=XLPZDZSJCBIRKX5LGIKLLP2CVHUM2X5SRBGNA6U2FGCDN2UXIPDQ, seg_pos={0 0}, upload=551730]: Uploading piece... Dec 31 21:06:19.400004671 (629h37m19.271087716s) [noise=true , address=145.79.227.6:4259 , piece_id=2LGQDOPBK4WUOOKTSFTZKHHLFZ3IMRBC2HCHFMCWOKETMZTFEAFA, seg_pos={0 0}, upload=551730]: Uploading piece... Dec 31 21:06:19.401527831 (629h37m19.272610876s) [noise=true , address=77.90.31.18:4808 , piece_id=4FAKZB3YUM4Y7PATZAWHD7SJOCTM4K7G75E7ARWKRRJUYNS3TZAQ, seg_pos={0 0}, upload=551730]: Uploading piece... Dec 31 21:06:19.402400904 (629h37m19.273483949s) [noise=true , address=145.79.227.37:4565 , piece_id=IAPFHAF5L2YZS2KYURPMYX3UT7FVXI7EKYX4QEJ7N7G7VOU3KTNA, seg_pos={0 0}, upload=551730]: Uploading piece... Dec 31 21:06:21.101007053 (629h37m20.972090098s) [upload=551730]: Next returned nil segment Dec 31 21:06:21.101022401 (629h37m20.972105436s) [upload=551730]: Waiting for error group managing segments... Dec 31 21:06:21.385360411 (629h37m21.256443446s) [noise=true , address=145.79.227.8:4331 , piece_id=VEIWQMSN2RVO3HWYKVPO2GRUQBDKOSCVGSLU3TGHCHC6YRXL4HAQ, seg_pos={0 0}, upload=551730]: Done uploading piece. err: <nil> Dec 31 21:06:21.386079980 (629h37m21.257163015s) [noise=true , address=145.79.227.21:4474 , piece_id=VVRXFDYZR2HSEYIHQ4KRR2TJ7MA45GLXXUAPRCU2QE47YXWP63RA, seg_pos={0 0}, upload=551730]: Done uploading piece. err: <nil> Dec 31 21:06:21.392677413 (629h37m21.263760478s) [noise=true , address=145.79.227.27:4690 , piece_id=POOEPKVNRGSZ6QHOEVYUCOOUGQZCCPVJFZ5G5GVSZSE6JVC2QGWQ, seg_pos={0 0}, upload=551730]: Done uploading piece. err: <nil> Dec 31 21:06:21.393084087 (629h37m21.264167122s) [noise=true , address=145.79.227.17:4970 , piece_id=37X56AFL2F7EY77MOIAMIT7ZWFXHWNZVBK2WHQ5J55UFEQCQIFJQ, seg_pos={0 0}, upload=551730]: Done uploading piece. err: <nil> Dec 31 21:06:21.393326629 (629h37m21.264409674s) [noise=true , address=145.79.227.6:4259 , piece_id=2LGQDOPBK4WUOOKTSFTZKHHLFZ3IMRBC2HCHFMCWOKETMZTFEAFA, seg_pos={0 0}, upload=551730]: Done uploading piece. err: <nil> Dec 31 21:06:21.404225282 (629h37m21.275308317s) [noise=true , address=145.79.227.37:4565 , piece_id=IAPFHAF5L2YZS2KYURPMYX3UT7FVXI7EKYX4QEJ7N7G7VOU3KTNA, seg_pos={0 0}, upload=551730]: Done uploading piece. err: <nil> Dec 31 21:06:21.407043318 (629h37m21.278126353s) [noise=true , address=145.79.227.36:4519 , piece_id=OCJNGU7QE7TW3VBDQ53TLZXIUEQGBZFYQNDPCNAXX4HLFX2WQYZA, seg_pos={0 0}, upload=551730]: Done uploading piece. err: <nil> Dec 31 21:06:21.411401707 (629h37m21.282484742s) [noise=true , address=145.79.227.2:4115 , piece_id=ZH3P4BLJCINMEOAXB27FGCQHFJXX2AQK5GICCBCVXX4GFXYYTFVQ, seg_pos={0 0}, upload=551730]: Done uploading piece. err: <nil> Dec 31 21:06:21.415461036 (629h37m21.286544081s) [noise=true , address=145.79.227.22:4509 , piece_id=XLPZDZSJCBIRKX5LGIKLLP2CVHUM2X5SRBGNA6U2FGCDN2UXIPDQ, seg_pos={0 0}, upload=551730]: Done uploading piece. err: <nil> Dec 31 21:06:21.556582891 (629h37m21.427665936s) [noise=true , address=77.90.31.18:4808 , piece_id=4FAKZB3YUM4Y7PATZAWHD7SJOCTM4K7G75E7ARWKRRJUYNS3TZAQ, seg_pos={0 0}, upload=551730]: Done uploading piece. err: <nil> Dec 31 21:06:21.721535795 (629h37m21.59261884s) [noise=true , address=45.91.116.8:4045 , piece_id=H4EO22GSLMHTDAOC4HXKEJ2ZRATC2GHH7YDLJ2PHRJ4FPYP2Q2FQ, seg_pos={0 0}, upload=551730]: Done uploading piece. err: <nil> Dec 31 21:06:21.775331980 (629h37m21.646415016s) [noise=true , address=45.91.116.10:4082 , piece_id=VVVRZMWTKJ334XCSMYKQECFKFSSMFBRIKTCUJHQZUIFZNHA3ZT5Q, seg_pos={0 0}, upload=551730]: Done uploading piece. err: <nil> Dec 31 21:06:21.807810725 (629h37m21.67889377s) [noise=true , address=45.91.116.4:4937 , piece_id=C6IM64R7L5IYDPISRRHSDUXD6TFMD6CTH3BNULTVQEFCBKJOGP3A, seg_pos={0 0}, upload=551730]: Done uploading piece. err: <nil> Dec 31 21:06:21.807850861 (629h37m21.678933916s) [seg_pos={0 0}, upload=551730]: Segment reached optimal threshold of 13 pieces. Dec 31 21:06:21.808089556 (629h37m21.679172601s) [noise=true , address=45.91.116.14:4191 , piece_id=XCCCHRXHATZGD2VUXYUMTFZTQ5FACPDX22UG3YSZUW4SJBCBXV5A, seg_pos={0 0}, upload=551730]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Dec 31 21:06:21.808097911 (629h37m21.679180966s) [noise=true , address=45.91.116.14:4191 , piece_id=XCCCHRXHATZGD2VUXYUMTFZTQ5FACPDX22UG3YSZUW4SJBCBXV5A, seg_pos={0 0}, upload=551730]: Long tail cancellation due to optimal threshold Dec 31 21:06:21.808156182 (629h37m21.679239228s) [noise=true , address=45.91.116.7:4011 , piece_id=OODB666DEWAW2CHBCHNETCEHMRLGNHPNGSZMKKWOA7K4TDRT6ZWA, seg_pos={0 0}, upload=551730]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 31 21:06:21.808172042 (629h37m21.679255087s) [noise=true , address=45.91.116.7:4011 , piece_id=OODB666DEWAW2CHBCHNETCEHMRLGNHPNGSZMKKWOA7K4TDRT6ZWA, seg_pos={0 0}, upload=551730]: Long tail cancellation due to optimal threshold Dec 31 21:06:21.808323540 (629h37m21.679406575s) [noise=true , address=145.79.227.44:4193 , piece_id=3JQX6CG4J4QRQYNLVXWOCQXB4WZRR5OPJ5OBIT4JI3JAXJYK733Q, seg_pos={0 0}, upload=551730]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 31 21:06:21.808330124 (629h37m21.679413159s) [noise=true , address=145.79.227.44:4193 , piece_id=3JQX6CG4J4QRQYNLVXWOCQXB4WZRR5OPJ5OBIT4JI3JAXJYK733Q, seg_pos={0 0}, upload=551730]: Long tail cancellation due to optimal threshold Dec 31 21:06:21.808540735 (629h37m21.67962377s) [seg_pos={0 0}, upload=551730]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 21:06:21.808632328 (629h37m21.679715364s) [upload=551730]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 21:06:21.808637007 (629h37m21.679720052s) [upload=551730]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 21:06:22.105013640 (629h37m21.976096675s) [upload=551730]: Done uploading segments... Dec 31 21:06:22.105031735 (629h37m21.97611477s) [upload=551730]: Upload finished. err: <nil> Duration: 5.944201871s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [XX...X..XX...X...XX....X......X....XX........X...X.......X..] duration: 5.848714892s trace: https://jaeger.dev.storj.io/trace/6b4a006b8b7aa49b FAIL (2025-12-31 21:07:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: VVVRZMWTKJ334XCSMYKQECFKFSSMFBRIKTCUJHQZUIFZNHA3ZT5Q) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: H4EO22GSLMHTDAOC4HXKEJ2ZRATC2GHH7YDLJ2PHRJ4FPYP2Q2FQ) context deadline exceeded ======LOGS====== Dec 31 21:06:19.130592534 (629h37m19.001675569s) [upload=551730]: Starting upload Dec 31 21:06:19.130672245 (629h37m19.00175529s) [upload=551730]: Done starting upload Dec 31 21:06:19.130755443 (629h37m19.001838488s) [upload=551730]: Uploading segments... Dec 31 21:06:19.130901432 (629h37m19.001984467s) [upload=551730]: Got next segment. Inline: false Dec 31 21:06:19.130950555 (629h37m19.0020336s) [upload=551730]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 21:06:19.130955284 (629h37m19.002038319s) [upload=551730]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 21:06:19.381665200 (629h37m19.252748265s) [seg_pos={0 0}, upload=551730]: Begin upload segment... Dec 31 21:06:19.381770831 (629h37m19.252853876s) [seg_pos={0 0}, upload=551730]: Done begin upload segment. Dec 31 21:06:19.381851134 (629h37m19.252934179s) [noise=true , address=145.79.227.21:4474 , piece_id=VVRXFDYZR2HSEYIHQ4KRR2TJ7MA45GLXXUAPRCU2QE47YXWP63RA, seg_pos={0 0}, upload=551730]: Uploading piece... Dec 31 21:06:19.385528566 (629h37m19.256611601s) [noise=true , address=45.91.116.8:4045 , piece_id=H4EO22GSLMHTDAOC4HXKEJ2ZRATC2GHH7YDLJ2PHRJ4FPYP2Q2FQ, seg_pos={0 0}, upload=551730]: Uploading piece... Dec 31 21:06:19.386657856 (629h37m19.257740891s) [noise=true , address=145.79.227.8:4331 , piece_id=VEIWQMSN2RVO3HWYKVPO2GRUQBDKOSCVGSLU3TGHCHC6YRXL4HAQ, seg_pos={0 0}, upload=551730]: Uploading piece... Dec 31 21:06:19.387654513 (629h37m19.258737549s) [noise=true , address=145.79.227.36:4519 , piece_id=OCJNGU7QE7TW3VBDQ53TLZXIUEQGBZFYQNDPCNAXX4HLFX2WQYZA, seg_pos={0 0}, upload=551730]: Uploading piece... Dec 31 21:06:19.388655089 (629h37m19.259738134s) [noise=true , address=145.79.227.27:4690 , piece_id=POOEPKVNRGSZ6QHOEVYUCOOUGQZCCPVJFZ5G5GVSZSE6JVC2QGWQ, seg_pos={0 0}, upload=551730]: Uploading piece... Dec 31 21:06:19.390073680 (629h37m19.261156716s) [noise=true , address=145.79.227.2:4115 , piece_id=ZH3P4BLJCINMEOAXB27FGCQHFJXX2AQK5GICCBCVXX4GFXYYTFVQ, seg_pos={0 0}, upload=551730]: Uploading piece... Dec 31 21:06:19.391092029 (629h37m19.262175074s) [noise=true , address=45.91.116.7:4011 , piece_id=OODB666DEWAW2CHBCHNETCEHMRLGNHPNGSZMKKWOA7K4TDRT6ZWA, seg_pos={0 0}, upload=551730]: Uploading piece... Dec 31 21:06:19.392162568 (629h37m19.263245603s) [noise=true , address=45.91.116.14:4191 , piece_id=XCCCHRXHATZGD2VUXYUMTFZTQ5FACPDX22UG3YSZUW4SJBCBXV5A, seg_pos={0 0}, upload=551730]: Uploading piece... Dec 31 21:06:19.393340190 (629h37m19.264423225s) [noise=true , address=145.79.227.17:4970 , piece_id=37X56AFL2F7EY77MOIAMIT7ZWFXHWNZVBK2WHQ5J55UFEQCQIFJQ, seg_pos={0 0}, upload=551730]: Uploading piece... Dec 31 21:06:19.395205221 (629h37m19.266288256s) [noise=true , address=45.91.116.4:4937 , piece_id=C6IM64R7L5IYDPISRRHSDUXD6TFMD6CTH3BNULTVQEFCBKJOGP3A, seg_pos={0 0}, upload=551730]: Uploading piece... Dec 31 21:06:19.396253277 (629h37m19.267336312s) [noise=true , address=45.91.116.10:4082 , piece_id=VVVRZMWTKJ334XCSMYKQECFKFSSMFBRIKTCUJHQZUIFZNHA3ZT5Q, seg_pos={0 0}, upload=551730]: Uploading piece... Dec 31 21:06:19.397275324 (629h37m19.268358359s) [noise=true , address=145.79.227.44:4193 , piece_id=3JQX6CG4J4QRQYNLVXWOCQXB4WZRR5OPJ5OBIT4JI3JAXJYK733Q, seg_pos={0 0}, upload=551730]: Uploading piece... Dec 31 21:06:19.398334380 (629h37m19.269417415s) [noise=true , address=145.79.227.22:4509 , piece_id=XLPZDZSJCBIRKX5LGIKLLP2CVHUM2X5SRBGNA6U2FGCDN2UXIPDQ, seg_pos={0 0}, upload=551730]: Uploading piece... Dec 31 21:06:19.400004671 (629h37m19.271087716s) [noise=true , address=145.79.227.6:4259 , piece_id=2LGQDOPBK4WUOOKTSFTZKHHLFZ3IMRBC2HCHFMCWOKETMZTFEAFA, seg_pos={0 0}, upload=551730]: Uploading piece... Dec 31 21:06:19.401527831 (629h37m19.272610876s) [noise=true , address=77.90.31.18:4808 , piece_id=4FAKZB3YUM4Y7PATZAWHD7SJOCTM4K7G75E7ARWKRRJUYNS3TZAQ, seg_pos={0 0}, upload=551730]: Uploading piece... Dec 31 21:06:19.402400904 (629h37m19.273483949s) [noise=true , address=145.79.227.37:4565 , piece_id=IAPFHAF5L2YZS2KYURPMYX3UT7FVXI7EKYX4QEJ7N7G7VOU3KTNA, seg_pos={0 0}, upload=551730]: Uploading piece... Dec 31 21:06:21.101007053 (629h37m20.972090098s) [upload=551730]: Next returned nil segment Dec 31 21:06:21.101022401 (629h37m20.972105436s) [upload=551730]: Waiting for error group managing segments... Dec 31 21:06:21.385360411 (629h37m21.256443446s) [noise=true , address=145.79.227.8:4331 , piece_id=VEIWQMSN2RVO3HWYKVPO2GRUQBDKOSCVGSLU3TGHCHC6YRXL4HAQ, seg_pos={0 0}, upload=551730]: Done uploading piece. err: <nil> Dec 31 21:06:21.386079980 (629h37m21.257163015s) [noise=true , address=145.79.227.21:4474 , piece_id=VVRXFDYZR2HSEYIHQ4KRR2TJ7MA45GLXXUAPRCU2QE47YXWP63RA, seg_pos={0 0}, upload=551730]: Done uploading piece. err: <nil> Dec 31 21:06:21.392677413 (629h37m21.263760478s) [noise=true , address=145.79.227.27:4690 , piece_id=POOEPKVNRGSZ6QHOEVYUCOOUGQZCCPVJFZ5G5GVSZSE6JVC2QGWQ, seg_pos={0 0}, upload=551730]: Done uploading piece. err: <nil> Dec 31 21:06:21.393084087 (629h37m21.264167122s) [noise=true , address=145.79.227.17:4970 , piece_id=37X56AFL2F7EY77MOIAMIT7ZWFXHWNZVBK2WHQ5J55UFEQCQIFJQ, seg_pos={0 0}, upload=551730]: Done uploading piece. err: <nil> Dec 31 21:06:21.393326629 (629h37m21.264409674s) [noise=true , address=145.79.227.6:4259 , piece_id=2LGQDOPBK4WUOOKTSFTZKHHLFZ3IMRBC2HCHFMCWOKETMZTFEAFA, seg_pos={0 0}, upload=551730]: Done uploading piece. err: <nil> Dec 31 21:06:21.404225282 (629h37m21.275308317s) [noise=true , address=145.79.227.37:4565 , piece_id=IAPFHAF5L2YZS2KYURPMYX3UT7FVXI7EKYX4QEJ7N7G7VOU3KTNA, seg_pos={0 0}, upload=551730]: Done uploading piece. err: <nil> Dec 31 21:06:21.407043318 (629h37m21.278126353s) [noise=true , address=145.79.227.36:4519 , piece_id=OCJNGU7QE7TW3VBDQ53TLZXIUEQGBZFYQNDPCNAXX4HLFX2WQYZA, seg_pos={0 0}, upload=551730]: Done uploading piece. err: <nil> Dec 31 21:06:21.411401707 (629h37m21.282484742s) [noise=true , address=145.79.227.2:4115 , piece_id=ZH3P4BLJCINMEOAXB27FGCQHFJXX2AQK5GICCBCVXX4GFXYYTFVQ, seg_pos={0 0}, upload=551730]: Done uploading piece. err: <nil> Dec 31 21:06:21.415461036 (629h37m21.286544081s) [noise=true , address=145.79.227.22:4509 , piece_id=XLPZDZSJCBIRKX5LGIKLLP2CVHUM2X5SRBGNA6U2FGCDN2UXIPDQ, seg_pos={0 0}, upload=551730]: Done uploading piece. err: <nil> Dec 31 21:06:21.556582891 (629h37m21.427665936s) [noise=true , address=77.90.31.18:4808 , piece_id=4FAKZB3YUM4Y7PATZAWHD7SJOCTM4K7G75E7ARWKRRJUYNS3TZAQ, seg_pos={0 0}, upload=551730]: Done uploading piece. err: <nil> Dec 31 21:06:21.721535795 (629h37m21.59261884s) [noise=true , address=45.91.116.8:4045 , piece_id=H4EO22GSLMHTDAOC4HXKEJ2ZRATC2GHH7YDLJ2PHRJ4FPYP2Q2FQ, seg_pos={0 0}, upload=551730]: Done uploading piece. err: <nil> Dec 31 21:06:21.775331980 (629h37m21.646415016s) [noise=true , address=45.91.116.10:4082 , piece_id=VVVRZMWTKJ334XCSMYKQECFKFSSMFBRIKTCUJHQZUIFZNHA3ZT5Q, seg_pos={0 0}, upload=551730]: Done uploading piece. err: <nil> Dec 31 21:06:21.807810725 (629h37m21.67889377s) [noise=true , address=45.91.116.4:4937 , piece_id=C6IM64R7L5IYDPISRRHSDUXD6TFMD6CTH3BNULTVQEFCBKJOGP3A, seg_pos={0 0}, upload=551730]: Done uploading piece. err: <nil> Dec 31 21:06:21.807850861 (629h37m21.678933916s) [seg_pos={0 0}, upload=551730]: Segment reached optimal threshold of 13 pieces. Dec 31 21:06:21.808089556 (629h37m21.679172601s) [noise=true , address=45.91.116.14:4191 , piece_id=XCCCHRXHATZGD2VUXYUMTFZTQ5FACPDX22UG3YSZUW4SJBCBXV5A, seg_pos={0 0}, upload=551730]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Dec 31 21:06:21.808097911 (629h37m21.679180966s) [noise=true , address=45.91.116.14:4191 , piece_id=XCCCHRXHATZGD2VUXYUMTFZTQ5FACPDX22UG3YSZUW4SJBCBXV5A, seg_pos={0 0}, upload=551730]: Long tail cancellation due to optimal threshold Dec 31 21:06:21.808156182 (629h37m21.679239228s) [noise=true , address=45.91.116.7:4011 , piece_id=OODB666DEWAW2CHBCHNETCEHMRLGNHPNGSZMKKWOA7K4TDRT6ZWA, seg_pos={0 0}, upload=551730]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 31 21:06:21.808172042 (629h37m21.679255087s) [noise=true , address=45.91.116.7:4011 , piece_id=OODB666DEWAW2CHBCHNETCEHMRLGNHPNGSZMKKWOA7K4TDRT6ZWA, seg_pos={0 0}, upload=551730]: Long tail cancellation due to optimal threshold Dec 31 21:06:21.808323540 (629h37m21.679406575s) [noise=true , address=145.79.227.44:4193 , piece_id=3JQX6CG4J4QRQYNLVXWOCQXB4WZRR5OPJ5OBIT4JI3JAXJYK733Q, seg_pos={0 0}, upload=551730]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 31 21:06:21.808330124 (629h37m21.679413159s) [noise=true , address=145.79.227.44:4193 , piece_id=3JQX6CG4J4QRQYNLVXWOCQXB4WZRR5OPJ5OBIT4JI3JAXJYK733Q, seg_pos={0 0}, upload=551730]: Long tail cancellation due to optimal threshold Dec 31 21:06:21.808540735 (629h37m21.67962377s) [seg_pos={0 0}, upload=551730]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 21:06:21.808632328 (629h37m21.679715364s) [upload=551730]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 21:06:21.808637007 (629h37m21.679720052s) [upload=551730]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 21:06:22.105013640 (629h37m21.976096675s) [upload=551730]: Done uploading segments... Dec 31 21:06:22.105031735 (629h37m21.97611477s) [upload=551730]: Upload finished. err: <nil> Duration: 5.848743868s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.XX...X..XX...X...XX....X......X....XX........X...X.......X.] duration: 5.845120667s trace: https://jaeger.dev.storj.io/trace/6b4a006b8b7aa49b FAIL (2025-12-31 21:06:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: VVVRZMWTKJ334XCSMYKQECFKFSSMFBRIKTCUJHQZUIFZNHA3ZT5Q) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: H4EO22GSLMHTDAOC4HXKEJ2ZRATC2GHH7YDLJ2PHRJ4FPYP2Q2FQ) context deadline exceeded ======LOGS====== Dec 31 21:06:19.130592534 (629h37m19.001675569s) [upload=551730]: Starting upload Dec 31 21:06:19.130672245 (629h37m19.00175529s) [upload=551730]: Done starting upload Dec 31 21:06:19.130755443 (629h37m19.001838488s) [upload=551730]: Uploading segments... Dec 31 21:06:19.130901432 (629h37m19.001984467s) [upload=551730]: Got next segment. Inline: false Dec 31 21:06:19.130950555 (629h37m19.0020336s) [upload=551730]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 21:06:19.130955284 (629h37m19.002038319s) [upload=551730]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 21:06:19.381665200 (629h37m19.252748265s) [seg_pos={0 0}, upload=551730]: Begin upload segment... Dec 31 21:06:19.381770831 (629h37m19.252853876s) [seg_pos={0 0}, upload=551730]: Done begin upload segment. Dec 31 21:06:19.381851134 (629h37m19.252934179s) [noise=true , address=145.79.227.21:4474 , piece_id=VVRXFDYZR2HSEYIHQ4KRR2TJ7MA45GLXXUAPRCU2QE47YXWP63RA, seg_pos={0 0}, upload=551730]: Uploading piece... Dec 31 21:06:19.385528566 (629h37m19.256611601s) [noise=true , address=45.91.116.8:4045 , piece_id=H4EO22GSLMHTDAOC4HXKEJ2ZRATC2GHH7YDLJ2PHRJ4FPYP2Q2FQ, seg_pos={0 0}, upload=551730]: Uploading piece... Dec 31 21:06:19.386657856 (629h37m19.257740891s) [noise=true , address=145.79.227.8:4331 , piece_id=VEIWQMSN2RVO3HWYKVPO2GRUQBDKOSCVGSLU3TGHCHC6YRXL4HAQ, seg_pos={0 0}, upload=551730]: Uploading piece... Dec 31 21:06:19.387654513 (629h37m19.258737549s) [noise=true , address=145.79.227.36:4519 , piece_id=OCJNGU7QE7TW3VBDQ53TLZXIUEQGBZFYQNDPCNAXX4HLFX2WQYZA, seg_pos={0 0}, upload=551730]: Uploading piece... Dec 31 21:06:19.388655089 (629h37m19.259738134s) [noise=true , address=145.79.227.27:4690 , piece_id=POOEPKVNRGSZ6QHOEVYUCOOUGQZCCPVJFZ5G5GVSZSE6JVC2QGWQ, seg_pos={0 0}, upload=551730]: Uploading piece... Dec 31 21:06:19.390073680 (629h37m19.261156716s) [noise=true , address=145.79.227.2:4115 , piece_id=ZH3P4BLJCINMEOAXB27FGCQHFJXX2AQK5GICCBCVXX4GFXYYTFVQ, seg_pos={0 0}, upload=551730]: Uploading piece... Dec 31 21:06:19.391092029 (629h37m19.262175074s) [noise=true , address=45.91.116.7:4011 , piece_id=OODB666DEWAW2CHBCHNETCEHMRLGNHPNGSZMKKWOA7K4TDRT6ZWA, seg_pos={0 0}, upload=551730]: Uploading piece... Dec 31 21:06:19.392162568 (629h37m19.263245603s) [noise=true , address=45.91.116.14:4191 , piece_id=XCCCHRXHATZGD2VUXYUMTFZTQ5FACPDX22UG3YSZUW4SJBCBXV5A, seg_pos={0 0}, upload=551730]: Uploading piece... Dec 31 21:06:19.393340190 (629h37m19.264423225s) [noise=true , address=145.79.227.17:4970 , piece_id=37X56AFL2F7EY77MOIAMIT7ZWFXHWNZVBK2WHQ5J55UFEQCQIFJQ, seg_pos={0 0}, upload=551730]: Uploading piece... Dec 31 21:06:19.395205221 (629h37m19.266288256s) [noise=true , address=45.91.116.4:4937 , piece_id=C6IM64R7L5IYDPISRRHSDUXD6TFMD6CTH3BNULTVQEFCBKJOGP3A, seg_pos={0 0}, upload=551730]: Uploading piece... Dec 31 21:06:19.396253277 (629h37m19.267336312s) [noise=true , address=45.91.116.10:4082 , piece_id=VVVRZMWTKJ334XCSMYKQECFKFSSMFBRIKTCUJHQZUIFZNHA3ZT5Q, seg_pos={0 0}, upload=551730]: Uploading piece... Dec 31 21:06:19.397275324 (629h37m19.268358359s) [noise=true , address=145.79.227.44:4193 , piece_id=3JQX6CG4J4QRQYNLVXWOCQXB4WZRR5OPJ5OBIT4JI3JAXJYK733Q, seg_pos={0 0}, upload=551730]: Uploading piece... Dec 31 21:06:19.398334380 (629h37m19.269417415s) [noise=true , address=145.79.227.22:4509 , piece_id=XLPZDZSJCBIRKX5LGIKLLP2CVHUM2X5SRBGNA6U2FGCDN2UXIPDQ, seg_pos={0 0}, upload=551730]: Uploading piece... Dec 31 21:06:19.400004671 (629h37m19.271087716s) [noise=true , address=145.79.227.6:4259 , piece_id=2LGQDOPBK4WUOOKTSFTZKHHLFZ3IMRBC2HCHFMCWOKETMZTFEAFA, seg_pos={0 0}, upload=551730]: Uploading piece... Dec 31 21:06:19.401527831 (629h37m19.272610876s) [noise=true , address=77.90.31.18:4808 , piece_id=4FAKZB3YUM4Y7PATZAWHD7SJOCTM4K7G75E7ARWKRRJUYNS3TZAQ, seg_pos={0 0}, upload=551730]: Uploading piece... Dec 31 21:06:19.402400904 (629h37m19.273483949s) [noise=true , address=145.79.227.37:4565 , piece_id=IAPFHAF5L2YZS2KYURPMYX3UT7FVXI7EKYX4QEJ7N7G7VOU3KTNA, seg_pos={0 0}, upload=551730]: Uploading piece... Dec 31 21:06:21.101007053 (629h37m20.972090098s) [upload=551730]: Next returned nil segment Dec 31 21:06:21.101022401 (629h37m20.972105436s) [upload=551730]: Waiting for error group managing segments... Dec 31 21:06:21.385360411 (629h37m21.256443446s) [noise=true , address=145.79.227.8:4331 , piece_id=VEIWQMSN2RVO3HWYKVPO2GRUQBDKOSCVGSLU3TGHCHC6YRXL4HAQ, seg_pos={0 0}, upload=551730]: Done uploading piece. err: <nil> Dec 31 21:06:21.386079980 (629h37m21.257163015s) [noise=true , address=145.79.227.21:4474 , piece_id=VVRXFDYZR2HSEYIHQ4KRR2TJ7MA45GLXXUAPRCU2QE47YXWP63RA, seg_pos={0 0}, upload=551730]: Done uploading piece. err: <nil> Dec 31 21:06:21.392677413 (629h37m21.263760478s) [noise=true , address=145.79.227.27:4690 , piece_id=POOEPKVNRGSZ6QHOEVYUCOOUGQZCCPVJFZ5G5GVSZSE6JVC2QGWQ, seg_pos={0 0}, upload=551730]: Done uploading piece. err: <nil> Dec 31 21:06:21.393084087 (629h37m21.264167122s) [noise=true , address=145.79.227.17:4970 , piece_id=37X56AFL2F7EY77MOIAMIT7ZWFXHWNZVBK2WHQ5J55UFEQCQIFJQ, seg_pos={0 0}, upload=551730]: Done uploading piece. err: <nil> Dec 31 21:06:21.393326629 (629h37m21.264409674s) [noise=true , address=145.79.227.6:4259 , piece_id=2LGQDOPBK4WUOOKTSFTZKHHLFZ3IMRBC2HCHFMCWOKETMZTFEAFA, seg_pos={0 0}, upload=551730]: Done uploading piece. err: <nil> Dec 31 21:06:21.404225282 (629h37m21.275308317s) [noise=true , address=145.79.227.37:4565 , piece_id=IAPFHAF5L2YZS2KYURPMYX3UT7FVXI7EKYX4QEJ7N7G7VOU3KTNA, seg_pos={0 0}, upload=551730]: Done uploading piece. err: <nil> Dec 31 21:06:21.407043318 (629h37m21.278126353s) [noise=true , address=145.79.227.36:4519 , piece_id=OCJNGU7QE7TW3VBDQ53TLZXIUEQGBZFYQNDPCNAXX4HLFX2WQYZA, seg_pos={0 0}, upload=551730]: Done uploading piece. err: <nil> Dec 31 21:06:21.411401707 (629h37m21.282484742s) [noise=true , address=145.79.227.2:4115 , piece_id=ZH3P4BLJCINMEOAXB27FGCQHFJXX2AQK5GICCBCVXX4GFXYYTFVQ, seg_pos={0 0}, upload=551730]: Done uploading piece. err: <nil> Dec 31 21:06:21.415461036 (629h37m21.286544081s) [noise=true , address=145.79.227.22:4509 , piece_id=XLPZDZSJCBIRKX5LGIKLLP2CVHUM2X5SRBGNA6U2FGCDN2UXIPDQ, seg_pos={0 0}, upload=551730]: Done uploading piece. err: <nil> Dec 31 21:06:21.556582891 (629h37m21.427665936s) [noise=true , address=77.90.31.18:4808 , piece_id=4FAKZB3YUM4Y7PATZAWHD7SJOCTM4K7G75E7ARWKRRJUYNS3TZAQ, seg_pos={0 0}, upload=551730]: Done uploading piece. err: <nil> Dec 31 21:06:21.721535795 (629h37m21.59261884s) [noise=true , address=45.91.116.8:4045 , piece_id=H4EO22GSLMHTDAOC4HXKEJ2ZRATC2GHH7YDLJ2PHRJ4FPYP2Q2FQ, seg_pos={0 0}, upload=551730]: Done uploading piece. err: <nil> Dec 31 21:06:21.775331980 (629h37m21.646415016s) [noise=true , address=45.91.116.10:4082 , piece_id=VVVRZMWTKJ334XCSMYKQECFKFSSMFBRIKTCUJHQZUIFZNHA3ZT5Q, seg_pos={0 0}, upload=551730]: Done uploading piece. err: <nil> Dec 31 21:06:21.807810725 (629h37m21.67889377s) [noise=true , address=45.91.116.4:4937 , piece_id=C6IM64R7L5IYDPISRRHSDUXD6TFMD6CTH3BNULTVQEFCBKJOGP3A, seg_pos={0 0}, upload=551730]: Done uploading piece. err: <nil> Dec 31 21:06:21.807850861 (629h37m21.678933916s) [seg_pos={0 0}, upload=551730]: Segment reached optimal threshold of 13 pieces. Dec 31 21:06:21.808089556 (629h37m21.679172601s) [noise=true , address=45.91.116.14:4191 , piece_id=XCCCHRXHATZGD2VUXYUMTFZTQ5FACPDX22UG3YSZUW4SJBCBXV5A, seg_pos={0 0}, upload=551730]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Dec 31 21:06:21.808097911 (629h37m21.679180966s) [noise=true , address=45.91.116.14:4191 , piece_id=XCCCHRXHATZGD2VUXYUMTFZTQ5FACPDX22UG3YSZUW4SJBCBXV5A, seg_pos={0 0}, upload=551730]: Long tail cancellation due to optimal threshold Dec 31 21:06:21.808156182 (629h37m21.679239228s) [noise=true , address=45.91.116.7:4011 , piece_id=OODB666DEWAW2CHBCHNETCEHMRLGNHPNGSZMKKWOA7K4TDRT6ZWA, seg_pos={0 0}, upload=551730]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 31 21:06:21.808172042 (629h37m21.679255087s) [noise=true , address=45.91.116.7:4011 , piece_id=OODB666DEWAW2CHBCHNETCEHMRLGNHPNGSZMKKWOA7K4TDRT6ZWA, seg_pos={0 0}, upload=551730]: Long tail cancellation due to optimal threshold Dec 31 21:06:21.808323540 (629h37m21.679406575s) [noise=true , address=145.79.227.44:4193 , piece_id=3JQX6CG4J4QRQYNLVXWOCQXB4WZRR5OPJ5OBIT4JI3JAXJYK733Q, seg_pos={0 0}, upload=551730]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 31 21:06:21.808330124 (629h37m21.679413159s) [noise=true , address=145.79.227.44:4193 , piece_id=3JQX6CG4J4QRQYNLVXWOCQXB4WZRR5OPJ5OBIT4JI3JAXJYK733Q, seg_pos={0 0}, upload=551730]: Long tail cancellation due to optimal threshold Dec 31 21:06:21.808540735 (629h37m21.67962377s) [seg_pos={0 0}, upload=551730]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 21:06:21.808632328 (629h37m21.679715364s) [upload=551730]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 21:06:21.808637007 (629h37m21.679720052s) [upload=551730]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 21:06:22.105013640 (629h37m21.976096675s) [upload=551730]: Done uploading segments... Dec 31 21:06:22.105031735 (629h37m21.97611477s) [upload=551730]: Upload finished. err: <nil> Duration: 5.845134362s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [..XX...X..XX...X...XX....X......X....XX........X...X.......X] duration: 30.020300035s trace: https://jaeger.dev.storj.io/trace/6b4a006b8b7aa49b FAIL (2025-12-31 21:06:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: VVVRZMWTKJ334XCSMYKQECFKFSSMFBRIKTCUJHQZUIFZNHA3ZT5Q) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: H4EO22GSLMHTDAOC4HXKEJ2ZRATC2GHH7YDLJ2PHRJ4FPYP2Q2FQ) context deadline exceeded ======LOGS====== Dec 31 21:06:19.130592534 (629h37m19.001675569s) [upload=551730]: Starting upload Dec 31 21:06:19.130672245 (629h37m19.00175529s) [upload=551730]: Done starting upload Dec 31 21:06:19.130755443 (629h37m19.001838488s) [upload=551730]: Uploading segments... Dec 31 21:06:19.130901432 (629h37m19.001984467s) [upload=551730]: Got next segment. Inline: false Dec 31 21:06:19.130950555 (629h37m19.0020336s) [upload=551730]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 21:06:19.130955284 (629h37m19.002038319s) [upload=551730]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 21:06:19.381665200 (629h37m19.252748265s) [seg_pos={0 0}, upload=551730]: Begin upload segment... Dec 31 21:06:19.381770831 (629h37m19.252853876s) [seg_pos={0 0}, upload=551730]: Done begin upload segment. Dec 31 21:06:19.381851134 (629h37m19.252934179s) [noise=true , address=145.79.227.21:4474 , piece_id=VVRXFDYZR2HSEYIHQ4KRR2TJ7MA45GLXXUAPRCU2QE47YXWP63RA, seg_pos={0 0}, upload=551730]: Uploading piece... Dec 31 21:06:19.385528566 (629h37m19.256611601s) [noise=true , address=45.91.116.8:4045 , piece_id=H4EO22GSLMHTDAOC4HXKEJ2ZRATC2GHH7YDLJ2PHRJ4FPYP2Q2FQ, seg_pos={0 0}, upload=551730]: Uploading piece... Dec 31 21:06:19.386657856 (629h37m19.257740891s) [noise=true , address=145.79.227.8:4331 , piece_id=VEIWQMSN2RVO3HWYKVPO2GRUQBDKOSCVGSLU3TGHCHC6YRXL4HAQ, seg_pos={0 0}, upload=551730]: Uploading piece... Dec 31 21:06:19.387654513 (629h37m19.258737549s) [noise=true , address=145.79.227.36:4519 , piece_id=OCJNGU7QE7TW3VBDQ53TLZXIUEQGBZFYQNDPCNAXX4HLFX2WQYZA, seg_pos={0 0}, upload=551730]: Uploading piece... Dec 31 21:06:19.388655089 (629h37m19.259738134s) [noise=true , address=145.79.227.27:4690 , piece_id=POOEPKVNRGSZ6QHOEVYUCOOUGQZCCPVJFZ5G5GVSZSE6JVC2QGWQ, seg_pos={0 0}, upload=551730]: Uploading piece... Dec 31 21:06:19.390073680 (629h37m19.261156716s) [noise=true , address=145.79.227.2:4115 , piece_id=ZH3P4BLJCINMEOAXB27FGCQHFJXX2AQK5GICCBCVXX4GFXYYTFVQ, seg_pos={0 0}, upload=551730]: Uploading piece... Dec 31 21:06:19.391092029 (629h37m19.262175074s) [noise=true , address=45.91.116.7:4011 , piece_id=OODB666DEWAW2CHBCHNETCEHMRLGNHPNGSZMKKWOA7K4TDRT6ZWA, seg_pos={0 0}, upload=551730]: Uploading piece... Dec 31 21:06:19.392162568 (629h37m19.263245603s) [noise=true , address=45.91.116.14:4191 , piece_id=XCCCHRXHATZGD2VUXYUMTFZTQ5FACPDX22UG3YSZUW4SJBCBXV5A, seg_pos={0 0}, upload=551730]: Uploading piece... Dec 31 21:06:19.393340190 (629h37m19.264423225s) [noise=true , address=145.79.227.17:4970 , piece_id=37X56AFL2F7EY77MOIAMIT7ZWFXHWNZVBK2WHQ5J55UFEQCQIFJQ, seg_pos={0 0}, upload=551730]: Uploading piece... Dec 31 21:06:19.395205221 (629h37m19.266288256s) [noise=true , address=45.91.116.4:4937 , piece_id=C6IM64R7L5IYDPISRRHSDUXD6TFMD6CTH3BNULTVQEFCBKJOGP3A, seg_pos={0 0}, upload=551730]: Uploading piece... Dec 31 21:06:19.396253277 (629h37m19.267336312s) [noise=true , address=45.91.116.10:4082 , piece_id=VVVRZMWTKJ334XCSMYKQECFKFSSMFBRIKTCUJHQZUIFZNHA3ZT5Q, seg_pos={0 0}, upload=551730]: Uploading piece... Dec 31 21:06:19.397275324 (629h37m19.268358359s) [noise=true , address=145.79.227.44:4193 , piece_id=3JQX6CG4J4QRQYNLVXWOCQXB4WZRR5OPJ5OBIT4JI3JAXJYK733Q, seg_pos={0 0}, upload=551730]: Uploading piece... Dec 31 21:06:19.398334380 (629h37m19.269417415s) [noise=true , address=145.79.227.22:4509 , piece_id=XLPZDZSJCBIRKX5LGIKLLP2CVHUM2X5SRBGNA6U2FGCDN2UXIPDQ, seg_pos={0 0}, upload=551730]: Uploading piece... Dec 31 21:06:19.400004671 (629h37m19.271087716s) [noise=true , address=145.79.227.6:4259 , piece_id=2LGQDOPBK4WUOOKTSFTZKHHLFZ3IMRBC2HCHFMCWOKETMZTFEAFA, seg_pos={0 0}, upload=551730]: Uploading piece... Dec 31 21:06:19.401527831 (629h37m19.272610876s) [noise=true , address=77.90.31.18:4808 , piece_id=4FAKZB3YUM4Y7PATZAWHD7SJOCTM4K7G75E7ARWKRRJUYNS3TZAQ, seg_pos={0 0}, upload=551730]: Uploading piece... Dec 31 21:06:19.402400904 (629h37m19.273483949s) [noise=true , address=145.79.227.37:4565 , piece_id=IAPFHAF5L2YZS2KYURPMYX3UT7FVXI7EKYX4QEJ7N7G7VOU3KTNA, seg_pos={0 0}, upload=551730]: Uploading piece... Dec 31 21:06:21.101007053 (629h37m20.972090098s) [upload=551730]: Next returned nil segment Dec 31 21:06:21.101022401 (629h37m20.972105436s) [upload=551730]: Waiting for error group managing segments... Dec 31 21:06:21.385360411 (629h37m21.256443446s) [noise=true , address=145.79.227.8:4331 , piece_id=VEIWQMSN2RVO3HWYKVPO2GRUQBDKOSCVGSLU3TGHCHC6YRXL4HAQ, seg_pos={0 0}, upload=551730]: Done uploading piece. err: <nil> Dec 31 21:06:21.386079980 (629h37m21.257163015s) [noise=true , address=145.79.227.21:4474 , piece_id=VVRXFDYZR2HSEYIHQ4KRR2TJ7MA45GLXXUAPRCU2QE47YXWP63RA, seg_pos={0 0}, upload=551730]: Done uploading piece. err: <nil> Dec 31 21:06:21.392677413 (629h37m21.263760478s) [noise=true , address=145.79.227.27:4690 , piece_id=POOEPKVNRGSZ6QHOEVYUCOOUGQZCCPVJFZ5G5GVSZSE6JVC2QGWQ, seg_pos={0 0}, upload=551730]: Done uploading piece. err: <nil> Dec 31 21:06:21.393084087 (629h37m21.264167122s) [noise=true , address=145.79.227.17:4970 , piece_id=37X56AFL2F7EY77MOIAMIT7ZWFXHWNZVBK2WHQ5J55UFEQCQIFJQ, seg_pos={0 0}, upload=551730]: Done uploading piece. err: <nil> Dec 31 21:06:21.393326629 (629h37m21.264409674s) [noise=true , address=145.79.227.6:4259 , piece_id=2LGQDOPBK4WUOOKTSFTZKHHLFZ3IMRBC2HCHFMCWOKETMZTFEAFA, seg_pos={0 0}, upload=551730]: Done uploading piece. err: <nil> Dec 31 21:06:21.404225282 (629h37m21.275308317s) [noise=true , address=145.79.227.37:4565 , piece_id=IAPFHAF5L2YZS2KYURPMYX3UT7FVXI7EKYX4QEJ7N7G7VOU3KTNA, seg_pos={0 0}, upload=551730]: Done uploading piece. err: <nil> Dec 31 21:06:21.407043318 (629h37m21.278126353s) [noise=true , address=145.79.227.36:4519 , piece_id=OCJNGU7QE7TW3VBDQ53TLZXIUEQGBZFYQNDPCNAXX4HLFX2WQYZA, seg_pos={0 0}, upload=551730]: Done uploading piece. err: <nil> Dec 31 21:06:21.411401707 (629h37m21.282484742s) [noise=true , address=145.79.227.2:4115 , piece_id=ZH3P4BLJCINMEOAXB27FGCQHFJXX2AQK5GICCBCVXX4GFXYYTFVQ, seg_pos={0 0}, upload=551730]: Done uploading piece. err: <nil> Dec 31 21:06:21.415461036 (629h37m21.286544081s) [noise=true , address=145.79.227.22:4509 , piece_id=XLPZDZSJCBIRKX5LGIKLLP2CVHUM2X5SRBGNA6U2FGCDN2UXIPDQ, seg_pos={0 0}, upload=551730]: Done uploading piece. err: <nil> Dec 31 21:06:21.556582891 (629h37m21.427665936s) [noise=true , address=77.90.31.18:4808 , piece_id=4FAKZB3YUM4Y7PATZAWHD7SJOCTM4K7G75E7ARWKRRJUYNS3TZAQ, seg_pos={0 0}, upload=551730]: Done uploading piece. err: <nil> Dec 31 21:06:21.721535795 (629h37m21.59261884s) [noise=true , address=45.91.116.8:4045 , piece_id=H4EO22GSLMHTDAOC4HXKEJ2ZRATC2GHH7YDLJ2PHRJ4FPYP2Q2FQ, seg_pos={0 0}, upload=551730]: Done uploading piece. err: <nil> Dec 31 21:06:21.775331980 (629h37m21.646415016s) [noise=true , address=45.91.116.10:4082 , piece_id=VVVRZMWTKJ334XCSMYKQECFKFSSMFBRIKTCUJHQZUIFZNHA3ZT5Q, seg_pos={0 0}, upload=551730]: Done uploading piece. err: <nil> Dec 31 21:06:21.807810725 (629h37m21.67889377s) [noise=true , address=45.91.116.4:4937 , piece_id=C6IM64R7L5IYDPISRRHSDUXD6TFMD6CTH3BNULTVQEFCBKJOGP3A, seg_pos={0 0}, upload=551730]: Done uploading piece. err: <nil> Dec 31 21:06:21.807850861 (629h37m21.678933916s) [seg_pos={0 0}, upload=551730]: Segment reached optimal threshold of 13 pieces. Dec 31 21:06:21.808089556 (629h37m21.679172601s) [noise=true , address=45.91.116.14:4191 , piece_id=XCCCHRXHATZGD2VUXYUMTFZTQ5FACPDX22UG3YSZUW4SJBCBXV5A, seg_pos={0 0}, upload=551730]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Dec 31 21:06:21.808097911 (629h37m21.679180966s) [noise=true , address=45.91.116.14:4191 , piece_id=XCCCHRXHATZGD2VUXYUMTFZTQ5FACPDX22UG3YSZUW4SJBCBXV5A, seg_pos={0 0}, upload=551730]: Long tail cancellation due to optimal threshold Dec 31 21:06:21.808156182 (629h37m21.679239228s) [noise=true , address=45.91.116.7:4011 , piece_id=OODB666DEWAW2CHBCHNETCEHMRLGNHPNGSZMKKWOA7K4TDRT6ZWA, seg_pos={0 0}, upload=551730]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 31 21:06:21.808172042 (629h37m21.679255087s) [noise=true , address=45.91.116.7:4011 , piece_id=OODB666DEWAW2CHBCHNETCEHMRLGNHPNGSZMKKWOA7K4TDRT6ZWA, seg_pos={0 0}, upload=551730]: Long tail cancellation due to optimal threshold Dec 31 21:06:21.808323540 (629h37m21.679406575s) [noise=true , address=145.79.227.44:4193 , piece_id=3JQX6CG4J4QRQYNLVXWOCQXB4WZRR5OPJ5OBIT4JI3JAXJYK733Q, seg_pos={0 0}, upload=551730]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 31 21:06:21.808330124 (629h37m21.679413159s) [noise=true , address=145.79.227.44:4193 , piece_id=3JQX6CG4J4QRQYNLVXWOCQXB4WZRR5OPJ5OBIT4JI3JAXJYK733Q, seg_pos={0 0}, upload=551730]: Long tail cancellation due to optimal threshold Dec 31 21:06:21.808540735 (629h37m21.67962377s) [seg_pos={0 0}, upload=551730]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 21:06:21.808632328 (629h37m21.679715364s) [upload=551730]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 21:06:21.808637007 (629h37m21.679720052s) [upload=551730]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 21:06:22.105013640 (629h37m21.976096675s) [upload=551730]: Done uploading segments... Dec 31 21:06:22.105031735 (629h37m21.97611477s) [upload=551730]: Upload finished. err: <nil> Duration: 30.020312078s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [...XX...X..XX...X...XX....X......X....XX........X...X.......] duration: 19.112959622s trace: https://jaeger.dev.storj.io/trace/6a06bfb17a3eab86 FAIL (2025-12-31 21:06:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: N3NUE7USLSFXLOIQX5QK5DSHNEYKJ2ADY5BF2D5XIQLQT3A5QX4Q) context deadline exceeded; piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: XVZ47S6DWTP4RSNSRUYOQU2MXGWNENDQRURCGORTT3VS3EGAAULA) context deadline exceeded ======LOGS====== Dec 31 21:00:33.062484089 (629h31m32.933567134s) [upload=551645]: Starting upload Dec 31 21:00:33.062550977 (629h31m32.933634022s) [upload=551645]: Done starting upload Dec 31 21:00:33.062646960 (629h31m32.933730005s) [upload=551645]: Uploading segments... Dec 31 21:00:33.062867289 (629h31m32.933950344s) [upload=551645]: Got next segment. Inline: false Dec 31 21:00:33.062913737 (629h31m32.933996782s) [upload=551645]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 21:00:33.062920930 (629h31m32.934003975s) [upload=551645]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 21:00:33.308226897 (629h31m33.179309943s) [seg_pos={0 0}, upload=551645]: Begin upload segment... Dec 31 21:00:33.308288425 (629h31m33.17937146s) [seg_pos={0 0}, upload=551645]: Done begin upload segment. Dec 31 21:00:33.308328721 (629h31m33.179411757s) [noise=true , address=145.79.227.14:4838 , piece_id=ATMRWYCCAMQAGYHXEY35VKRGSWMXCDRX7672PAS3TGENEKAX3ERA, seg_pos={0 0}, upload=551645]: Uploading piece... Dec 31 21:00:33.308459671 (629h31m33.179542706s) [noise=true , address=45.91.116.6:4976 , piece_id=ANDKYMH5VNO56ULC2LV3SP4AE4YTRJTIDQBMMAEHA23M3V2XEMOQ, seg_pos={0 0}, upload=551645]: Uploading piece... Dec 31 21:00:33.309041228 (629h31m33.180124273s) [noise=true , address=145.79.227.3:4150 , piece_id=ZL5QB43XTVIH4MYDFDATAOWUXSJ5VGVX3J3ABKXELB2HCAKURVEQ, seg_pos={0 0}, upload=551645]: Uploading piece... Dec 31 21:00:33.309531581 (629h31m33.180614646s) [noise=true , address=77.90.31.18:4808 , piece_id=Z7PKRDZ7VIVO5J3ZAR6REFW762HU6HINQA6G7G22GHZTPJ27QD7A, seg_pos={0 0}, upload=551645]: Uploading piece... Dec 31 21:00:33.309965959 (629h31m33.181048994s) [noise=true , address=145.79.227.6:4258 , piece_id=FOEXYLWMFRDJO6PKPROVWS2IBH5PVNUJTJYPN7ZSIB3I2I5EZDMQ, seg_pos={0 0}, upload=551645]: Uploading piece... Dec 31 21:00:33.310036904 (629h31m33.181119949s) [noise=true , address=145.79.227.4:4185 , piece_id=VSEPIGN7C5WID6ZD3X4724ZJOAVFKAXKXBVN5IRKHUATGFCQBDTQ, seg_pos={0 0}, upload=551645]: Uploading piece... Dec 31 21:00:33.310381490 (629h31m33.181464525s) [noise=true , address=45.91.116.9:4263 , piece_id=ET643B724W5RNM57VSQ3K6LHDFXN5FRIG7B67PVR3YS3EKZVCJPQ, seg_pos={0 0}, upload=551645]: Uploading piece... Dec 31 21:00:33.310445202 (629h31m33.181528247s) [noise=true , address=77.90.31.13:4629 , piece_id=N3NUE7USLSFXLOIQX5QK5DSHNEYKJ2ADY5BF2D5XIQLQT3A5QX4Q, seg_pos={0 0}, upload=551645]: Uploading piece... Dec 31 21:00:33.310688455 (629h31m33.1817715s) [noise=true , address=145.79.227.48:4303 , piece_id=62GX234FSROKN7PTPZC7W57GBK5V4CSONCJXNFJMWVVAX6UDP6YQ, seg_pos={0 0}, upload=551645]: Uploading piece... Dec 31 21:00:33.310892773 (629h31m33.181975818s) [noise=true , address=145.79.227.11:4730 , piece_id=LXHASBRZGSXXTDTIAETVSNH5CKKTOAMNVZH7EY2SRHZPUD5UFZDA, seg_pos={0 0}, upload=551645]: Uploading piece... Dec 31 21:00:33.311087024 (629h31m33.182170069s) [noise=true , address=45.91.116.14:4191 , piece_id=PSS2TMVYGRBI24Y6KB2ICMITG7FASE47AHYCJ6BLO5SRZCRW3B3Q, seg_pos={0 0}, upload=551645]: Uploading piece... Dec 31 21:00:33.311344393 (629h31m33.182427428s) [noise=true , address=45.91.116.10:4083 , piece_id=F4MLMY4SBH5FP3MTAKRIZX74VSSPT75VGXAOEEWNLTXFRN7DVBMA, seg_pos={0 0}, upload=551645]: Uploading piece... Dec 31 21:00:33.311571516 (629h31m33.182654551s) [noise=true , address=145.79.227.52:4446 , piece_id=US2JZVJYXT3XDADF32ZYN4GBDY5D7MLCHJ26SSX2GT4GCT563ZJA, seg_pos={0 0}, upload=551645]: Uploading piece... Dec 31 21:00:33.311836701 (629h31m33.182919747s) [noise=true , address=145.79.227.51:4409 , piece_id=AUH6T5UAXYGTS7TLKM7NKMXTLEO2CUOROEDSQ6KUYQLTVFNSC4DQ, seg_pos={0 0}, upload=551645]: Uploading piece... Dec 31 21:00:33.312324300 (629h31m33.183407335s) [noise=true , address=145.79.227.38:4978 , piece_id=Q4EVAOSFIHXOPTICQ626OQYC2GVY6P4TEXHIRDONWOI4CH7SFZUA, seg_pos={0 0}, upload=551645]: Uploading piece... Dec 31 21:00:33.312852185 (629h31m33.18393523s) [noise=true , address=45.91.116.8:4047 , piece_id=XVZ47S6DWTP4RSNSRUYOQU2MXGWNENDQRURCGORTT3VS3EGAAULA, seg_pos={0 0}, upload=551645]: Uploading piece... Dec 31 21:00:35.000749965 (629h31m34.87183302s) [upload=551645]: Next returned nil segment Dec 31 21:00:35.000788249 (629h31m34.871871294s) [upload=551645]: Waiting for error group managing segments... Dec 31 21:00:35.286902650 (629h31m35.157985695s) [noise=true , address=145.79.227.52:4446 , piece_id=US2JZVJYXT3XDADF32ZYN4GBDY5D7MLCHJ26SSX2GT4GCT563ZJA, seg_pos={0 0}, upload=551645]: Done uploading piece. err: <nil> Dec 31 21:00:35.301466231 (629h31m35.172549276s) [noise=true , address=145.79.227.38:4978 , piece_id=Q4EVAOSFIHXOPTICQ626OQYC2GVY6P4TEXHIRDONWOI4CH7SFZUA, seg_pos={0 0}, upload=551645]: Done uploading piece. err: <nil> Dec 31 21:00:35.303155738 (629h31m35.174238783s) [noise=true , address=145.79.227.6:4258 , piece_id=FOEXYLWMFRDJO6PKPROVWS2IBH5PVNUJTJYPN7ZSIB3I2I5EZDMQ, seg_pos={0 0}, upload=551645]: Done uploading piece. err: <nil> Dec 31 21:00:35.304701741 (629h31m35.175784786s) [noise=true , address=145.79.227.48:4303 , piece_id=62GX234FSROKN7PTPZC7W57GBK5V4CSONCJXNFJMWVVAX6UDP6YQ, seg_pos={0 0}, upload=551645]: Done uploading piece. err: <nil> Dec 31 21:00:35.317064482 (629h31m35.188147517s) [noise=true , address=145.79.227.11:4730 , piece_id=LXHASBRZGSXXTDTIAETVSNH5CKKTOAMNVZH7EY2SRHZPUD5UFZDA, seg_pos={0 0}, upload=551645]: Done uploading piece. err: <nil> Dec 31 21:00:35.326163107 (629h31m35.197246152s) [noise=true , address=145.79.227.14:4838 , piece_id=ATMRWYCCAMQAGYHXEY35VKRGSWMXCDRX7672PAS3TGENEKAX3ERA, seg_pos={0 0}, upload=551645]: Done uploading piece. err: <nil> Dec 31 21:00:35.328699697 (629h31m35.199782732s) [noise=true , address=145.79.227.3:4150 , piece_id=ZL5QB43XTVIH4MYDFDATAOWUXSJ5VGVX3J3ABKXELB2HCAKURVEQ, seg_pos={0 0}, upload=551645]: Done uploading piece. err: <nil> Dec 31 21:00:35.382078097 (629h31m35.253161142s) [noise=true , address=145.79.227.51:4409 , piece_id=AUH6T5UAXYGTS7TLKM7NKMXTLEO2CUOROEDSQ6KUYQLTVFNSC4DQ, seg_pos={0 0}, upload=551645]: Done uploading piece. err: <nil> Dec 31 21:00:35.382112613 (629h31m35.253195658s) [noise=true , address=145.79.227.4:4185 , piece_id=VSEPIGN7C5WID6ZD3X4724ZJOAVFKAXKXBVN5IRKHUATGFCQBDTQ, seg_pos={0 0}, upload=551645]: Done uploading piece. err: <nil> Dec 31 21:00:35.462783822 (629h31m35.333866867s) [noise=true , address=77.90.31.13:4629 , piece_id=N3NUE7USLSFXLOIQX5QK5DSHNEYKJ2ADY5BF2D5XIQLQT3A5QX4Q, seg_pos={0 0}, upload=551645]: Done uploading piece. err: <nil> Dec 31 21:00:35.597934097 (629h31m35.469017132s) [noise=true , address=77.90.31.18:4808 , piece_id=Z7PKRDZ7VIVO5J3ZAR6REFW762HU6HINQA6G7G22GHZTPJ27QD7A, seg_pos={0 0}, upload=551645]: Done uploading piece. err: <nil> Dec 31 21:00:35.623200927 (629h31m35.494283972s) [noise=true , address=45.91.116.6:4976 , piece_id=ANDKYMH5VNO56ULC2LV3SP4AE4YTRJTIDQBMMAEHA23M3V2XEMOQ, seg_pos={0 0}, upload=551645]: Done uploading piece. err: <nil> Dec 31 21:00:35.640167584 (629h31m35.511250629s) [noise=true , address=45.91.116.8:4047 , piece_id=XVZ47S6DWTP4RSNSRUYOQU2MXGWNENDQRURCGORTT3VS3EGAAULA, seg_pos={0 0}, upload=551645]: Done uploading piece. err: <nil> Dec 31 21:00:35.640193945 (629h31m35.51127699s) [seg_pos={0 0}, upload=551645]: Segment reached optimal threshold of 13 pieces. Dec 31 21:00:35.640658019 (629h31m35.511741064s) [noise=true , address=45.91.116.14:4191 , piece_id=PSS2TMVYGRBI24Y6KB2ICMITG7FASE47AHYCJ6BLO5SRZCRW3B3Q, seg_pos={0 0}, upload=551645]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Dec 31 21:00:35.640678898 (629h31m35.511761933s) [noise=true , address=45.91.116.14:4191 , piece_id=PSS2TMVYGRBI24Y6KB2ICMITG7FASE47AHYCJ6BLO5SRZCRW3B3Q, seg_pos={0 0}, upload=551645]: Long tail cancellation due to optimal threshold Dec 31 21:00:35.640863509 (629h31m35.511946544s) [noise=true , address=45.91.116.9:4263 , piece_id=ET643B724W5RNM57VSQ3K6LHDFXN5FRIG7B67PVR3YS3EKZVCJPQ, seg_pos={0 0}, upload=551645]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 31 21:00:35.640874010 (629h31m35.511957055s) [noise=true , address=45.91.116.9:4263 , piece_id=ET643B724W5RNM57VSQ3K6LHDFXN5FRIG7B67PVR3YS3EKZVCJPQ, seg_pos={0 0}, upload=551645]: Long tail cancellation due to optimal threshold Dec 31 21:00:35.641931734 (629h31m35.513014769s) [noise=true , address=45.91.116.10:4083 , piece_id=F4MLMY4SBH5FP3MTAKRIZX74VSSPT75VGXAOEEWNLTXFRN7DVBMA, seg_pos={0 0}, upload=551645]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: context canceled; context canceled Dec 31 21:00:35.641989414 (629h31m35.513072479s) [noise=true , address=45.91.116.10:4083 , piece_id=F4MLMY4SBH5FP3MTAKRIZX74VSSPT75VGXAOEEWNLTXFRN7DVBMA, seg_pos={0 0}, upload=551645]: Long tail cancellation due to optimal threshold Dec 31 21:00:35.642038657 (629h31m35.513121702s) [seg_pos={0 0}, upload=551645]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 21:00:35.642145671 (629h31m35.513228706s) [upload=551645]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 21:00:35.642152544 (629h31m35.513235589s) [upload=551645]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 21:00:35.928264791 (629h31m35.799347826s) [upload=551645]: Done uploading segments... Dec 31 21:00:35.928283407 (629h31m35.799366442s) [upload=551645]: Upload finished. err: <nil> Duration: 19.113008295s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [....XX...X..XX...X...XX....X......X....XX........X...X......] duration: 6.235560071s trace: https://jaeger.dev.storj.io/trace/6a06bfb17a3eab86 FAIL (2025-12-31 21:05:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: N3NUE7USLSFXLOIQX5QK5DSHNEYKJ2ADY5BF2D5XIQLQT3A5QX4Q) context deadline exceeded; piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: XVZ47S6DWTP4RSNSRUYOQU2MXGWNENDQRURCGORTT3VS3EGAAULA) context deadline exceeded ======LOGS====== Dec 31 21:00:33.062484089 (629h31m32.933567134s) [upload=551645]: Starting upload Dec 31 21:00:33.062550977 (629h31m32.933634022s) [upload=551645]: Done starting upload Dec 31 21:00:33.062646960 (629h31m32.933730005s) [upload=551645]: Uploading segments... Dec 31 21:00:33.062867289 (629h31m32.933950344s) [upload=551645]: Got next segment. Inline: false Dec 31 21:00:33.062913737 (629h31m32.933996782s) [upload=551645]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 21:00:33.062920930 (629h31m32.934003975s) [upload=551645]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 21:00:33.308226897 (629h31m33.179309943s) [seg_pos={0 0}, upload=551645]: Begin upload segment... Dec 31 21:00:33.308288425 (629h31m33.17937146s) [seg_pos={0 0}, upload=551645]: Done begin upload segment. Dec 31 21:00:33.308328721 (629h31m33.179411757s) [noise=true , address=145.79.227.14:4838 , piece_id=ATMRWYCCAMQAGYHXEY35VKRGSWMXCDRX7672PAS3TGENEKAX3ERA, seg_pos={0 0}, upload=551645]: Uploading piece... Dec 31 21:00:33.308459671 (629h31m33.179542706s) [noise=true , address=45.91.116.6:4976 , piece_id=ANDKYMH5VNO56ULC2LV3SP4AE4YTRJTIDQBMMAEHA23M3V2XEMOQ, seg_pos={0 0}, upload=551645]: Uploading piece... Dec 31 21:00:33.309041228 (629h31m33.180124273s) [noise=true , address=145.79.227.3:4150 , piece_id=ZL5QB43XTVIH4MYDFDATAOWUXSJ5VGVX3J3ABKXELB2HCAKURVEQ, seg_pos={0 0}, upload=551645]: Uploading piece... Dec 31 21:00:33.309531581 (629h31m33.180614646s) [noise=true , address=77.90.31.18:4808 , piece_id=Z7PKRDZ7VIVO5J3ZAR6REFW762HU6HINQA6G7G22GHZTPJ27QD7A, seg_pos={0 0}, upload=551645]: Uploading piece... Dec 31 21:00:33.309965959 (629h31m33.181048994s) [noise=true , address=145.79.227.6:4258 , piece_id=FOEXYLWMFRDJO6PKPROVWS2IBH5PVNUJTJYPN7ZSIB3I2I5EZDMQ, seg_pos={0 0}, upload=551645]: Uploading piece... Dec 31 21:00:33.310036904 (629h31m33.181119949s) [noise=true , address=145.79.227.4:4185 , piece_id=VSEPIGN7C5WID6ZD3X4724ZJOAVFKAXKXBVN5IRKHUATGFCQBDTQ, seg_pos={0 0}, upload=551645]: Uploading piece... Dec 31 21:00:33.310381490 (629h31m33.181464525s) [noise=true , address=45.91.116.9:4263 , piece_id=ET643B724W5RNM57VSQ3K6LHDFXN5FRIG7B67PVR3YS3EKZVCJPQ, seg_pos={0 0}, upload=551645]: Uploading piece... Dec 31 21:00:33.310445202 (629h31m33.181528247s) [noise=true , address=77.90.31.13:4629 , piece_id=N3NUE7USLSFXLOIQX5QK5DSHNEYKJ2ADY5BF2D5XIQLQT3A5QX4Q, seg_pos={0 0}, upload=551645]: Uploading piece... Dec 31 21:00:33.310688455 (629h31m33.1817715s) [noise=true , address=145.79.227.48:4303 , piece_id=62GX234FSROKN7PTPZC7W57GBK5V4CSONCJXNFJMWVVAX6UDP6YQ, seg_pos={0 0}, upload=551645]: Uploading piece... Dec 31 21:00:33.310892773 (629h31m33.181975818s) [noise=true , address=145.79.227.11:4730 , piece_id=LXHASBRZGSXXTDTIAETVSNH5CKKTOAMNVZH7EY2SRHZPUD5UFZDA, seg_pos={0 0}, upload=551645]: Uploading piece... Dec 31 21:00:33.311087024 (629h31m33.182170069s) [noise=true , address=45.91.116.14:4191 , piece_id=PSS2TMVYGRBI24Y6KB2ICMITG7FASE47AHYCJ6BLO5SRZCRW3B3Q, seg_pos={0 0}, upload=551645]: Uploading piece... Dec 31 21:00:33.311344393 (629h31m33.182427428s) [noise=true , address=45.91.116.10:4083 , piece_id=F4MLMY4SBH5FP3MTAKRIZX74VSSPT75VGXAOEEWNLTXFRN7DVBMA, seg_pos={0 0}, upload=551645]: Uploading piece... Dec 31 21:00:33.311571516 (629h31m33.182654551s) [noise=true , address=145.79.227.52:4446 , piece_id=US2JZVJYXT3XDADF32ZYN4GBDY5D7MLCHJ26SSX2GT4GCT563ZJA, seg_pos={0 0}, upload=551645]: Uploading piece... Dec 31 21:00:33.311836701 (629h31m33.182919747s) [noise=true , address=145.79.227.51:4409 , piece_id=AUH6T5UAXYGTS7TLKM7NKMXTLEO2CUOROEDSQ6KUYQLTVFNSC4DQ, seg_pos={0 0}, upload=551645]: Uploading piece... Dec 31 21:00:33.312324300 (629h31m33.183407335s) [noise=true , address=145.79.227.38:4978 , piece_id=Q4EVAOSFIHXOPTICQ626OQYC2GVY6P4TEXHIRDONWOI4CH7SFZUA, seg_pos={0 0}, upload=551645]: Uploading piece... Dec 31 21:00:33.312852185 (629h31m33.18393523s) [noise=true , address=45.91.116.8:4047 , piece_id=XVZ47S6DWTP4RSNSRUYOQU2MXGWNENDQRURCGORTT3VS3EGAAULA, seg_pos={0 0}, upload=551645]: Uploading piece... Dec 31 21:00:35.000749965 (629h31m34.87183302s) [upload=551645]: Next returned nil segment Dec 31 21:00:35.000788249 (629h31m34.871871294s) [upload=551645]: Waiting for error group managing segments... Dec 31 21:00:35.286902650 (629h31m35.157985695s) [noise=true , address=145.79.227.52:4446 , piece_id=US2JZVJYXT3XDADF32ZYN4GBDY5D7MLCHJ26SSX2GT4GCT563ZJA, seg_pos={0 0}, upload=551645]: Done uploading piece. err: <nil> Dec 31 21:00:35.301466231 (629h31m35.172549276s) [noise=true , address=145.79.227.38:4978 , piece_id=Q4EVAOSFIHXOPTICQ626OQYC2GVY6P4TEXHIRDONWOI4CH7SFZUA, seg_pos={0 0}, upload=551645]: Done uploading piece. err: <nil> Dec 31 21:00:35.303155738 (629h31m35.174238783s) [noise=true , address=145.79.227.6:4258 , piece_id=FOEXYLWMFRDJO6PKPROVWS2IBH5PVNUJTJYPN7ZSIB3I2I5EZDMQ, seg_pos={0 0}, upload=551645]: Done uploading piece. err: <nil> Dec 31 21:00:35.304701741 (629h31m35.175784786s) [noise=true , address=145.79.227.48:4303 , piece_id=62GX234FSROKN7PTPZC7W57GBK5V4CSONCJXNFJMWVVAX6UDP6YQ, seg_pos={0 0}, upload=551645]: Done uploading piece. err: <nil> Dec 31 21:00:35.317064482 (629h31m35.188147517s) [noise=true , address=145.79.227.11:4730 , piece_id=LXHASBRZGSXXTDTIAETVSNH5CKKTOAMNVZH7EY2SRHZPUD5UFZDA, seg_pos={0 0}, upload=551645]: Done uploading piece. err: <nil> Dec 31 21:00:35.326163107 (629h31m35.197246152s) [noise=true , address=145.79.227.14:4838 , piece_id=ATMRWYCCAMQAGYHXEY35VKRGSWMXCDRX7672PAS3TGENEKAX3ERA, seg_pos={0 0}, upload=551645]: Done uploading piece. err: <nil> Dec 31 21:00:35.328699697 (629h31m35.199782732s) [noise=true , address=145.79.227.3:4150 , piece_id=ZL5QB43XTVIH4MYDFDATAOWUXSJ5VGVX3J3ABKXELB2HCAKURVEQ, seg_pos={0 0}, upload=551645]: Done uploading piece. err: <nil> Dec 31 21:00:35.382078097 (629h31m35.253161142s) [noise=true , address=145.79.227.51:4409 , piece_id=AUH6T5UAXYGTS7TLKM7NKMXTLEO2CUOROEDSQ6KUYQLTVFNSC4DQ, seg_pos={0 0}, upload=551645]: Done uploading piece. err: <nil> Dec 31 21:00:35.382112613 (629h31m35.253195658s) [noise=true , address=145.79.227.4:4185 , piece_id=VSEPIGN7C5WID6ZD3X4724ZJOAVFKAXKXBVN5IRKHUATGFCQBDTQ, seg_pos={0 0}, upload=551645]: Done uploading piece. err: <nil> Dec 31 21:00:35.462783822 (629h31m35.333866867s) [noise=true , address=77.90.31.13:4629 , piece_id=N3NUE7USLSFXLOIQX5QK5DSHNEYKJ2ADY5BF2D5XIQLQT3A5QX4Q, seg_pos={0 0}, upload=551645]: Done uploading piece. err: <nil> Dec 31 21:00:35.597934097 (629h31m35.469017132s) [noise=true , address=77.90.31.18:4808 , piece_id=Z7PKRDZ7VIVO5J3ZAR6REFW762HU6HINQA6G7G22GHZTPJ27QD7A, seg_pos={0 0}, upload=551645]: Done uploading piece. err: <nil> Dec 31 21:00:35.623200927 (629h31m35.494283972s) [noise=true , address=45.91.116.6:4976 , piece_id=ANDKYMH5VNO56ULC2LV3SP4AE4YTRJTIDQBMMAEHA23M3V2XEMOQ, seg_pos={0 0}, upload=551645]: Done uploading piece. err: <nil> Dec 31 21:00:35.640167584 (629h31m35.511250629s) [noise=true , address=45.91.116.8:4047 , piece_id=XVZ47S6DWTP4RSNSRUYOQU2MXGWNENDQRURCGORTT3VS3EGAAULA, seg_pos={0 0}, upload=551645]: Done uploading piece. err: <nil> Dec 31 21:00:35.640193945 (629h31m35.51127699s) [seg_pos={0 0}, upload=551645]: Segment reached optimal threshold of 13 pieces. Dec 31 21:00:35.640658019 (629h31m35.511741064s) [noise=true , address=45.91.116.14:4191 , piece_id=PSS2TMVYGRBI24Y6KB2ICMITG7FASE47AHYCJ6BLO5SRZCRW3B3Q, seg_pos={0 0}, upload=551645]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Dec 31 21:00:35.640678898 (629h31m35.511761933s) [noise=true , address=45.91.116.14:4191 , piece_id=PSS2TMVYGRBI24Y6KB2ICMITG7FASE47AHYCJ6BLO5SRZCRW3B3Q, seg_pos={0 0}, upload=551645]: Long tail cancellation due to optimal threshold Dec 31 21:00:35.640863509 (629h31m35.511946544s) [noise=true , address=45.91.116.9:4263 , piece_id=ET643B724W5RNM57VSQ3K6LHDFXN5FRIG7B67PVR3YS3EKZVCJPQ, seg_pos={0 0}, upload=551645]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 31 21:00:35.640874010 (629h31m35.511957055s) [noise=true , address=45.91.116.9:4263 , piece_id=ET643B724W5RNM57VSQ3K6LHDFXN5FRIG7B67PVR3YS3EKZVCJPQ, seg_pos={0 0}, upload=551645]: Long tail cancellation due to optimal threshold Dec 31 21:00:35.641931734 (629h31m35.513014769s) [noise=true , address=45.91.116.10:4083 , piece_id=F4MLMY4SBH5FP3MTAKRIZX74VSSPT75VGXAOEEWNLTXFRN7DVBMA, seg_pos={0 0}, upload=551645]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: context canceled; context canceled Dec 31 21:00:35.641989414 (629h31m35.513072479s) [noise=true , address=45.91.116.10:4083 , piece_id=F4MLMY4SBH5FP3MTAKRIZX74VSSPT75VGXAOEEWNLTXFRN7DVBMA, seg_pos={0 0}, upload=551645]: Long tail cancellation due to optimal threshold Dec 31 21:00:35.642038657 (629h31m35.513121702s) [seg_pos={0 0}, upload=551645]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 21:00:35.642145671 (629h31m35.513228706s) [upload=551645]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 21:00:35.642152544 (629h31m35.513235589s) [upload=551645]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 21:00:35.928264791 (629h31m35.799347826s) [upload=551645]: Done uploading segments... Dec 31 21:00:35.928283407 (629h31m35.799366442s) [upload=551645]: Upload finished. err: <nil> Duration: 6.235593043s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [.....XX...X..XX...X...XX....X......X....XX........X...X.....] duration: 5.746625438s trace: https://jaeger.dev.storj.io/trace/6a06bfb17a3eab86 FAIL (2025-12-31 21:04:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: N3NUE7USLSFXLOIQX5QK5DSHNEYKJ2ADY5BF2D5XIQLQT3A5QX4Q) context deadline exceeded; piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: XVZ47S6DWTP4RSNSRUYOQU2MXGWNENDQRURCGORTT3VS3EGAAULA) context deadline exceeded ======LOGS====== Dec 31 21:00:33.062484089 (629h31m32.933567134s) [upload=551645]: Starting upload Dec 31 21:00:33.062550977 (629h31m32.933634022s) [upload=551645]: Done starting upload Dec 31 21:00:33.062646960 (629h31m32.933730005s) [upload=551645]: Uploading segments... Dec 31 21:00:33.062867289 (629h31m32.933950344s) [upload=551645]: Got next segment. Inline: false Dec 31 21:00:33.062913737 (629h31m32.933996782s) [upload=551645]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 21:00:33.062920930 (629h31m32.934003975s) [upload=551645]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 21:00:33.308226897 (629h31m33.179309943s) [seg_pos={0 0}, upload=551645]: Begin upload segment... Dec 31 21:00:33.308288425 (629h31m33.17937146s) [seg_pos={0 0}, upload=551645]: Done begin upload segment. Dec 31 21:00:33.308328721 (629h31m33.179411757s) [noise=true , address=145.79.227.14:4838 , piece_id=ATMRWYCCAMQAGYHXEY35VKRGSWMXCDRX7672PAS3TGENEKAX3ERA, seg_pos={0 0}, upload=551645]: Uploading piece... Dec 31 21:00:33.308459671 (629h31m33.179542706s) [noise=true , address=45.91.116.6:4976 , piece_id=ANDKYMH5VNO56ULC2LV3SP4AE4YTRJTIDQBMMAEHA23M3V2XEMOQ, seg_pos={0 0}, upload=551645]: Uploading piece... Dec 31 21:00:33.309041228 (629h31m33.180124273s) [noise=true , address=145.79.227.3:4150 , piece_id=ZL5QB43XTVIH4MYDFDATAOWUXSJ5VGVX3J3ABKXELB2HCAKURVEQ, seg_pos={0 0}, upload=551645]: Uploading piece... Dec 31 21:00:33.309531581 (629h31m33.180614646s) [noise=true , address=77.90.31.18:4808 , piece_id=Z7PKRDZ7VIVO5J3ZAR6REFW762HU6HINQA6G7G22GHZTPJ27QD7A, seg_pos={0 0}, upload=551645]: Uploading piece... Dec 31 21:00:33.309965959 (629h31m33.181048994s) [noise=true , address=145.79.227.6:4258 , piece_id=FOEXYLWMFRDJO6PKPROVWS2IBH5PVNUJTJYPN7ZSIB3I2I5EZDMQ, seg_pos={0 0}, upload=551645]: Uploading piece... Dec 31 21:00:33.310036904 (629h31m33.181119949s) [noise=true , address=145.79.227.4:4185 , piece_id=VSEPIGN7C5WID6ZD3X4724ZJOAVFKAXKXBVN5IRKHUATGFCQBDTQ, seg_pos={0 0}, upload=551645]: Uploading piece... Dec 31 21:00:33.310381490 (629h31m33.181464525s) [noise=true , address=45.91.116.9:4263 , piece_id=ET643B724W5RNM57VSQ3K6LHDFXN5FRIG7B67PVR3YS3EKZVCJPQ, seg_pos={0 0}, upload=551645]: Uploading piece... Dec 31 21:00:33.310445202 (629h31m33.181528247s) [noise=true , address=77.90.31.13:4629 , piece_id=N3NUE7USLSFXLOIQX5QK5DSHNEYKJ2ADY5BF2D5XIQLQT3A5QX4Q, seg_pos={0 0}, upload=551645]: Uploading piece... Dec 31 21:00:33.310688455 (629h31m33.1817715s) [noise=true , address=145.79.227.48:4303 , piece_id=62GX234FSROKN7PTPZC7W57GBK5V4CSONCJXNFJMWVVAX6UDP6YQ, seg_pos={0 0}, upload=551645]: Uploading piece... Dec 31 21:00:33.310892773 (629h31m33.181975818s) [noise=true , address=145.79.227.11:4730 , piece_id=LXHASBRZGSXXTDTIAETVSNH5CKKTOAMNVZH7EY2SRHZPUD5UFZDA, seg_pos={0 0}, upload=551645]: Uploading piece... Dec 31 21:00:33.311087024 (629h31m33.182170069s) [noise=true , address=45.91.116.14:4191 , piece_id=PSS2TMVYGRBI24Y6KB2ICMITG7FASE47AHYCJ6BLO5SRZCRW3B3Q, seg_pos={0 0}, upload=551645]: Uploading piece... Dec 31 21:00:33.311344393 (629h31m33.182427428s) [noise=true , address=45.91.116.10:4083 , piece_id=F4MLMY4SBH5FP3MTAKRIZX74VSSPT75VGXAOEEWNLTXFRN7DVBMA, seg_pos={0 0}, upload=551645]: Uploading piece... Dec 31 21:00:33.311571516 (629h31m33.182654551s) [noise=true , address=145.79.227.52:4446 , piece_id=US2JZVJYXT3XDADF32ZYN4GBDY5D7MLCHJ26SSX2GT4GCT563ZJA, seg_pos={0 0}, upload=551645]: Uploading piece... Dec 31 21:00:33.311836701 (629h31m33.182919747s) [noise=true , address=145.79.227.51:4409 , piece_id=AUH6T5UAXYGTS7TLKM7NKMXTLEO2CUOROEDSQ6KUYQLTVFNSC4DQ, seg_pos={0 0}, upload=551645]: Uploading piece... Dec 31 21:00:33.312324300 (629h31m33.183407335s) [noise=true , address=145.79.227.38:4978 , piece_id=Q4EVAOSFIHXOPTICQ626OQYC2GVY6P4TEXHIRDONWOI4CH7SFZUA, seg_pos={0 0}, upload=551645]: Uploading piece... Dec 31 21:00:33.312852185 (629h31m33.18393523s) [noise=true , address=45.91.116.8:4047 , piece_id=XVZ47S6DWTP4RSNSRUYOQU2MXGWNENDQRURCGORTT3VS3EGAAULA, seg_pos={0 0}, upload=551645]: Uploading piece... Dec 31 21:00:35.000749965 (629h31m34.87183302s) [upload=551645]: Next returned nil segment Dec 31 21:00:35.000788249 (629h31m34.871871294s) [upload=551645]: Waiting for error group managing segments... Dec 31 21:00:35.286902650 (629h31m35.157985695s) [noise=true , address=145.79.227.52:4446 , piece_id=US2JZVJYXT3XDADF32ZYN4GBDY5D7MLCHJ26SSX2GT4GCT563ZJA, seg_pos={0 0}, upload=551645]: Done uploading piece. err: <nil> Dec 31 21:00:35.301466231 (629h31m35.172549276s) [noise=true , address=145.79.227.38:4978 , piece_id=Q4EVAOSFIHXOPTICQ626OQYC2GVY6P4TEXHIRDONWOI4CH7SFZUA, seg_pos={0 0}, upload=551645]: Done uploading piece. err: <nil> Dec 31 21:00:35.303155738 (629h31m35.174238783s) [noise=true , address=145.79.227.6:4258 , piece_id=FOEXYLWMFRDJO6PKPROVWS2IBH5PVNUJTJYPN7ZSIB3I2I5EZDMQ, seg_pos={0 0}, upload=551645]: Done uploading piece. err: <nil> Dec 31 21:00:35.304701741 (629h31m35.175784786s) [noise=true , address=145.79.227.48:4303 , piece_id=62GX234FSROKN7PTPZC7W57GBK5V4CSONCJXNFJMWVVAX6UDP6YQ, seg_pos={0 0}, upload=551645]: Done uploading piece. err: <nil> Dec 31 21:00:35.317064482 (629h31m35.188147517s) [noise=true , address=145.79.227.11:4730 , piece_id=LXHASBRZGSXXTDTIAETVSNH5CKKTOAMNVZH7EY2SRHZPUD5UFZDA, seg_pos={0 0}, upload=551645]: Done uploading piece. err: <nil> Dec 31 21:00:35.326163107 (629h31m35.197246152s) [noise=true , address=145.79.227.14:4838 , piece_id=ATMRWYCCAMQAGYHXEY35VKRGSWMXCDRX7672PAS3TGENEKAX3ERA, seg_pos={0 0}, upload=551645]: Done uploading piece. err: <nil> Dec 31 21:00:35.328699697 (629h31m35.199782732s) [noise=true , address=145.79.227.3:4150 , piece_id=ZL5QB43XTVIH4MYDFDATAOWUXSJ5VGVX3J3ABKXELB2HCAKURVEQ, seg_pos={0 0}, upload=551645]: Done uploading piece. err: <nil> Dec 31 21:00:35.382078097 (629h31m35.253161142s) [noise=true , address=145.79.227.51:4409 , piece_id=AUH6T5UAXYGTS7TLKM7NKMXTLEO2CUOROEDSQ6KUYQLTVFNSC4DQ, seg_pos={0 0}, upload=551645]: Done uploading piece. err: <nil> Dec 31 21:00:35.382112613 (629h31m35.253195658s) [noise=true , address=145.79.227.4:4185 , piece_id=VSEPIGN7C5WID6ZD3X4724ZJOAVFKAXKXBVN5IRKHUATGFCQBDTQ, seg_pos={0 0}, upload=551645]: Done uploading piece. err: <nil> Dec 31 21:00:35.462783822 (629h31m35.333866867s) [noise=true , address=77.90.31.13:4629 , piece_id=N3NUE7USLSFXLOIQX5QK5DSHNEYKJ2ADY5BF2D5XIQLQT3A5QX4Q, seg_pos={0 0}, upload=551645]: Done uploading piece. err: <nil> Dec 31 21:00:35.597934097 (629h31m35.469017132s) [noise=true , address=77.90.31.18:4808 , piece_id=Z7PKRDZ7VIVO5J3ZAR6REFW762HU6HINQA6G7G22GHZTPJ27QD7A, seg_pos={0 0}, upload=551645]: Done uploading piece. err: <nil> Dec 31 21:00:35.623200927 (629h31m35.494283972s) [noise=true , address=45.91.116.6:4976 , piece_id=ANDKYMH5VNO56ULC2LV3SP4AE4YTRJTIDQBMMAEHA23M3V2XEMOQ, seg_pos={0 0}, upload=551645]: Done uploading piece. err: <nil> Dec 31 21:00:35.640167584 (629h31m35.511250629s) [noise=true , address=45.91.116.8:4047 , piece_id=XVZ47S6DWTP4RSNSRUYOQU2MXGWNENDQRURCGORTT3VS3EGAAULA, seg_pos={0 0}, upload=551645]: Done uploading piece. err: <nil> Dec 31 21:00:35.640193945 (629h31m35.51127699s) [seg_pos={0 0}, upload=551645]: Segment reached optimal threshold of 13 pieces. Dec 31 21:00:35.640658019 (629h31m35.511741064s) [noise=true , address=45.91.116.14:4191 , piece_id=PSS2TMVYGRBI24Y6KB2ICMITG7FASE47AHYCJ6BLO5SRZCRW3B3Q, seg_pos={0 0}, upload=551645]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Dec 31 21:00:35.640678898 (629h31m35.511761933s) [noise=true , address=45.91.116.14:4191 , piece_id=PSS2TMVYGRBI24Y6KB2ICMITG7FASE47AHYCJ6BLO5SRZCRW3B3Q, seg_pos={0 0}, upload=551645]: Long tail cancellation due to optimal threshold Dec 31 21:00:35.640863509 (629h31m35.511946544s) [noise=true , address=45.91.116.9:4263 , piece_id=ET643B724W5RNM57VSQ3K6LHDFXN5FRIG7B67PVR3YS3EKZVCJPQ, seg_pos={0 0}, upload=551645]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 31 21:00:35.640874010 (629h31m35.511957055s) [noise=true , address=45.91.116.9:4263 , piece_id=ET643B724W5RNM57VSQ3K6LHDFXN5FRIG7B67PVR3YS3EKZVCJPQ, seg_pos={0 0}, upload=551645]: Long tail cancellation due to optimal threshold Dec 31 21:00:35.641931734 (629h31m35.513014769s) [noise=true , address=45.91.116.10:4083 , piece_id=F4MLMY4SBH5FP3MTAKRIZX74VSSPT75VGXAOEEWNLTXFRN7DVBMA, seg_pos={0 0}, upload=551645]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: context canceled; context canceled Dec 31 21:00:35.641989414 (629h31m35.513072479s) [noise=true , address=45.91.116.10:4083 , piece_id=F4MLMY4SBH5FP3MTAKRIZX74VSSPT75VGXAOEEWNLTXFRN7DVBMA, seg_pos={0 0}, upload=551645]: Long tail cancellation due to optimal threshold Dec 31 21:00:35.642038657 (629h31m35.513121702s) [seg_pos={0 0}, upload=551645]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 21:00:35.642145671 (629h31m35.513228706s) [upload=551645]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 21:00:35.642152544 (629h31m35.513235589s) [upload=551645]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 21:00:35.928264791 (629h31m35.799347826s) [upload=551645]: Done uploading segments... Dec 31 21:00:35.928283407 (629h31m35.799366442s) [upload=551645]: Upload finished. err: <nil> Duration: 5.746646207s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [......XX...X..XX...X...XX....X......X....XX........X...X....] duration: 6.753159778s trace: https://jaeger.dev.storj.io/trace/6a06bfb17a3eab86 FAIL (2025-12-31 21:03:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: N3NUE7USLSFXLOIQX5QK5DSHNEYKJ2ADY5BF2D5XIQLQT3A5QX4Q) context deadline exceeded; piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: XVZ47S6DWTP4RSNSRUYOQU2MXGWNENDQRURCGORTT3VS3EGAAULA) context deadline exceeded ======LOGS====== Dec 31 21:00:33.062484089 (629h31m32.933567134s) [upload=551645]: Starting upload Dec 31 21:00:33.062550977 (629h31m32.933634022s) [upload=551645]: Done starting upload Dec 31 21:00:33.062646960 (629h31m32.933730005s) [upload=551645]: Uploading segments... Dec 31 21:00:33.062867289 (629h31m32.933950344s) [upload=551645]: Got next segment. Inline: false Dec 31 21:00:33.062913737 (629h31m32.933996782s) [upload=551645]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 21:00:33.062920930 (629h31m32.934003975s) [upload=551645]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 21:00:33.308226897 (629h31m33.179309943s) [seg_pos={0 0}, upload=551645]: Begin upload segment... Dec 31 21:00:33.308288425 (629h31m33.17937146s) [seg_pos={0 0}, upload=551645]: Done begin upload segment. Dec 31 21:00:33.308328721 (629h31m33.179411757s) [noise=true , address=145.79.227.14:4838 , piece_id=ATMRWYCCAMQAGYHXEY35VKRGSWMXCDRX7672PAS3TGENEKAX3ERA, seg_pos={0 0}, upload=551645]: Uploading piece... Dec 31 21:00:33.308459671 (629h31m33.179542706s) [noise=true , address=45.91.116.6:4976 , piece_id=ANDKYMH5VNO56ULC2LV3SP4AE4YTRJTIDQBMMAEHA23M3V2XEMOQ, seg_pos={0 0}, upload=551645]: Uploading piece... Dec 31 21:00:33.309041228 (629h31m33.180124273s) [noise=true , address=145.79.227.3:4150 , piece_id=ZL5QB43XTVIH4MYDFDATAOWUXSJ5VGVX3J3ABKXELB2HCAKURVEQ, seg_pos={0 0}, upload=551645]: Uploading piece... Dec 31 21:00:33.309531581 (629h31m33.180614646s) [noise=true , address=77.90.31.18:4808 , piece_id=Z7PKRDZ7VIVO5J3ZAR6REFW762HU6HINQA6G7G22GHZTPJ27QD7A, seg_pos={0 0}, upload=551645]: Uploading piece... Dec 31 21:00:33.309965959 (629h31m33.181048994s) [noise=true , address=145.79.227.6:4258 , piece_id=FOEXYLWMFRDJO6PKPROVWS2IBH5PVNUJTJYPN7ZSIB3I2I5EZDMQ, seg_pos={0 0}, upload=551645]: Uploading piece... Dec 31 21:00:33.310036904 (629h31m33.181119949s) [noise=true , address=145.79.227.4:4185 , piece_id=VSEPIGN7C5WID6ZD3X4724ZJOAVFKAXKXBVN5IRKHUATGFCQBDTQ, seg_pos={0 0}, upload=551645]: Uploading piece... Dec 31 21:00:33.310381490 (629h31m33.181464525s) [noise=true , address=45.91.116.9:4263 , piece_id=ET643B724W5RNM57VSQ3K6LHDFXN5FRIG7B67PVR3YS3EKZVCJPQ, seg_pos={0 0}, upload=551645]: Uploading piece... Dec 31 21:00:33.310445202 (629h31m33.181528247s) [noise=true , address=77.90.31.13:4629 , piece_id=N3NUE7USLSFXLOIQX5QK5DSHNEYKJ2ADY5BF2D5XIQLQT3A5QX4Q, seg_pos={0 0}, upload=551645]: Uploading piece... Dec 31 21:00:33.310688455 (629h31m33.1817715s) [noise=true , address=145.79.227.48:4303 , piece_id=62GX234FSROKN7PTPZC7W57GBK5V4CSONCJXNFJMWVVAX6UDP6YQ, seg_pos={0 0}, upload=551645]: Uploading piece... Dec 31 21:00:33.310892773 (629h31m33.181975818s) [noise=true , address=145.79.227.11:4730 , piece_id=LXHASBRZGSXXTDTIAETVSNH5CKKTOAMNVZH7EY2SRHZPUD5UFZDA, seg_pos={0 0}, upload=551645]: Uploading piece... Dec 31 21:00:33.311087024 (629h31m33.182170069s) [noise=true , address=45.91.116.14:4191 , piece_id=PSS2TMVYGRBI24Y6KB2ICMITG7FASE47AHYCJ6BLO5SRZCRW3B3Q, seg_pos={0 0}, upload=551645]: Uploading piece... Dec 31 21:00:33.311344393 (629h31m33.182427428s) [noise=true , address=45.91.116.10:4083 , piece_id=F4MLMY4SBH5FP3MTAKRIZX74VSSPT75VGXAOEEWNLTXFRN7DVBMA, seg_pos={0 0}, upload=551645]: Uploading piece... Dec 31 21:00:33.311571516 (629h31m33.182654551s) [noise=true , address=145.79.227.52:4446 , piece_id=US2JZVJYXT3XDADF32ZYN4GBDY5D7MLCHJ26SSX2GT4GCT563ZJA, seg_pos={0 0}, upload=551645]: Uploading piece... Dec 31 21:00:33.311836701 (629h31m33.182919747s) [noise=true , address=145.79.227.51:4409 , piece_id=AUH6T5UAXYGTS7TLKM7NKMXTLEO2CUOROEDSQ6KUYQLTVFNSC4DQ, seg_pos={0 0}, upload=551645]: Uploading piece... Dec 31 21:00:33.312324300 (629h31m33.183407335s) [noise=true , address=145.79.227.38:4978 , piece_id=Q4EVAOSFIHXOPTICQ626OQYC2GVY6P4TEXHIRDONWOI4CH7SFZUA, seg_pos={0 0}, upload=551645]: Uploading piece... Dec 31 21:00:33.312852185 (629h31m33.18393523s) [noise=true , address=45.91.116.8:4047 , piece_id=XVZ47S6DWTP4RSNSRUYOQU2MXGWNENDQRURCGORTT3VS3EGAAULA, seg_pos={0 0}, upload=551645]: Uploading piece... Dec 31 21:00:35.000749965 (629h31m34.87183302s) [upload=551645]: Next returned nil segment Dec 31 21:00:35.000788249 (629h31m34.871871294s) [upload=551645]: Waiting for error group managing segments... Dec 31 21:00:35.286902650 (629h31m35.157985695s) [noise=true , address=145.79.227.52:4446 , piece_id=US2JZVJYXT3XDADF32ZYN4GBDY5D7MLCHJ26SSX2GT4GCT563ZJA, seg_pos={0 0}, upload=551645]: Done uploading piece. err: <nil> Dec 31 21:00:35.301466231 (629h31m35.172549276s) [noise=true , address=145.79.227.38:4978 , piece_id=Q4EVAOSFIHXOPTICQ626OQYC2GVY6P4TEXHIRDONWOI4CH7SFZUA, seg_pos={0 0}, upload=551645]: Done uploading piece. err: <nil> Dec 31 21:00:35.303155738 (629h31m35.174238783s) [noise=true , address=145.79.227.6:4258 , piece_id=FOEXYLWMFRDJO6PKPROVWS2IBH5PVNUJTJYPN7ZSIB3I2I5EZDMQ, seg_pos={0 0}, upload=551645]: Done uploading piece. err: <nil> Dec 31 21:00:35.304701741 (629h31m35.175784786s) [noise=true , address=145.79.227.48:4303 , piece_id=62GX234FSROKN7PTPZC7W57GBK5V4CSONCJXNFJMWVVAX6UDP6YQ, seg_pos={0 0}, upload=551645]: Done uploading piece. err: <nil> Dec 31 21:00:35.317064482 (629h31m35.188147517s) [noise=true , address=145.79.227.11:4730 , piece_id=LXHASBRZGSXXTDTIAETVSNH5CKKTOAMNVZH7EY2SRHZPUD5UFZDA, seg_pos={0 0}, upload=551645]: Done uploading piece. err: <nil> Dec 31 21:00:35.326163107 (629h31m35.197246152s) [noise=true , address=145.79.227.14:4838 , piece_id=ATMRWYCCAMQAGYHXEY35VKRGSWMXCDRX7672PAS3TGENEKAX3ERA, seg_pos={0 0}, upload=551645]: Done uploading piece. err: <nil> Dec 31 21:00:35.328699697 (629h31m35.199782732s) [noise=true , address=145.79.227.3:4150 , piece_id=ZL5QB43XTVIH4MYDFDATAOWUXSJ5VGVX3J3ABKXELB2HCAKURVEQ, seg_pos={0 0}, upload=551645]: Done uploading piece. err: <nil> Dec 31 21:00:35.382078097 (629h31m35.253161142s) [noise=true , address=145.79.227.51:4409 , piece_id=AUH6T5UAXYGTS7TLKM7NKMXTLEO2CUOROEDSQ6KUYQLTVFNSC4DQ, seg_pos={0 0}, upload=551645]: Done uploading piece. err: <nil> Dec 31 21:00:35.382112613 (629h31m35.253195658s) [noise=true , address=145.79.227.4:4185 , piece_id=VSEPIGN7C5WID6ZD3X4724ZJOAVFKAXKXBVN5IRKHUATGFCQBDTQ, seg_pos={0 0}, upload=551645]: Done uploading piece. err: <nil> Dec 31 21:00:35.462783822 (629h31m35.333866867s) [noise=true , address=77.90.31.13:4629 , piece_id=N3NUE7USLSFXLOIQX5QK5DSHNEYKJ2ADY5BF2D5XIQLQT3A5QX4Q, seg_pos={0 0}, upload=551645]: Done uploading piece. err: <nil> Dec 31 21:00:35.597934097 (629h31m35.469017132s) [noise=true , address=77.90.31.18:4808 , piece_id=Z7PKRDZ7VIVO5J3ZAR6REFW762HU6HINQA6G7G22GHZTPJ27QD7A, seg_pos={0 0}, upload=551645]: Done uploading piece. err: <nil> Dec 31 21:00:35.623200927 (629h31m35.494283972s) [noise=true , address=45.91.116.6:4976 , piece_id=ANDKYMH5VNO56ULC2LV3SP4AE4YTRJTIDQBMMAEHA23M3V2XEMOQ, seg_pos={0 0}, upload=551645]: Done uploading piece. err: <nil> Dec 31 21:00:35.640167584 (629h31m35.511250629s) [noise=true , address=45.91.116.8:4047 , piece_id=XVZ47S6DWTP4RSNSRUYOQU2MXGWNENDQRURCGORTT3VS3EGAAULA, seg_pos={0 0}, upload=551645]: Done uploading piece. err: <nil> Dec 31 21:00:35.640193945 (629h31m35.51127699s) [seg_pos={0 0}, upload=551645]: Segment reached optimal threshold of 13 pieces. Dec 31 21:00:35.640658019 (629h31m35.511741064s) [noise=true , address=45.91.116.14:4191 , piece_id=PSS2TMVYGRBI24Y6KB2ICMITG7FASE47AHYCJ6BLO5SRZCRW3B3Q, seg_pos={0 0}, upload=551645]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Dec 31 21:00:35.640678898 (629h31m35.511761933s) [noise=true , address=45.91.116.14:4191 , piece_id=PSS2TMVYGRBI24Y6KB2ICMITG7FASE47AHYCJ6BLO5SRZCRW3B3Q, seg_pos={0 0}, upload=551645]: Long tail cancellation due to optimal threshold Dec 31 21:00:35.640863509 (629h31m35.511946544s) [noise=true , address=45.91.116.9:4263 , piece_id=ET643B724W5RNM57VSQ3K6LHDFXN5FRIG7B67PVR3YS3EKZVCJPQ, seg_pos={0 0}, upload=551645]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 31 21:00:35.640874010 (629h31m35.511957055s) [noise=true , address=45.91.116.9:4263 , piece_id=ET643B724W5RNM57VSQ3K6LHDFXN5FRIG7B67PVR3YS3EKZVCJPQ, seg_pos={0 0}, upload=551645]: Long tail cancellation due to optimal threshold Dec 31 21:00:35.641931734 (629h31m35.513014769s) [noise=true , address=45.91.116.10:4083 , piece_id=F4MLMY4SBH5FP3MTAKRIZX74VSSPT75VGXAOEEWNLTXFRN7DVBMA, seg_pos={0 0}, upload=551645]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: context canceled; context canceled Dec 31 21:00:35.641989414 (629h31m35.513072479s) [noise=true , address=45.91.116.10:4083 , piece_id=F4MLMY4SBH5FP3MTAKRIZX74VSSPT75VGXAOEEWNLTXFRN7DVBMA, seg_pos={0 0}, upload=551645]: Long tail cancellation due to optimal threshold Dec 31 21:00:35.642038657 (629h31m35.513121702s) [seg_pos={0 0}, upload=551645]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 21:00:35.642145671 (629h31m35.513228706s) [upload=551645]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 21:00:35.642152544 (629h31m35.513235589s) [upload=551645]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 21:00:35.928264791 (629h31m35.799347826s) [upload=551645]: Done uploading segments... Dec 31 21:00:35.928283407 (629h31m35.799366442s) [upload=551645]: Upload finished. err: <nil> Duration: 6.753187491s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [.......XX...X..XX...X...XX....X......X....XX........X...X...] duration: 6.294293544s trace: https://jaeger.dev.storj.io/trace/6a06bfb17a3eab86 FAIL (2025-12-31 21:02:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: N3NUE7USLSFXLOIQX5QK5DSHNEYKJ2ADY5BF2D5XIQLQT3A5QX4Q) context deadline exceeded; piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: XVZ47S6DWTP4RSNSRUYOQU2MXGWNENDQRURCGORTT3VS3EGAAULA) context deadline exceeded ======LOGS====== Dec 31 21:00:33.062484089 (629h31m32.933567134s) [upload=551645]: Starting upload Dec 31 21:00:33.062550977 (629h31m32.933634022s) [upload=551645]: Done starting upload Dec 31 21:00:33.062646960 (629h31m32.933730005s) [upload=551645]: Uploading segments... Dec 31 21:00:33.062867289 (629h31m32.933950344s) [upload=551645]: Got next segment. Inline: false Dec 31 21:00:33.062913737 (629h31m32.933996782s) [upload=551645]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 21:00:33.062920930 (629h31m32.934003975s) [upload=551645]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 21:00:33.308226897 (629h31m33.179309943s) [seg_pos={0 0}, upload=551645]: Begin upload segment... Dec 31 21:00:33.308288425 (629h31m33.17937146s) [seg_pos={0 0}, upload=551645]: Done begin upload segment. Dec 31 21:00:33.308328721 (629h31m33.179411757s) [noise=true , address=145.79.227.14:4838 , piece_id=ATMRWYCCAMQAGYHXEY35VKRGSWMXCDRX7672PAS3TGENEKAX3ERA, seg_pos={0 0}, upload=551645]: Uploading piece... Dec 31 21:00:33.308459671 (629h31m33.179542706s) [noise=true , address=45.91.116.6:4976 , piece_id=ANDKYMH5VNO56ULC2LV3SP4AE4YTRJTIDQBMMAEHA23M3V2XEMOQ, seg_pos={0 0}, upload=551645]: Uploading piece... Dec 31 21:00:33.309041228 (629h31m33.180124273s) [noise=true , address=145.79.227.3:4150 , piece_id=ZL5QB43XTVIH4MYDFDATAOWUXSJ5VGVX3J3ABKXELB2HCAKURVEQ, seg_pos={0 0}, upload=551645]: Uploading piece... Dec 31 21:00:33.309531581 (629h31m33.180614646s) [noise=true , address=77.90.31.18:4808 , piece_id=Z7PKRDZ7VIVO5J3ZAR6REFW762HU6HINQA6G7G22GHZTPJ27QD7A, seg_pos={0 0}, upload=551645]: Uploading piece... Dec 31 21:00:33.309965959 (629h31m33.181048994s) [noise=true , address=145.79.227.6:4258 , piece_id=FOEXYLWMFRDJO6PKPROVWS2IBH5PVNUJTJYPN7ZSIB3I2I5EZDMQ, seg_pos={0 0}, upload=551645]: Uploading piece... Dec 31 21:00:33.310036904 (629h31m33.181119949s) [noise=true , address=145.79.227.4:4185 , piece_id=VSEPIGN7C5WID6ZD3X4724ZJOAVFKAXKXBVN5IRKHUATGFCQBDTQ, seg_pos={0 0}, upload=551645]: Uploading piece... Dec 31 21:00:33.310381490 (629h31m33.181464525s) [noise=true , address=45.91.116.9:4263 , piece_id=ET643B724W5RNM57VSQ3K6LHDFXN5FRIG7B67PVR3YS3EKZVCJPQ, seg_pos={0 0}, upload=551645]: Uploading piece... Dec 31 21:00:33.310445202 (629h31m33.181528247s) [noise=true , address=77.90.31.13:4629 , piece_id=N3NUE7USLSFXLOIQX5QK5DSHNEYKJ2ADY5BF2D5XIQLQT3A5QX4Q, seg_pos={0 0}, upload=551645]: Uploading piece... Dec 31 21:00:33.310688455 (629h31m33.1817715s) [noise=true , address=145.79.227.48:4303 , piece_id=62GX234FSROKN7PTPZC7W57GBK5V4CSONCJXNFJMWVVAX6UDP6YQ, seg_pos={0 0}, upload=551645]: Uploading piece... Dec 31 21:00:33.310892773 (629h31m33.181975818s) [noise=true , address=145.79.227.11:4730 , piece_id=LXHASBRZGSXXTDTIAETVSNH5CKKTOAMNVZH7EY2SRHZPUD5UFZDA, seg_pos={0 0}, upload=551645]: Uploading piece... Dec 31 21:00:33.311087024 (629h31m33.182170069s) [noise=true , address=45.91.116.14:4191 , piece_id=PSS2TMVYGRBI24Y6KB2ICMITG7FASE47AHYCJ6BLO5SRZCRW3B3Q, seg_pos={0 0}, upload=551645]: Uploading piece... Dec 31 21:00:33.311344393 (629h31m33.182427428s) [noise=true , address=45.91.116.10:4083 , piece_id=F4MLMY4SBH5FP3MTAKRIZX74VSSPT75VGXAOEEWNLTXFRN7DVBMA, seg_pos={0 0}, upload=551645]: Uploading piece... Dec 31 21:00:33.311571516 (629h31m33.182654551s) [noise=true , address=145.79.227.52:4446 , piece_id=US2JZVJYXT3XDADF32ZYN4GBDY5D7MLCHJ26SSX2GT4GCT563ZJA, seg_pos={0 0}, upload=551645]: Uploading piece... Dec 31 21:00:33.311836701 (629h31m33.182919747s) [noise=true , address=145.79.227.51:4409 , piece_id=AUH6T5UAXYGTS7TLKM7NKMXTLEO2CUOROEDSQ6KUYQLTVFNSC4DQ, seg_pos={0 0}, upload=551645]: Uploading piece... Dec 31 21:00:33.312324300 (629h31m33.183407335s) [noise=true , address=145.79.227.38:4978 , piece_id=Q4EVAOSFIHXOPTICQ626OQYC2GVY6P4TEXHIRDONWOI4CH7SFZUA, seg_pos={0 0}, upload=551645]: Uploading piece... Dec 31 21:00:33.312852185 (629h31m33.18393523s) [noise=true , address=45.91.116.8:4047 , piece_id=XVZ47S6DWTP4RSNSRUYOQU2MXGWNENDQRURCGORTT3VS3EGAAULA, seg_pos={0 0}, upload=551645]: Uploading piece... Dec 31 21:00:35.000749965 (629h31m34.87183302s) [upload=551645]: Next returned nil segment Dec 31 21:00:35.000788249 (629h31m34.871871294s) [upload=551645]: Waiting for error group managing segments... Dec 31 21:00:35.286902650 (629h31m35.157985695s) [noise=true , address=145.79.227.52:4446 , piece_id=US2JZVJYXT3XDADF32ZYN4GBDY5D7MLCHJ26SSX2GT4GCT563ZJA, seg_pos={0 0}, upload=551645]: Done uploading piece. err: <nil> Dec 31 21:00:35.301466231 (629h31m35.172549276s) [noise=true , address=145.79.227.38:4978 , piece_id=Q4EVAOSFIHXOPTICQ626OQYC2GVY6P4TEXHIRDONWOI4CH7SFZUA, seg_pos={0 0}, upload=551645]: Done uploading piece. err: <nil> Dec 31 21:00:35.303155738 (629h31m35.174238783s) [noise=true , address=145.79.227.6:4258 , piece_id=FOEXYLWMFRDJO6PKPROVWS2IBH5PVNUJTJYPN7ZSIB3I2I5EZDMQ, seg_pos={0 0}, upload=551645]: Done uploading piece. err: <nil> Dec 31 21:00:35.304701741 (629h31m35.175784786s) [noise=true , address=145.79.227.48:4303 , piece_id=62GX234FSROKN7PTPZC7W57GBK5V4CSONCJXNFJMWVVAX6UDP6YQ, seg_pos={0 0}, upload=551645]: Done uploading piece. err: <nil> Dec 31 21:00:35.317064482 (629h31m35.188147517s) [noise=true , address=145.79.227.11:4730 , piece_id=LXHASBRZGSXXTDTIAETVSNH5CKKTOAMNVZH7EY2SRHZPUD5UFZDA, seg_pos={0 0}, upload=551645]: Done uploading piece. err: <nil> Dec 31 21:00:35.326163107 (629h31m35.197246152s) [noise=true , address=145.79.227.14:4838 , piece_id=ATMRWYCCAMQAGYHXEY35VKRGSWMXCDRX7672PAS3TGENEKAX3ERA, seg_pos={0 0}, upload=551645]: Done uploading piece. err: <nil> Dec 31 21:00:35.328699697 (629h31m35.199782732s) [noise=true , address=145.79.227.3:4150 , piece_id=ZL5QB43XTVIH4MYDFDATAOWUXSJ5VGVX3J3ABKXELB2HCAKURVEQ, seg_pos={0 0}, upload=551645]: Done uploading piece. err: <nil> Dec 31 21:00:35.382078097 (629h31m35.253161142s) [noise=true , address=145.79.227.51:4409 , piece_id=AUH6T5UAXYGTS7TLKM7NKMXTLEO2CUOROEDSQ6KUYQLTVFNSC4DQ, seg_pos={0 0}, upload=551645]: Done uploading piece. err: <nil> Dec 31 21:00:35.382112613 (629h31m35.253195658s) [noise=true , address=145.79.227.4:4185 , piece_id=VSEPIGN7C5WID6ZD3X4724ZJOAVFKAXKXBVN5IRKHUATGFCQBDTQ, seg_pos={0 0}, upload=551645]: Done uploading piece. err: <nil> Dec 31 21:00:35.462783822 (629h31m35.333866867s) [noise=true , address=77.90.31.13:4629 , piece_id=N3NUE7USLSFXLOIQX5QK5DSHNEYKJ2ADY5BF2D5XIQLQT3A5QX4Q, seg_pos={0 0}, upload=551645]: Done uploading piece. err: <nil> Dec 31 21:00:35.597934097 (629h31m35.469017132s) [noise=true , address=77.90.31.18:4808 , piece_id=Z7PKRDZ7VIVO5J3ZAR6REFW762HU6HINQA6G7G22GHZTPJ27QD7A, seg_pos={0 0}, upload=551645]: Done uploading piece. err: <nil> Dec 31 21:00:35.623200927 (629h31m35.494283972s) [noise=true , address=45.91.116.6:4976 , piece_id=ANDKYMH5VNO56ULC2LV3SP4AE4YTRJTIDQBMMAEHA23M3V2XEMOQ, seg_pos={0 0}, upload=551645]: Done uploading piece. err: <nil> Dec 31 21:00:35.640167584 (629h31m35.511250629s) [noise=true , address=45.91.116.8:4047 , piece_id=XVZ47S6DWTP4RSNSRUYOQU2MXGWNENDQRURCGORTT3VS3EGAAULA, seg_pos={0 0}, upload=551645]: Done uploading piece. err: <nil> Dec 31 21:00:35.640193945 (629h31m35.51127699s) [seg_pos={0 0}, upload=551645]: Segment reached optimal threshold of 13 pieces. Dec 31 21:00:35.640658019 (629h31m35.511741064s) [noise=true , address=45.91.116.14:4191 , piece_id=PSS2TMVYGRBI24Y6KB2ICMITG7FASE47AHYCJ6BLO5SRZCRW3B3Q, seg_pos={0 0}, upload=551645]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Dec 31 21:00:35.640678898 (629h31m35.511761933s) [noise=true , address=45.91.116.14:4191 , piece_id=PSS2TMVYGRBI24Y6KB2ICMITG7FASE47AHYCJ6BLO5SRZCRW3B3Q, seg_pos={0 0}, upload=551645]: Long tail cancellation due to optimal threshold Dec 31 21:00:35.640863509 (629h31m35.511946544s) [noise=true , address=45.91.116.9:4263 , piece_id=ET643B724W5RNM57VSQ3K6LHDFXN5FRIG7B67PVR3YS3EKZVCJPQ, seg_pos={0 0}, upload=551645]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 31 21:00:35.640874010 (629h31m35.511957055s) [noise=true , address=45.91.116.9:4263 , piece_id=ET643B724W5RNM57VSQ3K6LHDFXN5FRIG7B67PVR3YS3EKZVCJPQ, seg_pos={0 0}, upload=551645]: Long tail cancellation due to optimal threshold Dec 31 21:00:35.641931734 (629h31m35.513014769s) [noise=true , address=45.91.116.10:4083 , piece_id=F4MLMY4SBH5FP3MTAKRIZX74VSSPT75VGXAOEEWNLTXFRN7DVBMA, seg_pos={0 0}, upload=551645]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: context canceled; context canceled Dec 31 21:00:35.641989414 (629h31m35.513072479s) [noise=true , address=45.91.116.10:4083 , piece_id=F4MLMY4SBH5FP3MTAKRIZX74VSSPT75VGXAOEEWNLTXFRN7DVBMA, seg_pos={0 0}, upload=551645]: Long tail cancellation due to optimal threshold Dec 31 21:00:35.642038657 (629h31m35.513121702s) [seg_pos={0 0}, upload=551645]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 21:00:35.642145671 (629h31m35.513228706s) [upload=551645]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 21:00:35.642152544 (629h31m35.513235589s) [upload=551645]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 21:00:35.928264791 (629h31m35.799347826s) [upload=551645]: Done uploading segments... Dec 31 21:00:35.928283407 (629h31m35.799366442s) [upload=551645]: Upload finished. err: <nil> Duration: 6.294332979s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [X.......XX...X..XX...X...XX....X......X....XX........X...X..] duration: 5.864319484s trace: https://jaeger.dev.storj.io/trace/6a06bfb17a3eab86 FAIL (2025-12-31 21:02:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: N3NUE7USLSFXLOIQX5QK5DSHNEYKJ2ADY5BF2D5XIQLQT3A5QX4Q) context deadline exceeded; piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: XVZ47S6DWTP4RSNSRUYOQU2MXGWNENDQRURCGORTT3VS3EGAAULA) context deadline exceeded ======LOGS====== Dec 31 21:00:33.062484089 (629h31m32.933567134s) [upload=551645]: Starting upload Dec 31 21:00:33.062550977 (629h31m32.933634022s) [upload=551645]: Done starting upload Dec 31 21:00:33.062646960 (629h31m32.933730005s) [upload=551645]: Uploading segments... Dec 31 21:00:33.062867289 (629h31m32.933950344s) [upload=551645]: Got next segment. Inline: false Dec 31 21:00:33.062913737 (629h31m32.933996782s) [upload=551645]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 21:00:33.062920930 (629h31m32.934003975s) [upload=551645]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 21:00:33.308226897 (629h31m33.179309943s) [seg_pos={0 0}, upload=551645]: Begin upload segment... Dec 31 21:00:33.308288425 (629h31m33.17937146s) [seg_pos={0 0}, upload=551645]: Done begin upload segment. Dec 31 21:00:33.308328721 (629h31m33.179411757s) [noise=true , address=145.79.227.14:4838 , piece_id=ATMRWYCCAMQAGYHXEY35VKRGSWMXCDRX7672PAS3TGENEKAX3ERA, seg_pos={0 0}, upload=551645]: Uploading piece... Dec 31 21:00:33.308459671 (629h31m33.179542706s) [noise=true , address=45.91.116.6:4976 , piece_id=ANDKYMH5VNO56ULC2LV3SP4AE4YTRJTIDQBMMAEHA23M3V2XEMOQ, seg_pos={0 0}, upload=551645]: Uploading piece... Dec 31 21:00:33.309041228 (629h31m33.180124273s) [noise=true , address=145.79.227.3:4150 , piece_id=ZL5QB43XTVIH4MYDFDATAOWUXSJ5VGVX3J3ABKXELB2HCAKURVEQ, seg_pos={0 0}, upload=551645]: Uploading piece... Dec 31 21:00:33.309531581 (629h31m33.180614646s) [noise=true , address=77.90.31.18:4808 , piece_id=Z7PKRDZ7VIVO5J3ZAR6REFW762HU6HINQA6G7G22GHZTPJ27QD7A, seg_pos={0 0}, upload=551645]: Uploading piece... Dec 31 21:00:33.309965959 (629h31m33.181048994s) [noise=true , address=145.79.227.6:4258 , piece_id=FOEXYLWMFRDJO6PKPROVWS2IBH5PVNUJTJYPN7ZSIB3I2I5EZDMQ, seg_pos={0 0}, upload=551645]: Uploading piece... Dec 31 21:00:33.310036904 (629h31m33.181119949s) [noise=true , address=145.79.227.4:4185 , piece_id=VSEPIGN7C5WID6ZD3X4724ZJOAVFKAXKXBVN5IRKHUATGFCQBDTQ, seg_pos={0 0}, upload=551645]: Uploading piece... Dec 31 21:00:33.310381490 (629h31m33.181464525s) [noise=true , address=45.91.116.9:4263 , piece_id=ET643B724W5RNM57VSQ3K6LHDFXN5FRIG7B67PVR3YS3EKZVCJPQ, seg_pos={0 0}, upload=551645]: Uploading piece... Dec 31 21:00:33.310445202 (629h31m33.181528247s) [noise=true , address=77.90.31.13:4629 , piece_id=N3NUE7USLSFXLOIQX5QK5DSHNEYKJ2ADY5BF2D5XIQLQT3A5QX4Q, seg_pos={0 0}, upload=551645]: Uploading piece... Dec 31 21:00:33.310688455 (629h31m33.1817715s) [noise=true , address=145.79.227.48:4303 , piece_id=62GX234FSROKN7PTPZC7W57GBK5V4CSONCJXNFJMWVVAX6UDP6YQ, seg_pos={0 0}, upload=551645]: Uploading piece... Dec 31 21:00:33.310892773 (629h31m33.181975818s) [noise=true , address=145.79.227.11:4730 , piece_id=LXHASBRZGSXXTDTIAETVSNH5CKKTOAMNVZH7EY2SRHZPUD5UFZDA, seg_pos={0 0}, upload=551645]: Uploading piece... Dec 31 21:00:33.311087024 (629h31m33.182170069s) [noise=true , address=45.91.116.14:4191 , piece_id=PSS2TMVYGRBI24Y6KB2ICMITG7FASE47AHYCJ6BLO5SRZCRW3B3Q, seg_pos={0 0}, upload=551645]: Uploading piece... Dec 31 21:00:33.311344393 (629h31m33.182427428s) [noise=true , address=45.91.116.10:4083 , piece_id=F4MLMY4SBH5FP3MTAKRIZX74VSSPT75VGXAOEEWNLTXFRN7DVBMA, seg_pos={0 0}, upload=551645]: Uploading piece... Dec 31 21:00:33.311571516 (629h31m33.182654551s) [noise=true , address=145.79.227.52:4446 , piece_id=US2JZVJYXT3XDADF32ZYN4GBDY5D7MLCHJ26SSX2GT4GCT563ZJA, seg_pos={0 0}, upload=551645]: Uploading piece... Dec 31 21:00:33.311836701 (629h31m33.182919747s) [noise=true , address=145.79.227.51:4409 , piece_id=AUH6T5UAXYGTS7TLKM7NKMXTLEO2CUOROEDSQ6KUYQLTVFNSC4DQ, seg_pos={0 0}, upload=551645]: Uploading piece... Dec 31 21:00:33.312324300 (629h31m33.183407335s) [noise=true , address=145.79.227.38:4978 , piece_id=Q4EVAOSFIHXOPTICQ626OQYC2GVY6P4TEXHIRDONWOI4CH7SFZUA, seg_pos={0 0}, upload=551645]: Uploading piece... Dec 31 21:00:33.312852185 (629h31m33.18393523s) [noise=true , address=45.91.116.8:4047 , piece_id=XVZ47S6DWTP4RSNSRUYOQU2MXGWNENDQRURCGORTT3VS3EGAAULA, seg_pos={0 0}, upload=551645]: Uploading piece... Dec 31 21:00:35.000749965 (629h31m34.87183302s) [upload=551645]: Next returned nil segment Dec 31 21:00:35.000788249 (629h31m34.871871294s) [upload=551645]: Waiting for error group managing segments... Dec 31 21:00:35.286902650 (629h31m35.157985695s) [noise=true , address=145.79.227.52:4446 , piece_id=US2JZVJYXT3XDADF32ZYN4GBDY5D7MLCHJ26SSX2GT4GCT563ZJA, seg_pos={0 0}, upload=551645]: Done uploading piece. err: <nil> Dec 31 21:00:35.301466231 (629h31m35.172549276s) [noise=true , address=145.79.227.38:4978 , piece_id=Q4EVAOSFIHXOPTICQ626OQYC2GVY6P4TEXHIRDONWOI4CH7SFZUA, seg_pos={0 0}, upload=551645]: Done uploading piece. err: <nil> Dec 31 21:00:35.303155738 (629h31m35.174238783s) [noise=true , address=145.79.227.6:4258 , piece_id=FOEXYLWMFRDJO6PKPROVWS2IBH5PVNUJTJYPN7ZSIB3I2I5EZDMQ, seg_pos={0 0}, upload=551645]: Done uploading piece. err: <nil> Dec 31 21:00:35.304701741 (629h31m35.175784786s) [noise=true , address=145.79.227.48:4303 , piece_id=62GX234FSROKN7PTPZC7W57GBK5V4CSONCJXNFJMWVVAX6UDP6YQ, seg_pos={0 0}, upload=551645]: Done uploading piece. err: <nil> Dec 31 21:00:35.317064482 (629h31m35.188147517s) [noise=true , address=145.79.227.11:4730 , piece_id=LXHASBRZGSXXTDTIAETVSNH5CKKTOAMNVZH7EY2SRHZPUD5UFZDA, seg_pos={0 0}, upload=551645]: Done uploading piece. err: <nil> Dec 31 21:00:35.326163107 (629h31m35.197246152s) [noise=true , address=145.79.227.14:4838 , piece_id=ATMRWYCCAMQAGYHXEY35VKRGSWMXCDRX7672PAS3TGENEKAX3ERA, seg_pos={0 0}, upload=551645]: Done uploading piece. err: <nil> Dec 31 21:00:35.328699697 (629h31m35.199782732s) [noise=true , address=145.79.227.3:4150 , piece_id=ZL5QB43XTVIH4MYDFDATAOWUXSJ5VGVX3J3ABKXELB2HCAKURVEQ, seg_pos={0 0}, upload=551645]: Done uploading piece. err: <nil> Dec 31 21:00:35.382078097 (629h31m35.253161142s) [noise=true , address=145.79.227.51:4409 , piece_id=AUH6T5UAXYGTS7TLKM7NKMXTLEO2CUOROEDSQ6KUYQLTVFNSC4DQ, seg_pos={0 0}, upload=551645]: Done uploading piece. err: <nil> Dec 31 21:00:35.382112613 (629h31m35.253195658s) [noise=true , address=145.79.227.4:4185 , piece_id=VSEPIGN7C5WID6ZD3X4724ZJOAVFKAXKXBVN5IRKHUATGFCQBDTQ, seg_pos={0 0}, upload=551645]: Done uploading piece. err: <nil> Dec 31 21:00:35.462783822 (629h31m35.333866867s) [noise=true , address=77.90.31.13:4629 , piece_id=N3NUE7USLSFXLOIQX5QK5DSHNEYKJ2ADY5BF2D5XIQLQT3A5QX4Q, seg_pos={0 0}, upload=551645]: Done uploading piece. err: <nil> Dec 31 21:00:35.597934097 (629h31m35.469017132s) [noise=true , address=77.90.31.18:4808 , piece_id=Z7PKRDZ7VIVO5J3ZAR6REFW762HU6HINQA6G7G22GHZTPJ27QD7A, seg_pos={0 0}, upload=551645]: Done uploading piece. err: <nil> Dec 31 21:00:35.623200927 (629h31m35.494283972s) [noise=true , address=45.91.116.6:4976 , piece_id=ANDKYMH5VNO56ULC2LV3SP4AE4YTRJTIDQBMMAEHA23M3V2XEMOQ, seg_pos={0 0}, upload=551645]: Done uploading piece. err: <nil> Dec 31 21:00:35.640167584 (629h31m35.511250629s) [noise=true , address=45.91.116.8:4047 , piece_id=XVZ47S6DWTP4RSNSRUYOQU2MXGWNENDQRURCGORTT3VS3EGAAULA, seg_pos={0 0}, upload=551645]: Done uploading piece. err: <nil> Dec 31 21:00:35.640193945 (629h31m35.51127699s) [seg_pos={0 0}, upload=551645]: Segment reached optimal threshold of 13 pieces. Dec 31 21:00:35.640658019 (629h31m35.511741064s) [noise=true , address=45.91.116.14:4191 , piece_id=PSS2TMVYGRBI24Y6KB2ICMITG7FASE47AHYCJ6BLO5SRZCRW3B3Q, seg_pos={0 0}, upload=551645]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Dec 31 21:00:35.640678898 (629h31m35.511761933s) [noise=true , address=45.91.116.14:4191 , piece_id=PSS2TMVYGRBI24Y6KB2ICMITG7FASE47AHYCJ6BLO5SRZCRW3B3Q, seg_pos={0 0}, upload=551645]: Long tail cancellation due to optimal threshold Dec 31 21:00:35.640863509 (629h31m35.511946544s) [noise=true , address=45.91.116.9:4263 , piece_id=ET643B724W5RNM57VSQ3K6LHDFXN5FRIG7B67PVR3YS3EKZVCJPQ, seg_pos={0 0}, upload=551645]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 31 21:00:35.640874010 (629h31m35.511957055s) [noise=true , address=45.91.116.9:4263 , piece_id=ET643B724W5RNM57VSQ3K6LHDFXN5FRIG7B67PVR3YS3EKZVCJPQ, seg_pos={0 0}, upload=551645]: Long tail cancellation due to optimal threshold Dec 31 21:00:35.641931734 (629h31m35.513014769s) [noise=true , address=45.91.116.10:4083 , piece_id=F4MLMY4SBH5FP3MTAKRIZX74VSSPT75VGXAOEEWNLTXFRN7DVBMA, seg_pos={0 0}, upload=551645]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: context canceled; context canceled Dec 31 21:00:35.641989414 (629h31m35.513072479s) [noise=true , address=45.91.116.10:4083 , piece_id=F4MLMY4SBH5FP3MTAKRIZX74VSSPT75VGXAOEEWNLTXFRN7DVBMA, seg_pos={0 0}, upload=551645]: Long tail cancellation due to optimal threshold Dec 31 21:00:35.642038657 (629h31m35.513121702s) [seg_pos={0 0}, upload=551645]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 21:00:35.642145671 (629h31m35.513228706s) [upload=551645]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 21:00:35.642152544 (629h31m35.513235589s) [upload=551645]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 21:00:35.928264791 (629h31m35.799347826s) [upload=551645]: Done uploading segments... Dec 31 21:00:35.928283407 (629h31m35.799366442s) [upload=551645]: Upload finished. err: <nil> Duration: 5.864347768s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.X.......XX...X..XX...X...XX....X......X....XX........X...X.] duration: 6.262931031s trace: https://jaeger.dev.storj.io/trace/6a06bfb17a3eab86 FAIL (2025-12-31 21:01:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: N3NUE7USLSFXLOIQX5QK5DSHNEYKJ2ADY5BF2D5XIQLQT3A5QX4Q) context deadline exceeded; piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: XVZ47S6DWTP4RSNSRUYOQU2MXGWNENDQRURCGORTT3VS3EGAAULA) context deadline exceeded ======LOGS====== Dec 31 21:00:33.062484089 (629h31m32.933567134s) [upload=551645]: Starting upload Dec 31 21:00:33.062550977 (629h31m32.933634022s) [upload=551645]: Done starting upload Dec 31 21:00:33.062646960 (629h31m32.933730005s) [upload=551645]: Uploading segments... Dec 31 21:00:33.062867289 (629h31m32.933950344s) [upload=551645]: Got next segment. Inline: false Dec 31 21:00:33.062913737 (629h31m32.933996782s) [upload=551645]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 21:00:33.062920930 (629h31m32.934003975s) [upload=551645]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 21:00:33.308226897 (629h31m33.179309943s) [seg_pos={0 0}, upload=551645]: Begin upload segment... Dec 31 21:00:33.308288425 (629h31m33.17937146s) [seg_pos={0 0}, upload=551645]: Done begin upload segment. Dec 31 21:00:33.308328721 (629h31m33.179411757s) [noise=true , address=145.79.227.14:4838 , piece_id=ATMRWYCCAMQAGYHXEY35VKRGSWMXCDRX7672PAS3TGENEKAX3ERA, seg_pos={0 0}, upload=551645]: Uploading piece... Dec 31 21:00:33.308459671 (629h31m33.179542706s) [noise=true , address=45.91.116.6:4976 , piece_id=ANDKYMH5VNO56ULC2LV3SP4AE4YTRJTIDQBMMAEHA23M3V2XEMOQ, seg_pos={0 0}, upload=551645]: Uploading piece... Dec 31 21:00:33.309041228 (629h31m33.180124273s) [noise=true , address=145.79.227.3:4150 , piece_id=ZL5QB43XTVIH4MYDFDATAOWUXSJ5VGVX3J3ABKXELB2HCAKURVEQ, seg_pos={0 0}, upload=551645]: Uploading piece... Dec 31 21:00:33.309531581 (629h31m33.180614646s) [noise=true , address=77.90.31.18:4808 , piece_id=Z7PKRDZ7VIVO5J3ZAR6REFW762HU6HINQA6G7G22GHZTPJ27QD7A, seg_pos={0 0}, upload=551645]: Uploading piece... Dec 31 21:00:33.309965959 (629h31m33.181048994s) [noise=true , address=145.79.227.6:4258 , piece_id=FOEXYLWMFRDJO6PKPROVWS2IBH5PVNUJTJYPN7ZSIB3I2I5EZDMQ, seg_pos={0 0}, upload=551645]: Uploading piece... Dec 31 21:00:33.310036904 (629h31m33.181119949s) [noise=true , address=145.79.227.4:4185 , piece_id=VSEPIGN7C5WID6ZD3X4724ZJOAVFKAXKXBVN5IRKHUATGFCQBDTQ, seg_pos={0 0}, upload=551645]: Uploading piece... Dec 31 21:00:33.310381490 (629h31m33.181464525s) [noise=true , address=45.91.116.9:4263 , piece_id=ET643B724W5RNM57VSQ3K6LHDFXN5FRIG7B67PVR3YS3EKZVCJPQ, seg_pos={0 0}, upload=551645]: Uploading piece... Dec 31 21:00:33.310445202 (629h31m33.181528247s) [noise=true , address=77.90.31.13:4629 , piece_id=N3NUE7USLSFXLOIQX5QK5DSHNEYKJ2ADY5BF2D5XIQLQT3A5QX4Q, seg_pos={0 0}, upload=551645]: Uploading piece... Dec 31 21:00:33.310688455 (629h31m33.1817715s) [noise=true , address=145.79.227.48:4303 , piece_id=62GX234FSROKN7PTPZC7W57GBK5V4CSONCJXNFJMWVVAX6UDP6YQ, seg_pos={0 0}, upload=551645]: Uploading piece... Dec 31 21:00:33.310892773 (629h31m33.181975818s) [noise=true , address=145.79.227.11:4730 , piece_id=LXHASBRZGSXXTDTIAETVSNH5CKKTOAMNVZH7EY2SRHZPUD5UFZDA, seg_pos={0 0}, upload=551645]: Uploading piece... Dec 31 21:00:33.311087024 (629h31m33.182170069s) [noise=true , address=45.91.116.14:4191 , piece_id=PSS2TMVYGRBI24Y6KB2ICMITG7FASE47AHYCJ6BLO5SRZCRW3B3Q, seg_pos={0 0}, upload=551645]: Uploading piece... Dec 31 21:00:33.311344393 (629h31m33.182427428s) [noise=true , address=45.91.116.10:4083 , piece_id=F4MLMY4SBH5FP3MTAKRIZX74VSSPT75VGXAOEEWNLTXFRN7DVBMA, seg_pos={0 0}, upload=551645]: Uploading piece... Dec 31 21:00:33.311571516 (629h31m33.182654551s) [noise=true , address=145.79.227.52:4446 , piece_id=US2JZVJYXT3XDADF32ZYN4GBDY5D7MLCHJ26SSX2GT4GCT563ZJA, seg_pos={0 0}, upload=551645]: Uploading piece... Dec 31 21:00:33.311836701 (629h31m33.182919747s) [noise=true , address=145.79.227.51:4409 , piece_id=AUH6T5UAXYGTS7TLKM7NKMXTLEO2CUOROEDSQ6KUYQLTVFNSC4DQ, seg_pos={0 0}, upload=551645]: Uploading piece... Dec 31 21:00:33.312324300 (629h31m33.183407335s) [noise=true , address=145.79.227.38:4978 , piece_id=Q4EVAOSFIHXOPTICQ626OQYC2GVY6P4TEXHIRDONWOI4CH7SFZUA, seg_pos={0 0}, upload=551645]: Uploading piece... Dec 31 21:00:33.312852185 (629h31m33.18393523s) [noise=true , address=45.91.116.8:4047 , piece_id=XVZ47S6DWTP4RSNSRUYOQU2MXGWNENDQRURCGORTT3VS3EGAAULA, seg_pos={0 0}, upload=551645]: Uploading piece... Dec 31 21:00:35.000749965 (629h31m34.87183302s) [upload=551645]: Next returned nil segment Dec 31 21:00:35.000788249 (629h31m34.871871294s) [upload=551645]: Waiting for error group managing segments... Dec 31 21:00:35.286902650 (629h31m35.157985695s) [noise=true , address=145.79.227.52:4446 , piece_id=US2JZVJYXT3XDADF32ZYN4GBDY5D7MLCHJ26SSX2GT4GCT563ZJA, seg_pos={0 0}, upload=551645]: Done uploading piece. err: <nil> Dec 31 21:00:35.301466231 (629h31m35.172549276s) [noise=true , address=145.79.227.38:4978 , piece_id=Q4EVAOSFIHXOPTICQ626OQYC2GVY6P4TEXHIRDONWOI4CH7SFZUA, seg_pos={0 0}, upload=551645]: Done uploading piece. err: <nil> Dec 31 21:00:35.303155738 (629h31m35.174238783s) [noise=true , address=145.79.227.6:4258 , piece_id=FOEXYLWMFRDJO6PKPROVWS2IBH5PVNUJTJYPN7ZSIB3I2I5EZDMQ, seg_pos={0 0}, upload=551645]: Done uploading piece. err: <nil> Dec 31 21:00:35.304701741 (629h31m35.175784786s) [noise=true , address=145.79.227.48:4303 , piece_id=62GX234FSROKN7PTPZC7W57GBK5V4CSONCJXNFJMWVVAX6UDP6YQ, seg_pos={0 0}, upload=551645]: Done uploading piece. err: <nil> Dec 31 21:00:35.317064482 (629h31m35.188147517s) [noise=true , address=145.79.227.11:4730 , piece_id=LXHASBRZGSXXTDTIAETVSNH5CKKTOAMNVZH7EY2SRHZPUD5UFZDA, seg_pos={0 0}, upload=551645]: Done uploading piece. err: <nil> Dec 31 21:00:35.326163107 (629h31m35.197246152s) [noise=true , address=145.79.227.14:4838 , piece_id=ATMRWYCCAMQAGYHXEY35VKRGSWMXCDRX7672PAS3TGENEKAX3ERA, seg_pos={0 0}, upload=551645]: Done uploading piece. err: <nil> Dec 31 21:00:35.328699697 (629h31m35.199782732s) [noise=true , address=145.79.227.3:4150 , piece_id=ZL5QB43XTVIH4MYDFDATAOWUXSJ5VGVX3J3ABKXELB2HCAKURVEQ, seg_pos={0 0}, upload=551645]: Done uploading piece. err: <nil> Dec 31 21:00:35.382078097 (629h31m35.253161142s) [noise=true , address=145.79.227.51:4409 , piece_id=AUH6T5UAXYGTS7TLKM7NKMXTLEO2CUOROEDSQ6KUYQLTVFNSC4DQ, seg_pos={0 0}, upload=551645]: Done uploading piece. err: <nil> Dec 31 21:00:35.382112613 (629h31m35.253195658s) [noise=true , address=145.79.227.4:4185 , piece_id=VSEPIGN7C5WID6ZD3X4724ZJOAVFKAXKXBVN5IRKHUATGFCQBDTQ, seg_pos={0 0}, upload=551645]: Done uploading piece. err: <nil> Dec 31 21:00:35.462783822 (629h31m35.333866867s) [noise=true , address=77.90.31.13:4629 , piece_id=N3NUE7USLSFXLOIQX5QK5DSHNEYKJ2ADY5BF2D5XIQLQT3A5QX4Q, seg_pos={0 0}, upload=551645]: Done uploading piece. err: <nil> Dec 31 21:00:35.597934097 (629h31m35.469017132s) [noise=true , address=77.90.31.18:4808 , piece_id=Z7PKRDZ7VIVO5J3ZAR6REFW762HU6HINQA6G7G22GHZTPJ27QD7A, seg_pos={0 0}, upload=551645]: Done uploading piece. err: <nil> Dec 31 21:00:35.623200927 (629h31m35.494283972s) [noise=true , address=45.91.116.6:4976 , piece_id=ANDKYMH5VNO56ULC2LV3SP4AE4YTRJTIDQBMMAEHA23M3V2XEMOQ, seg_pos={0 0}, upload=551645]: Done uploading piece. err: <nil> Dec 31 21:00:35.640167584 (629h31m35.511250629s) [noise=true , address=45.91.116.8:4047 , piece_id=XVZ47S6DWTP4RSNSRUYOQU2MXGWNENDQRURCGORTT3VS3EGAAULA, seg_pos={0 0}, upload=551645]: Done uploading piece. err: <nil> Dec 31 21:00:35.640193945 (629h31m35.51127699s) [seg_pos={0 0}, upload=551645]: Segment reached optimal threshold of 13 pieces. Dec 31 21:00:35.640658019 (629h31m35.511741064s) [noise=true , address=45.91.116.14:4191 , piece_id=PSS2TMVYGRBI24Y6KB2ICMITG7FASE47AHYCJ6BLO5SRZCRW3B3Q, seg_pos={0 0}, upload=551645]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Dec 31 21:00:35.640678898 (629h31m35.511761933s) [noise=true , address=45.91.116.14:4191 , piece_id=PSS2TMVYGRBI24Y6KB2ICMITG7FASE47AHYCJ6BLO5SRZCRW3B3Q, seg_pos={0 0}, upload=551645]: Long tail cancellation due to optimal threshold Dec 31 21:00:35.640863509 (629h31m35.511946544s) [noise=true , address=45.91.116.9:4263 , piece_id=ET643B724W5RNM57VSQ3K6LHDFXN5FRIG7B67PVR3YS3EKZVCJPQ, seg_pos={0 0}, upload=551645]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 31 21:00:35.640874010 (629h31m35.511957055s) [noise=true , address=45.91.116.9:4263 , piece_id=ET643B724W5RNM57VSQ3K6LHDFXN5FRIG7B67PVR3YS3EKZVCJPQ, seg_pos={0 0}, upload=551645]: Long tail cancellation due to optimal threshold Dec 31 21:00:35.641931734 (629h31m35.513014769s) [noise=true , address=45.91.116.10:4083 , piece_id=F4MLMY4SBH5FP3MTAKRIZX74VSSPT75VGXAOEEWNLTXFRN7DVBMA, seg_pos={0 0}, upload=551645]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: context canceled; context canceled Dec 31 21:00:35.641989414 (629h31m35.513072479s) [noise=true , address=45.91.116.10:4083 , piece_id=F4MLMY4SBH5FP3MTAKRIZX74VSSPT75VGXAOEEWNLTXFRN7DVBMA, seg_pos={0 0}, upload=551645]: Long tail cancellation due to optimal threshold Dec 31 21:00:35.642038657 (629h31m35.513121702s) [seg_pos={0 0}, upload=551645]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 21:00:35.642145671 (629h31m35.513228706s) [upload=551645]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 21:00:35.642152544 (629h31m35.513235589s) [upload=551645]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 21:00:35.928264791 (629h31m35.799347826s) [upload=551645]: Done uploading segments... Dec 31 21:00:35.928283407 (629h31m35.799366442s) [upload=551645]: Upload finished. err: <nil> Duration: 6.262960938s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [..X.......XX...X..XX...X...XX....X......X....XX........X...X] duration: 30.003960626s trace: https://jaeger.dev.storj.io/trace/6a06bfb17a3eab86 FAIL (2025-12-31 21:01:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: N3NUE7USLSFXLOIQX5QK5DSHNEYKJ2ADY5BF2D5XIQLQT3A5QX4Q) context deadline exceeded; piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: XVZ47S6DWTP4RSNSRUYOQU2MXGWNENDQRURCGORTT3VS3EGAAULA) context deadline exceeded ======LOGS====== Dec 31 21:00:33.062484089 (629h31m32.933567134s) [upload=551645]: Starting upload Dec 31 21:00:33.062550977 (629h31m32.933634022s) [upload=551645]: Done starting upload Dec 31 21:00:33.062646960 (629h31m32.933730005s) [upload=551645]: Uploading segments... Dec 31 21:00:33.062867289 (629h31m32.933950344s) [upload=551645]: Got next segment. Inline: false Dec 31 21:00:33.062913737 (629h31m32.933996782s) [upload=551645]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 21:00:33.062920930 (629h31m32.934003975s) [upload=551645]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 21:00:33.308226897 (629h31m33.179309943s) [seg_pos={0 0}, upload=551645]: Begin upload segment... Dec 31 21:00:33.308288425 (629h31m33.17937146s) [seg_pos={0 0}, upload=551645]: Done begin upload segment. Dec 31 21:00:33.308328721 (629h31m33.179411757s) [noise=true , address=145.79.227.14:4838 , piece_id=ATMRWYCCAMQAGYHXEY35VKRGSWMXCDRX7672PAS3TGENEKAX3ERA, seg_pos={0 0}, upload=551645]: Uploading piece... Dec 31 21:00:33.308459671 (629h31m33.179542706s) [noise=true , address=45.91.116.6:4976 , piece_id=ANDKYMH5VNO56ULC2LV3SP4AE4YTRJTIDQBMMAEHA23M3V2XEMOQ, seg_pos={0 0}, upload=551645]: Uploading piece... Dec 31 21:00:33.309041228 (629h31m33.180124273s) [noise=true , address=145.79.227.3:4150 , piece_id=ZL5QB43XTVIH4MYDFDATAOWUXSJ5VGVX3J3ABKXELB2HCAKURVEQ, seg_pos={0 0}, upload=551645]: Uploading piece... Dec 31 21:00:33.309531581 (629h31m33.180614646s) [noise=true , address=77.90.31.18:4808 , piece_id=Z7PKRDZ7VIVO5J3ZAR6REFW762HU6HINQA6G7G22GHZTPJ27QD7A, seg_pos={0 0}, upload=551645]: Uploading piece... Dec 31 21:00:33.309965959 (629h31m33.181048994s) [noise=true , address=145.79.227.6:4258 , piece_id=FOEXYLWMFRDJO6PKPROVWS2IBH5PVNUJTJYPN7ZSIB3I2I5EZDMQ, seg_pos={0 0}, upload=551645]: Uploading piece... Dec 31 21:00:33.310036904 (629h31m33.181119949s) [noise=true , address=145.79.227.4:4185 , piece_id=VSEPIGN7C5WID6ZD3X4724ZJOAVFKAXKXBVN5IRKHUATGFCQBDTQ, seg_pos={0 0}, upload=551645]: Uploading piece... Dec 31 21:00:33.310381490 (629h31m33.181464525s) [noise=true , address=45.91.116.9:4263 , piece_id=ET643B724W5RNM57VSQ3K6LHDFXN5FRIG7B67PVR3YS3EKZVCJPQ, seg_pos={0 0}, upload=551645]: Uploading piece... Dec 31 21:00:33.310445202 (629h31m33.181528247s) [noise=true , address=77.90.31.13:4629 , piece_id=N3NUE7USLSFXLOIQX5QK5DSHNEYKJ2ADY5BF2D5XIQLQT3A5QX4Q, seg_pos={0 0}, upload=551645]: Uploading piece... Dec 31 21:00:33.310688455 (629h31m33.1817715s) [noise=true , address=145.79.227.48:4303 , piece_id=62GX234FSROKN7PTPZC7W57GBK5V4CSONCJXNFJMWVVAX6UDP6YQ, seg_pos={0 0}, upload=551645]: Uploading piece... Dec 31 21:00:33.310892773 (629h31m33.181975818s) [noise=true , address=145.79.227.11:4730 , piece_id=LXHASBRZGSXXTDTIAETVSNH5CKKTOAMNVZH7EY2SRHZPUD5UFZDA, seg_pos={0 0}, upload=551645]: Uploading piece... Dec 31 21:00:33.311087024 (629h31m33.182170069s) [noise=true , address=45.91.116.14:4191 , piece_id=PSS2TMVYGRBI24Y6KB2ICMITG7FASE47AHYCJ6BLO5SRZCRW3B3Q, seg_pos={0 0}, upload=551645]: Uploading piece... Dec 31 21:00:33.311344393 (629h31m33.182427428s) [noise=true , address=45.91.116.10:4083 , piece_id=F4MLMY4SBH5FP3MTAKRIZX74VSSPT75VGXAOEEWNLTXFRN7DVBMA, seg_pos={0 0}, upload=551645]: Uploading piece... Dec 31 21:00:33.311571516 (629h31m33.182654551s) [noise=true , address=145.79.227.52:4446 , piece_id=US2JZVJYXT3XDADF32ZYN4GBDY5D7MLCHJ26SSX2GT4GCT563ZJA, seg_pos={0 0}, upload=551645]: Uploading piece... Dec 31 21:00:33.311836701 (629h31m33.182919747s) [noise=true , address=145.79.227.51:4409 , piece_id=AUH6T5UAXYGTS7TLKM7NKMXTLEO2CUOROEDSQ6KUYQLTVFNSC4DQ, seg_pos={0 0}, upload=551645]: Uploading piece... Dec 31 21:00:33.312324300 (629h31m33.183407335s) [noise=true , address=145.79.227.38:4978 , piece_id=Q4EVAOSFIHXOPTICQ626OQYC2GVY6P4TEXHIRDONWOI4CH7SFZUA, seg_pos={0 0}, upload=551645]: Uploading piece... Dec 31 21:00:33.312852185 (629h31m33.18393523s) [noise=true , address=45.91.116.8:4047 , piece_id=XVZ47S6DWTP4RSNSRUYOQU2MXGWNENDQRURCGORTT3VS3EGAAULA, seg_pos={0 0}, upload=551645]: Uploading piece... Dec 31 21:00:35.000749965 (629h31m34.87183302s) [upload=551645]: Next returned nil segment Dec 31 21:00:35.000788249 (629h31m34.871871294s) [upload=551645]: Waiting for error group managing segments... Dec 31 21:00:35.286902650 (629h31m35.157985695s) [noise=true , address=145.79.227.52:4446 , piece_id=US2JZVJYXT3XDADF32ZYN4GBDY5D7MLCHJ26SSX2GT4GCT563ZJA, seg_pos={0 0}, upload=551645]: Done uploading piece. err: <nil> Dec 31 21:00:35.301466231 (629h31m35.172549276s) [noise=true , address=145.79.227.38:4978 , piece_id=Q4EVAOSFIHXOPTICQ626OQYC2GVY6P4TEXHIRDONWOI4CH7SFZUA, seg_pos={0 0}, upload=551645]: Done uploading piece. err: <nil> Dec 31 21:00:35.303155738 (629h31m35.174238783s) [noise=true , address=145.79.227.6:4258 , piece_id=FOEXYLWMFRDJO6PKPROVWS2IBH5PVNUJTJYPN7ZSIB3I2I5EZDMQ, seg_pos={0 0}, upload=551645]: Done uploading piece. err: <nil> Dec 31 21:00:35.304701741 (629h31m35.175784786s) [noise=true , address=145.79.227.48:4303 , piece_id=62GX234FSROKN7PTPZC7W57GBK5V4CSONCJXNFJMWVVAX6UDP6YQ, seg_pos={0 0}, upload=551645]: Done uploading piece. err: <nil> Dec 31 21:00:35.317064482 (629h31m35.188147517s) [noise=true , address=145.79.227.11:4730 , piece_id=LXHASBRZGSXXTDTIAETVSNH5CKKTOAMNVZH7EY2SRHZPUD5UFZDA, seg_pos={0 0}, upload=551645]: Done uploading piece. err: <nil> Dec 31 21:00:35.326163107 (629h31m35.197246152s) [noise=true , address=145.79.227.14:4838 , piece_id=ATMRWYCCAMQAGYHXEY35VKRGSWMXCDRX7672PAS3TGENEKAX3ERA, seg_pos={0 0}, upload=551645]: Done uploading piece. err: <nil> Dec 31 21:00:35.328699697 (629h31m35.199782732s) [noise=true , address=145.79.227.3:4150 , piece_id=ZL5QB43XTVIH4MYDFDATAOWUXSJ5VGVX3J3ABKXELB2HCAKURVEQ, seg_pos={0 0}, upload=551645]: Done uploading piece. err: <nil> Dec 31 21:00:35.382078097 (629h31m35.253161142s) [noise=true , address=145.79.227.51:4409 , piece_id=AUH6T5UAXYGTS7TLKM7NKMXTLEO2CUOROEDSQ6KUYQLTVFNSC4DQ, seg_pos={0 0}, upload=551645]: Done uploading piece. err: <nil> Dec 31 21:00:35.382112613 (629h31m35.253195658s) [noise=true , address=145.79.227.4:4185 , piece_id=VSEPIGN7C5WID6ZD3X4724ZJOAVFKAXKXBVN5IRKHUATGFCQBDTQ, seg_pos={0 0}, upload=551645]: Done uploading piece. err: <nil> Dec 31 21:00:35.462783822 (629h31m35.333866867s) [noise=true , address=77.90.31.13:4629 , piece_id=N3NUE7USLSFXLOIQX5QK5DSHNEYKJ2ADY5BF2D5XIQLQT3A5QX4Q, seg_pos={0 0}, upload=551645]: Done uploading piece. err: <nil> Dec 31 21:00:35.597934097 (629h31m35.469017132s) [noise=true , address=77.90.31.18:4808 , piece_id=Z7PKRDZ7VIVO5J3ZAR6REFW762HU6HINQA6G7G22GHZTPJ27QD7A, seg_pos={0 0}, upload=551645]: Done uploading piece. err: <nil> Dec 31 21:00:35.623200927 (629h31m35.494283972s) [noise=true , address=45.91.116.6:4976 , piece_id=ANDKYMH5VNO56ULC2LV3SP4AE4YTRJTIDQBMMAEHA23M3V2XEMOQ, seg_pos={0 0}, upload=551645]: Done uploading piece. err: <nil> Dec 31 21:00:35.640167584 (629h31m35.511250629s) [noise=true , address=45.91.116.8:4047 , piece_id=XVZ47S6DWTP4RSNSRUYOQU2MXGWNENDQRURCGORTT3VS3EGAAULA, seg_pos={0 0}, upload=551645]: Done uploading piece. err: <nil> Dec 31 21:00:35.640193945 (629h31m35.51127699s) [seg_pos={0 0}, upload=551645]: Segment reached optimal threshold of 13 pieces. Dec 31 21:00:35.640658019 (629h31m35.511741064s) [noise=true , address=45.91.116.14:4191 , piece_id=PSS2TMVYGRBI24Y6KB2ICMITG7FASE47AHYCJ6BLO5SRZCRW3B3Q, seg_pos={0 0}, upload=551645]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Dec 31 21:00:35.640678898 (629h31m35.511761933s) [noise=true , address=45.91.116.14:4191 , piece_id=PSS2TMVYGRBI24Y6KB2ICMITG7FASE47AHYCJ6BLO5SRZCRW3B3Q, seg_pos={0 0}, upload=551645]: Long tail cancellation due to optimal threshold Dec 31 21:00:35.640863509 (629h31m35.511946544s) [noise=true , address=45.91.116.9:4263 , piece_id=ET643B724W5RNM57VSQ3K6LHDFXN5FRIG7B67PVR3YS3EKZVCJPQ, seg_pos={0 0}, upload=551645]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 31 21:00:35.640874010 (629h31m35.511957055s) [noise=true , address=45.91.116.9:4263 , piece_id=ET643B724W5RNM57VSQ3K6LHDFXN5FRIG7B67PVR3YS3EKZVCJPQ, seg_pos={0 0}, upload=551645]: Long tail cancellation due to optimal threshold Dec 31 21:00:35.641931734 (629h31m35.513014769s) [noise=true , address=45.91.116.10:4083 , piece_id=F4MLMY4SBH5FP3MTAKRIZX74VSSPT75VGXAOEEWNLTXFRN7DVBMA, seg_pos={0 0}, upload=551645]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: context canceled; context canceled Dec 31 21:00:35.641989414 (629h31m35.513072479s) [noise=true , address=45.91.116.10:4083 , piece_id=F4MLMY4SBH5FP3MTAKRIZX74VSSPT75VGXAOEEWNLTXFRN7DVBMA, seg_pos={0 0}, upload=551645]: Long tail cancellation due to optimal threshold Dec 31 21:00:35.642038657 (629h31m35.513121702s) [seg_pos={0 0}, upload=551645]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 21:00:35.642145671 (629h31m35.513228706s) [upload=551645]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 21:00:35.642152544 (629h31m35.513235589s) [upload=551645]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 21:00:35.928264791 (629h31m35.799347826s) [upload=551645]: Done uploading segments... Dec 31 21:00:35.928283407 (629h31m35.799366442s) [upload=551645]: Upload finished. err: <nil> Duration: 30.00398916s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [...X.......XX...X..XX...X...XX....X......X....XX........X...] duration: 17.440146851s trace: https://jaeger.dev.storj.io/trace/62e53f89d015a867 FAIL (2025-12-31 21:00:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, Piece ID: 2M6QALYWFXXX4J4OPQV2QZRQZJENFSG7G73DXURKWRBBSMI7YOIA) context deadline exceeded; piecestore: (Node ID: 12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, Piece ID: Y36LIA6U3G2CEV7HQWQRME5EA2GKA3QPOWI6ZP5EQAQCWC56KW6Q) context deadline exceeded ======LOGS====== Dec 31 20:57:47.327402587 (629h28m47.198485632s) [upload=551605]: Starting upload Dec 31 20:57:47.327456269 (629h28m47.198539304s) [upload=551605]: Done starting upload Dec 31 20:57:47.327528276 (629h28m47.198611312s) [upload=551605]: Uploading segments... Dec 31 20:57:47.327633827 (629h28m47.198716862s) [upload=551605]: Got next segment. Inline: false Dec 31 20:57:47.327661971 (629h28m47.198745006s) [upload=551605]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 20:57:47.327666089 (629h28m47.198749124s) [upload=551605]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 20:57:47.578796046 (629h28m47.449879101s) [seg_pos={0 0}, upload=551605]: Begin upload segment... Dec 31 20:57:47.578849297 (629h28m47.449932332s) [seg_pos={0 0}, upload=551605]: Done begin upload segment. Dec 31 20:57:47.578884985 (629h28m47.44996802s) [noise=true , address=145.79.227.23:4547 , piece_id=LGROBKN2JBULMYC7MXDOYBP744ASAPVABA3H5W6EUZLWL5EIJ6DA, seg_pos={0 0}, upload=551605]: Uploading piece... Dec 31 20:57:47.578899523 (629h28m47.449982568s) [noise=true , address=45.91.116.13:4154 , piece_id=Q5WKC5UOGLRDPEMMWNKRXTZEFRHYYE5PRNWBDQAZQHFQ343SYRPQ, seg_pos={0 0}, upload=551605]: Uploading piece... Dec 31 20:57:47.579003671 (629h28m47.450086706s) [noise=true , address=45.91.116.5:4300 , piece_id=EUOXBKHHU3ZIE4XO6TSEJKHGRZQ6ELEOUIM4MCPR3VPJBB6O5E2Q, seg_pos={0 0}, upload=551605]: Uploading piece... Dec 31 20:57:47.579702872 (629h28m47.450785917s) [noise=true , address=145.79.227.52:4446 , piece_id=K6QZ3I7XMKB52VDX47M5PN7TLGZKFNDALS25FS2QB7TFEQU2FEAQ, seg_pos={0 0}, upload=551605]: Uploading piece... Dec 31 20:57:47.580341078 (629h28m47.451424123s) [noise=true , address=45.91.116.10:4081 , piece_id=LJGGEMYZTSJZXXZN7R47LVFJFLDN4E7D4MMDM4MFLQSCJOFSLBTA, seg_pos={0 0}, upload=551605]: Uploading piece... Dec 31 20:57:47.580797046 (629h28m47.451880081s) [noise=true , address=145.79.227.48:4303 , piece_id=UMAHKL7H3VYZEMTAEAXZTJBE6YMGRAPXKSMBNU7CLRJCVAP4MJXQ, seg_pos={0 0}, upload=551605]: Uploading piece... Dec 31 20:57:47.581194753 (629h28m47.452277798s) [noise=true , address=145.79.227.7:4295 , piece_id=HO2EQPC7MDGVZGWWCZIXXCCAAUQTLL2WGD5AQUQBTGSOJG62SDTA, seg_pos={0 0}, upload=551605]: Uploading piece... Dec 31 20:57:47.581571922 (629h28m47.452654957s) [noise=true , address=145.79.227.51:4409 , piece_id=Y47GETDCED7TXBB46AKCQFFBN53R6F27FSADUPWHYTNOYEYSZHQQ, seg_pos={0 0}, upload=551605]: Uploading piece... Dec 31 20:57:47.582067826 (629h28m47.453150871s) [noise=true , address=77.90.31.19:4841 , piece_id=2M6QALYWFXXX4J4OPQV2QZRQZJENFSG7G73DXURKWRBBSMI7YOIA, seg_pos={0 0}, upload=551605]: Uploading piece... Dec 31 20:57:47.582519295 (629h28m47.45360234s) [noise=true , address=77.90.31.21:4877 , piece_id=Y36LIA6U3G2CEV7HQWQRME5EA2GKA3QPOWI6ZP5EQAQCWC56KW6Q, seg_pos={0 0}, upload=551605]: Uploading piece... Dec 31 20:57:47.584070550 (629h28m47.455153585s) [noise=true , address=145.79.227.37:4565 , piece_id=7YB7O2IZHG67C2K437BFWGEPQJEMNRBNHD2VL7VDCZB4XQLTJ53A, seg_pos={0 0}, upload=551605]: Uploading piece... Dec 31 20:57:47.584613043 (629h28m47.455696088s) [noise=true , address=145.79.227.8:4330 , piece_id=WHZY3ZBXCHKXTL5PCYJGHMDFJ2K23HBVYDHXHYA3GYUJU46FFFXA, seg_pos={0 0}, upload=551605]: Uploading piece... Dec 31 20:57:47.586283874 (629h28m47.457366919s) [noise=true , address=145.79.227.38:4977 , piece_id=5JK6U5TLD6AOB4F257BSBLVLDE7ZGIH2MSHMUQJNLSYBC2M2AIIQ, seg_pos={0 0}, upload=551605]: Uploading piece... Dec 31 20:57:47.587787478 (629h28m47.458870523s) [noise=true , address=45.91.116.7:4012 , piece_id=AMRJDR4NB2NY333C65JPIUNTWXMEYYCB6TT6P3NHZZHZWIV5WVEQ, seg_pos={0 0}, upload=551605]: Uploading piece... Dec 31 20:57:47.589235975 (629h28m47.46031901s) [noise=true , address=145.79.227.50:4373 , piece_id=W7QHTEJFMLTMCQK5VQRP6BQTVFFEZVNP6M72GGGF6CPTDYYOVAUA, seg_pos={0 0}, upload=551605]: Uploading piece... Dec 31 20:57:47.590735520 (629h28m47.461818565s) [noise=true , address=145.79.227.14:4840 , piece_id=NOVTDNZOYNIWK7KPFX5RZ2HCJ3PHALLIZHMJUIX5PKZEKYCIAFXA, seg_pos={0 0}, upload=551605]: Uploading piece... Dec 31 20:57:49.259688133 (629h28m49.130771168s) [upload=551605]: Next returned nil segment Dec 31 20:57:49.259695617 (629h28m49.130778662s) [upload=551605]: Waiting for error group managing segments... Dec 31 20:57:49.556057161 (629h28m49.427140206s) [noise=true , address=145.79.227.52:4446 , piece_id=K6QZ3I7XMKB52VDX47M5PN7TLGZKFNDALS25FS2QB7TFEQU2FEAQ, seg_pos={0 0}, upload=551605]: Done uploading piece. err: <nil> Dec 31 20:57:49.564586322 (629h28m49.435669357s) [noise=true , address=145.79.227.48:4303 , piece_id=UMAHKL7H3VYZEMTAEAXZTJBE6YMGRAPXKSMBNU7CLRJCVAP4MJXQ, seg_pos={0 0}, upload=551605]: Done uploading piece. err: <nil> Dec 31 20:57:49.573752445 (629h28m49.44483548s) [noise=true , address=145.79.227.7:4295 , piece_id=HO2EQPC7MDGVZGWWCZIXXCCAAUQTLL2WGD5AQUQBTGSOJG62SDTA, seg_pos={0 0}, upload=551605]: Done uploading piece. err: <nil> Dec 31 20:57:49.574983680 (629h28m49.446066725s) [noise=true , address=145.79.227.8:4330 , piece_id=WHZY3ZBXCHKXTL5PCYJGHMDFJ2K23HBVYDHXHYA3GYUJU46FFFXA, seg_pos={0 0}, upload=551605]: Done uploading piece. err: <nil> Dec 31 20:57:49.576010686 (629h28m49.447093721s) [noise=true , address=145.79.227.38:4977 , piece_id=5JK6U5TLD6AOB4F257BSBLVLDE7ZGIH2MSHMUQJNLSYBC2M2AIIQ, seg_pos={0 0}, upload=551605]: Done uploading piece. err: <nil> Dec 31 20:57:49.586001309 (629h28m49.457084355s) [noise=true , address=145.79.227.14:4840 , piece_id=NOVTDNZOYNIWK7KPFX5RZ2HCJ3PHALLIZHMJUIX5PKZEKYCIAFXA, seg_pos={0 0}, upload=551605]: Done uploading piece. err: <nil> Dec 31 20:57:49.588212950 (629h28m49.459295995s) [noise=true , address=145.79.227.37:4565 , piece_id=7YB7O2IZHG67C2K437BFWGEPQJEMNRBNHD2VL7VDCZB4XQLTJ53A, seg_pos={0 0}, upload=551605]: Done uploading piece. err: <nil> Dec 31 20:57:49.591130777 (629h28m49.462213812s) [noise=true , address=145.79.227.23:4547 , piece_id=LGROBKN2JBULMYC7MXDOYBP744ASAPVABA3H5W6EUZLWL5EIJ6DA, seg_pos={0 0}, upload=551605]: Done uploading piece. err: <nil> Dec 31 20:57:49.607651224 (629h28m49.478734259s) [noise=true , address=145.79.227.50:4373 , piece_id=W7QHTEJFMLTMCQK5VQRP6BQTVFFEZVNP6M72GGGF6CPTDYYOVAUA, seg_pos={0 0}, upload=551605]: Done uploading piece. err: <nil> Dec 31 20:57:49.656485591 (629h28m49.527568626s) [noise=true , address=145.79.227.51:4409 , piece_id=Y47GETDCED7TXBB46AKCQFFBN53R6F27FSADUPWHYTNOYEYSZHQQ, seg_pos={0 0}, upload=551605]: Done uploading piece. err: <nil> Dec 31 20:57:49.735331545 (629h28m49.60641459s) [noise=true , address=77.90.31.21:4877 , piece_id=Y36LIA6U3G2CEV7HQWQRME5EA2GKA3QPOWI6ZP5EQAQCWC56KW6Q, seg_pos={0 0}, upload=551605]: Done uploading piece. err: <nil> Dec 31 20:57:49.800718531 (629h28m49.671801566s) [noise=true , address=77.90.31.19:4841 , piece_id=2M6QALYWFXXX4J4OPQV2QZRQZJENFSG7G73DXURKWRBBSMI7YOIA, seg_pos={0 0}, upload=551605]: Done uploading piece. err: <nil> Dec 31 20:57:49.883556196 (629h28m49.754639231s) [noise=true , address=45.91.116.5:4300 , piece_id=EUOXBKHHU3ZIE4XO6TSEJKHGRZQ6ELEOUIM4MCPR3VPJBB6O5E2Q, seg_pos={0 0}, upload=551605]: Done uploading piece. err: <nil> Dec 31 20:57:49.883581123 (629h28m49.754664158s) [seg_pos={0 0}, upload=551605]: Segment reached optimal threshold of 13 pieces. Dec 31 20:57:49.883814238 (629h28m49.754897283s) [noise=true , address=45.91.116.10:4081 , piece_id=LJGGEMYZTSJZXXZN7R47LVFJFLDN4E7D4MMDM4MFLQSCJOFSLBTA, seg_pos={0 0}, upload=551605]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF Dec 31 20:57:49.883823675 (629h28m49.75490672s) [noise=true , address=45.91.116.10:4081 , piece_id=LJGGEMYZTSJZXXZN7R47LVFJFLDN4E7D4MMDM4MFLQSCJOFSLBTA, seg_pos={0 0}, upload=551605]: Long tail cancellation due to optimal threshold Dec 31 20:57:49.883955737 (629h28m49.755038772s) [noise=true , address=45.91.116.7:4012 , piece_id=AMRJDR4NB2NY333C65JPIUNTWXMEYYCB6TT6P3NHZZHZWIV5WVEQ, seg_pos={0 0}, upload=551605]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: write tcp 188.166.225.170:60304->45.91.116.7:4012: use of closed network connection; write tcp 188.166.225.170:60304->45.91.116.7:4012: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60304->45.91.116.7:4012: use of closed network connection Dec 31 20:57:49.883971917 (629h28m49.755054962s) [noise=true , address=45.91.116.7:4012 , piece_id=AMRJDR4NB2NY333C65JPIUNTWXMEYYCB6TT6P3NHZZHZWIV5WVEQ, seg_pos={0 0}, upload=551605]: Long tail cancellation due to optimal threshold Dec 31 20:57:49.884109970 (629h28m49.755193015s) [noise=true , address=45.91.116.13:4154 , piece_id=Q5WKC5UOGLRDPEMMWNKRXTZEFRHYYE5PRNWBDQAZQHFQ343SYRPQ, seg_pos={0 0}, upload=551605]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Dec 31 20:57:49.884121993 (629h28m49.755205068s) [noise=true , address=45.91.116.13:4154 , piece_id=Q5WKC5UOGLRDPEMMWNKRXTZEFRHYYE5PRNWBDQAZQHFQ343SYRPQ, seg_pos={0 0}, upload=551605]: Long tail cancellation due to optimal threshold Dec 31 20:57:49.884141570 (629h28m49.755224605s) [seg_pos={0 0}, upload=551605]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 20:57:49.884217695 (629h28m49.75530073s) [upload=551605]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 20:57:49.884222524 (629h28m49.755305579s) [upload=551605]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 20:57:50.189548398 (629h28m50.060631444s) [upload=551605]: Done uploading segments... Dec 31 20:57:50.189570912 (629h28m50.060653947s) [upload=551605]: Upload finished. err: <nil> Duration: 17.440174013s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [....X.......XX...X..XX...X...XX....X......X....XX........X..] duration: 5.843623213s trace: https://jaeger.dev.storj.io/trace/62e53f89d015a867 FAIL (2025-12-31 20:59:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, Piece ID: 2M6QALYWFXXX4J4OPQV2QZRQZJENFSG7G73DXURKWRBBSMI7YOIA) context deadline exceeded; piecestore: (Node ID: 12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, Piece ID: Y36LIA6U3G2CEV7HQWQRME5EA2GKA3QPOWI6ZP5EQAQCWC56KW6Q) context deadline exceeded ======LOGS====== Dec 31 20:57:47.327402587 (629h28m47.198485632s) [upload=551605]: Starting upload Dec 31 20:57:47.327456269 (629h28m47.198539304s) [upload=551605]: Done starting upload Dec 31 20:57:47.327528276 (629h28m47.198611312s) [upload=551605]: Uploading segments... Dec 31 20:57:47.327633827 (629h28m47.198716862s) [upload=551605]: Got next segment. Inline: false Dec 31 20:57:47.327661971 (629h28m47.198745006s) [upload=551605]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 20:57:47.327666089 (629h28m47.198749124s) [upload=551605]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 20:57:47.578796046 (629h28m47.449879101s) [seg_pos={0 0}, upload=551605]: Begin upload segment... Dec 31 20:57:47.578849297 (629h28m47.449932332s) [seg_pos={0 0}, upload=551605]: Done begin upload segment. Dec 31 20:57:47.578884985 (629h28m47.44996802s) [noise=true , address=145.79.227.23:4547 , piece_id=LGROBKN2JBULMYC7MXDOYBP744ASAPVABA3H5W6EUZLWL5EIJ6DA, seg_pos={0 0}, upload=551605]: Uploading piece... Dec 31 20:57:47.578899523 (629h28m47.449982568s) [noise=true , address=45.91.116.13:4154 , piece_id=Q5WKC5UOGLRDPEMMWNKRXTZEFRHYYE5PRNWBDQAZQHFQ343SYRPQ, seg_pos={0 0}, upload=551605]: Uploading piece... Dec 31 20:57:47.579003671 (629h28m47.450086706s) [noise=true , address=45.91.116.5:4300 , piece_id=EUOXBKHHU3ZIE4XO6TSEJKHGRZQ6ELEOUIM4MCPR3VPJBB6O5E2Q, seg_pos={0 0}, upload=551605]: Uploading piece... Dec 31 20:57:47.579702872 (629h28m47.450785917s) [noise=true , address=145.79.227.52:4446 , piece_id=K6QZ3I7XMKB52VDX47M5PN7TLGZKFNDALS25FS2QB7TFEQU2FEAQ, seg_pos={0 0}, upload=551605]: Uploading piece... Dec 31 20:57:47.580341078 (629h28m47.451424123s) [noise=true , address=45.91.116.10:4081 , piece_id=LJGGEMYZTSJZXXZN7R47LVFJFLDN4E7D4MMDM4MFLQSCJOFSLBTA, seg_pos={0 0}, upload=551605]: Uploading piece... Dec 31 20:57:47.580797046 (629h28m47.451880081s) [noise=true , address=145.79.227.48:4303 , piece_id=UMAHKL7H3VYZEMTAEAXZTJBE6YMGRAPXKSMBNU7CLRJCVAP4MJXQ, seg_pos={0 0}, upload=551605]: Uploading piece... Dec 31 20:57:47.581194753 (629h28m47.452277798s) [noise=true , address=145.79.227.7:4295 , piece_id=HO2EQPC7MDGVZGWWCZIXXCCAAUQTLL2WGD5AQUQBTGSOJG62SDTA, seg_pos={0 0}, upload=551605]: Uploading piece... Dec 31 20:57:47.581571922 (629h28m47.452654957s) [noise=true , address=145.79.227.51:4409 , piece_id=Y47GETDCED7TXBB46AKCQFFBN53R6F27FSADUPWHYTNOYEYSZHQQ, seg_pos={0 0}, upload=551605]: Uploading piece... Dec 31 20:57:47.582067826 (629h28m47.453150871s) [noise=true , address=77.90.31.19:4841 , piece_id=2M6QALYWFXXX4J4OPQV2QZRQZJENFSG7G73DXURKWRBBSMI7YOIA, seg_pos={0 0}, upload=551605]: Uploading piece... Dec 31 20:57:47.582519295 (629h28m47.45360234s) [noise=true , address=77.90.31.21:4877 , piece_id=Y36LIA6U3G2CEV7HQWQRME5EA2GKA3QPOWI6ZP5EQAQCWC56KW6Q, seg_pos={0 0}, upload=551605]: Uploading piece... Dec 31 20:57:47.584070550 (629h28m47.455153585s) [noise=true , address=145.79.227.37:4565 , piece_id=7YB7O2IZHG67C2K437BFWGEPQJEMNRBNHD2VL7VDCZB4XQLTJ53A, seg_pos={0 0}, upload=551605]: Uploading piece... Dec 31 20:57:47.584613043 (629h28m47.455696088s) [noise=true , address=145.79.227.8:4330 , piece_id=WHZY3ZBXCHKXTL5PCYJGHMDFJ2K23HBVYDHXHYA3GYUJU46FFFXA, seg_pos={0 0}, upload=551605]: Uploading piece... Dec 31 20:57:47.586283874 (629h28m47.457366919s) [noise=true , address=145.79.227.38:4977 , piece_id=5JK6U5TLD6AOB4F257BSBLVLDE7ZGIH2MSHMUQJNLSYBC2M2AIIQ, seg_pos={0 0}, upload=551605]: Uploading piece... Dec 31 20:57:47.587787478 (629h28m47.458870523s) [noise=true , address=45.91.116.7:4012 , piece_id=AMRJDR4NB2NY333C65JPIUNTWXMEYYCB6TT6P3NHZZHZWIV5WVEQ, seg_pos={0 0}, upload=551605]: Uploading piece... Dec 31 20:57:47.589235975 (629h28m47.46031901s) [noise=true , address=145.79.227.50:4373 , piece_id=W7QHTEJFMLTMCQK5VQRP6BQTVFFEZVNP6M72GGGF6CPTDYYOVAUA, seg_pos={0 0}, upload=551605]: Uploading piece... Dec 31 20:57:47.590735520 (629h28m47.461818565s) [noise=true , address=145.79.227.14:4840 , piece_id=NOVTDNZOYNIWK7KPFX5RZ2HCJ3PHALLIZHMJUIX5PKZEKYCIAFXA, seg_pos={0 0}, upload=551605]: Uploading piece... Dec 31 20:57:49.259688133 (629h28m49.130771168s) [upload=551605]: Next returned nil segment Dec 31 20:57:49.259695617 (629h28m49.130778662s) [upload=551605]: Waiting for error group managing segments... Dec 31 20:57:49.556057161 (629h28m49.427140206s) [noise=true , address=145.79.227.52:4446 , piece_id=K6QZ3I7XMKB52VDX47M5PN7TLGZKFNDALS25FS2QB7TFEQU2FEAQ, seg_pos={0 0}, upload=551605]: Done uploading piece. err: <nil> Dec 31 20:57:49.564586322 (629h28m49.435669357s) [noise=true , address=145.79.227.48:4303 , piece_id=UMAHKL7H3VYZEMTAEAXZTJBE6YMGRAPXKSMBNU7CLRJCVAP4MJXQ, seg_pos={0 0}, upload=551605]: Done uploading piece. err: <nil> Dec 31 20:57:49.573752445 (629h28m49.44483548s) [noise=true , address=145.79.227.7:4295 , piece_id=HO2EQPC7MDGVZGWWCZIXXCCAAUQTLL2WGD5AQUQBTGSOJG62SDTA, seg_pos={0 0}, upload=551605]: Done uploading piece. err: <nil> Dec 31 20:57:49.574983680 (629h28m49.446066725s) [noise=true , address=145.79.227.8:4330 , piece_id=WHZY3ZBXCHKXTL5PCYJGHMDFJ2K23HBVYDHXHYA3GYUJU46FFFXA, seg_pos={0 0}, upload=551605]: Done uploading piece. err: <nil> Dec 31 20:57:49.576010686 (629h28m49.447093721s) [noise=true , address=145.79.227.38:4977 , piece_id=5JK6U5TLD6AOB4F257BSBLVLDE7ZGIH2MSHMUQJNLSYBC2M2AIIQ, seg_pos={0 0}, upload=551605]: Done uploading piece. err: <nil> Dec 31 20:57:49.586001309 (629h28m49.457084355s) [noise=true , address=145.79.227.14:4840 , piece_id=NOVTDNZOYNIWK7KPFX5RZ2HCJ3PHALLIZHMJUIX5PKZEKYCIAFXA, seg_pos={0 0}, upload=551605]: Done uploading piece. err: <nil> Dec 31 20:57:49.588212950 (629h28m49.459295995s) [noise=true , address=145.79.227.37:4565 , piece_id=7YB7O2IZHG67C2K437BFWGEPQJEMNRBNHD2VL7VDCZB4XQLTJ53A, seg_pos={0 0}, upload=551605]: Done uploading piece. err: <nil> Dec 31 20:57:49.591130777 (629h28m49.462213812s) [noise=true , address=145.79.227.23:4547 , piece_id=LGROBKN2JBULMYC7MXDOYBP744ASAPVABA3H5W6EUZLWL5EIJ6DA, seg_pos={0 0}, upload=551605]: Done uploading piece. err: <nil> Dec 31 20:57:49.607651224 (629h28m49.478734259s) [noise=true , address=145.79.227.50:4373 , piece_id=W7QHTEJFMLTMCQK5VQRP6BQTVFFEZVNP6M72GGGF6CPTDYYOVAUA, seg_pos={0 0}, upload=551605]: Done uploading piece. err: <nil> Dec 31 20:57:49.656485591 (629h28m49.527568626s) [noise=true , address=145.79.227.51:4409 , piece_id=Y47GETDCED7TXBB46AKCQFFBN53R6F27FSADUPWHYTNOYEYSZHQQ, seg_pos={0 0}, upload=551605]: Done uploading piece. err: <nil> Dec 31 20:57:49.735331545 (629h28m49.60641459s) [noise=true , address=77.90.31.21:4877 , piece_id=Y36LIA6U3G2CEV7HQWQRME5EA2GKA3QPOWI6ZP5EQAQCWC56KW6Q, seg_pos={0 0}, upload=551605]: Done uploading piece. err: <nil> Dec 31 20:57:49.800718531 (629h28m49.671801566s) [noise=true , address=77.90.31.19:4841 , piece_id=2M6QALYWFXXX4J4OPQV2QZRQZJENFSG7G73DXURKWRBBSMI7YOIA, seg_pos={0 0}, upload=551605]: Done uploading piece. err: <nil> Dec 31 20:57:49.883556196 (629h28m49.754639231s) [noise=true , address=45.91.116.5:4300 , piece_id=EUOXBKHHU3ZIE4XO6TSEJKHGRZQ6ELEOUIM4MCPR3VPJBB6O5E2Q, seg_pos={0 0}, upload=551605]: Done uploading piece. err: <nil> Dec 31 20:57:49.883581123 (629h28m49.754664158s) [seg_pos={0 0}, upload=551605]: Segment reached optimal threshold of 13 pieces. Dec 31 20:57:49.883814238 (629h28m49.754897283s) [noise=true , address=45.91.116.10:4081 , piece_id=LJGGEMYZTSJZXXZN7R47LVFJFLDN4E7D4MMDM4MFLQSCJOFSLBTA, seg_pos={0 0}, upload=551605]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF Dec 31 20:57:49.883823675 (629h28m49.75490672s) [noise=true , address=45.91.116.10:4081 , piece_id=LJGGEMYZTSJZXXZN7R47LVFJFLDN4E7D4MMDM4MFLQSCJOFSLBTA, seg_pos={0 0}, upload=551605]: Long tail cancellation due to optimal threshold Dec 31 20:57:49.883955737 (629h28m49.755038772s) [noise=true , address=45.91.116.7:4012 , piece_id=AMRJDR4NB2NY333C65JPIUNTWXMEYYCB6TT6P3NHZZHZWIV5WVEQ, seg_pos={0 0}, upload=551605]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: write tcp 188.166.225.170:60304->45.91.116.7:4012: use of closed network connection; write tcp 188.166.225.170:60304->45.91.116.7:4012: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60304->45.91.116.7:4012: use of closed network connection Dec 31 20:57:49.883971917 (629h28m49.755054962s) [noise=true , address=45.91.116.7:4012 , piece_id=AMRJDR4NB2NY333C65JPIUNTWXMEYYCB6TT6P3NHZZHZWIV5WVEQ, seg_pos={0 0}, upload=551605]: Long tail cancellation due to optimal threshold Dec 31 20:57:49.884109970 (629h28m49.755193015s) [noise=true , address=45.91.116.13:4154 , piece_id=Q5WKC5UOGLRDPEMMWNKRXTZEFRHYYE5PRNWBDQAZQHFQ343SYRPQ, seg_pos={0 0}, upload=551605]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Dec 31 20:57:49.884121993 (629h28m49.755205068s) [noise=true , address=45.91.116.13:4154 , piece_id=Q5WKC5UOGLRDPEMMWNKRXTZEFRHYYE5PRNWBDQAZQHFQ343SYRPQ, seg_pos={0 0}, upload=551605]: Long tail cancellation due to optimal threshold Dec 31 20:57:49.884141570 (629h28m49.755224605s) [seg_pos={0 0}, upload=551605]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 20:57:49.884217695 (629h28m49.75530073s) [upload=551605]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 20:57:49.884222524 (629h28m49.755305579s) [upload=551605]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 20:57:50.189548398 (629h28m50.060631444s) [upload=551605]: Done uploading segments... Dec 31 20:57:50.189570912 (629h28m50.060653947s) [upload=551605]: Upload finished. err: <nil> Duration: 5.843650385s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [.....X.......XX...X..XX...X...XX....X......X....XX........X.] duration: 6.055891996s trace: https://jaeger.dev.storj.io/trace/62e53f89d015a867 FAIL (2025-12-31 20:58:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, Piece ID: 2M6QALYWFXXX4J4OPQV2QZRQZJENFSG7G73DXURKWRBBSMI7YOIA) context deadline exceeded; piecestore: (Node ID: 12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, Piece ID: Y36LIA6U3G2CEV7HQWQRME5EA2GKA3QPOWI6ZP5EQAQCWC56KW6Q) context deadline exceeded ======LOGS====== Dec 31 20:57:47.327402587 (629h28m47.198485632s) [upload=551605]: Starting upload Dec 31 20:57:47.327456269 (629h28m47.198539304s) [upload=551605]: Done starting upload Dec 31 20:57:47.327528276 (629h28m47.198611312s) [upload=551605]: Uploading segments... Dec 31 20:57:47.327633827 (629h28m47.198716862s) [upload=551605]: Got next segment. Inline: false Dec 31 20:57:47.327661971 (629h28m47.198745006s) [upload=551605]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 20:57:47.327666089 (629h28m47.198749124s) [upload=551605]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 20:57:47.578796046 (629h28m47.449879101s) [seg_pos={0 0}, upload=551605]: Begin upload segment... Dec 31 20:57:47.578849297 (629h28m47.449932332s) [seg_pos={0 0}, upload=551605]: Done begin upload segment. Dec 31 20:57:47.578884985 (629h28m47.44996802s) [noise=true , address=145.79.227.23:4547 , piece_id=LGROBKN2JBULMYC7MXDOYBP744ASAPVABA3H5W6EUZLWL5EIJ6DA, seg_pos={0 0}, upload=551605]: Uploading piece... Dec 31 20:57:47.578899523 (629h28m47.449982568s) [noise=true , address=45.91.116.13:4154 , piece_id=Q5WKC5UOGLRDPEMMWNKRXTZEFRHYYE5PRNWBDQAZQHFQ343SYRPQ, seg_pos={0 0}, upload=551605]: Uploading piece... Dec 31 20:57:47.579003671 (629h28m47.450086706s) [noise=true , address=45.91.116.5:4300 , piece_id=EUOXBKHHU3ZIE4XO6TSEJKHGRZQ6ELEOUIM4MCPR3VPJBB6O5E2Q, seg_pos={0 0}, upload=551605]: Uploading piece... Dec 31 20:57:47.579702872 (629h28m47.450785917s) [noise=true , address=145.79.227.52:4446 , piece_id=K6QZ3I7XMKB52VDX47M5PN7TLGZKFNDALS25FS2QB7TFEQU2FEAQ, seg_pos={0 0}, upload=551605]: Uploading piece... Dec 31 20:57:47.580341078 (629h28m47.451424123s) [noise=true , address=45.91.116.10:4081 , piece_id=LJGGEMYZTSJZXXZN7R47LVFJFLDN4E7D4MMDM4MFLQSCJOFSLBTA, seg_pos={0 0}, upload=551605]: Uploading piece... Dec 31 20:57:47.580797046 (629h28m47.451880081s) [noise=true , address=145.79.227.48:4303 , piece_id=UMAHKL7H3VYZEMTAEAXZTJBE6YMGRAPXKSMBNU7CLRJCVAP4MJXQ, seg_pos={0 0}, upload=551605]: Uploading piece... Dec 31 20:57:47.581194753 (629h28m47.452277798s) [noise=true , address=145.79.227.7:4295 , piece_id=HO2EQPC7MDGVZGWWCZIXXCCAAUQTLL2WGD5AQUQBTGSOJG62SDTA, seg_pos={0 0}, upload=551605]: Uploading piece... Dec 31 20:57:47.581571922 (629h28m47.452654957s) [noise=true , address=145.79.227.51:4409 , piece_id=Y47GETDCED7TXBB46AKCQFFBN53R6F27FSADUPWHYTNOYEYSZHQQ, seg_pos={0 0}, upload=551605]: Uploading piece... Dec 31 20:57:47.582067826 (629h28m47.453150871s) [noise=true , address=77.90.31.19:4841 , piece_id=2M6QALYWFXXX4J4OPQV2QZRQZJENFSG7G73DXURKWRBBSMI7YOIA, seg_pos={0 0}, upload=551605]: Uploading piece... Dec 31 20:57:47.582519295 (629h28m47.45360234s) [noise=true , address=77.90.31.21:4877 , piece_id=Y36LIA6U3G2CEV7HQWQRME5EA2GKA3QPOWI6ZP5EQAQCWC56KW6Q, seg_pos={0 0}, upload=551605]: Uploading piece... Dec 31 20:57:47.584070550 (629h28m47.455153585s) [noise=true , address=145.79.227.37:4565 , piece_id=7YB7O2IZHG67C2K437BFWGEPQJEMNRBNHD2VL7VDCZB4XQLTJ53A, seg_pos={0 0}, upload=551605]: Uploading piece... Dec 31 20:57:47.584613043 (629h28m47.455696088s) [noise=true , address=145.79.227.8:4330 , piece_id=WHZY3ZBXCHKXTL5PCYJGHMDFJ2K23HBVYDHXHYA3GYUJU46FFFXA, seg_pos={0 0}, upload=551605]: Uploading piece... Dec 31 20:57:47.586283874 (629h28m47.457366919s) [noise=true , address=145.79.227.38:4977 , piece_id=5JK6U5TLD6AOB4F257BSBLVLDE7ZGIH2MSHMUQJNLSYBC2M2AIIQ, seg_pos={0 0}, upload=551605]: Uploading piece... Dec 31 20:57:47.587787478 (629h28m47.458870523s) [noise=true , address=45.91.116.7:4012 , piece_id=AMRJDR4NB2NY333C65JPIUNTWXMEYYCB6TT6P3NHZZHZWIV5WVEQ, seg_pos={0 0}, upload=551605]: Uploading piece... Dec 31 20:57:47.589235975 (629h28m47.46031901s) [noise=true , address=145.79.227.50:4373 , piece_id=W7QHTEJFMLTMCQK5VQRP6BQTVFFEZVNP6M72GGGF6CPTDYYOVAUA, seg_pos={0 0}, upload=551605]: Uploading piece... Dec 31 20:57:47.590735520 (629h28m47.461818565s) [noise=true , address=145.79.227.14:4840 , piece_id=NOVTDNZOYNIWK7KPFX5RZ2HCJ3PHALLIZHMJUIX5PKZEKYCIAFXA, seg_pos={0 0}, upload=551605]: Uploading piece... Dec 31 20:57:49.259688133 (629h28m49.130771168s) [upload=551605]: Next returned nil segment Dec 31 20:57:49.259695617 (629h28m49.130778662s) [upload=551605]: Waiting for error group managing segments... Dec 31 20:57:49.556057161 (629h28m49.427140206s) [noise=true , address=145.79.227.52:4446 , piece_id=K6QZ3I7XMKB52VDX47M5PN7TLGZKFNDALS25FS2QB7TFEQU2FEAQ, seg_pos={0 0}, upload=551605]: Done uploading piece. err: <nil> Dec 31 20:57:49.564586322 (629h28m49.435669357s) [noise=true , address=145.79.227.48:4303 , piece_id=UMAHKL7H3VYZEMTAEAXZTJBE6YMGRAPXKSMBNU7CLRJCVAP4MJXQ, seg_pos={0 0}, upload=551605]: Done uploading piece. err: <nil> Dec 31 20:57:49.573752445 (629h28m49.44483548s) [noise=true , address=145.79.227.7:4295 , piece_id=HO2EQPC7MDGVZGWWCZIXXCCAAUQTLL2WGD5AQUQBTGSOJG62SDTA, seg_pos={0 0}, upload=551605]: Done uploading piece. err: <nil> Dec 31 20:57:49.574983680 (629h28m49.446066725s) [noise=true , address=145.79.227.8:4330 , piece_id=WHZY3ZBXCHKXTL5PCYJGHMDFJ2K23HBVYDHXHYA3GYUJU46FFFXA, seg_pos={0 0}, upload=551605]: Done uploading piece. err: <nil> Dec 31 20:57:49.576010686 (629h28m49.447093721s) [noise=true , address=145.79.227.38:4977 , piece_id=5JK6U5TLD6AOB4F257BSBLVLDE7ZGIH2MSHMUQJNLSYBC2M2AIIQ, seg_pos={0 0}, upload=551605]: Done uploading piece. err: <nil> Dec 31 20:57:49.586001309 (629h28m49.457084355s) [noise=true , address=145.79.227.14:4840 , piece_id=NOVTDNZOYNIWK7KPFX5RZ2HCJ3PHALLIZHMJUIX5PKZEKYCIAFXA, seg_pos={0 0}, upload=551605]: Done uploading piece. err: <nil> Dec 31 20:57:49.588212950 (629h28m49.459295995s) [noise=true , address=145.79.227.37:4565 , piece_id=7YB7O2IZHG67C2K437BFWGEPQJEMNRBNHD2VL7VDCZB4XQLTJ53A, seg_pos={0 0}, upload=551605]: Done uploading piece. err: <nil> Dec 31 20:57:49.591130777 (629h28m49.462213812s) [noise=true , address=145.79.227.23:4547 , piece_id=LGROBKN2JBULMYC7MXDOYBP744ASAPVABA3H5W6EUZLWL5EIJ6DA, seg_pos={0 0}, upload=551605]: Done uploading piece. err: <nil> Dec 31 20:57:49.607651224 (629h28m49.478734259s) [noise=true , address=145.79.227.50:4373 , piece_id=W7QHTEJFMLTMCQK5VQRP6BQTVFFEZVNP6M72GGGF6CPTDYYOVAUA, seg_pos={0 0}, upload=551605]: Done uploading piece. err: <nil> Dec 31 20:57:49.656485591 (629h28m49.527568626s) [noise=true , address=145.79.227.51:4409 , piece_id=Y47GETDCED7TXBB46AKCQFFBN53R6F27FSADUPWHYTNOYEYSZHQQ, seg_pos={0 0}, upload=551605]: Done uploading piece. err: <nil> Dec 31 20:57:49.735331545 (629h28m49.60641459s) [noise=true , address=77.90.31.21:4877 , piece_id=Y36LIA6U3G2CEV7HQWQRME5EA2GKA3QPOWI6ZP5EQAQCWC56KW6Q, seg_pos={0 0}, upload=551605]: Done uploading piece. err: <nil> Dec 31 20:57:49.800718531 (629h28m49.671801566s) [noise=true , address=77.90.31.19:4841 , piece_id=2M6QALYWFXXX4J4OPQV2QZRQZJENFSG7G73DXURKWRBBSMI7YOIA, seg_pos={0 0}, upload=551605]: Done uploading piece. err: <nil> Dec 31 20:57:49.883556196 (629h28m49.754639231s) [noise=true , address=45.91.116.5:4300 , piece_id=EUOXBKHHU3ZIE4XO6TSEJKHGRZQ6ELEOUIM4MCPR3VPJBB6O5E2Q, seg_pos={0 0}, upload=551605]: Done uploading piece. err: <nil> Dec 31 20:57:49.883581123 (629h28m49.754664158s) [seg_pos={0 0}, upload=551605]: Segment reached optimal threshold of 13 pieces. Dec 31 20:57:49.883814238 (629h28m49.754897283s) [noise=true , address=45.91.116.10:4081 , piece_id=LJGGEMYZTSJZXXZN7R47LVFJFLDN4E7D4MMDM4MFLQSCJOFSLBTA, seg_pos={0 0}, upload=551605]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF Dec 31 20:57:49.883823675 (629h28m49.75490672s) [noise=true , address=45.91.116.10:4081 , piece_id=LJGGEMYZTSJZXXZN7R47LVFJFLDN4E7D4MMDM4MFLQSCJOFSLBTA, seg_pos={0 0}, upload=551605]: Long tail cancellation due to optimal threshold Dec 31 20:57:49.883955737 (629h28m49.755038772s) [noise=true , address=45.91.116.7:4012 , piece_id=AMRJDR4NB2NY333C65JPIUNTWXMEYYCB6TT6P3NHZZHZWIV5WVEQ, seg_pos={0 0}, upload=551605]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: write tcp 188.166.225.170:60304->45.91.116.7:4012: use of closed network connection; write tcp 188.166.225.170:60304->45.91.116.7:4012: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60304->45.91.116.7:4012: use of closed network connection Dec 31 20:57:49.883971917 (629h28m49.755054962s) [noise=true , address=45.91.116.7:4012 , piece_id=AMRJDR4NB2NY333C65JPIUNTWXMEYYCB6TT6P3NHZZHZWIV5WVEQ, seg_pos={0 0}, upload=551605]: Long tail cancellation due to optimal threshold Dec 31 20:57:49.884109970 (629h28m49.755193015s) [noise=true , address=45.91.116.13:4154 , piece_id=Q5WKC5UOGLRDPEMMWNKRXTZEFRHYYE5PRNWBDQAZQHFQ343SYRPQ, seg_pos={0 0}, upload=551605]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Dec 31 20:57:49.884121993 (629h28m49.755205068s) [noise=true , address=45.91.116.13:4154 , piece_id=Q5WKC5UOGLRDPEMMWNKRXTZEFRHYYE5PRNWBDQAZQHFQ343SYRPQ, seg_pos={0 0}, upload=551605]: Long tail cancellation due to optimal threshold Dec 31 20:57:49.884141570 (629h28m49.755224605s) [seg_pos={0 0}, upload=551605]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 20:57:49.884217695 (629h28m49.75530073s) [upload=551605]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 20:57:49.884222524 (629h28m49.755305579s) [upload=551605]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 20:57:50.189548398 (629h28m50.060631444s) [upload=551605]: Done uploading segments... Dec 31 20:57:50.189570912 (629h28m50.060653947s) [upload=551605]: Upload finished. err: <nil> Duration: 6.055916414s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [X.....X.......XX...X..XX...X...XX....X......X....XX........X] duration: 30.00316646s trace: https://jaeger.dev.storj.io/trace/62e53f89d015a867 FAIL (2025-12-31 20:58:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, Piece ID: 2M6QALYWFXXX4J4OPQV2QZRQZJENFSG7G73DXURKWRBBSMI7YOIA) context deadline exceeded; piecestore: (Node ID: 12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, Piece ID: Y36LIA6U3G2CEV7HQWQRME5EA2GKA3QPOWI6ZP5EQAQCWC56KW6Q) context deadline exceeded ======LOGS====== Dec 31 20:57:47.327402587 (629h28m47.198485632s) [upload=551605]: Starting upload Dec 31 20:57:47.327456269 (629h28m47.198539304s) [upload=551605]: Done starting upload Dec 31 20:57:47.327528276 (629h28m47.198611312s) [upload=551605]: Uploading segments... Dec 31 20:57:47.327633827 (629h28m47.198716862s) [upload=551605]: Got next segment. Inline: false Dec 31 20:57:47.327661971 (629h28m47.198745006s) [upload=551605]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 20:57:47.327666089 (629h28m47.198749124s) [upload=551605]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 20:57:47.578796046 (629h28m47.449879101s) [seg_pos={0 0}, upload=551605]: Begin upload segment... Dec 31 20:57:47.578849297 (629h28m47.449932332s) [seg_pos={0 0}, upload=551605]: Done begin upload segment. Dec 31 20:57:47.578884985 (629h28m47.44996802s) [noise=true , address=145.79.227.23:4547 , piece_id=LGROBKN2JBULMYC7MXDOYBP744ASAPVABA3H5W6EUZLWL5EIJ6DA, seg_pos={0 0}, upload=551605]: Uploading piece... Dec 31 20:57:47.578899523 (629h28m47.449982568s) [noise=true , address=45.91.116.13:4154 , piece_id=Q5WKC5UOGLRDPEMMWNKRXTZEFRHYYE5PRNWBDQAZQHFQ343SYRPQ, seg_pos={0 0}, upload=551605]: Uploading piece... Dec 31 20:57:47.579003671 (629h28m47.450086706s) [noise=true , address=45.91.116.5:4300 , piece_id=EUOXBKHHU3ZIE4XO6TSEJKHGRZQ6ELEOUIM4MCPR3VPJBB6O5E2Q, seg_pos={0 0}, upload=551605]: Uploading piece... Dec 31 20:57:47.579702872 (629h28m47.450785917s) [noise=true , address=145.79.227.52:4446 , piece_id=K6QZ3I7XMKB52VDX47M5PN7TLGZKFNDALS25FS2QB7TFEQU2FEAQ, seg_pos={0 0}, upload=551605]: Uploading piece... Dec 31 20:57:47.580341078 (629h28m47.451424123s) [noise=true , address=45.91.116.10:4081 , piece_id=LJGGEMYZTSJZXXZN7R47LVFJFLDN4E7D4MMDM4MFLQSCJOFSLBTA, seg_pos={0 0}, upload=551605]: Uploading piece... Dec 31 20:57:47.580797046 (629h28m47.451880081s) [noise=true , address=145.79.227.48:4303 , piece_id=UMAHKL7H3VYZEMTAEAXZTJBE6YMGRAPXKSMBNU7CLRJCVAP4MJXQ, seg_pos={0 0}, upload=551605]: Uploading piece... Dec 31 20:57:47.581194753 (629h28m47.452277798s) [noise=true , address=145.79.227.7:4295 , piece_id=HO2EQPC7MDGVZGWWCZIXXCCAAUQTLL2WGD5AQUQBTGSOJG62SDTA, seg_pos={0 0}, upload=551605]: Uploading piece... Dec 31 20:57:47.581571922 (629h28m47.452654957s) [noise=true , address=145.79.227.51:4409 , piece_id=Y47GETDCED7TXBB46AKCQFFBN53R6F27FSADUPWHYTNOYEYSZHQQ, seg_pos={0 0}, upload=551605]: Uploading piece... Dec 31 20:57:47.582067826 (629h28m47.453150871s) [noise=true , address=77.90.31.19:4841 , piece_id=2M6QALYWFXXX4J4OPQV2QZRQZJENFSG7G73DXURKWRBBSMI7YOIA, seg_pos={0 0}, upload=551605]: Uploading piece... Dec 31 20:57:47.582519295 (629h28m47.45360234s) [noise=true , address=77.90.31.21:4877 , piece_id=Y36LIA6U3G2CEV7HQWQRME5EA2GKA3QPOWI6ZP5EQAQCWC56KW6Q, seg_pos={0 0}, upload=551605]: Uploading piece... Dec 31 20:57:47.584070550 (629h28m47.455153585s) [noise=true , address=145.79.227.37:4565 , piece_id=7YB7O2IZHG67C2K437BFWGEPQJEMNRBNHD2VL7VDCZB4XQLTJ53A, seg_pos={0 0}, upload=551605]: Uploading piece... Dec 31 20:57:47.584613043 (629h28m47.455696088s) [noise=true , address=145.79.227.8:4330 , piece_id=WHZY3ZBXCHKXTL5PCYJGHMDFJ2K23HBVYDHXHYA3GYUJU46FFFXA, seg_pos={0 0}, upload=551605]: Uploading piece... Dec 31 20:57:47.586283874 (629h28m47.457366919s) [noise=true , address=145.79.227.38:4977 , piece_id=5JK6U5TLD6AOB4F257BSBLVLDE7ZGIH2MSHMUQJNLSYBC2M2AIIQ, seg_pos={0 0}, upload=551605]: Uploading piece... Dec 31 20:57:47.587787478 (629h28m47.458870523s) [noise=true , address=45.91.116.7:4012 , piece_id=AMRJDR4NB2NY333C65JPIUNTWXMEYYCB6TT6P3NHZZHZWIV5WVEQ, seg_pos={0 0}, upload=551605]: Uploading piece... Dec 31 20:57:47.589235975 (629h28m47.46031901s) [noise=true , address=145.79.227.50:4373 , piece_id=W7QHTEJFMLTMCQK5VQRP6BQTVFFEZVNP6M72GGGF6CPTDYYOVAUA, seg_pos={0 0}, upload=551605]: Uploading piece... Dec 31 20:57:47.590735520 (629h28m47.461818565s) [noise=true , address=145.79.227.14:4840 , piece_id=NOVTDNZOYNIWK7KPFX5RZ2HCJ3PHALLIZHMJUIX5PKZEKYCIAFXA, seg_pos={0 0}, upload=551605]: Uploading piece... Dec 31 20:57:49.259688133 (629h28m49.130771168s) [upload=551605]: Next returned nil segment Dec 31 20:57:49.259695617 (629h28m49.130778662s) [upload=551605]: Waiting for error group managing segments... Dec 31 20:57:49.556057161 (629h28m49.427140206s) [noise=true , address=145.79.227.52:4446 , piece_id=K6QZ3I7XMKB52VDX47M5PN7TLGZKFNDALS25FS2QB7TFEQU2FEAQ, seg_pos={0 0}, upload=551605]: Done uploading piece. err: <nil> Dec 31 20:57:49.564586322 (629h28m49.435669357s) [noise=true , address=145.79.227.48:4303 , piece_id=UMAHKL7H3VYZEMTAEAXZTJBE6YMGRAPXKSMBNU7CLRJCVAP4MJXQ, seg_pos={0 0}, upload=551605]: Done uploading piece. err: <nil> Dec 31 20:57:49.573752445 (629h28m49.44483548s) [noise=true , address=145.79.227.7:4295 , piece_id=HO2EQPC7MDGVZGWWCZIXXCCAAUQTLL2WGD5AQUQBTGSOJG62SDTA, seg_pos={0 0}, upload=551605]: Done uploading piece. err: <nil> Dec 31 20:57:49.574983680 (629h28m49.446066725s) [noise=true , address=145.79.227.8:4330 , piece_id=WHZY3ZBXCHKXTL5PCYJGHMDFJ2K23HBVYDHXHYA3GYUJU46FFFXA, seg_pos={0 0}, upload=551605]: Done uploading piece. err: <nil> Dec 31 20:57:49.576010686 (629h28m49.447093721s) [noise=true , address=145.79.227.38:4977 , piece_id=5JK6U5TLD6AOB4F257BSBLVLDE7ZGIH2MSHMUQJNLSYBC2M2AIIQ, seg_pos={0 0}, upload=551605]: Done uploading piece. err: <nil> Dec 31 20:57:49.586001309 (629h28m49.457084355s) [noise=true , address=145.79.227.14:4840 , piece_id=NOVTDNZOYNIWK7KPFX5RZ2HCJ3PHALLIZHMJUIX5PKZEKYCIAFXA, seg_pos={0 0}, upload=551605]: Done uploading piece. err: <nil> Dec 31 20:57:49.588212950 (629h28m49.459295995s) [noise=true , address=145.79.227.37:4565 , piece_id=7YB7O2IZHG67C2K437BFWGEPQJEMNRBNHD2VL7VDCZB4XQLTJ53A, seg_pos={0 0}, upload=551605]: Done uploading piece. err: <nil> Dec 31 20:57:49.591130777 (629h28m49.462213812s) [noise=true , address=145.79.227.23:4547 , piece_id=LGROBKN2JBULMYC7MXDOYBP744ASAPVABA3H5W6EUZLWL5EIJ6DA, seg_pos={0 0}, upload=551605]: Done uploading piece. err: <nil> Dec 31 20:57:49.607651224 (629h28m49.478734259s) [noise=true , address=145.79.227.50:4373 , piece_id=W7QHTEJFMLTMCQK5VQRP6BQTVFFEZVNP6M72GGGF6CPTDYYOVAUA, seg_pos={0 0}, upload=551605]: Done uploading piece. err: <nil> Dec 31 20:57:49.656485591 (629h28m49.527568626s) [noise=true , address=145.79.227.51:4409 , piece_id=Y47GETDCED7TXBB46AKCQFFBN53R6F27FSADUPWHYTNOYEYSZHQQ, seg_pos={0 0}, upload=551605]: Done uploading piece. err: <nil> Dec 31 20:57:49.735331545 (629h28m49.60641459s) [noise=true , address=77.90.31.21:4877 , piece_id=Y36LIA6U3G2CEV7HQWQRME5EA2GKA3QPOWI6ZP5EQAQCWC56KW6Q, seg_pos={0 0}, upload=551605]: Done uploading piece. err: <nil> Dec 31 20:57:49.800718531 (629h28m49.671801566s) [noise=true , address=77.90.31.19:4841 , piece_id=2M6QALYWFXXX4J4OPQV2QZRQZJENFSG7G73DXURKWRBBSMI7YOIA, seg_pos={0 0}, upload=551605]: Done uploading piece. err: <nil> Dec 31 20:57:49.883556196 (629h28m49.754639231s) [noise=true , address=45.91.116.5:4300 , piece_id=EUOXBKHHU3ZIE4XO6TSEJKHGRZQ6ELEOUIM4MCPR3VPJBB6O5E2Q, seg_pos={0 0}, upload=551605]: Done uploading piece. err: <nil> Dec 31 20:57:49.883581123 (629h28m49.754664158s) [seg_pos={0 0}, upload=551605]: Segment reached optimal threshold of 13 pieces. Dec 31 20:57:49.883814238 (629h28m49.754897283s) [noise=true , address=45.91.116.10:4081 , piece_id=LJGGEMYZTSJZXXZN7R47LVFJFLDN4E7D4MMDM4MFLQSCJOFSLBTA, seg_pos={0 0}, upload=551605]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF Dec 31 20:57:49.883823675 (629h28m49.75490672s) [noise=true , address=45.91.116.10:4081 , piece_id=LJGGEMYZTSJZXXZN7R47LVFJFLDN4E7D4MMDM4MFLQSCJOFSLBTA, seg_pos={0 0}, upload=551605]: Long tail cancellation due to optimal threshold Dec 31 20:57:49.883955737 (629h28m49.755038772s) [noise=true , address=45.91.116.7:4012 , piece_id=AMRJDR4NB2NY333C65JPIUNTWXMEYYCB6TT6P3NHZZHZWIV5WVEQ, seg_pos={0 0}, upload=551605]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: write tcp 188.166.225.170:60304->45.91.116.7:4012: use of closed network connection; write tcp 188.166.225.170:60304->45.91.116.7:4012: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60304->45.91.116.7:4012: use of closed network connection Dec 31 20:57:49.883971917 (629h28m49.755054962s) [noise=true , address=45.91.116.7:4012 , piece_id=AMRJDR4NB2NY333C65JPIUNTWXMEYYCB6TT6P3NHZZHZWIV5WVEQ, seg_pos={0 0}, upload=551605]: Long tail cancellation due to optimal threshold Dec 31 20:57:49.884109970 (629h28m49.755193015s) [noise=true , address=45.91.116.13:4154 , piece_id=Q5WKC5UOGLRDPEMMWNKRXTZEFRHYYE5PRNWBDQAZQHFQ343SYRPQ, seg_pos={0 0}, upload=551605]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Dec 31 20:57:49.884121993 (629h28m49.755205068s) [noise=true , address=45.91.116.13:4154 , piece_id=Q5WKC5UOGLRDPEMMWNKRXTZEFRHYYE5PRNWBDQAZQHFQ343SYRPQ, seg_pos={0 0}, upload=551605]: Long tail cancellation due to optimal threshold Dec 31 20:57:49.884141570 (629h28m49.755224605s) [seg_pos={0 0}, upload=551605]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 20:57:49.884217695 (629h28m49.75530073s) [upload=551605]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 20:57:49.884222524 (629h28m49.755305579s) [upload=551605]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 20:57:50.189548398 (629h28m50.060631444s) [upload=551605]: Done uploading segments... Dec 31 20:57:50.189570912 (629h28m50.060653947s) [upload=551605]: Upload finished. err: <nil> Duration: 30.0031878s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [.X.....X.......XX...X..XX...X...XX....X......X....XX........] duration: 6.148770196s trace: https://jaeger.dev.storj.io/trace/3c6720cb0382e95c FAIL (2025-12-31 20:56:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: Z6EYRVJCXCEIIN7GW6KFGRVEZSIWCQ4FFXT34Q5QIDYEIFW2QD5A) context deadline exceeded; piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: PSYLX6PNTZAP4UHHSXMGWNDAQPLOUKRPWKKDXZEYLZDACYI2DNTQ) context deadline exceeded ======LOGS====== 1 20:51:23.460064987 (629h22m23.331148022s) [upload=551506]: Starting upload Dec 31 20:51:23.460124451 (629h22m23.331207486s) [upload=551506]: Done starting upload Dec 31 20:51:23.460173073 (629h22m23.331256108s) [upload=551506]: Uploading segments... Dec 31 20:51:23.460298723 (629h22m23.331381758s) [upload=551506]: Got next segment. Inline: false Dec 31 20:51:23.460327628 (629h22m23.331410663s) [upload=551506]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 20:51:23.460332086 (629h22m23.331415121s) [upload=551506]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 20:51:23.708004630 (629h22m23.579087665s) [seg_pos={0 0}, upload=551506]: Begin upload segment... Dec 31 20:51:23.708053202 (629h22m23.579136237s) [seg_pos={0 0}, upload=551506]: Done begin upload segment. Dec 31 20:51:23.708079312 (629h22m23.579162347s) [noise=true , address=145.79.227.27:4689 , piece_id=XM2WI4TG2UL4TCFDRKVHRSQRE2BK2EBV6Y6JSUA6EZS3TJNNDX3Q, seg_pos={0 0}, upload=551506]: Uploading piece... Dec 31 20:51:23.708297587 (629h22m23.579380632s) [noise=true , address=145.79.227.23:4547 , piece_id=TX7OGE7PRC424AZPTCSQVSN2OCX2GDQMMXANIKZEPEISPNBQ7ADQ, seg_pos={0 0}, upload=551506]: Uploading piece... Dec 31 20:51:23.709027757 (629h22m23.580110803s) [noise=true , address=45.91.116.9:4261 , piece_id=JZFL73GDR56U4KVELCJTTXQNMWKGOCHWSSKWBE5ZB4CHEBPGRCXQ, seg_pos={0 0}, upload=551506]: Uploading piece... Dec 31 20:51:23.709691221 (629h22m23.580774266s) [noise=true , address=45.91.116.5:4297 , piece_id=PSYLX6PNTZAP4UHHSXMGWNDAQPLOUKRPWKKDXZEYLZDACYI2DNTQ, seg_pos={0 0}, upload=551506]: Uploading piece... Dec 31 20:51:23.710208286 (629h22m23.581291321s) [noise=true , address=45.91.116.10:4084 , piece_id=B3RPVHCWDSDXJFBLFI2HE2RD7ECKJLQT37Q3MDHCFLXXD3WNLWPA, seg_pos={0 0}, upload=551506]: Uploading piece... Dec 31 20:51:23.710215549 (629h22m23.581298584s) [noise=true , address=45.91.116.4:4940 , piece_id=23IH4MXTUDM37G4YBB7XGFJKOYEEXC4WBSOW4WJE4CXN7HVCJL3A, seg_pos={0 0}, upload=551506]: Uploading piece... Dec 31 20:51:23.710515751 (629h22m23.581598786s) [noise=true , address=145.79.227.15:4874 , piece_id=6RDUE67ASMARP4OCLPC5BPS6A64OOOKIAGUC6Q3F45R5UZJEJAJQ, seg_pos={0 0}, upload=551506]: Uploading piece... Dec 31 20:51:23.710671818 (629h22m23.581754853s) [noise=true , address=145.79.227.51:4411 , piece_id=BUUUPVH5YP7ZDUG44OU5RUTUXBJ6N4BESJSSYUUTOILNH4AJZWGA, seg_pos={0 0}, upload=551506]: Uploading piece... Dec 31 20:51:23.710782529 (629h22m23.581865565s) [noise=true , address=77.90.31.10:4519 , piece_id=FBS27JGWQEG3LNUBOIOPQCVZOKN5RY3BJBDXXQC2HG3PZZ46CZWA, seg_pos={0 0}, upload=551506]: Uploading piece... Dec 31 20:51:23.711055869 (629h22m23.582138914s) [noise=true , address=145.79.227.45:4229 , piece_id=OQVGZDBYFJWANIC7GAFUFWJWZS3LIT6CEWWXT5SPLR7M7VHEGH2A, seg_pos={0 0}, upload=551506]: Uploading piece... Dec 31 20:51:23.711130170 (629h22m23.582213216s) [noise=true , address=145.79.227.17:4970 , piece_id=3U7GCTZ6ZPJSDY7QJRRYLNOUY64LYBB7NA2AQRN3RTDTCWKL5H5A, seg_pos={0 0}, upload=551506]: Uploading piece... Dec 31 20:51:23.711404543 (629h22m23.582487578s) [noise=true , address=77.90.31.12:4590 , piece_id=X6F6LRMFT3VSDQFB2K723QPZ2724RXYFSWOWQMWZFGAUIVO5BNKQ, seg_pos={0 0}, upload=551506]: Uploading piece... Dec 31 20:51:23.711627709 (629h22m23.582710744s) [noise=true , address=145.79.227.34:4906 , piece_id=IAS4I7S3PYHZ5ODX2E6GLUBKWPTUBMZSIC5JU4I32LDZ2675U7LA, seg_pos={0 0}, upload=551506]: Uploading piece... Dec 31 20:51:23.711680569 (629h22m23.582763604s) [noise=true , address=145.79.227.2:4114 , piece_id=COATTNXHVC5ZR5MWZO7ZZOXSKRIEHWLV564VWSLWPELML2F5HQOA, seg_pos={0 0}, upload=551506]: Uploading piece... Dec 31 20:51:23.711969510 (629h22m23.583052545s) [noise=true , address=45.91.116.8:4045 , piece_id=KHFRRWXG2X32QMMYBXIAKNXLFNGCCL4UKPZHRHLLYRPGJWAJFVNQ, seg_pos={0 0}, upload=551506]: Uploading piece... Dec 31 20:51:23.711981542 (629h22m23.583064587s) [noise=true , address=77.90.31.21:4880 , piece_id=Z6EYRVJCXCEIIN7GW6KFGRVEZSIWCQ4FFXT34Q5QIDYEIFW2QD5A, seg_pos={0 0}, upload=551506]: Uploading piece... Dec 31 20:51:25.404572075 (629h22m25.27565514s) [upload=551506]: Next returned nil segment Dec 31 20:51:25.404596711 (629h22m25.275679776s) [upload=551506]: Waiting for error group managing segments... Dec 31 20:51:25.686038758 (629h22m25.557121803s) [noise=true , address=145.79.227.2:4114 , piece_id=COATTNXHVC5ZR5MWZO7ZZOXSKRIEHWLV564VWSLWPELML2F5HQOA, seg_pos={0 0}, upload=551506]: Done uploading piece. err: <nil> Dec 31 20:51:25.687291203 (629h22m25.558374248s) [noise=true , address=145.79.227.23:4547 , piece_id=TX7OGE7PRC424AZPTCSQVSN2OCX2GDQMMXANIKZEPEISPNBQ7ADQ, seg_pos={0 0}, upload=551506]: Done uploading piece. err: <nil> Dec 31 20:51:25.699906604 (629h22m25.570989649s) [noise=true , address=145.79.227.15:4874 , piece_id=6RDUE67ASMARP4OCLPC5BPS6A64OOOKIAGUC6Q3F45R5UZJEJAJQ, seg_pos={0 0}, upload=551506]: Done uploading piece. err: <nil> Dec 31 20:51:25.705013989 (629h22m25.576097024s) [noise=true , address=145.79.227.45:4229 , piece_id=OQVGZDBYFJWANIC7GAFUFWJWZS3LIT6CEWWXT5SPLR7M7VHEGH2A, seg_pos={0 0}, upload=551506]: Done uploading piece. err: <nil> Dec 31 20:51:25.768154168 (629h22m25.639237203s) [noise=true , address=145.79.227.17:4970 , piece_id=3U7GCTZ6ZPJSDY7QJRRYLNOUY64LYBB7NA2AQRN3RTDTCWKL5H5A, seg_pos={0 0}, upload=551506]: Done uploading piece. err: <nil> Dec 31 20:51:25.785619824 (629h22m25.656702859s) [noise=true , address=145.79.227.51:4411 , piece_id=BUUUPVH5YP7ZDUG44OU5RUTUXBJ6N4BESJSSYUUTOILNH4AJZWGA, seg_pos={0 0}, upload=551506]: Done uploading piece. err: <nil> Dec 31 20:51:25.806597689 (629h22m25.677680725s) [noise=true , address=145.79.227.27:4689 , piece_id=XM2WI4TG2UL4TCFDRKVHRSQRE2BK2EBV6Y6JSUA6EZS3TJNNDX3Q, seg_pos={0 0}, upload=551506]: Done uploading piece. err: <nil> Dec 31 20:51:25.883364705 (629h22m25.75444774s) [noise=true , address=77.90.31.12:4590 , piece_id=X6F6LRMFT3VSDQFB2K723QPZ2724RXYFSWOWQMWZFGAUIVO5BNKQ, seg_pos={0 0}, upload=551506]: Done uploading piece. err: <nil> Dec 31 20:51:25.984313866 (629h22m25.855396911s) [noise=true , address=77.90.31.21:4880 , piece_id=Z6EYRVJCXCEIIN7GW6KFGRVEZSIWCQ4FFXT34Q5QIDYEIFW2QD5A, seg_pos={0 0}, upload=551506]: Done uploading piece. err: <nil> Dec 31 20:51:25.986455264 (629h22m25.857538299s) [noise=true , address=77.90.31.10:4519 , piece_id=FBS27JGWQEG3LNUBOIOPQCVZOKN5RY3BJBDXXQC2HG3PZZ46CZWA, seg_pos={0 0}, upload=551506]: Done uploading piece. err: <nil> Dec 31 20:51:26.002926057 (629h22m25.874009102s) [noise=true , address=45.91.116.8:4045 , piece_id=KHFRRWXG2X32QMMYBXIAKNXLFNGCCL4UKPZHRHLLYRPGJWAJFVNQ, seg_pos={0 0}, upload=551506]: Done uploading piece. err: <nil> Dec 31 20:51:26.057447813 (629h22m25.928530848s) [noise=true , address=45.91.116.5:4297 , piece_id=PSYLX6PNTZAP4UHHSXMGWNDAQPLOUKRPWKKDXZEYLZDACYI2DNTQ, seg_pos={0 0}, upload=551506]: Done uploading piece. err: <nil> Dec 31 20:51:26.175629479 (629h22m26.046712524s) [noise=true , address=45.91.116.10:4084 , piece_id=B3RPVHCWDSDXJFBLFI2HE2RD7ECKJLQT37Q3MDHCFLXXD3WNLWPA, seg_pos={0 0}, upload=551506]: Done uploading piece. err: <nil> Dec 31 20:51:26.175653634 (629h22m26.046736669s) [seg_pos={0 0}, upload=551506]: Segment reached optimal threshold of 13 pieces. Dec 31 20:51:26.175909862 (629h22m26.046992908s) [noise=true , address=45.91.116.4:4940 , piece_id=23IH4MXTUDM37G4YBB7XGFJKOYEEXC4WBSOW4WJE4CXN7HVCJL3A, seg_pos={0 0}, upload=551506]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Dec 31 20:51:26.175921114 (629h22m26.047004149s) [noise=true , address=45.91.116.4:4940 , piece_id=23IH4MXTUDM37G4YBB7XGFJKOYEEXC4WBSOW4WJE4CXN7HVCJL3A, seg_pos={0 0}, upload=551506]: Long tail cancellation due to optimal threshold Dec 31 20:51:26.175954317 (629h22m26.047037362s) [noise=true , address=145.79.227.34:4906 , piece_id=IAS4I7S3PYHZ5ODX2E6GLUBKWPTUBMZSIC5JU4I32LDZ2675U7LA, seg_pos={0 0}, upload=551506]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 31 20:51:26.175964306 (629h22m26.047047351s) [noise=true , address=145.79.227.34:4906 , piece_id=IAS4I7S3PYHZ5ODX2E6GLUBKWPTUBMZSIC5JU4I32LDZ2675U7LA, seg_pos={0 0}, upload=551506]: Long tail cancellation due to optimal threshold Dec 31 20:51:26.176121806 (629h22m26.047204851s) [noise=true , address=45.91.116.9:4261 , piece_id=JZFL73GDR56U4KVELCJTTXQNMWKGOCHWSSKWBE5ZB4CHEBPGRCXQ, seg_pos={0 0}, upload=551506]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: write tcp 188.166.225.170:56626->45.91.116.9:4261: use of closed network connection; write tcp 188.166.225.170:56626->45.91.116.9:4261: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56626->45.91.116.9:4261: use of closed network connection Dec 31 20:51:26.176128408 (629h22m26.047211443s) [noise=true , address=45.91.116.9:4261 , piece_id=JZFL73GDR56U4KVELCJTTXQNMWKGOCHWSSKWBE5ZB4CHEBPGRCXQ, seg_pos={0 0}, upload=551506]: Long tail cancellation due to optimal threshold Dec 31 20:51:26.176146222 (629h22m26.047229257s) [seg_pos={0 0}, upload=551506]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 20:51:26.176216866 (629h22m26.047299911s) [upload=551506]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 20:51:26.176221676 (629h22m26.047304721s) [upload=551506]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 20:51:26.474867771 (629h22m26.345950806s) [upload=551506]: Done uploading segments... Dec 31 20:51:26.474891285 (629h22m26.34597433s) [upload=551506]: Upload finished. err: <nil> Duration: 6.148793981s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [..X.....X.......XX...X..XX...X...XX....X......X....XX.......] duration: 16.058543453s trace: https://jaeger.dev.storj.io/trace/3c6720cb0382e95c FAIL (2025-12-31 20:56:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: Z6EYRVJCXCEIIN7GW6KFGRVEZSIWCQ4FFXT34Q5QIDYEIFW2QD5A) context deadline exceeded; piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: PSYLX6PNTZAP4UHHSXMGWNDAQPLOUKRPWKKDXZEYLZDACYI2DNTQ) context deadline exceeded ======LOGS====== 1 20:51:23.460064987 (629h22m23.331148022s) [upload=551506]: Starting upload Dec 31 20:51:23.460124451 (629h22m23.331207486s) [upload=551506]: Done starting upload Dec 31 20:51:23.460173073 (629h22m23.331256108s) [upload=551506]: Uploading segments... Dec 31 20:51:23.460298723 (629h22m23.331381758s) [upload=551506]: Got next segment. Inline: false Dec 31 20:51:23.460327628 (629h22m23.331410663s) [upload=551506]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 20:51:23.460332086 (629h22m23.331415121s) [upload=551506]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 20:51:23.708004630 (629h22m23.579087665s) [seg_pos={0 0}, upload=551506]: Begin upload segment... Dec 31 20:51:23.708053202 (629h22m23.579136237s) [seg_pos={0 0}, upload=551506]: Done begin upload segment. Dec 31 20:51:23.708079312 (629h22m23.579162347s) [noise=true , address=145.79.227.27:4689 , piece_id=XM2WI4TG2UL4TCFDRKVHRSQRE2BK2EBV6Y6JSUA6EZS3TJNNDX3Q, seg_pos={0 0}, upload=551506]: Uploading piece... Dec 31 20:51:23.708297587 (629h22m23.579380632s) [noise=true , address=145.79.227.23:4547 , piece_id=TX7OGE7PRC424AZPTCSQVSN2OCX2GDQMMXANIKZEPEISPNBQ7ADQ, seg_pos={0 0}, upload=551506]: Uploading piece... Dec 31 20:51:23.709027757 (629h22m23.580110803s) [noise=true , address=45.91.116.9:4261 , piece_id=JZFL73GDR56U4KVELCJTTXQNMWKGOCHWSSKWBE5ZB4CHEBPGRCXQ, seg_pos={0 0}, upload=551506]: Uploading piece... Dec 31 20:51:23.709691221 (629h22m23.580774266s) [noise=true , address=45.91.116.5:4297 , piece_id=PSYLX6PNTZAP4UHHSXMGWNDAQPLOUKRPWKKDXZEYLZDACYI2DNTQ, seg_pos={0 0}, upload=551506]: Uploading piece... Dec 31 20:51:23.710208286 (629h22m23.581291321s) [noise=true , address=45.91.116.10:4084 , piece_id=B3RPVHCWDSDXJFBLFI2HE2RD7ECKJLQT37Q3MDHCFLXXD3WNLWPA, seg_pos={0 0}, upload=551506]: Uploading piece... Dec 31 20:51:23.710215549 (629h22m23.581298584s) [noise=true , address=45.91.116.4:4940 , piece_id=23IH4MXTUDM37G4YBB7XGFJKOYEEXC4WBSOW4WJE4CXN7HVCJL3A, seg_pos={0 0}, upload=551506]: Uploading piece... Dec 31 20:51:23.710515751 (629h22m23.581598786s) [noise=true , address=145.79.227.15:4874 , piece_id=6RDUE67ASMARP4OCLPC5BPS6A64OOOKIAGUC6Q3F45R5UZJEJAJQ, seg_pos={0 0}, upload=551506]: Uploading piece... Dec 31 20:51:23.710671818 (629h22m23.581754853s) [noise=true , address=145.79.227.51:4411 , piece_id=BUUUPVH5YP7ZDUG44OU5RUTUXBJ6N4BESJSSYUUTOILNH4AJZWGA, seg_pos={0 0}, upload=551506]: Uploading piece... Dec 31 20:51:23.710782529 (629h22m23.581865565s) [noise=true , address=77.90.31.10:4519 , piece_id=FBS27JGWQEG3LNUBOIOPQCVZOKN5RY3BJBDXXQC2HG3PZZ46CZWA, seg_pos={0 0}, upload=551506]: Uploading piece... Dec 31 20:51:23.711055869 (629h22m23.582138914s) [noise=true , address=145.79.227.45:4229 , piece_id=OQVGZDBYFJWANIC7GAFUFWJWZS3LIT6CEWWXT5SPLR7M7VHEGH2A, seg_pos={0 0}, upload=551506]: Uploading piece... Dec 31 20:51:23.711130170 (629h22m23.582213216s) [noise=true , address=145.79.227.17:4970 , piece_id=3U7GCTZ6ZPJSDY7QJRRYLNOUY64LYBB7NA2AQRN3RTDTCWKL5H5A, seg_pos={0 0}, upload=551506]: Uploading piece... Dec 31 20:51:23.711404543 (629h22m23.582487578s) [noise=true , address=77.90.31.12:4590 , piece_id=X6F6LRMFT3VSDQFB2K723QPZ2724RXYFSWOWQMWZFGAUIVO5BNKQ, seg_pos={0 0}, upload=551506]: Uploading piece... Dec 31 20:51:23.711627709 (629h22m23.582710744s) [noise=true , address=145.79.227.34:4906 , piece_id=IAS4I7S3PYHZ5ODX2E6GLUBKWPTUBMZSIC5JU4I32LDZ2675U7LA, seg_pos={0 0}, upload=551506]: Uploading piece... Dec 31 20:51:23.711680569 (629h22m23.582763604s) [noise=true , address=145.79.227.2:4114 , piece_id=COATTNXHVC5ZR5MWZO7ZZOXSKRIEHWLV564VWSLWPELML2F5HQOA, seg_pos={0 0}, upload=551506]: Uploading piece... Dec 31 20:51:23.711969510 (629h22m23.583052545s) [noise=true , address=45.91.116.8:4045 , piece_id=KHFRRWXG2X32QMMYBXIAKNXLFNGCCL4UKPZHRHLLYRPGJWAJFVNQ, seg_pos={0 0}, upload=551506]: Uploading piece... Dec 31 20:51:23.711981542 (629h22m23.583064587s) [noise=true , address=77.90.31.21:4880 , piece_id=Z6EYRVJCXCEIIN7GW6KFGRVEZSIWCQ4FFXT34Q5QIDYEIFW2QD5A, seg_pos={0 0}, upload=551506]: Uploading piece... Dec 31 20:51:25.404572075 (629h22m25.27565514s) [upload=551506]: Next returned nil segment Dec 31 20:51:25.404596711 (629h22m25.275679776s) [upload=551506]: Waiting for error group managing segments... Dec 31 20:51:25.686038758 (629h22m25.557121803s) [noise=true , address=145.79.227.2:4114 , piece_id=COATTNXHVC5ZR5MWZO7ZZOXSKRIEHWLV564VWSLWPELML2F5HQOA, seg_pos={0 0}, upload=551506]: Done uploading piece. err: <nil> Dec 31 20:51:25.687291203 (629h22m25.558374248s) [noise=true , address=145.79.227.23:4547 , piece_id=TX7OGE7PRC424AZPTCSQVSN2OCX2GDQMMXANIKZEPEISPNBQ7ADQ, seg_pos={0 0}, upload=551506]: Done uploading piece. err: <nil> Dec 31 20:51:25.699906604 (629h22m25.570989649s) [noise=true , address=145.79.227.15:4874 , piece_id=6RDUE67ASMARP4OCLPC5BPS6A64OOOKIAGUC6Q3F45R5UZJEJAJQ, seg_pos={0 0}, upload=551506]: Done uploading piece. err: <nil> Dec 31 20:51:25.705013989 (629h22m25.576097024s) [noise=true , address=145.79.227.45:4229 , piece_id=OQVGZDBYFJWANIC7GAFUFWJWZS3LIT6CEWWXT5SPLR7M7VHEGH2A, seg_pos={0 0}, upload=551506]: Done uploading piece. err: <nil> Dec 31 20:51:25.768154168 (629h22m25.639237203s) [noise=true , address=145.79.227.17:4970 , piece_id=3U7GCTZ6ZPJSDY7QJRRYLNOUY64LYBB7NA2AQRN3RTDTCWKL5H5A, seg_pos={0 0}, upload=551506]: Done uploading piece. err: <nil> Dec 31 20:51:25.785619824 (629h22m25.656702859s) [noise=true , address=145.79.227.51:4411 , piece_id=BUUUPVH5YP7ZDUG44OU5RUTUXBJ6N4BESJSSYUUTOILNH4AJZWGA, seg_pos={0 0}, upload=551506]: Done uploading piece. err: <nil> Dec 31 20:51:25.806597689 (629h22m25.677680725s) [noise=true , address=145.79.227.27:4689 , piece_id=XM2WI4TG2UL4TCFDRKVHRSQRE2BK2EBV6Y6JSUA6EZS3TJNNDX3Q, seg_pos={0 0}, upload=551506]: Done uploading piece. err: <nil> Dec 31 20:51:25.883364705 (629h22m25.75444774s) [noise=true , address=77.90.31.12:4590 , piece_id=X6F6LRMFT3VSDQFB2K723QPZ2724RXYFSWOWQMWZFGAUIVO5BNKQ, seg_pos={0 0}, upload=551506]: Done uploading piece. err: <nil> Dec 31 20:51:25.984313866 (629h22m25.855396911s) [noise=true , address=77.90.31.21:4880 , piece_id=Z6EYRVJCXCEIIN7GW6KFGRVEZSIWCQ4FFXT34Q5QIDYEIFW2QD5A, seg_pos={0 0}, upload=551506]: Done uploading piece. err: <nil> Dec 31 20:51:25.986455264 (629h22m25.857538299s) [noise=true , address=77.90.31.10:4519 , piece_id=FBS27JGWQEG3LNUBOIOPQCVZOKN5RY3BJBDXXQC2HG3PZZ46CZWA, seg_pos={0 0}, upload=551506]: Done uploading piece. err: <nil> Dec 31 20:51:26.002926057 (629h22m25.874009102s) [noise=true , address=45.91.116.8:4045 , piece_id=KHFRRWXG2X32QMMYBXIAKNXLFNGCCL4UKPZHRHLLYRPGJWAJFVNQ, seg_pos={0 0}, upload=551506]: Done uploading piece. err: <nil> Dec 31 20:51:26.057447813 (629h22m25.928530848s) [noise=true , address=45.91.116.5:4297 , piece_id=PSYLX6PNTZAP4UHHSXMGWNDAQPLOUKRPWKKDXZEYLZDACYI2DNTQ, seg_pos={0 0}, upload=551506]: Done uploading piece. err: <nil> Dec 31 20:51:26.175629479 (629h22m26.046712524s) [noise=true , address=45.91.116.10:4084 , piece_id=B3RPVHCWDSDXJFBLFI2HE2RD7ECKJLQT37Q3MDHCFLXXD3WNLWPA, seg_pos={0 0}, upload=551506]: Done uploading piece. err: <nil> Dec 31 20:51:26.175653634 (629h22m26.046736669s) [seg_pos={0 0}, upload=551506]: Segment reached optimal threshold of 13 pieces. Dec 31 20:51:26.175909862 (629h22m26.046992908s) [noise=true , address=45.91.116.4:4940 , piece_id=23IH4MXTUDM37G4YBB7XGFJKOYEEXC4WBSOW4WJE4CXN7HVCJL3A, seg_pos={0 0}, upload=551506]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Dec 31 20:51:26.175921114 (629h22m26.047004149s) [noise=true , address=45.91.116.4:4940 , piece_id=23IH4MXTUDM37G4YBB7XGFJKOYEEXC4WBSOW4WJE4CXN7HVCJL3A, seg_pos={0 0}, upload=551506]: Long tail cancellation due to optimal threshold Dec 31 20:51:26.175954317 (629h22m26.047037362s) [noise=true , address=145.79.227.34:4906 , piece_id=IAS4I7S3PYHZ5ODX2E6GLUBKWPTUBMZSIC5JU4I32LDZ2675U7LA, seg_pos={0 0}, upload=551506]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 31 20:51:26.175964306 (629h22m26.047047351s) [noise=true , address=145.79.227.34:4906 , piece_id=IAS4I7S3PYHZ5ODX2E6GLUBKWPTUBMZSIC5JU4I32LDZ2675U7LA, seg_pos={0 0}, upload=551506]: Long tail cancellation due to optimal threshold Dec 31 20:51:26.176121806 (629h22m26.047204851s) [noise=true , address=45.91.116.9:4261 , piece_id=JZFL73GDR56U4KVELCJTTXQNMWKGOCHWSSKWBE5ZB4CHEBPGRCXQ, seg_pos={0 0}, upload=551506]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: write tcp 188.166.225.170:56626->45.91.116.9:4261: use of closed network connection; write tcp 188.166.225.170:56626->45.91.116.9:4261: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56626->45.91.116.9:4261: use of closed network connection Dec 31 20:51:26.176128408 (629h22m26.047211443s) [noise=true , address=45.91.116.9:4261 , piece_id=JZFL73GDR56U4KVELCJTTXQNMWKGOCHWSSKWBE5ZB4CHEBPGRCXQ, seg_pos={0 0}, upload=551506]: Long tail cancellation due to optimal threshold Dec 31 20:51:26.176146222 (629h22m26.047229257s) [seg_pos={0 0}, upload=551506]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 20:51:26.176216866 (629h22m26.047299911s) [upload=551506]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 20:51:26.176221676 (629h22m26.047304721s) [upload=551506]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 20:51:26.474867771 (629h22m26.345950806s) [upload=551506]: Done uploading segments... Dec 31 20:51:26.474891285 (629h22m26.34597433s) [upload=551506]: Upload finished. err: <nil> Duration: 16.058574582s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [X..X.....X.......XX...X..XX...X...XX....X......X....XX......] duration: 6.06400709s trace: https://jaeger.dev.storj.io/trace/3c6720cb0382e95c FAIL (2025-12-31 20:55:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: Z6EYRVJCXCEIIN7GW6KFGRVEZSIWCQ4FFXT34Q5QIDYEIFW2QD5A) context deadline exceeded; piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: PSYLX6PNTZAP4UHHSXMGWNDAQPLOUKRPWKKDXZEYLZDACYI2DNTQ) context deadline exceeded ======LOGS====== 1 20:51:23.460064987 (629h22m23.331148022s) [upload=551506]: Starting upload Dec 31 20:51:23.460124451 (629h22m23.331207486s) [upload=551506]: Done starting upload Dec 31 20:51:23.460173073 (629h22m23.331256108s) [upload=551506]: Uploading segments... Dec 31 20:51:23.460298723 (629h22m23.331381758s) [upload=551506]: Got next segment. Inline: false Dec 31 20:51:23.460327628 (629h22m23.331410663s) [upload=551506]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 20:51:23.460332086 (629h22m23.331415121s) [upload=551506]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 20:51:23.708004630 (629h22m23.579087665s) [seg_pos={0 0}, upload=551506]: Begin upload segment... Dec 31 20:51:23.708053202 (629h22m23.579136237s) [seg_pos={0 0}, upload=551506]: Done begin upload segment. Dec 31 20:51:23.708079312 (629h22m23.579162347s) [noise=true , address=145.79.227.27:4689 , piece_id=XM2WI4TG2UL4TCFDRKVHRSQRE2BK2EBV6Y6JSUA6EZS3TJNNDX3Q, seg_pos={0 0}, upload=551506]: Uploading piece... Dec 31 20:51:23.708297587 (629h22m23.579380632s) [noise=true , address=145.79.227.23:4547 , piece_id=TX7OGE7PRC424AZPTCSQVSN2OCX2GDQMMXANIKZEPEISPNBQ7ADQ, seg_pos={0 0}, upload=551506]: Uploading piece... Dec 31 20:51:23.709027757 (629h22m23.580110803s) [noise=true , address=45.91.116.9:4261 , piece_id=JZFL73GDR56U4KVELCJTTXQNMWKGOCHWSSKWBE5ZB4CHEBPGRCXQ, seg_pos={0 0}, upload=551506]: Uploading piece... Dec 31 20:51:23.709691221 (629h22m23.580774266s) [noise=true , address=45.91.116.5:4297 , piece_id=PSYLX6PNTZAP4UHHSXMGWNDAQPLOUKRPWKKDXZEYLZDACYI2DNTQ, seg_pos={0 0}, upload=551506]: Uploading piece... Dec 31 20:51:23.710208286 (629h22m23.581291321s) [noise=true , address=45.91.116.10:4084 , piece_id=B3RPVHCWDSDXJFBLFI2HE2RD7ECKJLQT37Q3MDHCFLXXD3WNLWPA, seg_pos={0 0}, upload=551506]: Uploading piece... Dec 31 20:51:23.710215549 (629h22m23.581298584s) [noise=true , address=45.91.116.4:4940 , piece_id=23IH4MXTUDM37G4YBB7XGFJKOYEEXC4WBSOW4WJE4CXN7HVCJL3A, seg_pos={0 0}, upload=551506]: Uploading piece... Dec 31 20:51:23.710515751 (629h22m23.581598786s) [noise=true , address=145.79.227.15:4874 , piece_id=6RDUE67ASMARP4OCLPC5BPS6A64OOOKIAGUC6Q3F45R5UZJEJAJQ, seg_pos={0 0}, upload=551506]: Uploading piece... Dec 31 20:51:23.710671818 (629h22m23.581754853s) [noise=true , address=145.79.227.51:4411 , piece_id=BUUUPVH5YP7ZDUG44OU5RUTUXBJ6N4BESJSSYUUTOILNH4AJZWGA, seg_pos={0 0}, upload=551506]: Uploading piece... Dec 31 20:51:23.710782529 (629h22m23.581865565s) [noise=true , address=77.90.31.10:4519 , piece_id=FBS27JGWQEG3LNUBOIOPQCVZOKN5RY3BJBDXXQC2HG3PZZ46CZWA, seg_pos={0 0}, upload=551506]: Uploading piece... Dec 31 20:51:23.711055869 (629h22m23.582138914s) [noise=true , address=145.79.227.45:4229 , piece_id=OQVGZDBYFJWANIC7GAFUFWJWZS3LIT6CEWWXT5SPLR7M7VHEGH2A, seg_pos={0 0}, upload=551506]: Uploading piece... Dec 31 20:51:23.711130170 (629h22m23.582213216s) [noise=true , address=145.79.227.17:4970 , piece_id=3U7GCTZ6ZPJSDY7QJRRYLNOUY64LYBB7NA2AQRN3RTDTCWKL5H5A, seg_pos={0 0}, upload=551506]: Uploading piece... Dec 31 20:51:23.711404543 (629h22m23.582487578s) [noise=true , address=77.90.31.12:4590 , piece_id=X6F6LRMFT3VSDQFB2K723QPZ2724RXYFSWOWQMWZFGAUIVO5BNKQ, seg_pos={0 0}, upload=551506]: Uploading piece... Dec 31 20:51:23.711627709 (629h22m23.582710744s) [noise=true , address=145.79.227.34:4906 , piece_id=IAS4I7S3PYHZ5ODX2E6GLUBKWPTUBMZSIC5JU4I32LDZ2675U7LA, seg_pos={0 0}, upload=551506]: Uploading piece... Dec 31 20:51:23.711680569 (629h22m23.582763604s) [noise=true , address=145.79.227.2:4114 , piece_id=COATTNXHVC5ZR5MWZO7ZZOXSKRIEHWLV564VWSLWPELML2F5HQOA, seg_pos={0 0}, upload=551506]: Uploading piece... Dec 31 20:51:23.711969510 (629h22m23.583052545s) [noise=true , address=45.91.116.8:4045 , piece_id=KHFRRWXG2X32QMMYBXIAKNXLFNGCCL4UKPZHRHLLYRPGJWAJFVNQ, seg_pos={0 0}, upload=551506]: Uploading piece... Dec 31 20:51:23.711981542 (629h22m23.583064587s) [noise=true , address=77.90.31.21:4880 , piece_id=Z6EYRVJCXCEIIN7GW6KFGRVEZSIWCQ4FFXT34Q5QIDYEIFW2QD5A, seg_pos={0 0}, upload=551506]: Uploading piece... Dec 31 20:51:25.404572075 (629h22m25.27565514s) [upload=551506]: Next returned nil segment Dec 31 20:51:25.404596711 (629h22m25.275679776s) [upload=551506]: Waiting for error group managing segments... Dec 31 20:51:25.686038758 (629h22m25.557121803s) [noise=true , address=145.79.227.2:4114 , piece_id=COATTNXHVC5ZR5MWZO7ZZOXSKRIEHWLV564VWSLWPELML2F5HQOA, seg_pos={0 0}, upload=551506]: Done uploading piece. err: <nil> Dec 31 20:51:25.687291203 (629h22m25.558374248s) [noise=true , address=145.79.227.23:4547 , piece_id=TX7OGE7PRC424AZPTCSQVSN2OCX2GDQMMXANIKZEPEISPNBQ7ADQ, seg_pos={0 0}, upload=551506]: Done uploading piece. err: <nil> Dec 31 20:51:25.699906604 (629h22m25.570989649s) [noise=true , address=145.79.227.15:4874 , piece_id=6RDUE67ASMARP4OCLPC5BPS6A64OOOKIAGUC6Q3F45R5UZJEJAJQ, seg_pos={0 0}, upload=551506]: Done uploading piece. err: <nil> Dec 31 20:51:25.705013989 (629h22m25.576097024s) [noise=true , address=145.79.227.45:4229 , piece_id=OQVGZDBYFJWANIC7GAFUFWJWZS3LIT6CEWWXT5SPLR7M7VHEGH2A, seg_pos={0 0}, upload=551506]: Done uploading piece. err: <nil> Dec 31 20:51:25.768154168 (629h22m25.639237203s) [noise=true , address=145.79.227.17:4970 , piece_id=3U7GCTZ6ZPJSDY7QJRRYLNOUY64LYBB7NA2AQRN3RTDTCWKL5H5A, seg_pos={0 0}, upload=551506]: Done uploading piece. err: <nil> Dec 31 20:51:25.785619824 (629h22m25.656702859s) [noise=true , address=145.79.227.51:4411 , piece_id=BUUUPVH5YP7ZDUG44OU5RUTUXBJ6N4BESJSSYUUTOILNH4AJZWGA, seg_pos={0 0}, upload=551506]: Done uploading piece. err: <nil> Dec 31 20:51:25.806597689 (629h22m25.677680725s) [noise=true , address=145.79.227.27:4689 , piece_id=XM2WI4TG2UL4TCFDRKVHRSQRE2BK2EBV6Y6JSUA6EZS3TJNNDX3Q, seg_pos={0 0}, upload=551506]: Done uploading piece. err: <nil> Dec 31 20:51:25.883364705 (629h22m25.75444774s) [noise=true , address=77.90.31.12:4590 , piece_id=X6F6LRMFT3VSDQFB2K723QPZ2724RXYFSWOWQMWZFGAUIVO5BNKQ, seg_pos={0 0}, upload=551506]: Done uploading piece. err: <nil> Dec 31 20:51:25.984313866 (629h22m25.855396911s) [noise=true , address=77.90.31.21:4880 , piece_id=Z6EYRVJCXCEIIN7GW6KFGRVEZSIWCQ4FFXT34Q5QIDYEIFW2QD5A, seg_pos={0 0}, upload=551506]: Done uploading piece. err: <nil> Dec 31 20:51:25.986455264 (629h22m25.857538299s) [noise=true , address=77.90.31.10:4519 , piece_id=FBS27JGWQEG3LNUBOIOPQCVZOKN5RY3BJBDXXQC2HG3PZZ46CZWA, seg_pos={0 0}, upload=551506]: Done uploading piece. err: <nil> Dec 31 20:51:26.002926057 (629h22m25.874009102s) [noise=true , address=45.91.116.8:4045 , piece_id=KHFRRWXG2X32QMMYBXIAKNXLFNGCCL4UKPZHRHLLYRPGJWAJFVNQ, seg_pos={0 0}, upload=551506]: Done uploading piece. err: <nil> Dec 31 20:51:26.057447813 (629h22m25.928530848s) [noise=true , address=45.91.116.5:4297 , piece_id=PSYLX6PNTZAP4UHHSXMGWNDAQPLOUKRPWKKDXZEYLZDACYI2DNTQ, seg_pos={0 0}, upload=551506]: Done uploading piece. err: <nil> Dec 31 20:51:26.175629479 (629h22m26.046712524s) [noise=true , address=45.91.116.10:4084 , piece_id=B3RPVHCWDSDXJFBLFI2HE2RD7ECKJLQT37Q3MDHCFLXXD3WNLWPA, seg_pos={0 0}, upload=551506]: Done uploading piece. err: <nil> Dec 31 20:51:26.175653634 (629h22m26.046736669s) [seg_pos={0 0}, upload=551506]: Segment reached optimal threshold of 13 pieces. Dec 31 20:51:26.175909862 (629h22m26.046992908s) [noise=true , address=45.91.116.4:4940 , piece_id=23IH4MXTUDM37G4YBB7XGFJKOYEEXC4WBSOW4WJE4CXN7HVCJL3A, seg_pos={0 0}, upload=551506]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Dec 31 20:51:26.175921114 (629h22m26.047004149s) [noise=true , address=45.91.116.4:4940 , piece_id=23IH4MXTUDM37G4YBB7XGFJKOYEEXC4WBSOW4WJE4CXN7HVCJL3A, seg_pos={0 0}, upload=551506]: Long tail cancellation due to optimal threshold Dec 31 20:51:26.175954317 (629h22m26.047037362s) [noise=true , address=145.79.227.34:4906 , piece_id=IAS4I7S3PYHZ5ODX2E6GLUBKWPTUBMZSIC5JU4I32LDZ2675U7LA, seg_pos={0 0}, upload=551506]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 31 20:51:26.175964306 (629h22m26.047047351s) [noise=true , address=145.79.227.34:4906 , piece_id=IAS4I7S3PYHZ5ODX2E6GLUBKWPTUBMZSIC5JU4I32LDZ2675U7LA, seg_pos={0 0}, upload=551506]: Long tail cancellation due to optimal threshold Dec 31 20:51:26.176121806 (629h22m26.047204851s) [noise=true , address=45.91.116.9:4261 , piece_id=JZFL73GDR56U4KVELCJTTXQNMWKGOCHWSSKWBE5ZB4CHEBPGRCXQ, seg_pos={0 0}, upload=551506]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: write tcp 188.166.225.170:56626->45.91.116.9:4261: use of closed network connection; write tcp 188.166.225.170:56626->45.91.116.9:4261: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56626->45.91.116.9:4261: use of closed network connection Dec 31 20:51:26.176128408 (629h22m26.047211443s) [noise=true , address=45.91.116.9:4261 , piece_id=JZFL73GDR56U4KVELCJTTXQNMWKGOCHWSSKWBE5ZB4CHEBPGRCXQ, seg_pos={0 0}, upload=551506]: Long tail cancellation due to optimal threshold Dec 31 20:51:26.176146222 (629h22m26.047229257s) [seg_pos={0 0}, upload=551506]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 20:51:26.176216866 (629h22m26.047299911s) [upload=551506]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 20:51:26.176221676 (629h22m26.047304721s) [upload=551506]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 20:51:26.474867771 (629h22m26.345950806s) [upload=551506]: Done uploading segments... Dec 31 20:51:26.474891285 (629h22m26.34597433s) [upload=551506]: Upload finished. err: <nil> Duration: 6.064039301s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.X..X.....X.......XX...X..XX...X...XX....X......X....XX.....] duration: 26.598050855s trace: https://jaeger.dev.storj.io/trace/3c6720cb0382e95c FAIL (2025-12-31 20:54:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: Z6EYRVJCXCEIIN7GW6KFGRVEZSIWCQ4FFXT34Q5QIDYEIFW2QD5A) context deadline exceeded; piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: PSYLX6PNTZAP4UHHSXMGWNDAQPLOUKRPWKKDXZEYLZDACYI2DNTQ) context deadline exceeded ======LOGS====== 1 20:51:23.460064987 (629h22m23.331148022s) [upload=551506]: Starting upload Dec 31 20:51:23.460124451 (629h22m23.331207486s) [upload=551506]: Done starting upload Dec 31 20:51:23.460173073 (629h22m23.331256108s) [upload=551506]: Uploading segments... Dec 31 20:51:23.460298723 (629h22m23.331381758s) [upload=551506]: Got next segment. Inline: false Dec 31 20:51:23.460327628 (629h22m23.331410663s) [upload=551506]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 20:51:23.460332086 (629h22m23.331415121s) [upload=551506]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 20:51:23.708004630 (629h22m23.579087665s) [seg_pos={0 0}, upload=551506]: Begin upload segment... Dec 31 20:51:23.708053202 (629h22m23.579136237s) [seg_pos={0 0}, upload=551506]: Done begin upload segment. Dec 31 20:51:23.708079312 (629h22m23.579162347s) [noise=true , address=145.79.227.27:4689 , piece_id=XM2WI4TG2UL4TCFDRKVHRSQRE2BK2EBV6Y6JSUA6EZS3TJNNDX3Q, seg_pos={0 0}, upload=551506]: Uploading piece... Dec 31 20:51:23.708297587 (629h22m23.579380632s) [noise=true , address=145.79.227.23:4547 , piece_id=TX7OGE7PRC424AZPTCSQVSN2OCX2GDQMMXANIKZEPEISPNBQ7ADQ, seg_pos={0 0}, upload=551506]: Uploading piece... Dec 31 20:51:23.709027757 (629h22m23.580110803s) [noise=true , address=45.91.116.9:4261 , piece_id=JZFL73GDR56U4KVELCJTTXQNMWKGOCHWSSKWBE5ZB4CHEBPGRCXQ, seg_pos={0 0}, upload=551506]: Uploading piece... Dec 31 20:51:23.709691221 (629h22m23.580774266s) [noise=true , address=45.91.116.5:4297 , piece_id=PSYLX6PNTZAP4UHHSXMGWNDAQPLOUKRPWKKDXZEYLZDACYI2DNTQ, seg_pos={0 0}, upload=551506]: Uploading piece... Dec 31 20:51:23.710208286 (629h22m23.581291321s) [noise=true , address=45.91.116.10:4084 , piece_id=B3RPVHCWDSDXJFBLFI2HE2RD7ECKJLQT37Q3MDHCFLXXD3WNLWPA, seg_pos={0 0}, upload=551506]: Uploading piece... Dec 31 20:51:23.710215549 (629h22m23.581298584s) [noise=true , address=45.91.116.4:4940 , piece_id=23IH4MXTUDM37G4YBB7XGFJKOYEEXC4WBSOW4WJE4CXN7HVCJL3A, seg_pos={0 0}, upload=551506]: Uploading piece... Dec 31 20:51:23.710515751 (629h22m23.581598786s) [noise=true , address=145.79.227.15:4874 , piece_id=6RDUE67ASMARP4OCLPC5BPS6A64OOOKIAGUC6Q3F45R5UZJEJAJQ, seg_pos={0 0}, upload=551506]: Uploading piece... Dec 31 20:51:23.710671818 (629h22m23.581754853s) [noise=true , address=145.79.227.51:4411 , piece_id=BUUUPVH5YP7ZDUG44OU5RUTUXBJ6N4BESJSSYUUTOILNH4AJZWGA, seg_pos={0 0}, upload=551506]: Uploading piece... Dec 31 20:51:23.710782529 (629h22m23.581865565s) [noise=true , address=77.90.31.10:4519 , piece_id=FBS27JGWQEG3LNUBOIOPQCVZOKN5RY3BJBDXXQC2HG3PZZ46CZWA, seg_pos={0 0}, upload=551506]: Uploading piece... Dec 31 20:51:23.711055869 (629h22m23.582138914s) [noise=true , address=145.79.227.45:4229 , piece_id=OQVGZDBYFJWANIC7GAFUFWJWZS3LIT6CEWWXT5SPLR7M7VHEGH2A, seg_pos={0 0}, upload=551506]: Uploading piece... Dec 31 20:51:23.711130170 (629h22m23.582213216s) [noise=true , address=145.79.227.17:4970 , piece_id=3U7GCTZ6ZPJSDY7QJRRYLNOUY64LYBB7NA2AQRN3RTDTCWKL5H5A, seg_pos={0 0}, upload=551506]: Uploading piece... Dec 31 20:51:23.711404543 (629h22m23.582487578s) [noise=true , address=77.90.31.12:4590 , piece_id=X6F6LRMFT3VSDQFB2K723QPZ2724RXYFSWOWQMWZFGAUIVO5BNKQ, seg_pos={0 0}, upload=551506]: Uploading piece... Dec 31 20:51:23.711627709 (629h22m23.582710744s) [noise=true , address=145.79.227.34:4906 , piece_id=IAS4I7S3PYHZ5ODX2E6GLUBKWPTUBMZSIC5JU4I32LDZ2675U7LA, seg_pos={0 0}, upload=551506]: Uploading piece... Dec 31 20:51:23.711680569 (629h22m23.582763604s) [noise=true , address=145.79.227.2:4114 , piece_id=COATTNXHVC5ZR5MWZO7ZZOXSKRIEHWLV564VWSLWPELML2F5HQOA, seg_pos={0 0}, upload=551506]: Uploading piece... Dec 31 20:51:23.711969510 (629h22m23.583052545s) [noise=true , address=45.91.116.8:4045 , piece_id=KHFRRWXG2X32QMMYBXIAKNXLFNGCCL4UKPZHRHLLYRPGJWAJFVNQ, seg_pos={0 0}, upload=551506]: Uploading piece... Dec 31 20:51:23.711981542 (629h22m23.583064587s) [noise=true , address=77.90.31.21:4880 , piece_id=Z6EYRVJCXCEIIN7GW6KFGRVEZSIWCQ4FFXT34Q5QIDYEIFW2QD5A, seg_pos={0 0}, upload=551506]: Uploading piece... Dec 31 20:51:25.404572075 (629h22m25.27565514s) [upload=551506]: Next returned nil segment Dec 31 20:51:25.404596711 (629h22m25.275679776s) [upload=551506]: Waiting for error group managing segments... Dec 31 20:51:25.686038758 (629h22m25.557121803s) [noise=true , address=145.79.227.2:4114 , piece_id=COATTNXHVC5ZR5MWZO7ZZOXSKRIEHWLV564VWSLWPELML2F5HQOA, seg_pos={0 0}, upload=551506]: Done uploading piece. err: <nil> Dec 31 20:51:25.687291203 (629h22m25.558374248s) [noise=true , address=145.79.227.23:4547 , piece_id=TX7OGE7PRC424AZPTCSQVSN2OCX2GDQMMXANIKZEPEISPNBQ7ADQ, seg_pos={0 0}, upload=551506]: Done uploading piece. err: <nil> Dec 31 20:51:25.699906604 (629h22m25.570989649s) [noise=true , address=145.79.227.15:4874 , piece_id=6RDUE67ASMARP4OCLPC5BPS6A64OOOKIAGUC6Q3F45R5UZJEJAJQ, seg_pos={0 0}, upload=551506]: Done uploading piece. err: <nil> Dec 31 20:51:25.705013989 (629h22m25.576097024s) [noise=true , address=145.79.227.45:4229 , piece_id=OQVGZDBYFJWANIC7GAFUFWJWZS3LIT6CEWWXT5SPLR7M7VHEGH2A, seg_pos={0 0}, upload=551506]: Done uploading piece. err: <nil> Dec 31 20:51:25.768154168 (629h22m25.639237203s) [noise=true , address=145.79.227.17:4970 , piece_id=3U7GCTZ6ZPJSDY7QJRRYLNOUY64LYBB7NA2AQRN3RTDTCWKL5H5A, seg_pos={0 0}, upload=551506]: Done uploading piece. err: <nil> Dec 31 20:51:25.785619824 (629h22m25.656702859s) [noise=true , address=145.79.227.51:4411 , piece_id=BUUUPVH5YP7ZDUG44OU5RUTUXBJ6N4BESJSSYUUTOILNH4AJZWGA, seg_pos={0 0}, upload=551506]: Done uploading piece. err: <nil> Dec 31 20:51:25.806597689 (629h22m25.677680725s) [noise=true , address=145.79.227.27:4689 , piece_id=XM2WI4TG2UL4TCFDRKVHRSQRE2BK2EBV6Y6JSUA6EZS3TJNNDX3Q, seg_pos={0 0}, upload=551506]: Done uploading piece. err: <nil> Dec 31 20:51:25.883364705 (629h22m25.75444774s) [noise=true , address=77.90.31.12:4590 , piece_id=X6F6LRMFT3VSDQFB2K723QPZ2724RXYFSWOWQMWZFGAUIVO5BNKQ, seg_pos={0 0}, upload=551506]: Done uploading piece. err: <nil> Dec 31 20:51:25.984313866 (629h22m25.855396911s) [noise=true , address=77.90.31.21:4880 , piece_id=Z6EYRVJCXCEIIN7GW6KFGRVEZSIWCQ4FFXT34Q5QIDYEIFW2QD5A, seg_pos={0 0}, upload=551506]: Done uploading piece. err: <nil> Dec 31 20:51:25.986455264 (629h22m25.857538299s) [noise=true , address=77.90.31.10:4519 , piece_id=FBS27JGWQEG3LNUBOIOPQCVZOKN5RY3BJBDXXQC2HG3PZZ46CZWA, seg_pos={0 0}, upload=551506]: Done uploading piece. err: <nil> Dec 31 20:51:26.002926057 (629h22m25.874009102s) [noise=true , address=45.91.116.8:4045 , piece_id=KHFRRWXG2X32QMMYBXIAKNXLFNGCCL4UKPZHRHLLYRPGJWAJFVNQ, seg_pos={0 0}, upload=551506]: Done uploading piece. err: <nil> Dec 31 20:51:26.057447813 (629h22m25.928530848s) [noise=true , address=45.91.116.5:4297 , piece_id=PSYLX6PNTZAP4UHHSXMGWNDAQPLOUKRPWKKDXZEYLZDACYI2DNTQ, seg_pos={0 0}, upload=551506]: Done uploading piece. err: <nil> Dec 31 20:51:26.175629479 (629h22m26.046712524s) [noise=true , address=45.91.116.10:4084 , piece_id=B3RPVHCWDSDXJFBLFI2HE2RD7ECKJLQT37Q3MDHCFLXXD3WNLWPA, seg_pos={0 0}, upload=551506]: Done uploading piece. err: <nil> Dec 31 20:51:26.175653634 (629h22m26.046736669s) [seg_pos={0 0}, upload=551506]: Segment reached optimal threshold of 13 pieces. Dec 31 20:51:26.175909862 (629h22m26.046992908s) [noise=true , address=45.91.116.4:4940 , piece_id=23IH4MXTUDM37G4YBB7XGFJKOYEEXC4WBSOW4WJE4CXN7HVCJL3A, seg_pos={0 0}, upload=551506]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Dec 31 20:51:26.175921114 (629h22m26.047004149s) [noise=true , address=45.91.116.4:4940 , piece_id=23IH4MXTUDM37G4YBB7XGFJKOYEEXC4WBSOW4WJE4CXN7HVCJL3A, seg_pos={0 0}, upload=551506]: Long tail cancellation due to optimal threshold Dec 31 20:51:26.175954317 (629h22m26.047037362s) [noise=true , address=145.79.227.34:4906 , piece_id=IAS4I7S3PYHZ5ODX2E6GLUBKWPTUBMZSIC5JU4I32LDZ2675U7LA, seg_pos={0 0}, upload=551506]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 31 20:51:26.175964306 (629h22m26.047047351s) [noise=true , address=145.79.227.34:4906 , piece_id=IAS4I7S3PYHZ5ODX2E6GLUBKWPTUBMZSIC5JU4I32LDZ2675U7LA, seg_pos={0 0}, upload=551506]: Long tail cancellation due to optimal threshold Dec 31 20:51:26.176121806 (629h22m26.047204851s) [noise=true , address=45.91.116.9:4261 , piece_id=JZFL73GDR56U4KVELCJTTXQNMWKGOCHWSSKWBE5ZB4CHEBPGRCXQ, seg_pos={0 0}, upload=551506]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: write tcp 188.166.225.170:56626->45.91.116.9:4261: use of closed network connection; write tcp 188.166.225.170:56626->45.91.116.9:4261: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56626->45.91.116.9:4261: use of closed network connection Dec 31 20:51:26.176128408 (629h22m26.047211443s) [noise=true , address=45.91.116.9:4261 , piece_id=JZFL73GDR56U4KVELCJTTXQNMWKGOCHWSSKWBE5ZB4CHEBPGRCXQ, seg_pos={0 0}, upload=551506]: Long tail cancellation due to optimal threshold Dec 31 20:51:26.176146222 (629h22m26.047229257s) [seg_pos={0 0}, upload=551506]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 20:51:26.176216866 (629h22m26.047299911s) [upload=551506]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 20:51:26.176221676 (629h22m26.047304721s) [upload=551506]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 20:51:26.474867771 (629h22m26.345950806s) [upload=551506]: Done uploading segments... Dec 31 20:51:26.474891285 (629h22m26.34597433s) [upload=551506]: Upload finished. err: <nil> Duration: 26.598070132s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [..X..X.....X.......XX...X..XX...X...XX....X......X....XX....] duration: 6.875839203s trace: https://jaeger.dev.storj.io/trace/3c6720cb0382e95c FAIL (2025-12-31 20:54:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: Z6EYRVJCXCEIIN7GW6KFGRVEZSIWCQ4FFXT34Q5QIDYEIFW2QD5A) context deadline exceeded; piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: PSYLX6PNTZAP4UHHSXMGWNDAQPLOUKRPWKKDXZEYLZDACYI2DNTQ) context deadline exceeded ======LOGS====== 1 20:51:23.460064987 (629h22m23.331148022s) [upload=551506]: Starting upload Dec 31 20:51:23.460124451 (629h22m23.331207486s) [upload=551506]: Done starting upload Dec 31 20:51:23.460173073 (629h22m23.331256108s) [upload=551506]: Uploading segments... Dec 31 20:51:23.460298723 (629h22m23.331381758s) [upload=551506]: Got next segment. Inline: false Dec 31 20:51:23.460327628 (629h22m23.331410663s) [upload=551506]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 20:51:23.460332086 (629h22m23.331415121s) [upload=551506]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 20:51:23.708004630 (629h22m23.579087665s) [seg_pos={0 0}, upload=551506]: Begin upload segment... Dec 31 20:51:23.708053202 (629h22m23.579136237s) [seg_pos={0 0}, upload=551506]: Done begin upload segment. Dec 31 20:51:23.708079312 (629h22m23.579162347s) [noise=true , address=145.79.227.27:4689 , piece_id=XM2WI4TG2UL4TCFDRKVHRSQRE2BK2EBV6Y6JSUA6EZS3TJNNDX3Q, seg_pos={0 0}, upload=551506]: Uploading piece... Dec 31 20:51:23.708297587 (629h22m23.579380632s) [noise=true , address=145.79.227.23:4547 , piece_id=TX7OGE7PRC424AZPTCSQVSN2OCX2GDQMMXANIKZEPEISPNBQ7ADQ, seg_pos={0 0}, upload=551506]: Uploading piece... Dec 31 20:51:23.709027757 (629h22m23.580110803s) [noise=true , address=45.91.116.9:4261 , piece_id=JZFL73GDR56U4KVELCJTTXQNMWKGOCHWSSKWBE5ZB4CHEBPGRCXQ, seg_pos={0 0}, upload=551506]: Uploading piece... Dec 31 20:51:23.709691221 (629h22m23.580774266s) [noise=true , address=45.91.116.5:4297 , piece_id=PSYLX6PNTZAP4UHHSXMGWNDAQPLOUKRPWKKDXZEYLZDACYI2DNTQ, seg_pos={0 0}, upload=551506]: Uploading piece... Dec 31 20:51:23.710208286 (629h22m23.581291321s) [noise=true , address=45.91.116.10:4084 , piece_id=B3RPVHCWDSDXJFBLFI2HE2RD7ECKJLQT37Q3MDHCFLXXD3WNLWPA, seg_pos={0 0}, upload=551506]: Uploading piece... Dec 31 20:51:23.710215549 (629h22m23.581298584s) [noise=true , address=45.91.116.4:4940 , piece_id=23IH4MXTUDM37G4YBB7XGFJKOYEEXC4WBSOW4WJE4CXN7HVCJL3A, seg_pos={0 0}, upload=551506]: Uploading piece... Dec 31 20:51:23.710515751 (629h22m23.581598786s) [noise=true , address=145.79.227.15:4874 , piece_id=6RDUE67ASMARP4OCLPC5BPS6A64OOOKIAGUC6Q3F45R5UZJEJAJQ, seg_pos={0 0}, upload=551506]: Uploading piece... Dec 31 20:51:23.710671818 (629h22m23.581754853s) [noise=true , address=145.79.227.51:4411 , piece_id=BUUUPVH5YP7ZDUG44OU5RUTUXBJ6N4BESJSSYUUTOILNH4AJZWGA, seg_pos={0 0}, upload=551506]: Uploading piece... Dec 31 20:51:23.710782529 (629h22m23.581865565s) [noise=true , address=77.90.31.10:4519 , piece_id=FBS27JGWQEG3LNUBOIOPQCVZOKN5RY3BJBDXXQC2HG3PZZ46CZWA, seg_pos={0 0}, upload=551506]: Uploading piece... Dec 31 20:51:23.711055869 (629h22m23.582138914s) [noise=true , address=145.79.227.45:4229 , piece_id=OQVGZDBYFJWANIC7GAFUFWJWZS3LIT6CEWWXT5SPLR7M7VHEGH2A, seg_pos={0 0}, upload=551506]: Uploading piece... Dec 31 20:51:23.711130170 (629h22m23.582213216s) [noise=true , address=145.79.227.17:4970 , piece_id=3U7GCTZ6ZPJSDY7QJRRYLNOUY64LYBB7NA2AQRN3RTDTCWKL5H5A, seg_pos={0 0}, upload=551506]: Uploading piece... Dec 31 20:51:23.711404543 (629h22m23.582487578s) [noise=true , address=77.90.31.12:4590 , piece_id=X6F6LRMFT3VSDQFB2K723QPZ2724RXYFSWOWQMWZFGAUIVO5BNKQ, seg_pos={0 0}, upload=551506]: Uploading piece... Dec 31 20:51:23.711627709 (629h22m23.582710744s) [noise=true , address=145.79.227.34:4906 , piece_id=IAS4I7S3PYHZ5ODX2E6GLUBKWPTUBMZSIC5JU4I32LDZ2675U7LA, seg_pos={0 0}, upload=551506]: Uploading piece... Dec 31 20:51:23.711680569 (629h22m23.582763604s) [noise=true , address=145.79.227.2:4114 , piece_id=COATTNXHVC5ZR5MWZO7ZZOXSKRIEHWLV564VWSLWPELML2F5HQOA, seg_pos={0 0}, upload=551506]: Uploading piece... Dec 31 20:51:23.711969510 (629h22m23.583052545s) [noise=true , address=45.91.116.8:4045 , piece_id=KHFRRWXG2X32QMMYBXIAKNXLFNGCCL4UKPZHRHLLYRPGJWAJFVNQ, seg_pos={0 0}, upload=551506]: Uploading piece... Dec 31 20:51:23.711981542 (629h22m23.583064587s) [noise=true , address=77.90.31.21:4880 , piece_id=Z6EYRVJCXCEIIN7GW6KFGRVEZSIWCQ4FFXT34Q5QIDYEIFW2QD5A, seg_pos={0 0}, upload=551506]: Uploading piece... Dec 31 20:51:25.404572075 (629h22m25.27565514s) [upload=551506]: Next returned nil segment Dec 31 20:51:25.404596711 (629h22m25.275679776s) [upload=551506]: Waiting for error group managing segments... Dec 31 20:51:25.686038758 (629h22m25.557121803s) [noise=true , address=145.79.227.2:4114 , piece_id=COATTNXHVC5ZR5MWZO7ZZOXSKRIEHWLV564VWSLWPELML2F5HQOA, seg_pos={0 0}, upload=551506]: Done uploading piece. err: <nil> Dec 31 20:51:25.687291203 (629h22m25.558374248s) [noise=true , address=145.79.227.23:4547 , piece_id=TX7OGE7PRC424AZPTCSQVSN2OCX2GDQMMXANIKZEPEISPNBQ7ADQ, seg_pos={0 0}, upload=551506]: Done uploading piece. err: <nil> Dec 31 20:51:25.699906604 (629h22m25.570989649s) [noise=true , address=145.79.227.15:4874 , piece_id=6RDUE67ASMARP4OCLPC5BPS6A64OOOKIAGUC6Q3F45R5UZJEJAJQ, seg_pos={0 0}, upload=551506]: Done uploading piece. err: <nil> Dec 31 20:51:25.705013989 (629h22m25.576097024s) [noise=true , address=145.79.227.45:4229 , piece_id=OQVGZDBYFJWANIC7GAFUFWJWZS3LIT6CEWWXT5SPLR7M7VHEGH2A, seg_pos={0 0}, upload=551506]: Done uploading piece. err: <nil> Dec 31 20:51:25.768154168 (629h22m25.639237203s) [noise=true , address=145.79.227.17:4970 , piece_id=3U7GCTZ6ZPJSDY7QJRRYLNOUY64LYBB7NA2AQRN3RTDTCWKL5H5A, seg_pos={0 0}, upload=551506]: Done uploading piece. err: <nil> Dec 31 20:51:25.785619824 (629h22m25.656702859s) [noise=true , address=145.79.227.51:4411 , piece_id=BUUUPVH5YP7ZDUG44OU5RUTUXBJ6N4BESJSSYUUTOILNH4AJZWGA, seg_pos={0 0}, upload=551506]: Done uploading piece. err: <nil> Dec 31 20:51:25.806597689 (629h22m25.677680725s) [noise=true , address=145.79.227.27:4689 , piece_id=XM2WI4TG2UL4TCFDRKVHRSQRE2BK2EBV6Y6JSUA6EZS3TJNNDX3Q, seg_pos={0 0}, upload=551506]: Done uploading piece. err: <nil> Dec 31 20:51:25.883364705 (629h22m25.75444774s) [noise=true , address=77.90.31.12:4590 , piece_id=X6F6LRMFT3VSDQFB2K723QPZ2724RXYFSWOWQMWZFGAUIVO5BNKQ, seg_pos={0 0}, upload=551506]: Done uploading piece. err: <nil> Dec 31 20:51:25.984313866 (629h22m25.855396911s) [noise=true , address=77.90.31.21:4880 , piece_id=Z6EYRVJCXCEIIN7GW6KFGRVEZSIWCQ4FFXT34Q5QIDYEIFW2QD5A, seg_pos={0 0}, upload=551506]: Done uploading piece. err: <nil> Dec 31 20:51:25.986455264 (629h22m25.857538299s) [noise=true , address=77.90.31.10:4519 , piece_id=FBS27JGWQEG3LNUBOIOPQCVZOKN5RY3BJBDXXQC2HG3PZZ46CZWA, seg_pos={0 0}, upload=551506]: Done uploading piece. err: <nil> Dec 31 20:51:26.002926057 (629h22m25.874009102s) [noise=true , address=45.91.116.8:4045 , piece_id=KHFRRWXG2X32QMMYBXIAKNXLFNGCCL4UKPZHRHLLYRPGJWAJFVNQ, seg_pos={0 0}, upload=551506]: Done uploading piece. err: <nil> Dec 31 20:51:26.057447813 (629h22m25.928530848s) [noise=true , address=45.91.116.5:4297 , piece_id=PSYLX6PNTZAP4UHHSXMGWNDAQPLOUKRPWKKDXZEYLZDACYI2DNTQ, seg_pos={0 0}, upload=551506]: Done uploading piece. err: <nil> Dec 31 20:51:26.175629479 (629h22m26.046712524s) [noise=true , address=45.91.116.10:4084 , piece_id=B3RPVHCWDSDXJFBLFI2HE2RD7ECKJLQT37Q3MDHCFLXXD3WNLWPA, seg_pos={0 0}, upload=551506]: Done uploading piece. err: <nil> Dec 31 20:51:26.175653634 (629h22m26.046736669s) [seg_pos={0 0}, upload=551506]: Segment reached optimal threshold of 13 pieces. Dec 31 20:51:26.175909862 (629h22m26.046992908s) [noise=true , address=45.91.116.4:4940 , piece_id=23IH4MXTUDM37G4YBB7XGFJKOYEEXC4WBSOW4WJE4CXN7HVCJL3A, seg_pos={0 0}, upload=551506]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Dec 31 20:51:26.175921114 (629h22m26.047004149s) [noise=true , address=45.91.116.4:4940 , piece_id=23IH4MXTUDM37G4YBB7XGFJKOYEEXC4WBSOW4WJE4CXN7HVCJL3A, seg_pos={0 0}, upload=551506]: Long tail cancellation due to optimal threshold Dec 31 20:51:26.175954317 (629h22m26.047037362s) [noise=true , address=145.79.227.34:4906 , piece_id=IAS4I7S3PYHZ5ODX2E6GLUBKWPTUBMZSIC5JU4I32LDZ2675U7LA, seg_pos={0 0}, upload=551506]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 31 20:51:26.175964306 (629h22m26.047047351s) [noise=true , address=145.79.227.34:4906 , piece_id=IAS4I7S3PYHZ5ODX2E6GLUBKWPTUBMZSIC5JU4I32LDZ2675U7LA, seg_pos={0 0}, upload=551506]: Long tail cancellation due to optimal threshold Dec 31 20:51:26.176121806 (629h22m26.047204851s) [noise=true , address=45.91.116.9:4261 , piece_id=JZFL73GDR56U4KVELCJTTXQNMWKGOCHWSSKWBE5ZB4CHEBPGRCXQ, seg_pos={0 0}, upload=551506]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: write tcp 188.166.225.170:56626->45.91.116.9:4261: use of closed network connection; write tcp 188.166.225.170:56626->45.91.116.9:4261: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56626->45.91.116.9:4261: use of closed network connection Dec 31 20:51:26.176128408 (629h22m26.047211443s) [noise=true , address=45.91.116.9:4261 , piece_id=JZFL73GDR56U4KVELCJTTXQNMWKGOCHWSSKWBE5ZB4CHEBPGRCXQ, seg_pos={0 0}, upload=551506]: Long tail cancellation due to optimal threshold Dec 31 20:51:26.176146222 (629h22m26.047229257s) [seg_pos={0 0}, upload=551506]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 20:51:26.176216866 (629h22m26.047299911s) [upload=551506]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 20:51:26.176221676 (629h22m26.047304721s) [upload=551506]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 20:51:26.474867771 (629h22m26.345950806s) [upload=551506]: Done uploading segments... Dec 31 20:51:26.474891285 (629h22m26.34597433s) [upload=551506]: Upload finished. err: <nil> Duration: 6.87586396s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)