Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1m0.002882386s trace: https://jaeger.dev.storj.io/trace/499b96822b8dda80 FAIL (2025-12-28 20:33:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12S7WBBQ1Nq8jbeGrZiRKTk23CdJC31hPW2vU1aaM5SsHeVyeRH, Piece ID: V4P3KCUDLI543TSROBYUMVTQKKSQDDVSKSJSXKEZTQAHTXUJ2G4Q) context deadline exceeded; piecestore: (Node ID: 1kjCEPtjvcJ3D7BBKRSRirPybBwkRZ1dJwyAQCadzCvaLYvWhz, Piece ID: 4OULHJ6Z3QJ3BOPDY74TJSKVJG23TOEUCJCUEGY24I5C3MUTAVLQ) context deadline exceeded; piecestore: (Node ID: 1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c, Piece ID: LIYXCPAZOUPKXLKRH3EOWREXUY2LAJXSRZFQILM2QIU7HTW7SY2Q) context deadline exceeded ======LOGS====== ue , address=145.79.227.23:4580 , piece_id=OHYI2SFM6TKJE4LHVNGO2PVGZDNFIQDAIQGKG4FOEHTTASBWA5ZQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.496428825 (557h3m1.36751187s) [noise=true , address=77.90.31.19:4867 , piece_id=LDAWD2ZFG3BZATDRY7MMNWYIX6TOHQ2OIUPCAWAYN266GNUUKZ5A, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.496861509 (557h3m1.367944544s) [noise=true , address=145.79.227.19:4105 , piece_id=RY35FSM6NTWPNF3JE43FZRSNXXYZPUN2B6LRNNUOG6QQR6XSU36Q, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.496899933 (557h3m1.367982978s) [noise=true , address=77.90.31.19:4858 , piece_id=LIYXCPAZOUPKXLKRH3EOWREXUY2LAJXSRZFQILM2QIU7HTW7SY2Q, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497229089 (557h3m1.368312134s) [noise=true , address=77.90.31.17:4784 , piece_id=JV2ROD6MN65B6LZKOL3S7ATJWXAY6FBQSZWITA6IHAXELY7R42KA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497326194 (557h3m1.368409239s) [noise=true , address=145.79.227.33:4904 , piece_id=QCMN4OSBWIWS4JRLIGQIGDSHZFUF2JWEPVXQ4FQ4G3LG5FBCBAWQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497664128 (557h3m1.368747173s) [noise=true , address=145.79.227.34:4918 , piece_id=SPDT2KI2774UJMQIUSVTHXCDF7PD2FBG64BXUYW67UMLPE6H7SBQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497731405 (557h3m1.36881445s) [noise=true , address=77.90.31.12:4602 , piece_id=2YPYKKTQ6ID6VTI4AUZQEG2SY7JYPRSAPU4JLF7DWSOSBBO74LWQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.498205939 (557h3m1.369288984s) [noise=true , address=77.90.31.16:4745 , piece_id=CZBFU33BHMP2BEZIDJKW3J526537HR6HP7OCT2KMAQVY7LGZLPZQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.498424295 (557h3m1.36950734s) [noise=true , address=77.90.31.21:4895 , piece_id=V4P3KCUDLI543TSROBYUMVTQKKSQDDVSKSJSXKEZTQAHTXUJ2G4Q, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.498685422 (557h3m1.369768457s) [noise=true , address=145.79.227.35:4496 , piece_id=JH4XQBT7XYEJXWBGPPF6SJWCPLV77GBOLGOA53C3WUQMWHZXVZQA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499066348 (557h3m1.370149393s) [noise=true , address=77.90.31.11:4563 , piece_id=DIFJPHSKHROTLKCHP6HQBWT7XGE77HVRZTH2354MV2UEWZLHVGCQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499185725 (557h3m1.37026877s) [noise=true , address=145.79.227.30:4768 , piece_id=KK4LEV6RG3IGLYXID5MJF3KMNSJ7WS5JS45PNOOJ26LXLYAUCWOA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499468122 (557h3m1.370551157s) [noise=true , address=77.90.31.14:4688 , piece_id=4OULHJ6Z3QJ3BOPDY74TJSKVJG23TOEUCJCUEGY24I5C3MUTAVLQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499901047 (557h3m1.370984102s) [noise=true , address=45.91.116.5:4330 , piece_id=EVHLSVVR5JGY43TLSKVEQLQ6KDBEZENVAFL6YT7ZQ77Y7X2TXIKA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.500546175 (557h3m1.37162921s) [noise=true , address=145.79.227.43:4171 , piece_id=YF5DRDPGRRVYFL5SWMH2PKTSV3C72BIWKUDSAIZTXOIVOTYKRMNA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:02.979572381 (557h3m2.850655436s) [upload=488362]: Next returned nil segment Dec 28 20:32:02.979609411 (557h3m2.850692456s) [upload=488362]: Waiting for error group managing segments... Dec 28 20:32:03.416896733 (557h3m3.287979778s) [noise=true , address=145.79.227.23:4580 , piece_id=OHYI2SFM6TKJE4LHVNGO2PVGZDNFIQDAIQGKG4FOEHTTASBWA5ZQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.552263655 (557h3m3.4233467s) [noise=true , address=77.90.31.17:4784 , piece_id=JV2ROD6MN65B6LZKOL3S7ATJWXAY6FBQSZWITA6IHAXELY7R42KA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.573564856 (557h3m3.444647902s) [noise=true , address=77.90.31.12:4602 , piece_id=2YPYKKTQ6ID6VTI4AUZQEG2SY7JYPRSAPU4JLF7DWSOSBBO74LWQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.598560923 (557h3m3.469643968s) [noise=true , address=77.90.31.19:4867 , piece_id=LDAWD2ZFG3BZATDRY7MMNWYIX6TOHQ2OIUPCAWAYN266GNUUKZ5A, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.626925925 (557h3m3.49800897s) [noise=true , address=77.90.31.16:4745 , piece_id=CZBFU33BHMP2BEZIDJKW3J526537HR6HP7OCT2KMAQVY7LGZLPZQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.709061947 (557h3m3.580144992s) [noise=true , address=77.90.31.14:4688 , piece_id=4OULHJ6Z3QJ3BOPDY74TJSKVJG23TOEUCJCUEGY24I5C3MUTAVLQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.788879373 (557h3m3.659962418s) [noise=true , address=77.90.31.21:4895 , piece_id=V4P3KCUDLI543TSROBYUMVTQKKSQDDVSKSJSXKEZTQAHTXUJ2G4Q, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.217616523 (557h3m4.088699569s) [noise=true , address=77.90.31.19:4858 , piece_id=LIYXCPAZOUPKXLKRH3EOWREXUY2LAJXSRZFQILM2QIU7HTW7SY2Q, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.220337625 (557h3m4.09142067s) [noise=true , address=77.90.31.11:4563 , piece_id=DIFJPHSKHROTLKCHP6HQBWT7XGE77HVRZTH2354MV2UEWZLHVGCQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.303958744 (557h3m4.175041789s) [noise=true , address=145.79.227.44:4213 , piece_id=UEAQDBBCP7DIG6ZD3EEM2CKAL4W5QKW23LWFUAMSVUW5ZS2PJDRQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.312794398 (557h3m4.183877444s) [noise=true , address=145.79.227.34:4918 , piece_id=SPDT2KI2774UJMQIUSVTHXCDF7PD2FBG64BXUYW67UMLPE6H7SBQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.349308798 (557h3m4.220391833s) [noise=true , address=145.79.227.40:4058 , piece_id=EQ4LA7T2BCGWY76K3NJI4X77OQJFJED6H4VNGXDQMILU6IVQWOMQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.371518379 (557h3m4.242601424s) [noise=true , address=145.79.227.43:4171 , piece_id=YF5DRDPGRRVYFL5SWMH2PKTSV3C72BIWKUDSAIZTXOIVOTYKRMNA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.376232316 (557h3m4.247315361s) [noise=true , address=145.79.227.30:4768 , piece_id=KK4LEV6RG3IGLYXID5MJF3KMNSJ7WS5JS45PNOOJ26LXLYAUCWOA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.376257844 (557h3m4.247340879s) [seg_pos={0 0}, upload=488362]: Segment reached optimal threshold of 14 pieces. Dec 28 20:32:04.376575799 (557h3m4.247658834s) [noise=true , address=145.79.227.33:4904 , piece_id=QCMN4OSBWIWS4JRLIGQIGDSHZFUF2JWEPVXQ4FQ4G3LG5FBCBAWQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1dC7mC2Cbxmys7ukEkELyF9Us7F1qtptSP6yDJh1EfjVJ6uqsC): protocol: expected piece hash; context canceled; EOF Dec 28 20:32:04.376585127 (557h3m4.247668163s) [noise=true , address=145.79.227.33:4904 , piece_id=QCMN4OSBWIWS4JRLIGQIGDSHZFUF2JWEPVXQ4FQ4G3LG5FBCBAWQ, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.376764359 (557h3m4.247847404s) [noise=true , address=145.79.227.19:4105 , piece_id=RY35FSM6NTWPNF3JE43FZRSNXXYZPUN2B6LRNNUOG6QQR6XSU36Q, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GhDrKMcDEg2RxamqKtiHphpgNdvtZYdHXrjzGsNJFjBFYeEen): protocol: write tcp 188.166.225.170:38838->145.79.227.19:4105: use of closed network connection; write tcp 188.166.225.170:38838->145.79.227.19:4105: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38838->145.79.227.19:4105: use of closed network connection Dec 28 20:32:04.376772865 (557h3m4.2478559s) [noise=true , address=145.79.227.19:4105 , piece_id=RY35FSM6NTWPNF3JE43FZRSNXXYZPUN2B6LRNNUOG6QQR6XSU36Q, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.376926658 (557h3m4.248009703s) [noise=true , address=145.79.227.35:4496 , piece_id=JH4XQBT7XYEJXWBGPPF6SJWCPLV77GBOLGOA53C3WUQMWHZXVZQA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V8XhSWkyGUVYFR8MykifFAP1YAfGCwzqucDcMMfzw3xsVW77Z): protocol: expected piece hash; context canceled; EOF Dec 28 20:32:04.377165332 (557h3m4.248248377s) [noise=true , address=145.79.227.35:4496 , piece_id=JH4XQBT7XYEJXWBGPPF6SJWCPLV77GBOLGOA53C3WUQMWHZXVZQA, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.377212682 (557h3m4.248295848s) [noise=true , address=45.91.116.5:4330 , piece_id=EVHLSVVR5JGY43TLSKVEQLQ6KDBEZENVAFL6YT7ZQ77Y7X2TXIKA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12MK9aVDRwfWPfckFR6RfPUL1Nrm9bwfsgV7TXCKGbtadHebWvW): protocol: read tcp 188.166.225.170:55416->45.91.116.5:4330: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 20:32:04.377289859 (557h3m4.248372924s) [noise=true , address=45.91.116.5:4330 , piece_id=EVHLSVVR5JGY43TLSKVEQLQ6KDBEZENVAFL6YT7ZQ77Y7X2TXIKA, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.377387736 (557h3m4.248470781s) [seg_pos={0 0}, upload=488362]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 20:32:04.377599729 (557h3m4.248682774s) [upload=488362]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 20:32:04.377613535 (557h3m4.24869662s) [upload=488362]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 20:32:04.675090855 (557h3m4.5461739s) [upload=488362]: Done uploading segments... Dec 28 20:32:04.675121233 (557h3m4.546204268s) [upload=488362]: Upload finished. err: <nil> Duration: 1m0.002894038s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......X..........................X...........X.............] duration: 10.279907573s trace: https://jaeger.dev.storj.io/trace/3425d363be458a81 FAIL (2025-12-28 20:32:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: XB2K7RONOVEBPA2XGY36WIE57MWUUBKVAHEGXHAZWNDRL22WHH5Q) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: HHINRA5KN56ZF2KW7IEG4BUGXM6UKMNUM3TM4Y3HFADVWASFEHFA) context deadline exceeded ======LOGS====== Dec 28 20:22:57.518307420 (556h53m57.389390455s) [upload=488235]: Starting upload Dec 28 20:22:57.518369106 (556h53m57.389452182s) [upload=488235]: Done starting upload Dec 28 20:22:57.518427107 (556h53m57.389510152s) [upload=488235]: Uploading segments... Dec 28 20:22:57.518663427 (556h53m57.389746473s) [upload=488235]: Got next segment. Inline: false Dec 28 20:22:57.518709805 (556h53m57.38979284s) [upload=488235]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 20:22:57.518716318 (556h53m57.389799353s) [upload=488235]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 20:22:57.767659750 (556h53m57.638742815s) [seg_pos={0 0}, upload=488235]: Begin upload segment... Dec 28 20:22:57.767735904 (556h53m57.638818949s) [seg_pos={0 0}, upload=488235]: Done begin upload segment. Dec 28 20:22:57.767792252 (556h53m57.638875287s) [noise=true , address=77.90.31.21:4880 , piece_id=V4PF2AQHKOK3GCF6B4UDNRJW7WYQ5QVOC66HUJMOPOOLPZAQIM4A, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.767877795 (556h53m57.63896085s) [noise=true , address=145.79.227.51:4409 , piece_id=C342XC375VQVI23V5JESCTOES4AF66QWHIETEOBVV3KGRLIP2X5Q, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.768512193 (556h53m57.639595238s) [noise=true , address=145.79.227.18:4007 , piece_id=NB6ADK5TXELRFCYMKDF6C3QZRZQUL5XUPVQ7AVVYP32UY2GZA6SA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.768539074 (556h53m57.639622119s) [noise=true , address=45.91.116.5:4297 , piece_id=FNX3RGGDGJZL5DOZRL2ZF2K2JHRAHWAWHIN6NQ4ONUKPVC6J4ECA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.769129117 (556h53m57.640212152s) [noise=true , address=145.79.227.41:4086 , piece_id=YRHLQZ64IEO7C26HZTCDY2T2PRYUFP6CSN2EOBJS7XAE4I7MXLHQ, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.769521375 (556h53m57.64060441s) [noise=true , address=145.79.227.12:4765 , piece_id=IESWSXDQFYKKHDOEHSYQSVMTBYLAJWNQ5TPWGNQPMLHO23CXYXTQ, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.770627170 (556h53m57.641710205s) [noise=true , address=77.90.31.13:4627 , piece_id=XB2K7RONOVEBPA2XGY36WIE57MWUUBKVAHEGXHAZWNDRL22WHH5Q, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.771369815 (556h53m57.64245285s) [noise=true , address=145.79.227.50:4375 , piece_id=DMDQHEH7XJT334MFCFAPNCGLSJ4UWE3U74MTDFROMTM4BDAI4IHA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.772492252 (556h53m57.643575297s) [noise=true , address=145.79.227.36:4519 , piece_id=YIPTLMBP7UV4GLPNA74M4343U4JEE7433ANXUHKYIJLZ36T25TXA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.773128333 (556h53m57.644211368s) [noise=true , address=77.90.31.17:4770 , piece_id=MH6LXF5JVTVI3U3N7W73U4ULOARVA6G2CRNUQLQ3V2UE6HUDGCUA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.775274971 (556h53m57.646358006s) [noise=true , address=145.79.227.40:4051 , piece_id=PZ3547GPNB3TRB24RBQVWLHLOC5JQVFUMG4ZZJDA3BBESZN3XFYA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.777926579 (556h53m57.649009615s) [noise=true , address=145.79.227.42:4123 , piece_id=TN4ZO3UUKIFRAP44SANXU6HKESFMIE45S5XTWAIKHJW5SA6C2KFA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.780387376 (556h53m57.651470421s) [noise=true , address=45.91.116.13:4155 , piece_id=HHINRA5KN56ZF2KW7IEG4BUGXM6UKMNUM3TM4Y3HFADVWASFEHFA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.781667764 (556h53m57.652750809s) [noise=true , address=77.90.31.15:4697 , piece_id=25QRUIDDPIZ7CSGKDAXLMZIWJHE6HHITH6FU4H2GA2GDBI4PR5KA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.783365686 (556h53m57.654448731s) [noise=true , address=77.90.31.11:4556 , piece_id=QRRGWDO7EHFVVQ6X4BTVY7HQRRDF5UCHKCFRLO2FOOYLTEDNJR2A, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.784872727 (556h53m57.655955762s) [noise=true , address=145.79.227.21:4473 , piece_id=OHS3FHPTXD3EKQNBJLLGKENQIJPKPGEFTVIRVQYXO3PZQEVQILGQ, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:59.482505035 (556h53m59.35358808s) [upload=488235]: Next returned nil segment Dec 28 20:22:59.482533810 (556h53m59.353616845s) [upload=488235]: Waiting for error group managing segments... Dec 28 20:22:59.769352743 (556h53m59.640435799s) [noise=true , address=145.79.227.18:4007 , piece_id=NB6ADK5TXELRFCYMKDF6C3QZRZQUL5XUPVQ7AVVYP32UY2GZA6SA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.782950737 (556h53m59.654033782s) [noise=true , address=145.79.227.51:4409 , piece_id=C342XC375VQVI23V5JESCTOES4AF66QWHIETEOBVV3KGRLIP2X5Q, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.789678698 (556h53m59.660761733s) [noise=true , address=145.79.227.12:4765 , piece_id=IESWSXDQFYKKHDOEHSYQSVMTBYLAJWNQ5TPWGNQPMLHO23CXYXTQ, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.802190252 (556h53m59.673273287s) [noise=true , address=145.79.227.42:4123 , piece_id=TN4ZO3UUKIFRAP44SANXU6HKESFMIE45S5XTWAIKHJW5SA6C2KFA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.804531330 (556h53m59.675614375s) [noise=true , address=145.79.227.50:4375 , piece_id=DMDQHEH7XJT334MFCFAPNCGLSJ4UWE3U74MTDFROMTM4BDAI4IHA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.810012928 (556h53m59.681095963s) [noise=true , address=145.79.227.21:4473 , piece_id=OHS3FHPTXD3EKQNBJLLGKENQIJPKPGEFTVIRVQYXO3PZQEVQILGQ, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.812050608 (556h53m59.683133643s) [noise=true , address=145.79.227.36:4519 , piece_id=YIPTLMBP7UV4GLPNA74M4343U4JEE7433ANXUHKYIJLZ36T25TXA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.812993363 (556h53m59.684076398s) [noise=true , address=145.79.227.41:4086 , piece_id=YRHLQZ64IEO7C26HZTCDY2T2PRYUFP6CSN2EOBJS7XAE4I7MXLHQ, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.946324830 (556h53m59.817407865s) [noise=true , address=77.90.31.13:4627 , piece_id=XB2K7RONOVEBPA2XGY36WIE57MWUUBKVAHEGXHAZWNDRL22WHH5Q, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.965566208 (556h53m59.836649243s) [noise=true , address=77.90.31.17:4770 , piece_id=MH6LXF5JVTVI3U3N7W73U4ULOARVA6G2CRNUQLQ3V2UE6HUDGCUA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:23:00.024382405 (556h53m59.89546544s) [noise=true , address=77.90.31.21:4880 , piece_id=V4PF2AQHKOK3GCF6B4UDNRJW7WYQ5QVOC66HUJMOPOOLPZAQIM4A, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:23:00.056351529 (556h53m59.927434574s) [noise=true , address=45.91.116.5:4297 , piece_id=FNX3RGGDGJZL5DOZRL2ZF2K2JHRAHWAWHIN6NQ4ONUKPVC6J4ECA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:23:00.067006559 (556h53m59.938089594s) [noise=true , address=45.91.116.13:4155 , piece_id=HHINRA5KN56ZF2KW7IEG4BUGXM6UKMNUM3TM4Y3HFADVWASFEHFA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:23:00.067025144 (556h53m59.938108179s) [seg_pos={0 0}, upload=488235]: Segment reached optimal threshold of 13 pieces. Dec 28 20:23:00.067214014 (556h53m59.938297059s) [noise=true , address=77.90.31.11:4556 , piece_id=QRRGWDO7EHFVVQ6X4BTVY7HQRRDF5UCHKCFRLO2FOOYLTEDNJR2A, seg_pos={0 0}, upload=488235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: write tcp 188.166.225.170:60580->77.90.31.11:4556: use of closed network connection; write tcp 188.166.225.170:60580->77.90.31.11:4556: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60580->77.90.31.11:4556: use of closed network connection Dec 28 20:23:00.067225536 (556h53m59.938308571s) [noise=true , address=77.90.31.11:4556 , piece_id=QRRGWDO7EHFVVQ6X4BTVY7HQRRDF5UCHKCFRLO2FOOYLTEDNJR2A, seg_pos={0 0}, upload=488235]: Long tail cancellation due to optimal threshold Dec 28 20:23:00.067306319 (556h53m59.938389364s) [noise=true , address=77.90.31.15:4697 , piece_id=25QRUIDDPIZ7CSGKDAXLMZIWJHE6HHITH6FU4H2GA2GDBI4PR5KA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Dec 28 20:23:00.067314354 (556h53m59.938397389s) [noise=true , address=77.90.31.15:4697 , piece_id=25QRUIDDPIZ7CSGKDAXLMZIWJHE6HHITH6FU4H2GA2GDBI4PR5KA, seg_pos={0 0}, upload=488235]: Long tail cancellation due to optimal threshold Dec 28 20:23:00.067582064 (556h53m59.938665119s) [noise=true , address=145.79.227.40:4051 , piece_id=PZ3547GPNB3TRB24RBQVWLHLOC5JQVFUMG4ZZJDA3BBESZN3XFYA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 28 20:23:00.067589408 (556h53m59.938672453s) [noise=true , address=145.79.227.40:4051 , piece_id=PZ3547GPNB3TRB24RBQVWLHLOC5JQVFUMG4ZZJDA3BBESZN3XFYA, seg_pos={0 0}, upload=488235]: Long tail cancellation due to optimal threshold Dec 28 20:23:00.067605620 (556h53m59.938688655s) [seg_pos={0 0}, upload=488235]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 20:23:00.067681614 (556h53m59.938764659s) [upload=488235]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 20:23:00.067686654 (556h53m59.938769699s) [upload=488235]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 20:23:00.354014782 (556h54m0.225097827s) [upload=488235]: Done uploading segments... Dec 28 20:23:00.354034600 (556h54m0.225117635s) [upload=488235]: Upload finished. err: <nil> Duration: 10.279929916s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [........X..........................X...........X............] duration: 6.302933279s trace: https://jaeger.dev.storj.io/trace/3425d363be458a81 FAIL (2025-12-28 20:32:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: XB2K7RONOVEBPA2XGY36WIE57MWUUBKVAHEGXHAZWNDRL22WHH5Q) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: HHINRA5KN56ZF2KW7IEG4BUGXM6UKMNUM3TM4Y3HFADVWASFEHFA) context deadline exceeded ======LOGS====== Dec 28 20:22:57.518307420 (556h53m57.389390455s) [upload=488235]: Starting upload Dec 28 20:22:57.518369106 (556h53m57.389452182s) [upload=488235]: Done starting upload Dec 28 20:22:57.518427107 (556h53m57.389510152s) [upload=488235]: Uploading segments... Dec 28 20:22:57.518663427 (556h53m57.389746473s) [upload=488235]: Got next segment. Inline: false Dec 28 20:22:57.518709805 (556h53m57.38979284s) [upload=488235]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 20:22:57.518716318 (556h53m57.389799353s) [upload=488235]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 20:22:57.767659750 (556h53m57.638742815s) [seg_pos={0 0}, upload=488235]: Begin upload segment... Dec 28 20:22:57.767735904 (556h53m57.638818949s) [seg_pos={0 0}, upload=488235]: Done begin upload segment. Dec 28 20:22:57.767792252 (556h53m57.638875287s) [noise=true , address=77.90.31.21:4880 , piece_id=V4PF2AQHKOK3GCF6B4UDNRJW7WYQ5QVOC66HUJMOPOOLPZAQIM4A, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.767877795 (556h53m57.63896085s) [noise=true , address=145.79.227.51:4409 , piece_id=C342XC375VQVI23V5JESCTOES4AF66QWHIETEOBVV3KGRLIP2X5Q, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.768512193 (556h53m57.639595238s) [noise=true , address=145.79.227.18:4007 , piece_id=NB6ADK5TXELRFCYMKDF6C3QZRZQUL5XUPVQ7AVVYP32UY2GZA6SA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.768539074 (556h53m57.639622119s) [noise=true , address=45.91.116.5:4297 , piece_id=FNX3RGGDGJZL5DOZRL2ZF2K2JHRAHWAWHIN6NQ4ONUKPVC6J4ECA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.769129117 (556h53m57.640212152s) [noise=true , address=145.79.227.41:4086 , piece_id=YRHLQZ64IEO7C26HZTCDY2T2PRYUFP6CSN2EOBJS7XAE4I7MXLHQ, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.769521375 (556h53m57.64060441s) [noise=true , address=145.79.227.12:4765 , piece_id=IESWSXDQFYKKHDOEHSYQSVMTBYLAJWNQ5TPWGNQPMLHO23CXYXTQ, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.770627170 (556h53m57.641710205s) [noise=true , address=77.90.31.13:4627 , piece_id=XB2K7RONOVEBPA2XGY36WIE57MWUUBKVAHEGXHAZWNDRL22WHH5Q, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.771369815 (556h53m57.64245285s) [noise=true , address=145.79.227.50:4375 , piece_id=DMDQHEH7XJT334MFCFAPNCGLSJ4UWE3U74MTDFROMTM4BDAI4IHA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.772492252 (556h53m57.643575297s) [noise=true , address=145.79.227.36:4519 , piece_id=YIPTLMBP7UV4GLPNA74M4343U4JEE7433ANXUHKYIJLZ36T25TXA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.773128333 (556h53m57.644211368s) [noise=true , address=77.90.31.17:4770 , piece_id=MH6LXF5JVTVI3U3N7W73U4ULOARVA6G2CRNUQLQ3V2UE6HUDGCUA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.775274971 (556h53m57.646358006s) [noise=true , address=145.79.227.40:4051 , piece_id=PZ3547GPNB3TRB24RBQVWLHLOC5JQVFUMG4ZZJDA3BBESZN3XFYA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.777926579 (556h53m57.649009615s) [noise=true , address=145.79.227.42:4123 , piece_id=TN4ZO3UUKIFRAP44SANXU6HKESFMIE45S5XTWAIKHJW5SA6C2KFA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.780387376 (556h53m57.651470421s) [noise=true , address=45.91.116.13:4155 , piece_id=HHINRA5KN56ZF2KW7IEG4BUGXM6UKMNUM3TM4Y3HFADVWASFEHFA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.781667764 (556h53m57.652750809s) [noise=true , address=77.90.31.15:4697 , piece_id=25QRUIDDPIZ7CSGKDAXLMZIWJHE6HHITH6FU4H2GA2GDBI4PR5KA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.783365686 (556h53m57.654448731s) [noise=true , address=77.90.31.11:4556 , piece_id=QRRGWDO7EHFVVQ6X4BTVY7HQRRDF5UCHKCFRLO2FOOYLTEDNJR2A, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.784872727 (556h53m57.655955762s) [noise=true , address=145.79.227.21:4473 , piece_id=OHS3FHPTXD3EKQNBJLLGKENQIJPKPGEFTVIRVQYXO3PZQEVQILGQ, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:59.482505035 (556h53m59.35358808s) [upload=488235]: Next returned nil segment Dec 28 20:22:59.482533810 (556h53m59.353616845s) [upload=488235]: Waiting for error group managing segments... Dec 28 20:22:59.769352743 (556h53m59.640435799s) [noise=true , address=145.79.227.18:4007 , piece_id=NB6ADK5TXELRFCYMKDF6C3QZRZQUL5XUPVQ7AVVYP32UY2GZA6SA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.782950737 (556h53m59.654033782s) [noise=true , address=145.79.227.51:4409 , piece_id=C342XC375VQVI23V5JESCTOES4AF66QWHIETEOBVV3KGRLIP2X5Q, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.789678698 (556h53m59.660761733s) [noise=true , address=145.79.227.12:4765 , piece_id=IESWSXDQFYKKHDOEHSYQSVMTBYLAJWNQ5TPWGNQPMLHO23CXYXTQ, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.802190252 (556h53m59.673273287s) [noise=true , address=145.79.227.42:4123 , piece_id=TN4ZO3UUKIFRAP44SANXU6HKESFMIE45S5XTWAIKHJW5SA6C2KFA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.804531330 (556h53m59.675614375s) [noise=true , address=145.79.227.50:4375 , piece_id=DMDQHEH7XJT334MFCFAPNCGLSJ4UWE3U74MTDFROMTM4BDAI4IHA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.810012928 (556h53m59.681095963s) [noise=true , address=145.79.227.21:4473 , piece_id=OHS3FHPTXD3EKQNBJLLGKENQIJPKPGEFTVIRVQYXO3PZQEVQILGQ, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.812050608 (556h53m59.683133643s) [noise=true , address=145.79.227.36:4519 , piece_id=YIPTLMBP7UV4GLPNA74M4343U4JEE7433ANXUHKYIJLZ36T25TXA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.812993363 (556h53m59.684076398s) [noise=true , address=145.79.227.41:4086 , piece_id=YRHLQZ64IEO7C26HZTCDY2T2PRYUFP6CSN2EOBJS7XAE4I7MXLHQ, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.946324830 (556h53m59.817407865s) [noise=true , address=77.90.31.13:4627 , piece_id=XB2K7RONOVEBPA2XGY36WIE57MWUUBKVAHEGXHAZWNDRL22WHH5Q, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.965566208 (556h53m59.836649243s) [noise=true , address=77.90.31.17:4770 , piece_id=MH6LXF5JVTVI3U3N7W73U4ULOARVA6G2CRNUQLQ3V2UE6HUDGCUA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:23:00.024382405 (556h53m59.89546544s) [noise=true , address=77.90.31.21:4880 , piece_id=V4PF2AQHKOK3GCF6B4UDNRJW7WYQ5QVOC66HUJMOPOOLPZAQIM4A, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:23:00.056351529 (556h53m59.927434574s) [noise=true , address=45.91.116.5:4297 , piece_id=FNX3RGGDGJZL5DOZRL2ZF2K2JHRAHWAWHIN6NQ4ONUKPVC6J4ECA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:23:00.067006559 (556h53m59.938089594s) [noise=true , address=45.91.116.13:4155 , piece_id=HHINRA5KN56ZF2KW7IEG4BUGXM6UKMNUM3TM4Y3HFADVWASFEHFA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:23:00.067025144 (556h53m59.938108179s) [seg_pos={0 0}, upload=488235]: Segment reached optimal threshold of 13 pieces. Dec 28 20:23:00.067214014 (556h53m59.938297059s) [noise=true , address=77.90.31.11:4556 , piece_id=QRRGWDO7EHFVVQ6X4BTVY7HQRRDF5UCHKCFRLO2FOOYLTEDNJR2A, seg_pos={0 0}, upload=488235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: write tcp 188.166.225.170:60580->77.90.31.11:4556: use of closed network connection; write tcp 188.166.225.170:60580->77.90.31.11:4556: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60580->77.90.31.11:4556: use of closed network connection Dec 28 20:23:00.067225536 (556h53m59.938308571s) [noise=true , address=77.90.31.11:4556 , piece_id=QRRGWDO7EHFVVQ6X4BTVY7HQRRDF5UCHKCFRLO2FOOYLTEDNJR2A, seg_pos={0 0}, upload=488235]: Long tail cancellation due to optimal threshold Dec 28 20:23:00.067306319 (556h53m59.938389364s) [noise=true , address=77.90.31.15:4697 , piece_id=25QRUIDDPIZ7CSGKDAXLMZIWJHE6HHITH6FU4H2GA2GDBI4PR5KA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Dec 28 20:23:00.067314354 (556h53m59.938397389s) [noise=true , address=77.90.31.15:4697 , piece_id=25QRUIDDPIZ7CSGKDAXLMZIWJHE6HHITH6FU4H2GA2GDBI4PR5KA, seg_pos={0 0}, upload=488235]: Long tail cancellation due to optimal threshold Dec 28 20:23:00.067582064 (556h53m59.938665119s) [noise=true , address=145.79.227.40:4051 , piece_id=PZ3547GPNB3TRB24RBQVWLHLOC5JQVFUMG4ZZJDA3BBESZN3XFYA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 28 20:23:00.067589408 (556h53m59.938672453s) [noise=true , address=145.79.227.40:4051 , piece_id=PZ3547GPNB3TRB24RBQVWLHLOC5JQVFUMG4ZZJDA3BBESZN3XFYA, seg_pos={0 0}, upload=488235]: Long tail cancellation due to optimal threshold Dec 28 20:23:00.067605620 (556h53m59.938688655s) [seg_pos={0 0}, upload=488235]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 20:23:00.067681614 (556h53m59.938764659s) [upload=488235]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 20:23:00.067686654 (556h53m59.938769699s) [upload=488235]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 20:23:00.354014782 (556h54m0.225097827s) [upload=488235]: Done uploading segments... Dec 28 20:23:00.354034600 (556h54m0.225117635s) [upload=488235]: Upload finished. err: <nil> Duration: 6.303043088s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........X..........................X...........X...........] duration: 7.456667189s trace: https://jaeger.dev.storj.io/trace/3425d363be458a81 FAIL (2025-12-28 20:31:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: XB2K7RONOVEBPA2XGY36WIE57MWUUBKVAHEGXHAZWNDRL22WHH5Q) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: HHINRA5KN56ZF2KW7IEG4BUGXM6UKMNUM3TM4Y3HFADVWASFEHFA) context deadline exceeded ======LOGS====== Dec 28 20:22:57.518307420 (556h53m57.389390455s) [upload=488235]: Starting upload Dec 28 20:22:57.518369106 (556h53m57.389452182s) [upload=488235]: Done starting upload Dec 28 20:22:57.518427107 (556h53m57.389510152s) [upload=488235]: Uploading segments... Dec 28 20:22:57.518663427 (556h53m57.389746473s) [upload=488235]: Got next segment. Inline: false Dec 28 20:22:57.518709805 (556h53m57.38979284s) [upload=488235]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 20:22:57.518716318 (556h53m57.389799353s) [upload=488235]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 20:22:57.767659750 (556h53m57.638742815s) [seg_pos={0 0}, upload=488235]: Begin upload segment... Dec 28 20:22:57.767735904 (556h53m57.638818949s) [seg_pos={0 0}, upload=488235]: Done begin upload segment. Dec 28 20:22:57.767792252 (556h53m57.638875287s) [noise=true , address=77.90.31.21:4880 , piece_id=V4PF2AQHKOK3GCF6B4UDNRJW7WYQ5QVOC66HUJMOPOOLPZAQIM4A, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.767877795 (556h53m57.63896085s) [noise=true , address=145.79.227.51:4409 , piece_id=C342XC375VQVI23V5JESCTOES4AF66QWHIETEOBVV3KGRLIP2X5Q, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.768512193 (556h53m57.639595238s) [noise=true , address=145.79.227.18:4007 , piece_id=NB6ADK5TXELRFCYMKDF6C3QZRZQUL5XUPVQ7AVVYP32UY2GZA6SA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.768539074 (556h53m57.639622119s) [noise=true , address=45.91.116.5:4297 , piece_id=FNX3RGGDGJZL5DOZRL2ZF2K2JHRAHWAWHIN6NQ4ONUKPVC6J4ECA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.769129117 (556h53m57.640212152s) [noise=true , address=145.79.227.41:4086 , piece_id=YRHLQZ64IEO7C26HZTCDY2T2PRYUFP6CSN2EOBJS7XAE4I7MXLHQ, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.769521375 (556h53m57.64060441s) [noise=true , address=145.79.227.12:4765 , piece_id=IESWSXDQFYKKHDOEHSYQSVMTBYLAJWNQ5TPWGNQPMLHO23CXYXTQ, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.770627170 (556h53m57.641710205s) [noise=true , address=77.90.31.13:4627 , piece_id=XB2K7RONOVEBPA2XGY36WIE57MWUUBKVAHEGXHAZWNDRL22WHH5Q, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.771369815 (556h53m57.64245285s) [noise=true , address=145.79.227.50:4375 , piece_id=DMDQHEH7XJT334MFCFAPNCGLSJ4UWE3U74MTDFROMTM4BDAI4IHA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.772492252 (556h53m57.643575297s) [noise=true , address=145.79.227.36:4519 , piece_id=YIPTLMBP7UV4GLPNA74M4343U4JEE7433ANXUHKYIJLZ36T25TXA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.773128333 (556h53m57.644211368s) [noise=true , address=77.90.31.17:4770 , piece_id=MH6LXF5JVTVI3U3N7W73U4ULOARVA6G2CRNUQLQ3V2UE6HUDGCUA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.775274971 (556h53m57.646358006s) [noise=true , address=145.79.227.40:4051 , piece_id=PZ3547GPNB3TRB24RBQVWLHLOC5JQVFUMG4ZZJDA3BBESZN3XFYA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.777926579 (556h53m57.649009615s) [noise=true , address=145.79.227.42:4123 , piece_id=TN4ZO3UUKIFRAP44SANXU6HKESFMIE45S5XTWAIKHJW5SA6C2KFA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.780387376 (556h53m57.651470421s) [noise=true , address=45.91.116.13:4155 , piece_id=HHINRA5KN56ZF2KW7IEG4BUGXM6UKMNUM3TM4Y3HFADVWASFEHFA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.781667764 (556h53m57.652750809s) [noise=true , address=77.90.31.15:4697 , piece_id=25QRUIDDPIZ7CSGKDAXLMZIWJHE6HHITH6FU4H2GA2GDBI4PR5KA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.783365686 (556h53m57.654448731s) [noise=true , address=77.90.31.11:4556 , piece_id=QRRGWDO7EHFVVQ6X4BTVY7HQRRDF5UCHKCFRLO2FOOYLTEDNJR2A, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.784872727 (556h53m57.655955762s) [noise=true , address=145.79.227.21:4473 , piece_id=OHS3FHPTXD3EKQNBJLLGKENQIJPKPGEFTVIRVQYXO3PZQEVQILGQ, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:59.482505035 (556h53m59.35358808s) [upload=488235]: Next returned nil segment Dec 28 20:22:59.482533810 (556h53m59.353616845s) [upload=488235]: Waiting for error group managing segments... Dec 28 20:22:59.769352743 (556h53m59.640435799s) [noise=true , address=145.79.227.18:4007 , piece_id=NB6ADK5TXELRFCYMKDF6C3QZRZQUL5XUPVQ7AVVYP32UY2GZA6SA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.782950737 (556h53m59.654033782s) [noise=true , address=145.79.227.51:4409 , piece_id=C342XC375VQVI23V5JESCTOES4AF66QWHIETEOBVV3KGRLIP2X5Q, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.789678698 (556h53m59.660761733s) [noise=true , address=145.79.227.12:4765 , piece_id=IESWSXDQFYKKHDOEHSYQSVMTBYLAJWNQ5TPWGNQPMLHO23CXYXTQ, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.802190252 (556h53m59.673273287s) [noise=true , address=145.79.227.42:4123 , piece_id=TN4ZO3UUKIFRAP44SANXU6HKESFMIE45S5XTWAIKHJW5SA6C2KFA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.804531330 (556h53m59.675614375s) [noise=true , address=145.79.227.50:4375 , piece_id=DMDQHEH7XJT334MFCFAPNCGLSJ4UWE3U74MTDFROMTM4BDAI4IHA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.810012928 (556h53m59.681095963s) [noise=true , address=145.79.227.21:4473 , piece_id=OHS3FHPTXD3EKQNBJLLGKENQIJPKPGEFTVIRVQYXO3PZQEVQILGQ, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.812050608 (556h53m59.683133643s) [noise=true , address=145.79.227.36:4519 , piece_id=YIPTLMBP7UV4GLPNA74M4343U4JEE7433ANXUHKYIJLZ36T25TXA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.812993363 (556h53m59.684076398s) [noise=true , address=145.79.227.41:4086 , piece_id=YRHLQZ64IEO7C26HZTCDY2T2PRYUFP6CSN2EOBJS7XAE4I7MXLHQ, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.946324830 (556h53m59.817407865s) [noise=true , address=77.90.31.13:4627 , piece_id=XB2K7RONOVEBPA2XGY36WIE57MWUUBKVAHEGXHAZWNDRL22WHH5Q, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.965566208 (556h53m59.836649243s) [noise=true , address=77.90.31.17:4770 , piece_id=MH6LXF5JVTVI3U3N7W73U4ULOARVA6G2CRNUQLQ3V2UE6HUDGCUA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:23:00.024382405 (556h53m59.89546544s) [noise=true , address=77.90.31.21:4880 , piece_id=V4PF2AQHKOK3GCF6B4UDNRJW7WYQ5QVOC66HUJMOPOOLPZAQIM4A, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:23:00.056351529 (556h53m59.927434574s) [noise=true , address=45.91.116.5:4297 , piece_id=FNX3RGGDGJZL5DOZRL2ZF2K2JHRAHWAWHIN6NQ4ONUKPVC6J4ECA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:23:00.067006559 (556h53m59.938089594s) [noise=true , address=45.91.116.13:4155 , piece_id=HHINRA5KN56ZF2KW7IEG4BUGXM6UKMNUM3TM4Y3HFADVWASFEHFA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:23:00.067025144 (556h53m59.938108179s) [seg_pos={0 0}, upload=488235]: Segment reached optimal threshold of 13 pieces. Dec 28 20:23:00.067214014 (556h53m59.938297059s) [noise=true , address=77.90.31.11:4556 , piece_id=QRRGWDO7EHFVVQ6X4BTVY7HQRRDF5UCHKCFRLO2FOOYLTEDNJR2A, seg_pos={0 0}, upload=488235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: write tcp 188.166.225.170:60580->77.90.31.11:4556: use of closed network connection; write tcp 188.166.225.170:60580->77.90.31.11:4556: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60580->77.90.31.11:4556: use of closed network connection Dec 28 20:23:00.067225536 (556h53m59.938308571s) [noise=true , address=77.90.31.11:4556 , piece_id=QRRGWDO7EHFVVQ6X4BTVY7HQRRDF5UCHKCFRLO2FOOYLTEDNJR2A, seg_pos={0 0}, upload=488235]: Long tail cancellation due to optimal threshold Dec 28 20:23:00.067306319 (556h53m59.938389364s) [noise=true , address=77.90.31.15:4697 , piece_id=25QRUIDDPIZ7CSGKDAXLMZIWJHE6HHITH6FU4H2GA2GDBI4PR5KA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Dec 28 20:23:00.067314354 (556h53m59.938397389s) [noise=true , address=77.90.31.15:4697 , piece_id=25QRUIDDPIZ7CSGKDAXLMZIWJHE6HHITH6FU4H2GA2GDBI4PR5KA, seg_pos={0 0}, upload=488235]: Long tail cancellation due to optimal threshold Dec 28 20:23:00.067582064 (556h53m59.938665119s) [noise=true , address=145.79.227.40:4051 , piece_id=PZ3547GPNB3TRB24RBQVWLHLOC5JQVFUMG4ZZJDA3BBESZN3XFYA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 28 20:23:00.067589408 (556h53m59.938672453s) [noise=true , address=145.79.227.40:4051 , piece_id=PZ3547GPNB3TRB24RBQVWLHLOC5JQVFUMG4ZZJDA3BBESZN3XFYA, seg_pos={0 0}, upload=488235]: Long tail cancellation due to optimal threshold Dec 28 20:23:00.067605620 (556h53m59.938688655s) [seg_pos={0 0}, upload=488235]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 20:23:00.067681614 (556h53m59.938764659s) [upload=488235]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 20:23:00.067686654 (556h53m59.938769699s) [upload=488235]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 20:23:00.354014782 (556h54m0.225097827s) [upload=488235]: Done uploading segments... Dec 28 20:23:00.354034600 (556h54m0.225117635s) [upload=488235]: Upload finished. err: <nil> Duration: 7.456691265s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........X..........................X...........X..........] duration: 5.848361324s trace: https://jaeger.dev.storj.io/trace/3425d363be458a81 FAIL (2025-12-28 20:30:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: XB2K7RONOVEBPA2XGY36WIE57MWUUBKVAHEGXHAZWNDRL22WHH5Q) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: HHINRA5KN56ZF2KW7IEG4BUGXM6UKMNUM3TM4Y3HFADVWASFEHFA) context deadline exceeded ======LOGS====== Dec 28 20:22:57.518307420 (556h53m57.389390455s) [upload=488235]: Starting upload Dec 28 20:22:57.518369106 (556h53m57.389452182s) [upload=488235]: Done starting upload Dec 28 20:22:57.518427107 (556h53m57.389510152s) [upload=488235]: Uploading segments... Dec 28 20:22:57.518663427 (556h53m57.389746473s) [upload=488235]: Got next segment. Inline: false Dec 28 20:22:57.518709805 (556h53m57.38979284s) [upload=488235]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 20:22:57.518716318 (556h53m57.389799353s) [upload=488235]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 20:22:57.767659750 (556h53m57.638742815s) [seg_pos={0 0}, upload=488235]: Begin upload segment... Dec 28 20:22:57.767735904 (556h53m57.638818949s) [seg_pos={0 0}, upload=488235]: Done begin upload segment. Dec 28 20:22:57.767792252 (556h53m57.638875287s) [noise=true , address=77.90.31.21:4880 , piece_id=V4PF2AQHKOK3GCF6B4UDNRJW7WYQ5QVOC66HUJMOPOOLPZAQIM4A, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.767877795 (556h53m57.63896085s) [noise=true , address=145.79.227.51:4409 , piece_id=C342XC375VQVI23V5JESCTOES4AF66QWHIETEOBVV3KGRLIP2X5Q, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.768512193 (556h53m57.639595238s) [noise=true , address=145.79.227.18:4007 , piece_id=NB6ADK5TXELRFCYMKDF6C3QZRZQUL5XUPVQ7AVVYP32UY2GZA6SA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.768539074 (556h53m57.639622119s) [noise=true , address=45.91.116.5:4297 , piece_id=FNX3RGGDGJZL5DOZRL2ZF2K2JHRAHWAWHIN6NQ4ONUKPVC6J4ECA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.769129117 (556h53m57.640212152s) [noise=true , address=145.79.227.41:4086 , piece_id=YRHLQZ64IEO7C26HZTCDY2T2PRYUFP6CSN2EOBJS7XAE4I7MXLHQ, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.769521375 (556h53m57.64060441s) [noise=true , address=145.79.227.12:4765 , piece_id=IESWSXDQFYKKHDOEHSYQSVMTBYLAJWNQ5TPWGNQPMLHO23CXYXTQ, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.770627170 (556h53m57.641710205s) [noise=true , address=77.90.31.13:4627 , piece_id=XB2K7RONOVEBPA2XGY36WIE57MWUUBKVAHEGXHAZWNDRL22WHH5Q, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.771369815 (556h53m57.64245285s) [noise=true , address=145.79.227.50:4375 , piece_id=DMDQHEH7XJT334MFCFAPNCGLSJ4UWE3U74MTDFROMTM4BDAI4IHA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.772492252 (556h53m57.643575297s) [noise=true , address=145.79.227.36:4519 , piece_id=YIPTLMBP7UV4GLPNA74M4343U4JEE7433ANXUHKYIJLZ36T25TXA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.773128333 (556h53m57.644211368s) [noise=true , address=77.90.31.17:4770 , piece_id=MH6LXF5JVTVI3U3N7W73U4ULOARVA6G2CRNUQLQ3V2UE6HUDGCUA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.775274971 (556h53m57.646358006s) [noise=true , address=145.79.227.40:4051 , piece_id=PZ3547GPNB3TRB24RBQVWLHLOC5JQVFUMG4ZZJDA3BBESZN3XFYA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.777926579 (556h53m57.649009615s) [noise=true , address=145.79.227.42:4123 , piece_id=TN4ZO3UUKIFRAP44SANXU6HKESFMIE45S5XTWAIKHJW5SA6C2KFA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.780387376 (556h53m57.651470421s) [noise=true , address=45.91.116.13:4155 , piece_id=HHINRA5KN56ZF2KW7IEG4BUGXM6UKMNUM3TM4Y3HFADVWASFEHFA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.781667764 (556h53m57.652750809s) [noise=true , address=77.90.31.15:4697 , piece_id=25QRUIDDPIZ7CSGKDAXLMZIWJHE6HHITH6FU4H2GA2GDBI4PR5KA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.783365686 (556h53m57.654448731s) [noise=true , address=77.90.31.11:4556 , piece_id=QRRGWDO7EHFVVQ6X4BTVY7HQRRDF5UCHKCFRLO2FOOYLTEDNJR2A, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.784872727 (556h53m57.655955762s) [noise=true , address=145.79.227.21:4473 , piece_id=OHS3FHPTXD3EKQNBJLLGKENQIJPKPGEFTVIRVQYXO3PZQEVQILGQ, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:59.482505035 (556h53m59.35358808s) [upload=488235]: Next returned nil segment Dec 28 20:22:59.482533810 (556h53m59.353616845s) [upload=488235]: Waiting for error group managing segments... Dec 28 20:22:59.769352743 (556h53m59.640435799s) [noise=true , address=145.79.227.18:4007 , piece_id=NB6ADK5TXELRFCYMKDF6C3QZRZQUL5XUPVQ7AVVYP32UY2GZA6SA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.782950737 (556h53m59.654033782s) [noise=true , address=145.79.227.51:4409 , piece_id=C342XC375VQVI23V5JESCTOES4AF66QWHIETEOBVV3KGRLIP2X5Q, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.789678698 (556h53m59.660761733s) [noise=true , address=145.79.227.12:4765 , piece_id=IESWSXDQFYKKHDOEHSYQSVMTBYLAJWNQ5TPWGNQPMLHO23CXYXTQ, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.802190252 (556h53m59.673273287s) [noise=true , address=145.79.227.42:4123 , piece_id=TN4ZO3UUKIFRAP44SANXU6HKESFMIE45S5XTWAIKHJW5SA6C2KFA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.804531330 (556h53m59.675614375s) [noise=true , address=145.79.227.50:4375 , piece_id=DMDQHEH7XJT334MFCFAPNCGLSJ4UWE3U74MTDFROMTM4BDAI4IHA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.810012928 (556h53m59.681095963s) [noise=true , address=145.79.227.21:4473 , piece_id=OHS3FHPTXD3EKQNBJLLGKENQIJPKPGEFTVIRVQYXO3PZQEVQILGQ, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.812050608 (556h53m59.683133643s) [noise=true , address=145.79.227.36:4519 , piece_id=YIPTLMBP7UV4GLPNA74M4343U4JEE7433ANXUHKYIJLZ36T25TXA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.812993363 (556h53m59.684076398s) [noise=true , address=145.79.227.41:4086 , piece_id=YRHLQZ64IEO7C26HZTCDY2T2PRYUFP6CSN2EOBJS7XAE4I7MXLHQ, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.946324830 (556h53m59.817407865s) [noise=true , address=77.90.31.13:4627 , piece_id=XB2K7RONOVEBPA2XGY36WIE57MWUUBKVAHEGXHAZWNDRL22WHH5Q, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.965566208 (556h53m59.836649243s) [noise=true , address=77.90.31.17:4770 , piece_id=MH6LXF5JVTVI3U3N7W73U4ULOARVA6G2CRNUQLQ3V2UE6HUDGCUA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:23:00.024382405 (556h53m59.89546544s) [noise=true , address=77.90.31.21:4880 , piece_id=V4PF2AQHKOK3GCF6B4UDNRJW7WYQ5QVOC66HUJMOPOOLPZAQIM4A, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:23:00.056351529 (556h53m59.927434574s) [noise=true , address=45.91.116.5:4297 , piece_id=FNX3RGGDGJZL5DOZRL2ZF2K2JHRAHWAWHIN6NQ4ONUKPVC6J4ECA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:23:00.067006559 (556h53m59.938089594s) [noise=true , address=45.91.116.13:4155 , piece_id=HHINRA5KN56ZF2KW7IEG4BUGXM6UKMNUM3TM4Y3HFADVWASFEHFA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:23:00.067025144 (556h53m59.938108179s) [seg_pos={0 0}, upload=488235]: Segment reached optimal threshold of 13 pieces. Dec 28 20:23:00.067214014 (556h53m59.938297059s) [noise=true , address=77.90.31.11:4556 , piece_id=QRRGWDO7EHFVVQ6X4BTVY7HQRRDF5UCHKCFRLO2FOOYLTEDNJR2A, seg_pos={0 0}, upload=488235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: write tcp 188.166.225.170:60580->77.90.31.11:4556: use of closed network connection; write tcp 188.166.225.170:60580->77.90.31.11:4556: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60580->77.90.31.11:4556: use of closed network connection Dec 28 20:23:00.067225536 (556h53m59.938308571s) [noise=true , address=77.90.31.11:4556 , piece_id=QRRGWDO7EHFVVQ6X4BTVY7HQRRDF5UCHKCFRLO2FOOYLTEDNJR2A, seg_pos={0 0}, upload=488235]: Long tail cancellation due to optimal threshold Dec 28 20:23:00.067306319 (556h53m59.938389364s) [noise=true , address=77.90.31.15:4697 , piece_id=25QRUIDDPIZ7CSGKDAXLMZIWJHE6HHITH6FU4H2GA2GDBI4PR5KA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Dec 28 20:23:00.067314354 (556h53m59.938397389s) [noise=true , address=77.90.31.15:4697 , piece_id=25QRUIDDPIZ7CSGKDAXLMZIWJHE6HHITH6FU4H2GA2GDBI4PR5KA, seg_pos={0 0}, upload=488235]: Long tail cancellation due to optimal threshold Dec 28 20:23:00.067582064 (556h53m59.938665119s) [noise=true , address=145.79.227.40:4051 , piece_id=PZ3547GPNB3TRB24RBQVWLHLOC5JQVFUMG4ZZJDA3BBESZN3XFYA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 28 20:23:00.067589408 (556h53m59.938672453s) [noise=true , address=145.79.227.40:4051 , piece_id=PZ3547GPNB3TRB24RBQVWLHLOC5JQVFUMG4ZZJDA3BBESZN3XFYA, seg_pos={0 0}, upload=488235]: Long tail cancellation due to optimal threshold Dec 28 20:23:00.067605620 (556h53m59.938688655s) [seg_pos={0 0}, upload=488235]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 20:23:00.067681614 (556h53m59.938764659s) [upload=488235]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 20:23:00.067686654 (556h53m59.938769699s) [upload=488235]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 20:23:00.354014782 (556h54m0.225097827s) [upload=488235]: Done uploading segments... Dec 28 20:23:00.354034600 (556h54m0.225117635s) [upload=488235]: Upload finished. err: <nil> Duration: 5.848388977s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........X..........................X...........X.........] duration: 7.621271797s trace: https://jaeger.dev.storj.io/trace/3425d363be458a81 FAIL (2025-12-28 20:29:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: XB2K7RONOVEBPA2XGY36WIE57MWUUBKVAHEGXHAZWNDRL22WHH5Q) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: HHINRA5KN56ZF2KW7IEG4BUGXM6UKMNUM3TM4Y3HFADVWASFEHFA) context deadline exceeded ======LOGS====== Dec 28 20:22:57.518307420 (556h53m57.389390455s) [upload=488235]: Starting upload Dec 28 20:22:57.518369106 (556h53m57.389452182s) [upload=488235]: Done starting upload Dec 28 20:22:57.518427107 (556h53m57.389510152s) [upload=488235]: Uploading segments... Dec 28 20:22:57.518663427 (556h53m57.389746473s) [upload=488235]: Got next segment. Inline: false Dec 28 20:22:57.518709805 (556h53m57.38979284s) [upload=488235]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 20:22:57.518716318 (556h53m57.389799353s) [upload=488235]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 20:22:57.767659750 (556h53m57.638742815s) [seg_pos={0 0}, upload=488235]: Begin upload segment... Dec 28 20:22:57.767735904 (556h53m57.638818949s) [seg_pos={0 0}, upload=488235]: Done begin upload segment. Dec 28 20:22:57.767792252 (556h53m57.638875287s) [noise=true , address=77.90.31.21:4880 , piece_id=V4PF2AQHKOK3GCF6B4UDNRJW7WYQ5QVOC66HUJMOPOOLPZAQIM4A, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.767877795 (556h53m57.63896085s) [noise=true , address=145.79.227.51:4409 , piece_id=C342XC375VQVI23V5JESCTOES4AF66QWHIETEOBVV3KGRLIP2X5Q, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.768512193 (556h53m57.639595238s) [noise=true , address=145.79.227.18:4007 , piece_id=NB6ADK5TXELRFCYMKDF6C3QZRZQUL5XUPVQ7AVVYP32UY2GZA6SA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.768539074 (556h53m57.639622119s) [noise=true , address=45.91.116.5:4297 , piece_id=FNX3RGGDGJZL5DOZRL2ZF2K2JHRAHWAWHIN6NQ4ONUKPVC6J4ECA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.769129117 (556h53m57.640212152s) [noise=true , address=145.79.227.41:4086 , piece_id=YRHLQZ64IEO7C26HZTCDY2T2PRYUFP6CSN2EOBJS7XAE4I7MXLHQ, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.769521375 (556h53m57.64060441s) [noise=true , address=145.79.227.12:4765 , piece_id=IESWSXDQFYKKHDOEHSYQSVMTBYLAJWNQ5TPWGNQPMLHO23CXYXTQ, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.770627170 (556h53m57.641710205s) [noise=true , address=77.90.31.13:4627 , piece_id=XB2K7RONOVEBPA2XGY36WIE57MWUUBKVAHEGXHAZWNDRL22WHH5Q, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.771369815 (556h53m57.64245285s) [noise=true , address=145.79.227.50:4375 , piece_id=DMDQHEH7XJT334MFCFAPNCGLSJ4UWE3U74MTDFROMTM4BDAI4IHA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.772492252 (556h53m57.643575297s) [noise=true , address=145.79.227.36:4519 , piece_id=YIPTLMBP7UV4GLPNA74M4343U4JEE7433ANXUHKYIJLZ36T25TXA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.773128333 (556h53m57.644211368s) [noise=true , address=77.90.31.17:4770 , piece_id=MH6LXF5JVTVI3U3N7W73U4ULOARVA6G2CRNUQLQ3V2UE6HUDGCUA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.775274971 (556h53m57.646358006s) [noise=true , address=145.79.227.40:4051 , piece_id=PZ3547GPNB3TRB24RBQVWLHLOC5JQVFUMG4ZZJDA3BBESZN3XFYA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.777926579 (556h53m57.649009615s) [noise=true , address=145.79.227.42:4123 , piece_id=TN4ZO3UUKIFRAP44SANXU6HKESFMIE45S5XTWAIKHJW5SA6C2KFA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.780387376 (556h53m57.651470421s) [noise=true , address=45.91.116.13:4155 , piece_id=HHINRA5KN56ZF2KW7IEG4BUGXM6UKMNUM3TM4Y3HFADVWASFEHFA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.781667764 (556h53m57.652750809s) [noise=true , address=77.90.31.15:4697 , piece_id=25QRUIDDPIZ7CSGKDAXLMZIWJHE6HHITH6FU4H2GA2GDBI4PR5KA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.783365686 (556h53m57.654448731s) [noise=true , address=77.90.31.11:4556 , piece_id=QRRGWDO7EHFVVQ6X4BTVY7HQRRDF5UCHKCFRLO2FOOYLTEDNJR2A, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.784872727 (556h53m57.655955762s) [noise=true , address=145.79.227.21:4473 , piece_id=OHS3FHPTXD3EKQNBJLLGKENQIJPKPGEFTVIRVQYXO3PZQEVQILGQ, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:59.482505035 (556h53m59.35358808s) [upload=488235]: Next returned nil segment Dec 28 20:22:59.482533810 (556h53m59.353616845s) [upload=488235]: Waiting for error group managing segments... Dec 28 20:22:59.769352743 (556h53m59.640435799s) [noise=true , address=145.79.227.18:4007 , piece_id=NB6ADK5TXELRFCYMKDF6C3QZRZQUL5XUPVQ7AVVYP32UY2GZA6SA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.782950737 (556h53m59.654033782s) [noise=true , address=145.79.227.51:4409 , piece_id=C342XC375VQVI23V5JESCTOES4AF66QWHIETEOBVV3KGRLIP2X5Q, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.789678698 (556h53m59.660761733s) [noise=true , address=145.79.227.12:4765 , piece_id=IESWSXDQFYKKHDOEHSYQSVMTBYLAJWNQ5TPWGNQPMLHO23CXYXTQ, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.802190252 (556h53m59.673273287s) [noise=true , address=145.79.227.42:4123 , piece_id=TN4ZO3UUKIFRAP44SANXU6HKESFMIE45S5XTWAIKHJW5SA6C2KFA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.804531330 (556h53m59.675614375s) [noise=true , address=145.79.227.50:4375 , piece_id=DMDQHEH7XJT334MFCFAPNCGLSJ4UWE3U74MTDFROMTM4BDAI4IHA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.810012928 (556h53m59.681095963s) [noise=true , address=145.79.227.21:4473 , piece_id=OHS3FHPTXD3EKQNBJLLGKENQIJPKPGEFTVIRVQYXO3PZQEVQILGQ, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.812050608 (556h53m59.683133643s) [noise=true , address=145.79.227.36:4519 , piece_id=YIPTLMBP7UV4GLPNA74M4343U4JEE7433ANXUHKYIJLZ36T25TXA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.812993363 (556h53m59.684076398s) [noise=true , address=145.79.227.41:4086 , piece_id=YRHLQZ64IEO7C26HZTCDY2T2PRYUFP6CSN2EOBJS7XAE4I7MXLHQ, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.946324830 (556h53m59.817407865s) [noise=true , address=77.90.31.13:4627 , piece_id=XB2K7RONOVEBPA2XGY36WIE57MWUUBKVAHEGXHAZWNDRL22WHH5Q, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.965566208 (556h53m59.836649243s) [noise=true , address=77.90.31.17:4770 , piece_id=MH6LXF5JVTVI3U3N7W73U4ULOARVA6G2CRNUQLQ3V2UE6HUDGCUA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:23:00.024382405 (556h53m59.89546544s) [noise=true , address=77.90.31.21:4880 , piece_id=V4PF2AQHKOK3GCF6B4UDNRJW7WYQ5QVOC66HUJMOPOOLPZAQIM4A, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:23:00.056351529 (556h53m59.927434574s) [noise=true , address=45.91.116.5:4297 , piece_id=FNX3RGGDGJZL5DOZRL2ZF2K2JHRAHWAWHIN6NQ4ONUKPVC6J4ECA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:23:00.067006559 (556h53m59.938089594s) [noise=true , address=45.91.116.13:4155 , piece_id=HHINRA5KN56ZF2KW7IEG4BUGXM6UKMNUM3TM4Y3HFADVWASFEHFA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:23:00.067025144 (556h53m59.938108179s) [seg_pos={0 0}, upload=488235]: Segment reached optimal threshold of 13 pieces. Dec 28 20:23:00.067214014 (556h53m59.938297059s) [noise=true , address=77.90.31.11:4556 , piece_id=QRRGWDO7EHFVVQ6X4BTVY7HQRRDF5UCHKCFRLO2FOOYLTEDNJR2A, seg_pos={0 0}, upload=488235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: write tcp 188.166.225.170:60580->77.90.31.11:4556: use of closed network connection; write tcp 188.166.225.170:60580->77.90.31.11:4556: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60580->77.90.31.11:4556: use of closed network connection Dec 28 20:23:00.067225536 (556h53m59.938308571s) [noise=true , address=77.90.31.11:4556 , piece_id=QRRGWDO7EHFVVQ6X4BTVY7HQRRDF5UCHKCFRLO2FOOYLTEDNJR2A, seg_pos={0 0}, upload=488235]: Long tail cancellation due to optimal threshold Dec 28 20:23:00.067306319 (556h53m59.938389364s) [noise=true , address=77.90.31.15:4697 , piece_id=25QRUIDDPIZ7CSGKDAXLMZIWJHE6HHITH6FU4H2GA2GDBI4PR5KA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Dec 28 20:23:00.067314354 (556h53m59.938397389s) [noise=true , address=77.90.31.15:4697 , piece_id=25QRUIDDPIZ7CSGKDAXLMZIWJHE6HHITH6FU4H2GA2GDBI4PR5KA, seg_pos={0 0}, upload=488235]: Long tail cancellation due to optimal threshold Dec 28 20:23:00.067582064 (556h53m59.938665119s) [noise=true , address=145.79.227.40:4051 , piece_id=PZ3547GPNB3TRB24RBQVWLHLOC5JQVFUMG4ZZJDA3BBESZN3XFYA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 28 20:23:00.067589408 (556h53m59.938672453s) [noise=true , address=145.79.227.40:4051 , piece_id=PZ3547GPNB3TRB24RBQVWLHLOC5JQVFUMG4ZZJDA3BBESZN3XFYA, seg_pos={0 0}, upload=488235]: Long tail cancellation due to optimal threshold Dec 28 20:23:00.067605620 (556h53m59.938688655s) [seg_pos={0 0}, upload=488235]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 20:23:00.067681614 (556h53m59.938764659s) [upload=488235]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 20:23:00.067686654 (556h53m59.938769699s) [upload=488235]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 20:23:00.354014782 (556h54m0.225097827s) [upload=488235]: Done uploading segments... Dec 28 20:23:00.354034600 (556h54m0.225117635s) [upload=488235]: Upload finished. err: <nil> Duration: 7.621305661s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [............X..........................X...........X........] duration: 6.123378173s trace: https://jaeger.dev.storj.io/trace/3425d363be458a81 FAIL (2025-12-28 20:29:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: XB2K7RONOVEBPA2XGY36WIE57MWUUBKVAHEGXHAZWNDRL22WHH5Q) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: HHINRA5KN56ZF2KW7IEG4BUGXM6UKMNUM3TM4Y3HFADVWASFEHFA) context deadline exceeded ======LOGS====== Dec 28 20:22:57.518307420 (556h53m57.389390455s) [upload=488235]: Starting upload Dec 28 20:22:57.518369106 (556h53m57.389452182s) [upload=488235]: Done starting upload Dec 28 20:22:57.518427107 (556h53m57.389510152s) [upload=488235]: Uploading segments... Dec 28 20:22:57.518663427 (556h53m57.389746473s) [upload=488235]: Got next segment. Inline: false Dec 28 20:22:57.518709805 (556h53m57.38979284s) [upload=488235]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 20:22:57.518716318 (556h53m57.389799353s) [upload=488235]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 20:22:57.767659750 (556h53m57.638742815s) [seg_pos={0 0}, upload=488235]: Begin upload segment... Dec 28 20:22:57.767735904 (556h53m57.638818949s) [seg_pos={0 0}, upload=488235]: Done begin upload segment. Dec 28 20:22:57.767792252 (556h53m57.638875287s) [noise=true , address=77.90.31.21:4880 , piece_id=V4PF2AQHKOK3GCF6B4UDNRJW7WYQ5QVOC66HUJMOPOOLPZAQIM4A, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.767877795 (556h53m57.63896085s) [noise=true , address=145.79.227.51:4409 , piece_id=C342XC375VQVI23V5JESCTOES4AF66QWHIETEOBVV3KGRLIP2X5Q, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.768512193 (556h53m57.639595238s) [noise=true , address=145.79.227.18:4007 , piece_id=NB6ADK5TXELRFCYMKDF6C3QZRZQUL5XUPVQ7AVVYP32UY2GZA6SA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.768539074 (556h53m57.639622119s) [noise=true , address=45.91.116.5:4297 , piece_id=FNX3RGGDGJZL5DOZRL2ZF2K2JHRAHWAWHIN6NQ4ONUKPVC6J4ECA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.769129117 (556h53m57.640212152s) [noise=true , address=145.79.227.41:4086 , piece_id=YRHLQZ64IEO7C26HZTCDY2T2PRYUFP6CSN2EOBJS7XAE4I7MXLHQ, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.769521375 (556h53m57.64060441s) [noise=true , address=145.79.227.12:4765 , piece_id=IESWSXDQFYKKHDOEHSYQSVMTBYLAJWNQ5TPWGNQPMLHO23CXYXTQ, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.770627170 (556h53m57.641710205s) [noise=true , address=77.90.31.13:4627 , piece_id=XB2K7RONOVEBPA2XGY36WIE57MWUUBKVAHEGXHAZWNDRL22WHH5Q, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.771369815 (556h53m57.64245285s) [noise=true , address=145.79.227.50:4375 , piece_id=DMDQHEH7XJT334MFCFAPNCGLSJ4UWE3U74MTDFROMTM4BDAI4IHA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.772492252 (556h53m57.643575297s) [noise=true , address=145.79.227.36:4519 , piece_id=YIPTLMBP7UV4GLPNA74M4343U4JEE7433ANXUHKYIJLZ36T25TXA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.773128333 (556h53m57.644211368s) [noise=true , address=77.90.31.17:4770 , piece_id=MH6LXF5JVTVI3U3N7W73U4ULOARVA6G2CRNUQLQ3V2UE6HUDGCUA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.775274971 (556h53m57.646358006s) [noise=true , address=145.79.227.40:4051 , piece_id=PZ3547GPNB3TRB24RBQVWLHLOC5JQVFUMG4ZZJDA3BBESZN3XFYA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.777926579 (556h53m57.649009615s) [noise=true , address=145.79.227.42:4123 , piece_id=TN4ZO3UUKIFRAP44SANXU6HKESFMIE45S5XTWAIKHJW5SA6C2KFA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.780387376 (556h53m57.651470421s) [noise=true , address=45.91.116.13:4155 , piece_id=HHINRA5KN56ZF2KW7IEG4BUGXM6UKMNUM3TM4Y3HFADVWASFEHFA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.781667764 (556h53m57.652750809s) [noise=true , address=77.90.31.15:4697 , piece_id=25QRUIDDPIZ7CSGKDAXLMZIWJHE6HHITH6FU4H2GA2GDBI4PR5KA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.783365686 (556h53m57.654448731s) [noise=true , address=77.90.31.11:4556 , piece_id=QRRGWDO7EHFVVQ6X4BTVY7HQRRDF5UCHKCFRLO2FOOYLTEDNJR2A, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.784872727 (556h53m57.655955762s) [noise=true , address=145.79.227.21:4473 , piece_id=OHS3FHPTXD3EKQNBJLLGKENQIJPKPGEFTVIRVQYXO3PZQEVQILGQ, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:59.482505035 (556h53m59.35358808s) [upload=488235]: Next returned nil segment Dec 28 20:22:59.482533810 (556h53m59.353616845s) [upload=488235]: Waiting for error group managing segments... Dec 28 20:22:59.769352743 (556h53m59.640435799s) [noise=true , address=145.79.227.18:4007 , piece_id=NB6ADK5TXELRFCYMKDF6C3QZRZQUL5XUPVQ7AVVYP32UY2GZA6SA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.782950737 (556h53m59.654033782s) [noise=true , address=145.79.227.51:4409 , piece_id=C342XC375VQVI23V5JESCTOES4AF66QWHIETEOBVV3KGRLIP2X5Q, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.789678698 (556h53m59.660761733s) [noise=true , address=145.79.227.12:4765 , piece_id=IESWSXDQFYKKHDOEHSYQSVMTBYLAJWNQ5TPWGNQPMLHO23CXYXTQ, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.802190252 (556h53m59.673273287s) [noise=true , address=145.79.227.42:4123 , piece_id=TN4ZO3UUKIFRAP44SANXU6HKESFMIE45S5XTWAIKHJW5SA6C2KFA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.804531330 (556h53m59.675614375s) [noise=true , address=145.79.227.50:4375 , piece_id=DMDQHEH7XJT334MFCFAPNCGLSJ4UWE3U74MTDFROMTM4BDAI4IHA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.810012928 (556h53m59.681095963s) [noise=true , address=145.79.227.21:4473 , piece_id=OHS3FHPTXD3EKQNBJLLGKENQIJPKPGEFTVIRVQYXO3PZQEVQILGQ, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.812050608 (556h53m59.683133643s) [noise=true , address=145.79.227.36:4519 , piece_id=YIPTLMBP7UV4GLPNA74M4343U4JEE7433ANXUHKYIJLZ36T25TXA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.812993363 (556h53m59.684076398s) [noise=true , address=145.79.227.41:4086 , piece_id=YRHLQZ64IEO7C26HZTCDY2T2PRYUFP6CSN2EOBJS7XAE4I7MXLHQ, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.946324830 (556h53m59.817407865s) [noise=true , address=77.90.31.13:4627 , piece_id=XB2K7RONOVEBPA2XGY36WIE57MWUUBKVAHEGXHAZWNDRL22WHH5Q, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.965566208 (556h53m59.836649243s) [noise=true , address=77.90.31.17:4770 , piece_id=MH6LXF5JVTVI3U3N7W73U4ULOARVA6G2CRNUQLQ3V2UE6HUDGCUA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:23:00.024382405 (556h53m59.89546544s) [noise=true , address=77.90.31.21:4880 , piece_id=V4PF2AQHKOK3GCF6B4UDNRJW7WYQ5QVOC66HUJMOPOOLPZAQIM4A, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:23:00.056351529 (556h53m59.927434574s) [noise=true , address=45.91.116.5:4297 , piece_id=FNX3RGGDGJZL5DOZRL2ZF2K2JHRAHWAWHIN6NQ4ONUKPVC6J4ECA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:23:00.067006559 (556h53m59.938089594s) [noise=true , address=45.91.116.13:4155 , piece_id=HHINRA5KN56ZF2KW7IEG4BUGXM6UKMNUM3TM4Y3HFADVWASFEHFA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:23:00.067025144 (556h53m59.938108179s) [seg_pos={0 0}, upload=488235]: Segment reached optimal threshold of 13 pieces. Dec 28 20:23:00.067214014 (556h53m59.938297059s) [noise=true , address=77.90.31.11:4556 , piece_id=QRRGWDO7EHFVVQ6X4BTVY7HQRRDF5UCHKCFRLO2FOOYLTEDNJR2A, seg_pos={0 0}, upload=488235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: write tcp 188.166.225.170:60580->77.90.31.11:4556: use of closed network connection; write tcp 188.166.225.170:60580->77.90.31.11:4556: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60580->77.90.31.11:4556: use of closed network connection Dec 28 20:23:00.067225536 (556h53m59.938308571s) [noise=true , address=77.90.31.11:4556 , piece_id=QRRGWDO7EHFVVQ6X4BTVY7HQRRDF5UCHKCFRLO2FOOYLTEDNJR2A, seg_pos={0 0}, upload=488235]: Long tail cancellation due to optimal threshold Dec 28 20:23:00.067306319 (556h53m59.938389364s) [noise=true , address=77.90.31.15:4697 , piece_id=25QRUIDDPIZ7CSGKDAXLMZIWJHE6HHITH6FU4H2GA2GDBI4PR5KA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Dec 28 20:23:00.067314354 (556h53m59.938397389s) [noise=true , address=77.90.31.15:4697 , piece_id=25QRUIDDPIZ7CSGKDAXLMZIWJHE6HHITH6FU4H2GA2GDBI4PR5KA, seg_pos={0 0}, upload=488235]: Long tail cancellation due to optimal threshold Dec 28 20:23:00.067582064 (556h53m59.938665119s) [noise=true , address=145.79.227.40:4051 , piece_id=PZ3547GPNB3TRB24RBQVWLHLOC5JQVFUMG4ZZJDA3BBESZN3XFYA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 28 20:23:00.067589408 (556h53m59.938672453s) [noise=true , address=145.79.227.40:4051 , piece_id=PZ3547GPNB3TRB24RBQVWLHLOC5JQVFUMG4ZZJDA3BBESZN3XFYA, seg_pos={0 0}, upload=488235]: Long tail cancellation due to optimal threshold Dec 28 20:23:00.067605620 (556h53m59.938688655s) [seg_pos={0 0}, upload=488235]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 20:23:00.067681614 (556h53m59.938764659s) [upload=488235]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 20:23:00.067686654 (556h53m59.938769699s) [upload=488235]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 20:23:00.354014782 (556h54m0.225097827s) [upload=488235]: Done uploading segments... Dec 28 20:23:00.354034600 (556h54m0.225117635s) [upload=488235]: Upload finished. err: <nil> Duration: 6.123410645s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.............X..........................X...........X.......] duration: 5.998546724s trace: https://jaeger.dev.storj.io/trace/3425d363be458a81 FAIL (2025-12-28 20:28:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: XB2K7RONOVEBPA2XGY36WIE57MWUUBKVAHEGXHAZWNDRL22WHH5Q) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: HHINRA5KN56ZF2KW7IEG4BUGXM6UKMNUM3TM4Y3HFADVWASFEHFA) context deadline exceeded ======LOGS====== Dec 28 20:22:57.518307420 (556h53m57.389390455s) [upload=488235]: Starting upload Dec 28 20:22:57.518369106 (556h53m57.389452182s) [upload=488235]: Done starting upload Dec 28 20:22:57.518427107 (556h53m57.389510152s) [upload=488235]: Uploading segments... Dec 28 20:22:57.518663427 (556h53m57.389746473s) [upload=488235]: Got next segment. Inline: false Dec 28 20:22:57.518709805 (556h53m57.38979284s) [upload=488235]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 20:22:57.518716318 (556h53m57.389799353s) [upload=488235]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 20:22:57.767659750 (556h53m57.638742815s) [seg_pos={0 0}, upload=488235]: Begin upload segment... Dec 28 20:22:57.767735904 (556h53m57.638818949s) [seg_pos={0 0}, upload=488235]: Done begin upload segment. Dec 28 20:22:57.767792252 (556h53m57.638875287s) [noise=true , address=77.90.31.21:4880 , piece_id=V4PF2AQHKOK3GCF6B4UDNRJW7WYQ5QVOC66HUJMOPOOLPZAQIM4A, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.767877795 (556h53m57.63896085s) [noise=true , address=145.79.227.51:4409 , piece_id=C342XC375VQVI23V5JESCTOES4AF66QWHIETEOBVV3KGRLIP2X5Q, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.768512193 (556h53m57.639595238s) [noise=true , address=145.79.227.18:4007 , piece_id=NB6ADK5TXELRFCYMKDF6C3QZRZQUL5XUPVQ7AVVYP32UY2GZA6SA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.768539074 (556h53m57.639622119s) [noise=true , address=45.91.116.5:4297 , piece_id=FNX3RGGDGJZL5DOZRL2ZF2K2JHRAHWAWHIN6NQ4ONUKPVC6J4ECA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.769129117 (556h53m57.640212152s) [noise=true , address=145.79.227.41:4086 , piece_id=YRHLQZ64IEO7C26HZTCDY2T2PRYUFP6CSN2EOBJS7XAE4I7MXLHQ, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.769521375 (556h53m57.64060441s) [noise=true , address=145.79.227.12:4765 , piece_id=IESWSXDQFYKKHDOEHSYQSVMTBYLAJWNQ5TPWGNQPMLHO23CXYXTQ, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.770627170 (556h53m57.641710205s) [noise=true , address=77.90.31.13:4627 , piece_id=XB2K7RONOVEBPA2XGY36WIE57MWUUBKVAHEGXHAZWNDRL22WHH5Q, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.771369815 (556h53m57.64245285s) [noise=true , address=145.79.227.50:4375 , piece_id=DMDQHEH7XJT334MFCFAPNCGLSJ4UWE3U74MTDFROMTM4BDAI4IHA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.772492252 (556h53m57.643575297s) [noise=true , address=145.79.227.36:4519 , piece_id=YIPTLMBP7UV4GLPNA74M4343U4JEE7433ANXUHKYIJLZ36T25TXA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.773128333 (556h53m57.644211368s) [noise=true , address=77.90.31.17:4770 , piece_id=MH6LXF5JVTVI3U3N7W73U4ULOARVA6G2CRNUQLQ3V2UE6HUDGCUA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.775274971 (556h53m57.646358006s) [noise=true , address=145.79.227.40:4051 , piece_id=PZ3547GPNB3TRB24RBQVWLHLOC5JQVFUMG4ZZJDA3BBESZN3XFYA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.777926579 (556h53m57.649009615s) [noise=true , address=145.79.227.42:4123 , piece_id=TN4ZO3UUKIFRAP44SANXU6HKESFMIE45S5XTWAIKHJW5SA6C2KFA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.780387376 (556h53m57.651470421s) [noise=true , address=45.91.116.13:4155 , piece_id=HHINRA5KN56ZF2KW7IEG4BUGXM6UKMNUM3TM4Y3HFADVWASFEHFA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.781667764 (556h53m57.652750809s) [noise=true , address=77.90.31.15:4697 , piece_id=25QRUIDDPIZ7CSGKDAXLMZIWJHE6HHITH6FU4H2GA2GDBI4PR5KA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.783365686 (556h53m57.654448731s) [noise=true , address=77.90.31.11:4556 , piece_id=QRRGWDO7EHFVVQ6X4BTVY7HQRRDF5UCHKCFRLO2FOOYLTEDNJR2A, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.784872727 (556h53m57.655955762s) [noise=true , address=145.79.227.21:4473 , piece_id=OHS3FHPTXD3EKQNBJLLGKENQIJPKPGEFTVIRVQYXO3PZQEVQILGQ, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:59.482505035 (556h53m59.35358808s) [upload=488235]: Next returned nil segment Dec 28 20:22:59.482533810 (556h53m59.353616845s) [upload=488235]: Waiting for error group managing segments... Dec 28 20:22:59.769352743 (556h53m59.640435799s) [noise=true , address=145.79.227.18:4007 , piece_id=NB6ADK5TXELRFCYMKDF6C3QZRZQUL5XUPVQ7AVVYP32UY2GZA6SA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.782950737 (556h53m59.654033782s) [noise=true , address=145.79.227.51:4409 , piece_id=C342XC375VQVI23V5JESCTOES4AF66QWHIETEOBVV3KGRLIP2X5Q, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.789678698 (556h53m59.660761733s) [noise=true , address=145.79.227.12:4765 , piece_id=IESWSXDQFYKKHDOEHSYQSVMTBYLAJWNQ5TPWGNQPMLHO23CXYXTQ, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.802190252 (556h53m59.673273287s) [noise=true , address=145.79.227.42:4123 , piece_id=TN4ZO3UUKIFRAP44SANXU6HKESFMIE45S5XTWAIKHJW5SA6C2KFA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.804531330 (556h53m59.675614375s) [noise=true , address=145.79.227.50:4375 , piece_id=DMDQHEH7XJT334MFCFAPNCGLSJ4UWE3U74MTDFROMTM4BDAI4IHA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.810012928 (556h53m59.681095963s) [noise=true , address=145.79.227.21:4473 , piece_id=OHS3FHPTXD3EKQNBJLLGKENQIJPKPGEFTVIRVQYXO3PZQEVQILGQ, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.812050608 (556h53m59.683133643s) [noise=true , address=145.79.227.36:4519 , piece_id=YIPTLMBP7UV4GLPNA74M4343U4JEE7433ANXUHKYIJLZ36T25TXA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.812993363 (556h53m59.684076398s) [noise=true , address=145.79.227.41:4086 , piece_id=YRHLQZ64IEO7C26HZTCDY2T2PRYUFP6CSN2EOBJS7XAE4I7MXLHQ, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.946324830 (556h53m59.817407865s) [noise=true , address=77.90.31.13:4627 , piece_id=XB2K7RONOVEBPA2XGY36WIE57MWUUBKVAHEGXHAZWNDRL22WHH5Q, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.965566208 (556h53m59.836649243s) [noise=true , address=77.90.31.17:4770 , piece_id=MH6LXF5JVTVI3U3N7W73U4ULOARVA6G2CRNUQLQ3V2UE6HUDGCUA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:23:00.024382405 (556h53m59.89546544s) [noise=true , address=77.90.31.21:4880 , piece_id=V4PF2AQHKOK3GCF6B4UDNRJW7WYQ5QVOC66HUJMOPOOLPZAQIM4A, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:23:00.056351529 (556h53m59.927434574s) [noise=true , address=45.91.116.5:4297 , piece_id=FNX3RGGDGJZL5DOZRL2ZF2K2JHRAHWAWHIN6NQ4ONUKPVC6J4ECA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:23:00.067006559 (556h53m59.938089594s) [noise=true , address=45.91.116.13:4155 , piece_id=HHINRA5KN56ZF2KW7IEG4BUGXM6UKMNUM3TM4Y3HFADVWASFEHFA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:23:00.067025144 (556h53m59.938108179s) [seg_pos={0 0}, upload=488235]: Segment reached optimal threshold of 13 pieces. Dec 28 20:23:00.067214014 (556h53m59.938297059s) [noise=true , address=77.90.31.11:4556 , piece_id=QRRGWDO7EHFVVQ6X4BTVY7HQRRDF5UCHKCFRLO2FOOYLTEDNJR2A, seg_pos={0 0}, upload=488235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: write tcp 188.166.225.170:60580->77.90.31.11:4556: use of closed network connection; write tcp 188.166.225.170:60580->77.90.31.11:4556: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60580->77.90.31.11:4556: use of closed network connection Dec 28 20:23:00.067225536 (556h53m59.938308571s) [noise=true , address=77.90.31.11:4556 , piece_id=QRRGWDO7EHFVVQ6X4BTVY7HQRRDF5UCHKCFRLO2FOOYLTEDNJR2A, seg_pos={0 0}, upload=488235]: Long tail cancellation due to optimal threshold Dec 28 20:23:00.067306319 (556h53m59.938389364s) [noise=true , address=77.90.31.15:4697 , piece_id=25QRUIDDPIZ7CSGKDAXLMZIWJHE6HHITH6FU4H2GA2GDBI4PR5KA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Dec 28 20:23:00.067314354 (556h53m59.938397389s) [noise=true , address=77.90.31.15:4697 , piece_id=25QRUIDDPIZ7CSGKDAXLMZIWJHE6HHITH6FU4H2GA2GDBI4PR5KA, seg_pos={0 0}, upload=488235]: Long tail cancellation due to optimal threshold Dec 28 20:23:00.067582064 (556h53m59.938665119s) [noise=true , address=145.79.227.40:4051 , piece_id=PZ3547GPNB3TRB24RBQVWLHLOC5JQVFUMG4ZZJDA3BBESZN3XFYA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 28 20:23:00.067589408 (556h53m59.938672453s) [noise=true , address=145.79.227.40:4051 , piece_id=PZ3547GPNB3TRB24RBQVWLHLOC5JQVFUMG4ZZJDA3BBESZN3XFYA, seg_pos={0 0}, upload=488235]: Long tail cancellation due to optimal threshold Dec 28 20:23:00.067605620 (556h53m59.938688655s) [seg_pos={0 0}, upload=488235]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 20:23:00.067681614 (556h53m59.938764659s) [upload=488235]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 20:23:00.067686654 (556h53m59.938769699s) [upload=488235]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 20:23:00.354014782 (556h54m0.225097827s) [upload=488235]: Done uploading segments... Dec 28 20:23:00.354034600 (556h54m0.225117635s) [upload=488235]: Upload finished. err: <nil> Duration: 5.998573995s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..............X..........................X...........X......] duration: 11.086692143s trace: https://jaeger.dev.storj.io/trace/3425d363be458a81 FAIL (2025-12-28 20:27:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: XB2K7RONOVEBPA2XGY36WIE57MWUUBKVAHEGXHAZWNDRL22WHH5Q) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: HHINRA5KN56ZF2KW7IEG4BUGXM6UKMNUM3TM4Y3HFADVWASFEHFA) context deadline exceeded ======LOGS====== Dec 28 20:22:57.518307420 (556h53m57.389390455s) [upload=488235]: Starting upload Dec 28 20:22:57.518369106 (556h53m57.389452182s) [upload=488235]: Done starting upload Dec 28 20:22:57.518427107 (556h53m57.389510152s) [upload=488235]: Uploading segments... Dec 28 20:22:57.518663427 (556h53m57.389746473s) [upload=488235]: Got next segment. Inline: false Dec 28 20:22:57.518709805 (556h53m57.38979284s) [upload=488235]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 20:22:57.518716318 (556h53m57.389799353s) [upload=488235]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 20:22:57.767659750 (556h53m57.638742815s) [seg_pos={0 0}, upload=488235]: Begin upload segment... Dec 28 20:22:57.767735904 (556h53m57.638818949s) [seg_pos={0 0}, upload=488235]: Done begin upload segment. Dec 28 20:22:57.767792252 (556h53m57.638875287s) [noise=true , address=77.90.31.21:4880 , piece_id=V4PF2AQHKOK3GCF6B4UDNRJW7WYQ5QVOC66HUJMOPOOLPZAQIM4A, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.767877795 (556h53m57.63896085s) [noise=true , address=145.79.227.51:4409 , piece_id=C342XC375VQVI23V5JESCTOES4AF66QWHIETEOBVV3KGRLIP2X5Q, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.768512193 (556h53m57.639595238s) [noise=true , address=145.79.227.18:4007 , piece_id=NB6ADK5TXELRFCYMKDF6C3QZRZQUL5XUPVQ7AVVYP32UY2GZA6SA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.768539074 (556h53m57.639622119s) [noise=true , address=45.91.116.5:4297 , piece_id=FNX3RGGDGJZL5DOZRL2ZF2K2JHRAHWAWHIN6NQ4ONUKPVC6J4ECA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.769129117 (556h53m57.640212152s) [noise=true , address=145.79.227.41:4086 , piece_id=YRHLQZ64IEO7C26HZTCDY2T2PRYUFP6CSN2EOBJS7XAE4I7MXLHQ, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.769521375 (556h53m57.64060441s) [noise=true , address=145.79.227.12:4765 , piece_id=IESWSXDQFYKKHDOEHSYQSVMTBYLAJWNQ5TPWGNQPMLHO23CXYXTQ, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.770627170 (556h53m57.641710205s) [noise=true , address=77.90.31.13:4627 , piece_id=XB2K7RONOVEBPA2XGY36WIE57MWUUBKVAHEGXHAZWNDRL22WHH5Q, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.771369815 (556h53m57.64245285s) [noise=true , address=145.79.227.50:4375 , piece_id=DMDQHEH7XJT334MFCFAPNCGLSJ4UWE3U74MTDFROMTM4BDAI4IHA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.772492252 (556h53m57.643575297s) [noise=true , address=145.79.227.36:4519 , piece_id=YIPTLMBP7UV4GLPNA74M4343U4JEE7433ANXUHKYIJLZ36T25TXA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.773128333 (556h53m57.644211368s) [noise=true , address=77.90.31.17:4770 , piece_id=MH6LXF5JVTVI3U3N7W73U4ULOARVA6G2CRNUQLQ3V2UE6HUDGCUA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.775274971 (556h53m57.646358006s) [noise=true , address=145.79.227.40:4051 , piece_id=PZ3547GPNB3TRB24RBQVWLHLOC5JQVFUMG4ZZJDA3BBESZN3XFYA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.777926579 (556h53m57.649009615s) [noise=true , address=145.79.227.42:4123 , piece_id=TN4ZO3UUKIFRAP44SANXU6HKESFMIE45S5XTWAIKHJW5SA6C2KFA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.780387376 (556h53m57.651470421s) [noise=true , address=45.91.116.13:4155 , piece_id=HHINRA5KN56ZF2KW7IEG4BUGXM6UKMNUM3TM4Y3HFADVWASFEHFA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.781667764 (556h53m57.652750809s) [noise=true , address=77.90.31.15:4697 , piece_id=25QRUIDDPIZ7CSGKDAXLMZIWJHE6HHITH6FU4H2GA2GDBI4PR5KA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.783365686 (556h53m57.654448731s) [noise=true , address=77.90.31.11:4556 , piece_id=QRRGWDO7EHFVVQ6X4BTVY7HQRRDF5UCHKCFRLO2FOOYLTEDNJR2A, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.784872727 (556h53m57.655955762s) [noise=true , address=145.79.227.21:4473 , piece_id=OHS3FHPTXD3EKQNBJLLGKENQIJPKPGEFTVIRVQYXO3PZQEVQILGQ, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:59.482505035 (556h53m59.35358808s) [upload=488235]: Next returned nil segment Dec 28 20:22:59.482533810 (556h53m59.353616845s) [upload=488235]: Waiting for error group managing segments... Dec 28 20:22:59.769352743 (556h53m59.640435799s) [noise=true , address=145.79.227.18:4007 , piece_id=NB6ADK5TXELRFCYMKDF6C3QZRZQUL5XUPVQ7AVVYP32UY2GZA6SA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.782950737 (556h53m59.654033782s) [noise=true , address=145.79.227.51:4409 , piece_id=C342XC375VQVI23V5JESCTOES4AF66QWHIETEOBVV3KGRLIP2X5Q, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.789678698 (556h53m59.660761733s) [noise=true , address=145.79.227.12:4765 , piece_id=IESWSXDQFYKKHDOEHSYQSVMTBYLAJWNQ5TPWGNQPMLHO23CXYXTQ, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.802190252 (556h53m59.673273287s) [noise=true , address=145.79.227.42:4123 , piece_id=TN4ZO3UUKIFRAP44SANXU6HKESFMIE45S5XTWAIKHJW5SA6C2KFA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.804531330 (556h53m59.675614375s) [noise=true , address=145.79.227.50:4375 , piece_id=DMDQHEH7XJT334MFCFAPNCGLSJ4UWE3U74MTDFROMTM4BDAI4IHA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.810012928 (556h53m59.681095963s) [noise=true , address=145.79.227.21:4473 , piece_id=OHS3FHPTXD3EKQNBJLLGKENQIJPKPGEFTVIRVQYXO3PZQEVQILGQ, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.812050608 (556h53m59.683133643s) [noise=true , address=145.79.227.36:4519 , piece_id=YIPTLMBP7UV4GLPNA74M4343U4JEE7433ANXUHKYIJLZ36T25TXA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.812993363 (556h53m59.684076398s) [noise=true , address=145.79.227.41:4086 , piece_id=YRHLQZ64IEO7C26HZTCDY2T2PRYUFP6CSN2EOBJS7XAE4I7MXLHQ, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.946324830 (556h53m59.817407865s) [noise=true , address=77.90.31.13:4627 , piece_id=XB2K7RONOVEBPA2XGY36WIE57MWUUBKVAHEGXHAZWNDRL22WHH5Q, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.965566208 (556h53m59.836649243s) [noise=true , address=77.90.31.17:4770 , piece_id=MH6LXF5JVTVI3U3N7W73U4ULOARVA6G2CRNUQLQ3V2UE6HUDGCUA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:23:00.024382405 (556h53m59.89546544s) [noise=true , address=77.90.31.21:4880 , piece_id=V4PF2AQHKOK3GCF6B4UDNRJW7WYQ5QVOC66HUJMOPOOLPZAQIM4A, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:23:00.056351529 (556h53m59.927434574s) [noise=true , address=45.91.116.5:4297 , piece_id=FNX3RGGDGJZL5DOZRL2ZF2K2JHRAHWAWHIN6NQ4ONUKPVC6J4ECA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:23:00.067006559 (556h53m59.938089594s) [noise=true , address=45.91.116.13:4155 , piece_id=HHINRA5KN56ZF2KW7IEG4BUGXM6UKMNUM3TM4Y3HFADVWASFEHFA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:23:00.067025144 (556h53m59.938108179s) [seg_pos={0 0}, upload=488235]: Segment reached optimal threshold of 13 pieces. Dec 28 20:23:00.067214014 (556h53m59.938297059s) [noise=true , address=77.90.31.11:4556 , piece_id=QRRGWDO7EHFVVQ6X4BTVY7HQRRDF5UCHKCFRLO2FOOYLTEDNJR2A, seg_pos={0 0}, upload=488235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: write tcp 188.166.225.170:60580->77.90.31.11:4556: use of closed network connection; write tcp 188.166.225.170:60580->77.90.31.11:4556: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60580->77.90.31.11:4556: use of closed network connection Dec 28 20:23:00.067225536 (556h53m59.938308571s) [noise=true , address=77.90.31.11:4556 , piece_id=QRRGWDO7EHFVVQ6X4BTVY7HQRRDF5UCHKCFRLO2FOOYLTEDNJR2A, seg_pos={0 0}, upload=488235]: Long tail cancellation due to optimal threshold Dec 28 20:23:00.067306319 (556h53m59.938389364s) [noise=true , address=77.90.31.15:4697 , piece_id=25QRUIDDPIZ7CSGKDAXLMZIWJHE6HHITH6FU4H2GA2GDBI4PR5KA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Dec 28 20:23:00.067314354 (556h53m59.938397389s) [noise=true , address=77.90.31.15:4697 , piece_id=25QRUIDDPIZ7CSGKDAXLMZIWJHE6HHITH6FU4H2GA2GDBI4PR5KA, seg_pos={0 0}, upload=488235]: Long tail cancellation due to optimal threshold Dec 28 20:23:00.067582064 (556h53m59.938665119s) [noise=true , address=145.79.227.40:4051 , piece_id=PZ3547GPNB3TRB24RBQVWLHLOC5JQVFUMG4ZZJDA3BBESZN3XFYA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 28 20:23:00.067589408 (556h53m59.938672453s) [noise=true , address=145.79.227.40:4051 , piece_id=PZ3547GPNB3TRB24RBQVWLHLOC5JQVFUMG4ZZJDA3BBESZN3XFYA, seg_pos={0 0}, upload=488235]: Long tail cancellation due to optimal threshold Dec 28 20:23:00.067605620 (556h53m59.938688655s) [seg_pos={0 0}, upload=488235]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 20:23:00.067681614 (556h53m59.938764659s) [upload=488235]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 20:23:00.067686654 (556h53m59.938769699s) [upload=488235]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 20:23:00.354014782 (556h54m0.225097827s) [upload=488235]: Done uploading segments... Dec 28 20:23:00.354034600 (556h54m0.225117635s) [upload=488235]: Upload finished. err: <nil> Duration: 11.086715566s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [X..............X..........................X...........X.....] duration: 6.08714837s trace: https://jaeger.dev.storj.io/trace/3425d363be458a81 FAIL (2025-12-28 20:27:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: XB2K7RONOVEBPA2XGY36WIE57MWUUBKVAHEGXHAZWNDRL22WHH5Q) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: HHINRA5KN56ZF2KW7IEG4BUGXM6UKMNUM3TM4Y3HFADVWASFEHFA) context deadline exceeded ======LOGS====== Dec 28 20:22:57.518307420 (556h53m57.389390455s) [upload=488235]: Starting upload Dec 28 20:22:57.518369106 (556h53m57.389452182s) [upload=488235]: Done starting upload Dec 28 20:22:57.518427107 (556h53m57.389510152s) [upload=488235]: Uploading segments... Dec 28 20:22:57.518663427 (556h53m57.389746473s) [upload=488235]: Got next segment. Inline: false Dec 28 20:22:57.518709805 (556h53m57.38979284s) [upload=488235]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 20:22:57.518716318 (556h53m57.389799353s) [upload=488235]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 20:22:57.767659750 (556h53m57.638742815s) [seg_pos={0 0}, upload=488235]: Begin upload segment... Dec 28 20:22:57.767735904 (556h53m57.638818949s) [seg_pos={0 0}, upload=488235]: Done begin upload segment. Dec 28 20:22:57.767792252 (556h53m57.638875287s) [noise=true , address=77.90.31.21:4880 , piece_id=V4PF2AQHKOK3GCF6B4UDNRJW7WYQ5QVOC66HUJMOPOOLPZAQIM4A, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.767877795 (556h53m57.63896085s) [noise=true , address=145.79.227.51:4409 , piece_id=C342XC375VQVI23V5JESCTOES4AF66QWHIETEOBVV3KGRLIP2X5Q, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.768512193 (556h53m57.639595238s) [noise=true , address=145.79.227.18:4007 , piece_id=NB6ADK5TXELRFCYMKDF6C3QZRZQUL5XUPVQ7AVVYP32UY2GZA6SA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.768539074 (556h53m57.639622119s) [noise=true , address=45.91.116.5:4297 , piece_id=FNX3RGGDGJZL5DOZRL2ZF2K2JHRAHWAWHIN6NQ4ONUKPVC6J4ECA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.769129117 (556h53m57.640212152s) [noise=true , address=145.79.227.41:4086 , piece_id=YRHLQZ64IEO7C26HZTCDY2T2PRYUFP6CSN2EOBJS7XAE4I7MXLHQ, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.769521375 (556h53m57.64060441s) [noise=true , address=145.79.227.12:4765 , piece_id=IESWSXDQFYKKHDOEHSYQSVMTBYLAJWNQ5TPWGNQPMLHO23CXYXTQ, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.770627170 (556h53m57.641710205s) [noise=true , address=77.90.31.13:4627 , piece_id=XB2K7RONOVEBPA2XGY36WIE57MWUUBKVAHEGXHAZWNDRL22WHH5Q, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.771369815 (556h53m57.64245285s) [noise=true , address=145.79.227.50:4375 , piece_id=DMDQHEH7XJT334MFCFAPNCGLSJ4UWE3U74MTDFROMTM4BDAI4IHA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.772492252 (556h53m57.643575297s) [noise=true , address=145.79.227.36:4519 , piece_id=YIPTLMBP7UV4GLPNA74M4343U4JEE7433ANXUHKYIJLZ36T25TXA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.773128333 (556h53m57.644211368s) [noise=true , address=77.90.31.17:4770 , piece_id=MH6LXF5JVTVI3U3N7W73U4ULOARVA6G2CRNUQLQ3V2UE6HUDGCUA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.775274971 (556h53m57.646358006s) [noise=true , address=145.79.227.40:4051 , piece_id=PZ3547GPNB3TRB24RBQVWLHLOC5JQVFUMG4ZZJDA3BBESZN3XFYA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.777926579 (556h53m57.649009615s) [noise=true , address=145.79.227.42:4123 , piece_id=TN4ZO3UUKIFRAP44SANXU6HKESFMIE45S5XTWAIKHJW5SA6C2KFA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.780387376 (556h53m57.651470421s) [noise=true , address=45.91.116.13:4155 , piece_id=HHINRA5KN56ZF2KW7IEG4BUGXM6UKMNUM3TM4Y3HFADVWASFEHFA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.781667764 (556h53m57.652750809s) [noise=true , address=77.90.31.15:4697 , piece_id=25QRUIDDPIZ7CSGKDAXLMZIWJHE6HHITH6FU4H2GA2GDBI4PR5KA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.783365686 (556h53m57.654448731s) [noise=true , address=77.90.31.11:4556 , piece_id=QRRGWDO7EHFVVQ6X4BTVY7HQRRDF5UCHKCFRLO2FOOYLTEDNJR2A, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.784872727 (556h53m57.655955762s) [noise=true , address=145.79.227.21:4473 , piece_id=OHS3FHPTXD3EKQNBJLLGKENQIJPKPGEFTVIRVQYXO3PZQEVQILGQ, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:59.482505035 (556h53m59.35358808s) [upload=488235]: Next returned nil segment Dec 28 20:22:59.482533810 (556h53m59.353616845s) [upload=488235]: Waiting for error group managing segments... Dec 28 20:22:59.769352743 (556h53m59.640435799s) [noise=true , address=145.79.227.18:4007 , piece_id=NB6ADK5TXELRFCYMKDF6C3QZRZQUL5XUPVQ7AVVYP32UY2GZA6SA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.782950737 (556h53m59.654033782s) [noise=true , address=145.79.227.51:4409 , piece_id=C342XC375VQVI23V5JESCTOES4AF66QWHIETEOBVV3KGRLIP2X5Q, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.789678698 (556h53m59.660761733s) [noise=true , address=145.79.227.12:4765 , piece_id=IESWSXDQFYKKHDOEHSYQSVMTBYLAJWNQ5TPWGNQPMLHO23CXYXTQ, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.802190252 (556h53m59.673273287s) [noise=true , address=145.79.227.42:4123 , piece_id=TN4ZO3UUKIFRAP44SANXU6HKESFMIE45S5XTWAIKHJW5SA6C2KFA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.804531330 (556h53m59.675614375s) [noise=true , address=145.79.227.50:4375 , piece_id=DMDQHEH7XJT334MFCFAPNCGLSJ4UWE3U74MTDFROMTM4BDAI4IHA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.810012928 (556h53m59.681095963s) [noise=true , address=145.79.227.21:4473 , piece_id=OHS3FHPTXD3EKQNBJLLGKENQIJPKPGEFTVIRVQYXO3PZQEVQILGQ, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.812050608 (556h53m59.683133643s) [noise=true , address=145.79.227.36:4519 , piece_id=YIPTLMBP7UV4GLPNA74M4343U4JEE7433ANXUHKYIJLZ36T25TXA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.812993363 (556h53m59.684076398s) [noise=true , address=145.79.227.41:4086 , piece_id=YRHLQZ64IEO7C26HZTCDY2T2PRYUFP6CSN2EOBJS7XAE4I7MXLHQ, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.946324830 (556h53m59.817407865s) [noise=true , address=77.90.31.13:4627 , piece_id=XB2K7RONOVEBPA2XGY36WIE57MWUUBKVAHEGXHAZWNDRL22WHH5Q, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.965566208 (556h53m59.836649243s) [noise=true , address=77.90.31.17:4770 , piece_id=MH6LXF5JVTVI3U3N7W73U4ULOARVA6G2CRNUQLQ3V2UE6HUDGCUA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:23:00.024382405 (556h53m59.89546544s) [noise=true , address=77.90.31.21:4880 , piece_id=V4PF2AQHKOK3GCF6B4UDNRJW7WYQ5QVOC66HUJMOPOOLPZAQIM4A, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:23:00.056351529 (556h53m59.927434574s) [noise=true , address=45.91.116.5:4297 , piece_id=FNX3RGGDGJZL5DOZRL2ZF2K2JHRAHWAWHIN6NQ4ONUKPVC6J4ECA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:23:00.067006559 (556h53m59.938089594s) [noise=true , address=45.91.116.13:4155 , piece_id=HHINRA5KN56ZF2KW7IEG4BUGXM6UKMNUM3TM4Y3HFADVWASFEHFA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:23:00.067025144 (556h53m59.938108179s) [seg_pos={0 0}, upload=488235]: Segment reached optimal threshold of 13 pieces. Dec 28 20:23:00.067214014 (556h53m59.938297059s) [noise=true , address=77.90.31.11:4556 , piece_id=QRRGWDO7EHFVVQ6X4BTVY7HQRRDF5UCHKCFRLO2FOOYLTEDNJR2A, seg_pos={0 0}, upload=488235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: write tcp 188.166.225.170:60580->77.90.31.11:4556: use of closed network connection; write tcp 188.166.225.170:60580->77.90.31.11:4556: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60580->77.90.31.11:4556: use of closed network connection Dec 28 20:23:00.067225536 (556h53m59.938308571s) [noise=true , address=77.90.31.11:4556 , piece_id=QRRGWDO7EHFVVQ6X4BTVY7HQRRDF5UCHKCFRLO2FOOYLTEDNJR2A, seg_pos={0 0}, upload=488235]: Long tail cancellation due to optimal threshold Dec 28 20:23:00.067306319 (556h53m59.938389364s) [noise=true , address=77.90.31.15:4697 , piece_id=25QRUIDDPIZ7CSGKDAXLMZIWJHE6HHITH6FU4H2GA2GDBI4PR5KA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Dec 28 20:23:00.067314354 (556h53m59.938397389s) [noise=true , address=77.90.31.15:4697 , piece_id=25QRUIDDPIZ7CSGKDAXLMZIWJHE6HHITH6FU4H2GA2GDBI4PR5KA, seg_pos={0 0}, upload=488235]: Long tail cancellation due to optimal threshold Dec 28 20:23:00.067582064 (556h53m59.938665119s) [noise=true , address=145.79.227.40:4051 , piece_id=PZ3547GPNB3TRB24RBQVWLHLOC5JQVFUMG4ZZJDA3BBESZN3XFYA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 28 20:23:00.067589408 (556h53m59.938672453s) [noise=true , address=145.79.227.40:4051 , piece_id=PZ3547GPNB3TRB24RBQVWLHLOC5JQVFUMG4ZZJDA3BBESZN3XFYA, seg_pos={0 0}, upload=488235]: Long tail cancellation due to optimal threshold Dec 28 20:23:00.067605620 (556h53m59.938688655s) [seg_pos={0 0}, upload=488235]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 20:23:00.067681614 (556h53m59.938764659s) [upload=488235]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 20:23:00.067686654 (556h53m59.938769699s) [upload=488235]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 20:23:00.354014782 (556h54m0.225097827s) [upload=488235]: Done uploading segments... Dec 28 20:23:00.354034600 (556h54m0.225117635s) [upload=488235]: Upload finished. err: <nil> Duration: 6.087177294s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [XX..............X..........................X...........X....] duration: 6.021869935s trace: https://jaeger.dev.storj.io/trace/3425d363be458a81 FAIL (2025-12-28 20:26:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: XB2K7RONOVEBPA2XGY36WIE57MWUUBKVAHEGXHAZWNDRL22WHH5Q) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: HHINRA5KN56ZF2KW7IEG4BUGXM6UKMNUM3TM4Y3HFADVWASFEHFA) context deadline exceeded ======LOGS====== Dec 28 20:22:57.518307420 (556h53m57.389390455s) [upload=488235]: Starting upload Dec 28 20:22:57.518369106 (556h53m57.389452182s) [upload=488235]: Done starting upload Dec 28 20:22:57.518427107 (556h53m57.389510152s) [upload=488235]: Uploading segments... Dec 28 20:22:57.518663427 (556h53m57.389746473s) [upload=488235]: Got next segment. Inline: false Dec 28 20:22:57.518709805 (556h53m57.38979284s) [upload=488235]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 20:22:57.518716318 (556h53m57.389799353s) [upload=488235]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 20:22:57.767659750 (556h53m57.638742815s) [seg_pos={0 0}, upload=488235]: Begin upload segment... Dec 28 20:22:57.767735904 (556h53m57.638818949s) [seg_pos={0 0}, upload=488235]: Done begin upload segment. Dec 28 20:22:57.767792252 (556h53m57.638875287s) [noise=true , address=77.90.31.21:4880 , piece_id=V4PF2AQHKOK3GCF6B4UDNRJW7WYQ5QVOC66HUJMOPOOLPZAQIM4A, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.767877795 (556h53m57.63896085s) [noise=true , address=145.79.227.51:4409 , piece_id=C342XC375VQVI23V5JESCTOES4AF66QWHIETEOBVV3KGRLIP2X5Q, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.768512193 (556h53m57.639595238s) [noise=true , address=145.79.227.18:4007 , piece_id=NB6ADK5TXELRFCYMKDF6C3QZRZQUL5XUPVQ7AVVYP32UY2GZA6SA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.768539074 (556h53m57.639622119s) [noise=true , address=45.91.116.5:4297 , piece_id=FNX3RGGDGJZL5DOZRL2ZF2K2JHRAHWAWHIN6NQ4ONUKPVC6J4ECA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.769129117 (556h53m57.640212152s) [noise=true , address=145.79.227.41:4086 , piece_id=YRHLQZ64IEO7C26HZTCDY2T2PRYUFP6CSN2EOBJS7XAE4I7MXLHQ, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.769521375 (556h53m57.64060441s) [noise=true , address=145.79.227.12:4765 , piece_id=IESWSXDQFYKKHDOEHSYQSVMTBYLAJWNQ5TPWGNQPMLHO23CXYXTQ, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.770627170 (556h53m57.641710205s) [noise=true , address=77.90.31.13:4627 , piece_id=XB2K7RONOVEBPA2XGY36WIE57MWUUBKVAHEGXHAZWNDRL22WHH5Q, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.771369815 (556h53m57.64245285s) [noise=true , address=145.79.227.50:4375 , piece_id=DMDQHEH7XJT334MFCFAPNCGLSJ4UWE3U74MTDFROMTM4BDAI4IHA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.772492252 (556h53m57.643575297s) [noise=true , address=145.79.227.36:4519 , piece_id=YIPTLMBP7UV4GLPNA74M4343U4JEE7433ANXUHKYIJLZ36T25TXA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.773128333 (556h53m57.644211368s) [noise=true , address=77.90.31.17:4770 , piece_id=MH6LXF5JVTVI3U3N7W73U4ULOARVA6G2CRNUQLQ3V2UE6HUDGCUA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.775274971 (556h53m57.646358006s) [noise=true , address=145.79.227.40:4051 , piece_id=PZ3547GPNB3TRB24RBQVWLHLOC5JQVFUMG4ZZJDA3BBESZN3XFYA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.777926579 (556h53m57.649009615s) [noise=true , address=145.79.227.42:4123 , piece_id=TN4ZO3UUKIFRAP44SANXU6HKESFMIE45S5XTWAIKHJW5SA6C2KFA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.780387376 (556h53m57.651470421s) [noise=true , address=45.91.116.13:4155 , piece_id=HHINRA5KN56ZF2KW7IEG4BUGXM6UKMNUM3TM4Y3HFADVWASFEHFA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.781667764 (556h53m57.652750809s) [noise=true , address=77.90.31.15:4697 , piece_id=25QRUIDDPIZ7CSGKDAXLMZIWJHE6HHITH6FU4H2GA2GDBI4PR5KA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.783365686 (556h53m57.654448731s) [noise=true , address=77.90.31.11:4556 , piece_id=QRRGWDO7EHFVVQ6X4BTVY7HQRRDF5UCHKCFRLO2FOOYLTEDNJR2A, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.784872727 (556h53m57.655955762s) [noise=true , address=145.79.227.21:4473 , piece_id=OHS3FHPTXD3EKQNBJLLGKENQIJPKPGEFTVIRVQYXO3PZQEVQILGQ, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:59.482505035 (556h53m59.35358808s) [upload=488235]: Next returned nil segment Dec 28 20:22:59.482533810 (556h53m59.353616845s) [upload=488235]: Waiting for error group managing segments... Dec 28 20:22:59.769352743 (556h53m59.640435799s) [noise=true , address=145.79.227.18:4007 , piece_id=NB6ADK5TXELRFCYMKDF6C3QZRZQUL5XUPVQ7AVVYP32UY2GZA6SA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.782950737 (556h53m59.654033782s) [noise=true , address=145.79.227.51:4409 , piece_id=C342XC375VQVI23V5JESCTOES4AF66QWHIETEOBVV3KGRLIP2X5Q, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.789678698 (556h53m59.660761733s) [noise=true , address=145.79.227.12:4765 , piece_id=IESWSXDQFYKKHDOEHSYQSVMTBYLAJWNQ5TPWGNQPMLHO23CXYXTQ, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.802190252 (556h53m59.673273287s) [noise=true , address=145.79.227.42:4123 , piece_id=TN4ZO3UUKIFRAP44SANXU6HKESFMIE45S5XTWAIKHJW5SA6C2KFA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.804531330 (556h53m59.675614375s) [noise=true , address=145.79.227.50:4375 , piece_id=DMDQHEH7XJT334MFCFAPNCGLSJ4UWE3U74MTDFROMTM4BDAI4IHA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.810012928 (556h53m59.681095963s) [noise=true , address=145.79.227.21:4473 , piece_id=OHS3FHPTXD3EKQNBJLLGKENQIJPKPGEFTVIRVQYXO3PZQEVQILGQ, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.812050608 (556h53m59.683133643s) [noise=true , address=145.79.227.36:4519 , piece_id=YIPTLMBP7UV4GLPNA74M4343U4JEE7433ANXUHKYIJLZ36T25TXA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.812993363 (556h53m59.684076398s) [noise=true , address=145.79.227.41:4086 , piece_id=YRHLQZ64IEO7C26HZTCDY2T2PRYUFP6CSN2EOBJS7XAE4I7MXLHQ, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.946324830 (556h53m59.817407865s) [noise=true , address=77.90.31.13:4627 , piece_id=XB2K7RONOVEBPA2XGY36WIE57MWUUBKVAHEGXHAZWNDRL22WHH5Q, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.965566208 (556h53m59.836649243s) [noise=true , address=77.90.31.17:4770 , piece_id=MH6LXF5JVTVI3U3N7W73U4ULOARVA6G2CRNUQLQ3V2UE6HUDGCUA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:23:00.024382405 (556h53m59.89546544s) [noise=true , address=77.90.31.21:4880 , piece_id=V4PF2AQHKOK3GCF6B4UDNRJW7WYQ5QVOC66HUJMOPOOLPZAQIM4A, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:23:00.056351529 (556h53m59.927434574s) [noise=true , address=45.91.116.5:4297 , piece_id=FNX3RGGDGJZL5DOZRL2ZF2K2JHRAHWAWHIN6NQ4ONUKPVC6J4ECA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:23:00.067006559 (556h53m59.938089594s) [noise=true , address=45.91.116.13:4155 , piece_id=HHINRA5KN56ZF2KW7IEG4BUGXM6UKMNUM3TM4Y3HFADVWASFEHFA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:23:00.067025144 (556h53m59.938108179s) [seg_pos={0 0}, upload=488235]: Segment reached optimal threshold of 13 pieces. Dec 28 20:23:00.067214014 (556h53m59.938297059s) [noise=true , address=77.90.31.11:4556 , piece_id=QRRGWDO7EHFVVQ6X4BTVY7HQRRDF5UCHKCFRLO2FOOYLTEDNJR2A, seg_pos={0 0}, upload=488235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: write tcp 188.166.225.170:60580->77.90.31.11:4556: use of closed network connection; write tcp 188.166.225.170:60580->77.90.31.11:4556: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60580->77.90.31.11:4556: use of closed network connection Dec 28 20:23:00.067225536 (556h53m59.938308571s) [noise=true , address=77.90.31.11:4556 , piece_id=QRRGWDO7EHFVVQ6X4BTVY7HQRRDF5UCHKCFRLO2FOOYLTEDNJR2A, seg_pos={0 0}, upload=488235]: Long tail cancellation due to optimal threshold Dec 28 20:23:00.067306319 (556h53m59.938389364s) [noise=true , address=77.90.31.15:4697 , piece_id=25QRUIDDPIZ7CSGKDAXLMZIWJHE6HHITH6FU4H2GA2GDBI4PR5KA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Dec 28 20:23:00.067314354 (556h53m59.938397389s) [noise=true , address=77.90.31.15:4697 , piece_id=25QRUIDDPIZ7CSGKDAXLMZIWJHE6HHITH6FU4H2GA2GDBI4PR5KA, seg_pos={0 0}, upload=488235]: Long tail cancellation due to optimal threshold Dec 28 20:23:00.067582064 (556h53m59.938665119s) [noise=true , address=145.79.227.40:4051 , piece_id=PZ3547GPNB3TRB24RBQVWLHLOC5JQVFUMG4ZZJDA3BBESZN3XFYA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 28 20:23:00.067589408 (556h53m59.938672453s) [noise=true , address=145.79.227.40:4051 , piece_id=PZ3547GPNB3TRB24RBQVWLHLOC5JQVFUMG4ZZJDA3BBESZN3XFYA, seg_pos={0 0}, upload=488235]: Long tail cancellation due to optimal threshold Dec 28 20:23:00.067605620 (556h53m59.938688655s) [seg_pos={0 0}, upload=488235]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 20:23:00.067681614 (556h53m59.938764659s) [upload=488235]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 20:23:00.067686654 (556h53m59.938769699s) [upload=488235]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 20:23:00.354014782 (556h54m0.225097827s) [upload=488235]: Done uploading segments... Dec 28 20:23:00.354034600 (556h54m0.225117635s) [upload=488235]: Upload finished. err: <nil> Duration: 6.021888019s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.XX..............X..........................X...........X...] duration: 6.042530476s trace: https://jaeger.dev.storj.io/trace/3425d363be458a81 FAIL (2025-12-28 20:25:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: XB2K7RONOVEBPA2XGY36WIE57MWUUBKVAHEGXHAZWNDRL22WHH5Q) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: HHINRA5KN56ZF2KW7IEG4BUGXM6UKMNUM3TM4Y3HFADVWASFEHFA) context deadline exceeded ======LOGS====== Dec 28 20:22:57.518307420 (556h53m57.389390455s) [upload=488235]: Starting upload Dec 28 20:22:57.518369106 (556h53m57.389452182s) [upload=488235]: Done starting upload Dec 28 20:22:57.518427107 (556h53m57.389510152s) [upload=488235]: Uploading segments... Dec 28 20:22:57.518663427 (556h53m57.389746473s) [upload=488235]: Got next segment. Inline: false Dec 28 20:22:57.518709805 (556h53m57.38979284s) [upload=488235]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 20:22:57.518716318 (556h53m57.389799353s) [upload=488235]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 20:22:57.767659750 (556h53m57.638742815s) [seg_pos={0 0}, upload=488235]: Begin upload segment... Dec 28 20:22:57.767735904 (556h53m57.638818949s) [seg_pos={0 0}, upload=488235]: Done begin upload segment. Dec 28 20:22:57.767792252 (556h53m57.638875287s) [noise=true , address=77.90.31.21:4880 , piece_id=V4PF2AQHKOK3GCF6B4UDNRJW7WYQ5QVOC66HUJMOPOOLPZAQIM4A, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.767877795 (556h53m57.63896085s) [noise=true , address=145.79.227.51:4409 , piece_id=C342XC375VQVI23V5JESCTOES4AF66QWHIETEOBVV3KGRLIP2X5Q, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.768512193 (556h53m57.639595238s) [noise=true , address=145.79.227.18:4007 , piece_id=NB6ADK5TXELRFCYMKDF6C3QZRZQUL5XUPVQ7AVVYP32UY2GZA6SA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.768539074 (556h53m57.639622119s) [noise=true , address=45.91.116.5:4297 , piece_id=FNX3RGGDGJZL5DOZRL2ZF2K2JHRAHWAWHIN6NQ4ONUKPVC6J4ECA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.769129117 (556h53m57.640212152s) [noise=true , address=145.79.227.41:4086 , piece_id=YRHLQZ64IEO7C26HZTCDY2T2PRYUFP6CSN2EOBJS7XAE4I7MXLHQ, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.769521375 (556h53m57.64060441s) [noise=true , address=145.79.227.12:4765 , piece_id=IESWSXDQFYKKHDOEHSYQSVMTBYLAJWNQ5TPWGNQPMLHO23CXYXTQ, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.770627170 (556h53m57.641710205s) [noise=true , address=77.90.31.13:4627 , piece_id=XB2K7RONOVEBPA2XGY36WIE57MWUUBKVAHEGXHAZWNDRL22WHH5Q, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.771369815 (556h53m57.64245285s) [noise=true , address=145.79.227.50:4375 , piece_id=DMDQHEH7XJT334MFCFAPNCGLSJ4UWE3U74MTDFROMTM4BDAI4IHA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.772492252 (556h53m57.643575297s) [noise=true , address=145.79.227.36:4519 , piece_id=YIPTLMBP7UV4GLPNA74M4343U4JEE7433ANXUHKYIJLZ36T25TXA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.773128333 (556h53m57.644211368s) [noise=true , address=77.90.31.17:4770 , piece_id=MH6LXF5JVTVI3U3N7W73U4ULOARVA6G2CRNUQLQ3V2UE6HUDGCUA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.775274971 (556h53m57.646358006s) [noise=true , address=145.79.227.40:4051 , piece_id=PZ3547GPNB3TRB24RBQVWLHLOC5JQVFUMG4ZZJDA3BBESZN3XFYA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.777926579 (556h53m57.649009615s) [noise=true , address=145.79.227.42:4123 , piece_id=TN4ZO3UUKIFRAP44SANXU6HKESFMIE45S5XTWAIKHJW5SA6C2KFA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.780387376 (556h53m57.651470421s) [noise=true , address=45.91.116.13:4155 , piece_id=HHINRA5KN56ZF2KW7IEG4BUGXM6UKMNUM3TM4Y3HFADVWASFEHFA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.781667764 (556h53m57.652750809s) [noise=true , address=77.90.31.15:4697 , piece_id=25QRUIDDPIZ7CSGKDAXLMZIWJHE6HHITH6FU4H2GA2GDBI4PR5KA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.783365686 (556h53m57.654448731s) [noise=true , address=77.90.31.11:4556 , piece_id=QRRGWDO7EHFVVQ6X4BTVY7HQRRDF5UCHKCFRLO2FOOYLTEDNJR2A, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.784872727 (556h53m57.655955762s) [noise=true , address=145.79.227.21:4473 , piece_id=OHS3FHPTXD3EKQNBJLLGKENQIJPKPGEFTVIRVQYXO3PZQEVQILGQ, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:59.482505035 (556h53m59.35358808s) [upload=488235]: Next returned nil segment Dec 28 20:22:59.482533810 (556h53m59.353616845s) [upload=488235]: Waiting for error group managing segments... Dec 28 20:22:59.769352743 (556h53m59.640435799s) [noise=true , address=145.79.227.18:4007 , piece_id=NB6ADK5TXELRFCYMKDF6C3QZRZQUL5XUPVQ7AVVYP32UY2GZA6SA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.782950737 (556h53m59.654033782s) [noise=true , address=145.79.227.51:4409 , piece_id=C342XC375VQVI23V5JESCTOES4AF66QWHIETEOBVV3KGRLIP2X5Q, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.789678698 (556h53m59.660761733s) [noise=true , address=145.79.227.12:4765 , piece_id=IESWSXDQFYKKHDOEHSYQSVMTBYLAJWNQ5TPWGNQPMLHO23CXYXTQ, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.802190252 (556h53m59.673273287s) [noise=true , address=145.79.227.42:4123 , piece_id=TN4ZO3UUKIFRAP44SANXU6HKESFMIE45S5XTWAIKHJW5SA6C2KFA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.804531330 (556h53m59.675614375s) [noise=true , address=145.79.227.50:4375 , piece_id=DMDQHEH7XJT334MFCFAPNCGLSJ4UWE3U74MTDFROMTM4BDAI4IHA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.810012928 (556h53m59.681095963s) [noise=true , address=145.79.227.21:4473 , piece_id=OHS3FHPTXD3EKQNBJLLGKENQIJPKPGEFTVIRVQYXO3PZQEVQILGQ, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.812050608 (556h53m59.683133643s) [noise=true , address=145.79.227.36:4519 , piece_id=YIPTLMBP7UV4GLPNA74M4343U4JEE7433ANXUHKYIJLZ36T25TXA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.812993363 (556h53m59.684076398s) [noise=true , address=145.79.227.41:4086 , piece_id=YRHLQZ64IEO7C26HZTCDY2T2PRYUFP6CSN2EOBJS7XAE4I7MXLHQ, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.946324830 (556h53m59.817407865s) [noise=true , address=77.90.31.13:4627 , piece_id=XB2K7RONOVEBPA2XGY36WIE57MWUUBKVAHEGXHAZWNDRL22WHH5Q, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.965566208 (556h53m59.836649243s) [noise=true , address=77.90.31.17:4770 , piece_id=MH6LXF5JVTVI3U3N7W73U4ULOARVA6G2CRNUQLQ3V2UE6HUDGCUA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:23:00.024382405 (556h53m59.89546544s) [noise=true , address=77.90.31.21:4880 , piece_id=V4PF2AQHKOK3GCF6B4UDNRJW7WYQ5QVOC66HUJMOPOOLPZAQIM4A, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:23:00.056351529 (556h53m59.927434574s) [noise=true , address=45.91.116.5:4297 , piece_id=FNX3RGGDGJZL5DOZRL2ZF2K2JHRAHWAWHIN6NQ4ONUKPVC6J4ECA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:23:00.067006559 (556h53m59.938089594s) [noise=true , address=45.91.116.13:4155 , piece_id=HHINRA5KN56ZF2KW7IEG4BUGXM6UKMNUM3TM4Y3HFADVWASFEHFA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:23:00.067025144 (556h53m59.938108179s) [seg_pos={0 0}, upload=488235]: Segment reached optimal threshold of 13 pieces. Dec 28 20:23:00.067214014 (556h53m59.938297059s) [noise=true , address=77.90.31.11:4556 , piece_id=QRRGWDO7EHFVVQ6X4BTVY7HQRRDF5UCHKCFRLO2FOOYLTEDNJR2A, seg_pos={0 0}, upload=488235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: write tcp 188.166.225.170:60580->77.90.31.11:4556: use of closed network connection; write tcp 188.166.225.170:60580->77.90.31.11:4556: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60580->77.90.31.11:4556: use of closed network connection Dec 28 20:23:00.067225536 (556h53m59.938308571s) [noise=true , address=77.90.31.11:4556 , piece_id=QRRGWDO7EHFVVQ6X4BTVY7HQRRDF5UCHKCFRLO2FOOYLTEDNJR2A, seg_pos={0 0}, upload=488235]: Long tail cancellation due to optimal threshold Dec 28 20:23:00.067306319 (556h53m59.938389364s) [noise=true , address=77.90.31.15:4697 , piece_id=25QRUIDDPIZ7CSGKDAXLMZIWJHE6HHITH6FU4H2GA2GDBI4PR5KA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Dec 28 20:23:00.067314354 (556h53m59.938397389s) [noise=true , address=77.90.31.15:4697 , piece_id=25QRUIDDPIZ7CSGKDAXLMZIWJHE6HHITH6FU4H2GA2GDBI4PR5KA, seg_pos={0 0}, upload=488235]: Long tail cancellation due to optimal threshold Dec 28 20:23:00.067582064 (556h53m59.938665119s) [noise=true , address=145.79.227.40:4051 , piece_id=PZ3547GPNB3TRB24RBQVWLHLOC5JQVFUMG4ZZJDA3BBESZN3XFYA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 28 20:23:00.067589408 (556h53m59.938672453s) [noise=true , address=145.79.227.40:4051 , piece_id=PZ3547GPNB3TRB24RBQVWLHLOC5JQVFUMG4ZZJDA3BBESZN3XFYA, seg_pos={0 0}, upload=488235]: Long tail cancellation due to optimal threshold Dec 28 20:23:00.067605620 (556h53m59.938688655s) [seg_pos={0 0}, upload=488235]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 20:23:00.067681614 (556h53m59.938764659s) [upload=488235]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 20:23:00.067686654 (556h53m59.938769699s) [upload=488235]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 20:23:00.354014782 (556h54m0.225097827s) [upload=488235]: Done uploading segments... Dec 28 20:23:00.354034600 (556h54m0.225117635s) [upload=488235]: Upload finished. err: <nil> Duration: 6.042598707s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..XX..............X..........................X...........X..] duration: 23.17607787s trace: https://jaeger.dev.storj.io/trace/3425d363be458a81 FAIL (2025-12-28 20:24:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: XB2K7RONOVEBPA2XGY36WIE57MWUUBKVAHEGXHAZWNDRL22WHH5Q) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: HHINRA5KN56ZF2KW7IEG4BUGXM6UKMNUM3TM4Y3HFADVWASFEHFA) context deadline exceeded ======LOGS====== Dec 28 20:22:57.518307420 (556h53m57.389390455s) [upload=488235]: Starting upload Dec 28 20:22:57.518369106 (556h53m57.389452182s) [upload=488235]: Done starting upload Dec 28 20:22:57.518427107 (556h53m57.389510152s) [upload=488235]: Uploading segments... Dec 28 20:22:57.518663427 (556h53m57.389746473s) [upload=488235]: Got next segment. Inline: false Dec 28 20:22:57.518709805 (556h53m57.38979284s) [upload=488235]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 20:22:57.518716318 (556h53m57.389799353s) [upload=488235]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 20:22:57.767659750 (556h53m57.638742815s) [seg_pos={0 0}, upload=488235]: Begin upload segment... Dec 28 20:22:57.767735904 (556h53m57.638818949s) [seg_pos={0 0}, upload=488235]: Done begin upload segment. Dec 28 20:22:57.767792252 (556h53m57.638875287s) [noise=true , address=77.90.31.21:4880 , piece_id=V4PF2AQHKOK3GCF6B4UDNRJW7WYQ5QVOC66HUJMOPOOLPZAQIM4A, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.767877795 (556h53m57.63896085s) [noise=true , address=145.79.227.51:4409 , piece_id=C342XC375VQVI23V5JESCTOES4AF66QWHIETEOBVV3KGRLIP2X5Q, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.768512193 (556h53m57.639595238s) [noise=true , address=145.79.227.18:4007 , piece_id=NB6ADK5TXELRFCYMKDF6C3QZRZQUL5XUPVQ7AVVYP32UY2GZA6SA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.768539074 (556h53m57.639622119s) [noise=true , address=45.91.116.5:4297 , piece_id=FNX3RGGDGJZL5DOZRL2ZF2K2JHRAHWAWHIN6NQ4ONUKPVC6J4ECA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.769129117 (556h53m57.640212152s) [noise=true , address=145.79.227.41:4086 , piece_id=YRHLQZ64IEO7C26HZTCDY2T2PRYUFP6CSN2EOBJS7XAE4I7MXLHQ, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.769521375 (556h53m57.64060441s) [noise=true , address=145.79.227.12:4765 , piece_id=IESWSXDQFYKKHDOEHSYQSVMTBYLAJWNQ5TPWGNQPMLHO23CXYXTQ, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.770627170 (556h53m57.641710205s) [noise=true , address=77.90.31.13:4627 , piece_id=XB2K7RONOVEBPA2XGY36WIE57MWUUBKVAHEGXHAZWNDRL22WHH5Q, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.771369815 (556h53m57.64245285s) [noise=true , address=145.79.227.50:4375 , piece_id=DMDQHEH7XJT334MFCFAPNCGLSJ4UWE3U74MTDFROMTM4BDAI4IHA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.772492252 (556h53m57.643575297s) [noise=true , address=145.79.227.36:4519 , piece_id=YIPTLMBP7UV4GLPNA74M4343U4JEE7433ANXUHKYIJLZ36T25TXA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.773128333 (556h53m57.644211368s) [noise=true , address=77.90.31.17:4770 , piece_id=MH6LXF5JVTVI3U3N7W73U4ULOARVA6G2CRNUQLQ3V2UE6HUDGCUA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.775274971 (556h53m57.646358006s) [noise=true , address=145.79.227.40:4051 , piece_id=PZ3547GPNB3TRB24RBQVWLHLOC5JQVFUMG4ZZJDA3BBESZN3XFYA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.777926579 (556h53m57.649009615s) [noise=true , address=145.79.227.42:4123 , piece_id=TN4ZO3UUKIFRAP44SANXU6HKESFMIE45S5XTWAIKHJW5SA6C2KFA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.780387376 (556h53m57.651470421s) [noise=true , address=45.91.116.13:4155 , piece_id=HHINRA5KN56ZF2KW7IEG4BUGXM6UKMNUM3TM4Y3HFADVWASFEHFA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.781667764 (556h53m57.652750809s) [noise=true , address=77.90.31.15:4697 , piece_id=25QRUIDDPIZ7CSGKDAXLMZIWJHE6HHITH6FU4H2GA2GDBI4PR5KA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.783365686 (556h53m57.654448731s) [noise=true , address=77.90.31.11:4556 , piece_id=QRRGWDO7EHFVVQ6X4BTVY7HQRRDF5UCHKCFRLO2FOOYLTEDNJR2A, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.784872727 (556h53m57.655955762s) [noise=true , address=145.79.227.21:4473 , piece_id=OHS3FHPTXD3EKQNBJLLGKENQIJPKPGEFTVIRVQYXO3PZQEVQILGQ, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:59.482505035 (556h53m59.35358808s) [upload=488235]: Next returned nil segment Dec 28 20:22:59.482533810 (556h53m59.353616845s) [upload=488235]: Waiting for error group managing segments... Dec 28 20:22:59.769352743 (556h53m59.640435799s) [noise=true , address=145.79.227.18:4007 , piece_id=NB6ADK5TXELRFCYMKDF6C3QZRZQUL5XUPVQ7AVVYP32UY2GZA6SA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.782950737 (556h53m59.654033782s) [noise=true , address=145.79.227.51:4409 , piece_id=C342XC375VQVI23V5JESCTOES4AF66QWHIETEOBVV3KGRLIP2X5Q, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.789678698 (556h53m59.660761733s) [noise=true , address=145.79.227.12:4765 , piece_id=IESWSXDQFYKKHDOEHSYQSVMTBYLAJWNQ5TPWGNQPMLHO23CXYXTQ, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.802190252 (556h53m59.673273287s) [noise=true , address=145.79.227.42:4123 , piece_id=TN4ZO3UUKIFRAP44SANXU6HKESFMIE45S5XTWAIKHJW5SA6C2KFA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.804531330 (556h53m59.675614375s) [noise=true , address=145.79.227.50:4375 , piece_id=DMDQHEH7XJT334MFCFAPNCGLSJ4UWE3U74MTDFROMTM4BDAI4IHA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.810012928 (556h53m59.681095963s) [noise=true , address=145.79.227.21:4473 , piece_id=OHS3FHPTXD3EKQNBJLLGKENQIJPKPGEFTVIRVQYXO3PZQEVQILGQ, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.812050608 (556h53m59.683133643s) [noise=true , address=145.79.227.36:4519 , piece_id=YIPTLMBP7UV4GLPNA74M4343U4JEE7433ANXUHKYIJLZ36T25TXA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.812993363 (556h53m59.684076398s) [noise=true , address=145.79.227.41:4086 , piece_id=YRHLQZ64IEO7C26HZTCDY2T2PRYUFP6CSN2EOBJS7XAE4I7MXLHQ, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.946324830 (556h53m59.817407865s) [noise=true , address=77.90.31.13:4627 , piece_id=XB2K7RONOVEBPA2XGY36WIE57MWUUBKVAHEGXHAZWNDRL22WHH5Q, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.965566208 (556h53m59.836649243s) [noise=true , address=77.90.31.17:4770 , piece_id=MH6LXF5JVTVI3U3N7W73U4ULOARVA6G2CRNUQLQ3V2UE6HUDGCUA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:23:00.024382405 (556h53m59.89546544s) [noise=true , address=77.90.31.21:4880 , piece_id=V4PF2AQHKOK3GCF6B4UDNRJW7WYQ5QVOC66HUJMOPOOLPZAQIM4A, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:23:00.056351529 (556h53m59.927434574s) [noise=true , address=45.91.116.5:4297 , piece_id=FNX3RGGDGJZL5DOZRL2ZF2K2JHRAHWAWHIN6NQ4ONUKPVC6J4ECA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:23:00.067006559 (556h53m59.938089594s) [noise=true , address=45.91.116.13:4155 , piece_id=HHINRA5KN56ZF2KW7IEG4BUGXM6UKMNUM3TM4Y3HFADVWASFEHFA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:23:00.067025144 (556h53m59.938108179s) [seg_pos={0 0}, upload=488235]: Segment reached optimal threshold of 13 pieces. Dec 28 20:23:00.067214014 (556h53m59.938297059s) [noise=true , address=77.90.31.11:4556 , piece_id=QRRGWDO7EHFVVQ6X4BTVY7HQRRDF5UCHKCFRLO2FOOYLTEDNJR2A, seg_pos={0 0}, upload=488235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: write tcp 188.166.225.170:60580->77.90.31.11:4556: use of closed network connection; write tcp 188.166.225.170:60580->77.90.31.11:4556: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60580->77.90.31.11:4556: use of closed network connection Dec 28 20:23:00.067225536 (556h53m59.938308571s) [noise=true , address=77.90.31.11:4556 , piece_id=QRRGWDO7EHFVVQ6X4BTVY7HQRRDF5UCHKCFRLO2FOOYLTEDNJR2A, seg_pos={0 0}, upload=488235]: Long tail cancellation due to optimal threshold Dec 28 20:23:00.067306319 (556h53m59.938389364s) [noise=true , address=77.90.31.15:4697 , piece_id=25QRUIDDPIZ7CSGKDAXLMZIWJHE6HHITH6FU4H2GA2GDBI4PR5KA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Dec 28 20:23:00.067314354 (556h53m59.938397389s) [noise=true , address=77.90.31.15:4697 , piece_id=25QRUIDDPIZ7CSGKDAXLMZIWJHE6HHITH6FU4H2GA2GDBI4PR5KA, seg_pos={0 0}, upload=488235]: Long tail cancellation due to optimal threshold Dec 28 20:23:00.067582064 (556h53m59.938665119s) [noise=true , address=145.79.227.40:4051 , piece_id=PZ3547GPNB3TRB24RBQVWLHLOC5JQVFUMG4ZZJDA3BBESZN3XFYA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 28 20:23:00.067589408 (556h53m59.938672453s) [noise=true , address=145.79.227.40:4051 , piece_id=PZ3547GPNB3TRB24RBQVWLHLOC5JQVFUMG4ZZJDA3BBESZN3XFYA, seg_pos={0 0}, upload=488235]: Long tail cancellation due to optimal threshold Dec 28 20:23:00.067605620 (556h53m59.938688655s) [seg_pos={0 0}, upload=488235]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 20:23:00.067681614 (556h53m59.938764659s) [upload=488235]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 20:23:00.067686654 (556h53m59.938769699s) [upload=488235]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 20:23:00.354014782 (556h54m0.225097827s) [upload=488235]: Done uploading segments... Dec 28 20:23:00.354034600 (556h54m0.225117635s) [upload=488235]: Upload finished. err: <nil> Duration: 23.176109361s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 21.965806753s trace: https://jaeger.dev.storj.io/trace/10e1641b228b2a1f FAIL (2025-12-28 20:24:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 40:05.744568479 (556h11m5.615651514s) [noise=true , address=145.79.227.28:4744 , piece_id=42LII6EOG3CMBTIL5I4RZQPDYU52TKJN3PGWEQXROBYMNDFYIIVA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.745099351 (556h11m5.616182396s) [noise=true , address=145.79.227.23:4550 , piece_id=NWIC722LZUHJSAMPWSKCLPRIDGWPVUEYMS5KRAMX3IIMD2KR4XGQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.745777401 (556h11m5.616860436s) [noise=true , address=77.90.31.15:4722 , piece_id=AF7F2OA247CJIDFTEU6HEOIYGOINEYPMII53BKJQXI7UZ26MJIUA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.746374739 (556h11m5.617457784s) [noise=true , address=77.90.31.10:4522 , piece_id=JOHOMHCNYXF4XQGLN2NJUSWSUO3XFBRRMYD27SLIZAK2RCT3ZC2A, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.746934555 (556h11m5.618017601s) [noise=true , address=77.90.31.21:4896 , piece_id=ATTC7BZUT4HAAEAZCHFZQE4INV2LLSWR2YCUNVA73LPJCW3YL35A, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.747556861 (556h11m5.618640146s) [noise=true , address=64.49.13.26:4956 , piece_id=FFHUPOOM7PLEABRHFH6H2SI6WXJBU2XFCLZJUBV4CXUJ6EP4PI2Q, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.748964150 (556h11m5.620047195s) [noise=true , address=145.79.227.34:4916 , piece_id=SPYSFWMJ752FC5JGRE36G54S2FHG4DQFCV6CIZONQHJDXDCN52VQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.751119314 (556h11m5.622202359s) [noise=true , address=77.90.31.18:4816 , piece_id=YT3PSYNQTDQBVBRUDNOXJ2XCN6EFQA54WBSNL6MZMXFXR7XNCHIA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.753413222 (556h11m5.624496257s) [noise=true , address=77.90.31.11:4582 , piece_id=ZRE24R27TX4E5DTV2POX7EJANQ4CM75D6ZOUS6SKM3QHHQCKRBRA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.755230482 (556h11m5.626313527s) [noise=true , address=64.49.13.154:4932 , piece_id=QB5WM5RZ2P2GC6RIGGM34IANVEI2SNDJ2PO556JXKTDBX5HLD4YA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.757598362 (556h11m5.628681407s) [noise=true , address=145.79.227.35:4510 , piece_id=P275TU367DTYXBQOGGV5THFUPHJUWGSM5IMJQ753VE4MX4IPOONQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.759644679 (556h11m5.630727734s) [noise=true , address=145.79.227.46:4626 , piece_id=OQOTWYLVIU7GJBBLYWQOEEFKR3TNAUOXZS6UAFJWJ32SIOTXNXEA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.761217594 (556h11m5.632300639s) [noise=true , address=145.79.227.43:4163 , piece_id=77BKLLVD25UMTDQZW46CA2N225I7O4HCHCRRPQ7HLJGRHOLGSDHA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.762138507 (556h11m5.633221552s) [noise=true , address=77.90.31.19:4860 , piece_id=5FUYVK2Z6LBZOAGFI2NOTAAUIV4NMBI5ERMA5RH4BMONNHABZLDQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.764304321 (556h11m5.635387356s) [noise=true , address=145.79.227.30:4784 , piece_id=MKCRSFWVFVGOAYCK5M66HO3OT7OMYWLVR2KMDXUGBS2VXXNCUSSQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.765421217 (556h11m5.636504252s) [noise=true , address=145.79.227.44:4216 , piece_id=GSBWC2LXUOZYOVD55KNQYA5Z3S2TCYROBB5O2XZIIVFUQMJQMYNA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:07.082099826 (556h11m6.953182871s) [upload=487609]: Next returned nil segment Dec 28 19:40:07.082116707 (556h11m6.953199752s) [upload=487609]: Waiting for error group managing segments... Dec 28 19:40:07.415884144 (556h11m7.286967179s) [noise=true , address=64.49.13.154:4932 , piece_id=QB5WM5RZ2P2GC6RIGGM34IANVEI2SNDJ2PO556JXKTDBX5HLD4YA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.615434252 (556h11m7.486517297s) [noise=true , address=145.79.227.23:4550 , piece_id=NWIC722LZUHJSAMPWSKCLPRIDGWPVUEYMS5KRAMX3IIMD2KR4XGQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.647182077 (556h11m7.518265112s) [noise=true , address=145.79.227.28:4744 , piece_id=42LII6EOG3CMBTIL5I4RZQPDYU52TKJN3PGWEQXROBYMNDFYIIVA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.678458624 (556h11m7.549541659s) [noise=true , address=145.79.227.46:4626 , piece_id=OQOTWYLVIU7GJBBLYWQOEEFKR3TNAUOXZS6UAFJWJ32SIOTXNXEA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.793406049 (556h11m7.664489084s) [noise=true , address=77.90.31.19:4860 , piece_id=5FUYVK2Z6LBZOAGFI2NOTAAUIV4NMBI5ERMA5RH4BMONNHABZLDQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.802506498 (556h11m7.673589543s) [noise=true , address=77.90.31.21:4896 , piece_id=ATTC7BZUT4HAAEAZCHFZQE4INV2LLSWR2YCUNVA73LPJCW3YL35A, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.853663242 (556h11m7.724746287s) [noise=true , address=77.90.31.10:4522 , piece_id=JOHOMHCNYXF4XQGLN2NJUSWSUO3XFBRRMYD27SLIZAK2RCT3ZC2A, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.880639500 (556h11m7.751722535s) [noise=true , address=77.90.31.11:4582 , piece_id=ZRE24R27TX4E5DTV2POX7EJANQ4CM75D6ZOUS6SKM3QHHQCKRBRA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.904835446 (556h11m7.775918491s) [noise=true , address=77.90.31.18:4816 , piece_id=YT3PSYNQTDQBVBRUDNOXJ2XCN6EFQA54WBSNL6MZMXFXR7XNCHIA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.988914260 (556h11m7.859997295s) [noise=true , address=77.90.31.15:4722 , piece_id=AF7F2OA247CJIDFTEU6HEOIYGOINEYPMII53BKJQXI7UZ26MJIUA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.484927449 (556h11m8.356010484s) [noise=true , address=145.79.227.35:4510 , piece_id=P275TU367DTYXBQOGGV5THFUPHJUWGSM5IMJQ753VE4MX4IPOONQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.610317849 (556h11m8.481400894s) [noise=true , address=145.79.227.44:4216 , piece_id=GSBWC2LXUOZYOVD55KNQYA5Z3S2TCYROBB5O2XZIIVFUQMJQMYNA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.611969885 (556h11m8.48305292s) [noise=true , address=145.79.227.34:4916 , piece_id=SPYSFWMJ752FC5JGRE36G54S2FHG4DQFCV6CIZONQHJDXDCN52VQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.636098801 (556h11m8.507181836s) [noise=true , address=145.79.227.30:4784 , piece_id=MKCRSFWVFVGOAYCK5M66HO3OT7OMYWLVR2KMDXUGBS2VXXNCUSSQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.636122136 (556h11m8.507205181s) [seg_pos={0 0}, upload=487609]: Segment reached optimal threshold of 14 pieces. Dec 28 19:40:08.636333929 (556h11m8.507416974s) [noise=true , address=64.49.13.26:4956 , piece_id=FFHUPOOM7PLEABRHFH6H2SI6WXJBU2XFCLZJUBV4CXUJ6EP4PI2Q, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TYHhjpfyBKkenSwviyQCUZpSBbnaFCGBFRkM4fyKLDuxKx4Ju): protocol: expected piece hash; context canceled; EOF Dec 28 19:40:08.636342175 (556h11m8.50742521s) [noise=true , address=64.49.13.26:4956 , piece_id=FFHUPOOM7PLEABRHFH6H2SI6WXJBU2XFCLZJUBV4CXUJ6EP4PI2Q, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636585959 (556h11m8.507668994s) [noise=true , address=145.79.227.43:4163 , piece_id=77BKLLVD25UMTDQZW46CA2N225I7O4HCHCRRPQ7HLJGRHOLGSDHA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CgPQ5hkrJdqbJvGarxKjXvYYnrXQqapL5BH2rNJNvnhpRi3pw): protocol: expected piece hash; context canceled; EOF Dec 28 19:40:08.636592172 (556h11m8.507675207s) [noise=true , address=145.79.227.43:4163 , piece_id=77BKLLVD25UMTDQZW46CA2N225I7O4HCHCRRPQ7HLJGRHOLGSDHA, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636681511 (556h11m8.507764546s) [noise=true , address=145.79.227.19:4083 , piece_id=GL5XL7T2QQABD5364AT4AFV6RZNOWCTVNSMAVIFJU5SUQFHOPZKQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KZ5a6CQ9tUTrC6EYE7M8yg6p9fK95CN8zQLQmP4tdyz4rBfGh): protocol: expected piece hash; context canceled; EOF Dec 28 19:40:08.636687052 (556h11m8.507770087s) [noise=true , address=145.79.227.19:4083 , piece_id=GL5XL7T2QQABD5364AT4AFV6RZNOWCTVNSMAVIFJU5SUQFHOPZKQ, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636810327 (556h11m8.507893362s) [noise=true , address=145.79.227.17:4992 , piece_id=GUF7AUFCFRRPY5PSYS7DRPFQLGT4ZEL5C7P23US2VI54KYWSUCUQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hbt8HKTunpPKyAzH4XBwGQqTKzcgqPpoWwKZm9JMtaK8P7aVp): protocol: write tcp 188.166.225.170:51210->145.79.227.17:4992: use of closed network connection; write tcp 188.166.225.170:51210->145.79.227.17:4992: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51210->145.79.227.17:4992: use of closed network connection Dec 28 19:40:08.636815957 (556h11m8.507899002s) [noise=true , address=145.79.227.17:4992 , piece_id=GUF7AUFCFRRPY5PSYS7DRPFQLGT4ZEL5C7P23US2VI54KYWSUCUQ, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636834313 (556h11m8.507917358s) [seg_pos={0 0}, upload=487609]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 19:40:08.636907953 (556h11m8.507990998s) [upload=487609]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 19:40:08.636913644 (556h11m8.507996689s) [upload=487609]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 19:40:08.924991397 (556h11m8.796074442s) [upload=487609]: Done uploading segments... Dec 28 19:40:08.925014742 (556h11m8.796097777s) [upload=487609]: Upload finished. err: <nil> Duration: 21.965834125s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...XX..............X..........................X...........X.] duration: 5.8736106s trace: https://jaeger.dev.storj.io/trace/3425d363be458a81 FAIL (2025-12-28 20:24:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: XB2K7RONOVEBPA2XGY36WIE57MWUUBKVAHEGXHAZWNDRL22WHH5Q) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: HHINRA5KN56ZF2KW7IEG4BUGXM6UKMNUM3TM4Y3HFADVWASFEHFA) context deadline exceeded ======LOGS====== Dec 28 20:22:57.518307420 (556h53m57.389390455s) [upload=488235]: Starting upload Dec 28 20:22:57.518369106 (556h53m57.389452182s) [upload=488235]: Done starting upload Dec 28 20:22:57.518427107 (556h53m57.389510152s) [upload=488235]: Uploading segments... Dec 28 20:22:57.518663427 (556h53m57.389746473s) [upload=488235]: Got next segment. Inline: false Dec 28 20:22:57.518709805 (556h53m57.38979284s) [upload=488235]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 20:22:57.518716318 (556h53m57.389799353s) [upload=488235]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 20:22:57.767659750 (556h53m57.638742815s) [seg_pos={0 0}, upload=488235]: Begin upload segment... Dec 28 20:22:57.767735904 (556h53m57.638818949s) [seg_pos={0 0}, upload=488235]: Done begin upload segment. Dec 28 20:22:57.767792252 (556h53m57.638875287s) [noise=true , address=77.90.31.21:4880 , piece_id=V4PF2AQHKOK3GCF6B4UDNRJW7WYQ5QVOC66HUJMOPOOLPZAQIM4A, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.767877795 (556h53m57.63896085s) [noise=true , address=145.79.227.51:4409 , piece_id=C342XC375VQVI23V5JESCTOES4AF66QWHIETEOBVV3KGRLIP2X5Q, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.768512193 (556h53m57.639595238s) [noise=true , address=145.79.227.18:4007 , piece_id=NB6ADK5TXELRFCYMKDF6C3QZRZQUL5XUPVQ7AVVYP32UY2GZA6SA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.768539074 (556h53m57.639622119s) [noise=true , address=45.91.116.5:4297 , piece_id=FNX3RGGDGJZL5DOZRL2ZF2K2JHRAHWAWHIN6NQ4ONUKPVC6J4ECA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.769129117 (556h53m57.640212152s) [noise=true , address=145.79.227.41:4086 , piece_id=YRHLQZ64IEO7C26HZTCDY2T2PRYUFP6CSN2EOBJS7XAE4I7MXLHQ, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.769521375 (556h53m57.64060441s) [noise=true , address=145.79.227.12:4765 , piece_id=IESWSXDQFYKKHDOEHSYQSVMTBYLAJWNQ5TPWGNQPMLHO23CXYXTQ, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.770627170 (556h53m57.641710205s) [noise=true , address=77.90.31.13:4627 , piece_id=XB2K7RONOVEBPA2XGY36WIE57MWUUBKVAHEGXHAZWNDRL22WHH5Q, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.771369815 (556h53m57.64245285s) [noise=true , address=145.79.227.50:4375 , piece_id=DMDQHEH7XJT334MFCFAPNCGLSJ4UWE3U74MTDFROMTM4BDAI4IHA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.772492252 (556h53m57.643575297s) [noise=true , address=145.79.227.36:4519 , piece_id=YIPTLMBP7UV4GLPNA74M4343U4JEE7433ANXUHKYIJLZ36T25TXA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.773128333 (556h53m57.644211368s) [noise=true , address=77.90.31.17:4770 , piece_id=MH6LXF5JVTVI3U3N7W73U4ULOARVA6G2CRNUQLQ3V2UE6HUDGCUA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.775274971 (556h53m57.646358006s) [noise=true , address=145.79.227.40:4051 , piece_id=PZ3547GPNB3TRB24RBQVWLHLOC5JQVFUMG4ZZJDA3BBESZN3XFYA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.777926579 (556h53m57.649009615s) [noise=true , address=145.79.227.42:4123 , piece_id=TN4ZO3UUKIFRAP44SANXU6HKESFMIE45S5XTWAIKHJW5SA6C2KFA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.780387376 (556h53m57.651470421s) [noise=true , address=45.91.116.13:4155 , piece_id=HHINRA5KN56ZF2KW7IEG4BUGXM6UKMNUM3TM4Y3HFADVWASFEHFA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.781667764 (556h53m57.652750809s) [noise=true , address=77.90.31.15:4697 , piece_id=25QRUIDDPIZ7CSGKDAXLMZIWJHE6HHITH6FU4H2GA2GDBI4PR5KA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.783365686 (556h53m57.654448731s) [noise=true , address=77.90.31.11:4556 , piece_id=QRRGWDO7EHFVVQ6X4BTVY7HQRRDF5UCHKCFRLO2FOOYLTEDNJR2A, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.784872727 (556h53m57.655955762s) [noise=true , address=145.79.227.21:4473 , piece_id=OHS3FHPTXD3EKQNBJLLGKENQIJPKPGEFTVIRVQYXO3PZQEVQILGQ, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:59.482505035 (556h53m59.35358808s) [upload=488235]: Next returned nil segment Dec 28 20:22:59.482533810 (556h53m59.353616845s) [upload=488235]: Waiting for error group managing segments... Dec 28 20:22:59.769352743 (556h53m59.640435799s) [noise=true , address=145.79.227.18:4007 , piece_id=NB6ADK5TXELRFCYMKDF6C3QZRZQUL5XUPVQ7AVVYP32UY2GZA6SA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.782950737 (556h53m59.654033782s) [noise=true , address=145.79.227.51:4409 , piece_id=C342XC375VQVI23V5JESCTOES4AF66QWHIETEOBVV3KGRLIP2X5Q, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.789678698 (556h53m59.660761733s) [noise=true , address=145.79.227.12:4765 , piece_id=IESWSXDQFYKKHDOEHSYQSVMTBYLAJWNQ5TPWGNQPMLHO23CXYXTQ, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.802190252 (556h53m59.673273287s) [noise=true , address=145.79.227.42:4123 , piece_id=TN4ZO3UUKIFRAP44SANXU6HKESFMIE45S5XTWAIKHJW5SA6C2KFA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.804531330 (556h53m59.675614375s) [noise=true , address=145.79.227.50:4375 , piece_id=DMDQHEH7XJT334MFCFAPNCGLSJ4UWE3U74MTDFROMTM4BDAI4IHA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.810012928 (556h53m59.681095963s) [noise=true , address=145.79.227.21:4473 , piece_id=OHS3FHPTXD3EKQNBJLLGKENQIJPKPGEFTVIRVQYXO3PZQEVQILGQ, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.812050608 (556h53m59.683133643s) [noise=true , address=145.79.227.36:4519 , piece_id=YIPTLMBP7UV4GLPNA74M4343U4JEE7433ANXUHKYIJLZ36T25TXA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.812993363 (556h53m59.684076398s) [noise=true , address=145.79.227.41:4086 , piece_id=YRHLQZ64IEO7C26HZTCDY2T2PRYUFP6CSN2EOBJS7XAE4I7MXLHQ, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.946324830 (556h53m59.817407865s) [noise=true , address=77.90.31.13:4627 , piece_id=XB2K7RONOVEBPA2XGY36WIE57MWUUBKVAHEGXHAZWNDRL22WHH5Q, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.965566208 (556h53m59.836649243s) [noise=true , address=77.90.31.17:4770 , piece_id=MH6LXF5JVTVI3U3N7W73U4ULOARVA6G2CRNUQLQ3V2UE6HUDGCUA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:23:00.024382405 (556h53m59.89546544s) [noise=true , address=77.90.31.21:4880 , piece_id=V4PF2AQHKOK3GCF6B4UDNRJW7WYQ5QVOC66HUJMOPOOLPZAQIM4A, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:23:00.056351529 (556h53m59.927434574s) [noise=true , address=45.91.116.5:4297 , piece_id=FNX3RGGDGJZL5DOZRL2ZF2K2JHRAHWAWHIN6NQ4ONUKPVC6J4ECA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:23:00.067006559 (556h53m59.938089594s) [noise=true , address=45.91.116.13:4155 , piece_id=HHINRA5KN56ZF2KW7IEG4BUGXM6UKMNUM3TM4Y3HFADVWASFEHFA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:23:00.067025144 (556h53m59.938108179s) [seg_pos={0 0}, upload=488235]: Segment reached optimal threshold of 13 pieces. Dec 28 20:23:00.067214014 (556h53m59.938297059s) [noise=true , address=77.90.31.11:4556 , piece_id=QRRGWDO7EHFVVQ6X4BTVY7HQRRDF5UCHKCFRLO2FOOYLTEDNJR2A, seg_pos={0 0}, upload=488235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: write tcp 188.166.225.170:60580->77.90.31.11:4556: use of closed network connection; write tcp 188.166.225.170:60580->77.90.31.11:4556: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60580->77.90.31.11:4556: use of closed network connection Dec 28 20:23:00.067225536 (556h53m59.938308571s) [noise=true , address=77.90.31.11:4556 , piece_id=QRRGWDO7EHFVVQ6X4BTVY7HQRRDF5UCHKCFRLO2FOOYLTEDNJR2A, seg_pos={0 0}, upload=488235]: Long tail cancellation due to optimal threshold Dec 28 20:23:00.067306319 (556h53m59.938389364s) [noise=true , address=77.90.31.15:4697 , piece_id=25QRUIDDPIZ7CSGKDAXLMZIWJHE6HHITH6FU4H2GA2GDBI4PR5KA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Dec 28 20:23:00.067314354 (556h53m59.938397389s) [noise=true , address=77.90.31.15:4697 , piece_id=25QRUIDDPIZ7CSGKDAXLMZIWJHE6HHITH6FU4H2GA2GDBI4PR5KA, seg_pos={0 0}, upload=488235]: Long tail cancellation due to optimal threshold Dec 28 20:23:00.067582064 (556h53m59.938665119s) [noise=true , address=145.79.227.40:4051 , piece_id=PZ3547GPNB3TRB24RBQVWLHLOC5JQVFUMG4ZZJDA3BBESZN3XFYA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 28 20:23:00.067589408 (556h53m59.938672453s) [noise=true , address=145.79.227.40:4051 , piece_id=PZ3547GPNB3TRB24RBQVWLHLOC5JQVFUMG4ZZJDA3BBESZN3XFYA, seg_pos={0 0}, upload=488235]: Long tail cancellation due to optimal threshold Dec 28 20:23:00.067605620 (556h53m59.938688655s) [seg_pos={0 0}, upload=488235]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 20:23:00.067681614 (556h53m59.938764659s) [upload=488235]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 20:23:00.067686654 (556h53m59.938769699s) [upload=488235]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 20:23:00.354014782 (556h54m0.225097827s) [upload=488235]: Done uploading segments... Dec 28 20:23:00.354034600 (556h54m0.225117635s) [upload=488235]: Upload finished. err: <nil> Duration: 5.873641418s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 19.238494732s trace: https://jaeger.dev.storj.io/trace/10e1641b228b2a1f FAIL (2025-12-28 20:23:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 40:05.744568479 (556h11m5.615651514s) [noise=true , address=145.79.227.28:4744 , piece_id=42LII6EOG3CMBTIL5I4RZQPDYU52TKJN3PGWEQXROBYMNDFYIIVA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.745099351 (556h11m5.616182396s) [noise=true , address=145.79.227.23:4550 , piece_id=NWIC722LZUHJSAMPWSKCLPRIDGWPVUEYMS5KRAMX3IIMD2KR4XGQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.745777401 (556h11m5.616860436s) [noise=true , address=77.90.31.15:4722 , piece_id=AF7F2OA247CJIDFTEU6HEOIYGOINEYPMII53BKJQXI7UZ26MJIUA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.746374739 (556h11m5.617457784s) [noise=true , address=77.90.31.10:4522 , piece_id=JOHOMHCNYXF4XQGLN2NJUSWSUO3XFBRRMYD27SLIZAK2RCT3ZC2A, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.746934555 (556h11m5.618017601s) [noise=true , address=77.90.31.21:4896 , piece_id=ATTC7BZUT4HAAEAZCHFZQE4INV2LLSWR2YCUNVA73LPJCW3YL35A, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.747556861 (556h11m5.618640146s) [noise=true , address=64.49.13.26:4956 , piece_id=FFHUPOOM7PLEABRHFH6H2SI6WXJBU2XFCLZJUBV4CXUJ6EP4PI2Q, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.748964150 (556h11m5.620047195s) [noise=true , address=145.79.227.34:4916 , piece_id=SPYSFWMJ752FC5JGRE36G54S2FHG4DQFCV6CIZONQHJDXDCN52VQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.751119314 (556h11m5.622202359s) [noise=true , address=77.90.31.18:4816 , piece_id=YT3PSYNQTDQBVBRUDNOXJ2XCN6EFQA54WBSNL6MZMXFXR7XNCHIA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.753413222 (556h11m5.624496257s) [noise=true , address=77.90.31.11:4582 , piece_id=ZRE24R27TX4E5DTV2POX7EJANQ4CM75D6ZOUS6SKM3QHHQCKRBRA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.755230482 (556h11m5.626313527s) [noise=true , address=64.49.13.154:4932 , piece_id=QB5WM5RZ2P2GC6RIGGM34IANVEI2SNDJ2PO556JXKTDBX5HLD4YA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.757598362 (556h11m5.628681407s) [noise=true , address=145.79.227.35:4510 , piece_id=P275TU367DTYXBQOGGV5THFUPHJUWGSM5IMJQ753VE4MX4IPOONQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.759644679 (556h11m5.630727734s) [noise=true , address=145.79.227.46:4626 , piece_id=OQOTWYLVIU7GJBBLYWQOEEFKR3TNAUOXZS6UAFJWJ32SIOTXNXEA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.761217594 (556h11m5.632300639s) [noise=true , address=145.79.227.43:4163 , piece_id=77BKLLVD25UMTDQZW46CA2N225I7O4HCHCRRPQ7HLJGRHOLGSDHA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.762138507 (556h11m5.633221552s) [noise=true , address=77.90.31.19:4860 , piece_id=5FUYVK2Z6LBZOAGFI2NOTAAUIV4NMBI5ERMA5RH4BMONNHABZLDQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.764304321 (556h11m5.635387356s) [noise=true , address=145.79.227.30:4784 , piece_id=MKCRSFWVFVGOAYCK5M66HO3OT7OMYWLVR2KMDXUGBS2VXXNCUSSQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.765421217 (556h11m5.636504252s) [noise=true , address=145.79.227.44:4216 , piece_id=GSBWC2LXUOZYOVD55KNQYA5Z3S2TCYROBB5O2XZIIVFUQMJQMYNA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:07.082099826 (556h11m6.953182871s) [upload=487609]: Next returned nil segment Dec 28 19:40:07.082116707 (556h11m6.953199752s) [upload=487609]: Waiting for error group managing segments... Dec 28 19:40:07.415884144 (556h11m7.286967179s) [noise=true , address=64.49.13.154:4932 , piece_id=QB5WM5RZ2P2GC6RIGGM34IANVEI2SNDJ2PO556JXKTDBX5HLD4YA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.615434252 (556h11m7.486517297s) [noise=true , address=145.79.227.23:4550 , piece_id=NWIC722LZUHJSAMPWSKCLPRIDGWPVUEYMS5KRAMX3IIMD2KR4XGQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.647182077 (556h11m7.518265112s) [noise=true , address=145.79.227.28:4744 , piece_id=42LII6EOG3CMBTIL5I4RZQPDYU52TKJN3PGWEQXROBYMNDFYIIVA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.678458624 (556h11m7.549541659s) [noise=true , address=145.79.227.46:4626 , piece_id=OQOTWYLVIU7GJBBLYWQOEEFKR3TNAUOXZS6UAFJWJ32SIOTXNXEA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.793406049 (556h11m7.664489084s) [noise=true , address=77.90.31.19:4860 , piece_id=5FUYVK2Z6LBZOAGFI2NOTAAUIV4NMBI5ERMA5RH4BMONNHABZLDQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.802506498 (556h11m7.673589543s) [noise=true , address=77.90.31.21:4896 , piece_id=ATTC7BZUT4HAAEAZCHFZQE4INV2LLSWR2YCUNVA73LPJCW3YL35A, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.853663242 (556h11m7.724746287s) [noise=true , address=77.90.31.10:4522 , piece_id=JOHOMHCNYXF4XQGLN2NJUSWSUO3XFBRRMYD27SLIZAK2RCT3ZC2A, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.880639500 (556h11m7.751722535s) [noise=true , address=77.90.31.11:4582 , piece_id=ZRE24R27TX4E5DTV2POX7EJANQ4CM75D6ZOUS6SKM3QHHQCKRBRA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.904835446 (556h11m7.775918491s) [noise=true , address=77.90.31.18:4816 , piece_id=YT3PSYNQTDQBVBRUDNOXJ2XCN6EFQA54WBSNL6MZMXFXR7XNCHIA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.988914260 (556h11m7.859997295s) [noise=true , address=77.90.31.15:4722 , piece_id=AF7F2OA247CJIDFTEU6HEOIYGOINEYPMII53BKJQXI7UZ26MJIUA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.484927449 (556h11m8.356010484s) [noise=true , address=145.79.227.35:4510 , piece_id=P275TU367DTYXBQOGGV5THFUPHJUWGSM5IMJQ753VE4MX4IPOONQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.610317849 (556h11m8.481400894s) [noise=true , address=145.79.227.44:4216 , piece_id=GSBWC2LXUOZYOVD55KNQYA5Z3S2TCYROBB5O2XZIIVFUQMJQMYNA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.611969885 (556h11m8.48305292s) [noise=true , address=145.79.227.34:4916 , piece_id=SPYSFWMJ752FC5JGRE36G54S2FHG4DQFCV6CIZONQHJDXDCN52VQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.636098801 (556h11m8.507181836s) [noise=true , address=145.79.227.30:4784 , piece_id=MKCRSFWVFVGOAYCK5M66HO3OT7OMYWLVR2KMDXUGBS2VXXNCUSSQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.636122136 (556h11m8.507205181s) [seg_pos={0 0}, upload=487609]: Segment reached optimal threshold of 14 pieces. Dec 28 19:40:08.636333929 (556h11m8.507416974s) [noise=true , address=64.49.13.26:4956 , piece_id=FFHUPOOM7PLEABRHFH6H2SI6WXJBU2XFCLZJUBV4CXUJ6EP4PI2Q, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TYHhjpfyBKkenSwviyQCUZpSBbnaFCGBFRkM4fyKLDuxKx4Ju): protocol: expected piece hash; context canceled; EOF Dec 28 19:40:08.636342175 (556h11m8.50742521s) [noise=true , address=64.49.13.26:4956 , piece_id=FFHUPOOM7PLEABRHFH6H2SI6WXJBU2XFCLZJUBV4CXUJ6EP4PI2Q, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636585959 (556h11m8.507668994s) [noise=true , address=145.79.227.43:4163 , piece_id=77BKLLVD25UMTDQZW46CA2N225I7O4HCHCRRPQ7HLJGRHOLGSDHA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CgPQ5hkrJdqbJvGarxKjXvYYnrXQqapL5BH2rNJNvnhpRi3pw): protocol: expected piece hash; context canceled; EOF Dec 28 19:40:08.636592172 (556h11m8.507675207s) [noise=true , address=145.79.227.43:4163 , piece_id=77BKLLVD25UMTDQZW46CA2N225I7O4HCHCRRPQ7HLJGRHOLGSDHA, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636681511 (556h11m8.507764546s) [noise=true , address=145.79.227.19:4083 , piece_id=GL5XL7T2QQABD5364AT4AFV6RZNOWCTVNSMAVIFJU5SUQFHOPZKQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KZ5a6CQ9tUTrC6EYE7M8yg6p9fK95CN8zQLQmP4tdyz4rBfGh): protocol: expected piece hash; context canceled; EOF Dec 28 19:40:08.636687052 (556h11m8.507770087s) [noise=true , address=145.79.227.19:4083 , piece_id=GL5XL7T2QQABD5364AT4AFV6RZNOWCTVNSMAVIFJU5SUQFHOPZKQ, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636810327 (556h11m8.507893362s) [noise=true , address=145.79.227.17:4992 , piece_id=GUF7AUFCFRRPY5PSYS7DRPFQLGT4ZEL5C7P23US2VI54KYWSUCUQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hbt8HKTunpPKyAzH4XBwGQqTKzcgqPpoWwKZm9JMtaK8P7aVp): protocol: write tcp 188.166.225.170:51210->145.79.227.17:4992: use of closed network connection; write tcp 188.166.225.170:51210->145.79.227.17:4992: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51210->145.79.227.17:4992: use of closed network connection Dec 28 19:40:08.636815957 (556h11m8.507899002s) [noise=true , address=145.79.227.17:4992 , piece_id=GUF7AUFCFRRPY5PSYS7DRPFQLGT4ZEL5C7P23US2VI54KYWSUCUQ, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636834313 (556h11m8.507917358s) [seg_pos={0 0}, upload=487609]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 19:40:08.636907953 (556h11m8.507990998s) [upload=487609]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 19:40:08.636913644 (556h11m8.507996689s) [upload=487609]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 19:40:08.924991397 (556h11m8.796074442s) [upload=487609]: Done uploading segments... Dec 28 19:40:08.925014742 (556h11m8.796097777s) [upload=487609]: Upload finished. err: <nil> Duration: 19.238527204s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [....XX..............X..........................X...........X] duration: 30.002683133s trace: https://jaeger.dev.storj.io/trace/3425d363be458a81 FAIL (2025-12-28 20:23:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: XB2K7RONOVEBPA2XGY36WIE57MWUUBKVAHEGXHAZWNDRL22WHH5Q) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: HHINRA5KN56ZF2KW7IEG4BUGXM6UKMNUM3TM4Y3HFADVWASFEHFA) context deadline exceeded ======LOGS====== Dec 28 20:22:57.518307420 (556h53m57.389390455s) [upload=488235]: Starting upload Dec 28 20:22:57.518369106 (556h53m57.389452182s) [upload=488235]: Done starting upload Dec 28 20:22:57.518427107 (556h53m57.389510152s) [upload=488235]: Uploading segments... Dec 28 20:22:57.518663427 (556h53m57.389746473s) [upload=488235]: Got next segment. Inline: false Dec 28 20:22:57.518709805 (556h53m57.38979284s) [upload=488235]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 20:22:57.518716318 (556h53m57.389799353s) [upload=488235]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 20:22:57.767659750 (556h53m57.638742815s) [seg_pos={0 0}, upload=488235]: Begin upload segment... Dec 28 20:22:57.767735904 (556h53m57.638818949s) [seg_pos={0 0}, upload=488235]: Done begin upload segment. Dec 28 20:22:57.767792252 (556h53m57.638875287s) [noise=true , address=77.90.31.21:4880 , piece_id=V4PF2AQHKOK3GCF6B4UDNRJW7WYQ5QVOC66HUJMOPOOLPZAQIM4A, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.767877795 (556h53m57.63896085s) [noise=true , address=145.79.227.51:4409 , piece_id=C342XC375VQVI23V5JESCTOES4AF66QWHIETEOBVV3KGRLIP2X5Q, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.768512193 (556h53m57.639595238s) [noise=true , address=145.79.227.18:4007 , piece_id=NB6ADK5TXELRFCYMKDF6C3QZRZQUL5XUPVQ7AVVYP32UY2GZA6SA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.768539074 (556h53m57.639622119s) [noise=true , address=45.91.116.5:4297 , piece_id=FNX3RGGDGJZL5DOZRL2ZF2K2JHRAHWAWHIN6NQ4ONUKPVC6J4ECA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.769129117 (556h53m57.640212152s) [noise=true , address=145.79.227.41:4086 , piece_id=YRHLQZ64IEO7C26HZTCDY2T2PRYUFP6CSN2EOBJS7XAE4I7MXLHQ, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.769521375 (556h53m57.64060441s) [noise=true , address=145.79.227.12:4765 , piece_id=IESWSXDQFYKKHDOEHSYQSVMTBYLAJWNQ5TPWGNQPMLHO23CXYXTQ, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.770627170 (556h53m57.641710205s) [noise=true , address=77.90.31.13:4627 , piece_id=XB2K7RONOVEBPA2XGY36WIE57MWUUBKVAHEGXHAZWNDRL22WHH5Q, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.771369815 (556h53m57.64245285s) [noise=true , address=145.79.227.50:4375 , piece_id=DMDQHEH7XJT334MFCFAPNCGLSJ4UWE3U74MTDFROMTM4BDAI4IHA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.772492252 (556h53m57.643575297s) [noise=true , address=145.79.227.36:4519 , piece_id=YIPTLMBP7UV4GLPNA74M4343U4JEE7433ANXUHKYIJLZ36T25TXA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.773128333 (556h53m57.644211368s) [noise=true , address=77.90.31.17:4770 , piece_id=MH6LXF5JVTVI3U3N7W73U4ULOARVA6G2CRNUQLQ3V2UE6HUDGCUA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.775274971 (556h53m57.646358006s) [noise=true , address=145.79.227.40:4051 , piece_id=PZ3547GPNB3TRB24RBQVWLHLOC5JQVFUMG4ZZJDA3BBESZN3XFYA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.777926579 (556h53m57.649009615s) [noise=true , address=145.79.227.42:4123 , piece_id=TN4ZO3UUKIFRAP44SANXU6HKESFMIE45S5XTWAIKHJW5SA6C2KFA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.780387376 (556h53m57.651470421s) [noise=true , address=45.91.116.13:4155 , piece_id=HHINRA5KN56ZF2KW7IEG4BUGXM6UKMNUM3TM4Y3HFADVWASFEHFA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.781667764 (556h53m57.652750809s) [noise=true , address=77.90.31.15:4697 , piece_id=25QRUIDDPIZ7CSGKDAXLMZIWJHE6HHITH6FU4H2GA2GDBI4PR5KA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.783365686 (556h53m57.654448731s) [noise=true , address=77.90.31.11:4556 , piece_id=QRRGWDO7EHFVVQ6X4BTVY7HQRRDF5UCHKCFRLO2FOOYLTEDNJR2A, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.784872727 (556h53m57.655955762s) [noise=true , address=145.79.227.21:4473 , piece_id=OHS3FHPTXD3EKQNBJLLGKENQIJPKPGEFTVIRVQYXO3PZQEVQILGQ, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:59.482505035 (556h53m59.35358808s) [upload=488235]: Next returned nil segment Dec 28 20:22:59.482533810 (556h53m59.353616845s) [upload=488235]: Waiting for error group managing segments... Dec 28 20:22:59.769352743 (556h53m59.640435799s) [noise=true , address=145.79.227.18:4007 , piece_id=NB6ADK5TXELRFCYMKDF6C3QZRZQUL5XUPVQ7AVVYP32UY2GZA6SA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.782950737 (556h53m59.654033782s) [noise=true , address=145.79.227.51:4409 , piece_id=C342XC375VQVI23V5JESCTOES4AF66QWHIETEOBVV3KGRLIP2X5Q, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.789678698 (556h53m59.660761733s) [noise=true , address=145.79.227.12:4765 , piece_id=IESWSXDQFYKKHDOEHSYQSVMTBYLAJWNQ5TPWGNQPMLHO23CXYXTQ, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.802190252 (556h53m59.673273287s) [noise=true , address=145.79.227.42:4123 , piece_id=TN4ZO3UUKIFRAP44SANXU6HKESFMIE45S5XTWAIKHJW5SA6C2KFA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.804531330 (556h53m59.675614375s) [noise=true , address=145.79.227.50:4375 , piece_id=DMDQHEH7XJT334MFCFAPNCGLSJ4UWE3U74MTDFROMTM4BDAI4IHA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.810012928 (556h53m59.681095963s) [noise=true , address=145.79.227.21:4473 , piece_id=OHS3FHPTXD3EKQNBJLLGKENQIJPKPGEFTVIRVQYXO3PZQEVQILGQ, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.812050608 (556h53m59.683133643s) [noise=true , address=145.79.227.36:4519 , piece_id=YIPTLMBP7UV4GLPNA74M4343U4JEE7433ANXUHKYIJLZ36T25TXA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.812993363 (556h53m59.684076398s) [noise=true , address=145.79.227.41:4086 , piece_id=YRHLQZ64IEO7C26HZTCDY2T2PRYUFP6CSN2EOBJS7XAE4I7MXLHQ, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.946324830 (556h53m59.817407865s) [noise=true , address=77.90.31.13:4627 , piece_id=XB2K7RONOVEBPA2XGY36WIE57MWUUBKVAHEGXHAZWNDRL22WHH5Q, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.965566208 (556h53m59.836649243s) [noise=true , address=77.90.31.17:4770 , piece_id=MH6LXF5JVTVI3U3N7W73U4ULOARVA6G2CRNUQLQ3V2UE6HUDGCUA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:23:00.024382405 (556h53m59.89546544s) [noise=true , address=77.90.31.21:4880 , piece_id=V4PF2AQHKOK3GCF6B4UDNRJW7WYQ5QVOC66HUJMOPOOLPZAQIM4A, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:23:00.056351529 (556h53m59.927434574s) [noise=true , address=45.91.116.5:4297 , piece_id=FNX3RGGDGJZL5DOZRL2ZF2K2JHRAHWAWHIN6NQ4ONUKPVC6J4ECA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:23:00.067006559 (556h53m59.938089594s) [noise=true , address=45.91.116.13:4155 , piece_id=HHINRA5KN56ZF2KW7IEG4BUGXM6UKMNUM3TM4Y3HFADVWASFEHFA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:23:00.067025144 (556h53m59.938108179s) [seg_pos={0 0}, upload=488235]: Segment reached optimal threshold of 13 pieces. Dec 28 20:23:00.067214014 (556h53m59.938297059s) [noise=true , address=77.90.31.11:4556 , piece_id=QRRGWDO7EHFVVQ6X4BTVY7HQRRDF5UCHKCFRLO2FOOYLTEDNJR2A, seg_pos={0 0}, upload=488235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: write tcp 188.166.225.170:60580->77.90.31.11:4556: use of closed network connection; write tcp 188.166.225.170:60580->77.90.31.11:4556: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60580->77.90.31.11:4556: use of closed network connection Dec 28 20:23:00.067225536 (556h53m59.938308571s) [noise=true , address=77.90.31.11:4556 , piece_id=QRRGWDO7EHFVVQ6X4BTVY7HQRRDF5UCHKCFRLO2FOOYLTEDNJR2A, seg_pos={0 0}, upload=488235]: Long tail cancellation due to optimal threshold Dec 28 20:23:00.067306319 (556h53m59.938389364s) [noise=true , address=77.90.31.15:4697 , piece_id=25QRUIDDPIZ7CSGKDAXLMZIWJHE6HHITH6FU4H2GA2GDBI4PR5KA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Dec 28 20:23:00.067314354 (556h53m59.938397389s) [noise=true , address=77.90.31.15:4697 , piece_id=25QRUIDDPIZ7CSGKDAXLMZIWJHE6HHITH6FU4H2GA2GDBI4PR5KA, seg_pos={0 0}, upload=488235]: Long tail cancellation due to optimal threshold Dec 28 20:23:00.067582064 (556h53m59.938665119s) [noise=true , address=145.79.227.40:4051 , piece_id=PZ3547GPNB3TRB24RBQVWLHLOC5JQVFUMG4ZZJDA3BBESZN3XFYA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 28 20:23:00.067589408 (556h53m59.938672453s) [noise=true , address=145.79.227.40:4051 , piece_id=PZ3547GPNB3TRB24RBQVWLHLOC5JQVFUMG4ZZJDA3BBESZN3XFYA, seg_pos={0 0}, upload=488235]: Long tail cancellation due to optimal threshold Dec 28 20:23:00.067605620 (556h53m59.938688655s) [seg_pos={0 0}, upload=488235]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 20:23:00.067681614 (556h53m59.938764659s) [upload=488235]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 20:23:00.067686654 (556h53m59.938769699s) [upload=488235]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 20:23:00.354014782 (556h54m0.225097827s) [upload=488235]: Done uploading segments... Dec 28 20:23:00.354034600 (556h54m0.225117635s) [upload=488235]: Upload finished. err: <nil> Duration: 30.002706588s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 33.782940266s trace: https://jaeger.dev.storj.io/trace/10e1641b228b2a1f FAIL (2025-12-28 20:22:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 40:05.744568479 (556h11m5.615651514s) [noise=true , address=145.79.227.28:4744 , piece_id=42LII6EOG3CMBTIL5I4RZQPDYU52TKJN3PGWEQXROBYMNDFYIIVA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.745099351 (556h11m5.616182396s) [noise=true , address=145.79.227.23:4550 , piece_id=NWIC722LZUHJSAMPWSKCLPRIDGWPVUEYMS5KRAMX3IIMD2KR4XGQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.745777401 (556h11m5.616860436s) [noise=true , address=77.90.31.15:4722 , piece_id=AF7F2OA247CJIDFTEU6HEOIYGOINEYPMII53BKJQXI7UZ26MJIUA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.746374739 (556h11m5.617457784s) [noise=true , address=77.90.31.10:4522 , piece_id=JOHOMHCNYXF4XQGLN2NJUSWSUO3XFBRRMYD27SLIZAK2RCT3ZC2A, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.746934555 (556h11m5.618017601s) [noise=true , address=77.90.31.21:4896 , piece_id=ATTC7BZUT4HAAEAZCHFZQE4INV2LLSWR2YCUNVA73LPJCW3YL35A, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.747556861 (556h11m5.618640146s) [noise=true , address=64.49.13.26:4956 , piece_id=FFHUPOOM7PLEABRHFH6H2SI6WXJBU2XFCLZJUBV4CXUJ6EP4PI2Q, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.748964150 (556h11m5.620047195s) [noise=true , address=145.79.227.34:4916 , piece_id=SPYSFWMJ752FC5JGRE36G54S2FHG4DQFCV6CIZONQHJDXDCN52VQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.751119314 (556h11m5.622202359s) [noise=true , address=77.90.31.18:4816 , piece_id=YT3PSYNQTDQBVBRUDNOXJ2XCN6EFQA54WBSNL6MZMXFXR7XNCHIA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.753413222 (556h11m5.624496257s) [noise=true , address=77.90.31.11:4582 , piece_id=ZRE24R27TX4E5DTV2POX7EJANQ4CM75D6ZOUS6SKM3QHHQCKRBRA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.755230482 (556h11m5.626313527s) [noise=true , address=64.49.13.154:4932 , piece_id=QB5WM5RZ2P2GC6RIGGM34IANVEI2SNDJ2PO556JXKTDBX5HLD4YA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.757598362 (556h11m5.628681407s) [noise=true , address=145.79.227.35:4510 , piece_id=P275TU367DTYXBQOGGV5THFUPHJUWGSM5IMJQ753VE4MX4IPOONQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.759644679 (556h11m5.630727734s) [noise=true , address=145.79.227.46:4626 , piece_id=OQOTWYLVIU7GJBBLYWQOEEFKR3TNAUOXZS6UAFJWJ32SIOTXNXEA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.761217594 (556h11m5.632300639s) [noise=true , address=145.79.227.43:4163 , piece_id=77BKLLVD25UMTDQZW46CA2N225I7O4HCHCRRPQ7HLJGRHOLGSDHA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.762138507 (556h11m5.633221552s) [noise=true , address=77.90.31.19:4860 , piece_id=5FUYVK2Z6LBZOAGFI2NOTAAUIV4NMBI5ERMA5RH4BMONNHABZLDQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.764304321 (556h11m5.635387356s) [noise=true , address=145.79.227.30:4784 , piece_id=MKCRSFWVFVGOAYCK5M66HO3OT7OMYWLVR2KMDXUGBS2VXXNCUSSQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.765421217 (556h11m5.636504252s) [noise=true , address=145.79.227.44:4216 , piece_id=GSBWC2LXUOZYOVD55KNQYA5Z3S2TCYROBB5O2XZIIVFUQMJQMYNA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:07.082099826 (556h11m6.953182871s) [upload=487609]: Next returned nil segment Dec 28 19:40:07.082116707 (556h11m6.953199752s) [upload=487609]: Waiting for error group managing segments... Dec 28 19:40:07.415884144 (556h11m7.286967179s) [noise=true , address=64.49.13.154:4932 , piece_id=QB5WM5RZ2P2GC6RIGGM34IANVEI2SNDJ2PO556JXKTDBX5HLD4YA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.615434252 (556h11m7.486517297s) [noise=true , address=145.79.227.23:4550 , piece_id=NWIC722LZUHJSAMPWSKCLPRIDGWPVUEYMS5KRAMX3IIMD2KR4XGQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.647182077 (556h11m7.518265112s) [noise=true , address=145.79.227.28:4744 , piece_id=42LII6EOG3CMBTIL5I4RZQPDYU52TKJN3PGWEQXROBYMNDFYIIVA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.678458624 (556h11m7.549541659s) [noise=true , address=145.79.227.46:4626 , piece_id=OQOTWYLVIU7GJBBLYWQOEEFKR3TNAUOXZS6UAFJWJ32SIOTXNXEA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.793406049 (556h11m7.664489084s) [noise=true , address=77.90.31.19:4860 , piece_id=5FUYVK2Z6LBZOAGFI2NOTAAUIV4NMBI5ERMA5RH4BMONNHABZLDQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.802506498 (556h11m7.673589543s) [noise=true , address=77.90.31.21:4896 , piece_id=ATTC7BZUT4HAAEAZCHFZQE4INV2LLSWR2YCUNVA73LPJCW3YL35A, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.853663242 (556h11m7.724746287s) [noise=true , address=77.90.31.10:4522 , piece_id=JOHOMHCNYXF4XQGLN2NJUSWSUO3XFBRRMYD27SLIZAK2RCT3ZC2A, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.880639500 (556h11m7.751722535s) [noise=true , address=77.90.31.11:4582 , piece_id=ZRE24R27TX4E5DTV2POX7EJANQ4CM75D6ZOUS6SKM3QHHQCKRBRA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.904835446 (556h11m7.775918491s) [noise=true , address=77.90.31.18:4816 , piece_id=YT3PSYNQTDQBVBRUDNOXJ2XCN6EFQA54WBSNL6MZMXFXR7XNCHIA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.988914260 (556h11m7.859997295s) [noise=true , address=77.90.31.15:4722 , piece_id=AF7F2OA247CJIDFTEU6HEOIYGOINEYPMII53BKJQXI7UZ26MJIUA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.484927449 (556h11m8.356010484s) [noise=true , address=145.79.227.35:4510 , piece_id=P275TU367DTYXBQOGGV5THFUPHJUWGSM5IMJQ753VE4MX4IPOONQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.610317849 (556h11m8.481400894s) [noise=true , address=145.79.227.44:4216 , piece_id=GSBWC2LXUOZYOVD55KNQYA5Z3S2TCYROBB5O2XZIIVFUQMJQMYNA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.611969885 (556h11m8.48305292s) [noise=true , address=145.79.227.34:4916 , piece_id=SPYSFWMJ752FC5JGRE36G54S2FHG4DQFCV6CIZONQHJDXDCN52VQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.636098801 (556h11m8.507181836s) [noise=true , address=145.79.227.30:4784 , piece_id=MKCRSFWVFVGOAYCK5M66HO3OT7OMYWLVR2KMDXUGBS2VXXNCUSSQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.636122136 (556h11m8.507205181s) [seg_pos={0 0}, upload=487609]: Segment reached optimal threshold of 14 pieces. Dec 28 19:40:08.636333929 (556h11m8.507416974s) [noise=true , address=64.49.13.26:4956 , piece_id=FFHUPOOM7PLEABRHFH6H2SI6WXJBU2XFCLZJUBV4CXUJ6EP4PI2Q, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TYHhjpfyBKkenSwviyQCUZpSBbnaFCGBFRkM4fyKLDuxKx4Ju): protocol: expected piece hash; context canceled; EOF Dec 28 19:40:08.636342175 (556h11m8.50742521s) [noise=true , address=64.49.13.26:4956 , piece_id=FFHUPOOM7PLEABRHFH6H2SI6WXJBU2XFCLZJUBV4CXUJ6EP4PI2Q, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636585959 (556h11m8.507668994s) [noise=true , address=145.79.227.43:4163 , piece_id=77BKLLVD25UMTDQZW46CA2N225I7O4HCHCRRPQ7HLJGRHOLGSDHA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CgPQ5hkrJdqbJvGarxKjXvYYnrXQqapL5BH2rNJNvnhpRi3pw): protocol: expected piece hash; context canceled; EOF Dec 28 19:40:08.636592172 (556h11m8.507675207s) [noise=true , address=145.79.227.43:4163 , piece_id=77BKLLVD25UMTDQZW46CA2N225I7O4HCHCRRPQ7HLJGRHOLGSDHA, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636681511 (556h11m8.507764546s) [noise=true , address=145.79.227.19:4083 , piece_id=GL5XL7T2QQABD5364AT4AFV6RZNOWCTVNSMAVIFJU5SUQFHOPZKQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KZ5a6CQ9tUTrC6EYE7M8yg6p9fK95CN8zQLQmP4tdyz4rBfGh): protocol: expected piece hash; context canceled; EOF Dec 28 19:40:08.636687052 (556h11m8.507770087s) [noise=true , address=145.79.227.19:4083 , piece_id=GL5XL7T2QQABD5364AT4AFV6RZNOWCTVNSMAVIFJU5SUQFHOPZKQ, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636810327 (556h11m8.507893362s) [noise=true , address=145.79.227.17:4992 , piece_id=GUF7AUFCFRRPY5PSYS7DRPFQLGT4ZEL5C7P23US2VI54KYWSUCUQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hbt8HKTunpPKyAzH4XBwGQqTKzcgqPpoWwKZm9JMtaK8P7aVp): protocol: write tcp 188.166.225.170:51210->145.79.227.17:4992: use of closed network connection; write tcp 188.166.225.170:51210->145.79.227.17:4992: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51210->145.79.227.17:4992: use of closed network connection Dec 28 19:40:08.636815957 (556h11m8.507899002s) [noise=true , address=145.79.227.17:4992 , piece_id=GUF7AUFCFRRPY5PSYS7DRPFQLGT4ZEL5C7P23US2VI54KYWSUCUQ, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636834313 (556h11m8.507917358s) [seg_pos={0 0}, upload=487609]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 19:40:08.636907953 (556h11m8.507990998s) [upload=487609]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 19:40:08.636913644 (556h11m8.507996689s) [upload=487609]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 19:40:08.924991397 (556h11m8.796074442s) [upload=487609]: Done uploading segments... Dec 28 19:40:08.925014742 (556h11m8.796097777s) [upload=487609]: Upload finished. err: <nil> Duration: 33.782970192s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....XX..............X..........................X...........] duration: 6.183793205s trace: https://jaeger.dev.storj.io/trace/3057620e25d11143 FAIL (2025-12-28 20:22:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 20:13:34.173546860 (556h44m34.044629895s) [upload=488098]: Starting upload Dec 28 20:13:34.173605531 (556h44m34.044688566s) [upload=488098]: Done starting upload Dec 28 20:13:34.173822354 (556h44m34.04490539s) [upload=488098]: Uploading segments... Dec 28 20:13:34.173860978 (556h44m34.044944013s) [upload=488098]: Got next segment. Inline: false Dec 28 20:13:34.173892047 (556h44m34.044975082s) [upload=488098]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 20:13:34.173896415 (556h44m34.04497945s) [upload=488098]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 20:13:34.422688850 (556h44m34.293771905s) [seg_pos={0 0}, upload=488098]: Begin upload segment... Dec 28 20:13:34.422745879 (556h44m34.293828944s) [seg_pos={0 0}, upload=488098]: Done begin upload segment. Dec 28 20:13:34.422771076 (556h44m34.293854141s) [noise=true , address=145.79.227.16:4909 , piece_id=G7U4H2HJWK4VGY42RUBKYGDBLRDZLWX6K772ULG2YAH5OVZ5JPKA, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.422842654 (556h44m34.293925699s) [noise=true , address=145.79.227.37:4566 , piece_id=AXTGBOXYJDN7QRFJWEMU3WBEF74KYERZS2B2SZXSPLZRU2TYC4JA, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.423548137 (556h44m34.294631192s) [noise=true , address=145.79.227.11:4730 , piece_id=6AXVYQY7IIQYL4OHBO3URZFJCC42ZWL3NEMD3JSFHLYMXNTQBMUQ, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.423555861 (556h44m34.294638906s) [noise=true , address=45.91.116.10:4081 , piece_id=GU7LAF6GLPTVIEOQ6GRHZI4VYYZMQUPKMAPOFBFCRQB344CUKKMQ, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.424230055 (556h44m34.29531309s) [noise=true , address=45.91.116.6:4976 , piece_id=4GCSWP27NMVW6VQFIOFBGBKHCZEB5VWC32CXMKKLVGXXFK3EHBXQ, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.424619267 (556h44m34.295702302s) [noise=true , address=145.79.227.36:4519 , piece_id=EJBDJEJAN7WG3HHTXBUEIW7QZ2VY6ZOXKNHBLHNYJ7L6ZVH6M7YA, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.424778910 (556h44m34.295861955s) [noise=true , address=77.90.31.11:4556 , piece_id=FANBXJICXHI33JVWYCLDDB76KSYXXFBTX4MLYXE6ICXQP2JAJAJA, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.425200092 (556h44m34.296283137s) [noise=true , address=77.90.31.18:4808 , piece_id=FIOQVOLXKF2CFNJ57BTCL4WTDPZPEEGAS2CDPF2PNTY27JVFTG6Q, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.425605254 (556h44m34.296688289s) [noise=true , address=77.90.31.17:4770 , piece_id=EA2QWTOVMLD5HKF6FGTB3TZXZZLSI4UVFTPD3MZZ2PAXF47VGJQA, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.426052345 (556h44m34.29713538s) [noise=true , address=145.79.227.51:4411 , piece_id=NHVE27MXBV6Y4VNYIT6G73S4WDAWVKACBQIOKTIOCGELPMLCY4CQ, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.426486622 (556h44m34.297569657s) [noise=true , address=45.91.116.13:4153 , piece_id=XCGSWRC2CZ2NLTPN5T5XTK4VHGIES67NB3FFCBXGI63DLGQ7V2EQ, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.426897945 (556h44m34.29798098s) [noise=true , address=45.91.116.4:4937 , piece_id=V6H4VACBOTZZMXQKCQRPJOBRFQVRF562HZC2GQGBLQGAGKFY5VNQ, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.427411794 (556h44m34.298494829s) [noise=true , address=145.79.227.45:4229 , piece_id=IRJAI3XSHMQYCOIFOMRNETTBXLLAEL3Z3YBOLPGEM2FOBF6S37XQ, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.427894844 (556h44m34.298977879s) [noise=true , address=145.79.227.21:4473 , piece_id=KAYPSFAGCFG3DUA4MVZAAHSH65MCNPDOH2M7GTAYILZO6EPFPXYQ, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.431922001 (556h44m34.303005036s) [noise=true , address=145.79.227.50:4374 , piece_id=DCLHZ4GR6U56WATYQRY7YFCFHO4JYMGLXSBY2OWEM3HSGPJHLDSA, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.436931210 (556h44m34.308014255s) [noise=true , address=145.79.227.38:4979 , piece_id=OUNZYJAAA32ZMIQOOCHK4YF4DCF354XUW4WUCHM3445666VXVLBQ, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:36.115225165 (556h44m35.98630821s) [upload=488098]: Next returned nil segment Dec 28 20:13:36.115243359 (556h44m35.986326395s) [upload=488098]: Waiting for error group managing segments... Dec 28 20:13:36.415982301 (556h44m36.287065346s) [noise=true , address=145.79.227.51:4411 , piece_id=NHVE27MXBV6Y4VNYIT6G73S4WDAWVKACBQIOKTIOCGELPMLCY4CQ, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.430495527 (556h44m36.301578562s) [noise=true , address=145.79.227.45:4229 , piece_id=IRJAI3XSHMQYCOIFOMRNETTBXLLAEL3Z3YBOLPGEM2FOBF6S37XQ, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.435192370 (556h44m36.306275406s) [noise=true , address=145.79.227.21:4473 , piece_id=KAYPSFAGCFG3DUA4MVZAAHSH65MCNPDOH2M7GTAYILZO6EPFPXYQ, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.435831557 (556h44m36.306914602s) [noise=true , address=145.79.227.38:4979 , piece_id=OUNZYJAAA32ZMIQOOCHK4YF4DCF354XUW4WUCHM3445666VXVLBQ, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.444124790 (556h44m36.315207825s) [noise=true , address=145.79.227.50:4374 , piece_id=DCLHZ4GR6U56WATYQRY7YFCFHO4JYMGLXSBY2OWEM3HSGPJHLDSA, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.450320487 (556h44m36.321403533s) [noise=true , address=145.79.227.11:4730 , piece_id=6AXVYQY7IIQYL4OHBO3URZFJCC42ZWL3NEMD3JSFHLYMXNTQBMUQ, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.452177964 (556h44m36.323260999s) [noise=true , address=145.79.227.36:4519 , piece_id=EJBDJEJAN7WG3HHTXBUEIW7QZ2VY6ZOXKNHBLHNYJ7L6ZVH6M7YA, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.462219385 (556h44m36.33330242s) [noise=true , address=145.79.227.37:4566 , piece_id=AXTGBOXYJDN7QRFJWEMU3WBEF74KYERZS2B2SZXSPLZRU2TYC4JA, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.485557775 (556h44m36.35664081s) [noise=true , address=145.79.227.16:4909 , piece_id=G7U4H2HJWK4VGY42RUBKYGDBLRDZLWX6K772ULG2YAH5OVZ5JPKA, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.583906940 (556h44m36.454989975s) [noise=true , address=77.90.31.11:4556 , piece_id=FANBXJICXHI33JVWYCLDDB76KSYXXFBTX4MLYXE6ICXQP2JAJAJA, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.658185680 (556h44m36.529268725s) [noise=true , address=45.91.116.4:4937 , piece_id=V6H4VACBOTZZMXQKCQRPJOBRFQVRF562HZC2GQGBLQGAGKFY5VNQ, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.715312460 (556h44m36.586395495s) [noise=true , address=45.91.116.13:4153 , piece_id=XCGSWRC2CZ2NLTPN5T5XTK4VHGIES67NB3FFCBXGI63DLGQ7V2EQ, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.737246156 (556h44m36.608329201s) [noise=true , address=45.91.116.10:4081 , piece_id=GU7LAF6GLPTVIEOQ6GRHZI4VYYZMQUPKMAPOFBFCRQB344CUKKMQ, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.737270231 (556h44m36.608353266s) [seg_pos={0 0}, upload=488098]: Segment reached optimal threshold of 13 pieces. Dec 28 20:13:36.737506872 (556h44m36.608589927s) [noise=true , address=77.90.31.18:4808 , piece_id=FIOQVOLXKF2CFNJ57BTCL4WTDPZPEEGAS2CDPF2PNTY27JVFTG6Q, seg_pos={0 0}, upload=488098]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: expected piece hash; context canceled; EOF Dec 28 20:13:36.737516099 (556h44m36.608599134s) [noise=true , address=77.90.31.18:4808 , piece_id=FIOQVOLXKF2CFNJ57BTCL4WTDPZPEEGAS2CDPF2PNTY27JVFTG6Q, seg_pos={0 0}, upload=488098]: Long tail cancellation due to optimal threshold Dec 28 20:13:36.737659642 (556h44m36.608742688s) [noise=true , address=45.91.116.6:4976 , piece_id=4GCSWP27NMVW6VQFIOFBGBKHCZEB5VWC32CXMKKLVGXXFK3EHBXQ, seg_pos={0 0}, upload=488098]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Dec 28 20:13:36.737707214 (556h44m36.608790259s) [noise=true , address=45.91.116.6:4976 , piece_id=4GCSWP27NMVW6VQFIOFBGBKHCZEB5VWC32CXMKKLVGXXFK3EHBXQ, seg_pos={0 0}, upload=488098]: Long tail cancellation due to optimal threshold Dec 28 20:13:36.737939255 (556h44m36.6090223s) [noise=true , address=77.90.31.17:4770 , piece_id=EA2QWTOVMLD5HKF6FGTB3TZXZZLSI4UVFTPD3MZZ2PAXF47VGJQA, seg_pos={0 0}, upload=488098]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: write tcp 188.166.225.170:53958->77.90.31.17:4770: use of closed network connection; write tcp 188.166.225.170:53958->77.90.31.17:4770: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53958->77.90.31.17:4770: use of closed network connection Dec 28 20:13:36.737965024 (556h44m36.609048079s) [noise=true , address=77.90.31.17:4770 , piece_id=EA2QWTOVMLD5HKF6FGTB3TZXZZLSI4UVFTPD3MZZ2PAXF47VGJQA, seg_pos={0 0}, upload=488098]: Long tail cancellation due to optimal threshold Dec 28 20:13:36.737999810 (556h44m36.609082846s) [seg_pos={0 0}, upload=488098]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 20:13:36.738101905 (556h44m36.60918495s) [upload=488098]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 20:13:36.738109249 (556h44m36.609192304s) [upload=488098]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 20:13:37.032410772 (556h44m36.903493817s) [upload=488098]: Done uploading segments... Dec 28 20:13:37.032430300 (556h44m36.903513345s) [upload=488098]: Upload finished. err: <nil> Duration: 6.183819305s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [......XX..............X..........................X..........] duration: 5.812176341s trace: https://jaeger.dev.storj.io/trace/3057620e25d11143 FAIL (2025-12-28 20:21:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 20:13:34.173546860 (556h44m34.044629895s) [upload=488098]: Starting upload Dec 28 20:13:34.173605531 (556h44m34.044688566s) [upload=488098]: Done starting upload Dec 28 20:13:34.173822354 (556h44m34.04490539s) [upload=488098]: Uploading segments... Dec 28 20:13:34.173860978 (556h44m34.044944013s) [upload=488098]: Got next segment. Inline: false Dec 28 20:13:34.173892047 (556h44m34.044975082s) [upload=488098]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 20:13:34.173896415 (556h44m34.04497945s) [upload=488098]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 20:13:34.422688850 (556h44m34.293771905s) [seg_pos={0 0}, upload=488098]: Begin upload segment... Dec 28 20:13:34.422745879 (556h44m34.293828944s) [seg_pos={0 0}, upload=488098]: Done begin upload segment. Dec 28 20:13:34.422771076 (556h44m34.293854141s) [noise=true , address=145.79.227.16:4909 , piece_id=G7U4H2HJWK4VGY42RUBKYGDBLRDZLWX6K772ULG2YAH5OVZ5JPKA, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.422842654 (556h44m34.293925699s) [noise=true , address=145.79.227.37:4566 , piece_id=AXTGBOXYJDN7QRFJWEMU3WBEF74KYERZS2B2SZXSPLZRU2TYC4JA, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.423548137 (556h44m34.294631192s) [noise=true , address=145.79.227.11:4730 , piece_id=6AXVYQY7IIQYL4OHBO3URZFJCC42ZWL3NEMD3JSFHLYMXNTQBMUQ, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.423555861 (556h44m34.294638906s) [noise=true , address=45.91.116.10:4081 , piece_id=GU7LAF6GLPTVIEOQ6GRHZI4VYYZMQUPKMAPOFBFCRQB344CUKKMQ, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.424230055 (556h44m34.29531309s) [noise=true , address=45.91.116.6:4976 , piece_id=4GCSWP27NMVW6VQFIOFBGBKHCZEB5VWC32CXMKKLVGXXFK3EHBXQ, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.424619267 (556h44m34.295702302s) [noise=true , address=145.79.227.36:4519 , piece_id=EJBDJEJAN7WG3HHTXBUEIW7QZ2VY6ZOXKNHBLHNYJ7L6ZVH6M7YA, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.424778910 (556h44m34.295861955s) [noise=true , address=77.90.31.11:4556 , piece_id=FANBXJICXHI33JVWYCLDDB76KSYXXFBTX4MLYXE6ICXQP2JAJAJA, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.425200092 (556h44m34.296283137s) [noise=true , address=77.90.31.18:4808 , piece_id=FIOQVOLXKF2CFNJ57BTCL4WTDPZPEEGAS2CDPF2PNTY27JVFTG6Q, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.425605254 (556h44m34.296688289s) [noise=true , address=77.90.31.17:4770 , piece_id=EA2QWTOVMLD5HKF6FGTB3TZXZZLSI4UVFTPD3MZZ2PAXF47VGJQA, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.426052345 (556h44m34.29713538s) [noise=true , address=145.79.227.51:4411 , piece_id=NHVE27MXBV6Y4VNYIT6G73S4WDAWVKACBQIOKTIOCGELPMLCY4CQ, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.426486622 (556h44m34.297569657s) [noise=true , address=45.91.116.13:4153 , piece_id=XCGSWRC2CZ2NLTPN5T5XTK4VHGIES67NB3FFCBXGI63DLGQ7V2EQ, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.426897945 (556h44m34.29798098s) [noise=true , address=45.91.116.4:4937 , piece_id=V6H4VACBOTZZMXQKCQRPJOBRFQVRF562HZC2GQGBLQGAGKFY5VNQ, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.427411794 (556h44m34.298494829s) [noise=true , address=145.79.227.45:4229 , piece_id=IRJAI3XSHMQYCOIFOMRNETTBXLLAEL3Z3YBOLPGEM2FOBF6S37XQ, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.427894844 (556h44m34.298977879s) [noise=true , address=145.79.227.21:4473 , piece_id=KAYPSFAGCFG3DUA4MVZAAHSH65MCNPDOH2M7GTAYILZO6EPFPXYQ, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.431922001 (556h44m34.303005036s) [noise=true , address=145.79.227.50:4374 , piece_id=DCLHZ4GR6U56WATYQRY7YFCFHO4JYMGLXSBY2OWEM3HSGPJHLDSA, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.436931210 (556h44m34.308014255s) [noise=true , address=145.79.227.38:4979 , piece_id=OUNZYJAAA32ZMIQOOCHK4YF4DCF354XUW4WUCHM3445666VXVLBQ, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:36.115225165 (556h44m35.98630821s) [upload=488098]: Next returned nil segment Dec 28 20:13:36.115243359 (556h44m35.986326395s) [upload=488098]: Waiting for error group managing segments... Dec 28 20:13:36.415982301 (556h44m36.287065346s) [noise=true , address=145.79.227.51:4411 , piece_id=NHVE27MXBV6Y4VNYIT6G73S4WDAWVKACBQIOKTIOCGELPMLCY4CQ, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.430495527 (556h44m36.301578562s) [noise=true , address=145.79.227.45:4229 , piece_id=IRJAI3XSHMQYCOIFOMRNETTBXLLAEL3Z3YBOLPGEM2FOBF6S37XQ, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.435192370 (556h44m36.306275406s) [noise=true , address=145.79.227.21:4473 , piece_id=KAYPSFAGCFG3DUA4MVZAAHSH65MCNPDOH2M7GTAYILZO6EPFPXYQ, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.435831557 (556h44m36.306914602s) [noise=true , address=145.79.227.38:4979 , piece_id=OUNZYJAAA32ZMIQOOCHK4YF4DCF354XUW4WUCHM3445666VXVLBQ, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.444124790 (556h44m36.315207825s) [noise=true , address=145.79.227.50:4374 , piece_id=DCLHZ4GR6U56WATYQRY7YFCFHO4JYMGLXSBY2OWEM3HSGPJHLDSA, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.450320487 (556h44m36.321403533s) [noise=true , address=145.79.227.11:4730 , piece_id=6AXVYQY7IIQYL4OHBO3URZFJCC42ZWL3NEMD3JSFHLYMXNTQBMUQ, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.452177964 (556h44m36.323260999s) [noise=true , address=145.79.227.36:4519 , piece_id=EJBDJEJAN7WG3HHTXBUEIW7QZ2VY6ZOXKNHBLHNYJ7L6ZVH6M7YA, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.462219385 (556h44m36.33330242s) [noise=true , address=145.79.227.37:4566 , piece_id=AXTGBOXYJDN7QRFJWEMU3WBEF74KYERZS2B2SZXSPLZRU2TYC4JA, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.485557775 (556h44m36.35664081s) [noise=true , address=145.79.227.16:4909 , piece_id=G7U4H2HJWK4VGY42RUBKYGDBLRDZLWX6K772ULG2YAH5OVZ5JPKA, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.583906940 (556h44m36.454989975s) [noise=true , address=77.90.31.11:4556 , piece_id=FANBXJICXHI33JVWYCLDDB76KSYXXFBTX4MLYXE6ICXQP2JAJAJA, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.658185680 (556h44m36.529268725s) [noise=true , address=45.91.116.4:4937 , piece_id=V6H4VACBOTZZMXQKCQRPJOBRFQVRF562HZC2GQGBLQGAGKFY5VNQ, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.715312460 (556h44m36.586395495s) [noise=true , address=45.91.116.13:4153 , piece_id=XCGSWRC2CZ2NLTPN5T5XTK4VHGIES67NB3FFCBXGI63DLGQ7V2EQ, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.737246156 (556h44m36.608329201s) [noise=true , address=45.91.116.10:4081 , piece_id=GU7LAF6GLPTVIEOQ6GRHZI4VYYZMQUPKMAPOFBFCRQB344CUKKMQ, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.737270231 (556h44m36.608353266s) [seg_pos={0 0}, upload=488098]: Segment reached optimal threshold of 13 pieces. Dec 28 20:13:36.737506872 (556h44m36.608589927s) [noise=true , address=77.90.31.18:4808 , piece_id=FIOQVOLXKF2CFNJ57BTCL4WTDPZPEEGAS2CDPF2PNTY27JVFTG6Q, seg_pos={0 0}, upload=488098]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: expected piece hash; context canceled; EOF Dec 28 20:13:36.737516099 (556h44m36.608599134s) [noise=true , address=77.90.31.18:4808 , piece_id=FIOQVOLXKF2CFNJ57BTCL4WTDPZPEEGAS2CDPF2PNTY27JVFTG6Q, seg_pos={0 0}, upload=488098]: Long tail cancellation due to optimal threshold Dec 28 20:13:36.737659642 (556h44m36.608742688s) [noise=true , address=45.91.116.6:4976 , piece_id=4GCSWP27NMVW6VQFIOFBGBKHCZEB5VWC32CXMKKLVGXXFK3EHBXQ, seg_pos={0 0}, upload=488098]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Dec 28 20:13:36.737707214 (556h44m36.608790259s) [noise=true , address=45.91.116.6:4976 , piece_id=4GCSWP27NMVW6VQFIOFBGBKHCZEB5VWC32CXMKKLVGXXFK3EHBXQ, seg_pos={0 0}, upload=488098]: Long tail cancellation due to optimal threshold Dec 28 20:13:36.737939255 (556h44m36.6090223s) [noise=true , address=77.90.31.17:4770 , piece_id=EA2QWTOVMLD5HKF6FGTB3TZXZZLSI4UVFTPD3MZZ2PAXF47VGJQA, seg_pos={0 0}, upload=488098]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: write tcp 188.166.225.170:53958->77.90.31.17:4770: use of closed network connection; write tcp 188.166.225.170:53958->77.90.31.17:4770: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53958->77.90.31.17:4770: use of closed network connection Dec 28 20:13:36.737965024 (556h44m36.609048079s) [noise=true , address=77.90.31.17:4770 , piece_id=EA2QWTOVMLD5HKF6FGTB3TZXZZLSI4UVFTPD3MZZ2PAXF47VGJQA, seg_pos={0 0}, upload=488098]: Long tail cancellation due to optimal threshold Dec 28 20:13:36.737999810 (556h44m36.609082846s) [seg_pos={0 0}, upload=488098]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 20:13:36.738101905 (556h44m36.60918495s) [upload=488098]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 20:13:36.738109249 (556h44m36.609192304s) [upload=488098]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 20:13:37.032410772 (556h44m36.903493817s) [upload=488098]: Done uploading segments... Dec 28 20:13:37.032430300 (556h44m36.903513345s) [upload=488098]: Upload finished. err: <nil> Duration: 5.812201249s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 22.636146526s trace: https://jaeger.dev.storj.io/trace/10e1641b228b2a1f FAIL (2025-12-28 20:21:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 40:05.744568479 (556h11m5.615651514s) [noise=true , address=145.79.227.28:4744 , piece_id=42LII6EOG3CMBTIL5I4RZQPDYU52TKJN3PGWEQXROBYMNDFYIIVA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.745099351 (556h11m5.616182396s) [noise=true , address=145.79.227.23:4550 , piece_id=NWIC722LZUHJSAMPWSKCLPRIDGWPVUEYMS5KRAMX3IIMD2KR4XGQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.745777401 (556h11m5.616860436s) [noise=true , address=77.90.31.15:4722 , piece_id=AF7F2OA247CJIDFTEU6HEOIYGOINEYPMII53BKJQXI7UZ26MJIUA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.746374739 (556h11m5.617457784s) [noise=true , address=77.90.31.10:4522 , piece_id=JOHOMHCNYXF4XQGLN2NJUSWSUO3XFBRRMYD27SLIZAK2RCT3ZC2A, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.746934555 (556h11m5.618017601s) [noise=true , address=77.90.31.21:4896 , piece_id=ATTC7BZUT4HAAEAZCHFZQE4INV2LLSWR2YCUNVA73LPJCW3YL35A, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.747556861 (556h11m5.618640146s) [noise=true , address=64.49.13.26:4956 , piece_id=FFHUPOOM7PLEABRHFH6H2SI6WXJBU2XFCLZJUBV4CXUJ6EP4PI2Q, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.748964150 (556h11m5.620047195s) [noise=true , address=145.79.227.34:4916 , piece_id=SPYSFWMJ752FC5JGRE36G54S2FHG4DQFCV6CIZONQHJDXDCN52VQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.751119314 (556h11m5.622202359s) [noise=true , address=77.90.31.18:4816 , piece_id=YT3PSYNQTDQBVBRUDNOXJ2XCN6EFQA54WBSNL6MZMXFXR7XNCHIA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.753413222 (556h11m5.624496257s) [noise=true , address=77.90.31.11:4582 , piece_id=ZRE24R27TX4E5DTV2POX7EJANQ4CM75D6ZOUS6SKM3QHHQCKRBRA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.755230482 (556h11m5.626313527s) [noise=true , address=64.49.13.154:4932 , piece_id=QB5WM5RZ2P2GC6RIGGM34IANVEI2SNDJ2PO556JXKTDBX5HLD4YA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.757598362 (556h11m5.628681407s) [noise=true , address=145.79.227.35:4510 , piece_id=P275TU367DTYXBQOGGV5THFUPHJUWGSM5IMJQ753VE4MX4IPOONQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.759644679 (556h11m5.630727734s) [noise=true , address=145.79.227.46:4626 , piece_id=OQOTWYLVIU7GJBBLYWQOEEFKR3TNAUOXZS6UAFJWJ32SIOTXNXEA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.761217594 (556h11m5.632300639s) [noise=true , address=145.79.227.43:4163 , piece_id=77BKLLVD25UMTDQZW46CA2N225I7O4HCHCRRPQ7HLJGRHOLGSDHA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.762138507 (556h11m5.633221552s) [noise=true , address=77.90.31.19:4860 , piece_id=5FUYVK2Z6LBZOAGFI2NOTAAUIV4NMBI5ERMA5RH4BMONNHABZLDQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.764304321 (556h11m5.635387356s) [noise=true , address=145.79.227.30:4784 , piece_id=MKCRSFWVFVGOAYCK5M66HO3OT7OMYWLVR2KMDXUGBS2VXXNCUSSQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.765421217 (556h11m5.636504252s) [noise=true , address=145.79.227.44:4216 , piece_id=GSBWC2LXUOZYOVD55KNQYA5Z3S2TCYROBB5O2XZIIVFUQMJQMYNA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:07.082099826 (556h11m6.953182871s) [upload=487609]: Next returned nil segment Dec 28 19:40:07.082116707 (556h11m6.953199752s) [upload=487609]: Waiting for error group managing segments... Dec 28 19:40:07.415884144 (556h11m7.286967179s) [noise=true , address=64.49.13.154:4932 , piece_id=QB5WM5RZ2P2GC6RIGGM34IANVEI2SNDJ2PO556JXKTDBX5HLD4YA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.615434252 (556h11m7.486517297s) [noise=true , address=145.79.227.23:4550 , piece_id=NWIC722LZUHJSAMPWSKCLPRIDGWPVUEYMS5KRAMX3IIMD2KR4XGQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.647182077 (556h11m7.518265112s) [noise=true , address=145.79.227.28:4744 , piece_id=42LII6EOG3CMBTIL5I4RZQPDYU52TKJN3PGWEQXROBYMNDFYIIVA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.678458624 (556h11m7.549541659s) [noise=true , address=145.79.227.46:4626 , piece_id=OQOTWYLVIU7GJBBLYWQOEEFKR3TNAUOXZS6UAFJWJ32SIOTXNXEA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.793406049 (556h11m7.664489084s) [noise=true , address=77.90.31.19:4860 , piece_id=5FUYVK2Z6LBZOAGFI2NOTAAUIV4NMBI5ERMA5RH4BMONNHABZLDQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.802506498 (556h11m7.673589543s) [noise=true , address=77.90.31.21:4896 , piece_id=ATTC7BZUT4HAAEAZCHFZQE4INV2LLSWR2YCUNVA73LPJCW3YL35A, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.853663242 (556h11m7.724746287s) [noise=true , address=77.90.31.10:4522 , piece_id=JOHOMHCNYXF4XQGLN2NJUSWSUO3XFBRRMYD27SLIZAK2RCT3ZC2A, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.880639500 (556h11m7.751722535s) [noise=true , address=77.90.31.11:4582 , piece_id=ZRE24R27TX4E5DTV2POX7EJANQ4CM75D6ZOUS6SKM3QHHQCKRBRA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.904835446 (556h11m7.775918491s) [noise=true , address=77.90.31.18:4816 , piece_id=YT3PSYNQTDQBVBRUDNOXJ2XCN6EFQA54WBSNL6MZMXFXR7XNCHIA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.988914260 (556h11m7.859997295s) [noise=true , address=77.90.31.15:4722 , piece_id=AF7F2OA247CJIDFTEU6HEOIYGOINEYPMII53BKJQXI7UZ26MJIUA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.484927449 (556h11m8.356010484s) [noise=true , address=145.79.227.35:4510 , piece_id=P275TU367DTYXBQOGGV5THFUPHJUWGSM5IMJQ753VE4MX4IPOONQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.610317849 (556h11m8.481400894s) [noise=true , address=145.79.227.44:4216 , piece_id=GSBWC2LXUOZYOVD55KNQYA5Z3S2TCYROBB5O2XZIIVFUQMJQMYNA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.611969885 (556h11m8.48305292s) [noise=true , address=145.79.227.34:4916 , piece_id=SPYSFWMJ752FC5JGRE36G54S2FHG4DQFCV6CIZONQHJDXDCN52VQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.636098801 (556h11m8.507181836s) [noise=true , address=145.79.227.30:4784 , piece_id=MKCRSFWVFVGOAYCK5M66HO3OT7OMYWLVR2KMDXUGBS2VXXNCUSSQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.636122136 (556h11m8.507205181s) [seg_pos={0 0}, upload=487609]: Segment reached optimal threshold of 14 pieces. Dec 28 19:40:08.636333929 (556h11m8.507416974s) [noise=true , address=64.49.13.26:4956 , piece_id=FFHUPOOM7PLEABRHFH6H2SI6WXJBU2XFCLZJUBV4CXUJ6EP4PI2Q, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TYHhjpfyBKkenSwviyQCUZpSBbnaFCGBFRkM4fyKLDuxKx4Ju): protocol: expected piece hash; context canceled; EOF Dec 28 19:40:08.636342175 (556h11m8.50742521s) [noise=true , address=64.49.13.26:4956 , piece_id=FFHUPOOM7PLEABRHFH6H2SI6WXJBU2XFCLZJUBV4CXUJ6EP4PI2Q, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636585959 (556h11m8.507668994s) [noise=true , address=145.79.227.43:4163 , piece_id=77BKLLVD25UMTDQZW46CA2N225I7O4HCHCRRPQ7HLJGRHOLGSDHA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CgPQ5hkrJdqbJvGarxKjXvYYnrXQqapL5BH2rNJNvnhpRi3pw): protocol: expected piece hash; context canceled; EOF Dec 28 19:40:08.636592172 (556h11m8.507675207s) [noise=true , address=145.79.227.43:4163 , piece_id=77BKLLVD25UMTDQZW46CA2N225I7O4HCHCRRPQ7HLJGRHOLGSDHA, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636681511 (556h11m8.507764546s) [noise=true , address=145.79.227.19:4083 , piece_id=GL5XL7T2QQABD5364AT4AFV6RZNOWCTVNSMAVIFJU5SUQFHOPZKQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KZ5a6CQ9tUTrC6EYE7M8yg6p9fK95CN8zQLQmP4tdyz4rBfGh): protocol: expected piece hash; context canceled; EOF Dec 28 19:40:08.636687052 (556h11m8.507770087s) [noise=true , address=145.79.227.19:4083 , piece_id=GL5XL7T2QQABD5364AT4AFV6RZNOWCTVNSMAVIFJU5SUQFHOPZKQ, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636810327 (556h11m8.507893362s) [noise=true , address=145.79.227.17:4992 , piece_id=GUF7AUFCFRRPY5PSYS7DRPFQLGT4ZEL5C7P23US2VI54KYWSUCUQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hbt8HKTunpPKyAzH4XBwGQqTKzcgqPpoWwKZm9JMtaK8P7aVp): protocol: write tcp 188.166.225.170:51210->145.79.227.17:4992: use of closed network connection; write tcp 188.166.225.170:51210->145.79.227.17:4992: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51210->145.79.227.17:4992: use of closed network connection Dec 28 19:40:08.636815957 (556h11m8.507899002s) [noise=true , address=145.79.227.17:4992 , piece_id=GUF7AUFCFRRPY5PSYS7DRPFQLGT4ZEL5C7P23US2VI54KYWSUCUQ, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636834313 (556h11m8.507917358s) [seg_pos={0 0}, upload=487609]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 19:40:08.636907953 (556h11m8.507990998s) [upload=487609]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 19:40:08.636913644 (556h11m8.507996689s) [upload=487609]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 19:40:08.924991397 (556h11m8.796074442s) [upload=487609]: Done uploading segments... Dec 28 19:40:08.925014742 (556h11m8.796097777s) [upload=487609]: Upload finished. err: <nil> Duration: 22.636259311s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [X......XX..............X..........................X.........] duration: 6.81720316s trace: https://jaeger.dev.storj.io/trace/3057620e25d11143 FAIL (2025-12-28 20:21:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 20:13:34.173546860 (556h44m34.044629895s) [upload=488098]: Starting upload Dec 28 20:13:34.173605531 (556h44m34.044688566s) [upload=488098]: Done starting upload Dec 28 20:13:34.173822354 (556h44m34.04490539s) [upload=488098]: Uploading segments... Dec 28 20:13:34.173860978 (556h44m34.044944013s) [upload=488098]: Got next segment. Inline: false Dec 28 20:13:34.173892047 (556h44m34.044975082s) [upload=488098]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 20:13:34.173896415 (556h44m34.04497945s) [upload=488098]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 20:13:34.422688850 (556h44m34.293771905s) [seg_pos={0 0}, upload=488098]: Begin upload segment... Dec 28 20:13:34.422745879 (556h44m34.293828944s) [seg_pos={0 0}, upload=488098]: Done begin upload segment. Dec 28 20:13:34.422771076 (556h44m34.293854141s) [noise=true , address=145.79.227.16:4909 , piece_id=G7U4H2HJWK4VGY42RUBKYGDBLRDZLWX6K772ULG2YAH5OVZ5JPKA, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.422842654 (556h44m34.293925699s) [noise=true , address=145.79.227.37:4566 , piece_id=AXTGBOXYJDN7QRFJWEMU3WBEF74KYERZS2B2SZXSPLZRU2TYC4JA, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.423548137 (556h44m34.294631192s) [noise=true , address=145.79.227.11:4730 , piece_id=6AXVYQY7IIQYL4OHBO3URZFJCC42ZWL3NEMD3JSFHLYMXNTQBMUQ, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.423555861 (556h44m34.294638906s) [noise=true , address=45.91.116.10:4081 , piece_id=GU7LAF6GLPTVIEOQ6GRHZI4VYYZMQUPKMAPOFBFCRQB344CUKKMQ, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.424230055 (556h44m34.29531309s) [noise=true , address=45.91.116.6:4976 , piece_id=4GCSWP27NMVW6VQFIOFBGBKHCZEB5VWC32CXMKKLVGXXFK3EHBXQ, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.424619267 (556h44m34.295702302s) [noise=true , address=145.79.227.36:4519 , piece_id=EJBDJEJAN7WG3HHTXBUEIW7QZ2VY6ZOXKNHBLHNYJ7L6ZVH6M7YA, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.424778910 (556h44m34.295861955s) [noise=true , address=77.90.31.11:4556 , piece_id=FANBXJICXHI33JVWYCLDDB76KSYXXFBTX4MLYXE6ICXQP2JAJAJA, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.425200092 (556h44m34.296283137s) [noise=true , address=77.90.31.18:4808 , piece_id=FIOQVOLXKF2CFNJ57BTCL4WTDPZPEEGAS2CDPF2PNTY27JVFTG6Q, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.425605254 (556h44m34.296688289s) [noise=true , address=77.90.31.17:4770 , piece_id=EA2QWTOVMLD5HKF6FGTB3TZXZZLSI4UVFTPD3MZZ2PAXF47VGJQA, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.426052345 (556h44m34.29713538s) [noise=true , address=145.79.227.51:4411 , piece_id=NHVE27MXBV6Y4VNYIT6G73S4WDAWVKACBQIOKTIOCGELPMLCY4CQ, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.426486622 (556h44m34.297569657s) [noise=true , address=45.91.116.13:4153 , piece_id=XCGSWRC2CZ2NLTPN5T5XTK4VHGIES67NB3FFCBXGI63DLGQ7V2EQ, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.426897945 (556h44m34.29798098s) [noise=true , address=45.91.116.4:4937 , piece_id=V6H4VACBOTZZMXQKCQRPJOBRFQVRF562HZC2GQGBLQGAGKFY5VNQ, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.427411794 (556h44m34.298494829s) [noise=true , address=145.79.227.45:4229 , piece_id=IRJAI3XSHMQYCOIFOMRNETTBXLLAEL3Z3YBOLPGEM2FOBF6S37XQ, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.427894844 (556h44m34.298977879s) [noise=true , address=145.79.227.21:4473 , piece_id=KAYPSFAGCFG3DUA4MVZAAHSH65MCNPDOH2M7GTAYILZO6EPFPXYQ, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.431922001 (556h44m34.303005036s) [noise=true , address=145.79.227.50:4374 , piece_id=DCLHZ4GR6U56WATYQRY7YFCFHO4JYMGLXSBY2OWEM3HSGPJHLDSA, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.436931210 (556h44m34.308014255s) [noise=true , address=145.79.227.38:4979 , piece_id=OUNZYJAAA32ZMIQOOCHK4YF4DCF354XUW4WUCHM3445666VXVLBQ, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:36.115225165 (556h44m35.98630821s) [upload=488098]: Next returned nil segment Dec 28 20:13:36.115243359 (556h44m35.986326395s) [upload=488098]: Waiting for error group managing segments... Dec 28 20:13:36.415982301 (556h44m36.287065346s) [noise=true , address=145.79.227.51:4411 , piece_id=NHVE27MXBV6Y4VNYIT6G73S4WDAWVKACBQIOKTIOCGELPMLCY4CQ, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.430495527 (556h44m36.301578562s) [noise=true , address=145.79.227.45:4229 , piece_id=IRJAI3XSHMQYCOIFOMRNETTBXLLAEL3Z3YBOLPGEM2FOBF6S37XQ, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.435192370 (556h44m36.306275406s) [noise=true , address=145.79.227.21:4473 , piece_id=KAYPSFAGCFG3DUA4MVZAAHSH65MCNPDOH2M7GTAYILZO6EPFPXYQ, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.435831557 (556h44m36.306914602s) [noise=true , address=145.79.227.38:4979 , piece_id=OUNZYJAAA32ZMIQOOCHK4YF4DCF354XUW4WUCHM3445666VXVLBQ, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.444124790 (556h44m36.315207825s) [noise=true , address=145.79.227.50:4374 , piece_id=DCLHZ4GR6U56WATYQRY7YFCFHO4JYMGLXSBY2OWEM3HSGPJHLDSA, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.450320487 (556h44m36.321403533s) [noise=true , address=145.79.227.11:4730 , piece_id=6AXVYQY7IIQYL4OHBO3URZFJCC42ZWL3NEMD3JSFHLYMXNTQBMUQ, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.452177964 (556h44m36.323260999s) [noise=true , address=145.79.227.36:4519 , piece_id=EJBDJEJAN7WG3HHTXBUEIW7QZ2VY6ZOXKNHBLHNYJ7L6ZVH6M7YA, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.462219385 (556h44m36.33330242s) [noise=true , address=145.79.227.37:4566 , piece_id=AXTGBOXYJDN7QRFJWEMU3WBEF74KYERZS2B2SZXSPLZRU2TYC4JA, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.485557775 (556h44m36.35664081s) [noise=true , address=145.79.227.16:4909 , piece_id=G7U4H2HJWK4VGY42RUBKYGDBLRDZLWX6K772ULG2YAH5OVZ5JPKA, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.583906940 (556h44m36.454989975s) [noise=true , address=77.90.31.11:4556 , piece_id=FANBXJICXHI33JVWYCLDDB76KSYXXFBTX4MLYXE6ICXQP2JAJAJA, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.658185680 (556h44m36.529268725s) [noise=true , address=45.91.116.4:4937 , piece_id=V6H4VACBOTZZMXQKCQRPJOBRFQVRF562HZC2GQGBLQGAGKFY5VNQ, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.715312460 (556h44m36.586395495s) [noise=true , address=45.91.116.13:4153 , piece_id=XCGSWRC2CZ2NLTPN5T5XTK4VHGIES67NB3FFCBXGI63DLGQ7V2EQ, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.737246156 (556h44m36.608329201s) [noise=true , address=45.91.116.10:4081 , piece_id=GU7LAF6GLPTVIEOQ6GRHZI4VYYZMQUPKMAPOFBFCRQB344CUKKMQ, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.737270231 (556h44m36.608353266s) [seg_pos={0 0}, upload=488098]: Segment reached optimal threshold of 13 pieces. Dec 28 20:13:36.737506872 (556h44m36.608589927s) [noise=true , address=77.90.31.18:4808 , piece_id=FIOQVOLXKF2CFNJ57BTCL4WTDPZPEEGAS2CDPF2PNTY27JVFTG6Q, seg_pos={0 0}, upload=488098]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: expected piece hash; context canceled; EOF Dec 28 20:13:36.737516099 (556h44m36.608599134s) [noise=true , address=77.90.31.18:4808 , piece_id=FIOQVOLXKF2CFNJ57BTCL4WTDPZPEEGAS2CDPF2PNTY27JVFTG6Q, seg_pos={0 0}, upload=488098]: Long tail cancellation due to optimal threshold Dec 28 20:13:36.737659642 (556h44m36.608742688s) [noise=true , address=45.91.116.6:4976 , piece_id=4GCSWP27NMVW6VQFIOFBGBKHCZEB5VWC32CXMKKLVGXXFK3EHBXQ, seg_pos={0 0}, upload=488098]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Dec 28 20:13:36.737707214 (556h44m36.608790259s) [noise=true , address=45.91.116.6:4976 , piece_id=4GCSWP27NMVW6VQFIOFBGBKHCZEB5VWC32CXMKKLVGXXFK3EHBXQ, seg_pos={0 0}, upload=488098]: Long tail cancellation due to optimal threshold Dec 28 20:13:36.737939255 (556h44m36.6090223s) [noise=true , address=77.90.31.17:4770 , piece_id=EA2QWTOVMLD5HKF6FGTB3TZXZZLSI4UVFTPD3MZZ2PAXF47VGJQA, seg_pos={0 0}, upload=488098]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: write tcp 188.166.225.170:53958->77.90.31.17:4770: use of closed network connection; write tcp 188.166.225.170:53958->77.90.31.17:4770: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53958->77.90.31.17:4770: use of closed network connection Dec 28 20:13:36.737965024 (556h44m36.609048079s) [noise=true , address=77.90.31.17:4770 , piece_id=EA2QWTOVMLD5HKF6FGTB3TZXZZLSI4UVFTPD3MZZ2PAXF47VGJQA, seg_pos={0 0}, upload=488098]: Long tail cancellation due to optimal threshold Dec 28 20:13:36.737999810 (556h44m36.609082846s) [seg_pos={0 0}, upload=488098]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 20:13:36.738101905 (556h44m36.60918495s) [upload=488098]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 20:13:36.738109249 (556h44m36.609192304s) [upload=488098]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 20:13:37.032410772 (556h44m36.903493817s) [upload=488098]: Done uploading segments... Dec 28 20:13:37.032430300 (556h44m36.903513345s) [upload=488098]: Upload finished. err: <nil> Duration: 6.817229641s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.X......XX..............X..........................X........] duration: 8.69455815s trace: https://jaeger.dev.storj.io/trace/3057620e25d11143 FAIL (2025-12-28 20:21:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 20:13:34.173546860 (556h44m34.044629895s) [upload=488098]: Starting upload Dec 28 20:13:34.173605531 (556h44m34.044688566s) [upload=488098]: Done starting upload Dec 28 20:13:34.173822354 (556h44m34.04490539s) [upload=488098]: Uploading segments... Dec 28 20:13:34.173860978 (556h44m34.044944013s) [upload=488098]: Got next segment. Inline: false Dec 28 20:13:34.173892047 (556h44m34.044975082s) [upload=488098]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 20:13:34.173896415 (556h44m34.04497945s) [upload=488098]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 20:13:34.422688850 (556h44m34.293771905s) [seg_pos={0 0}, upload=488098]: Begin upload segment... Dec 28 20:13:34.422745879 (556h44m34.293828944s) [seg_pos={0 0}, upload=488098]: Done begin upload segment. Dec 28 20:13:34.422771076 (556h44m34.293854141s) [noise=true , address=145.79.227.16:4909 , piece_id=G7U4H2HJWK4VGY42RUBKYGDBLRDZLWX6K772ULG2YAH5OVZ5JPKA, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.422842654 (556h44m34.293925699s) [noise=true , address=145.79.227.37:4566 , piece_id=AXTGBOXYJDN7QRFJWEMU3WBEF74KYERZS2B2SZXSPLZRU2TYC4JA, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.423548137 (556h44m34.294631192s) [noise=true , address=145.79.227.11:4730 , piece_id=6AXVYQY7IIQYL4OHBO3URZFJCC42ZWL3NEMD3JSFHLYMXNTQBMUQ, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.423555861 (556h44m34.294638906s) [noise=true , address=45.91.116.10:4081 , piece_id=GU7LAF6GLPTVIEOQ6GRHZI4VYYZMQUPKMAPOFBFCRQB344CUKKMQ, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.424230055 (556h44m34.29531309s) [noise=true , address=45.91.116.6:4976 , piece_id=4GCSWP27NMVW6VQFIOFBGBKHCZEB5VWC32CXMKKLVGXXFK3EHBXQ, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.424619267 (556h44m34.295702302s) [noise=true , address=145.79.227.36:4519 , piece_id=EJBDJEJAN7WG3HHTXBUEIW7QZ2VY6ZOXKNHBLHNYJ7L6ZVH6M7YA, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.424778910 (556h44m34.295861955s) [noise=true , address=77.90.31.11:4556 , piece_id=FANBXJICXHI33JVWYCLDDB76KSYXXFBTX4MLYXE6ICXQP2JAJAJA, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.425200092 (556h44m34.296283137s) [noise=true , address=77.90.31.18:4808 , piece_id=FIOQVOLXKF2CFNJ57BTCL4WTDPZPEEGAS2CDPF2PNTY27JVFTG6Q, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.425605254 (556h44m34.296688289s) [noise=true , address=77.90.31.17:4770 , piece_id=EA2QWTOVMLD5HKF6FGTB3TZXZZLSI4UVFTPD3MZZ2PAXF47VGJQA, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.426052345 (556h44m34.29713538s) [noise=true , address=145.79.227.51:4411 , piece_id=NHVE27MXBV6Y4VNYIT6G73S4WDAWVKACBQIOKTIOCGELPMLCY4CQ, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.426486622 (556h44m34.297569657s) [noise=true , address=45.91.116.13:4153 , piece_id=XCGSWRC2CZ2NLTPN5T5XTK4VHGIES67NB3FFCBXGI63DLGQ7V2EQ, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.426897945 (556h44m34.29798098s) [noise=true , address=45.91.116.4:4937 , piece_id=V6H4VACBOTZZMXQKCQRPJOBRFQVRF562HZC2GQGBLQGAGKFY5VNQ, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.427411794 (556h44m34.298494829s) [noise=true , address=145.79.227.45:4229 , piece_id=IRJAI3XSHMQYCOIFOMRNETTBXLLAEL3Z3YBOLPGEM2FOBF6S37XQ, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.427894844 (556h44m34.298977879s) [noise=true , address=145.79.227.21:4473 , piece_id=KAYPSFAGCFG3DUA4MVZAAHSH65MCNPDOH2M7GTAYILZO6EPFPXYQ, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.431922001 (556h44m34.303005036s) [noise=true , address=145.79.227.50:4374 , piece_id=DCLHZ4GR6U56WATYQRY7YFCFHO4JYMGLXSBY2OWEM3HSGPJHLDSA, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.436931210 (556h44m34.308014255s) [noise=true , address=145.79.227.38:4979 , piece_id=OUNZYJAAA32ZMIQOOCHK4YF4DCF354XUW4WUCHM3445666VXVLBQ, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:36.115225165 (556h44m35.98630821s) [upload=488098]: Next returned nil segment Dec 28 20:13:36.115243359 (556h44m35.986326395s) [upload=488098]: Waiting for error group managing segments... Dec 28 20:13:36.415982301 (556h44m36.287065346s) [noise=true , address=145.79.227.51:4411 , piece_id=NHVE27MXBV6Y4VNYIT6G73S4WDAWVKACBQIOKTIOCGELPMLCY4CQ, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.430495527 (556h44m36.301578562s) [noise=true , address=145.79.227.45:4229 , piece_id=IRJAI3XSHMQYCOIFOMRNETTBXLLAEL3Z3YBOLPGEM2FOBF6S37XQ, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.435192370 (556h44m36.306275406s) [noise=true , address=145.79.227.21:4473 , piece_id=KAYPSFAGCFG3DUA4MVZAAHSH65MCNPDOH2M7GTAYILZO6EPFPXYQ, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.435831557 (556h44m36.306914602s) [noise=true , address=145.79.227.38:4979 , piece_id=OUNZYJAAA32ZMIQOOCHK4YF4DCF354XUW4WUCHM3445666VXVLBQ, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.444124790 (556h44m36.315207825s) [noise=true , address=145.79.227.50:4374 , piece_id=DCLHZ4GR6U56WATYQRY7YFCFHO4JYMGLXSBY2OWEM3HSGPJHLDSA, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.450320487 (556h44m36.321403533s) [noise=true , address=145.79.227.11:4730 , piece_id=6AXVYQY7IIQYL4OHBO3URZFJCC42ZWL3NEMD3JSFHLYMXNTQBMUQ, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.452177964 (556h44m36.323260999s) [noise=true , address=145.79.227.36:4519 , piece_id=EJBDJEJAN7WG3HHTXBUEIW7QZ2VY6ZOXKNHBLHNYJ7L6ZVH6M7YA, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.462219385 (556h44m36.33330242s) [noise=true , address=145.79.227.37:4566 , piece_id=AXTGBOXYJDN7QRFJWEMU3WBEF74KYERZS2B2SZXSPLZRU2TYC4JA, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.485557775 (556h44m36.35664081s) [noise=true , address=145.79.227.16:4909 , piece_id=G7U4H2HJWK4VGY42RUBKYGDBLRDZLWX6K772ULG2YAH5OVZ5JPKA, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.583906940 (556h44m36.454989975s) [noise=true , address=77.90.31.11:4556 , piece_id=FANBXJICXHI33JVWYCLDDB76KSYXXFBTX4MLYXE6ICXQP2JAJAJA, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.658185680 (556h44m36.529268725s) [noise=true , address=45.91.116.4:4937 , piece_id=V6H4VACBOTZZMXQKCQRPJOBRFQVRF562HZC2GQGBLQGAGKFY5VNQ, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.715312460 (556h44m36.586395495s) [noise=true , address=45.91.116.13:4153 , piece_id=XCGSWRC2CZ2NLTPN5T5XTK4VHGIES67NB3FFCBXGI63DLGQ7V2EQ, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.737246156 (556h44m36.608329201s) [noise=true , address=45.91.116.10:4081 , piece_id=GU7LAF6GLPTVIEOQ6GRHZI4VYYZMQUPKMAPOFBFCRQB344CUKKMQ, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.737270231 (556h44m36.608353266s) [seg_pos={0 0}, upload=488098]: Segment reached optimal threshold of 13 pieces. Dec 28 20:13:36.737506872 (556h44m36.608589927s) [noise=true , address=77.90.31.18:4808 , piece_id=FIOQVOLXKF2CFNJ57BTCL4WTDPZPEEGAS2CDPF2PNTY27JVFTG6Q, seg_pos={0 0}, upload=488098]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: expected piece hash; context canceled; EOF Dec 28 20:13:36.737516099 (556h44m36.608599134s) [noise=true , address=77.90.31.18:4808 , piece_id=FIOQVOLXKF2CFNJ57BTCL4WTDPZPEEGAS2CDPF2PNTY27JVFTG6Q, seg_pos={0 0}, upload=488098]: Long tail cancellation due to optimal threshold Dec 28 20:13:36.737659642 (556h44m36.608742688s) [noise=true , address=45.91.116.6:4976 , piece_id=4GCSWP27NMVW6VQFIOFBGBKHCZEB5VWC32CXMKKLVGXXFK3EHBXQ, seg_pos={0 0}, upload=488098]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Dec 28 20:13:36.737707214 (556h44m36.608790259s) [noise=true , address=45.91.116.6:4976 , piece_id=4GCSWP27NMVW6VQFIOFBGBKHCZEB5VWC32CXMKKLVGXXFK3EHBXQ, seg_pos={0 0}, upload=488098]: Long tail cancellation due to optimal threshold Dec 28 20:13:36.737939255 (556h44m36.6090223s) [noise=true , address=77.90.31.17:4770 , piece_id=EA2QWTOVMLD5HKF6FGTB3TZXZZLSI4UVFTPD3MZZ2PAXF47VGJQA, seg_pos={0 0}, upload=488098]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: write tcp 188.166.225.170:53958->77.90.31.17:4770: use of closed network connection; write tcp 188.166.225.170:53958->77.90.31.17:4770: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53958->77.90.31.17:4770: use of closed network connection Dec 28 20:13:36.737965024 (556h44m36.609048079s) [noise=true , address=77.90.31.17:4770 , piece_id=EA2QWTOVMLD5HKF6FGTB3TZXZZLSI4UVFTPD3MZZ2PAXF47VGJQA, seg_pos={0 0}, upload=488098]: Long tail cancellation due to optimal threshold Dec 28 20:13:36.737999810 (556h44m36.609082846s) [seg_pos={0 0}, upload=488098]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 20:13:36.738101905 (556h44m36.60918495s) [upload=488098]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 20:13:36.738109249 (556h44m36.609192304s) [upload=488098]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 20:13:37.032410772 (556h44m36.903493817s) [upload=488098]: Done uploading segments... Dec 28 20:13:37.032430300 (556h44m36.903513345s) [upload=488098]: Upload finished. err: <nil> Duration: 8.694586424s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 30.89221837s trace: https://jaeger.dev.storj.io/trace/10e1641b228b2a1f FAIL (2025-12-28 20:20:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 40:05.744568479 (556h11m5.615651514s) [noise=true , address=145.79.227.28:4744 , piece_id=42LII6EOG3CMBTIL5I4RZQPDYU52TKJN3PGWEQXROBYMNDFYIIVA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.745099351 (556h11m5.616182396s) [noise=true , address=145.79.227.23:4550 , piece_id=NWIC722LZUHJSAMPWSKCLPRIDGWPVUEYMS5KRAMX3IIMD2KR4XGQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.745777401 (556h11m5.616860436s) [noise=true , address=77.90.31.15:4722 , piece_id=AF7F2OA247CJIDFTEU6HEOIYGOINEYPMII53BKJQXI7UZ26MJIUA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.746374739 (556h11m5.617457784s) [noise=true , address=77.90.31.10:4522 , piece_id=JOHOMHCNYXF4XQGLN2NJUSWSUO3XFBRRMYD27SLIZAK2RCT3ZC2A, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.746934555 (556h11m5.618017601s) [noise=true , address=77.90.31.21:4896 , piece_id=ATTC7BZUT4HAAEAZCHFZQE4INV2LLSWR2YCUNVA73LPJCW3YL35A, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.747556861 (556h11m5.618640146s) [noise=true , address=64.49.13.26:4956 , piece_id=FFHUPOOM7PLEABRHFH6H2SI6WXJBU2XFCLZJUBV4CXUJ6EP4PI2Q, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.748964150 (556h11m5.620047195s) [noise=true , address=145.79.227.34:4916 , piece_id=SPYSFWMJ752FC5JGRE36G54S2FHG4DQFCV6CIZONQHJDXDCN52VQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.751119314 (556h11m5.622202359s) [noise=true , address=77.90.31.18:4816 , piece_id=YT3PSYNQTDQBVBRUDNOXJ2XCN6EFQA54WBSNL6MZMXFXR7XNCHIA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.753413222 (556h11m5.624496257s) [noise=true , address=77.90.31.11:4582 , piece_id=ZRE24R27TX4E5DTV2POX7EJANQ4CM75D6ZOUS6SKM3QHHQCKRBRA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.755230482 (556h11m5.626313527s) [noise=true , address=64.49.13.154:4932 , piece_id=QB5WM5RZ2P2GC6RIGGM34IANVEI2SNDJ2PO556JXKTDBX5HLD4YA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.757598362 (556h11m5.628681407s) [noise=true , address=145.79.227.35:4510 , piece_id=P275TU367DTYXBQOGGV5THFUPHJUWGSM5IMJQ753VE4MX4IPOONQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.759644679 (556h11m5.630727734s) [noise=true , address=145.79.227.46:4626 , piece_id=OQOTWYLVIU7GJBBLYWQOEEFKR3TNAUOXZS6UAFJWJ32SIOTXNXEA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.761217594 (556h11m5.632300639s) [noise=true , address=145.79.227.43:4163 , piece_id=77BKLLVD25UMTDQZW46CA2N225I7O4HCHCRRPQ7HLJGRHOLGSDHA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.762138507 (556h11m5.633221552s) [noise=true , address=77.90.31.19:4860 , piece_id=5FUYVK2Z6LBZOAGFI2NOTAAUIV4NMBI5ERMA5RH4BMONNHABZLDQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.764304321 (556h11m5.635387356s) [noise=true , address=145.79.227.30:4784 , piece_id=MKCRSFWVFVGOAYCK5M66HO3OT7OMYWLVR2KMDXUGBS2VXXNCUSSQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.765421217 (556h11m5.636504252s) [noise=true , address=145.79.227.44:4216 , piece_id=GSBWC2LXUOZYOVD55KNQYA5Z3S2TCYROBB5O2XZIIVFUQMJQMYNA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:07.082099826 (556h11m6.953182871s) [upload=487609]: Next returned nil segment Dec 28 19:40:07.082116707 (556h11m6.953199752s) [upload=487609]: Waiting for error group managing segments... Dec 28 19:40:07.415884144 (556h11m7.286967179s) [noise=true , address=64.49.13.154:4932 , piece_id=QB5WM5RZ2P2GC6RIGGM34IANVEI2SNDJ2PO556JXKTDBX5HLD4YA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.615434252 (556h11m7.486517297s) [noise=true , address=145.79.227.23:4550 , piece_id=NWIC722LZUHJSAMPWSKCLPRIDGWPVUEYMS5KRAMX3IIMD2KR4XGQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.647182077 (556h11m7.518265112s) [noise=true , address=145.79.227.28:4744 , piece_id=42LII6EOG3CMBTIL5I4RZQPDYU52TKJN3PGWEQXROBYMNDFYIIVA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.678458624 (556h11m7.549541659s) [noise=true , address=145.79.227.46:4626 , piece_id=OQOTWYLVIU7GJBBLYWQOEEFKR3TNAUOXZS6UAFJWJ32SIOTXNXEA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.793406049 (556h11m7.664489084s) [noise=true , address=77.90.31.19:4860 , piece_id=5FUYVK2Z6LBZOAGFI2NOTAAUIV4NMBI5ERMA5RH4BMONNHABZLDQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.802506498 (556h11m7.673589543s) [noise=true , address=77.90.31.21:4896 , piece_id=ATTC7BZUT4HAAEAZCHFZQE4INV2LLSWR2YCUNVA73LPJCW3YL35A, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.853663242 (556h11m7.724746287s) [noise=true , address=77.90.31.10:4522 , piece_id=JOHOMHCNYXF4XQGLN2NJUSWSUO3XFBRRMYD27SLIZAK2RCT3ZC2A, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.880639500 (556h11m7.751722535s) [noise=true , address=77.90.31.11:4582 , piece_id=ZRE24R27TX4E5DTV2POX7EJANQ4CM75D6ZOUS6SKM3QHHQCKRBRA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.904835446 (556h11m7.775918491s) [noise=true , address=77.90.31.18:4816 , piece_id=YT3PSYNQTDQBVBRUDNOXJ2XCN6EFQA54WBSNL6MZMXFXR7XNCHIA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.988914260 (556h11m7.859997295s) [noise=true , address=77.90.31.15:4722 , piece_id=AF7F2OA247CJIDFTEU6HEOIYGOINEYPMII53BKJQXI7UZ26MJIUA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.484927449 (556h11m8.356010484s) [noise=true , address=145.79.227.35:4510 , piece_id=P275TU367DTYXBQOGGV5THFUPHJUWGSM5IMJQ753VE4MX4IPOONQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.610317849 (556h11m8.481400894s) [noise=true , address=145.79.227.44:4216 , piece_id=GSBWC2LXUOZYOVD55KNQYA5Z3S2TCYROBB5O2XZIIVFUQMJQMYNA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.611969885 (556h11m8.48305292s) [noise=true , address=145.79.227.34:4916 , piece_id=SPYSFWMJ752FC5JGRE36G54S2FHG4DQFCV6CIZONQHJDXDCN52VQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.636098801 (556h11m8.507181836s) [noise=true , address=145.79.227.30:4784 , piece_id=MKCRSFWVFVGOAYCK5M66HO3OT7OMYWLVR2KMDXUGBS2VXXNCUSSQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.636122136 (556h11m8.507205181s) [seg_pos={0 0}, upload=487609]: Segment reached optimal threshold of 14 pieces. Dec 28 19:40:08.636333929 (556h11m8.507416974s) [noise=true , address=64.49.13.26:4956 , piece_id=FFHUPOOM7PLEABRHFH6H2SI6WXJBU2XFCLZJUBV4CXUJ6EP4PI2Q, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TYHhjpfyBKkenSwviyQCUZpSBbnaFCGBFRkM4fyKLDuxKx4Ju): protocol: expected piece hash; context canceled; EOF Dec 28 19:40:08.636342175 (556h11m8.50742521s) [noise=true , address=64.49.13.26:4956 , piece_id=FFHUPOOM7PLEABRHFH6H2SI6WXJBU2XFCLZJUBV4CXUJ6EP4PI2Q, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636585959 (556h11m8.507668994s) [noise=true , address=145.79.227.43:4163 , piece_id=77BKLLVD25UMTDQZW46CA2N225I7O4HCHCRRPQ7HLJGRHOLGSDHA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CgPQ5hkrJdqbJvGarxKjXvYYnrXQqapL5BH2rNJNvnhpRi3pw): protocol: expected piece hash; context canceled; EOF Dec 28 19:40:08.636592172 (556h11m8.507675207s) [noise=true , address=145.79.227.43:4163 , piece_id=77BKLLVD25UMTDQZW46CA2N225I7O4HCHCRRPQ7HLJGRHOLGSDHA, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636681511 (556h11m8.507764546s) [noise=true , address=145.79.227.19:4083 , piece_id=GL5XL7T2QQABD5364AT4AFV6RZNOWCTVNSMAVIFJU5SUQFHOPZKQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KZ5a6CQ9tUTrC6EYE7M8yg6p9fK95CN8zQLQmP4tdyz4rBfGh): protocol: expected piece hash; context canceled; EOF Dec 28 19:40:08.636687052 (556h11m8.507770087s) [noise=true , address=145.79.227.19:4083 , piece_id=GL5XL7T2QQABD5364AT4AFV6RZNOWCTVNSMAVIFJU5SUQFHOPZKQ, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636810327 (556h11m8.507893362s) [noise=true , address=145.79.227.17:4992 , piece_id=GUF7AUFCFRRPY5PSYS7DRPFQLGT4ZEL5C7P23US2VI54KYWSUCUQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hbt8HKTunpPKyAzH4XBwGQqTKzcgqPpoWwKZm9JMtaK8P7aVp): protocol: write tcp 188.166.225.170:51210->145.79.227.17:4992: use of closed network connection; write tcp 188.166.225.170:51210->145.79.227.17:4992: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51210->145.79.227.17:4992: use of closed network connection Dec 28 19:40:08.636815957 (556h11m8.507899002s) [noise=true , address=145.79.227.17:4992 , piece_id=GUF7AUFCFRRPY5PSYS7DRPFQLGT4ZEL5C7P23US2VI54KYWSUCUQ, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636834313 (556h11m8.507917358s) [seg_pos={0 0}, upload=487609]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 19:40:08.636907953 (556h11m8.507990998s) [upload=487609]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 19:40:08.636913644 (556h11m8.507996689s) [upload=487609]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 19:40:08.924991397 (556h11m8.796074442s) [upload=487609]: Done uploading segments... Dec 28 19:40:08.925014742 (556h11m8.796097777s) [upload=487609]: Upload finished. err: <nil> Duration: 30.89224537s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..X......XX..............X..........................X.......] duration: 6.024945205s trace: https://jaeger.dev.storj.io/trace/3057620e25d11143 FAIL (2025-12-28 20:19:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 20:13:34.173546860 (556h44m34.044629895s) [upload=488098]: Starting upload Dec 28 20:13:34.173605531 (556h44m34.044688566s) [upload=488098]: Done starting upload Dec 28 20:13:34.173822354 (556h44m34.04490539s) [upload=488098]: Uploading segments... Dec 28 20:13:34.173860978 (556h44m34.044944013s) [upload=488098]: Got next segment. Inline: false Dec 28 20:13:34.173892047 (556h44m34.044975082s) [upload=488098]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 20:13:34.173896415 (556h44m34.04497945s) [upload=488098]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 20:13:34.422688850 (556h44m34.293771905s) [seg_pos={0 0}, upload=488098]: Begin upload segment... Dec 28 20:13:34.422745879 (556h44m34.293828944s) [seg_pos={0 0}, upload=488098]: Done begin upload segment. Dec 28 20:13:34.422771076 (556h44m34.293854141s) [noise=true , address=145.79.227.16:4909 , piece_id=G7U4H2HJWK4VGY42RUBKYGDBLRDZLWX6K772ULG2YAH5OVZ5JPKA, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.422842654 (556h44m34.293925699s) [noise=true , address=145.79.227.37:4566 , piece_id=AXTGBOXYJDN7QRFJWEMU3WBEF74KYERZS2B2SZXSPLZRU2TYC4JA, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.423548137 (556h44m34.294631192s) [noise=true , address=145.79.227.11:4730 , piece_id=6AXVYQY7IIQYL4OHBO3URZFJCC42ZWL3NEMD3JSFHLYMXNTQBMUQ, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.423555861 (556h44m34.294638906s) [noise=true , address=45.91.116.10:4081 , piece_id=GU7LAF6GLPTVIEOQ6GRHZI4VYYZMQUPKMAPOFBFCRQB344CUKKMQ, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.424230055 (556h44m34.29531309s) [noise=true , address=45.91.116.6:4976 , piece_id=4GCSWP27NMVW6VQFIOFBGBKHCZEB5VWC32CXMKKLVGXXFK3EHBXQ, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.424619267 (556h44m34.295702302s) [noise=true , address=145.79.227.36:4519 , piece_id=EJBDJEJAN7WG3HHTXBUEIW7QZ2VY6ZOXKNHBLHNYJ7L6ZVH6M7YA, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.424778910 (556h44m34.295861955s) [noise=true , address=77.90.31.11:4556 , piece_id=FANBXJICXHI33JVWYCLDDB76KSYXXFBTX4MLYXE6ICXQP2JAJAJA, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.425200092 (556h44m34.296283137s) [noise=true , address=77.90.31.18:4808 , piece_id=FIOQVOLXKF2CFNJ57BTCL4WTDPZPEEGAS2CDPF2PNTY27JVFTG6Q, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.425605254 (556h44m34.296688289s) [noise=true , address=77.90.31.17:4770 , piece_id=EA2QWTOVMLD5HKF6FGTB3TZXZZLSI4UVFTPD3MZZ2PAXF47VGJQA, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.426052345 (556h44m34.29713538s) [noise=true , address=145.79.227.51:4411 , piece_id=NHVE27MXBV6Y4VNYIT6G73S4WDAWVKACBQIOKTIOCGELPMLCY4CQ, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.426486622 (556h44m34.297569657s) [noise=true , address=45.91.116.13:4153 , piece_id=XCGSWRC2CZ2NLTPN5T5XTK4VHGIES67NB3FFCBXGI63DLGQ7V2EQ, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.426897945 (556h44m34.29798098s) [noise=true , address=45.91.116.4:4937 , piece_id=V6H4VACBOTZZMXQKCQRPJOBRFQVRF562HZC2GQGBLQGAGKFY5VNQ, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.427411794 (556h44m34.298494829s) [noise=true , address=145.79.227.45:4229 , piece_id=IRJAI3XSHMQYCOIFOMRNETTBXLLAEL3Z3YBOLPGEM2FOBF6S37XQ, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.427894844 (556h44m34.298977879s) [noise=true , address=145.79.227.21:4473 , piece_id=KAYPSFAGCFG3DUA4MVZAAHSH65MCNPDOH2M7GTAYILZO6EPFPXYQ, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.431922001 (556h44m34.303005036s) [noise=true , address=145.79.227.50:4374 , piece_id=DCLHZ4GR6U56WATYQRY7YFCFHO4JYMGLXSBY2OWEM3HSGPJHLDSA, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.436931210 (556h44m34.308014255s) [noise=true , address=145.79.227.38:4979 , piece_id=OUNZYJAAA32ZMIQOOCHK4YF4DCF354XUW4WUCHM3445666VXVLBQ, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:36.115225165 (556h44m35.98630821s) [upload=488098]: Next returned nil segment Dec 28 20:13:36.115243359 (556h44m35.986326395s) [upload=488098]: Waiting for error group managing segments... Dec 28 20:13:36.415982301 (556h44m36.287065346s) [noise=true , address=145.79.227.51:4411 , piece_id=NHVE27MXBV6Y4VNYIT6G73S4WDAWVKACBQIOKTIOCGELPMLCY4CQ, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.430495527 (556h44m36.301578562s) [noise=true , address=145.79.227.45:4229 , piece_id=IRJAI3XSHMQYCOIFOMRNETTBXLLAEL3Z3YBOLPGEM2FOBF6S37XQ, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.435192370 (556h44m36.306275406s) [noise=true , address=145.79.227.21:4473 , piece_id=KAYPSFAGCFG3DUA4MVZAAHSH65MCNPDOH2M7GTAYILZO6EPFPXYQ, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.435831557 (556h44m36.306914602s) [noise=true , address=145.79.227.38:4979 , piece_id=OUNZYJAAA32ZMIQOOCHK4YF4DCF354XUW4WUCHM3445666VXVLBQ, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.444124790 (556h44m36.315207825s) [noise=true , address=145.79.227.50:4374 , piece_id=DCLHZ4GR6U56WATYQRY7YFCFHO4JYMGLXSBY2OWEM3HSGPJHLDSA, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.450320487 (556h44m36.321403533s) [noise=true , address=145.79.227.11:4730 , piece_id=6AXVYQY7IIQYL4OHBO3URZFJCC42ZWL3NEMD3JSFHLYMXNTQBMUQ, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.452177964 (556h44m36.323260999s) [noise=true , address=145.79.227.36:4519 , piece_id=EJBDJEJAN7WG3HHTXBUEIW7QZ2VY6ZOXKNHBLHNYJ7L6ZVH6M7YA, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.462219385 (556h44m36.33330242s) [noise=true , address=145.79.227.37:4566 , piece_id=AXTGBOXYJDN7QRFJWEMU3WBEF74KYERZS2B2SZXSPLZRU2TYC4JA, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.485557775 (556h44m36.35664081s) [noise=true , address=145.79.227.16:4909 , piece_id=G7U4H2HJWK4VGY42RUBKYGDBLRDZLWX6K772ULG2YAH5OVZ5JPKA, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.583906940 (556h44m36.454989975s) [noise=true , address=77.90.31.11:4556 , piece_id=FANBXJICXHI33JVWYCLDDB76KSYXXFBTX4MLYXE6ICXQP2JAJAJA, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.658185680 (556h44m36.529268725s) [noise=true , address=45.91.116.4:4937 , piece_id=V6H4VACBOTZZMXQKCQRPJOBRFQVRF562HZC2GQGBLQGAGKFY5VNQ, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.715312460 (556h44m36.586395495s) [noise=true , address=45.91.116.13:4153 , piece_id=XCGSWRC2CZ2NLTPN5T5XTK4VHGIES67NB3FFCBXGI63DLGQ7V2EQ, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.737246156 (556h44m36.608329201s) [noise=true , address=45.91.116.10:4081 , piece_id=GU7LAF6GLPTVIEOQ6GRHZI4VYYZMQUPKMAPOFBFCRQB344CUKKMQ, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.737270231 (556h44m36.608353266s) [seg_pos={0 0}, upload=488098]: Segment reached optimal threshold of 13 pieces. Dec 28 20:13:36.737506872 (556h44m36.608589927s) [noise=true , address=77.90.31.18:4808 , piece_id=FIOQVOLXKF2CFNJ57BTCL4WTDPZPEEGAS2CDPF2PNTY27JVFTG6Q, seg_pos={0 0}, upload=488098]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: expected piece hash; context canceled; EOF Dec 28 20:13:36.737516099 (556h44m36.608599134s) [noise=true , address=77.90.31.18:4808 , piece_id=FIOQVOLXKF2CFNJ57BTCL4WTDPZPEEGAS2CDPF2PNTY27JVFTG6Q, seg_pos={0 0}, upload=488098]: Long tail cancellation due to optimal threshold Dec 28 20:13:36.737659642 (556h44m36.608742688s) [noise=true , address=45.91.116.6:4976 , piece_id=4GCSWP27NMVW6VQFIOFBGBKHCZEB5VWC32CXMKKLVGXXFK3EHBXQ, seg_pos={0 0}, upload=488098]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Dec 28 20:13:36.737707214 (556h44m36.608790259s) [noise=true , address=45.91.116.6:4976 , piece_id=4GCSWP27NMVW6VQFIOFBGBKHCZEB5VWC32CXMKKLVGXXFK3EHBXQ, seg_pos={0 0}, upload=488098]: Long tail cancellation due to optimal threshold Dec 28 20:13:36.737939255 (556h44m36.6090223s) [noise=true , address=77.90.31.17:4770 , piece_id=EA2QWTOVMLD5HKF6FGTB3TZXZZLSI4UVFTPD3MZZ2PAXF47VGJQA, seg_pos={0 0}, upload=488098]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: write tcp 188.166.225.170:53958->77.90.31.17:4770: use of closed network connection; write tcp 188.166.225.170:53958->77.90.31.17:4770: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53958->77.90.31.17:4770: use of closed network connection Dec 28 20:13:36.737965024 (556h44m36.609048079s) [noise=true , address=77.90.31.17:4770 , piece_id=EA2QWTOVMLD5HKF6FGTB3TZXZZLSI4UVFTPD3MZZ2PAXF47VGJQA, seg_pos={0 0}, upload=488098]: Long tail cancellation due to optimal threshold Dec 28 20:13:36.737999810 (556h44m36.609082846s) [seg_pos={0 0}, upload=488098]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 20:13:36.738101905 (556h44m36.60918495s) [upload=488098]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 20:13:36.738109249 (556h44m36.609192304s) [upload=488098]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 20:13:37.032410772 (556h44m36.903493817s) [upload=488098]: Done uploading segments... Dec 28 20:13:37.032430300 (556h44m36.903513345s) [upload=488098]: Upload finished. err: <nil> Duration: 6.02503259s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 24.460722735s trace: https://jaeger.dev.storj.io/trace/10e1641b228b2a1f FAIL (2025-12-28 20:19:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 40:05.744568479 (556h11m5.615651514s) [noise=true , address=145.79.227.28:4744 , piece_id=42LII6EOG3CMBTIL5I4RZQPDYU52TKJN3PGWEQXROBYMNDFYIIVA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.745099351 (556h11m5.616182396s) [noise=true , address=145.79.227.23:4550 , piece_id=NWIC722LZUHJSAMPWSKCLPRIDGWPVUEYMS5KRAMX3IIMD2KR4XGQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.745777401 (556h11m5.616860436s) [noise=true , address=77.90.31.15:4722 , piece_id=AF7F2OA247CJIDFTEU6HEOIYGOINEYPMII53BKJQXI7UZ26MJIUA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.746374739 (556h11m5.617457784s) [noise=true , address=77.90.31.10:4522 , piece_id=JOHOMHCNYXF4XQGLN2NJUSWSUO3XFBRRMYD27SLIZAK2RCT3ZC2A, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.746934555 (556h11m5.618017601s) [noise=true , address=77.90.31.21:4896 , piece_id=ATTC7BZUT4HAAEAZCHFZQE4INV2LLSWR2YCUNVA73LPJCW3YL35A, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.747556861 (556h11m5.618640146s) [noise=true , address=64.49.13.26:4956 , piece_id=FFHUPOOM7PLEABRHFH6H2SI6WXJBU2XFCLZJUBV4CXUJ6EP4PI2Q, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.748964150 (556h11m5.620047195s) [noise=true , address=145.79.227.34:4916 , piece_id=SPYSFWMJ752FC5JGRE36G54S2FHG4DQFCV6CIZONQHJDXDCN52VQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.751119314 (556h11m5.622202359s) [noise=true , address=77.90.31.18:4816 , piece_id=YT3PSYNQTDQBVBRUDNOXJ2XCN6EFQA54WBSNL6MZMXFXR7XNCHIA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.753413222 (556h11m5.624496257s) [noise=true , address=77.90.31.11:4582 , piece_id=ZRE24R27TX4E5DTV2POX7EJANQ4CM75D6ZOUS6SKM3QHHQCKRBRA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.755230482 (556h11m5.626313527s) [noise=true , address=64.49.13.154:4932 , piece_id=QB5WM5RZ2P2GC6RIGGM34IANVEI2SNDJ2PO556JXKTDBX5HLD4YA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.757598362 (556h11m5.628681407s) [noise=true , address=145.79.227.35:4510 , piece_id=P275TU367DTYXBQOGGV5THFUPHJUWGSM5IMJQ753VE4MX4IPOONQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.759644679 (556h11m5.630727734s) [noise=true , address=145.79.227.46:4626 , piece_id=OQOTWYLVIU7GJBBLYWQOEEFKR3TNAUOXZS6UAFJWJ32SIOTXNXEA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.761217594 (556h11m5.632300639s) [noise=true , address=145.79.227.43:4163 , piece_id=77BKLLVD25UMTDQZW46CA2N225I7O4HCHCRRPQ7HLJGRHOLGSDHA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.762138507 (556h11m5.633221552s) [noise=true , address=77.90.31.19:4860 , piece_id=5FUYVK2Z6LBZOAGFI2NOTAAUIV4NMBI5ERMA5RH4BMONNHABZLDQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.764304321 (556h11m5.635387356s) [noise=true , address=145.79.227.30:4784 , piece_id=MKCRSFWVFVGOAYCK5M66HO3OT7OMYWLVR2KMDXUGBS2VXXNCUSSQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.765421217 (556h11m5.636504252s) [noise=true , address=145.79.227.44:4216 , piece_id=GSBWC2LXUOZYOVD55KNQYA5Z3S2TCYROBB5O2XZIIVFUQMJQMYNA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:07.082099826 (556h11m6.953182871s) [upload=487609]: Next returned nil segment Dec 28 19:40:07.082116707 (556h11m6.953199752s) [upload=487609]: Waiting for error group managing segments... Dec 28 19:40:07.415884144 (556h11m7.286967179s) [noise=true , address=64.49.13.154:4932 , piece_id=QB5WM5RZ2P2GC6RIGGM34IANVEI2SNDJ2PO556JXKTDBX5HLD4YA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.615434252 (556h11m7.486517297s) [noise=true , address=145.79.227.23:4550 , piece_id=NWIC722LZUHJSAMPWSKCLPRIDGWPVUEYMS5KRAMX3IIMD2KR4XGQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.647182077 (556h11m7.518265112s) [noise=true , address=145.79.227.28:4744 , piece_id=42LII6EOG3CMBTIL5I4RZQPDYU52TKJN3PGWEQXROBYMNDFYIIVA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.678458624 (556h11m7.549541659s) [noise=true , address=145.79.227.46:4626 , piece_id=OQOTWYLVIU7GJBBLYWQOEEFKR3TNAUOXZS6UAFJWJ32SIOTXNXEA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.793406049 (556h11m7.664489084s) [noise=true , address=77.90.31.19:4860 , piece_id=5FUYVK2Z6LBZOAGFI2NOTAAUIV4NMBI5ERMA5RH4BMONNHABZLDQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.802506498 (556h11m7.673589543s) [noise=true , address=77.90.31.21:4896 , piece_id=ATTC7BZUT4HAAEAZCHFZQE4INV2LLSWR2YCUNVA73LPJCW3YL35A, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.853663242 (556h11m7.724746287s) [noise=true , address=77.90.31.10:4522 , piece_id=JOHOMHCNYXF4XQGLN2NJUSWSUO3XFBRRMYD27SLIZAK2RCT3ZC2A, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.880639500 (556h11m7.751722535s) [noise=true , address=77.90.31.11:4582 , piece_id=ZRE24R27TX4E5DTV2POX7EJANQ4CM75D6ZOUS6SKM3QHHQCKRBRA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.904835446 (556h11m7.775918491s) [noise=true , address=77.90.31.18:4816 , piece_id=YT3PSYNQTDQBVBRUDNOXJ2XCN6EFQA54WBSNL6MZMXFXR7XNCHIA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.988914260 (556h11m7.859997295s) [noise=true , address=77.90.31.15:4722 , piece_id=AF7F2OA247CJIDFTEU6HEOIYGOINEYPMII53BKJQXI7UZ26MJIUA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.484927449 (556h11m8.356010484s) [noise=true , address=145.79.227.35:4510 , piece_id=P275TU367DTYXBQOGGV5THFUPHJUWGSM5IMJQ753VE4MX4IPOONQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.610317849 (556h11m8.481400894s) [noise=true , address=145.79.227.44:4216 , piece_id=GSBWC2LXUOZYOVD55KNQYA5Z3S2TCYROBB5O2XZIIVFUQMJQMYNA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.611969885 (556h11m8.48305292s) [noise=true , address=145.79.227.34:4916 , piece_id=SPYSFWMJ752FC5JGRE36G54S2FHG4DQFCV6CIZONQHJDXDCN52VQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.636098801 (556h11m8.507181836s) [noise=true , address=145.79.227.30:4784 , piece_id=MKCRSFWVFVGOAYCK5M66HO3OT7OMYWLVR2KMDXUGBS2VXXNCUSSQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.636122136 (556h11m8.507205181s) [seg_pos={0 0}, upload=487609]: Segment reached optimal threshold of 14 pieces. Dec 28 19:40:08.636333929 (556h11m8.507416974s) [noise=true , address=64.49.13.26:4956 , piece_id=FFHUPOOM7PLEABRHFH6H2SI6WXJBU2XFCLZJUBV4CXUJ6EP4PI2Q, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TYHhjpfyBKkenSwviyQCUZpSBbnaFCGBFRkM4fyKLDuxKx4Ju): protocol: expected piece hash; context canceled; EOF Dec 28 19:40:08.636342175 (556h11m8.50742521s) [noise=true , address=64.49.13.26:4956 , piece_id=FFHUPOOM7PLEABRHFH6H2SI6WXJBU2XFCLZJUBV4CXUJ6EP4PI2Q, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636585959 (556h11m8.507668994s) [noise=true , address=145.79.227.43:4163 , piece_id=77BKLLVD25UMTDQZW46CA2N225I7O4HCHCRRPQ7HLJGRHOLGSDHA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CgPQ5hkrJdqbJvGarxKjXvYYnrXQqapL5BH2rNJNvnhpRi3pw): protocol: expected piece hash; context canceled; EOF Dec 28 19:40:08.636592172 (556h11m8.507675207s) [noise=true , address=145.79.227.43:4163 , piece_id=77BKLLVD25UMTDQZW46CA2N225I7O4HCHCRRPQ7HLJGRHOLGSDHA, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636681511 (556h11m8.507764546s) [noise=true , address=145.79.227.19:4083 , piece_id=GL5XL7T2QQABD5364AT4AFV6RZNOWCTVNSMAVIFJU5SUQFHOPZKQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KZ5a6CQ9tUTrC6EYE7M8yg6p9fK95CN8zQLQmP4tdyz4rBfGh): protocol: expected piece hash; context canceled; EOF Dec 28 19:40:08.636687052 (556h11m8.507770087s) [noise=true , address=145.79.227.19:4083 , piece_id=GL5XL7T2QQABD5364AT4AFV6RZNOWCTVNSMAVIFJU5SUQFHOPZKQ, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636810327 (556h11m8.507893362s) [noise=true , address=145.79.227.17:4992 , piece_id=GUF7AUFCFRRPY5PSYS7DRPFQLGT4ZEL5C7P23US2VI54KYWSUCUQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hbt8HKTunpPKyAzH4XBwGQqTKzcgqPpoWwKZm9JMtaK8P7aVp): protocol: write tcp 188.166.225.170:51210->145.79.227.17:4992: use of closed network connection; write tcp 188.166.225.170:51210->145.79.227.17:4992: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51210->145.79.227.17:4992: use of closed network connection Dec 28 19:40:08.636815957 (556h11m8.507899002s) [noise=true , address=145.79.227.17:4992 , piece_id=GUF7AUFCFRRPY5PSYS7DRPFQLGT4ZEL5C7P23US2VI54KYWSUCUQ, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636834313 (556h11m8.507917358s) [seg_pos={0 0}, upload=487609]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 19:40:08.636907953 (556h11m8.507990998s) [upload=487609]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 19:40:08.636913644 (556h11m8.507996689s) [upload=487609]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 19:40:08.924991397 (556h11m8.796074442s) [upload=487609]: Done uploading segments... Dec 28 19:40:08.925014742 (556h11m8.796097777s) [upload=487609]: Upload finished. err: <nil> Duration: 24.460749455s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [X..X......XX..............X..........................X......] duration: 6.06873083s trace: https://jaeger.dev.storj.io/trace/3057620e25d11143 FAIL (2025-12-28 20:18:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 20:13:34.173546860 (556h44m34.044629895s) [upload=488098]: Starting upload Dec 28 20:13:34.173605531 (556h44m34.044688566s) [upload=488098]: Done starting upload Dec 28 20:13:34.173822354 (556h44m34.04490539s) [upload=488098]: Uploading segments... Dec 28 20:13:34.173860978 (556h44m34.044944013s) [upload=488098]: Got next segment. Inline: false Dec 28 20:13:34.173892047 (556h44m34.044975082s) [upload=488098]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 20:13:34.173896415 (556h44m34.04497945s) [upload=488098]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 20:13:34.422688850 (556h44m34.293771905s) [seg_pos={0 0}, upload=488098]: Begin upload segment... Dec 28 20:13:34.422745879 (556h44m34.293828944s) [seg_pos={0 0}, upload=488098]: Done begin upload segment. Dec 28 20:13:34.422771076 (556h44m34.293854141s) [noise=true , address=145.79.227.16:4909 , piece_id=G7U4H2HJWK4VGY42RUBKYGDBLRDZLWX6K772ULG2YAH5OVZ5JPKA, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.422842654 (556h44m34.293925699s) [noise=true , address=145.79.227.37:4566 , piece_id=AXTGBOXYJDN7QRFJWEMU3WBEF74KYERZS2B2SZXSPLZRU2TYC4JA, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.423548137 (556h44m34.294631192s) [noise=true , address=145.79.227.11:4730 , piece_id=6AXVYQY7IIQYL4OHBO3URZFJCC42ZWL3NEMD3JSFHLYMXNTQBMUQ, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.423555861 (556h44m34.294638906s) [noise=true , address=45.91.116.10:4081 , piece_id=GU7LAF6GLPTVIEOQ6GRHZI4VYYZMQUPKMAPOFBFCRQB344CUKKMQ, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.424230055 (556h44m34.29531309s) [noise=true , address=45.91.116.6:4976 , piece_id=4GCSWP27NMVW6VQFIOFBGBKHCZEB5VWC32CXMKKLVGXXFK3EHBXQ, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.424619267 (556h44m34.295702302s) [noise=true , address=145.79.227.36:4519 , piece_id=EJBDJEJAN7WG3HHTXBUEIW7QZ2VY6ZOXKNHBLHNYJ7L6ZVH6M7YA, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.424778910 (556h44m34.295861955s) [noise=true , address=77.90.31.11:4556 , piece_id=FANBXJICXHI33JVWYCLDDB76KSYXXFBTX4MLYXE6ICXQP2JAJAJA, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.425200092 (556h44m34.296283137s) [noise=true , address=77.90.31.18:4808 , piece_id=FIOQVOLXKF2CFNJ57BTCL4WTDPZPEEGAS2CDPF2PNTY27JVFTG6Q, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.425605254 (556h44m34.296688289s) [noise=true , address=77.90.31.17:4770 , piece_id=EA2QWTOVMLD5HKF6FGTB3TZXZZLSI4UVFTPD3MZZ2PAXF47VGJQA, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.426052345 (556h44m34.29713538s) [noise=true , address=145.79.227.51:4411 , piece_id=NHVE27MXBV6Y4VNYIT6G73S4WDAWVKACBQIOKTIOCGELPMLCY4CQ, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.426486622 (556h44m34.297569657s) [noise=true , address=45.91.116.13:4153 , piece_id=XCGSWRC2CZ2NLTPN5T5XTK4VHGIES67NB3FFCBXGI63DLGQ7V2EQ, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.426897945 (556h44m34.29798098s) [noise=true , address=45.91.116.4:4937 , piece_id=V6H4VACBOTZZMXQKCQRPJOBRFQVRF562HZC2GQGBLQGAGKFY5VNQ, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.427411794 (556h44m34.298494829s) [noise=true , address=145.79.227.45:4229 , piece_id=IRJAI3XSHMQYCOIFOMRNETTBXLLAEL3Z3YBOLPGEM2FOBF6S37XQ, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.427894844 (556h44m34.298977879s) [noise=true , address=145.79.227.21:4473 , piece_id=KAYPSFAGCFG3DUA4MVZAAHSH65MCNPDOH2M7GTAYILZO6EPFPXYQ, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.431922001 (556h44m34.303005036s) [noise=true , address=145.79.227.50:4374 , piece_id=DCLHZ4GR6U56WATYQRY7YFCFHO4JYMGLXSBY2OWEM3HSGPJHLDSA, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.436931210 (556h44m34.308014255s) [noise=true , address=145.79.227.38:4979 , piece_id=OUNZYJAAA32ZMIQOOCHK4YF4DCF354XUW4WUCHM3445666VXVLBQ, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:36.115225165 (556h44m35.98630821s) [upload=488098]: Next returned nil segment Dec 28 20:13:36.115243359 (556h44m35.986326395s) [upload=488098]: Waiting for error group managing segments... Dec 28 20:13:36.415982301 (556h44m36.287065346s) [noise=true , address=145.79.227.51:4411 , piece_id=NHVE27MXBV6Y4VNYIT6G73S4WDAWVKACBQIOKTIOCGELPMLCY4CQ, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.430495527 (556h44m36.301578562s) [noise=true , address=145.79.227.45:4229 , piece_id=IRJAI3XSHMQYCOIFOMRNETTBXLLAEL3Z3YBOLPGEM2FOBF6S37XQ, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.435192370 (556h44m36.306275406s) [noise=true , address=145.79.227.21:4473 , piece_id=KAYPSFAGCFG3DUA4MVZAAHSH65MCNPDOH2M7GTAYILZO6EPFPXYQ, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.435831557 (556h44m36.306914602s) [noise=true , address=145.79.227.38:4979 , piece_id=OUNZYJAAA32ZMIQOOCHK4YF4DCF354XUW4WUCHM3445666VXVLBQ, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.444124790 (556h44m36.315207825s) [noise=true , address=145.79.227.50:4374 , piece_id=DCLHZ4GR6U56WATYQRY7YFCFHO4JYMGLXSBY2OWEM3HSGPJHLDSA, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.450320487 (556h44m36.321403533s) [noise=true , address=145.79.227.11:4730 , piece_id=6AXVYQY7IIQYL4OHBO3URZFJCC42ZWL3NEMD3JSFHLYMXNTQBMUQ, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.452177964 (556h44m36.323260999s) [noise=true , address=145.79.227.36:4519 , piece_id=EJBDJEJAN7WG3HHTXBUEIW7QZ2VY6ZOXKNHBLHNYJ7L6ZVH6M7YA, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.462219385 (556h44m36.33330242s) [noise=true , address=145.79.227.37:4566 , piece_id=AXTGBOXYJDN7QRFJWEMU3WBEF74KYERZS2B2SZXSPLZRU2TYC4JA, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.485557775 (556h44m36.35664081s) [noise=true , address=145.79.227.16:4909 , piece_id=G7U4H2HJWK4VGY42RUBKYGDBLRDZLWX6K772ULG2YAH5OVZ5JPKA, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.583906940 (556h44m36.454989975s) [noise=true , address=77.90.31.11:4556 , piece_id=FANBXJICXHI33JVWYCLDDB76KSYXXFBTX4MLYXE6ICXQP2JAJAJA, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.658185680 (556h44m36.529268725s) [noise=true , address=45.91.116.4:4937 , piece_id=V6H4VACBOTZZMXQKCQRPJOBRFQVRF562HZC2GQGBLQGAGKFY5VNQ, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.715312460 (556h44m36.586395495s) [noise=true , address=45.91.116.13:4153 , piece_id=XCGSWRC2CZ2NLTPN5T5XTK4VHGIES67NB3FFCBXGI63DLGQ7V2EQ, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.737246156 (556h44m36.608329201s) [noise=true , address=45.91.116.10:4081 , piece_id=GU7LAF6GLPTVIEOQ6GRHZI4VYYZMQUPKMAPOFBFCRQB344CUKKMQ, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.737270231 (556h44m36.608353266s) [seg_pos={0 0}, upload=488098]: Segment reached optimal threshold of 13 pieces. Dec 28 20:13:36.737506872 (556h44m36.608589927s) [noise=true , address=77.90.31.18:4808 , piece_id=FIOQVOLXKF2CFNJ57BTCL4WTDPZPEEGAS2CDPF2PNTY27JVFTG6Q, seg_pos={0 0}, upload=488098]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: expected piece hash; context canceled; EOF Dec 28 20:13:36.737516099 (556h44m36.608599134s) [noise=true , address=77.90.31.18:4808 , piece_id=FIOQVOLXKF2CFNJ57BTCL4WTDPZPEEGAS2CDPF2PNTY27JVFTG6Q, seg_pos={0 0}, upload=488098]: Long tail cancellation due to optimal threshold Dec 28 20:13:36.737659642 (556h44m36.608742688s) [noise=true , address=45.91.116.6:4976 , piece_id=4GCSWP27NMVW6VQFIOFBGBKHCZEB5VWC32CXMKKLVGXXFK3EHBXQ, seg_pos={0 0}, upload=488098]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Dec 28 20:13:36.737707214 (556h44m36.608790259s) [noise=true , address=45.91.116.6:4976 , piece_id=4GCSWP27NMVW6VQFIOFBGBKHCZEB5VWC32CXMKKLVGXXFK3EHBXQ, seg_pos={0 0}, upload=488098]: Long tail cancellation due to optimal threshold Dec 28 20:13:36.737939255 (556h44m36.6090223s) [noise=true , address=77.90.31.17:4770 , piece_id=EA2QWTOVMLD5HKF6FGTB3TZXZZLSI4UVFTPD3MZZ2PAXF47VGJQA, seg_pos={0 0}, upload=488098]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: write tcp 188.166.225.170:53958->77.90.31.17:4770: use of closed network connection; write tcp 188.166.225.170:53958->77.90.31.17:4770: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53958->77.90.31.17:4770: use of closed network connection Dec 28 20:13:36.737965024 (556h44m36.609048079s) [noise=true , address=77.90.31.17:4770 , piece_id=EA2QWTOVMLD5HKF6FGTB3TZXZZLSI4UVFTPD3MZZ2PAXF47VGJQA, seg_pos={0 0}, upload=488098]: Long tail cancellation due to optimal threshold Dec 28 20:13:36.737999810 (556h44m36.609082846s) [seg_pos={0 0}, upload=488098]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 20:13:36.738101905 (556h44m36.60918495s) [upload=488098]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 20:13:36.738109249 (556h44m36.609192304s) [upload=488098]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 20:13:37.032410772 (556h44m36.903493817s) [upload=488098]: Done uploading segments... Dec 28 20:13:37.032430300 (556h44m36.903513345s) [upload=488098]: Upload finished. err: <nil> Duration: 6.068757409s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 20.58474827s trace: https://jaeger.dev.storj.io/trace/10e1641b228b2a1f FAIL (2025-12-28 20:18:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 40:05.744568479 (556h11m5.615651514s) [noise=true , address=145.79.227.28:4744 , piece_id=42LII6EOG3CMBTIL5I4RZQPDYU52TKJN3PGWEQXROBYMNDFYIIVA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.745099351 (556h11m5.616182396s) [noise=true , address=145.79.227.23:4550 , piece_id=NWIC722LZUHJSAMPWSKCLPRIDGWPVUEYMS5KRAMX3IIMD2KR4XGQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.745777401 (556h11m5.616860436s) [noise=true , address=77.90.31.15:4722 , piece_id=AF7F2OA247CJIDFTEU6HEOIYGOINEYPMII53BKJQXI7UZ26MJIUA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.746374739 (556h11m5.617457784s) [noise=true , address=77.90.31.10:4522 , piece_id=JOHOMHCNYXF4XQGLN2NJUSWSUO3XFBRRMYD27SLIZAK2RCT3ZC2A, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.746934555 (556h11m5.618017601s) [noise=true , address=77.90.31.21:4896 , piece_id=ATTC7BZUT4HAAEAZCHFZQE4INV2LLSWR2YCUNVA73LPJCW3YL35A, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.747556861 (556h11m5.618640146s) [noise=true , address=64.49.13.26:4956 , piece_id=FFHUPOOM7PLEABRHFH6H2SI6WXJBU2XFCLZJUBV4CXUJ6EP4PI2Q, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.748964150 (556h11m5.620047195s) [noise=true , address=145.79.227.34:4916 , piece_id=SPYSFWMJ752FC5JGRE36G54S2FHG4DQFCV6CIZONQHJDXDCN52VQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.751119314 (556h11m5.622202359s) [noise=true , address=77.90.31.18:4816 , piece_id=YT3PSYNQTDQBVBRUDNOXJ2XCN6EFQA54WBSNL6MZMXFXR7XNCHIA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.753413222 (556h11m5.624496257s) [noise=true , address=77.90.31.11:4582 , piece_id=ZRE24R27TX4E5DTV2POX7EJANQ4CM75D6ZOUS6SKM3QHHQCKRBRA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.755230482 (556h11m5.626313527s) [noise=true , address=64.49.13.154:4932 , piece_id=QB5WM5RZ2P2GC6RIGGM34IANVEI2SNDJ2PO556JXKTDBX5HLD4YA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.757598362 (556h11m5.628681407s) [noise=true , address=145.79.227.35:4510 , piece_id=P275TU367DTYXBQOGGV5THFUPHJUWGSM5IMJQ753VE4MX4IPOONQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.759644679 (556h11m5.630727734s) [noise=true , address=145.79.227.46:4626 , piece_id=OQOTWYLVIU7GJBBLYWQOEEFKR3TNAUOXZS6UAFJWJ32SIOTXNXEA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.761217594 (556h11m5.632300639s) [noise=true , address=145.79.227.43:4163 , piece_id=77BKLLVD25UMTDQZW46CA2N225I7O4HCHCRRPQ7HLJGRHOLGSDHA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.762138507 (556h11m5.633221552s) [noise=true , address=77.90.31.19:4860 , piece_id=5FUYVK2Z6LBZOAGFI2NOTAAUIV4NMBI5ERMA5RH4BMONNHABZLDQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.764304321 (556h11m5.635387356s) [noise=true , address=145.79.227.30:4784 , piece_id=MKCRSFWVFVGOAYCK5M66HO3OT7OMYWLVR2KMDXUGBS2VXXNCUSSQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.765421217 (556h11m5.636504252s) [noise=true , address=145.79.227.44:4216 , piece_id=GSBWC2LXUOZYOVD55KNQYA5Z3S2TCYROBB5O2XZIIVFUQMJQMYNA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:07.082099826 (556h11m6.953182871s) [upload=487609]: Next returned nil segment Dec 28 19:40:07.082116707 (556h11m6.953199752s) [upload=487609]: Waiting for error group managing segments... Dec 28 19:40:07.415884144 (556h11m7.286967179s) [noise=true , address=64.49.13.154:4932 , piece_id=QB5WM5RZ2P2GC6RIGGM34IANVEI2SNDJ2PO556JXKTDBX5HLD4YA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.615434252 (556h11m7.486517297s) [noise=true , address=145.79.227.23:4550 , piece_id=NWIC722LZUHJSAMPWSKCLPRIDGWPVUEYMS5KRAMX3IIMD2KR4XGQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.647182077 (556h11m7.518265112s) [noise=true , address=145.79.227.28:4744 , piece_id=42LII6EOG3CMBTIL5I4RZQPDYU52TKJN3PGWEQXROBYMNDFYIIVA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.678458624 (556h11m7.549541659s) [noise=true , address=145.79.227.46:4626 , piece_id=OQOTWYLVIU7GJBBLYWQOEEFKR3TNAUOXZS6UAFJWJ32SIOTXNXEA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.793406049 (556h11m7.664489084s) [noise=true , address=77.90.31.19:4860 , piece_id=5FUYVK2Z6LBZOAGFI2NOTAAUIV4NMBI5ERMA5RH4BMONNHABZLDQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.802506498 (556h11m7.673589543s) [noise=true , address=77.90.31.21:4896 , piece_id=ATTC7BZUT4HAAEAZCHFZQE4INV2LLSWR2YCUNVA73LPJCW3YL35A, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.853663242 (556h11m7.724746287s) [noise=true , address=77.90.31.10:4522 , piece_id=JOHOMHCNYXF4XQGLN2NJUSWSUO3XFBRRMYD27SLIZAK2RCT3ZC2A, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.880639500 (556h11m7.751722535s) [noise=true , address=77.90.31.11:4582 , piece_id=ZRE24R27TX4E5DTV2POX7EJANQ4CM75D6ZOUS6SKM3QHHQCKRBRA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.904835446 (556h11m7.775918491s) [noise=true , address=77.90.31.18:4816 , piece_id=YT3PSYNQTDQBVBRUDNOXJ2XCN6EFQA54WBSNL6MZMXFXR7XNCHIA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.988914260 (556h11m7.859997295s) [noise=true , address=77.90.31.15:4722 , piece_id=AF7F2OA247CJIDFTEU6HEOIYGOINEYPMII53BKJQXI7UZ26MJIUA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.484927449 (556h11m8.356010484s) [noise=true , address=145.79.227.35:4510 , piece_id=P275TU367DTYXBQOGGV5THFUPHJUWGSM5IMJQ753VE4MX4IPOONQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.610317849 (556h11m8.481400894s) [noise=true , address=145.79.227.44:4216 , piece_id=GSBWC2LXUOZYOVD55KNQYA5Z3S2TCYROBB5O2XZIIVFUQMJQMYNA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.611969885 (556h11m8.48305292s) [noise=true , address=145.79.227.34:4916 , piece_id=SPYSFWMJ752FC5JGRE36G54S2FHG4DQFCV6CIZONQHJDXDCN52VQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.636098801 (556h11m8.507181836s) [noise=true , address=145.79.227.30:4784 , piece_id=MKCRSFWVFVGOAYCK5M66HO3OT7OMYWLVR2KMDXUGBS2VXXNCUSSQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.636122136 (556h11m8.507205181s) [seg_pos={0 0}, upload=487609]: Segment reached optimal threshold of 14 pieces. Dec 28 19:40:08.636333929 (556h11m8.507416974s) [noise=true , address=64.49.13.26:4956 , piece_id=FFHUPOOM7PLEABRHFH6H2SI6WXJBU2XFCLZJUBV4CXUJ6EP4PI2Q, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TYHhjpfyBKkenSwviyQCUZpSBbnaFCGBFRkM4fyKLDuxKx4Ju): protocol: expected piece hash; context canceled; EOF Dec 28 19:40:08.636342175 (556h11m8.50742521s) [noise=true , address=64.49.13.26:4956 , piece_id=FFHUPOOM7PLEABRHFH6H2SI6WXJBU2XFCLZJUBV4CXUJ6EP4PI2Q, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636585959 (556h11m8.507668994s) [noise=true , address=145.79.227.43:4163 , piece_id=77BKLLVD25UMTDQZW46CA2N225I7O4HCHCRRPQ7HLJGRHOLGSDHA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CgPQ5hkrJdqbJvGarxKjXvYYnrXQqapL5BH2rNJNvnhpRi3pw): protocol: expected piece hash; context canceled; EOF Dec 28 19:40:08.636592172 (556h11m8.507675207s) [noise=true , address=145.79.227.43:4163 , piece_id=77BKLLVD25UMTDQZW46CA2N225I7O4HCHCRRPQ7HLJGRHOLGSDHA, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636681511 (556h11m8.507764546s) [noise=true , address=145.79.227.19:4083 , piece_id=GL5XL7T2QQABD5364AT4AFV6RZNOWCTVNSMAVIFJU5SUQFHOPZKQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KZ5a6CQ9tUTrC6EYE7M8yg6p9fK95CN8zQLQmP4tdyz4rBfGh): protocol: expected piece hash; context canceled; EOF Dec 28 19:40:08.636687052 (556h11m8.507770087s) [noise=true , address=145.79.227.19:4083 , piece_id=GL5XL7T2QQABD5364AT4AFV6RZNOWCTVNSMAVIFJU5SUQFHOPZKQ, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636810327 (556h11m8.507893362s) [noise=true , address=145.79.227.17:4992 , piece_id=GUF7AUFCFRRPY5PSYS7DRPFQLGT4ZEL5C7P23US2VI54KYWSUCUQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hbt8HKTunpPKyAzH4XBwGQqTKzcgqPpoWwKZm9JMtaK8P7aVp): protocol: write tcp 188.166.225.170:51210->145.79.227.17:4992: use of closed network connection; write tcp 188.166.225.170:51210->145.79.227.17:4992: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51210->145.79.227.17:4992: use of closed network connection Dec 28 19:40:08.636815957 (556h11m8.507899002s) [noise=true , address=145.79.227.17:4992 , piece_id=GUF7AUFCFRRPY5PSYS7DRPFQLGT4ZEL5C7P23US2VI54KYWSUCUQ, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636834313 (556h11m8.507917358s) [seg_pos={0 0}, upload=487609]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 19:40:08.636907953 (556h11m8.507990998s) [upload=487609]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 19:40:08.636913644 (556h11m8.507996689s) [upload=487609]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 19:40:08.924991397 (556h11m8.796074442s) [upload=487609]: Done uploading segments... Dec 28 19:40:08.925014742 (556h11m8.796097777s) [upload=487609]: Upload finished. err: <nil> Duration: 20.58485835s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.X..X......XX..............X..........................X.....] duration: 5.916736246s trace: https://jaeger.dev.storj.io/trace/3057620e25d11143 FAIL (2025-12-28 20:18:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 20:13:34.173546860 (556h44m34.044629895s) [upload=488098]: Starting upload Dec 28 20:13:34.173605531 (556h44m34.044688566s) [upload=488098]: Done starting upload Dec 28 20:13:34.173822354 (556h44m34.04490539s) [upload=488098]: Uploading segments... Dec 28 20:13:34.173860978 (556h44m34.044944013s) [upload=488098]: Got next segment. Inline: false Dec 28 20:13:34.173892047 (556h44m34.044975082s) [upload=488098]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 20:13:34.173896415 (556h44m34.04497945s) [upload=488098]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 20:13:34.422688850 (556h44m34.293771905s) [seg_pos={0 0}, upload=488098]: Begin upload segment... Dec 28 20:13:34.422745879 (556h44m34.293828944s) [seg_pos={0 0}, upload=488098]: Done begin upload segment. Dec 28 20:13:34.422771076 (556h44m34.293854141s) [noise=true , address=145.79.227.16:4909 , piece_id=G7U4H2HJWK4VGY42RUBKYGDBLRDZLWX6K772ULG2YAH5OVZ5JPKA, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.422842654 (556h44m34.293925699s) [noise=true , address=145.79.227.37:4566 , piece_id=AXTGBOXYJDN7QRFJWEMU3WBEF74KYERZS2B2SZXSPLZRU2TYC4JA, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.423548137 (556h44m34.294631192s) [noise=true , address=145.79.227.11:4730 , piece_id=6AXVYQY7IIQYL4OHBO3URZFJCC42ZWL3NEMD3JSFHLYMXNTQBMUQ, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.423555861 (556h44m34.294638906s) [noise=true , address=45.91.116.10:4081 , piece_id=GU7LAF6GLPTVIEOQ6GRHZI4VYYZMQUPKMAPOFBFCRQB344CUKKMQ, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.424230055 (556h44m34.29531309s) [noise=true , address=45.91.116.6:4976 , piece_id=4GCSWP27NMVW6VQFIOFBGBKHCZEB5VWC32CXMKKLVGXXFK3EHBXQ, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.424619267 (556h44m34.295702302s) [noise=true , address=145.79.227.36:4519 , piece_id=EJBDJEJAN7WG3HHTXBUEIW7QZ2VY6ZOXKNHBLHNYJ7L6ZVH6M7YA, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.424778910 (556h44m34.295861955s) [noise=true , address=77.90.31.11:4556 , piece_id=FANBXJICXHI33JVWYCLDDB76KSYXXFBTX4MLYXE6ICXQP2JAJAJA, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.425200092 (556h44m34.296283137s) [noise=true , address=77.90.31.18:4808 , piece_id=FIOQVOLXKF2CFNJ57BTCL4WTDPZPEEGAS2CDPF2PNTY27JVFTG6Q, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.425605254 (556h44m34.296688289s) [noise=true , address=77.90.31.17:4770 , piece_id=EA2QWTOVMLD5HKF6FGTB3TZXZZLSI4UVFTPD3MZZ2PAXF47VGJQA, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.426052345 (556h44m34.29713538s) [noise=true , address=145.79.227.51:4411 , piece_id=NHVE27MXBV6Y4VNYIT6G73S4WDAWVKACBQIOKTIOCGELPMLCY4CQ, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.426486622 (556h44m34.297569657s) [noise=true , address=45.91.116.13:4153 , piece_id=XCGSWRC2CZ2NLTPN5T5XTK4VHGIES67NB3FFCBXGI63DLGQ7V2EQ, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.426897945 (556h44m34.29798098s) [noise=true , address=45.91.116.4:4937 , piece_id=V6H4VACBOTZZMXQKCQRPJOBRFQVRF562HZC2GQGBLQGAGKFY5VNQ, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.427411794 (556h44m34.298494829s) [noise=true , address=145.79.227.45:4229 , piece_id=IRJAI3XSHMQYCOIFOMRNETTBXLLAEL3Z3YBOLPGEM2FOBF6S37XQ, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.427894844 (556h44m34.298977879s) [noise=true , address=145.79.227.21:4473 , piece_id=KAYPSFAGCFG3DUA4MVZAAHSH65MCNPDOH2M7GTAYILZO6EPFPXYQ, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.431922001 (556h44m34.303005036s) [noise=true , address=145.79.227.50:4374 , piece_id=DCLHZ4GR6U56WATYQRY7YFCFHO4JYMGLXSBY2OWEM3HSGPJHLDSA, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.436931210 (556h44m34.308014255s) [noise=true , address=145.79.227.38:4979 , piece_id=OUNZYJAAA32ZMIQOOCHK4YF4DCF354XUW4WUCHM3445666VXVLBQ, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:36.115225165 (556h44m35.98630821s) [upload=488098]: Next returned nil segment Dec 28 20:13:36.115243359 (556h44m35.986326395s) [upload=488098]: Waiting for error group managing segments... Dec 28 20:13:36.415982301 (556h44m36.287065346s) [noise=true , address=145.79.227.51:4411 , piece_id=NHVE27MXBV6Y4VNYIT6G73S4WDAWVKACBQIOKTIOCGELPMLCY4CQ, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.430495527 (556h44m36.301578562s) [noise=true , address=145.79.227.45:4229 , piece_id=IRJAI3XSHMQYCOIFOMRNETTBXLLAEL3Z3YBOLPGEM2FOBF6S37XQ, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.435192370 (556h44m36.306275406s) [noise=true , address=145.79.227.21:4473 , piece_id=KAYPSFAGCFG3DUA4MVZAAHSH65MCNPDOH2M7GTAYILZO6EPFPXYQ, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.435831557 (556h44m36.306914602s) [noise=true , address=145.79.227.38:4979 , piece_id=OUNZYJAAA32ZMIQOOCHK4YF4DCF354XUW4WUCHM3445666VXVLBQ, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.444124790 (556h44m36.315207825s) [noise=true , address=145.79.227.50:4374 , piece_id=DCLHZ4GR6U56WATYQRY7YFCFHO4JYMGLXSBY2OWEM3HSGPJHLDSA, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.450320487 (556h44m36.321403533s) [noise=true , address=145.79.227.11:4730 , piece_id=6AXVYQY7IIQYL4OHBO3URZFJCC42ZWL3NEMD3JSFHLYMXNTQBMUQ, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.452177964 (556h44m36.323260999s) [noise=true , address=145.79.227.36:4519 , piece_id=EJBDJEJAN7WG3HHTXBUEIW7QZ2VY6ZOXKNHBLHNYJ7L6ZVH6M7YA, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.462219385 (556h44m36.33330242s) [noise=true , address=145.79.227.37:4566 , piece_id=AXTGBOXYJDN7QRFJWEMU3WBEF74KYERZS2B2SZXSPLZRU2TYC4JA, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.485557775 (556h44m36.35664081s) [noise=true , address=145.79.227.16:4909 , piece_id=G7U4H2HJWK4VGY42RUBKYGDBLRDZLWX6K772ULG2YAH5OVZ5JPKA, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.583906940 (556h44m36.454989975s) [noise=true , address=77.90.31.11:4556 , piece_id=FANBXJICXHI33JVWYCLDDB76KSYXXFBTX4MLYXE6ICXQP2JAJAJA, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.658185680 (556h44m36.529268725s) [noise=true , address=45.91.116.4:4937 , piece_id=V6H4VACBOTZZMXQKCQRPJOBRFQVRF562HZC2GQGBLQGAGKFY5VNQ, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.715312460 (556h44m36.586395495s) [noise=true , address=45.91.116.13:4153 , piece_id=XCGSWRC2CZ2NLTPN5T5XTK4VHGIES67NB3FFCBXGI63DLGQ7V2EQ, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.737246156 (556h44m36.608329201s) [noise=true , address=45.91.116.10:4081 , piece_id=GU7LAF6GLPTVIEOQ6GRHZI4VYYZMQUPKMAPOFBFCRQB344CUKKMQ, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.737270231 (556h44m36.608353266s) [seg_pos={0 0}, upload=488098]: Segment reached optimal threshold of 13 pieces. Dec 28 20:13:36.737506872 (556h44m36.608589927s) [noise=true , address=77.90.31.18:4808 , piece_id=FIOQVOLXKF2CFNJ57BTCL4WTDPZPEEGAS2CDPF2PNTY27JVFTG6Q, seg_pos={0 0}, upload=488098]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: expected piece hash; context canceled; EOF Dec 28 20:13:36.737516099 (556h44m36.608599134s) [noise=true , address=77.90.31.18:4808 , piece_id=FIOQVOLXKF2CFNJ57BTCL4WTDPZPEEGAS2CDPF2PNTY27JVFTG6Q, seg_pos={0 0}, upload=488098]: Long tail cancellation due to optimal threshold Dec 28 20:13:36.737659642 (556h44m36.608742688s) [noise=true , address=45.91.116.6:4976 , piece_id=4GCSWP27NMVW6VQFIOFBGBKHCZEB5VWC32CXMKKLVGXXFK3EHBXQ, seg_pos={0 0}, upload=488098]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Dec 28 20:13:36.737707214 (556h44m36.608790259s) [noise=true , address=45.91.116.6:4976 , piece_id=4GCSWP27NMVW6VQFIOFBGBKHCZEB5VWC32CXMKKLVGXXFK3EHBXQ, seg_pos={0 0}, upload=488098]: Long tail cancellation due to optimal threshold Dec 28 20:13:36.737939255 (556h44m36.6090223s) [noise=true , address=77.90.31.17:4770 , piece_id=EA2QWTOVMLD5HKF6FGTB3TZXZZLSI4UVFTPD3MZZ2PAXF47VGJQA, seg_pos={0 0}, upload=488098]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: write tcp 188.166.225.170:53958->77.90.31.17:4770: use of closed network connection; write tcp 188.166.225.170:53958->77.90.31.17:4770: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53958->77.90.31.17:4770: use of closed network connection Dec 28 20:13:36.737965024 (556h44m36.609048079s) [noise=true , address=77.90.31.17:4770 , piece_id=EA2QWTOVMLD5HKF6FGTB3TZXZZLSI4UVFTPD3MZZ2PAXF47VGJQA, seg_pos={0 0}, upload=488098]: Long tail cancellation due to optimal threshold Dec 28 20:13:36.737999810 (556h44m36.609082846s) [seg_pos={0 0}, upload=488098]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 20:13:36.738101905 (556h44m36.60918495s) [upload=488098]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 20:13:36.738109249 (556h44m36.609192304s) [upload=488098]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 20:13:37.032410772 (556h44m36.903493817s) [upload=488098]: Done uploading segments... Dec 28 20:13:37.032430300 (556h44m36.903513345s) [upload=488098]: Upload finished. err: <nil> Duration: 5.916766473s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 23.458792662s trace: https://jaeger.dev.storj.io/trace/10e1641b228b2a1f FAIL (2025-12-28 20:18:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 40:05.744568479 (556h11m5.615651514s) [noise=true , address=145.79.227.28:4744 , piece_id=42LII6EOG3CMBTIL5I4RZQPDYU52TKJN3PGWEQXROBYMNDFYIIVA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.745099351 (556h11m5.616182396s) [noise=true , address=145.79.227.23:4550 , piece_id=NWIC722LZUHJSAMPWSKCLPRIDGWPVUEYMS5KRAMX3IIMD2KR4XGQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.745777401 (556h11m5.616860436s) [noise=true , address=77.90.31.15:4722 , piece_id=AF7F2OA247CJIDFTEU6HEOIYGOINEYPMII53BKJQXI7UZ26MJIUA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.746374739 (556h11m5.617457784s) [noise=true , address=77.90.31.10:4522 , piece_id=JOHOMHCNYXF4XQGLN2NJUSWSUO3XFBRRMYD27SLIZAK2RCT3ZC2A, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.746934555 (556h11m5.618017601s) [noise=true , address=77.90.31.21:4896 , piece_id=ATTC7BZUT4HAAEAZCHFZQE4INV2LLSWR2YCUNVA73LPJCW3YL35A, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.747556861 (556h11m5.618640146s) [noise=true , address=64.49.13.26:4956 , piece_id=FFHUPOOM7PLEABRHFH6H2SI6WXJBU2XFCLZJUBV4CXUJ6EP4PI2Q, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.748964150 (556h11m5.620047195s) [noise=true , address=145.79.227.34:4916 , piece_id=SPYSFWMJ752FC5JGRE36G54S2FHG4DQFCV6CIZONQHJDXDCN52VQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.751119314 (556h11m5.622202359s) [noise=true , address=77.90.31.18:4816 , piece_id=YT3PSYNQTDQBVBRUDNOXJ2XCN6EFQA54WBSNL6MZMXFXR7XNCHIA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.753413222 (556h11m5.624496257s) [noise=true , address=77.90.31.11:4582 , piece_id=ZRE24R27TX4E5DTV2POX7EJANQ4CM75D6ZOUS6SKM3QHHQCKRBRA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.755230482 (556h11m5.626313527s) [noise=true , address=64.49.13.154:4932 , piece_id=QB5WM5RZ2P2GC6RIGGM34IANVEI2SNDJ2PO556JXKTDBX5HLD4YA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.757598362 (556h11m5.628681407s) [noise=true , address=145.79.227.35:4510 , piece_id=P275TU367DTYXBQOGGV5THFUPHJUWGSM5IMJQ753VE4MX4IPOONQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.759644679 (556h11m5.630727734s) [noise=true , address=145.79.227.46:4626 , piece_id=OQOTWYLVIU7GJBBLYWQOEEFKR3TNAUOXZS6UAFJWJ32SIOTXNXEA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.761217594 (556h11m5.632300639s) [noise=true , address=145.79.227.43:4163 , piece_id=77BKLLVD25UMTDQZW46CA2N225I7O4HCHCRRPQ7HLJGRHOLGSDHA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.762138507 (556h11m5.633221552s) [noise=true , address=77.90.31.19:4860 , piece_id=5FUYVK2Z6LBZOAGFI2NOTAAUIV4NMBI5ERMA5RH4BMONNHABZLDQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.764304321 (556h11m5.635387356s) [noise=true , address=145.79.227.30:4784 , piece_id=MKCRSFWVFVGOAYCK5M66HO3OT7OMYWLVR2KMDXUGBS2VXXNCUSSQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.765421217 (556h11m5.636504252s) [noise=true , address=145.79.227.44:4216 , piece_id=GSBWC2LXUOZYOVD55KNQYA5Z3S2TCYROBB5O2XZIIVFUQMJQMYNA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:07.082099826 (556h11m6.953182871s) [upload=487609]: Next returned nil segment Dec 28 19:40:07.082116707 (556h11m6.953199752s) [upload=487609]: Waiting for error group managing segments... Dec 28 19:40:07.415884144 (556h11m7.286967179s) [noise=true , address=64.49.13.154:4932 , piece_id=QB5WM5RZ2P2GC6RIGGM34IANVEI2SNDJ2PO556JXKTDBX5HLD4YA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.615434252 (556h11m7.486517297s) [noise=true , address=145.79.227.23:4550 , piece_id=NWIC722LZUHJSAMPWSKCLPRIDGWPVUEYMS5KRAMX3IIMD2KR4XGQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.647182077 (556h11m7.518265112s) [noise=true , address=145.79.227.28:4744 , piece_id=42LII6EOG3CMBTIL5I4RZQPDYU52TKJN3PGWEQXROBYMNDFYIIVA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.678458624 (556h11m7.549541659s) [noise=true , address=145.79.227.46:4626 , piece_id=OQOTWYLVIU7GJBBLYWQOEEFKR3TNAUOXZS6UAFJWJ32SIOTXNXEA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.793406049 (556h11m7.664489084s) [noise=true , address=77.90.31.19:4860 , piece_id=5FUYVK2Z6LBZOAGFI2NOTAAUIV4NMBI5ERMA5RH4BMONNHABZLDQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.802506498 (556h11m7.673589543s) [noise=true , address=77.90.31.21:4896 , piece_id=ATTC7BZUT4HAAEAZCHFZQE4INV2LLSWR2YCUNVA73LPJCW3YL35A, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.853663242 (556h11m7.724746287s) [noise=true , address=77.90.31.10:4522 , piece_id=JOHOMHCNYXF4XQGLN2NJUSWSUO3XFBRRMYD27SLIZAK2RCT3ZC2A, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.880639500 (556h11m7.751722535s) [noise=true , address=77.90.31.11:4582 , piece_id=ZRE24R27TX4E5DTV2POX7EJANQ4CM75D6ZOUS6SKM3QHHQCKRBRA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.904835446 (556h11m7.775918491s) [noise=true , address=77.90.31.18:4816 , piece_id=YT3PSYNQTDQBVBRUDNOXJ2XCN6EFQA54WBSNL6MZMXFXR7XNCHIA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.988914260 (556h11m7.859997295s) [noise=true , address=77.90.31.15:4722 , piece_id=AF7F2OA247CJIDFTEU6HEOIYGOINEYPMII53BKJQXI7UZ26MJIUA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.484927449 (556h11m8.356010484s) [noise=true , address=145.79.227.35:4510 , piece_id=P275TU367DTYXBQOGGV5THFUPHJUWGSM5IMJQ753VE4MX4IPOONQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.610317849 (556h11m8.481400894s) [noise=true , address=145.79.227.44:4216 , piece_id=GSBWC2LXUOZYOVD55KNQYA5Z3S2TCYROBB5O2XZIIVFUQMJQMYNA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.611969885 (556h11m8.48305292s) [noise=true , address=145.79.227.34:4916 , piece_id=SPYSFWMJ752FC5JGRE36G54S2FHG4DQFCV6CIZONQHJDXDCN52VQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.636098801 (556h11m8.507181836s) [noise=true , address=145.79.227.30:4784 , piece_id=MKCRSFWVFVGOAYCK5M66HO3OT7OMYWLVR2KMDXUGBS2VXXNCUSSQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.636122136 (556h11m8.507205181s) [seg_pos={0 0}, upload=487609]: Segment reached optimal threshold of 14 pieces. Dec 28 19:40:08.636333929 (556h11m8.507416974s) [noise=true , address=64.49.13.26:4956 , piece_id=FFHUPOOM7PLEABRHFH6H2SI6WXJBU2XFCLZJUBV4CXUJ6EP4PI2Q, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TYHhjpfyBKkenSwviyQCUZpSBbnaFCGBFRkM4fyKLDuxKx4Ju): protocol: expected piece hash; context canceled; EOF Dec 28 19:40:08.636342175 (556h11m8.50742521s) [noise=true , address=64.49.13.26:4956 , piece_id=FFHUPOOM7PLEABRHFH6H2SI6WXJBU2XFCLZJUBV4CXUJ6EP4PI2Q, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636585959 (556h11m8.507668994s) [noise=true , address=145.79.227.43:4163 , piece_id=77BKLLVD25UMTDQZW46CA2N225I7O4HCHCRRPQ7HLJGRHOLGSDHA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CgPQ5hkrJdqbJvGarxKjXvYYnrXQqapL5BH2rNJNvnhpRi3pw): protocol: expected piece hash; context canceled; EOF Dec 28 19:40:08.636592172 (556h11m8.507675207s) [noise=true , address=145.79.227.43:4163 , piece_id=77BKLLVD25UMTDQZW46CA2N225I7O4HCHCRRPQ7HLJGRHOLGSDHA, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636681511 (556h11m8.507764546s) [noise=true , address=145.79.227.19:4083 , piece_id=GL5XL7T2QQABD5364AT4AFV6RZNOWCTVNSMAVIFJU5SUQFHOPZKQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KZ5a6CQ9tUTrC6EYE7M8yg6p9fK95CN8zQLQmP4tdyz4rBfGh): protocol: expected piece hash; context canceled; EOF Dec 28 19:40:08.636687052 (556h11m8.507770087s) [noise=true , address=145.79.227.19:4083 , piece_id=GL5XL7T2QQABD5364AT4AFV6RZNOWCTVNSMAVIFJU5SUQFHOPZKQ, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636810327 (556h11m8.507893362s) [noise=true , address=145.79.227.17:4992 , piece_id=GUF7AUFCFRRPY5PSYS7DRPFQLGT4ZEL5C7P23US2VI54KYWSUCUQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hbt8HKTunpPKyAzH4XBwGQqTKzcgqPpoWwKZm9JMtaK8P7aVp): protocol: write tcp 188.166.225.170:51210->145.79.227.17:4992: use of closed network connection; write tcp 188.166.225.170:51210->145.79.227.17:4992: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51210->145.79.227.17:4992: use of closed network connection Dec 28 19:40:08.636815957 (556h11m8.507899002s) [noise=true , address=145.79.227.17:4992 , piece_id=GUF7AUFCFRRPY5PSYS7DRPFQLGT4ZEL5C7P23US2VI54KYWSUCUQ, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636834313 (556h11m8.507917358s) [seg_pos={0 0}, upload=487609]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 19:40:08.636907953 (556h11m8.507990998s) [upload=487609]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 19:40:08.636913644 (556h11m8.507996689s) [upload=487609]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 19:40:08.924991397 (556h11m8.796074442s) [upload=487609]: Done uploading segments... Dec 28 19:40:08.925014742 (556h11m8.796097777s) [upload=487609]: Upload finished. err: <nil> Duration: 23.458829353s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 18.708060378s trace: https://jaeger.dev.storj.io/trace/10e1641b228b2a1f FAIL (2025-12-28 20:17:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 40:05.744568479 (556h11m5.615651514s) [noise=true , address=145.79.227.28:4744 , piece_id=42LII6EOG3CMBTIL5I4RZQPDYU52TKJN3PGWEQXROBYMNDFYIIVA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.745099351 (556h11m5.616182396s) [noise=true , address=145.79.227.23:4550 , piece_id=NWIC722LZUHJSAMPWSKCLPRIDGWPVUEYMS5KRAMX3IIMD2KR4XGQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.745777401 (556h11m5.616860436s) [noise=true , address=77.90.31.15:4722 , piece_id=AF7F2OA247CJIDFTEU6HEOIYGOINEYPMII53BKJQXI7UZ26MJIUA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.746374739 (556h11m5.617457784s) [noise=true , address=77.90.31.10:4522 , piece_id=JOHOMHCNYXF4XQGLN2NJUSWSUO3XFBRRMYD27SLIZAK2RCT3ZC2A, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.746934555 (556h11m5.618017601s) [noise=true , address=77.90.31.21:4896 , piece_id=ATTC7BZUT4HAAEAZCHFZQE4INV2LLSWR2YCUNVA73LPJCW3YL35A, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.747556861 (556h11m5.618640146s) [noise=true , address=64.49.13.26:4956 , piece_id=FFHUPOOM7PLEABRHFH6H2SI6WXJBU2XFCLZJUBV4CXUJ6EP4PI2Q, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.748964150 (556h11m5.620047195s) [noise=true , address=145.79.227.34:4916 , piece_id=SPYSFWMJ752FC5JGRE36G54S2FHG4DQFCV6CIZONQHJDXDCN52VQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.751119314 (556h11m5.622202359s) [noise=true , address=77.90.31.18:4816 , piece_id=YT3PSYNQTDQBVBRUDNOXJ2XCN6EFQA54WBSNL6MZMXFXR7XNCHIA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.753413222 (556h11m5.624496257s) [noise=true , address=77.90.31.11:4582 , piece_id=ZRE24R27TX4E5DTV2POX7EJANQ4CM75D6ZOUS6SKM3QHHQCKRBRA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.755230482 (556h11m5.626313527s) [noise=true , address=64.49.13.154:4932 , piece_id=QB5WM5RZ2P2GC6RIGGM34IANVEI2SNDJ2PO556JXKTDBX5HLD4YA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.757598362 (556h11m5.628681407s) [noise=true , address=145.79.227.35:4510 , piece_id=P275TU367DTYXBQOGGV5THFUPHJUWGSM5IMJQ753VE4MX4IPOONQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.759644679 (556h11m5.630727734s) [noise=true , address=145.79.227.46:4626 , piece_id=OQOTWYLVIU7GJBBLYWQOEEFKR3TNAUOXZS6UAFJWJ32SIOTXNXEA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.761217594 (556h11m5.632300639s) [noise=true , address=145.79.227.43:4163 , piece_id=77BKLLVD25UMTDQZW46CA2N225I7O4HCHCRRPQ7HLJGRHOLGSDHA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.762138507 (556h11m5.633221552s) [noise=true , address=77.90.31.19:4860 , piece_id=5FUYVK2Z6LBZOAGFI2NOTAAUIV4NMBI5ERMA5RH4BMONNHABZLDQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.764304321 (556h11m5.635387356s) [noise=true , address=145.79.227.30:4784 , piece_id=MKCRSFWVFVGOAYCK5M66HO3OT7OMYWLVR2KMDXUGBS2VXXNCUSSQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.765421217 (556h11m5.636504252s) [noise=true , address=145.79.227.44:4216 , piece_id=GSBWC2LXUOZYOVD55KNQYA5Z3S2TCYROBB5O2XZIIVFUQMJQMYNA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:07.082099826 (556h11m6.953182871s) [upload=487609]: Next returned nil segment Dec 28 19:40:07.082116707 (556h11m6.953199752s) [upload=487609]: Waiting for error group managing segments... Dec 28 19:40:07.415884144 (556h11m7.286967179s) [noise=true , address=64.49.13.154:4932 , piece_id=QB5WM5RZ2P2GC6RIGGM34IANVEI2SNDJ2PO556JXKTDBX5HLD4YA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.615434252 (556h11m7.486517297s) [noise=true , address=145.79.227.23:4550 , piece_id=NWIC722LZUHJSAMPWSKCLPRIDGWPVUEYMS5KRAMX3IIMD2KR4XGQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.647182077 (556h11m7.518265112s) [noise=true , address=145.79.227.28:4744 , piece_id=42LII6EOG3CMBTIL5I4RZQPDYU52TKJN3PGWEQXROBYMNDFYIIVA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.678458624 (556h11m7.549541659s) [noise=true , address=145.79.227.46:4626 , piece_id=OQOTWYLVIU7GJBBLYWQOEEFKR3TNAUOXZS6UAFJWJ32SIOTXNXEA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.793406049 (556h11m7.664489084s) [noise=true , address=77.90.31.19:4860 , piece_id=5FUYVK2Z6LBZOAGFI2NOTAAUIV4NMBI5ERMA5RH4BMONNHABZLDQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.802506498 (556h11m7.673589543s) [noise=true , address=77.90.31.21:4896 , piece_id=ATTC7BZUT4HAAEAZCHFZQE4INV2LLSWR2YCUNVA73LPJCW3YL35A, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.853663242 (556h11m7.724746287s) [noise=true , address=77.90.31.10:4522 , piece_id=JOHOMHCNYXF4XQGLN2NJUSWSUO3XFBRRMYD27SLIZAK2RCT3ZC2A, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.880639500 (556h11m7.751722535s) [noise=true , address=77.90.31.11:4582 , piece_id=ZRE24R27TX4E5DTV2POX7EJANQ4CM75D6ZOUS6SKM3QHHQCKRBRA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.904835446 (556h11m7.775918491s) [noise=true , address=77.90.31.18:4816 , piece_id=YT3PSYNQTDQBVBRUDNOXJ2XCN6EFQA54WBSNL6MZMXFXR7XNCHIA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.988914260 (556h11m7.859997295s) [noise=true , address=77.90.31.15:4722 , piece_id=AF7F2OA247CJIDFTEU6HEOIYGOINEYPMII53BKJQXI7UZ26MJIUA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.484927449 (556h11m8.356010484s) [noise=true , address=145.79.227.35:4510 , piece_id=P275TU367DTYXBQOGGV5THFUPHJUWGSM5IMJQ753VE4MX4IPOONQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.610317849 (556h11m8.481400894s) [noise=true , address=145.79.227.44:4216 , piece_id=GSBWC2LXUOZYOVD55KNQYA5Z3S2TCYROBB5O2XZIIVFUQMJQMYNA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.611969885 (556h11m8.48305292s) [noise=true , address=145.79.227.34:4916 , piece_id=SPYSFWMJ752FC5JGRE36G54S2FHG4DQFCV6CIZONQHJDXDCN52VQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.636098801 (556h11m8.507181836s) [noise=true , address=145.79.227.30:4784 , piece_id=MKCRSFWVFVGOAYCK5M66HO3OT7OMYWLVR2KMDXUGBS2VXXNCUSSQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.636122136 (556h11m8.507205181s) [seg_pos={0 0}, upload=487609]: Segment reached optimal threshold of 14 pieces. Dec 28 19:40:08.636333929 (556h11m8.507416974s) [noise=true , address=64.49.13.26:4956 , piece_id=FFHUPOOM7PLEABRHFH6H2SI6WXJBU2XFCLZJUBV4CXUJ6EP4PI2Q, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TYHhjpfyBKkenSwviyQCUZpSBbnaFCGBFRkM4fyKLDuxKx4Ju): protocol: expected piece hash; context canceled; EOF Dec 28 19:40:08.636342175 (556h11m8.50742521s) [noise=true , address=64.49.13.26:4956 , piece_id=FFHUPOOM7PLEABRHFH6H2SI6WXJBU2XFCLZJUBV4CXUJ6EP4PI2Q, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636585959 (556h11m8.507668994s) [noise=true , address=145.79.227.43:4163 , piece_id=77BKLLVD25UMTDQZW46CA2N225I7O4HCHCRRPQ7HLJGRHOLGSDHA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CgPQ5hkrJdqbJvGarxKjXvYYnrXQqapL5BH2rNJNvnhpRi3pw): protocol: expected piece hash; context canceled; EOF Dec 28 19:40:08.636592172 (556h11m8.507675207s) [noise=true , address=145.79.227.43:4163 , piece_id=77BKLLVD25UMTDQZW46CA2N225I7O4HCHCRRPQ7HLJGRHOLGSDHA, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636681511 (556h11m8.507764546s) [noise=true , address=145.79.227.19:4083 , piece_id=GL5XL7T2QQABD5364AT4AFV6RZNOWCTVNSMAVIFJU5SUQFHOPZKQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KZ5a6CQ9tUTrC6EYE7M8yg6p9fK95CN8zQLQmP4tdyz4rBfGh): protocol: expected piece hash; context canceled; EOF Dec 28 19:40:08.636687052 (556h11m8.507770087s) [noise=true , address=145.79.227.19:4083 , piece_id=GL5XL7T2QQABD5364AT4AFV6RZNOWCTVNSMAVIFJU5SUQFHOPZKQ, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636810327 (556h11m8.507893362s) [noise=true , address=145.79.227.17:4992 , piece_id=GUF7AUFCFRRPY5PSYS7DRPFQLGT4ZEL5C7P23US2VI54KYWSUCUQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hbt8HKTunpPKyAzH4XBwGQqTKzcgqPpoWwKZm9JMtaK8P7aVp): protocol: write tcp 188.166.225.170:51210->145.79.227.17:4992: use of closed network connection; write tcp 188.166.225.170:51210->145.79.227.17:4992: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51210->145.79.227.17:4992: use of closed network connection Dec 28 19:40:08.636815957 (556h11m8.507899002s) [noise=true , address=145.79.227.17:4992 , piece_id=GUF7AUFCFRRPY5PSYS7DRPFQLGT4ZEL5C7P23US2VI54KYWSUCUQ, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636834313 (556h11m8.507917358s) [seg_pos={0 0}, upload=487609]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 19:40:08.636907953 (556h11m8.507990998s) [upload=487609]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 19:40:08.636913644 (556h11m8.507996689s) [upload=487609]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 19:40:08.924991397 (556h11m8.796074442s) [upload=487609]: Done uploading segments... Dec 28 19:40:08.925014742 (556h11m8.796097777s) [upload=487609]: Upload finished. err: <nil> Duration: 18.70818225s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..X..X......XX..............X..........................X....] duration: 6.195706484s trace: https://jaeger.dev.storj.io/trace/3057620e25d11143 FAIL (2025-12-28 20:17:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 20:13:34.173546860 (556h44m34.044629895s) [upload=488098]: Starting upload Dec 28 20:13:34.173605531 (556h44m34.044688566s) [upload=488098]: Done starting upload Dec 28 20:13:34.173822354 (556h44m34.04490539s) [upload=488098]: Uploading segments... Dec 28 20:13:34.173860978 (556h44m34.044944013s) [upload=488098]: Got next segment. Inline: false Dec 28 20:13:34.173892047 (556h44m34.044975082s) [upload=488098]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 20:13:34.173896415 (556h44m34.04497945s) [upload=488098]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 20:13:34.422688850 (556h44m34.293771905s) [seg_pos={0 0}, upload=488098]: Begin upload segment... Dec 28 20:13:34.422745879 (556h44m34.293828944s) [seg_pos={0 0}, upload=488098]: Done begin upload segment. Dec 28 20:13:34.422771076 (556h44m34.293854141s) [noise=true , address=145.79.227.16:4909 , piece_id=G7U4H2HJWK4VGY42RUBKYGDBLRDZLWX6K772ULG2YAH5OVZ5JPKA, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.422842654 (556h44m34.293925699s) [noise=true , address=145.79.227.37:4566 , piece_id=AXTGBOXYJDN7QRFJWEMU3WBEF74KYERZS2B2SZXSPLZRU2TYC4JA, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.423548137 (556h44m34.294631192s) [noise=true , address=145.79.227.11:4730 , piece_id=6AXVYQY7IIQYL4OHBO3URZFJCC42ZWL3NEMD3JSFHLYMXNTQBMUQ, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.423555861 (556h44m34.294638906s) [noise=true , address=45.91.116.10:4081 , piece_id=GU7LAF6GLPTVIEOQ6GRHZI4VYYZMQUPKMAPOFBFCRQB344CUKKMQ, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.424230055 (556h44m34.29531309s) [noise=true , address=45.91.116.6:4976 , piece_id=4GCSWP27NMVW6VQFIOFBGBKHCZEB5VWC32CXMKKLVGXXFK3EHBXQ, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.424619267 (556h44m34.295702302s) [noise=true , address=145.79.227.36:4519 , piece_id=EJBDJEJAN7WG3HHTXBUEIW7QZ2VY6ZOXKNHBLHNYJ7L6ZVH6M7YA, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.424778910 (556h44m34.295861955s) [noise=true , address=77.90.31.11:4556 , piece_id=FANBXJICXHI33JVWYCLDDB76KSYXXFBTX4MLYXE6ICXQP2JAJAJA, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.425200092 (556h44m34.296283137s) [noise=true , address=77.90.31.18:4808 , piece_id=FIOQVOLXKF2CFNJ57BTCL4WTDPZPEEGAS2CDPF2PNTY27JVFTG6Q, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.425605254 (556h44m34.296688289s) [noise=true , address=77.90.31.17:4770 , piece_id=EA2QWTOVMLD5HKF6FGTB3TZXZZLSI4UVFTPD3MZZ2PAXF47VGJQA, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.426052345 (556h44m34.29713538s) [noise=true , address=145.79.227.51:4411 , piece_id=NHVE27MXBV6Y4VNYIT6G73S4WDAWVKACBQIOKTIOCGELPMLCY4CQ, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.426486622 (556h44m34.297569657s) [noise=true , address=45.91.116.13:4153 , piece_id=XCGSWRC2CZ2NLTPN5T5XTK4VHGIES67NB3FFCBXGI63DLGQ7V2EQ, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.426897945 (556h44m34.29798098s) [noise=true , address=45.91.116.4:4937 , piece_id=V6H4VACBOTZZMXQKCQRPJOBRFQVRF562HZC2GQGBLQGAGKFY5VNQ, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.427411794 (556h44m34.298494829s) [noise=true , address=145.79.227.45:4229 , piece_id=IRJAI3XSHMQYCOIFOMRNETTBXLLAEL3Z3YBOLPGEM2FOBF6S37XQ, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.427894844 (556h44m34.298977879s) [noise=true , address=145.79.227.21:4473 , piece_id=KAYPSFAGCFG3DUA4MVZAAHSH65MCNPDOH2M7GTAYILZO6EPFPXYQ, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.431922001 (556h44m34.303005036s) [noise=true , address=145.79.227.50:4374 , piece_id=DCLHZ4GR6U56WATYQRY7YFCFHO4JYMGLXSBY2OWEM3HSGPJHLDSA, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.436931210 (556h44m34.308014255s) [noise=true , address=145.79.227.38:4979 , piece_id=OUNZYJAAA32ZMIQOOCHK4YF4DCF354XUW4WUCHM3445666VXVLBQ, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:36.115225165 (556h44m35.98630821s) [upload=488098]: Next returned nil segment Dec 28 20:13:36.115243359 (556h44m35.986326395s) [upload=488098]: Waiting for error group managing segments... Dec 28 20:13:36.415982301 (556h44m36.287065346s) [noise=true , address=145.79.227.51:4411 , piece_id=NHVE27MXBV6Y4VNYIT6G73S4WDAWVKACBQIOKTIOCGELPMLCY4CQ, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.430495527 (556h44m36.301578562s) [noise=true , address=145.79.227.45:4229 , piece_id=IRJAI3XSHMQYCOIFOMRNETTBXLLAEL3Z3YBOLPGEM2FOBF6S37XQ, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.435192370 (556h44m36.306275406s) [noise=true , address=145.79.227.21:4473 , piece_id=KAYPSFAGCFG3DUA4MVZAAHSH65MCNPDOH2M7GTAYILZO6EPFPXYQ, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.435831557 (556h44m36.306914602s) [noise=true , address=145.79.227.38:4979 , piece_id=OUNZYJAAA32ZMIQOOCHK4YF4DCF354XUW4WUCHM3445666VXVLBQ, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.444124790 (556h44m36.315207825s) [noise=true , address=145.79.227.50:4374 , piece_id=DCLHZ4GR6U56WATYQRY7YFCFHO4JYMGLXSBY2OWEM3HSGPJHLDSA, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.450320487 (556h44m36.321403533s) [noise=true , address=145.79.227.11:4730 , piece_id=6AXVYQY7IIQYL4OHBO3URZFJCC42ZWL3NEMD3JSFHLYMXNTQBMUQ, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.452177964 (556h44m36.323260999s) [noise=true , address=145.79.227.36:4519 , piece_id=EJBDJEJAN7WG3HHTXBUEIW7QZ2VY6ZOXKNHBLHNYJ7L6ZVH6M7YA, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.462219385 (556h44m36.33330242s) [noise=true , address=145.79.227.37:4566 , piece_id=AXTGBOXYJDN7QRFJWEMU3WBEF74KYERZS2B2SZXSPLZRU2TYC4JA, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.485557775 (556h44m36.35664081s) [noise=true , address=145.79.227.16:4909 , piece_id=G7U4H2HJWK4VGY42RUBKYGDBLRDZLWX6K772ULG2YAH5OVZ5JPKA, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.583906940 (556h44m36.454989975s) [noise=true , address=77.90.31.11:4556 , piece_id=FANBXJICXHI33JVWYCLDDB76KSYXXFBTX4MLYXE6ICXQP2JAJAJA, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.658185680 (556h44m36.529268725s) [noise=true , address=45.91.116.4:4937 , piece_id=V6H4VACBOTZZMXQKCQRPJOBRFQVRF562HZC2GQGBLQGAGKFY5VNQ, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.715312460 (556h44m36.586395495s) [noise=true , address=45.91.116.13:4153 , piece_id=XCGSWRC2CZ2NLTPN5T5XTK4VHGIES67NB3FFCBXGI63DLGQ7V2EQ, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.737246156 (556h44m36.608329201s) [noise=true , address=45.91.116.10:4081 , piece_id=GU7LAF6GLPTVIEOQ6GRHZI4VYYZMQUPKMAPOFBFCRQB344CUKKMQ, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.737270231 (556h44m36.608353266s) [seg_pos={0 0}, upload=488098]: Segment reached optimal threshold of 13 pieces. Dec 28 20:13:36.737506872 (556h44m36.608589927s) [noise=true , address=77.90.31.18:4808 , piece_id=FIOQVOLXKF2CFNJ57BTCL4WTDPZPEEGAS2CDPF2PNTY27JVFTG6Q, seg_pos={0 0}, upload=488098]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: expected piece hash; context canceled; EOF Dec 28 20:13:36.737516099 (556h44m36.608599134s) [noise=true , address=77.90.31.18:4808 , piece_id=FIOQVOLXKF2CFNJ57BTCL4WTDPZPEEGAS2CDPF2PNTY27JVFTG6Q, seg_pos={0 0}, upload=488098]: Long tail cancellation due to optimal threshold Dec 28 20:13:36.737659642 (556h44m36.608742688s) [noise=true , address=45.91.116.6:4976 , piece_id=4GCSWP27NMVW6VQFIOFBGBKHCZEB5VWC32CXMKKLVGXXFK3EHBXQ, seg_pos={0 0}, upload=488098]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Dec 28 20:13:36.737707214 (556h44m36.608790259s) [noise=true , address=45.91.116.6:4976 , piece_id=4GCSWP27NMVW6VQFIOFBGBKHCZEB5VWC32CXMKKLVGXXFK3EHBXQ, seg_pos={0 0}, upload=488098]: Long tail cancellation due to optimal threshold Dec 28 20:13:36.737939255 (556h44m36.6090223s) [noise=true , address=77.90.31.17:4770 , piece_id=EA2QWTOVMLD5HKF6FGTB3TZXZZLSI4UVFTPD3MZZ2PAXF47VGJQA, seg_pos={0 0}, upload=488098]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: write tcp 188.166.225.170:53958->77.90.31.17:4770: use of closed network connection; write tcp 188.166.225.170:53958->77.90.31.17:4770: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53958->77.90.31.17:4770: use of closed network connection Dec 28 20:13:36.737965024 (556h44m36.609048079s) [noise=true , address=77.90.31.17:4770 , piece_id=EA2QWTOVMLD5HKF6FGTB3TZXZZLSI4UVFTPD3MZZ2PAXF47VGJQA, seg_pos={0 0}, upload=488098]: Long tail cancellation due to optimal threshold Dec 28 20:13:36.737999810 (556h44m36.609082846s) [seg_pos={0 0}, upload=488098]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 20:13:36.738101905 (556h44m36.60918495s) [upload=488098]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 20:13:36.738109249 (556h44m36.609192304s) [upload=488098]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 20:13:37.032410772 (556h44m36.903493817s) [upload=488098]: Done uploading segments... Dec 28 20:13:37.032430300 (556h44m36.903513345s) [upload=488098]: Upload finished. err: <nil> Duration: 6.195784933s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [X..X..X......XX..............X..........................X...] duration: 9.735311222s trace: https://jaeger.dev.storj.io/trace/3057620e25d11143 FAIL (2025-12-28 20:16:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 20:13:34.173546860 (556h44m34.044629895s) [upload=488098]: Starting upload Dec 28 20:13:34.173605531 (556h44m34.044688566s) [upload=488098]: Done starting upload Dec 28 20:13:34.173822354 (556h44m34.04490539s) [upload=488098]: Uploading segments... Dec 28 20:13:34.173860978 (556h44m34.044944013s) [upload=488098]: Got next segment. Inline: false Dec 28 20:13:34.173892047 (556h44m34.044975082s) [upload=488098]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 20:13:34.173896415 (556h44m34.04497945s) [upload=488098]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 20:13:34.422688850 (556h44m34.293771905s) [seg_pos={0 0}, upload=488098]: Begin upload segment... Dec 28 20:13:34.422745879 (556h44m34.293828944s) [seg_pos={0 0}, upload=488098]: Done begin upload segment. Dec 28 20:13:34.422771076 (556h44m34.293854141s) [noise=true , address=145.79.227.16:4909 , piece_id=G7U4H2HJWK4VGY42RUBKYGDBLRDZLWX6K772ULG2YAH5OVZ5JPKA, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.422842654 (556h44m34.293925699s) [noise=true , address=145.79.227.37:4566 , piece_id=AXTGBOXYJDN7QRFJWEMU3WBEF74KYERZS2B2SZXSPLZRU2TYC4JA, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.423548137 (556h44m34.294631192s) [noise=true , address=145.79.227.11:4730 , piece_id=6AXVYQY7IIQYL4OHBO3URZFJCC42ZWL3NEMD3JSFHLYMXNTQBMUQ, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.423555861 (556h44m34.294638906s) [noise=true , address=45.91.116.10:4081 , piece_id=GU7LAF6GLPTVIEOQ6GRHZI4VYYZMQUPKMAPOFBFCRQB344CUKKMQ, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.424230055 (556h44m34.29531309s) [noise=true , address=45.91.116.6:4976 , piece_id=4GCSWP27NMVW6VQFIOFBGBKHCZEB5VWC32CXMKKLVGXXFK3EHBXQ, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.424619267 (556h44m34.295702302s) [noise=true , address=145.79.227.36:4519 , piece_id=EJBDJEJAN7WG3HHTXBUEIW7QZ2VY6ZOXKNHBLHNYJ7L6ZVH6M7YA, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.424778910 (556h44m34.295861955s) [noise=true , address=77.90.31.11:4556 , piece_id=FANBXJICXHI33JVWYCLDDB76KSYXXFBTX4MLYXE6ICXQP2JAJAJA, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.425200092 (556h44m34.296283137s) [noise=true , address=77.90.31.18:4808 , piece_id=FIOQVOLXKF2CFNJ57BTCL4WTDPZPEEGAS2CDPF2PNTY27JVFTG6Q, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.425605254 (556h44m34.296688289s) [noise=true , address=77.90.31.17:4770 , piece_id=EA2QWTOVMLD5HKF6FGTB3TZXZZLSI4UVFTPD3MZZ2PAXF47VGJQA, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.426052345 (556h44m34.29713538s) [noise=true , address=145.79.227.51:4411 , piece_id=NHVE27MXBV6Y4VNYIT6G73S4WDAWVKACBQIOKTIOCGELPMLCY4CQ, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.426486622 (556h44m34.297569657s) [noise=true , address=45.91.116.13:4153 , piece_id=XCGSWRC2CZ2NLTPN5T5XTK4VHGIES67NB3FFCBXGI63DLGQ7V2EQ, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.426897945 (556h44m34.29798098s) [noise=true , address=45.91.116.4:4937 , piece_id=V6H4VACBOTZZMXQKCQRPJOBRFQVRF562HZC2GQGBLQGAGKFY5VNQ, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.427411794 (556h44m34.298494829s) [noise=true , address=145.79.227.45:4229 , piece_id=IRJAI3XSHMQYCOIFOMRNETTBXLLAEL3Z3YBOLPGEM2FOBF6S37XQ, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.427894844 (556h44m34.298977879s) [noise=true , address=145.79.227.21:4473 , piece_id=KAYPSFAGCFG3DUA4MVZAAHSH65MCNPDOH2M7GTAYILZO6EPFPXYQ, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.431922001 (556h44m34.303005036s) [noise=true , address=145.79.227.50:4374 , piece_id=DCLHZ4GR6U56WATYQRY7YFCFHO4JYMGLXSBY2OWEM3HSGPJHLDSA, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.436931210 (556h44m34.308014255s) [noise=true , address=145.79.227.38:4979 , piece_id=OUNZYJAAA32ZMIQOOCHK4YF4DCF354XUW4WUCHM3445666VXVLBQ, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:36.115225165 (556h44m35.98630821s) [upload=488098]: Next returned nil segment Dec 28 20:13:36.115243359 (556h44m35.986326395s) [upload=488098]: Waiting for error group managing segments... Dec 28 20:13:36.415982301 (556h44m36.287065346s) [noise=true , address=145.79.227.51:4411 , piece_id=NHVE27MXBV6Y4VNYIT6G73S4WDAWVKACBQIOKTIOCGELPMLCY4CQ, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.430495527 (556h44m36.301578562s) [noise=true , address=145.79.227.45:4229 , piece_id=IRJAI3XSHMQYCOIFOMRNETTBXLLAEL3Z3YBOLPGEM2FOBF6S37XQ, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.435192370 (556h44m36.306275406s) [noise=true , address=145.79.227.21:4473 , piece_id=KAYPSFAGCFG3DUA4MVZAAHSH65MCNPDOH2M7GTAYILZO6EPFPXYQ, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.435831557 (556h44m36.306914602s) [noise=true , address=145.79.227.38:4979 , piece_id=OUNZYJAAA32ZMIQOOCHK4YF4DCF354XUW4WUCHM3445666VXVLBQ, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.444124790 (556h44m36.315207825s) [noise=true , address=145.79.227.50:4374 , piece_id=DCLHZ4GR6U56WATYQRY7YFCFHO4JYMGLXSBY2OWEM3HSGPJHLDSA, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.450320487 (556h44m36.321403533s) [noise=true , address=145.79.227.11:4730 , piece_id=6AXVYQY7IIQYL4OHBO3URZFJCC42ZWL3NEMD3JSFHLYMXNTQBMUQ, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.452177964 (556h44m36.323260999s) [noise=true , address=145.79.227.36:4519 , piece_id=EJBDJEJAN7WG3HHTXBUEIW7QZ2VY6ZOXKNHBLHNYJ7L6ZVH6M7YA, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.462219385 (556h44m36.33330242s) [noise=true , address=145.79.227.37:4566 , piece_id=AXTGBOXYJDN7QRFJWEMU3WBEF74KYERZS2B2SZXSPLZRU2TYC4JA, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.485557775 (556h44m36.35664081s) [noise=true , address=145.79.227.16:4909 , piece_id=G7U4H2HJWK4VGY42RUBKYGDBLRDZLWX6K772ULG2YAH5OVZ5JPKA, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.583906940 (556h44m36.454989975s) [noise=true , address=77.90.31.11:4556 , piece_id=FANBXJICXHI33JVWYCLDDB76KSYXXFBTX4MLYXE6ICXQP2JAJAJA, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.658185680 (556h44m36.529268725s) [noise=true , address=45.91.116.4:4937 , piece_id=V6H4VACBOTZZMXQKCQRPJOBRFQVRF562HZC2GQGBLQGAGKFY5VNQ, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.715312460 (556h44m36.586395495s) [noise=true , address=45.91.116.13:4153 , piece_id=XCGSWRC2CZ2NLTPN5T5XTK4VHGIES67NB3FFCBXGI63DLGQ7V2EQ, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.737246156 (556h44m36.608329201s) [noise=true , address=45.91.116.10:4081 , piece_id=GU7LAF6GLPTVIEOQ6GRHZI4VYYZMQUPKMAPOFBFCRQB344CUKKMQ, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.737270231 (556h44m36.608353266s) [seg_pos={0 0}, upload=488098]: Segment reached optimal threshold of 13 pieces. Dec 28 20:13:36.737506872 (556h44m36.608589927s) [noise=true , address=77.90.31.18:4808 , piece_id=FIOQVOLXKF2CFNJ57BTCL4WTDPZPEEGAS2CDPF2PNTY27JVFTG6Q, seg_pos={0 0}, upload=488098]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: expected piece hash; context canceled; EOF Dec 28 20:13:36.737516099 (556h44m36.608599134s) [noise=true , address=77.90.31.18:4808 , piece_id=FIOQVOLXKF2CFNJ57BTCL4WTDPZPEEGAS2CDPF2PNTY27JVFTG6Q, seg_pos={0 0}, upload=488098]: Long tail cancellation due to optimal threshold Dec 28 20:13:36.737659642 (556h44m36.608742688s) [noise=true , address=45.91.116.6:4976 , piece_id=4GCSWP27NMVW6VQFIOFBGBKHCZEB5VWC32CXMKKLVGXXFK3EHBXQ, seg_pos={0 0}, upload=488098]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Dec 28 20:13:36.737707214 (556h44m36.608790259s) [noise=true , address=45.91.116.6:4976 , piece_id=4GCSWP27NMVW6VQFIOFBGBKHCZEB5VWC32CXMKKLVGXXFK3EHBXQ, seg_pos={0 0}, upload=488098]: Long tail cancellation due to optimal threshold Dec 28 20:13:36.737939255 (556h44m36.6090223s) [noise=true , address=77.90.31.17:4770 , piece_id=EA2QWTOVMLD5HKF6FGTB3TZXZZLSI4UVFTPD3MZZ2PAXF47VGJQA, seg_pos={0 0}, upload=488098]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: write tcp 188.166.225.170:53958->77.90.31.17:4770: use of closed network connection; write tcp 188.166.225.170:53958->77.90.31.17:4770: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53958->77.90.31.17:4770: use of closed network connection Dec 28 20:13:36.737965024 (556h44m36.609048079s) [noise=true , address=77.90.31.17:4770 , piece_id=EA2QWTOVMLD5HKF6FGTB3TZXZZLSI4UVFTPD3MZZ2PAXF47VGJQA, seg_pos={0 0}, upload=488098]: Long tail cancellation due to optimal threshold Dec 28 20:13:36.737999810 (556h44m36.609082846s) [seg_pos={0 0}, upload=488098]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 20:13:36.738101905 (556h44m36.60918495s) [upload=488098]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 20:13:36.738109249 (556h44m36.609192304s) [upload=488098]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 20:13:37.032410772 (556h44m36.903493817s) [upload=488098]: Done uploading segments... Dec 28 20:13:37.032430300 (556h44m36.903513345s) [upload=488098]: Upload finished. err: <nil> Duration: 9.735345115s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 33.657264063s trace: https://jaeger.dev.storj.io/trace/10e1641b228b2a1f FAIL (2025-12-28 20:16:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 40:05.744568479 (556h11m5.615651514s) [noise=true , address=145.79.227.28:4744 , piece_id=42LII6EOG3CMBTIL5I4RZQPDYU52TKJN3PGWEQXROBYMNDFYIIVA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.745099351 (556h11m5.616182396s) [noise=true , address=145.79.227.23:4550 , piece_id=NWIC722LZUHJSAMPWSKCLPRIDGWPVUEYMS5KRAMX3IIMD2KR4XGQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.745777401 (556h11m5.616860436s) [noise=true , address=77.90.31.15:4722 , piece_id=AF7F2OA247CJIDFTEU6HEOIYGOINEYPMII53BKJQXI7UZ26MJIUA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.746374739 (556h11m5.617457784s) [noise=true , address=77.90.31.10:4522 , piece_id=JOHOMHCNYXF4XQGLN2NJUSWSUO3XFBRRMYD27SLIZAK2RCT3ZC2A, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.746934555 (556h11m5.618017601s) [noise=true , address=77.90.31.21:4896 , piece_id=ATTC7BZUT4HAAEAZCHFZQE4INV2LLSWR2YCUNVA73LPJCW3YL35A, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.747556861 (556h11m5.618640146s) [noise=true , address=64.49.13.26:4956 , piece_id=FFHUPOOM7PLEABRHFH6H2SI6WXJBU2XFCLZJUBV4CXUJ6EP4PI2Q, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.748964150 (556h11m5.620047195s) [noise=true , address=145.79.227.34:4916 , piece_id=SPYSFWMJ752FC5JGRE36G54S2FHG4DQFCV6CIZONQHJDXDCN52VQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.751119314 (556h11m5.622202359s) [noise=true , address=77.90.31.18:4816 , piece_id=YT3PSYNQTDQBVBRUDNOXJ2XCN6EFQA54WBSNL6MZMXFXR7XNCHIA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.753413222 (556h11m5.624496257s) [noise=true , address=77.90.31.11:4582 , piece_id=ZRE24R27TX4E5DTV2POX7EJANQ4CM75D6ZOUS6SKM3QHHQCKRBRA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.755230482 (556h11m5.626313527s) [noise=true , address=64.49.13.154:4932 , piece_id=QB5WM5RZ2P2GC6RIGGM34IANVEI2SNDJ2PO556JXKTDBX5HLD4YA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.757598362 (556h11m5.628681407s) [noise=true , address=145.79.227.35:4510 , piece_id=P275TU367DTYXBQOGGV5THFUPHJUWGSM5IMJQ753VE4MX4IPOONQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.759644679 (556h11m5.630727734s) [noise=true , address=145.79.227.46:4626 , piece_id=OQOTWYLVIU7GJBBLYWQOEEFKR3TNAUOXZS6UAFJWJ32SIOTXNXEA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.761217594 (556h11m5.632300639s) [noise=true , address=145.79.227.43:4163 , piece_id=77BKLLVD25UMTDQZW46CA2N225I7O4HCHCRRPQ7HLJGRHOLGSDHA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.762138507 (556h11m5.633221552s) [noise=true , address=77.90.31.19:4860 , piece_id=5FUYVK2Z6LBZOAGFI2NOTAAUIV4NMBI5ERMA5RH4BMONNHABZLDQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.764304321 (556h11m5.635387356s) [noise=true , address=145.79.227.30:4784 , piece_id=MKCRSFWVFVGOAYCK5M66HO3OT7OMYWLVR2KMDXUGBS2VXXNCUSSQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.765421217 (556h11m5.636504252s) [noise=true , address=145.79.227.44:4216 , piece_id=GSBWC2LXUOZYOVD55KNQYA5Z3S2TCYROBB5O2XZIIVFUQMJQMYNA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:07.082099826 (556h11m6.953182871s) [upload=487609]: Next returned nil segment Dec 28 19:40:07.082116707 (556h11m6.953199752s) [upload=487609]: Waiting for error group managing segments... Dec 28 19:40:07.415884144 (556h11m7.286967179s) [noise=true , address=64.49.13.154:4932 , piece_id=QB5WM5RZ2P2GC6RIGGM34IANVEI2SNDJ2PO556JXKTDBX5HLD4YA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.615434252 (556h11m7.486517297s) [noise=true , address=145.79.227.23:4550 , piece_id=NWIC722LZUHJSAMPWSKCLPRIDGWPVUEYMS5KRAMX3IIMD2KR4XGQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.647182077 (556h11m7.518265112s) [noise=true , address=145.79.227.28:4744 , piece_id=42LII6EOG3CMBTIL5I4RZQPDYU52TKJN3PGWEQXROBYMNDFYIIVA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.678458624 (556h11m7.549541659s) [noise=true , address=145.79.227.46:4626 , piece_id=OQOTWYLVIU7GJBBLYWQOEEFKR3TNAUOXZS6UAFJWJ32SIOTXNXEA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.793406049 (556h11m7.664489084s) [noise=true , address=77.90.31.19:4860 , piece_id=5FUYVK2Z6LBZOAGFI2NOTAAUIV4NMBI5ERMA5RH4BMONNHABZLDQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.802506498 (556h11m7.673589543s) [noise=true , address=77.90.31.21:4896 , piece_id=ATTC7BZUT4HAAEAZCHFZQE4INV2LLSWR2YCUNVA73LPJCW3YL35A, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.853663242 (556h11m7.724746287s) [noise=true , address=77.90.31.10:4522 , piece_id=JOHOMHCNYXF4XQGLN2NJUSWSUO3XFBRRMYD27SLIZAK2RCT3ZC2A, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.880639500 (556h11m7.751722535s) [noise=true , address=77.90.31.11:4582 , piece_id=ZRE24R27TX4E5DTV2POX7EJANQ4CM75D6ZOUS6SKM3QHHQCKRBRA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.904835446 (556h11m7.775918491s) [noise=true , address=77.90.31.18:4816 , piece_id=YT3PSYNQTDQBVBRUDNOXJ2XCN6EFQA54WBSNL6MZMXFXR7XNCHIA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.988914260 (556h11m7.859997295s) [noise=true , address=77.90.31.15:4722 , piece_id=AF7F2OA247CJIDFTEU6HEOIYGOINEYPMII53BKJQXI7UZ26MJIUA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.484927449 (556h11m8.356010484s) [noise=true , address=145.79.227.35:4510 , piece_id=P275TU367DTYXBQOGGV5THFUPHJUWGSM5IMJQ753VE4MX4IPOONQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.610317849 (556h11m8.481400894s) [noise=true , address=145.79.227.44:4216 , piece_id=GSBWC2LXUOZYOVD55KNQYA5Z3S2TCYROBB5O2XZIIVFUQMJQMYNA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.611969885 (556h11m8.48305292s) [noise=true , address=145.79.227.34:4916 , piece_id=SPYSFWMJ752FC5JGRE36G54S2FHG4DQFCV6CIZONQHJDXDCN52VQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.636098801 (556h11m8.507181836s) [noise=true , address=145.79.227.30:4784 , piece_id=MKCRSFWVFVGOAYCK5M66HO3OT7OMYWLVR2KMDXUGBS2VXXNCUSSQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.636122136 (556h11m8.507205181s) [seg_pos={0 0}, upload=487609]: Segment reached optimal threshold of 14 pieces. Dec 28 19:40:08.636333929 (556h11m8.507416974s) [noise=true , address=64.49.13.26:4956 , piece_id=FFHUPOOM7PLEABRHFH6H2SI6WXJBU2XFCLZJUBV4CXUJ6EP4PI2Q, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TYHhjpfyBKkenSwviyQCUZpSBbnaFCGBFRkM4fyKLDuxKx4Ju): protocol: expected piece hash; context canceled; EOF Dec 28 19:40:08.636342175 (556h11m8.50742521s) [noise=true , address=64.49.13.26:4956 , piece_id=FFHUPOOM7PLEABRHFH6H2SI6WXJBU2XFCLZJUBV4CXUJ6EP4PI2Q, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636585959 (556h11m8.507668994s) [noise=true , address=145.79.227.43:4163 , piece_id=77BKLLVD25UMTDQZW46CA2N225I7O4HCHCRRPQ7HLJGRHOLGSDHA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CgPQ5hkrJdqbJvGarxKjXvYYnrXQqapL5BH2rNJNvnhpRi3pw): protocol: expected piece hash; context canceled; EOF Dec 28 19:40:08.636592172 (556h11m8.507675207s) [noise=true , address=145.79.227.43:4163 , piece_id=77BKLLVD25UMTDQZW46CA2N225I7O4HCHCRRPQ7HLJGRHOLGSDHA, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636681511 (556h11m8.507764546s) [noise=true , address=145.79.227.19:4083 , piece_id=GL5XL7T2QQABD5364AT4AFV6RZNOWCTVNSMAVIFJU5SUQFHOPZKQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KZ5a6CQ9tUTrC6EYE7M8yg6p9fK95CN8zQLQmP4tdyz4rBfGh): protocol: expected piece hash; context canceled; EOF Dec 28 19:40:08.636687052 (556h11m8.507770087s) [noise=true , address=145.79.227.19:4083 , piece_id=GL5XL7T2QQABD5364AT4AFV6RZNOWCTVNSMAVIFJU5SUQFHOPZKQ, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636810327 (556h11m8.507893362s) [noise=true , address=145.79.227.17:4992 , piece_id=GUF7AUFCFRRPY5PSYS7DRPFQLGT4ZEL5C7P23US2VI54KYWSUCUQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hbt8HKTunpPKyAzH4XBwGQqTKzcgqPpoWwKZm9JMtaK8P7aVp): protocol: write tcp 188.166.225.170:51210->145.79.227.17:4992: use of closed network connection; write tcp 188.166.225.170:51210->145.79.227.17:4992: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51210->145.79.227.17:4992: use of closed network connection Dec 28 19:40:08.636815957 (556h11m8.507899002s) [noise=true , address=145.79.227.17:4992 , piece_id=GUF7AUFCFRRPY5PSYS7DRPFQLGT4ZEL5C7P23US2VI54KYWSUCUQ, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636834313 (556h11m8.507917358s) [seg_pos={0 0}, upload=487609]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 19:40:08.636907953 (556h11m8.507990998s) [upload=487609]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 19:40:08.636913644 (556h11m8.507996689s) [upload=487609]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 19:40:08.924991397 (556h11m8.796074442s) [upload=487609]: Done uploading segments... Dec 28 19:40:08.925014742 (556h11m8.796097777s) [upload=487609]: Upload finished. err: <nil> Duration: 33.65728895s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [XX..X..X......XX..............X..........................X..] duration: 5.929275854s trace: https://jaeger.dev.storj.io/trace/3057620e25d11143 FAIL (2025-12-28 20:15:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 20:13:34.173546860 (556h44m34.044629895s) [upload=488098]: Starting upload Dec 28 20:13:34.173605531 (556h44m34.044688566s) [upload=488098]: Done starting upload Dec 28 20:13:34.173822354 (556h44m34.04490539s) [upload=488098]: Uploading segments... Dec 28 20:13:34.173860978 (556h44m34.044944013s) [upload=488098]: Got next segment. Inline: false Dec 28 20:13:34.173892047 (556h44m34.044975082s) [upload=488098]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 20:13:34.173896415 (556h44m34.04497945s) [upload=488098]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 20:13:34.422688850 (556h44m34.293771905s) [seg_pos={0 0}, upload=488098]: Begin upload segment... Dec 28 20:13:34.422745879 (556h44m34.293828944s) [seg_pos={0 0}, upload=488098]: Done begin upload segment. Dec 28 20:13:34.422771076 (556h44m34.293854141s) [noise=true , address=145.79.227.16:4909 , piece_id=G7U4H2HJWK4VGY42RUBKYGDBLRDZLWX6K772ULG2YAH5OVZ5JPKA, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.422842654 (556h44m34.293925699s) [noise=true , address=145.79.227.37:4566 , piece_id=AXTGBOXYJDN7QRFJWEMU3WBEF74KYERZS2B2SZXSPLZRU2TYC4JA, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.423548137 (556h44m34.294631192s) [noise=true , address=145.79.227.11:4730 , piece_id=6AXVYQY7IIQYL4OHBO3URZFJCC42ZWL3NEMD3JSFHLYMXNTQBMUQ, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.423555861 (556h44m34.294638906s) [noise=true , address=45.91.116.10:4081 , piece_id=GU7LAF6GLPTVIEOQ6GRHZI4VYYZMQUPKMAPOFBFCRQB344CUKKMQ, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.424230055 (556h44m34.29531309s) [noise=true , address=45.91.116.6:4976 , piece_id=4GCSWP27NMVW6VQFIOFBGBKHCZEB5VWC32CXMKKLVGXXFK3EHBXQ, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.424619267 (556h44m34.295702302s) [noise=true , address=145.79.227.36:4519 , piece_id=EJBDJEJAN7WG3HHTXBUEIW7QZ2VY6ZOXKNHBLHNYJ7L6ZVH6M7YA, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.424778910 (556h44m34.295861955s) [noise=true , address=77.90.31.11:4556 , piece_id=FANBXJICXHI33JVWYCLDDB76KSYXXFBTX4MLYXE6ICXQP2JAJAJA, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.425200092 (556h44m34.296283137s) [noise=true , address=77.90.31.18:4808 , piece_id=FIOQVOLXKF2CFNJ57BTCL4WTDPZPEEGAS2CDPF2PNTY27JVFTG6Q, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.425605254 (556h44m34.296688289s) [noise=true , address=77.90.31.17:4770 , piece_id=EA2QWTOVMLD5HKF6FGTB3TZXZZLSI4UVFTPD3MZZ2PAXF47VGJQA, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.426052345 (556h44m34.29713538s) [noise=true , address=145.79.227.51:4411 , piece_id=NHVE27MXBV6Y4VNYIT6G73S4WDAWVKACBQIOKTIOCGELPMLCY4CQ, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.426486622 (556h44m34.297569657s) [noise=true , address=45.91.116.13:4153 , piece_id=XCGSWRC2CZ2NLTPN5T5XTK4VHGIES67NB3FFCBXGI63DLGQ7V2EQ, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.426897945 (556h44m34.29798098s) [noise=true , address=45.91.116.4:4937 , piece_id=V6H4VACBOTZZMXQKCQRPJOBRFQVRF562HZC2GQGBLQGAGKFY5VNQ, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.427411794 (556h44m34.298494829s) [noise=true , address=145.79.227.45:4229 , piece_id=IRJAI3XSHMQYCOIFOMRNETTBXLLAEL3Z3YBOLPGEM2FOBF6S37XQ, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.427894844 (556h44m34.298977879s) [noise=true , address=145.79.227.21:4473 , piece_id=KAYPSFAGCFG3DUA4MVZAAHSH65MCNPDOH2M7GTAYILZO6EPFPXYQ, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.431922001 (556h44m34.303005036s) [noise=true , address=145.79.227.50:4374 , piece_id=DCLHZ4GR6U56WATYQRY7YFCFHO4JYMGLXSBY2OWEM3HSGPJHLDSA, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.436931210 (556h44m34.308014255s) [noise=true , address=145.79.227.38:4979 , piece_id=OUNZYJAAA32ZMIQOOCHK4YF4DCF354XUW4WUCHM3445666VXVLBQ, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:36.115225165 (556h44m35.98630821s) [upload=488098]: Next returned nil segment Dec 28 20:13:36.115243359 (556h44m35.986326395s) [upload=488098]: Waiting for error group managing segments... Dec 28 20:13:36.415982301 (556h44m36.287065346s) [noise=true , address=145.79.227.51:4411 , piece_id=NHVE27MXBV6Y4VNYIT6G73S4WDAWVKACBQIOKTIOCGELPMLCY4CQ, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.430495527 (556h44m36.301578562s) [noise=true , address=145.79.227.45:4229 , piece_id=IRJAI3XSHMQYCOIFOMRNETTBXLLAEL3Z3YBOLPGEM2FOBF6S37XQ, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.435192370 (556h44m36.306275406s) [noise=true , address=145.79.227.21:4473 , piece_id=KAYPSFAGCFG3DUA4MVZAAHSH65MCNPDOH2M7GTAYILZO6EPFPXYQ, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.435831557 (556h44m36.306914602s) [noise=true , address=145.79.227.38:4979 , piece_id=OUNZYJAAA32ZMIQOOCHK4YF4DCF354XUW4WUCHM3445666VXVLBQ, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.444124790 (556h44m36.315207825s) [noise=true , address=145.79.227.50:4374 , piece_id=DCLHZ4GR6U56WATYQRY7YFCFHO4JYMGLXSBY2OWEM3HSGPJHLDSA, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.450320487 (556h44m36.321403533s) [noise=true , address=145.79.227.11:4730 , piece_id=6AXVYQY7IIQYL4OHBO3URZFJCC42ZWL3NEMD3JSFHLYMXNTQBMUQ, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.452177964 (556h44m36.323260999s) [noise=true , address=145.79.227.36:4519 , piece_id=EJBDJEJAN7WG3HHTXBUEIW7QZ2VY6ZOXKNHBLHNYJ7L6ZVH6M7YA, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.462219385 (556h44m36.33330242s) [noise=true , address=145.79.227.37:4566 , piece_id=AXTGBOXYJDN7QRFJWEMU3WBEF74KYERZS2B2SZXSPLZRU2TYC4JA, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.485557775 (556h44m36.35664081s) [noise=true , address=145.79.227.16:4909 , piece_id=G7U4H2HJWK4VGY42RUBKYGDBLRDZLWX6K772ULG2YAH5OVZ5JPKA, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.583906940 (556h44m36.454989975s) [noise=true , address=77.90.31.11:4556 , piece_id=FANBXJICXHI33JVWYCLDDB76KSYXXFBTX4MLYXE6ICXQP2JAJAJA, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.658185680 (556h44m36.529268725s) [noise=true , address=45.91.116.4:4937 , piece_id=V6H4VACBOTZZMXQKCQRPJOBRFQVRF562HZC2GQGBLQGAGKFY5VNQ, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.715312460 (556h44m36.586395495s) [noise=true , address=45.91.116.13:4153 , piece_id=XCGSWRC2CZ2NLTPN5T5XTK4VHGIES67NB3FFCBXGI63DLGQ7V2EQ, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.737246156 (556h44m36.608329201s) [noise=true , address=45.91.116.10:4081 , piece_id=GU7LAF6GLPTVIEOQ6GRHZI4VYYZMQUPKMAPOFBFCRQB344CUKKMQ, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.737270231 (556h44m36.608353266s) [seg_pos={0 0}, upload=488098]: Segment reached optimal threshold of 13 pieces. Dec 28 20:13:36.737506872 (556h44m36.608589927s) [noise=true , address=77.90.31.18:4808 , piece_id=FIOQVOLXKF2CFNJ57BTCL4WTDPZPEEGAS2CDPF2PNTY27JVFTG6Q, seg_pos={0 0}, upload=488098]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: expected piece hash; context canceled; EOF Dec 28 20:13:36.737516099 (556h44m36.608599134s) [noise=true , address=77.90.31.18:4808 , piece_id=FIOQVOLXKF2CFNJ57BTCL4WTDPZPEEGAS2CDPF2PNTY27JVFTG6Q, seg_pos={0 0}, upload=488098]: Long tail cancellation due to optimal threshold Dec 28 20:13:36.737659642 (556h44m36.608742688s) [noise=true , address=45.91.116.6:4976 , piece_id=4GCSWP27NMVW6VQFIOFBGBKHCZEB5VWC32CXMKKLVGXXFK3EHBXQ, seg_pos={0 0}, upload=488098]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Dec 28 20:13:36.737707214 (556h44m36.608790259s) [noise=true , address=45.91.116.6:4976 , piece_id=4GCSWP27NMVW6VQFIOFBGBKHCZEB5VWC32CXMKKLVGXXFK3EHBXQ, seg_pos={0 0}, upload=488098]: Long tail cancellation due to optimal threshold Dec 28 20:13:36.737939255 (556h44m36.6090223s) [noise=true , address=77.90.31.17:4770 , piece_id=EA2QWTOVMLD5HKF6FGTB3TZXZZLSI4UVFTPD3MZZ2PAXF47VGJQA, seg_pos={0 0}, upload=488098]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: write tcp 188.166.225.170:53958->77.90.31.17:4770: use of closed network connection; write tcp 188.166.225.170:53958->77.90.31.17:4770: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53958->77.90.31.17:4770: use of closed network connection Dec 28 20:13:36.737965024 (556h44m36.609048079s) [noise=true , address=77.90.31.17:4770 , piece_id=EA2QWTOVMLD5HKF6FGTB3TZXZZLSI4UVFTPD3MZZ2PAXF47VGJQA, seg_pos={0 0}, upload=488098]: Long tail cancellation due to optimal threshold Dec 28 20:13:36.737999810 (556h44m36.609082846s) [seg_pos={0 0}, upload=488098]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 20:13:36.738101905 (556h44m36.60918495s) [upload=488098]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 20:13:36.738109249 (556h44m36.609192304s) [upload=488098]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 20:13:37.032410772 (556h44m36.903493817s) [upload=488098]: Done uploading segments... Dec 28 20:13:37.032430300 (556h44m36.903513345s) [upload=488098]: Upload finished. err: <nil> Duration: 5.929309838s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [XXX..X..X......XX..............X..........................X.] duration: 17.269152464s trace: https://jaeger.dev.storj.io/trace/3057620e25d11143 FAIL (2025-12-28 20:14:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 20:13:34.173546860 (556h44m34.044629895s) [upload=488098]: Starting upload Dec 28 20:13:34.173605531 (556h44m34.044688566s) [upload=488098]: Done starting upload Dec 28 20:13:34.173822354 (556h44m34.04490539s) [upload=488098]: Uploading segments... Dec 28 20:13:34.173860978 (556h44m34.044944013s) [upload=488098]: Got next segment. Inline: false Dec 28 20:13:34.173892047 (556h44m34.044975082s) [upload=488098]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 20:13:34.173896415 (556h44m34.04497945s) [upload=488098]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 20:13:34.422688850 (556h44m34.293771905s) [seg_pos={0 0}, upload=488098]: Begin upload segment... Dec 28 20:13:34.422745879 (556h44m34.293828944s) [seg_pos={0 0}, upload=488098]: Done begin upload segment. Dec 28 20:13:34.422771076 (556h44m34.293854141s) [noise=true , address=145.79.227.16:4909 , piece_id=G7U4H2HJWK4VGY42RUBKYGDBLRDZLWX6K772ULG2YAH5OVZ5JPKA, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.422842654 (556h44m34.293925699s) [noise=true , address=145.79.227.37:4566 , piece_id=AXTGBOXYJDN7QRFJWEMU3WBEF74KYERZS2B2SZXSPLZRU2TYC4JA, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.423548137 (556h44m34.294631192s) [noise=true , address=145.79.227.11:4730 , piece_id=6AXVYQY7IIQYL4OHBO3URZFJCC42ZWL3NEMD3JSFHLYMXNTQBMUQ, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.423555861 (556h44m34.294638906s) [noise=true , address=45.91.116.10:4081 , piece_id=GU7LAF6GLPTVIEOQ6GRHZI4VYYZMQUPKMAPOFBFCRQB344CUKKMQ, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.424230055 (556h44m34.29531309s) [noise=true , address=45.91.116.6:4976 , piece_id=4GCSWP27NMVW6VQFIOFBGBKHCZEB5VWC32CXMKKLVGXXFK3EHBXQ, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.424619267 (556h44m34.295702302s) [noise=true , address=145.79.227.36:4519 , piece_id=EJBDJEJAN7WG3HHTXBUEIW7QZ2VY6ZOXKNHBLHNYJ7L6ZVH6M7YA, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.424778910 (556h44m34.295861955s) [noise=true , address=77.90.31.11:4556 , piece_id=FANBXJICXHI33JVWYCLDDB76KSYXXFBTX4MLYXE6ICXQP2JAJAJA, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.425200092 (556h44m34.296283137s) [noise=true , address=77.90.31.18:4808 , piece_id=FIOQVOLXKF2CFNJ57BTCL4WTDPZPEEGAS2CDPF2PNTY27JVFTG6Q, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.425605254 (556h44m34.296688289s) [noise=true , address=77.90.31.17:4770 , piece_id=EA2QWTOVMLD5HKF6FGTB3TZXZZLSI4UVFTPD3MZZ2PAXF47VGJQA, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.426052345 (556h44m34.29713538s) [noise=true , address=145.79.227.51:4411 , piece_id=NHVE27MXBV6Y4VNYIT6G73S4WDAWVKACBQIOKTIOCGELPMLCY4CQ, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.426486622 (556h44m34.297569657s) [noise=true , address=45.91.116.13:4153 , piece_id=XCGSWRC2CZ2NLTPN5T5XTK4VHGIES67NB3FFCBXGI63DLGQ7V2EQ, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.426897945 (556h44m34.29798098s) [noise=true , address=45.91.116.4:4937 , piece_id=V6H4VACBOTZZMXQKCQRPJOBRFQVRF562HZC2GQGBLQGAGKFY5VNQ, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.427411794 (556h44m34.298494829s) [noise=true , address=145.79.227.45:4229 , piece_id=IRJAI3XSHMQYCOIFOMRNETTBXLLAEL3Z3YBOLPGEM2FOBF6S37XQ, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.427894844 (556h44m34.298977879s) [noise=true , address=145.79.227.21:4473 , piece_id=KAYPSFAGCFG3DUA4MVZAAHSH65MCNPDOH2M7GTAYILZO6EPFPXYQ, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.431922001 (556h44m34.303005036s) [noise=true , address=145.79.227.50:4374 , piece_id=DCLHZ4GR6U56WATYQRY7YFCFHO4JYMGLXSBY2OWEM3HSGPJHLDSA, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.436931210 (556h44m34.308014255s) [noise=true , address=145.79.227.38:4979 , piece_id=OUNZYJAAA32ZMIQOOCHK4YF4DCF354XUW4WUCHM3445666VXVLBQ, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:36.115225165 (556h44m35.98630821s) [upload=488098]: Next returned nil segment Dec 28 20:13:36.115243359 (556h44m35.986326395s) [upload=488098]: Waiting for error group managing segments... Dec 28 20:13:36.415982301 (556h44m36.287065346s) [noise=true , address=145.79.227.51:4411 , piece_id=NHVE27MXBV6Y4VNYIT6G73S4WDAWVKACBQIOKTIOCGELPMLCY4CQ, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.430495527 (556h44m36.301578562s) [noise=true , address=145.79.227.45:4229 , piece_id=IRJAI3XSHMQYCOIFOMRNETTBXLLAEL3Z3YBOLPGEM2FOBF6S37XQ, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.435192370 (556h44m36.306275406s) [noise=true , address=145.79.227.21:4473 , piece_id=KAYPSFAGCFG3DUA4MVZAAHSH65MCNPDOH2M7GTAYILZO6EPFPXYQ, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.435831557 (556h44m36.306914602s) [noise=true , address=145.79.227.38:4979 , piece_id=OUNZYJAAA32ZMIQOOCHK4YF4DCF354XUW4WUCHM3445666VXVLBQ, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.444124790 (556h44m36.315207825s) [noise=true , address=145.79.227.50:4374 , piece_id=DCLHZ4GR6U56WATYQRY7YFCFHO4JYMGLXSBY2OWEM3HSGPJHLDSA, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.450320487 (556h44m36.321403533s) [noise=true , address=145.79.227.11:4730 , piece_id=6AXVYQY7IIQYL4OHBO3URZFJCC42ZWL3NEMD3JSFHLYMXNTQBMUQ, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.452177964 (556h44m36.323260999s) [noise=true , address=145.79.227.36:4519 , piece_id=EJBDJEJAN7WG3HHTXBUEIW7QZ2VY6ZOXKNHBLHNYJ7L6ZVH6M7YA, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.462219385 (556h44m36.33330242s) [noise=true , address=145.79.227.37:4566 , piece_id=AXTGBOXYJDN7QRFJWEMU3WBEF74KYERZS2B2SZXSPLZRU2TYC4JA, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.485557775 (556h44m36.35664081s) [noise=true , address=145.79.227.16:4909 , piece_id=G7U4H2HJWK4VGY42RUBKYGDBLRDZLWX6K772ULG2YAH5OVZ5JPKA, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.583906940 (556h44m36.454989975s) [noise=true , address=77.90.31.11:4556 , piece_id=FANBXJICXHI33JVWYCLDDB76KSYXXFBTX4MLYXE6ICXQP2JAJAJA, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.658185680 (556h44m36.529268725s) [noise=true , address=45.91.116.4:4937 , piece_id=V6H4VACBOTZZMXQKCQRPJOBRFQVRF562HZC2GQGBLQGAGKFY5VNQ, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.715312460 (556h44m36.586395495s) [noise=true , address=45.91.116.13:4153 , piece_id=XCGSWRC2CZ2NLTPN5T5XTK4VHGIES67NB3FFCBXGI63DLGQ7V2EQ, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.737246156 (556h44m36.608329201s) [noise=true , address=45.91.116.10:4081 , piece_id=GU7LAF6GLPTVIEOQ6GRHZI4VYYZMQUPKMAPOFBFCRQB344CUKKMQ, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.737270231 (556h44m36.608353266s) [seg_pos={0 0}, upload=488098]: Segment reached optimal threshold of 13 pieces. Dec 28 20:13:36.737506872 (556h44m36.608589927s) [noise=true , address=77.90.31.18:4808 , piece_id=FIOQVOLXKF2CFNJ57BTCL4WTDPZPEEGAS2CDPF2PNTY27JVFTG6Q, seg_pos={0 0}, upload=488098]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: expected piece hash; context canceled; EOF Dec 28 20:13:36.737516099 (556h44m36.608599134s) [noise=true , address=77.90.31.18:4808 , piece_id=FIOQVOLXKF2CFNJ57BTCL4WTDPZPEEGAS2CDPF2PNTY27JVFTG6Q, seg_pos={0 0}, upload=488098]: Long tail cancellation due to optimal threshold Dec 28 20:13:36.737659642 (556h44m36.608742688s) [noise=true , address=45.91.116.6:4976 , piece_id=4GCSWP27NMVW6VQFIOFBGBKHCZEB5VWC32CXMKKLVGXXFK3EHBXQ, seg_pos={0 0}, upload=488098]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Dec 28 20:13:36.737707214 (556h44m36.608790259s) [noise=true , address=45.91.116.6:4976 , piece_id=4GCSWP27NMVW6VQFIOFBGBKHCZEB5VWC32CXMKKLVGXXFK3EHBXQ, seg_pos={0 0}, upload=488098]: Long tail cancellation due to optimal threshold Dec 28 20:13:36.737939255 (556h44m36.6090223s) [noise=true , address=77.90.31.17:4770 , piece_id=EA2QWTOVMLD5HKF6FGTB3TZXZZLSI4UVFTPD3MZZ2PAXF47VGJQA, seg_pos={0 0}, upload=488098]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: write tcp 188.166.225.170:53958->77.90.31.17:4770: use of closed network connection; write tcp 188.166.225.170:53958->77.90.31.17:4770: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53958->77.90.31.17:4770: use of closed network connection Dec 28 20:13:36.737965024 (556h44m36.609048079s) [noise=true , address=77.90.31.17:4770 , piece_id=EA2QWTOVMLD5HKF6FGTB3TZXZZLSI4UVFTPD3MZZ2PAXF47VGJQA, seg_pos={0 0}, upload=488098]: Long tail cancellation due to optimal threshold Dec 28 20:13:36.737999810 (556h44m36.609082846s) [seg_pos={0 0}, upload=488098]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 20:13:36.738101905 (556h44m36.60918495s) [upload=488098]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 20:13:36.738109249 (556h44m36.609192304s) [upload=488098]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 20:13:37.032410772 (556h44m36.903493817s) [upload=488098]: Done uploading segments... Dec 28 20:13:37.032430300 (556h44m36.903513345s) [upload=488098]: Upload finished. err: <nil> Duration: 17.269178766s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 15.406693828s trace: https://jaeger.dev.storj.io/trace/10e1641b228b2a1f FAIL (2025-12-28 20:14:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 40:05.744568479 (556h11m5.615651514s) [noise=true , address=145.79.227.28:4744 , piece_id=42LII6EOG3CMBTIL5I4RZQPDYU52TKJN3PGWEQXROBYMNDFYIIVA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.745099351 (556h11m5.616182396s) [noise=true , address=145.79.227.23:4550 , piece_id=NWIC722LZUHJSAMPWSKCLPRIDGWPVUEYMS5KRAMX3IIMD2KR4XGQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.745777401 (556h11m5.616860436s) [noise=true , address=77.90.31.15:4722 , piece_id=AF7F2OA247CJIDFTEU6HEOIYGOINEYPMII53BKJQXI7UZ26MJIUA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.746374739 (556h11m5.617457784s) [noise=true , address=77.90.31.10:4522 , piece_id=JOHOMHCNYXF4XQGLN2NJUSWSUO3XFBRRMYD27SLIZAK2RCT3ZC2A, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.746934555 (556h11m5.618017601s) [noise=true , address=77.90.31.21:4896 , piece_id=ATTC7BZUT4HAAEAZCHFZQE4INV2LLSWR2YCUNVA73LPJCW3YL35A, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.747556861 (556h11m5.618640146s) [noise=true , address=64.49.13.26:4956 , piece_id=FFHUPOOM7PLEABRHFH6H2SI6WXJBU2XFCLZJUBV4CXUJ6EP4PI2Q, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.748964150 (556h11m5.620047195s) [noise=true , address=145.79.227.34:4916 , piece_id=SPYSFWMJ752FC5JGRE36G54S2FHG4DQFCV6CIZONQHJDXDCN52VQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.751119314 (556h11m5.622202359s) [noise=true , address=77.90.31.18:4816 , piece_id=YT3PSYNQTDQBVBRUDNOXJ2XCN6EFQA54WBSNL6MZMXFXR7XNCHIA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.753413222 (556h11m5.624496257s) [noise=true , address=77.90.31.11:4582 , piece_id=ZRE24R27TX4E5DTV2POX7EJANQ4CM75D6ZOUS6SKM3QHHQCKRBRA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.755230482 (556h11m5.626313527s) [noise=true , address=64.49.13.154:4932 , piece_id=QB5WM5RZ2P2GC6RIGGM34IANVEI2SNDJ2PO556JXKTDBX5HLD4YA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.757598362 (556h11m5.628681407s) [noise=true , address=145.79.227.35:4510 , piece_id=P275TU367DTYXBQOGGV5THFUPHJUWGSM5IMJQ753VE4MX4IPOONQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.759644679 (556h11m5.630727734s) [noise=true , address=145.79.227.46:4626 , piece_id=OQOTWYLVIU7GJBBLYWQOEEFKR3TNAUOXZS6UAFJWJ32SIOTXNXEA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.761217594 (556h11m5.632300639s) [noise=true , address=145.79.227.43:4163 , piece_id=77BKLLVD25UMTDQZW46CA2N225I7O4HCHCRRPQ7HLJGRHOLGSDHA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.762138507 (556h11m5.633221552s) [noise=true , address=77.90.31.19:4860 , piece_id=5FUYVK2Z6LBZOAGFI2NOTAAUIV4NMBI5ERMA5RH4BMONNHABZLDQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.764304321 (556h11m5.635387356s) [noise=true , address=145.79.227.30:4784 , piece_id=MKCRSFWVFVGOAYCK5M66HO3OT7OMYWLVR2KMDXUGBS2VXXNCUSSQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.765421217 (556h11m5.636504252s) [noise=true , address=145.79.227.44:4216 , piece_id=GSBWC2LXUOZYOVD55KNQYA5Z3S2TCYROBB5O2XZIIVFUQMJQMYNA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:07.082099826 (556h11m6.953182871s) [upload=487609]: Next returned nil segment Dec 28 19:40:07.082116707 (556h11m6.953199752s) [upload=487609]: Waiting for error group managing segments... Dec 28 19:40:07.415884144 (556h11m7.286967179s) [noise=true , address=64.49.13.154:4932 , piece_id=QB5WM5RZ2P2GC6RIGGM34IANVEI2SNDJ2PO556JXKTDBX5HLD4YA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.615434252 (556h11m7.486517297s) [noise=true , address=145.79.227.23:4550 , piece_id=NWIC722LZUHJSAMPWSKCLPRIDGWPVUEYMS5KRAMX3IIMD2KR4XGQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.647182077 (556h11m7.518265112s) [noise=true , address=145.79.227.28:4744 , piece_id=42LII6EOG3CMBTIL5I4RZQPDYU52TKJN3PGWEQXROBYMNDFYIIVA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.678458624 (556h11m7.549541659s) [noise=true , address=145.79.227.46:4626 , piece_id=OQOTWYLVIU7GJBBLYWQOEEFKR3TNAUOXZS6UAFJWJ32SIOTXNXEA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.793406049 (556h11m7.664489084s) [noise=true , address=77.90.31.19:4860 , piece_id=5FUYVK2Z6LBZOAGFI2NOTAAUIV4NMBI5ERMA5RH4BMONNHABZLDQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.802506498 (556h11m7.673589543s) [noise=true , address=77.90.31.21:4896 , piece_id=ATTC7BZUT4HAAEAZCHFZQE4INV2LLSWR2YCUNVA73LPJCW3YL35A, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.853663242 (556h11m7.724746287s) [noise=true , address=77.90.31.10:4522 , piece_id=JOHOMHCNYXF4XQGLN2NJUSWSUO3XFBRRMYD27SLIZAK2RCT3ZC2A, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.880639500 (556h11m7.751722535s) [noise=true , address=77.90.31.11:4582 , piece_id=ZRE24R27TX4E5DTV2POX7EJANQ4CM75D6ZOUS6SKM3QHHQCKRBRA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.904835446 (556h11m7.775918491s) [noise=true , address=77.90.31.18:4816 , piece_id=YT3PSYNQTDQBVBRUDNOXJ2XCN6EFQA54WBSNL6MZMXFXR7XNCHIA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.988914260 (556h11m7.859997295s) [noise=true , address=77.90.31.15:4722 , piece_id=AF7F2OA247CJIDFTEU6HEOIYGOINEYPMII53BKJQXI7UZ26MJIUA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.484927449 (556h11m8.356010484s) [noise=true , address=145.79.227.35:4510 , piece_id=P275TU367DTYXBQOGGV5THFUPHJUWGSM5IMJQ753VE4MX4IPOONQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.610317849 (556h11m8.481400894s) [noise=true , address=145.79.227.44:4216 , piece_id=GSBWC2LXUOZYOVD55KNQYA5Z3S2TCYROBB5O2XZIIVFUQMJQMYNA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.611969885 (556h11m8.48305292s) [noise=true , address=145.79.227.34:4916 , piece_id=SPYSFWMJ752FC5JGRE36G54S2FHG4DQFCV6CIZONQHJDXDCN52VQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.636098801 (556h11m8.507181836s) [noise=true , address=145.79.227.30:4784 , piece_id=MKCRSFWVFVGOAYCK5M66HO3OT7OMYWLVR2KMDXUGBS2VXXNCUSSQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.636122136 (556h11m8.507205181s) [seg_pos={0 0}, upload=487609]: Segment reached optimal threshold of 14 pieces. Dec 28 19:40:08.636333929 (556h11m8.507416974s) [noise=true , address=64.49.13.26:4956 , piece_id=FFHUPOOM7PLEABRHFH6H2SI6WXJBU2XFCLZJUBV4CXUJ6EP4PI2Q, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TYHhjpfyBKkenSwviyQCUZpSBbnaFCGBFRkM4fyKLDuxKx4Ju): protocol: expected piece hash; context canceled; EOF Dec 28 19:40:08.636342175 (556h11m8.50742521s) [noise=true , address=64.49.13.26:4956 , piece_id=FFHUPOOM7PLEABRHFH6H2SI6WXJBU2XFCLZJUBV4CXUJ6EP4PI2Q, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636585959 (556h11m8.507668994s) [noise=true , address=145.79.227.43:4163 , piece_id=77BKLLVD25UMTDQZW46CA2N225I7O4HCHCRRPQ7HLJGRHOLGSDHA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CgPQ5hkrJdqbJvGarxKjXvYYnrXQqapL5BH2rNJNvnhpRi3pw): protocol: expected piece hash; context canceled; EOF Dec 28 19:40:08.636592172 (556h11m8.507675207s) [noise=true , address=145.79.227.43:4163 , piece_id=77BKLLVD25UMTDQZW46CA2N225I7O4HCHCRRPQ7HLJGRHOLGSDHA, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636681511 (556h11m8.507764546s) [noise=true , address=145.79.227.19:4083 , piece_id=GL5XL7T2QQABD5364AT4AFV6RZNOWCTVNSMAVIFJU5SUQFHOPZKQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KZ5a6CQ9tUTrC6EYE7M8yg6p9fK95CN8zQLQmP4tdyz4rBfGh): protocol: expected piece hash; context canceled; EOF Dec 28 19:40:08.636687052 (556h11m8.507770087s) [noise=true , address=145.79.227.19:4083 , piece_id=GL5XL7T2QQABD5364AT4AFV6RZNOWCTVNSMAVIFJU5SUQFHOPZKQ, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636810327 (556h11m8.507893362s) [noise=true , address=145.79.227.17:4992 , piece_id=GUF7AUFCFRRPY5PSYS7DRPFQLGT4ZEL5C7P23US2VI54KYWSUCUQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hbt8HKTunpPKyAzH4XBwGQqTKzcgqPpoWwKZm9JMtaK8P7aVp): protocol: write tcp 188.166.225.170:51210->145.79.227.17:4992: use of closed network connection; write tcp 188.166.225.170:51210->145.79.227.17:4992: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51210->145.79.227.17:4992: use of closed network connection Dec 28 19:40:08.636815957 (556h11m8.507899002s) [noise=true , address=145.79.227.17:4992 , piece_id=GUF7AUFCFRRPY5PSYS7DRPFQLGT4ZEL5C7P23US2VI54KYWSUCUQ, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636834313 (556h11m8.507917358s) [seg_pos={0 0}, upload=487609]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 19:40:08.636907953 (556h11m8.507990998s) [upload=487609]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 19:40:08.636913644 (556h11m8.507996689s) [upload=487609]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 19:40:08.924991397 (556h11m8.796074442s) [upload=487609]: Done uploading segments... Dec 28 19:40:08.925014742 (556h11m8.796097777s) [upload=487609]: Upload finished. err: <nil> Duration: 15.406726509s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [.XXX..X..X......XX..............X..........................X] duration: 30.0047615s trace: https://jaeger.dev.storj.io/trace/3057620e25d11143 FAIL (2025-12-28 20:14:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 20:13:34.173546860 (556h44m34.044629895s) [upload=488098]: Starting upload Dec 28 20:13:34.173605531 (556h44m34.044688566s) [upload=488098]: Done starting upload Dec 28 20:13:34.173822354 (556h44m34.04490539s) [upload=488098]: Uploading segments... Dec 28 20:13:34.173860978 (556h44m34.044944013s) [upload=488098]: Got next segment. Inline: false Dec 28 20:13:34.173892047 (556h44m34.044975082s) [upload=488098]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 20:13:34.173896415 (556h44m34.04497945s) [upload=488098]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 20:13:34.422688850 (556h44m34.293771905s) [seg_pos={0 0}, upload=488098]: Begin upload segment... Dec 28 20:13:34.422745879 (556h44m34.293828944s) [seg_pos={0 0}, upload=488098]: Done begin upload segment. Dec 28 20:13:34.422771076 (556h44m34.293854141s) [noise=true , address=145.79.227.16:4909 , piece_id=G7U4H2HJWK4VGY42RUBKYGDBLRDZLWX6K772ULG2YAH5OVZ5JPKA, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.422842654 (556h44m34.293925699s) [noise=true , address=145.79.227.37:4566 , piece_id=AXTGBOXYJDN7QRFJWEMU3WBEF74KYERZS2B2SZXSPLZRU2TYC4JA, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.423548137 (556h44m34.294631192s) [noise=true , address=145.79.227.11:4730 , piece_id=6AXVYQY7IIQYL4OHBO3URZFJCC42ZWL3NEMD3JSFHLYMXNTQBMUQ, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.423555861 (556h44m34.294638906s) [noise=true , address=45.91.116.10:4081 , piece_id=GU7LAF6GLPTVIEOQ6GRHZI4VYYZMQUPKMAPOFBFCRQB344CUKKMQ, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.424230055 (556h44m34.29531309s) [noise=true , address=45.91.116.6:4976 , piece_id=4GCSWP27NMVW6VQFIOFBGBKHCZEB5VWC32CXMKKLVGXXFK3EHBXQ, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.424619267 (556h44m34.295702302s) [noise=true , address=145.79.227.36:4519 , piece_id=EJBDJEJAN7WG3HHTXBUEIW7QZ2VY6ZOXKNHBLHNYJ7L6ZVH6M7YA, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.424778910 (556h44m34.295861955s) [noise=true , address=77.90.31.11:4556 , piece_id=FANBXJICXHI33JVWYCLDDB76KSYXXFBTX4MLYXE6ICXQP2JAJAJA, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.425200092 (556h44m34.296283137s) [noise=true , address=77.90.31.18:4808 , piece_id=FIOQVOLXKF2CFNJ57BTCL4WTDPZPEEGAS2CDPF2PNTY27JVFTG6Q, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.425605254 (556h44m34.296688289s) [noise=true , address=77.90.31.17:4770 , piece_id=EA2QWTOVMLD5HKF6FGTB3TZXZZLSI4UVFTPD3MZZ2PAXF47VGJQA, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.426052345 (556h44m34.29713538s) [noise=true , address=145.79.227.51:4411 , piece_id=NHVE27MXBV6Y4VNYIT6G73S4WDAWVKACBQIOKTIOCGELPMLCY4CQ, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.426486622 (556h44m34.297569657s) [noise=true , address=45.91.116.13:4153 , piece_id=XCGSWRC2CZ2NLTPN5T5XTK4VHGIES67NB3FFCBXGI63DLGQ7V2EQ, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.426897945 (556h44m34.29798098s) [noise=true , address=45.91.116.4:4937 , piece_id=V6H4VACBOTZZMXQKCQRPJOBRFQVRF562HZC2GQGBLQGAGKFY5VNQ, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.427411794 (556h44m34.298494829s) [noise=true , address=145.79.227.45:4229 , piece_id=IRJAI3XSHMQYCOIFOMRNETTBXLLAEL3Z3YBOLPGEM2FOBF6S37XQ, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.427894844 (556h44m34.298977879s) [noise=true , address=145.79.227.21:4473 , piece_id=KAYPSFAGCFG3DUA4MVZAAHSH65MCNPDOH2M7GTAYILZO6EPFPXYQ, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.431922001 (556h44m34.303005036s) [noise=true , address=145.79.227.50:4374 , piece_id=DCLHZ4GR6U56WATYQRY7YFCFHO4JYMGLXSBY2OWEM3HSGPJHLDSA, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:34.436931210 (556h44m34.308014255s) [noise=true , address=145.79.227.38:4979 , piece_id=OUNZYJAAA32ZMIQOOCHK4YF4DCF354XUW4WUCHM3445666VXVLBQ, seg_pos={0 0}, upload=488098]: Uploading piece... Dec 28 20:13:36.115225165 (556h44m35.98630821s) [upload=488098]: Next returned nil segment Dec 28 20:13:36.115243359 (556h44m35.986326395s) [upload=488098]: Waiting for error group managing segments... Dec 28 20:13:36.415982301 (556h44m36.287065346s) [noise=true , address=145.79.227.51:4411 , piece_id=NHVE27MXBV6Y4VNYIT6G73S4WDAWVKACBQIOKTIOCGELPMLCY4CQ, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.430495527 (556h44m36.301578562s) [noise=true , address=145.79.227.45:4229 , piece_id=IRJAI3XSHMQYCOIFOMRNETTBXLLAEL3Z3YBOLPGEM2FOBF6S37XQ, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.435192370 (556h44m36.306275406s) [noise=true , address=145.79.227.21:4473 , piece_id=KAYPSFAGCFG3DUA4MVZAAHSH65MCNPDOH2M7GTAYILZO6EPFPXYQ, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.435831557 (556h44m36.306914602s) [noise=true , address=145.79.227.38:4979 , piece_id=OUNZYJAAA32ZMIQOOCHK4YF4DCF354XUW4WUCHM3445666VXVLBQ, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.444124790 (556h44m36.315207825s) [noise=true , address=145.79.227.50:4374 , piece_id=DCLHZ4GR6U56WATYQRY7YFCFHO4JYMGLXSBY2OWEM3HSGPJHLDSA, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.450320487 (556h44m36.321403533s) [noise=true , address=145.79.227.11:4730 , piece_id=6AXVYQY7IIQYL4OHBO3URZFJCC42ZWL3NEMD3JSFHLYMXNTQBMUQ, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.452177964 (556h44m36.323260999s) [noise=true , address=145.79.227.36:4519 , piece_id=EJBDJEJAN7WG3HHTXBUEIW7QZ2VY6ZOXKNHBLHNYJ7L6ZVH6M7YA, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.462219385 (556h44m36.33330242s) [noise=true , address=145.79.227.37:4566 , piece_id=AXTGBOXYJDN7QRFJWEMU3WBEF74KYERZS2B2SZXSPLZRU2TYC4JA, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.485557775 (556h44m36.35664081s) [noise=true , address=145.79.227.16:4909 , piece_id=G7U4H2HJWK4VGY42RUBKYGDBLRDZLWX6K772ULG2YAH5OVZ5JPKA, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.583906940 (556h44m36.454989975s) [noise=true , address=77.90.31.11:4556 , piece_id=FANBXJICXHI33JVWYCLDDB76KSYXXFBTX4MLYXE6ICXQP2JAJAJA, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.658185680 (556h44m36.529268725s) [noise=true , address=45.91.116.4:4937 , piece_id=V6H4VACBOTZZMXQKCQRPJOBRFQVRF562HZC2GQGBLQGAGKFY5VNQ, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.715312460 (556h44m36.586395495s) [noise=true , address=45.91.116.13:4153 , piece_id=XCGSWRC2CZ2NLTPN5T5XTK4VHGIES67NB3FFCBXGI63DLGQ7V2EQ, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.737246156 (556h44m36.608329201s) [noise=true , address=45.91.116.10:4081 , piece_id=GU7LAF6GLPTVIEOQ6GRHZI4VYYZMQUPKMAPOFBFCRQB344CUKKMQ, seg_pos={0 0}, upload=488098]: Done uploading piece. err: <nil> Dec 28 20:13:36.737270231 (556h44m36.608353266s) [seg_pos={0 0}, upload=488098]: Segment reached optimal threshold of 13 pieces. Dec 28 20:13:36.737506872 (556h44m36.608589927s) [noise=true , address=77.90.31.18:4808 , piece_id=FIOQVOLXKF2CFNJ57BTCL4WTDPZPEEGAS2CDPF2PNTY27JVFTG6Q, seg_pos={0 0}, upload=488098]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: expected piece hash; context canceled; EOF Dec 28 20:13:36.737516099 (556h44m36.608599134s) [noise=true , address=77.90.31.18:4808 , piece_id=FIOQVOLXKF2CFNJ57BTCL4WTDPZPEEGAS2CDPF2PNTY27JVFTG6Q, seg_pos={0 0}, upload=488098]: Long tail cancellation due to optimal threshold Dec 28 20:13:36.737659642 (556h44m36.608742688s) [noise=true , address=45.91.116.6:4976 , piece_id=4GCSWP27NMVW6VQFIOFBGBKHCZEB5VWC32CXMKKLVGXXFK3EHBXQ, seg_pos={0 0}, upload=488098]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Dec 28 20:13:36.737707214 (556h44m36.608790259s) [noise=true , address=45.91.116.6:4976 , piece_id=4GCSWP27NMVW6VQFIOFBGBKHCZEB5VWC32CXMKKLVGXXFK3EHBXQ, seg_pos={0 0}, upload=488098]: Long tail cancellation due to optimal threshold Dec 28 20:13:36.737939255 (556h44m36.6090223s) [noise=true , address=77.90.31.17:4770 , piece_id=EA2QWTOVMLD5HKF6FGTB3TZXZZLSI4UVFTPD3MZZ2PAXF47VGJQA, seg_pos={0 0}, upload=488098]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: write tcp 188.166.225.170:53958->77.90.31.17:4770: use of closed network connection; write tcp 188.166.225.170:53958->77.90.31.17:4770: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53958->77.90.31.17:4770: use of closed network connection Dec 28 20:13:36.737965024 (556h44m36.609048079s) [noise=true , address=77.90.31.17:4770 , piece_id=EA2QWTOVMLD5HKF6FGTB3TZXZZLSI4UVFTPD3MZZ2PAXF47VGJQA, seg_pos={0 0}, upload=488098]: Long tail cancellation due to optimal threshold Dec 28 20:13:36.737999810 (556h44m36.609082846s) [seg_pos={0 0}, upload=488098]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 20:13:36.738101905 (556h44m36.60918495s) [upload=488098]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 20:13:36.738109249 (556h44m36.609192304s) [upload=488098]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 20:13:37.032410772 (556h44m36.903493817s) [upload=488098]: Done uploading segments... Dec 28 20:13:37.032430300 (556h44m36.903513345s) [upload=488098]: Upload finished. err: <nil> Duration: 30.004788441s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 15.343146853s trace: https://jaeger.dev.storj.io/trace/10e1641b228b2a1f FAIL (2025-12-28 20:13:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 40:05.744568479 (556h11m5.615651514s) [noise=true , address=145.79.227.28:4744 , piece_id=42LII6EOG3CMBTIL5I4RZQPDYU52TKJN3PGWEQXROBYMNDFYIIVA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.745099351 (556h11m5.616182396s) [noise=true , address=145.79.227.23:4550 , piece_id=NWIC722LZUHJSAMPWSKCLPRIDGWPVUEYMS5KRAMX3IIMD2KR4XGQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.745777401 (556h11m5.616860436s) [noise=true , address=77.90.31.15:4722 , piece_id=AF7F2OA247CJIDFTEU6HEOIYGOINEYPMII53BKJQXI7UZ26MJIUA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.746374739 (556h11m5.617457784s) [noise=true , address=77.90.31.10:4522 , piece_id=JOHOMHCNYXF4XQGLN2NJUSWSUO3XFBRRMYD27SLIZAK2RCT3ZC2A, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.746934555 (556h11m5.618017601s) [noise=true , address=77.90.31.21:4896 , piece_id=ATTC7BZUT4HAAEAZCHFZQE4INV2LLSWR2YCUNVA73LPJCW3YL35A, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.747556861 (556h11m5.618640146s) [noise=true , address=64.49.13.26:4956 , piece_id=FFHUPOOM7PLEABRHFH6H2SI6WXJBU2XFCLZJUBV4CXUJ6EP4PI2Q, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.748964150 (556h11m5.620047195s) [noise=true , address=145.79.227.34:4916 , piece_id=SPYSFWMJ752FC5JGRE36G54S2FHG4DQFCV6CIZONQHJDXDCN52VQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.751119314 (556h11m5.622202359s) [noise=true , address=77.90.31.18:4816 , piece_id=YT3PSYNQTDQBVBRUDNOXJ2XCN6EFQA54WBSNL6MZMXFXR7XNCHIA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.753413222 (556h11m5.624496257s) [noise=true , address=77.90.31.11:4582 , piece_id=ZRE24R27TX4E5DTV2POX7EJANQ4CM75D6ZOUS6SKM3QHHQCKRBRA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.755230482 (556h11m5.626313527s) [noise=true , address=64.49.13.154:4932 , piece_id=QB5WM5RZ2P2GC6RIGGM34IANVEI2SNDJ2PO556JXKTDBX5HLD4YA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.757598362 (556h11m5.628681407s) [noise=true , address=145.79.227.35:4510 , piece_id=P275TU367DTYXBQOGGV5THFUPHJUWGSM5IMJQ753VE4MX4IPOONQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.759644679 (556h11m5.630727734s) [noise=true , address=145.79.227.46:4626 , piece_id=OQOTWYLVIU7GJBBLYWQOEEFKR3TNAUOXZS6UAFJWJ32SIOTXNXEA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.761217594 (556h11m5.632300639s) [noise=true , address=145.79.227.43:4163 , piece_id=77BKLLVD25UMTDQZW46CA2N225I7O4HCHCRRPQ7HLJGRHOLGSDHA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.762138507 (556h11m5.633221552s) [noise=true , address=77.90.31.19:4860 , piece_id=5FUYVK2Z6LBZOAGFI2NOTAAUIV4NMBI5ERMA5RH4BMONNHABZLDQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.764304321 (556h11m5.635387356s) [noise=true , address=145.79.227.30:4784 , piece_id=MKCRSFWVFVGOAYCK5M66HO3OT7OMYWLVR2KMDXUGBS2VXXNCUSSQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.765421217 (556h11m5.636504252s) [noise=true , address=145.79.227.44:4216 , piece_id=GSBWC2LXUOZYOVD55KNQYA5Z3S2TCYROBB5O2XZIIVFUQMJQMYNA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:07.082099826 (556h11m6.953182871s) [upload=487609]: Next returned nil segment Dec 28 19:40:07.082116707 (556h11m6.953199752s) [upload=487609]: Waiting for error group managing segments... Dec 28 19:40:07.415884144 (556h11m7.286967179s) [noise=true , address=64.49.13.154:4932 , piece_id=QB5WM5RZ2P2GC6RIGGM34IANVEI2SNDJ2PO556JXKTDBX5HLD4YA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.615434252 (556h11m7.486517297s) [noise=true , address=145.79.227.23:4550 , piece_id=NWIC722LZUHJSAMPWSKCLPRIDGWPVUEYMS5KRAMX3IIMD2KR4XGQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.647182077 (556h11m7.518265112s) [noise=true , address=145.79.227.28:4744 , piece_id=42LII6EOG3CMBTIL5I4RZQPDYU52TKJN3PGWEQXROBYMNDFYIIVA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.678458624 (556h11m7.549541659s) [noise=true , address=145.79.227.46:4626 , piece_id=OQOTWYLVIU7GJBBLYWQOEEFKR3TNAUOXZS6UAFJWJ32SIOTXNXEA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.793406049 (556h11m7.664489084s) [noise=true , address=77.90.31.19:4860 , piece_id=5FUYVK2Z6LBZOAGFI2NOTAAUIV4NMBI5ERMA5RH4BMONNHABZLDQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.802506498 (556h11m7.673589543s) [noise=true , address=77.90.31.21:4896 , piece_id=ATTC7BZUT4HAAEAZCHFZQE4INV2LLSWR2YCUNVA73LPJCW3YL35A, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.853663242 (556h11m7.724746287s) [noise=true , address=77.90.31.10:4522 , piece_id=JOHOMHCNYXF4XQGLN2NJUSWSUO3XFBRRMYD27SLIZAK2RCT3ZC2A, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.880639500 (556h11m7.751722535s) [noise=true , address=77.90.31.11:4582 , piece_id=ZRE24R27TX4E5DTV2POX7EJANQ4CM75D6ZOUS6SKM3QHHQCKRBRA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.904835446 (556h11m7.775918491s) [noise=true , address=77.90.31.18:4816 , piece_id=YT3PSYNQTDQBVBRUDNOXJ2XCN6EFQA54WBSNL6MZMXFXR7XNCHIA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.988914260 (556h11m7.859997295s) [noise=true , address=77.90.31.15:4722 , piece_id=AF7F2OA247CJIDFTEU6HEOIYGOINEYPMII53BKJQXI7UZ26MJIUA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.484927449 (556h11m8.356010484s) [noise=true , address=145.79.227.35:4510 , piece_id=P275TU367DTYXBQOGGV5THFUPHJUWGSM5IMJQ753VE4MX4IPOONQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.610317849 (556h11m8.481400894s) [noise=true , address=145.79.227.44:4216 , piece_id=GSBWC2LXUOZYOVD55KNQYA5Z3S2TCYROBB5O2XZIIVFUQMJQMYNA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.611969885 (556h11m8.48305292s) [noise=true , address=145.79.227.34:4916 , piece_id=SPYSFWMJ752FC5JGRE36G54S2FHG4DQFCV6CIZONQHJDXDCN52VQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.636098801 (556h11m8.507181836s) [noise=true , address=145.79.227.30:4784 , piece_id=MKCRSFWVFVGOAYCK5M66HO3OT7OMYWLVR2KMDXUGBS2VXXNCUSSQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.636122136 (556h11m8.507205181s) [seg_pos={0 0}, upload=487609]: Segment reached optimal threshold of 14 pieces. Dec 28 19:40:08.636333929 (556h11m8.507416974s) [noise=true , address=64.49.13.26:4956 , piece_id=FFHUPOOM7PLEABRHFH6H2SI6WXJBU2XFCLZJUBV4CXUJ6EP4PI2Q, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TYHhjpfyBKkenSwviyQCUZpSBbnaFCGBFRkM4fyKLDuxKx4Ju): protocol: expected piece hash; context canceled; EOF Dec 28 19:40:08.636342175 (556h11m8.50742521s) [noise=true , address=64.49.13.26:4956 , piece_id=FFHUPOOM7PLEABRHFH6H2SI6WXJBU2XFCLZJUBV4CXUJ6EP4PI2Q, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636585959 (556h11m8.507668994s) [noise=true , address=145.79.227.43:4163 , piece_id=77BKLLVD25UMTDQZW46CA2N225I7O4HCHCRRPQ7HLJGRHOLGSDHA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CgPQ5hkrJdqbJvGarxKjXvYYnrXQqapL5BH2rNJNvnhpRi3pw): protocol: expected piece hash; context canceled; EOF Dec 28 19:40:08.636592172 (556h11m8.507675207s) [noise=true , address=145.79.227.43:4163 , piece_id=77BKLLVD25UMTDQZW46CA2N225I7O4HCHCRRPQ7HLJGRHOLGSDHA, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636681511 (556h11m8.507764546s) [noise=true , address=145.79.227.19:4083 , piece_id=GL5XL7T2QQABD5364AT4AFV6RZNOWCTVNSMAVIFJU5SUQFHOPZKQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KZ5a6CQ9tUTrC6EYE7M8yg6p9fK95CN8zQLQmP4tdyz4rBfGh): protocol: expected piece hash; context canceled; EOF Dec 28 19:40:08.636687052 (556h11m8.507770087s) [noise=true , address=145.79.227.19:4083 , piece_id=GL5XL7T2QQABD5364AT4AFV6RZNOWCTVNSMAVIFJU5SUQFHOPZKQ, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636810327 (556h11m8.507893362s) [noise=true , address=145.79.227.17:4992 , piece_id=GUF7AUFCFRRPY5PSYS7DRPFQLGT4ZEL5C7P23US2VI54KYWSUCUQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hbt8HKTunpPKyAzH4XBwGQqTKzcgqPpoWwKZm9JMtaK8P7aVp): protocol: write tcp 188.166.225.170:51210->145.79.227.17:4992: use of closed network connection; write tcp 188.166.225.170:51210->145.79.227.17:4992: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51210->145.79.227.17:4992: use of closed network connection Dec 28 19:40:08.636815957 (556h11m8.507899002s) [noise=true , address=145.79.227.17:4992 , piece_id=GUF7AUFCFRRPY5PSYS7DRPFQLGT4ZEL5C7P23US2VI54KYWSUCUQ, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636834313 (556h11m8.507917358s) [seg_pos={0 0}, upload=487609]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 19:40:08.636907953 (556h11m8.507990998s) [upload=487609]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 19:40:08.636913644 (556h11m8.507996689s) [upload=487609]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 19:40:08.924991397 (556h11m8.796074442s) [upload=487609]: Done uploading segments... Dec 28 19:40:08.925014742 (556h11m8.796097777s) [upload=487609]: Upload finished. err: <nil> Duration: 15.343181177s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 15.39253675s trace: https://jaeger.dev.storj.io/trace/10e1641b228b2a1f FAIL (2025-12-28 20:13:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 40:05.744568479 (556h11m5.615651514s) [noise=true , address=145.79.227.28:4744 , piece_id=42LII6EOG3CMBTIL5I4RZQPDYU52TKJN3PGWEQXROBYMNDFYIIVA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.745099351 (556h11m5.616182396s) [noise=true , address=145.79.227.23:4550 , piece_id=NWIC722LZUHJSAMPWSKCLPRIDGWPVUEYMS5KRAMX3IIMD2KR4XGQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.745777401 (556h11m5.616860436s) [noise=true , address=77.90.31.15:4722 , piece_id=AF7F2OA247CJIDFTEU6HEOIYGOINEYPMII53BKJQXI7UZ26MJIUA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.746374739 (556h11m5.617457784s) [noise=true , address=77.90.31.10:4522 , piece_id=JOHOMHCNYXF4XQGLN2NJUSWSUO3XFBRRMYD27SLIZAK2RCT3ZC2A, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.746934555 (556h11m5.618017601s) [noise=true , address=77.90.31.21:4896 , piece_id=ATTC7BZUT4HAAEAZCHFZQE4INV2LLSWR2YCUNVA73LPJCW3YL35A, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.747556861 (556h11m5.618640146s) [noise=true , address=64.49.13.26:4956 , piece_id=FFHUPOOM7PLEABRHFH6H2SI6WXJBU2XFCLZJUBV4CXUJ6EP4PI2Q, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.748964150 (556h11m5.620047195s) [noise=true , address=145.79.227.34:4916 , piece_id=SPYSFWMJ752FC5JGRE36G54S2FHG4DQFCV6CIZONQHJDXDCN52VQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.751119314 (556h11m5.622202359s) [noise=true , address=77.90.31.18:4816 , piece_id=YT3PSYNQTDQBVBRUDNOXJ2XCN6EFQA54WBSNL6MZMXFXR7XNCHIA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.753413222 (556h11m5.624496257s) [noise=true , address=77.90.31.11:4582 , piece_id=ZRE24R27TX4E5DTV2POX7EJANQ4CM75D6ZOUS6SKM3QHHQCKRBRA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.755230482 (556h11m5.626313527s) [noise=true , address=64.49.13.154:4932 , piece_id=QB5WM5RZ2P2GC6RIGGM34IANVEI2SNDJ2PO556JXKTDBX5HLD4YA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.757598362 (556h11m5.628681407s) [noise=true , address=145.79.227.35:4510 , piece_id=P275TU367DTYXBQOGGV5THFUPHJUWGSM5IMJQ753VE4MX4IPOONQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.759644679 (556h11m5.630727734s) [noise=true , address=145.79.227.46:4626 , piece_id=OQOTWYLVIU7GJBBLYWQOEEFKR3TNAUOXZS6UAFJWJ32SIOTXNXEA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.761217594 (556h11m5.632300639s) [noise=true , address=145.79.227.43:4163 , piece_id=77BKLLVD25UMTDQZW46CA2N225I7O4HCHCRRPQ7HLJGRHOLGSDHA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.762138507 (556h11m5.633221552s) [noise=true , address=77.90.31.19:4860 , piece_id=5FUYVK2Z6LBZOAGFI2NOTAAUIV4NMBI5ERMA5RH4BMONNHABZLDQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.764304321 (556h11m5.635387356s) [noise=true , address=145.79.227.30:4784 , piece_id=MKCRSFWVFVGOAYCK5M66HO3OT7OMYWLVR2KMDXUGBS2VXXNCUSSQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.765421217 (556h11m5.636504252s) [noise=true , address=145.79.227.44:4216 , piece_id=GSBWC2LXUOZYOVD55KNQYA5Z3S2TCYROBB5O2XZIIVFUQMJQMYNA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:07.082099826 (556h11m6.953182871s) [upload=487609]: Next returned nil segment Dec 28 19:40:07.082116707 (556h11m6.953199752s) [upload=487609]: Waiting for error group managing segments... Dec 28 19:40:07.415884144 (556h11m7.286967179s) [noise=true , address=64.49.13.154:4932 , piece_id=QB5WM5RZ2P2GC6RIGGM34IANVEI2SNDJ2PO556JXKTDBX5HLD4YA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.615434252 (556h11m7.486517297s) [noise=true , address=145.79.227.23:4550 , piece_id=NWIC722LZUHJSAMPWSKCLPRIDGWPVUEYMS5KRAMX3IIMD2KR4XGQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.647182077 (556h11m7.518265112s) [noise=true , address=145.79.227.28:4744 , piece_id=42LII6EOG3CMBTIL5I4RZQPDYU52TKJN3PGWEQXROBYMNDFYIIVA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.678458624 (556h11m7.549541659s) [noise=true , address=145.79.227.46:4626 , piece_id=OQOTWYLVIU7GJBBLYWQOEEFKR3TNAUOXZS6UAFJWJ32SIOTXNXEA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.793406049 (556h11m7.664489084s) [noise=true , address=77.90.31.19:4860 , piece_id=5FUYVK2Z6LBZOAGFI2NOTAAUIV4NMBI5ERMA5RH4BMONNHABZLDQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.802506498 (556h11m7.673589543s) [noise=true , address=77.90.31.21:4896 , piece_id=ATTC7BZUT4HAAEAZCHFZQE4INV2LLSWR2YCUNVA73LPJCW3YL35A, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.853663242 (556h11m7.724746287s) [noise=true , address=77.90.31.10:4522 , piece_id=JOHOMHCNYXF4XQGLN2NJUSWSUO3XFBRRMYD27SLIZAK2RCT3ZC2A, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.880639500 (556h11m7.751722535s) [noise=true , address=77.90.31.11:4582 , piece_id=ZRE24R27TX4E5DTV2POX7EJANQ4CM75D6ZOUS6SKM3QHHQCKRBRA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.904835446 (556h11m7.775918491s) [noise=true , address=77.90.31.18:4816 , piece_id=YT3PSYNQTDQBVBRUDNOXJ2XCN6EFQA54WBSNL6MZMXFXR7XNCHIA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.988914260 (556h11m7.859997295s) [noise=true , address=77.90.31.15:4722 , piece_id=AF7F2OA247CJIDFTEU6HEOIYGOINEYPMII53BKJQXI7UZ26MJIUA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.484927449 (556h11m8.356010484s) [noise=true , address=145.79.227.35:4510 , piece_id=P275TU367DTYXBQOGGV5THFUPHJUWGSM5IMJQ753VE4MX4IPOONQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.610317849 (556h11m8.481400894s) [noise=true , address=145.79.227.44:4216 , piece_id=GSBWC2LXUOZYOVD55KNQYA5Z3S2TCYROBB5O2XZIIVFUQMJQMYNA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.611969885 (556h11m8.48305292s) [noise=true , address=145.79.227.34:4916 , piece_id=SPYSFWMJ752FC5JGRE36G54S2FHG4DQFCV6CIZONQHJDXDCN52VQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.636098801 (556h11m8.507181836s) [noise=true , address=145.79.227.30:4784 , piece_id=MKCRSFWVFVGOAYCK5M66HO3OT7OMYWLVR2KMDXUGBS2VXXNCUSSQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.636122136 (556h11m8.507205181s) [seg_pos={0 0}, upload=487609]: Segment reached optimal threshold of 14 pieces. Dec 28 19:40:08.636333929 (556h11m8.507416974s) [noise=true , address=64.49.13.26:4956 , piece_id=FFHUPOOM7PLEABRHFH6H2SI6WXJBU2XFCLZJUBV4CXUJ6EP4PI2Q, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TYHhjpfyBKkenSwviyQCUZpSBbnaFCGBFRkM4fyKLDuxKx4Ju): protocol: expected piece hash; context canceled; EOF Dec 28 19:40:08.636342175 (556h11m8.50742521s) [noise=true , address=64.49.13.26:4956 , piece_id=FFHUPOOM7PLEABRHFH6H2SI6WXJBU2XFCLZJUBV4CXUJ6EP4PI2Q, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636585959 (556h11m8.507668994s) [noise=true , address=145.79.227.43:4163 , piece_id=77BKLLVD25UMTDQZW46CA2N225I7O4HCHCRRPQ7HLJGRHOLGSDHA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CgPQ5hkrJdqbJvGarxKjXvYYnrXQqapL5BH2rNJNvnhpRi3pw): protocol: expected piece hash; context canceled; EOF Dec 28 19:40:08.636592172 (556h11m8.507675207s) [noise=true , address=145.79.227.43:4163 , piece_id=77BKLLVD25UMTDQZW46CA2N225I7O4HCHCRRPQ7HLJGRHOLGSDHA, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636681511 (556h11m8.507764546s) [noise=true , address=145.79.227.19:4083 , piece_id=GL5XL7T2QQABD5364AT4AFV6RZNOWCTVNSMAVIFJU5SUQFHOPZKQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KZ5a6CQ9tUTrC6EYE7M8yg6p9fK95CN8zQLQmP4tdyz4rBfGh): protocol: expected piece hash; context canceled; EOF Dec 28 19:40:08.636687052 (556h11m8.507770087s) [noise=true , address=145.79.227.19:4083 , piece_id=GL5XL7T2QQABD5364AT4AFV6RZNOWCTVNSMAVIFJU5SUQFHOPZKQ, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636810327 (556h11m8.507893362s) [noise=true , address=145.79.227.17:4992 , piece_id=GUF7AUFCFRRPY5PSYS7DRPFQLGT4ZEL5C7P23US2VI54KYWSUCUQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hbt8HKTunpPKyAzH4XBwGQqTKzcgqPpoWwKZm9JMtaK8P7aVp): protocol: write tcp 188.166.225.170:51210->145.79.227.17:4992: use of closed network connection; write tcp 188.166.225.170:51210->145.79.227.17:4992: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51210->145.79.227.17:4992: use of closed network connection Dec 28 19:40:08.636815957 (556h11m8.507899002s) [noise=true , address=145.79.227.17:4992 , piece_id=GUF7AUFCFRRPY5PSYS7DRPFQLGT4ZEL5C7P23US2VI54KYWSUCUQ, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636834313 (556h11m8.507917358s) [seg_pos={0 0}, upload=487609]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 19:40:08.636907953 (556h11m8.507990998s) [upload=487609]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 19:40:08.636913644 (556h11m8.507996689s) [upload=487609]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 19:40:08.924991397 (556h11m8.796074442s) [upload=487609]: Done uploading segments... Dec 28 19:40:08.925014742 (556h11m8.796097777s) [upload=487609]: Upload finished. err: <nil> Duration: 15.392561499s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [..XXX..X..X......XX..............X..........................] duration: 6.021913692s trace: https://jaeger.dev.storj.io/trace/6de23ce395480be2 FAIL (2025-12-28 20:12:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: VL3CQZWQBCPGDPMEAGIQCCHVRPMV36BXHPHJ2LPJLIC6LDGWR5YA) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: NYOXGWDJWHYHW6BY3DQHFGISYSYQV4JT2H2XDJ4WLMUNE2IHN6LA) context deadline exceeded; piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: BQE24O2CURUFHQS4WSIG722BGBPCPOY32W6KAOECGV7JPCKWGOKA) context deadline exceeded ======LOGS====== Dec 28 19:51:31.138959699 (556h22m31.010042744s) [upload=487778]: Starting upload Dec 28 19:51:31.139044411 (556h22m31.010127456s) [upload=487778]: Done starting upload Dec 28 19:51:31.139127078 (556h22m31.010210113s) [upload=487778]: Uploading segments... Dec 28 19:51:31.139326739 (556h22m31.010409784s) [upload=487778]: Got next segment. Inline: false Dec 28 19:51:31.139377555 (556h22m31.0104606s) [upload=487778]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 19:51:31.139384197 (556h22m31.010467232s) [upload=487778]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 19:51:31.387475461 (556h22m31.258558496s) [seg_pos={0 0}, upload=487778]: Begin upload segment... Dec 28 19:51:31.387544984 (556h22m31.258628049s) [seg_pos={0 0}, upload=487778]: Done begin upload segment. Dec 28 19:51:31.387771626 (556h22m31.258854671s) [noise=true , address=45.91.116.7:4011 , piece_id=NQDPXLIESHRAWWBPVYOGPARTQ6PDN3TO7YCWTLXI4KBFD7JZFXXQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.388814712 (556h22m31.259897747s) [noise=true , address=77.90.31.16:4735 , piece_id=EBCUZV26NTWYQFRJEZQE3N5EU2FPNPTQRCEPR6FNK5YTRZRORP2Q, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.389204404 (556h22m31.260287449s) [noise=true , address=145.79.227.9:4366 , piece_id=PC3TE34FAHG6J6TWTEU7GKWFGP4DDUELNIT2RIAEKGTM6XDRWIZA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.389706850 (556h22m31.260789885s) [noise=true , address=145.79.227.21:4475 , piece_id=BBVRNDFL25BI3Q2CNE36E5GRAOXT55T4MQA2HTXX257HPKZ6HSPQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.390109167 (556h22m31.261192212s) [noise=true , address=145.79.227.51:4411 , piece_id=LICH4YP46TD6FZHHN4X4OFWKYBPPCFWSGHU63E2WP4AMD6HFZDNA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.390421501 (556h22m31.261504546s) [noise=true , address=77.90.31.10:4519 , piece_id=VL3CQZWQBCPGDPMEAGIQCCHVRPMV36BXHPHJ2LPJLIC6LDGWR5YA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.390852582 (556h22m31.261935617s) [noise=true , address=145.79.227.5:4221 , piece_id=O73W2GAUHPHLTWZTO7IJT4OBBXAX22MAABQ2MQSTZPNLPIQXHKRQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.392756678 (556h22m31.263839713s) [noise=true , address=45.91.116.6:4974 , piece_id=FSPEWZOEC4SHIHA6IJSXCHUCOL6GPFRDFUFXJPCQR7263Y2YFIMQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.393255988 (556h22m31.264339023s) [noise=true , address=145.79.227.2:4113 , piece_id=QJTHNBJ6SMI7GHV5634NHXQUD2P4TXERE77FO4ECEVYHQ6GUQ3FQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.393633607 (556h22m31.264716642s) [noise=true , address=145.79.227.41:4087 , piece_id=SILOBZTKKG4QOW6F2XRCEXG64GCTMC2ULONWTO2D3QFFL74FP5BQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.394230243 (556h22m31.265313278s) [noise=true , address=145.79.227.47:4265 , piece_id=43D7BZZXISPRFCKYXSTCZONLHLQ2Y6IRLFD7PDEUAJWYP6ID7DMQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.394697414 (556h22m31.265780449s) [noise=true , address=145.79.227.39:4013 , piece_id=PWYMH2SD6JOHCSM7HFDM4CHUD5C3MUCSCRML3BMZKHDZAFRM6UMA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.401284737 (556h22m31.272367792s) [noise=true , address=145.79.227.45:4229 , piece_id=AZOSHGLV2UUTXSX5NX27R3ZFIEOWS2CDGLSG34JZKCVWY4DLHUHQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.402581947 (556h22m31.273664982s) [noise=true , address=77.90.31.14:4661 , piece_id=XLUH6WZPNM2W67MLQ7KFKGM3Y4WL4TFJ6GSYKPX325FSDPKZCUUA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.403710777 (556h22m31.274793822s) [noise=true , address=45.91.116.5:4300 , piece_id=NYOXGWDJWHYHW6BY3DQHFGISYSYQV4JT2H2XDJ4WLMUNE2IHN6LA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.404110447 (556h22m31.275193492s) [noise=true , address=45.91.116.9:4261 , piece_id=BQE24O2CURUFHQS4WSIG722BGBPCPOY32W6KAOECGV7JPCKWGOKA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:33.086369496 (556h22m32.957452531s) [upload=487778]: Next returned nil segment Dec 28 19:51:33.086389915 (556h22m32.95747296s) [upload=487778]: Waiting for error group managing segments... Dec 28 19:51:33.394411997 (556h22m33.265495032s) [noise=true , address=145.79.227.21:4475 , piece_id=BBVRNDFL25BI3Q2CNE36E5GRAOXT55T4MQA2HTXX257HPKZ6HSPQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.398162920 (556h22m33.269245955s) [noise=true , address=145.79.227.39:4013 , piece_id=PWYMH2SD6JOHCSM7HFDM4CHUD5C3MUCSCRML3BMZKHDZAFRM6UMA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.403651411 (556h22m33.274734446s) [noise=true , address=145.79.227.45:4229 , piece_id=AZOSHGLV2UUTXSX5NX27R3ZFIEOWS2CDGLSG34JZKCVWY4DLHUHQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.406027926 (556h22m33.277110971s) [noise=true , address=145.79.227.51:4411 , piece_id=LICH4YP46TD6FZHHN4X4OFWKYBPPCFWSGHU63E2WP4AMD6HFZDNA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.418392551 (556h22m33.289475586s) [noise=true , address=145.79.227.47:4265 , piece_id=43D7BZZXISPRFCKYXSTCZONLHLQ2Y6IRLFD7PDEUAJWYP6ID7DMQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.420875018 (556h22m33.291958063s) [noise=true , address=145.79.227.41:4087 , piece_id=SILOBZTKKG4QOW6F2XRCEXG64GCTMC2ULONWTO2D3QFFL74FP5BQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.421544332 (556h22m33.292627377s) [noise=true , address=145.79.227.2:4113 , piece_id=QJTHNBJ6SMI7GHV5634NHXQUD2P4TXERE77FO4ECEVYHQ6GUQ3FQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.427849308 (556h22m33.298932353s) [noise=true , address=145.79.227.9:4366 , piece_id=PC3TE34FAHG6J6TWTEU7GKWFGP4DDUELNIT2RIAEKGTM6XDRWIZA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.649006914 (556h22m33.520089949s) [noise=true , address=45.91.116.9:4261 , piece_id=BQE24O2CURUFHQS4WSIG722BGBPCPOY32W6KAOECGV7JPCKWGOKA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.688630355 (556h22m33.55971339s) [noise=true , address=77.90.31.14:4661 , piece_id=XLUH6WZPNM2W67MLQ7KFKGM3Y4WL4TFJ6GSYKPX325FSDPKZCUUA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.694890445 (556h22m33.56597348s) [noise=true , address=45.91.116.5:4300 , piece_id=NYOXGWDJWHYHW6BY3DQHFGISYSYQV4JT2H2XDJ4WLMUNE2IHN6LA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.738432388 (556h22m33.609515433s) [noise=true , address=45.91.116.7:4011 , piece_id=NQDPXLIESHRAWWBPVYOGPARTQ6PDN3TO7YCWTLXI4KBFD7JZFXXQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:34.314051275 (556h22m34.18513431s) [noise=true , address=77.90.31.10:4519 , piece_id=VL3CQZWQBCPGDPMEAGIQCCHVRPMV36BXHPHJ2LPJLIC6LDGWR5YA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:34.314075711 (556h22m34.185158746s) [seg_pos={0 0}, upload=487778]: Segment reached optimal threshold of 13 pieces. Dec 28 19:51:34.314354963 (556h22m34.185437998s) [noise=true , address=45.91.116.6:4974 , piece_id=FSPEWZOEC4SHIHA6IJSXCHUCOL6GPFRDFUFXJPCQR7263Y2YFIMQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Dec 28 19:51:34.314365733 (556h22m34.185448778s) [noise=true , address=45.91.116.6:4974 , piece_id=FSPEWZOEC4SHIHA6IJSXCHUCOL6GPFRDFUFXJPCQR7263Y2YFIMQ, seg_pos={0 0}, upload=487778]: Long tail cancellation due to optimal threshold Dec 28 19:51:34.314429585 (556h22m34.18551263s) [noise=true , address=145.79.227.5:4221 , piece_id=O73W2GAUHPHLTWZTO7IJT4OBBXAX22MAABQ2MQSTZPNLPIQXHKRQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR): protocol: expected piece hash; context canceled; EOF Dec 28 19:51:34.314451267 (556h22m34.185534302s) [noise=true , address=145.79.227.5:4221 , piece_id=O73W2GAUHPHLTWZTO7IJT4OBBXAX22MAABQ2MQSTZPNLPIQXHKRQ, seg_pos={0 0}, upload=487778]: Long tail cancellation due to optimal threshold Dec 28 19:51:34.314521831 (556h22m34.185604866s) [noise=true , address=77.90.31.16:4735 , piece_id=EBCUZV26NTWYQFRJEZQE3N5EU2FPNPTQRCEPR6FNK5YTRZRORP2Q, seg_pos={0 0}, upload=487778]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 28 19:51:34.314530057 (556h22m34.185613092s) [noise=true , address=77.90.31.16:4735 , piece_id=EBCUZV26NTWYQFRJEZQE3N5EU2FPNPTQRCEPR6FNK5YTRZRORP2Q, seg_pos={0 0}, upload=487778]: Long tail cancellation due to optimal threshold Dec 28 19:51:34.314593758 (556h22m34.185676793s) [seg_pos={0 0}, upload=487778]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 19:51:34.314673079 (556h22m34.185756124s) [upload=487778]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 19:51:34.314678910 (556h22m34.185761965s) [upload=487778]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 19:51:34.609917501 (556h22m34.481000566s) [upload=487778]: Done uploading segments... Dec 28 19:51:34.609974519 (556h22m34.481057594s) [upload=487778]: Upload finished. err: <nil> Duration: 6.021940924s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 43.538975384s trace: https://jaeger.dev.storj.io/trace/10e1641b228b2a1f FAIL (2025-12-28 20:12:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 40:05.744568479 (556h11m5.615651514s) [noise=true , address=145.79.227.28:4744 , piece_id=42LII6EOG3CMBTIL5I4RZQPDYU52TKJN3PGWEQXROBYMNDFYIIVA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.745099351 (556h11m5.616182396s) [noise=true , address=145.79.227.23:4550 , piece_id=NWIC722LZUHJSAMPWSKCLPRIDGWPVUEYMS5KRAMX3IIMD2KR4XGQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.745777401 (556h11m5.616860436s) [noise=true , address=77.90.31.15:4722 , piece_id=AF7F2OA247CJIDFTEU6HEOIYGOINEYPMII53BKJQXI7UZ26MJIUA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.746374739 (556h11m5.617457784s) [noise=true , address=77.90.31.10:4522 , piece_id=JOHOMHCNYXF4XQGLN2NJUSWSUO3XFBRRMYD27SLIZAK2RCT3ZC2A, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.746934555 (556h11m5.618017601s) [noise=true , address=77.90.31.21:4896 , piece_id=ATTC7BZUT4HAAEAZCHFZQE4INV2LLSWR2YCUNVA73LPJCW3YL35A, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.747556861 (556h11m5.618640146s) [noise=true , address=64.49.13.26:4956 , piece_id=FFHUPOOM7PLEABRHFH6H2SI6WXJBU2XFCLZJUBV4CXUJ6EP4PI2Q, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.748964150 (556h11m5.620047195s) [noise=true , address=145.79.227.34:4916 , piece_id=SPYSFWMJ752FC5JGRE36G54S2FHG4DQFCV6CIZONQHJDXDCN52VQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.751119314 (556h11m5.622202359s) [noise=true , address=77.90.31.18:4816 , piece_id=YT3PSYNQTDQBVBRUDNOXJ2XCN6EFQA54WBSNL6MZMXFXR7XNCHIA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.753413222 (556h11m5.624496257s) [noise=true , address=77.90.31.11:4582 , piece_id=ZRE24R27TX4E5DTV2POX7EJANQ4CM75D6ZOUS6SKM3QHHQCKRBRA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.755230482 (556h11m5.626313527s) [noise=true , address=64.49.13.154:4932 , piece_id=QB5WM5RZ2P2GC6RIGGM34IANVEI2SNDJ2PO556JXKTDBX5HLD4YA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.757598362 (556h11m5.628681407s) [noise=true , address=145.79.227.35:4510 , piece_id=P275TU367DTYXBQOGGV5THFUPHJUWGSM5IMJQ753VE4MX4IPOONQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.759644679 (556h11m5.630727734s) [noise=true , address=145.79.227.46:4626 , piece_id=OQOTWYLVIU7GJBBLYWQOEEFKR3TNAUOXZS6UAFJWJ32SIOTXNXEA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.761217594 (556h11m5.632300639s) [noise=true , address=145.79.227.43:4163 , piece_id=77BKLLVD25UMTDQZW46CA2N225I7O4HCHCRRPQ7HLJGRHOLGSDHA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.762138507 (556h11m5.633221552s) [noise=true , address=77.90.31.19:4860 , piece_id=5FUYVK2Z6LBZOAGFI2NOTAAUIV4NMBI5ERMA5RH4BMONNHABZLDQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.764304321 (556h11m5.635387356s) [noise=true , address=145.79.227.30:4784 , piece_id=MKCRSFWVFVGOAYCK5M66HO3OT7OMYWLVR2KMDXUGBS2VXXNCUSSQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.765421217 (556h11m5.636504252s) [noise=true , address=145.79.227.44:4216 , piece_id=GSBWC2LXUOZYOVD55KNQYA5Z3S2TCYROBB5O2XZIIVFUQMJQMYNA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:07.082099826 (556h11m6.953182871s) [upload=487609]: Next returned nil segment Dec 28 19:40:07.082116707 (556h11m6.953199752s) [upload=487609]: Waiting for error group managing segments... Dec 28 19:40:07.415884144 (556h11m7.286967179s) [noise=true , address=64.49.13.154:4932 , piece_id=QB5WM5RZ2P2GC6RIGGM34IANVEI2SNDJ2PO556JXKTDBX5HLD4YA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.615434252 (556h11m7.486517297s) [noise=true , address=145.79.227.23:4550 , piece_id=NWIC722LZUHJSAMPWSKCLPRIDGWPVUEYMS5KRAMX3IIMD2KR4XGQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.647182077 (556h11m7.518265112s) [noise=true , address=145.79.227.28:4744 , piece_id=42LII6EOG3CMBTIL5I4RZQPDYU52TKJN3PGWEQXROBYMNDFYIIVA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.678458624 (556h11m7.549541659s) [noise=true , address=145.79.227.46:4626 , piece_id=OQOTWYLVIU7GJBBLYWQOEEFKR3TNAUOXZS6UAFJWJ32SIOTXNXEA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.793406049 (556h11m7.664489084s) [noise=true , address=77.90.31.19:4860 , piece_id=5FUYVK2Z6LBZOAGFI2NOTAAUIV4NMBI5ERMA5RH4BMONNHABZLDQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.802506498 (556h11m7.673589543s) [noise=true , address=77.90.31.21:4896 , piece_id=ATTC7BZUT4HAAEAZCHFZQE4INV2LLSWR2YCUNVA73LPJCW3YL35A, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.853663242 (556h11m7.724746287s) [noise=true , address=77.90.31.10:4522 , piece_id=JOHOMHCNYXF4XQGLN2NJUSWSUO3XFBRRMYD27SLIZAK2RCT3ZC2A, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.880639500 (556h11m7.751722535s) [noise=true , address=77.90.31.11:4582 , piece_id=ZRE24R27TX4E5DTV2POX7EJANQ4CM75D6ZOUS6SKM3QHHQCKRBRA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.904835446 (556h11m7.775918491s) [noise=true , address=77.90.31.18:4816 , piece_id=YT3PSYNQTDQBVBRUDNOXJ2XCN6EFQA54WBSNL6MZMXFXR7XNCHIA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.988914260 (556h11m7.859997295s) [noise=true , address=77.90.31.15:4722 , piece_id=AF7F2OA247CJIDFTEU6HEOIYGOINEYPMII53BKJQXI7UZ26MJIUA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.484927449 (556h11m8.356010484s) [noise=true , address=145.79.227.35:4510 , piece_id=P275TU367DTYXBQOGGV5THFUPHJUWGSM5IMJQ753VE4MX4IPOONQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.610317849 (556h11m8.481400894s) [noise=true , address=145.79.227.44:4216 , piece_id=GSBWC2LXUOZYOVD55KNQYA5Z3S2TCYROBB5O2XZIIVFUQMJQMYNA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.611969885 (556h11m8.48305292s) [noise=true , address=145.79.227.34:4916 , piece_id=SPYSFWMJ752FC5JGRE36G54S2FHG4DQFCV6CIZONQHJDXDCN52VQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.636098801 (556h11m8.507181836s) [noise=true , address=145.79.227.30:4784 , piece_id=MKCRSFWVFVGOAYCK5M66HO3OT7OMYWLVR2KMDXUGBS2VXXNCUSSQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.636122136 (556h11m8.507205181s) [seg_pos={0 0}, upload=487609]: Segment reached optimal threshold of 14 pieces. Dec 28 19:40:08.636333929 (556h11m8.507416974s) [noise=true , address=64.49.13.26:4956 , piece_id=FFHUPOOM7PLEABRHFH6H2SI6WXJBU2XFCLZJUBV4CXUJ6EP4PI2Q, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TYHhjpfyBKkenSwviyQCUZpSBbnaFCGBFRkM4fyKLDuxKx4Ju): protocol: expected piece hash; context canceled; EOF Dec 28 19:40:08.636342175 (556h11m8.50742521s) [noise=true , address=64.49.13.26:4956 , piece_id=FFHUPOOM7PLEABRHFH6H2SI6WXJBU2XFCLZJUBV4CXUJ6EP4PI2Q, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636585959 (556h11m8.507668994s) [noise=true , address=145.79.227.43:4163 , piece_id=77BKLLVD25UMTDQZW46CA2N225I7O4HCHCRRPQ7HLJGRHOLGSDHA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CgPQ5hkrJdqbJvGarxKjXvYYnrXQqapL5BH2rNJNvnhpRi3pw): protocol: expected piece hash; context canceled; EOF Dec 28 19:40:08.636592172 (556h11m8.507675207s) [noise=true , address=145.79.227.43:4163 , piece_id=77BKLLVD25UMTDQZW46CA2N225I7O4HCHCRRPQ7HLJGRHOLGSDHA, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636681511 (556h11m8.507764546s) [noise=true , address=145.79.227.19:4083 , piece_id=GL5XL7T2QQABD5364AT4AFV6RZNOWCTVNSMAVIFJU5SUQFHOPZKQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KZ5a6CQ9tUTrC6EYE7M8yg6p9fK95CN8zQLQmP4tdyz4rBfGh): protocol: expected piece hash; context canceled; EOF Dec 28 19:40:08.636687052 (556h11m8.507770087s) [noise=true , address=145.79.227.19:4083 , piece_id=GL5XL7T2QQABD5364AT4AFV6RZNOWCTVNSMAVIFJU5SUQFHOPZKQ, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636810327 (556h11m8.507893362s) [noise=true , address=145.79.227.17:4992 , piece_id=GUF7AUFCFRRPY5PSYS7DRPFQLGT4ZEL5C7P23US2VI54KYWSUCUQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hbt8HKTunpPKyAzH4XBwGQqTKzcgqPpoWwKZm9JMtaK8P7aVp): protocol: write tcp 188.166.225.170:51210->145.79.227.17:4992: use of closed network connection; write tcp 188.166.225.170:51210->145.79.227.17:4992: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51210->145.79.227.17:4992: use of closed network connection Dec 28 19:40:08.636815957 (556h11m8.507899002s) [noise=true , address=145.79.227.17:4992 , piece_id=GUF7AUFCFRRPY5PSYS7DRPFQLGT4ZEL5C7P23US2VI54KYWSUCUQ, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636834313 (556h11m8.507917358s) [seg_pos={0 0}, upload=487609]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 19:40:08.636907953 (556h11m8.507990998s) [upload=487609]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 19:40:08.636913644 (556h11m8.507996689s) [upload=487609]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 19:40:08.924991397 (556h11m8.796074442s) [upload=487609]: Done uploading segments... Dec 28 19:40:08.925014742 (556h11m8.796097777s) [upload=487609]: Upload finished. err: <nil> Duration: 43.539009208s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [...XXX..X..X......XX..............X.........................] duration: 5.886508346s trace: https://jaeger.dev.storj.io/trace/6de23ce395480be2 FAIL (2025-12-28 20:11:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: VL3CQZWQBCPGDPMEAGIQCCHVRPMV36BXHPHJ2LPJLIC6LDGWR5YA) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: NYOXGWDJWHYHW6BY3DQHFGISYSYQV4JT2H2XDJ4WLMUNE2IHN6LA) context deadline exceeded; piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: BQE24O2CURUFHQS4WSIG722BGBPCPOY32W6KAOECGV7JPCKWGOKA) context deadline exceeded ======LOGS====== Dec 28 19:51:31.138959699 (556h22m31.010042744s) [upload=487778]: Starting upload Dec 28 19:51:31.139044411 (556h22m31.010127456s) [upload=487778]: Done starting upload Dec 28 19:51:31.139127078 (556h22m31.010210113s) [upload=487778]: Uploading segments... Dec 28 19:51:31.139326739 (556h22m31.010409784s) [upload=487778]: Got next segment. Inline: false Dec 28 19:51:31.139377555 (556h22m31.0104606s) [upload=487778]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 19:51:31.139384197 (556h22m31.010467232s) [upload=487778]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 19:51:31.387475461 (556h22m31.258558496s) [seg_pos={0 0}, upload=487778]: Begin upload segment... Dec 28 19:51:31.387544984 (556h22m31.258628049s) [seg_pos={0 0}, upload=487778]: Done begin upload segment. Dec 28 19:51:31.387771626 (556h22m31.258854671s) [noise=true , address=45.91.116.7:4011 , piece_id=NQDPXLIESHRAWWBPVYOGPARTQ6PDN3TO7YCWTLXI4KBFD7JZFXXQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.388814712 (556h22m31.259897747s) [noise=true , address=77.90.31.16:4735 , piece_id=EBCUZV26NTWYQFRJEZQE3N5EU2FPNPTQRCEPR6FNK5YTRZRORP2Q, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.389204404 (556h22m31.260287449s) [noise=true , address=145.79.227.9:4366 , piece_id=PC3TE34FAHG6J6TWTEU7GKWFGP4DDUELNIT2RIAEKGTM6XDRWIZA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.389706850 (556h22m31.260789885s) [noise=true , address=145.79.227.21:4475 , piece_id=BBVRNDFL25BI3Q2CNE36E5GRAOXT55T4MQA2HTXX257HPKZ6HSPQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.390109167 (556h22m31.261192212s) [noise=true , address=145.79.227.51:4411 , piece_id=LICH4YP46TD6FZHHN4X4OFWKYBPPCFWSGHU63E2WP4AMD6HFZDNA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.390421501 (556h22m31.261504546s) [noise=true , address=77.90.31.10:4519 , piece_id=VL3CQZWQBCPGDPMEAGIQCCHVRPMV36BXHPHJ2LPJLIC6LDGWR5YA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.390852582 (556h22m31.261935617s) [noise=true , address=145.79.227.5:4221 , piece_id=O73W2GAUHPHLTWZTO7IJT4OBBXAX22MAABQ2MQSTZPNLPIQXHKRQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.392756678 (556h22m31.263839713s) [noise=true , address=45.91.116.6:4974 , piece_id=FSPEWZOEC4SHIHA6IJSXCHUCOL6GPFRDFUFXJPCQR7263Y2YFIMQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.393255988 (556h22m31.264339023s) [noise=true , address=145.79.227.2:4113 , piece_id=QJTHNBJ6SMI7GHV5634NHXQUD2P4TXERE77FO4ECEVYHQ6GUQ3FQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.393633607 (556h22m31.264716642s) [noise=true , address=145.79.227.41:4087 , piece_id=SILOBZTKKG4QOW6F2XRCEXG64GCTMC2ULONWTO2D3QFFL74FP5BQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.394230243 (556h22m31.265313278s) [noise=true , address=145.79.227.47:4265 , piece_id=43D7BZZXISPRFCKYXSTCZONLHLQ2Y6IRLFD7PDEUAJWYP6ID7DMQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.394697414 (556h22m31.265780449s) [noise=true , address=145.79.227.39:4013 , piece_id=PWYMH2SD6JOHCSM7HFDM4CHUD5C3MUCSCRML3BMZKHDZAFRM6UMA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.401284737 (556h22m31.272367792s) [noise=true , address=145.79.227.45:4229 , piece_id=AZOSHGLV2UUTXSX5NX27R3ZFIEOWS2CDGLSG34JZKCVWY4DLHUHQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.402581947 (556h22m31.273664982s) [noise=true , address=77.90.31.14:4661 , piece_id=XLUH6WZPNM2W67MLQ7KFKGM3Y4WL4TFJ6GSYKPX325FSDPKZCUUA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.403710777 (556h22m31.274793822s) [noise=true , address=45.91.116.5:4300 , piece_id=NYOXGWDJWHYHW6BY3DQHFGISYSYQV4JT2H2XDJ4WLMUNE2IHN6LA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.404110447 (556h22m31.275193492s) [noise=true , address=45.91.116.9:4261 , piece_id=BQE24O2CURUFHQS4WSIG722BGBPCPOY32W6KAOECGV7JPCKWGOKA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:33.086369496 (556h22m32.957452531s) [upload=487778]: Next returned nil segment Dec 28 19:51:33.086389915 (556h22m32.95747296s) [upload=487778]: Waiting for error group managing segments... Dec 28 19:51:33.394411997 (556h22m33.265495032s) [noise=true , address=145.79.227.21:4475 , piece_id=BBVRNDFL25BI3Q2CNE36E5GRAOXT55T4MQA2HTXX257HPKZ6HSPQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.398162920 (556h22m33.269245955s) [noise=true , address=145.79.227.39:4013 , piece_id=PWYMH2SD6JOHCSM7HFDM4CHUD5C3MUCSCRML3BMZKHDZAFRM6UMA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.403651411 (556h22m33.274734446s) [noise=true , address=145.79.227.45:4229 , piece_id=AZOSHGLV2UUTXSX5NX27R3ZFIEOWS2CDGLSG34JZKCVWY4DLHUHQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.406027926 (556h22m33.277110971s) [noise=true , address=145.79.227.51:4411 , piece_id=LICH4YP46TD6FZHHN4X4OFWKYBPPCFWSGHU63E2WP4AMD6HFZDNA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.418392551 (556h22m33.289475586s) [noise=true , address=145.79.227.47:4265 , piece_id=43D7BZZXISPRFCKYXSTCZONLHLQ2Y6IRLFD7PDEUAJWYP6ID7DMQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.420875018 (556h22m33.291958063s) [noise=true , address=145.79.227.41:4087 , piece_id=SILOBZTKKG4QOW6F2XRCEXG64GCTMC2ULONWTO2D3QFFL74FP5BQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.421544332 (556h22m33.292627377s) [noise=true , address=145.79.227.2:4113 , piece_id=QJTHNBJ6SMI7GHV5634NHXQUD2P4TXERE77FO4ECEVYHQ6GUQ3FQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.427849308 (556h22m33.298932353s) [noise=true , address=145.79.227.9:4366 , piece_id=PC3TE34FAHG6J6TWTEU7GKWFGP4DDUELNIT2RIAEKGTM6XDRWIZA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.649006914 (556h22m33.520089949s) [noise=true , address=45.91.116.9:4261 , piece_id=BQE24O2CURUFHQS4WSIG722BGBPCPOY32W6KAOECGV7JPCKWGOKA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.688630355 (556h22m33.55971339s) [noise=true , address=77.90.31.14:4661 , piece_id=XLUH6WZPNM2W67MLQ7KFKGM3Y4WL4TFJ6GSYKPX325FSDPKZCUUA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.694890445 (556h22m33.56597348s) [noise=true , address=45.91.116.5:4300 , piece_id=NYOXGWDJWHYHW6BY3DQHFGISYSYQV4JT2H2XDJ4WLMUNE2IHN6LA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.738432388 (556h22m33.609515433s) [noise=true , address=45.91.116.7:4011 , piece_id=NQDPXLIESHRAWWBPVYOGPARTQ6PDN3TO7YCWTLXI4KBFD7JZFXXQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:34.314051275 (556h22m34.18513431s) [noise=true , address=77.90.31.10:4519 , piece_id=VL3CQZWQBCPGDPMEAGIQCCHVRPMV36BXHPHJ2LPJLIC6LDGWR5YA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:34.314075711 (556h22m34.185158746s) [seg_pos={0 0}, upload=487778]: Segment reached optimal threshold of 13 pieces. Dec 28 19:51:34.314354963 (556h22m34.185437998s) [noise=true , address=45.91.116.6:4974 , piece_id=FSPEWZOEC4SHIHA6IJSXCHUCOL6GPFRDFUFXJPCQR7263Y2YFIMQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Dec 28 19:51:34.314365733 (556h22m34.185448778s) [noise=true , address=45.91.116.6:4974 , piece_id=FSPEWZOEC4SHIHA6IJSXCHUCOL6GPFRDFUFXJPCQR7263Y2YFIMQ, seg_pos={0 0}, upload=487778]: Long tail cancellation due to optimal threshold Dec 28 19:51:34.314429585 (556h22m34.18551263s) [noise=true , address=145.79.227.5:4221 , piece_id=O73W2GAUHPHLTWZTO7IJT4OBBXAX22MAABQ2MQSTZPNLPIQXHKRQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR): protocol: expected piece hash; context canceled; EOF Dec 28 19:51:34.314451267 (556h22m34.185534302s) [noise=true , address=145.79.227.5:4221 , piece_id=O73W2GAUHPHLTWZTO7IJT4OBBXAX22MAABQ2MQSTZPNLPIQXHKRQ, seg_pos={0 0}, upload=487778]: Long tail cancellation due to optimal threshold Dec 28 19:51:34.314521831 (556h22m34.185604866s) [noise=true , address=77.90.31.16:4735 , piece_id=EBCUZV26NTWYQFRJEZQE3N5EU2FPNPTQRCEPR6FNK5YTRZRORP2Q, seg_pos={0 0}, upload=487778]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 28 19:51:34.314530057 (556h22m34.185613092s) [noise=true , address=77.90.31.16:4735 , piece_id=EBCUZV26NTWYQFRJEZQE3N5EU2FPNPTQRCEPR6FNK5YTRZRORP2Q, seg_pos={0 0}, upload=487778]: Long tail cancellation due to optimal threshold Dec 28 19:51:34.314593758 (556h22m34.185676793s) [seg_pos={0 0}, upload=487778]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 19:51:34.314673079 (556h22m34.185756124s) [upload=487778]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 19:51:34.314678910 (556h22m34.185761965s) [upload=487778]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 19:51:34.609917501 (556h22m34.481000566s) [upload=487778]: Done uploading segments... Dec 28 19:51:34.609974519 (556h22m34.481057594s) [upload=487778]: Upload finished. err: <nil> Duration: 5.886539474s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [....XXX..X..X......XX..............X........................] duration: 6.139939245s trace: https://jaeger.dev.storj.io/trace/6de23ce395480be2 FAIL (2025-12-28 20:11:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: VL3CQZWQBCPGDPMEAGIQCCHVRPMV36BXHPHJ2LPJLIC6LDGWR5YA) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: NYOXGWDJWHYHW6BY3DQHFGISYSYQV4JT2H2XDJ4WLMUNE2IHN6LA) context deadline exceeded; piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: BQE24O2CURUFHQS4WSIG722BGBPCPOY32W6KAOECGV7JPCKWGOKA) context deadline exceeded ======LOGS====== Dec 28 19:51:31.138959699 (556h22m31.010042744s) [upload=487778]: Starting upload Dec 28 19:51:31.139044411 (556h22m31.010127456s) [upload=487778]: Done starting upload Dec 28 19:51:31.139127078 (556h22m31.010210113s) [upload=487778]: Uploading segments... Dec 28 19:51:31.139326739 (556h22m31.010409784s) [upload=487778]: Got next segment. Inline: false Dec 28 19:51:31.139377555 (556h22m31.0104606s) [upload=487778]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 19:51:31.139384197 (556h22m31.010467232s) [upload=487778]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 19:51:31.387475461 (556h22m31.258558496s) [seg_pos={0 0}, upload=487778]: Begin upload segment... Dec 28 19:51:31.387544984 (556h22m31.258628049s) [seg_pos={0 0}, upload=487778]: Done begin upload segment. Dec 28 19:51:31.387771626 (556h22m31.258854671s) [noise=true , address=45.91.116.7:4011 , piece_id=NQDPXLIESHRAWWBPVYOGPARTQ6PDN3TO7YCWTLXI4KBFD7JZFXXQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.388814712 (556h22m31.259897747s) [noise=true , address=77.90.31.16:4735 , piece_id=EBCUZV26NTWYQFRJEZQE3N5EU2FPNPTQRCEPR6FNK5YTRZRORP2Q, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.389204404 (556h22m31.260287449s) [noise=true , address=145.79.227.9:4366 , piece_id=PC3TE34FAHG6J6TWTEU7GKWFGP4DDUELNIT2RIAEKGTM6XDRWIZA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.389706850 (556h22m31.260789885s) [noise=true , address=145.79.227.21:4475 , piece_id=BBVRNDFL25BI3Q2CNE36E5GRAOXT55T4MQA2HTXX257HPKZ6HSPQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.390109167 (556h22m31.261192212s) [noise=true , address=145.79.227.51:4411 , piece_id=LICH4YP46TD6FZHHN4X4OFWKYBPPCFWSGHU63E2WP4AMD6HFZDNA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.390421501 (556h22m31.261504546s) [noise=true , address=77.90.31.10:4519 , piece_id=VL3CQZWQBCPGDPMEAGIQCCHVRPMV36BXHPHJ2LPJLIC6LDGWR5YA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.390852582 (556h22m31.261935617s) [noise=true , address=145.79.227.5:4221 , piece_id=O73W2GAUHPHLTWZTO7IJT4OBBXAX22MAABQ2MQSTZPNLPIQXHKRQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.392756678 (556h22m31.263839713s) [noise=true , address=45.91.116.6:4974 , piece_id=FSPEWZOEC4SHIHA6IJSXCHUCOL6GPFRDFUFXJPCQR7263Y2YFIMQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.393255988 (556h22m31.264339023s) [noise=true , address=145.79.227.2:4113 , piece_id=QJTHNBJ6SMI7GHV5634NHXQUD2P4TXERE77FO4ECEVYHQ6GUQ3FQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.393633607 (556h22m31.264716642s) [noise=true , address=145.79.227.41:4087 , piece_id=SILOBZTKKG4QOW6F2XRCEXG64GCTMC2ULONWTO2D3QFFL74FP5BQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.394230243 (556h22m31.265313278s) [noise=true , address=145.79.227.47:4265 , piece_id=43D7BZZXISPRFCKYXSTCZONLHLQ2Y6IRLFD7PDEUAJWYP6ID7DMQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.394697414 (556h22m31.265780449s) [noise=true , address=145.79.227.39:4013 , piece_id=PWYMH2SD6JOHCSM7HFDM4CHUD5C3MUCSCRML3BMZKHDZAFRM6UMA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.401284737 (556h22m31.272367792s) [noise=true , address=145.79.227.45:4229 , piece_id=AZOSHGLV2UUTXSX5NX27R3ZFIEOWS2CDGLSG34JZKCVWY4DLHUHQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.402581947 (556h22m31.273664982s) [noise=true , address=77.90.31.14:4661 , piece_id=XLUH6WZPNM2W67MLQ7KFKGM3Y4WL4TFJ6GSYKPX325FSDPKZCUUA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.403710777 (556h22m31.274793822s) [noise=true , address=45.91.116.5:4300 , piece_id=NYOXGWDJWHYHW6BY3DQHFGISYSYQV4JT2H2XDJ4WLMUNE2IHN6LA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.404110447 (556h22m31.275193492s) [noise=true , address=45.91.116.9:4261 , piece_id=BQE24O2CURUFHQS4WSIG722BGBPCPOY32W6KAOECGV7JPCKWGOKA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:33.086369496 (556h22m32.957452531s) [upload=487778]: Next returned nil segment Dec 28 19:51:33.086389915 (556h22m32.95747296s) [upload=487778]: Waiting for error group managing segments... Dec 28 19:51:33.394411997 (556h22m33.265495032s) [noise=true , address=145.79.227.21:4475 , piece_id=BBVRNDFL25BI3Q2CNE36E5GRAOXT55T4MQA2HTXX257HPKZ6HSPQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.398162920 (556h22m33.269245955s) [noise=true , address=145.79.227.39:4013 , piece_id=PWYMH2SD6JOHCSM7HFDM4CHUD5C3MUCSCRML3BMZKHDZAFRM6UMA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.403651411 (556h22m33.274734446s) [noise=true , address=145.79.227.45:4229 , piece_id=AZOSHGLV2UUTXSX5NX27R3ZFIEOWS2CDGLSG34JZKCVWY4DLHUHQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.406027926 (556h22m33.277110971s) [noise=true , address=145.79.227.51:4411 , piece_id=LICH4YP46TD6FZHHN4X4OFWKYBPPCFWSGHU63E2WP4AMD6HFZDNA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.418392551 (556h22m33.289475586s) [noise=true , address=145.79.227.47:4265 , piece_id=43D7BZZXISPRFCKYXSTCZONLHLQ2Y6IRLFD7PDEUAJWYP6ID7DMQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.420875018 (556h22m33.291958063s) [noise=true , address=145.79.227.41:4087 , piece_id=SILOBZTKKG4QOW6F2XRCEXG64GCTMC2ULONWTO2D3QFFL74FP5BQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.421544332 (556h22m33.292627377s) [noise=true , address=145.79.227.2:4113 , piece_id=QJTHNBJ6SMI7GHV5634NHXQUD2P4TXERE77FO4ECEVYHQ6GUQ3FQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.427849308 (556h22m33.298932353s) [noise=true , address=145.79.227.9:4366 , piece_id=PC3TE34FAHG6J6TWTEU7GKWFGP4DDUELNIT2RIAEKGTM6XDRWIZA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.649006914 (556h22m33.520089949s) [noise=true , address=45.91.116.9:4261 , piece_id=BQE24O2CURUFHQS4WSIG722BGBPCPOY32W6KAOECGV7JPCKWGOKA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.688630355 (556h22m33.55971339s) [noise=true , address=77.90.31.14:4661 , piece_id=XLUH6WZPNM2W67MLQ7KFKGM3Y4WL4TFJ6GSYKPX325FSDPKZCUUA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.694890445 (556h22m33.56597348s) [noise=true , address=45.91.116.5:4300 , piece_id=NYOXGWDJWHYHW6BY3DQHFGISYSYQV4JT2H2XDJ4WLMUNE2IHN6LA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.738432388 (556h22m33.609515433s) [noise=true , address=45.91.116.7:4011 , piece_id=NQDPXLIESHRAWWBPVYOGPARTQ6PDN3TO7YCWTLXI4KBFD7JZFXXQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:34.314051275 (556h22m34.18513431s) [noise=true , address=77.90.31.10:4519 , piece_id=VL3CQZWQBCPGDPMEAGIQCCHVRPMV36BXHPHJ2LPJLIC6LDGWR5YA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:34.314075711 (556h22m34.185158746s) [seg_pos={0 0}, upload=487778]: Segment reached optimal threshold of 13 pieces. Dec 28 19:51:34.314354963 (556h22m34.185437998s) [noise=true , address=45.91.116.6:4974 , piece_id=FSPEWZOEC4SHIHA6IJSXCHUCOL6GPFRDFUFXJPCQR7263Y2YFIMQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Dec 28 19:51:34.314365733 (556h22m34.185448778s) [noise=true , address=45.91.116.6:4974 , piece_id=FSPEWZOEC4SHIHA6IJSXCHUCOL6GPFRDFUFXJPCQR7263Y2YFIMQ, seg_pos={0 0}, upload=487778]: Long tail cancellation due to optimal threshold Dec 28 19:51:34.314429585 (556h22m34.18551263s) [noise=true , address=145.79.227.5:4221 , piece_id=O73W2GAUHPHLTWZTO7IJT4OBBXAX22MAABQ2MQSTZPNLPIQXHKRQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR): protocol: expected piece hash; context canceled; EOF Dec 28 19:51:34.314451267 (556h22m34.185534302s) [noise=true , address=145.79.227.5:4221 , piece_id=O73W2GAUHPHLTWZTO7IJT4OBBXAX22MAABQ2MQSTZPNLPIQXHKRQ, seg_pos={0 0}, upload=487778]: Long tail cancellation due to optimal threshold Dec 28 19:51:34.314521831 (556h22m34.185604866s) [noise=true , address=77.90.31.16:4735 , piece_id=EBCUZV26NTWYQFRJEZQE3N5EU2FPNPTQRCEPR6FNK5YTRZRORP2Q, seg_pos={0 0}, upload=487778]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 28 19:51:34.314530057 (556h22m34.185613092s) [noise=true , address=77.90.31.16:4735 , piece_id=EBCUZV26NTWYQFRJEZQE3N5EU2FPNPTQRCEPR6FNK5YTRZRORP2Q, seg_pos={0 0}, upload=487778]: Long tail cancellation due to optimal threshold Dec 28 19:51:34.314593758 (556h22m34.185676793s) [seg_pos={0 0}, upload=487778]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 19:51:34.314673079 (556h22m34.185756124s) [upload=487778]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 19:51:34.314678910 (556h22m34.185761965s) [upload=487778]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 19:51:34.609917501 (556h22m34.481000566s) [upload=487778]: Done uploading segments... Dec 28 19:51:34.609974519 (556h22m34.481057594s) [upload=487778]: Upload finished. err: <nil> Duration: 6.139966467s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 19.801272378s trace: https://jaeger.dev.storj.io/trace/10e1641b228b2a1f FAIL (2025-12-28 20:11:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 40:05.744568479 (556h11m5.615651514s) [noise=true , address=145.79.227.28:4744 , piece_id=42LII6EOG3CMBTIL5I4RZQPDYU52TKJN3PGWEQXROBYMNDFYIIVA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.745099351 (556h11m5.616182396s) [noise=true , address=145.79.227.23:4550 , piece_id=NWIC722LZUHJSAMPWSKCLPRIDGWPVUEYMS5KRAMX3IIMD2KR4XGQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.745777401 (556h11m5.616860436s) [noise=true , address=77.90.31.15:4722 , piece_id=AF7F2OA247CJIDFTEU6HEOIYGOINEYPMII53BKJQXI7UZ26MJIUA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.746374739 (556h11m5.617457784s) [noise=true , address=77.90.31.10:4522 , piece_id=JOHOMHCNYXF4XQGLN2NJUSWSUO3XFBRRMYD27SLIZAK2RCT3ZC2A, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.746934555 (556h11m5.618017601s) [noise=true , address=77.90.31.21:4896 , piece_id=ATTC7BZUT4HAAEAZCHFZQE4INV2LLSWR2YCUNVA73LPJCW3YL35A, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.747556861 (556h11m5.618640146s) [noise=true , address=64.49.13.26:4956 , piece_id=FFHUPOOM7PLEABRHFH6H2SI6WXJBU2XFCLZJUBV4CXUJ6EP4PI2Q, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.748964150 (556h11m5.620047195s) [noise=true , address=145.79.227.34:4916 , piece_id=SPYSFWMJ752FC5JGRE36G54S2FHG4DQFCV6CIZONQHJDXDCN52VQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.751119314 (556h11m5.622202359s) [noise=true , address=77.90.31.18:4816 , piece_id=YT3PSYNQTDQBVBRUDNOXJ2XCN6EFQA54WBSNL6MZMXFXR7XNCHIA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.753413222 (556h11m5.624496257s) [noise=true , address=77.90.31.11:4582 , piece_id=ZRE24R27TX4E5DTV2POX7EJANQ4CM75D6ZOUS6SKM3QHHQCKRBRA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.755230482 (556h11m5.626313527s) [noise=true , address=64.49.13.154:4932 , piece_id=QB5WM5RZ2P2GC6RIGGM34IANVEI2SNDJ2PO556JXKTDBX5HLD4YA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.757598362 (556h11m5.628681407s) [noise=true , address=145.79.227.35:4510 , piece_id=P275TU367DTYXBQOGGV5THFUPHJUWGSM5IMJQ753VE4MX4IPOONQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.759644679 (556h11m5.630727734s) [noise=true , address=145.79.227.46:4626 , piece_id=OQOTWYLVIU7GJBBLYWQOEEFKR3TNAUOXZS6UAFJWJ32SIOTXNXEA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.761217594 (556h11m5.632300639s) [noise=true , address=145.79.227.43:4163 , piece_id=77BKLLVD25UMTDQZW46CA2N225I7O4HCHCRRPQ7HLJGRHOLGSDHA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.762138507 (556h11m5.633221552s) [noise=true , address=77.90.31.19:4860 , piece_id=5FUYVK2Z6LBZOAGFI2NOTAAUIV4NMBI5ERMA5RH4BMONNHABZLDQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.764304321 (556h11m5.635387356s) [noise=true , address=145.79.227.30:4784 , piece_id=MKCRSFWVFVGOAYCK5M66HO3OT7OMYWLVR2KMDXUGBS2VXXNCUSSQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.765421217 (556h11m5.636504252s) [noise=true , address=145.79.227.44:4216 , piece_id=GSBWC2LXUOZYOVD55KNQYA5Z3S2TCYROBB5O2XZIIVFUQMJQMYNA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:07.082099826 (556h11m6.953182871s) [upload=487609]: Next returned nil segment Dec 28 19:40:07.082116707 (556h11m6.953199752s) [upload=487609]: Waiting for error group managing segments... Dec 28 19:40:07.415884144 (556h11m7.286967179s) [noise=true , address=64.49.13.154:4932 , piece_id=QB5WM5RZ2P2GC6RIGGM34IANVEI2SNDJ2PO556JXKTDBX5HLD4YA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.615434252 (556h11m7.486517297s) [noise=true , address=145.79.227.23:4550 , piece_id=NWIC722LZUHJSAMPWSKCLPRIDGWPVUEYMS5KRAMX3IIMD2KR4XGQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.647182077 (556h11m7.518265112s) [noise=true , address=145.79.227.28:4744 , piece_id=42LII6EOG3CMBTIL5I4RZQPDYU52TKJN3PGWEQXROBYMNDFYIIVA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.678458624 (556h11m7.549541659s) [noise=true , address=145.79.227.46:4626 , piece_id=OQOTWYLVIU7GJBBLYWQOEEFKR3TNAUOXZS6UAFJWJ32SIOTXNXEA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.793406049 (556h11m7.664489084s) [noise=true , address=77.90.31.19:4860 , piece_id=5FUYVK2Z6LBZOAGFI2NOTAAUIV4NMBI5ERMA5RH4BMONNHABZLDQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.802506498 (556h11m7.673589543s) [noise=true , address=77.90.31.21:4896 , piece_id=ATTC7BZUT4HAAEAZCHFZQE4INV2LLSWR2YCUNVA73LPJCW3YL35A, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.853663242 (556h11m7.724746287s) [noise=true , address=77.90.31.10:4522 , piece_id=JOHOMHCNYXF4XQGLN2NJUSWSUO3XFBRRMYD27SLIZAK2RCT3ZC2A, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.880639500 (556h11m7.751722535s) [noise=true , address=77.90.31.11:4582 , piece_id=ZRE24R27TX4E5DTV2POX7EJANQ4CM75D6ZOUS6SKM3QHHQCKRBRA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.904835446 (556h11m7.775918491s) [noise=true , address=77.90.31.18:4816 , piece_id=YT3PSYNQTDQBVBRUDNOXJ2XCN6EFQA54WBSNL6MZMXFXR7XNCHIA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.988914260 (556h11m7.859997295s) [noise=true , address=77.90.31.15:4722 , piece_id=AF7F2OA247CJIDFTEU6HEOIYGOINEYPMII53BKJQXI7UZ26MJIUA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.484927449 (556h11m8.356010484s) [noise=true , address=145.79.227.35:4510 , piece_id=P275TU367DTYXBQOGGV5THFUPHJUWGSM5IMJQ753VE4MX4IPOONQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.610317849 (556h11m8.481400894s) [noise=true , address=145.79.227.44:4216 , piece_id=GSBWC2LXUOZYOVD55KNQYA5Z3S2TCYROBB5O2XZIIVFUQMJQMYNA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.611969885 (556h11m8.48305292s) [noise=true , address=145.79.227.34:4916 , piece_id=SPYSFWMJ752FC5JGRE36G54S2FHG4DQFCV6CIZONQHJDXDCN52VQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.636098801 (556h11m8.507181836s) [noise=true , address=145.79.227.30:4784 , piece_id=MKCRSFWVFVGOAYCK5M66HO3OT7OMYWLVR2KMDXUGBS2VXXNCUSSQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.636122136 (556h11m8.507205181s) [seg_pos={0 0}, upload=487609]: Segment reached optimal threshold of 14 pieces. Dec 28 19:40:08.636333929 (556h11m8.507416974s) [noise=true , address=64.49.13.26:4956 , piece_id=FFHUPOOM7PLEABRHFH6H2SI6WXJBU2XFCLZJUBV4CXUJ6EP4PI2Q, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TYHhjpfyBKkenSwviyQCUZpSBbnaFCGBFRkM4fyKLDuxKx4Ju): protocol: expected piece hash; context canceled; EOF Dec 28 19:40:08.636342175 (556h11m8.50742521s) [noise=true , address=64.49.13.26:4956 , piece_id=FFHUPOOM7PLEABRHFH6H2SI6WXJBU2XFCLZJUBV4CXUJ6EP4PI2Q, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636585959 (556h11m8.507668994s) [noise=true , address=145.79.227.43:4163 , piece_id=77BKLLVD25UMTDQZW46CA2N225I7O4HCHCRRPQ7HLJGRHOLGSDHA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CgPQ5hkrJdqbJvGarxKjXvYYnrXQqapL5BH2rNJNvnhpRi3pw): protocol: expected piece hash; context canceled; EOF Dec 28 19:40:08.636592172 (556h11m8.507675207s) [noise=true , address=145.79.227.43:4163 , piece_id=77BKLLVD25UMTDQZW46CA2N225I7O4HCHCRRPQ7HLJGRHOLGSDHA, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636681511 (556h11m8.507764546s) [noise=true , address=145.79.227.19:4083 , piece_id=GL5XL7T2QQABD5364AT4AFV6RZNOWCTVNSMAVIFJU5SUQFHOPZKQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KZ5a6CQ9tUTrC6EYE7M8yg6p9fK95CN8zQLQmP4tdyz4rBfGh): protocol: expected piece hash; context canceled; EOF Dec 28 19:40:08.636687052 (556h11m8.507770087s) [noise=true , address=145.79.227.19:4083 , piece_id=GL5XL7T2QQABD5364AT4AFV6RZNOWCTVNSMAVIFJU5SUQFHOPZKQ, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636810327 (556h11m8.507893362s) [noise=true , address=145.79.227.17:4992 , piece_id=GUF7AUFCFRRPY5PSYS7DRPFQLGT4ZEL5C7P23US2VI54KYWSUCUQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hbt8HKTunpPKyAzH4XBwGQqTKzcgqPpoWwKZm9JMtaK8P7aVp): protocol: write tcp 188.166.225.170:51210->145.79.227.17:4992: use of closed network connection; write tcp 188.166.225.170:51210->145.79.227.17:4992: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51210->145.79.227.17:4992: use of closed network connection Dec 28 19:40:08.636815957 (556h11m8.507899002s) [noise=true , address=145.79.227.17:4992 , piece_id=GUF7AUFCFRRPY5PSYS7DRPFQLGT4ZEL5C7P23US2VI54KYWSUCUQ, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636834313 (556h11m8.507917358s) [seg_pos={0 0}, upload=487609]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 19:40:08.636907953 (556h11m8.507990998s) [upload=487609]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 19:40:08.636913644 (556h11m8.507996689s) [upload=487609]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 19:40:08.924991397 (556h11m8.796074442s) [upload=487609]: Done uploading segments... Dec 28 19:40:08.925014742 (556h11m8.796097777s) [upload=487609]: Upload finished. err: <nil> Duration: 19.801302135s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [X....XXX..X..X......XX..............X.......................] duration: 6.118934717s trace: https://jaeger.dev.storj.io/trace/6de23ce395480be2 FAIL (2025-12-28 20:10:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: VL3CQZWQBCPGDPMEAGIQCCHVRPMV36BXHPHJ2LPJLIC6LDGWR5YA) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: NYOXGWDJWHYHW6BY3DQHFGISYSYQV4JT2H2XDJ4WLMUNE2IHN6LA) context deadline exceeded; piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: BQE24O2CURUFHQS4WSIG722BGBPCPOY32W6KAOECGV7JPCKWGOKA) context deadline exceeded ======LOGS====== Dec 28 19:51:31.138959699 (556h22m31.010042744s) [upload=487778]: Starting upload Dec 28 19:51:31.139044411 (556h22m31.010127456s) [upload=487778]: Done starting upload Dec 28 19:51:31.139127078 (556h22m31.010210113s) [upload=487778]: Uploading segments... Dec 28 19:51:31.139326739 (556h22m31.010409784s) [upload=487778]: Got next segment. Inline: false Dec 28 19:51:31.139377555 (556h22m31.0104606s) [upload=487778]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 19:51:31.139384197 (556h22m31.010467232s) [upload=487778]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 19:51:31.387475461 (556h22m31.258558496s) [seg_pos={0 0}, upload=487778]: Begin upload segment... Dec 28 19:51:31.387544984 (556h22m31.258628049s) [seg_pos={0 0}, upload=487778]: Done begin upload segment. Dec 28 19:51:31.387771626 (556h22m31.258854671s) [noise=true , address=45.91.116.7:4011 , piece_id=NQDPXLIESHRAWWBPVYOGPARTQ6PDN3TO7YCWTLXI4KBFD7JZFXXQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.388814712 (556h22m31.259897747s) [noise=true , address=77.90.31.16:4735 , piece_id=EBCUZV26NTWYQFRJEZQE3N5EU2FPNPTQRCEPR6FNK5YTRZRORP2Q, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.389204404 (556h22m31.260287449s) [noise=true , address=145.79.227.9:4366 , piece_id=PC3TE34FAHG6J6TWTEU7GKWFGP4DDUELNIT2RIAEKGTM6XDRWIZA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.389706850 (556h22m31.260789885s) [noise=true , address=145.79.227.21:4475 , piece_id=BBVRNDFL25BI3Q2CNE36E5GRAOXT55T4MQA2HTXX257HPKZ6HSPQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.390109167 (556h22m31.261192212s) [noise=true , address=145.79.227.51:4411 , piece_id=LICH4YP46TD6FZHHN4X4OFWKYBPPCFWSGHU63E2WP4AMD6HFZDNA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.390421501 (556h22m31.261504546s) [noise=true , address=77.90.31.10:4519 , piece_id=VL3CQZWQBCPGDPMEAGIQCCHVRPMV36BXHPHJ2LPJLIC6LDGWR5YA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.390852582 (556h22m31.261935617s) [noise=true , address=145.79.227.5:4221 , piece_id=O73W2GAUHPHLTWZTO7IJT4OBBXAX22MAABQ2MQSTZPNLPIQXHKRQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.392756678 (556h22m31.263839713s) [noise=true , address=45.91.116.6:4974 , piece_id=FSPEWZOEC4SHIHA6IJSXCHUCOL6GPFRDFUFXJPCQR7263Y2YFIMQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.393255988 (556h22m31.264339023s) [noise=true , address=145.79.227.2:4113 , piece_id=QJTHNBJ6SMI7GHV5634NHXQUD2P4TXERE77FO4ECEVYHQ6GUQ3FQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.393633607 (556h22m31.264716642s) [noise=true , address=145.79.227.41:4087 , piece_id=SILOBZTKKG4QOW6F2XRCEXG64GCTMC2ULONWTO2D3QFFL74FP5BQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.394230243 (556h22m31.265313278s) [noise=true , address=145.79.227.47:4265 , piece_id=43D7BZZXISPRFCKYXSTCZONLHLQ2Y6IRLFD7PDEUAJWYP6ID7DMQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.394697414 (556h22m31.265780449s) [noise=true , address=145.79.227.39:4013 , piece_id=PWYMH2SD6JOHCSM7HFDM4CHUD5C3MUCSCRML3BMZKHDZAFRM6UMA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.401284737 (556h22m31.272367792s) [noise=true , address=145.79.227.45:4229 , piece_id=AZOSHGLV2UUTXSX5NX27R3ZFIEOWS2CDGLSG34JZKCVWY4DLHUHQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.402581947 (556h22m31.273664982s) [noise=true , address=77.90.31.14:4661 , piece_id=XLUH6WZPNM2W67MLQ7KFKGM3Y4WL4TFJ6GSYKPX325FSDPKZCUUA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.403710777 (556h22m31.274793822s) [noise=true , address=45.91.116.5:4300 , piece_id=NYOXGWDJWHYHW6BY3DQHFGISYSYQV4JT2H2XDJ4WLMUNE2IHN6LA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.404110447 (556h22m31.275193492s) [noise=true , address=45.91.116.9:4261 , piece_id=BQE24O2CURUFHQS4WSIG722BGBPCPOY32W6KAOECGV7JPCKWGOKA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:33.086369496 (556h22m32.957452531s) [upload=487778]: Next returned nil segment Dec 28 19:51:33.086389915 (556h22m32.95747296s) [upload=487778]: Waiting for error group managing segments... Dec 28 19:51:33.394411997 (556h22m33.265495032s) [noise=true , address=145.79.227.21:4475 , piece_id=BBVRNDFL25BI3Q2CNE36E5GRAOXT55T4MQA2HTXX257HPKZ6HSPQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.398162920 (556h22m33.269245955s) [noise=true , address=145.79.227.39:4013 , piece_id=PWYMH2SD6JOHCSM7HFDM4CHUD5C3MUCSCRML3BMZKHDZAFRM6UMA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.403651411 (556h22m33.274734446s) [noise=true , address=145.79.227.45:4229 , piece_id=AZOSHGLV2UUTXSX5NX27R3ZFIEOWS2CDGLSG34JZKCVWY4DLHUHQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.406027926 (556h22m33.277110971s) [noise=true , address=145.79.227.51:4411 , piece_id=LICH4YP46TD6FZHHN4X4OFWKYBPPCFWSGHU63E2WP4AMD6HFZDNA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.418392551 (556h22m33.289475586s) [noise=true , address=145.79.227.47:4265 , piece_id=43D7BZZXISPRFCKYXSTCZONLHLQ2Y6IRLFD7PDEUAJWYP6ID7DMQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.420875018 (556h22m33.291958063s) [noise=true , address=145.79.227.41:4087 , piece_id=SILOBZTKKG4QOW6F2XRCEXG64GCTMC2ULONWTO2D3QFFL74FP5BQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.421544332 (556h22m33.292627377s) [noise=true , address=145.79.227.2:4113 , piece_id=QJTHNBJ6SMI7GHV5634NHXQUD2P4TXERE77FO4ECEVYHQ6GUQ3FQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.427849308 (556h22m33.298932353s) [noise=true , address=145.79.227.9:4366 , piece_id=PC3TE34FAHG6J6TWTEU7GKWFGP4DDUELNIT2RIAEKGTM6XDRWIZA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.649006914 (556h22m33.520089949s) [noise=true , address=45.91.116.9:4261 , piece_id=BQE24O2CURUFHQS4WSIG722BGBPCPOY32W6KAOECGV7JPCKWGOKA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.688630355 (556h22m33.55971339s) [noise=true , address=77.90.31.14:4661 , piece_id=XLUH6WZPNM2W67MLQ7KFKGM3Y4WL4TFJ6GSYKPX325FSDPKZCUUA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.694890445 (556h22m33.56597348s) [noise=true , address=45.91.116.5:4300 , piece_id=NYOXGWDJWHYHW6BY3DQHFGISYSYQV4JT2H2XDJ4WLMUNE2IHN6LA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.738432388 (556h22m33.609515433s) [noise=true , address=45.91.116.7:4011 , piece_id=NQDPXLIESHRAWWBPVYOGPARTQ6PDN3TO7YCWTLXI4KBFD7JZFXXQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:34.314051275 (556h22m34.18513431s) [noise=true , address=77.90.31.10:4519 , piece_id=VL3CQZWQBCPGDPMEAGIQCCHVRPMV36BXHPHJ2LPJLIC6LDGWR5YA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:34.314075711 (556h22m34.185158746s) [seg_pos={0 0}, upload=487778]: Segment reached optimal threshold of 13 pieces. Dec 28 19:51:34.314354963 (556h22m34.185437998s) [noise=true , address=45.91.116.6:4974 , piece_id=FSPEWZOEC4SHIHA6IJSXCHUCOL6GPFRDFUFXJPCQR7263Y2YFIMQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Dec 28 19:51:34.314365733 (556h22m34.185448778s) [noise=true , address=45.91.116.6:4974 , piece_id=FSPEWZOEC4SHIHA6IJSXCHUCOL6GPFRDFUFXJPCQR7263Y2YFIMQ, seg_pos={0 0}, upload=487778]: Long tail cancellation due to optimal threshold Dec 28 19:51:34.314429585 (556h22m34.18551263s) [noise=true , address=145.79.227.5:4221 , piece_id=O73W2GAUHPHLTWZTO7IJT4OBBXAX22MAABQ2MQSTZPNLPIQXHKRQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR): protocol: expected piece hash; context canceled; EOF Dec 28 19:51:34.314451267 (556h22m34.185534302s) [noise=true , address=145.79.227.5:4221 , piece_id=O73W2GAUHPHLTWZTO7IJT4OBBXAX22MAABQ2MQSTZPNLPIQXHKRQ, seg_pos={0 0}, upload=487778]: Long tail cancellation due to optimal threshold Dec 28 19:51:34.314521831 (556h22m34.185604866s) [noise=true , address=77.90.31.16:4735 , piece_id=EBCUZV26NTWYQFRJEZQE3N5EU2FPNPTQRCEPR6FNK5YTRZRORP2Q, seg_pos={0 0}, upload=487778]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 28 19:51:34.314530057 (556h22m34.185613092s) [noise=true , address=77.90.31.16:4735 , piece_id=EBCUZV26NTWYQFRJEZQE3N5EU2FPNPTQRCEPR6FNK5YTRZRORP2Q, seg_pos={0 0}, upload=487778]: Long tail cancellation due to optimal threshold Dec 28 19:51:34.314593758 (556h22m34.185676793s) [seg_pos={0 0}, upload=487778]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 19:51:34.314673079 (556h22m34.185756124s) [upload=487778]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 19:51:34.314678910 (556h22m34.185761965s) [upload=487778]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 19:51:34.609917501 (556h22m34.481000566s) [upload=487778]: Done uploading segments... Dec 28 19:51:34.609974519 (556h22m34.481057594s) [upload=487778]: Upload finished. err: <nil> Duration: 6.118960285s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 15.233886941s trace: https://jaeger.dev.storj.io/trace/10e1641b228b2a1f FAIL (2025-12-28 20:10:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 40:05.744568479 (556h11m5.615651514s) [noise=true , address=145.79.227.28:4744 , piece_id=42LII6EOG3CMBTIL5I4RZQPDYU52TKJN3PGWEQXROBYMNDFYIIVA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.745099351 (556h11m5.616182396s) [noise=true , address=145.79.227.23:4550 , piece_id=NWIC722LZUHJSAMPWSKCLPRIDGWPVUEYMS5KRAMX3IIMD2KR4XGQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.745777401 (556h11m5.616860436s) [noise=true , address=77.90.31.15:4722 , piece_id=AF7F2OA247CJIDFTEU6HEOIYGOINEYPMII53BKJQXI7UZ26MJIUA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.746374739 (556h11m5.617457784s) [noise=true , address=77.90.31.10:4522 , piece_id=JOHOMHCNYXF4XQGLN2NJUSWSUO3XFBRRMYD27SLIZAK2RCT3ZC2A, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.746934555 (556h11m5.618017601s) [noise=true , address=77.90.31.21:4896 , piece_id=ATTC7BZUT4HAAEAZCHFZQE4INV2LLSWR2YCUNVA73LPJCW3YL35A, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.747556861 (556h11m5.618640146s) [noise=true , address=64.49.13.26:4956 , piece_id=FFHUPOOM7PLEABRHFH6H2SI6WXJBU2XFCLZJUBV4CXUJ6EP4PI2Q, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.748964150 (556h11m5.620047195s) [noise=true , address=145.79.227.34:4916 , piece_id=SPYSFWMJ752FC5JGRE36G54S2FHG4DQFCV6CIZONQHJDXDCN52VQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.751119314 (556h11m5.622202359s) [noise=true , address=77.90.31.18:4816 , piece_id=YT3PSYNQTDQBVBRUDNOXJ2XCN6EFQA54WBSNL6MZMXFXR7XNCHIA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.753413222 (556h11m5.624496257s) [noise=true , address=77.90.31.11:4582 , piece_id=ZRE24R27TX4E5DTV2POX7EJANQ4CM75D6ZOUS6SKM3QHHQCKRBRA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.755230482 (556h11m5.626313527s) [noise=true , address=64.49.13.154:4932 , piece_id=QB5WM5RZ2P2GC6RIGGM34IANVEI2SNDJ2PO556JXKTDBX5HLD4YA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.757598362 (556h11m5.628681407s) [noise=true , address=145.79.227.35:4510 , piece_id=P275TU367DTYXBQOGGV5THFUPHJUWGSM5IMJQ753VE4MX4IPOONQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.759644679 (556h11m5.630727734s) [noise=true , address=145.79.227.46:4626 , piece_id=OQOTWYLVIU7GJBBLYWQOEEFKR3TNAUOXZS6UAFJWJ32SIOTXNXEA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.761217594 (556h11m5.632300639s) [noise=true , address=145.79.227.43:4163 , piece_id=77BKLLVD25UMTDQZW46CA2N225I7O4HCHCRRPQ7HLJGRHOLGSDHA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.762138507 (556h11m5.633221552s) [noise=true , address=77.90.31.19:4860 , piece_id=5FUYVK2Z6LBZOAGFI2NOTAAUIV4NMBI5ERMA5RH4BMONNHABZLDQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.764304321 (556h11m5.635387356s) [noise=true , address=145.79.227.30:4784 , piece_id=MKCRSFWVFVGOAYCK5M66HO3OT7OMYWLVR2KMDXUGBS2VXXNCUSSQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.765421217 (556h11m5.636504252s) [noise=true , address=145.79.227.44:4216 , piece_id=GSBWC2LXUOZYOVD55KNQYA5Z3S2TCYROBB5O2XZIIVFUQMJQMYNA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:07.082099826 (556h11m6.953182871s) [upload=487609]: Next returned nil segment Dec 28 19:40:07.082116707 (556h11m6.953199752s) [upload=487609]: Waiting for error group managing segments... Dec 28 19:40:07.415884144 (556h11m7.286967179s) [noise=true , address=64.49.13.154:4932 , piece_id=QB5WM5RZ2P2GC6RIGGM34IANVEI2SNDJ2PO556JXKTDBX5HLD4YA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.615434252 (556h11m7.486517297s) [noise=true , address=145.79.227.23:4550 , piece_id=NWIC722LZUHJSAMPWSKCLPRIDGWPVUEYMS5KRAMX3IIMD2KR4XGQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.647182077 (556h11m7.518265112s) [noise=true , address=145.79.227.28:4744 , piece_id=42LII6EOG3CMBTIL5I4RZQPDYU52TKJN3PGWEQXROBYMNDFYIIVA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.678458624 (556h11m7.549541659s) [noise=true , address=145.79.227.46:4626 , piece_id=OQOTWYLVIU7GJBBLYWQOEEFKR3TNAUOXZS6UAFJWJ32SIOTXNXEA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.793406049 (556h11m7.664489084s) [noise=true , address=77.90.31.19:4860 , piece_id=5FUYVK2Z6LBZOAGFI2NOTAAUIV4NMBI5ERMA5RH4BMONNHABZLDQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.802506498 (556h11m7.673589543s) [noise=true , address=77.90.31.21:4896 , piece_id=ATTC7BZUT4HAAEAZCHFZQE4INV2LLSWR2YCUNVA73LPJCW3YL35A, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.853663242 (556h11m7.724746287s) [noise=true , address=77.90.31.10:4522 , piece_id=JOHOMHCNYXF4XQGLN2NJUSWSUO3XFBRRMYD27SLIZAK2RCT3ZC2A, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.880639500 (556h11m7.751722535s) [noise=true , address=77.90.31.11:4582 , piece_id=ZRE24R27TX4E5DTV2POX7EJANQ4CM75D6ZOUS6SKM3QHHQCKRBRA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.904835446 (556h11m7.775918491s) [noise=true , address=77.90.31.18:4816 , piece_id=YT3PSYNQTDQBVBRUDNOXJ2XCN6EFQA54WBSNL6MZMXFXR7XNCHIA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.988914260 (556h11m7.859997295s) [noise=true , address=77.90.31.15:4722 , piece_id=AF7F2OA247CJIDFTEU6HEOIYGOINEYPMII53BKJQXI7UZ26MJIUA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.484927449 (556h11m8.356010484s) [noise=true , address=145.79.227.35:4510 , piece_id=P275TU367DTYXBQOGGV5THFUPHJUWGSM5IMJQ753VE4MX4IPOONQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.610317849 (556h11m8.481400894s) [noise=true , address=145.79.227.44:4216 , piece_id=GSBWC2LXUOZYOVD55KNQYA5Z3S2TCYROBB5O2XZIIVFUQMJQMYNA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.611969885 (556h11m8.48305292s) [noise=true , address=145.79.227.34:4916 , piece_id=SPYSFWMJ752FC5JGRE36G54S2FHG4DQFCV6CIZONQHJDXDCN52VQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.636098801 (556h11m8.507181836s) [noise=true , address=145.79.227.30:4784 , piece_id=MKCRSFWVFVGOAYCK5M66HO3OT7OMYWLVR2KMDXUGBS2VXXNCUSSQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.636122136 (556h11m8.507205181s) [seg_pos={0 0}, upload=487609]: Segment reached optimal threshold of 14 pieces. Dec 28 19:40:08.636333929 (556h11m8.507416974s) [noise=true , address=64.49.13.26:4956 , piece_id=FFHUPOOM7PLEABRHFH6H2SI6WXJBU2XFCLZJUBV4CXUJ6EP4PI2Q, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TYHhjpfyBKkenSwviyQCUZpSBbnaFCGBFRkM4fyKLDuxKx4Ju): protocol: expected piece hash; context canceled; EOF Dec 28 19:40:08.636342175 (556h11m8.50742521s) [noise=true , address=64.49.13.26:4956 , piece_id=FFHUPOOM7PLEABRHFH6H2SI6WXJBU2XFCLZJUBV4CXUJ6EP4PI2Q, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636585959 (556h11m8.507668994s) [noise=true , address=145.79.227.43:4163 , piece_id=77BKLLVD25UMTDQZW46CA2N225I7O4HCHCRRPQ7HLJGRHOLGSDHA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CgPQ5hkrJdqbJvGarxKjXvYYnrXQqapL5BH2rNJNvnhpRi3pw): protocol: expected piece hash; context canceled; EOF Dec 28 19:40:08.636592172 (556h11m8.507675207s) [noise=true , address=145.79.227.43:4163 , piece_id=77BKLLVD25UMTDQZW46CA2N225I7O4HCHCRRPQ7HLJGRHOLGSDHA, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636681511 (556h11m8.507764546s) [noise=true , address=145.79.227.19:4083 , piece_id=GL5XL7T2QQABD5364AT4AFV6RZNOWCTVNSMAVIFJU5SUQFHOPZKQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KZ5a6CQ9tUTrC6EYE7M8yg6p9fK95CN8zQLQmP4tdyz4rBfGh): protocol: expected piece hash; context canceled; EOF Dec 28 19:40:08.636687052 (556h11m8.507770087s) [noise=true , address=145.79.227.19:4083 , piece_id=GL5XL7T2QQABD5364AT4AFV6RZNOWCTVNSMAVIFJU5SUQFHOPZKQ, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636810327 (556h11m8.507893362s) [noise=true , address=145.79.227.17:4992 , piece_id=GUF7AUFCFRRPY5PSYS7DRPFQLGT4ZEL5C7P23US2VI54KYWSUCUQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hbt8HKTunpPKyAzH4XBwGQqTKzcgqPpoWwKZm9JMtaK8P7aVp): protocol: write tcp 188.166.225.170:51210->145.79.227.17:4992: use of closed network connection; write tcp 188.166.225.170:51210->145.79.227.17:4992: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51210->145.79.227.17:4992: use of closed network connection Dec 28 19:40:08.636815957 (556h11m8.507899002s) [noise=true , address=145.79.227.17:4992 , piece_id=GUF7AUFCFRRPY5PSYS7DRPFQLGT4ZEL5C7P23US2VI54KYWSUCUQ, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636834313 (556h11m8.507917358s) [seg_pos={0 0}, upload=487609]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 19:40:08.636907953 (556h11m8.507990998s) [upload=487609]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 19:40:08.636913644 (556h11m8.507996689s) [upload=487609]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 19:40:08.924991397 (556h11m8.796074442s) [upload=487609]: Done uploading segments... Dec 28 19:40:08.925014742 (556h11m8.796097777s) [upload=487609]: Upload finished. err: <nil> Duration: 15.233968887s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 13.367580774s trace: https://jaeger.dev.storj.io/trace/10e1641b228b2a1f FAIL (2025-12-28 20:09:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 40:05.744568479 (556h11m5.615651514s) [noise=true , address=145.79.227.28:4744 , piece_id=42LII6EOG3CMBTIL5I4RZQPDYU52TKJN3PGWEQXROBYMNDFYIIVA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.745099351 (556h11m5.616182396s) [noise=true , address=145.79.227.23:4550 , piece_id=NWIC722LZUHJSAMPWSKCLPRIDGWPVUEYMS5KRAMX3IIMD2KR4XGQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.745777401 (556h11m5.616860436s) [noise=true , address=77.90.31.15:4722 , piece_id=AF7F2OA247CJIDFTEU6HEOIYGOINEYPMII53BKJQXI7UZ26MJIUA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.746374739 (556h11m5.617457784s) [noise=true , address=77.90.31.10:4522 , piece_id=JOHOMHCNYXF4XQGLN2NJUSWSUO3XFBRRMYD27SLIZAK2RCT3ZC2A, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.746934555 (556h11m5.618017601s) [noise=true , address=77.90.31.21:4896 , piece_id=ATTC7BZUT4HAAEAZCHFZQE4INV2LLSWR2YCUNVA73LPJCW3YL35A, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.747556861 (556h11m5.618640146s) [noise=true , address=64.49.13.26:4956 , piece_id=FFHUPOOM7PLEABRHFH6H2SI6WXJBU2XFCLZJUBV4CXUJ6EP4PI2Q, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.748964150 (556h11m5.620047195s) [noise=true , address=145.79.227.34:4916 , piece_id=SPYSFWMJ752FC5JGRE36G54S2FHG4DQFCV6CIZONQHJDXDCN52VQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.751119314 (556h11m5.622202359s) [noise=true , address=77.90.31.18:4816 , piece_id=YT3PSYNQTDQBVBRUDNOXJ2XCN6EFQA54WBSNL6MZMXFXR7XNCHIA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.753413222 (556h11m5.624496257s) [noise=true , address=77.90.31.11:4582 , piece_id=ZRE24R27TX4E5DTV2POX7EJANQ4CM75D6ZOUS6SKM3QHHQCKRBRA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.755230482 (556h11m5.626313527s) [noise=true , address=64.49.13.154:4932 , piece_id=QB5WM5RZ2P2GC6RIGGM34IANVEI2SNDJ2PO556JXKTDBX5HLD4YA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.757598362 (556h11m5.628681407s) [noise=true , address=145.79.227.35:4510 , piece_id=P275TU367DTYXBQOGGV5THFUPHJUWGSM5IMJQ753VE4MX4IPOONQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.759644679 (556h11m5.630727734s) [noise=true , address=145.79.227.46:4626 , piece_id=OQOTWYLVIU7GJBBLYWQOEEFKR3TNAUOXZS6UAFJWJ32SIOTXNXEA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.761217594 (556h11m5.632300639s) [noise=true , address=145.79.227.43:4163 , piece_id=77BKLLVD25UMTDQZW46CA2N225I7O4HCHCRRPQ7HLJGRHOLGSDHA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.762138507 (556h11m5.633221552s) [noise=true , address=77.90.31.19:4860 , piece_id=5FUYVK2Z6LBZOAGFI2NOTAAUIV4NMBI5ERMA5RH4BMONNHABZLDQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.764304321 (556h11m5.635387356s) [noise=true , address=145.79.227.30:4784 , piece_id=MKCRSFWVFVGOAYCK5M66HO3OT7OMYWLVR2KMDXUGBS2VXXNCUSSQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.765421217 (556h11m5.636504252s) [noise=true , address=145.79.227.44:4216 , piece_id=GSBWC2LXUOZYOVD55KNQYA5Z3S2TCYROBB5O2XZIIVFUQMJQMYNA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:07.082099826 (556h11m6.953182871s) [upload=487609]: Next returned nil segment Dec 28 19:40:07.082116707 (556h11m6.953199752s) [upload=487609]: Waiting for error group managing segments... Dec 28 19:40:07.415884144 (556h11m7.286967179s) [noise=true , address=64.49.13.154:4932 , piece_id=QB5WM5RZ2P2GC6RIGGM34IANVEI2SNDJ2PO556JXKTDBX5HLD4YA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.615434252 (556h11m7.486517297s) [noise=true , address=145.79.227.23:4550 , piece_id=NWIC722LZUHJSAMPWSKCLPRIDGWPVUEYMS5KRAMX3IIMD2KR4XGQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.647182077 (556h11m7.518265112s) [noise=true , address=145.79.227.28:4744 , piece_id=42LII6EOG3CMBTIL5I4RZQPDYU52TKJN3PGWEQXROBYMNDFYIIVA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.678458624 (556h11m7.549541659s) [noise=true , address=145.79.227.46:4626 , piece_id=OQOTWYLVIU7GJBBLYWQOEEFKR3TNAUOXZS6UAFJWJ32SIOTXNXEA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.793406049 (556h11m7.664489084s) [noise=true , address=77.90.31.19:4860 , piece_id=5FUYVK2Z6LBZOAGFI2NOTAAUIV4NMBI5ERMA5RH4BMONNHABZLDQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.802506498 (556h11m7.673589543s) [noise=true , address=77.90.31.21:4896 , piece_id=ATTC7BZUT4HAAEAZCHFZQE4INV2LLSWR2YCUNVA73LPJCW3YL35A, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.853663242 (556h11m7.724746287s) [noise=true , address=77.90.31.10:4522 , piece_id=JOHOMHCNYXF4XQGLN2NJUSWSUO3XFBRRMYD27SLIZAK2RCT3ZC2A, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.880639500 (556h11m7.751722535s) [noise=true , address=77.90.31.11:4582 , piece_id=ZRE24R27TX4E5DTV2POX7EJANQ4CM75D6ZOUS6SKM3QHHQCKRBRA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.904835446 (556h11m7.775918491s) [noise=true , address=77.90.31.18:4816 , piece_id=YT3PSYNQTDQBVBRUDNOXJ2XCN6EFQA54WBSNL6MZMXFXR7XNCHIA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.988914260 (556h11m7.859997295s) [noise=true , address=77.90.31.15:4722 , piece_id=AF7F2OA247CJIDFTEU6HEOIYGOINEYPMII53BKJQXI7UZ26MJIUA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.484927449 (556h11m8.356010484s) [noise=true , address=145.79.227.35:4510 , piece_id=P275TU367DTYXBQOGGV5THFUPHJUWGSM5IMJQ753VE4MX4IPOONQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.610317849 (556h11m8.481400894s) [noise=true , address=145.79.227.44:4216 , piece_id=GSBWC2LXUOZYOVD55KNQYA5Z3S2TCYROBB5O2XZIIVFUQMJQMYNA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.611969885 (556h11m8.48305292s) [noise=true , address=145.79.227.34:4916 , piece_id=SPYSFWMJ752FC5JGRE36G54S2FHG4DQFCV6CIZONQHJDXDCN52VQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.636098801 (556h11m8.507181836s) [noise=true , address=145.79.227.30:4784 , piece_id=MKCRSFWVFVGOAYCK5M66HO3OT7OMYWLVR2KMDXUGBS2VXXNCUSSQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.636122136 (556h11m8.507205181s) [seg_pos={0 0}, upload=487609]: Segment reached optimal threshold of 14 pieces. Dec 28 19:40:08.636333929 (556h11m8.507416974s) [noise=true , address=64.49.13.26:4956 , piece_id=FFHUPOOM7PLEABRHFH6H2SI6WXJBU2XFCLZJUBV4CXUJ6EP4PI2Q, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TYHhjpfyBKkenSwviyQCUZpSBbnaFCGBFRkM4fyKLDuxKx4Ju): protocol: expected piece hash; context canceled; EOF Dec 28 19:40:08.636342175 (556h11m8.50742521s) [noise=true , address=64.49.13.26:4956 , piece_id=FFHUPOOM7PLEABRHFH6H2SI6WXJBU2XFCLZJUBV4CXUJ6EP4PI2Q, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636585959 (556h11m8.507668994s) [noise=true , address=145.79.227.43:4163 , piece_id=77BKLLVD25UMTDQZW46CA2N225I7O4HCHCRRPQ7HLJGRHOLGSDHA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CgPQ5hkrJdqbJvGarxKjXvYYnrXQqapL5BH2rNJNvnhpRi3pw): protocol: expected piece hash; context canceled; EOF Dec 28 19:40:08.636592172 (556h11m8.507675207s) [noise=true , address=145.79.227.43:4163 , piece_id=77BKLLVD25UMTDQZW46CA2N225I7O4HCHCRRPQ7HLJGRHOLGSDHA, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636681511 (556h11m8.507764546s) [noise=true , address=145.79.227.19:4083 , piece_id=GL5XL7T2QQABD5364AT4AFV6RZNOWCTVNSMAVIFJU5SUQFHOPZKQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KZ5a6CQ9tUTrC6EYE7M8yg6p9fK95CN8zQLQmP4tdyz4rBfGh): protocol: expected piece hash; context canceled; EOF Dec 28 19:40:08.636687052 (556h11m8.507770087s) [noise=true , address=145.79.227.19:4083 , piece_id=GL5XL7T2QQABD5364AT4AFV6RZNOWCTVNSMAVIFJU5SUQFHOPZKQ, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636810327 (556h11m8.507893362s) [noise=true , address=145.79.227.17:4992 , piece_id=GUF7AUFCFRRPY5PSYS7DRPFQLGT4ZEL5C7P23US2VI54KYWSUCUQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hbt8HKTunpPKyAzH4XBwGQqTKzcgqPpoWwKZm9JMtaK8P7aVp): protocol: write tcp 188.166.225.170:51210->145.79.227.17:4992: use of closed network connection; write tcp 188.166.225.170:51210->145.79.227.17:4992: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51210->145.79.227.17:4992: use of closed network connection Dec 28 19:40:08.636815957 (556h11m8.507899002s) [noise=true , address=145.79.227.17:4992 , piece_id=GUF7AUFCFRRPY5PSYS7DRPFQLGT4ZEL5C7P23US2VI54KYWSUCUQ, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636834313 (556h11m8.507917358s) [seg_pos={0 0}, upload=487609]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 19:40:08.636907953 (556h11m8.507990998s) [upload=487609]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 19:40:08.636913644 (556h11m8.507996689s) [upload=487609]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 19:40:08.924991397 (556h11m8.796074442s) [upload=487609]: Done uploading segments... Dec 28 19:40:08.925014742 (556h11m8.796097777s) [upload=487609]: Upload finished. err: <nil> Duration: 13.36760546s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [XX....XXX..X..X......XX..............X......................] duration: 5.81754628s trace: https://jaeger.dev.storj.io/trace/6de23ce395480be2 FAIL (2025-12-28 20:09:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: VL3CQZWQBCPGDPMEAGIQCCHVRPMV36BXHPHJ2LPJLIC6LDGWR5YA) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: NYOXGWDJWHYHW6BY3DQHFGISYSYQV4JT2H2XDJ4WLMUNE2IHN6LA) context deadline exceeded; piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: BQE24O2CURUFHQS4WSIG722BGBPCPOY32W6KAOECGV7JPCKWGOKA) context deadline exceeded ======LOGS====== Dec 28 19:51:31.138959699 (556h22m31.010042744s) [upload=487778]: Starting upload Dec 28 19:51:31.139044411 (556h22m31.010127456s) [upload=487778]: Done starting upload Dec 28 19:51:31.139127078 (556h22m31.010210113s) [upload=487778]: Uploading segments... Dec 28 19:51:31.139326739 (556h22m31.010409784s) [upload=487778]: Got next segment. Inline: false Dec 28 19:51:31.139377555 (556h22m31.0104606s) [upload=487778]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 19:51:31.139384197 (556h22m31.010467232s) [upload=487778]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 19:51:31.387475461 (556h22m31.258558496s) [seg_pos={0 0}, upload=487778]: Begin upload segment... Dec 28 19:51:31.387544984 (556h22m31.258628049s) [seg_pos={0 0}, upload=487778]: Done begin upload segment. Dec 28 19:51:31.387771626 (556h22m31.258854671s) [noise=true , address=45.91.116.7:4011 , piece_id=NQDPXLIESHRAWWBPVYOGPARTQ6PDN3TO7YCWTLXI4KBFD7JZFXXQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.388814712 (556h22m31.259897747s) [noise=true , address=77.90.31.16:4735 , piece_id=EBCUZV26NTWYQFRJEZQE3N5EU2FPNPTQRCEPR6FNK5YTRZRORP2Q, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.389204404 (556h22m31.260287449s) [noise=true , address=145.79.227.9:4366 , piece_id=PC3TE34FAHG6J6TWTEU7GKWFGP4DDUELNIT2RIAEKGTM6XDRWIZA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.389706850 (556h22m31.260789885s) [noise=true , address=145.79.227.21:4475 , piece_id=BBVRNDFL25BI3Q2CNE36E5GRAOXT55T4MQA2HTXX257HPKZ6HSPQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.390109167 (556h22m31.261192212s) [noise=true , address=145.79.227.51:4411 , piece_id=LICH4YP46TD6FZHHN4X4OFWKYBPPCFWSGHU63E2WP4AMD6HFZDNA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.390421501 (556h22m31.261504546s) [noise=true , address=77.90.31.10:4519 , piece_id=VL3CQZWQBCPGDPMEAGIQCCHVRPMV36BXHPHJ2LPJLIC6LDGWR5YA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.390852582 (556h22m31.261935617s) [noise=true , address=145.79.227.5:4221 , piece_id=O73W2GAUHPHLTWZTO7IJT4OBBXAX22MAABQ2MQSTZPNLPIQXHKRQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.392756678 (556h22m31.263839713s) [noise=true , address=45.91.116.6:4974 , piece_id=FSPEWZOEC4SHIHA6IJSXCHUCOL6GPFRDFUFXJPCQR7263Y2YFIMQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.393255988 (556h22m31.264339023s) [noise=true , address=145.79.227.2:4113 , piece_id=QJTHNBJ6SMI7GHV5634NHXQUD2P4TXERE77FO4ECEVYHQ6GUQ3FQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.393633607 (556h22m31.264716642s) [noise=true , address=145.79.227.41:4087 , piece_id=SILOBZTKKG4QOW6F2XRCEXG64GCTMC2ULONWTO2D3QFFL74FP5BQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.394230243 (556h22m31.265313278s) [noise=true , address=145.79.227.47:4265 , piece_id=43D7BZZXISPRFCKYXSTCZONLHLQ2Y6IRLFD7PDEUAJWYP6ID7DMQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.394697414 (556h22m31.265780449s) [noise=true , address=145.79.227.39:4013 , piece_id=PWYMH2SD6JOHCSM7HFDM4CHUD5C3MUCSCRML3BMZKHDZAFRM6UMA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.401284737 (556h22m31.272367792s) [noise=true , address=145.79.227.45:4229 , piece_id=AZOSHGLV2UUTXSX5NX27R3ZFIEOWS2CDGLSG34JZKCVWY4DLHUHQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.402581947 (556h22m31.273664982s) [noise=true , address=77.90.31.14:4661 , piece_id=XLUH6WZPNM2W67MLQ7KFKGM3Y4WL4TFJ6GSYKPX325FSDPKZCUUA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.403710777 (556h22m31.274793822s) [noise=true , address=45.91.116.5:4300 , piece_id=NYOXGWDJWHYHW6BY3DQHFGISYSYQV4JT2H2XDJ4WLMUNE2IHN6LA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.404110447 (556h22m31.275193492s) [noise=true , address=45.91.116.9:4261 , piece_id=BQE24O2CURUFHQS4WSIG722BGBPCPOY32W6KAOECGV7JPCKWGOKA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:33.086369496 (556h22m32.957452531s) [upload=487778]: Next returned nil segment Dec 28 19:51:33.086389915 (556h22m32.95747296s) [upload=487778]: Waiting for error group managing segments... Dec 28 19:51:33.394411997 (556h22m33.265495032s) [noise=true , address=145.79.227.21:4475 , piece_id=BBVRNDFL25BI3Q2CNE36E5GRAOXT55T4MQA2HTXX257HPKZ6HSPQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.398162920 (556h22m33.269245955s) [noise=true , address=145.79.227.39:4013 , piece_id=PWYMH2SD6JOHCSM7HFDM4CHUD5C3MUCSCRML3BMZKHDZAFRM6UMA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.403651411 (556h22m33.274734446s) [noise=true , address=145.79.227.45:4229 , piece_id=AZOSHGLV2UUTXSX5NX27R3ZFIEOWS2CDGLSG34JZKCVWY4DLHUHQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.406027926 (556h22m33.277110971s) [noise=true , address=145.79.227.51:4411 , piece_id=LICH4YP46TD6FZHHN4X4OFWKYBPPCFWSGHU63E2WP4AMD6HFZDNA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.418392551 (556h22m33.289475586s) [noise=true , address=145.79.227.47:4265 , piece_id=43D7BZZXISPRFCKYXSTCZONLHLQ2Y6IRLFD7PDEUAJWYP6ID7DMQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.420875018 (556h22m33.291958063s) [noise=true , address=145.79.227.41:4087 , piece_id=SILOBZTKKG4QOW6F2XRCEXG64GCTMC2ULONWTO2D3QFFL74FP5BQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.421544332 (556h22m33.292627377s) [noise=true , address=145.79.227.2:4113 , piece_id=QJTHNBJ6SMI7GHV5634NHXQUD2P4TXERE77FO4ECEVYHQ6GUQ3FQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.427849308 (556h22m33.298932353s) [noise=true , address=145.79.227.9:4366 , piece_id=PC3TE34FAHG6J6TWTEU7GKWFGP4DDUELNIT2RIAEKGTM6XDRWIZA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.649006914 (556h22m33.520089949s) [noise=true , address=45.91.116.9:4261 , piece_id=BQE24O2CURUFHQS4WSIG722BGBPCPOY32W6KAOECGV7JPCKWGOKA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.688630355 (556h22m33.55971339s) [noise=true , address=77.90.31.14:4661 , piece_id=XLUH6WZPNM2W67MLQ7KFKGM3Y4WL4TFJ6GSYKPX325FSDPKZCUUA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.694890445 (556h22m33.56597348s) [noise=true , address=45.91.116.5:4300 , piece_id=NYOXGWDJWHYHW6BY3DQHFGISYSYQV4JT2H2XDJ4WLMUNE2IHN6LA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.738432388 (556h22m33.609515433s) [noise=true , address=45.91.116.7:4011 , piece_id=NQDPXLIESHRAWWBPVYOGPARTQ6PDN3TO7YCWTLXI4KBFD7JZFXXQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:34.314051275 (556h22m34.18513431s) [noise=true , address=77.90.31.10:4519 , piece_id=VL3CQZWQBCPGDPMEAGIQCCHVRPMV36BXHPHJ2LPJLIC6LDGWR5YA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:34.314075711 (556h22m34.185158746s) [seg_pos={0 0}, upload=487778]: Segment reached optimal threshold of 13 pieces. Dec 28 19:51:34.314354963 (556h22m34.185437998s) [noise=true , address=45.91.116.6:4974 , piece_id=FSPEWZOEC4SHIHA6IJSXCHUCOL6GPFRDFUFXJPCQR7263Y2YFIMQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Dec 28 19:51:34.314365733 (556h22m34.185448778s) [noise=true , address=45.91.116.6:4974 , piece_id=FSPEWZOEC4SHIHA6IJSXCHUCOL6GPFRDFUFXJPCQR7263Y2YFIMQ, seg_pos={0 0}, upload=487778]: Long tail cancellation due to optimal threshold Dec 28 19:51:34.314429585 (556h22m34.18551263s) [noise=true , address=145.79.227.5:4221 , piece_id=O73W2GAUHPHLTWZTO7IJT4OBBXAX22MAABQ2MQSTZPNLPIQXHKRQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR): protocol: expected piece hash; context canceled; EOF Dec 28 19:51:34.314451267 (556h22m34.185534302s) [noise=true , address=145.79.227.5:4221 , piece_id=O73W2GAUHPHLTWZTO7IJT4OBBXAX22MAABQ2MQSTZPNLPIQXHKRQ, seg_pos={0 0}, upload=487778]: Long tail cancellation due to optimal threshold Dec 28 19:51:34.314521831 (556h22m34.185604866s) [noise=true , address=77.90.31.16:4735 , piece_id=EBCUZV26NTWYQFRJEZQE3N5EU2FPNPTQRCEPR6FNK5YTRZRORP2Q, seg_pos={0 0}, upload=487778]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 28 19:51:34.314530057 (556h22m34.185613092s) [noise=true , address=77.90.31.16:4735 , piece_id=EBCUZV26NTWYQFRJEZQE3N5EU2FPNPTQRCEPR6FNK5YTRZRORP2Q, seg_pos={0 0}, upload=487778]: Long tail cancellation due to optimal threshold Dec 28 19:51:34.314593758 (556h22m34.185676793s) [seg_pos={0 0}, upload=487778]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 19:51:34.314673079 (556h22m34.185756124s) [upload=487778]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 19:51:34.314678910 (556h22m34.185761965s) [upload=487778]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 19:51:34.609917501 (556h22m34.481000566s) [upload=487778]: Done uploading segments... Dec 28 19:51:34.609974519 (556h22m34.481057594s) [upload=487778]: Upload finished. err: <nil> Duration: 5.817575816s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.XX....XXX..X..X......XX..............X.....................] duration: 6.042685567s trace: https://jaeger.dev.storj.io/trace/6de23ce395480be2 FAIL (2025-12-28 20:08:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: VL3CQZWQBCPGDPMEAGIQCCHVRPMV36BXHPHJ2LPJLIC6LDGWR5YA) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: NYOXGWDJWHYHW6BY3DQHFGISYSYQV4JT2H2XDJ4WLMUNE2IHN6LA) context deadline exceeded; piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: BQE24O2CURUFHQS4WSIG722BGBPCPOY32W6KAOECGV7JPCKWGOKA) context deadline exceeded ======LOGS====== Dec 28 19:51:31.138959699 (556h22m31.010042744s) [upload=487778]: Starting upload Dec 28 19:51:31.139044411 (556h22m31.010127456s) [upload=487778]: Done starting upload Dec 28 19:51:31.139127078 (556h22m31.010210113s) [upload=487778]: Uploading segments... Dec 28 19:51:31.139326739 (556h22m31.010409784s) [upload=487778]: Got next segment. Inline: false Dec 28 19:51:31.139377555 (556h22m31.0104606s) [upload=487778]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 19:51:31.139384197 (556h22m31.010467232s) [upload=487778]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 19:51:31.387475461 (556h22m31.258558496s) [seg_pos={0 0}, upload=487778]: Begin upload segment... Dec 28 19:51:31.387544984 (556h22m31.258628049s) [seg_pos={0 0}, upload=487778]: Done begin upload segment. Dec 28 19:51:31.387771626 (556h22m31.258854671s) [noise=true , address=45.91.116.7:4011 , piece_id=NQDPXLIESHRAWWBPVYOGPARTQ6PDN3TO7YCWTLXI4KBFD7JZFXXQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.388814712 (556h22m31.259897747s) [noise=true , address=77.90.31.16:4735 , piece_id=EBCUZV26NTWYQFRJEZQE3N5EU2FPNPTQRCEPR6FNK5YTRZRORP2Q, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.389204404 (556h22m31.260287449s) [noise=true , address=145.79.227.9:4366 , piece_id=PC3TE34FAHG6J6TWTEU7GKWFGP4DDUELNIT2RIAEKGTM6XDRWIZA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.389706850 (556h22m31.260789885s) [noise=true , address=145.79.227.21:4475 , piece_id=BBVRNDFL25BI3Q2CNE36E5GRAOXT55T4MQA2HTXX257HPKZ6HSPQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.390109167 (556h22m31.261192212s) [noise=true , address=145.79.227.51:4411 , piece_id=LICH4YP46TD6FZHHN4X4OFWKYBPPCFWSGHU63E2WP4AMD6HFZDNA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.390421501 (556h22m31.261504546s) [noise=true , address=77.90.31.10:4519 , piece_id=VL3CQZWQBCPGDPMEAGIQCCHVRPMV36BXHPHJ2LPJLIC6LDGWR5YA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.390852582 (556h22m31.261935617s) [noise=true , address=145.79.227.5:4221 , piece_id=O73W2GAUHPHLTWZTO7IJT4OBBXAX22MAABQ2MQSTZPNLPIQXHKRQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.392756678 (556h22m31.263839713s) [noise=true , address=45.91.116.6:4974 , piece_id=FSPEWZOEC4SHIHA6IJSXCHUCOL6GPFRDFUFXJPCQR7263Y2YFIMQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.393255988 (556h22m31.264339023s) [noise=true , address=145.79.227.2:4113 , piece_id=QJTHNBJ6SMI7GHV5634NHXQUD2P4TXERE77FO4ECEVYHQ6GUQ3FQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.393633607 (556h22m31.264716642s) [noise=true , address=145.79.227.41:4087 , piece_id=SILOBZTKKG4QOW6F2XRCEXG64GCTMC2ULONWTO2D3QFFL74FP5BQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.394230243 (556h22m31.265313278s) [noise=true , address=145.79.227.47:4265 , piece_id=43D7BZZXISPRFCKYXSTCZONLHLQ2Y6IRLFD7PDEUAJWYP6ID7DMQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.394697414 (556h22m31.265780449s) [noise=true , address=145.79.227.39:4013 , piece_id=PWYMH2SD6JOHCSM7HFDM4CHUD5C3MUCSCRML3BMZKHDZAFRM6UMA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.401284737 (556h22m31.272367792s) [noise=true , address=145.79.227.45:4229 , piece_id=AZOSHGLV2UUTXSX5NX27R3ZFIEOWS2CDGLSG34JZKCVWY4DLHUHQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.402581947 (556h22m31.273664982s) [noise=true , address=77.90.31.14:4661 , piece_id=XLUH6WZPNM2W67MLQ7KFKGM3Y4WL4TFJ6GSYKPX325FSDPKZCUUA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.403710777 (556h22m31.274793822s) [noise=true , address=45.91.116.5:4300 , piece_id=NYOXGWDJWHYHW6BY3DQHFGISYSYQV4JT2H2XDJ4WLMUNE2IHN6LA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.404110447 (556h22m31.275193492s) [noise=true , address=45.91.116.9:4261 , piece_id=BQE24O2CURUFHQS4WSIG722BGBPCPOY32W6KAOECGV7JPCKWGOKA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:33.086369496 (556h22m32.957452531s) [upload=487778]: Next returned nil segment Dec 28 19:51:33.086389915 (556h22m32.95747296s) [upload=487778]: Waiting for error group managing segments... Dec 28 19:51:33.394411997 (556h22m33.265495032s) [noise=true , address=145.79.227.21:4475 , piece_id=BBVRNDFL25BI3Q2CNE36E5GRAOXT55T4MQA2HTXX257HPKZ6HSPQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.398162920 (556h22m33.269245955s) [noise=true , address=145.79.227.39:4013 , piece_id=PWYMH2SD6JOHCSM7HFDM4CHUD5C3MUCSCRML3BMZKHDZAFRM6UMA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.403651411 (556h22m33.274734446s) [noise=true , address=145.79.227.45:4229 , piece_id=AZOSHGLV2UUTXSX5NX27R3ZFIEOWS2CDGLSG34JZKCVWY4DLHUHQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.406027926 (556h22m33.277110971s) [noise=true , address=145.79.227.51:4411 , piece_id=LICH4YP46TD6FZHHN4X4OFWKYBPPCFWSGHU63E2WP4AMD6HFZDNA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.418392551 (556h22m33.289475586s) [noise=true , address=145.79.227.47:4265 , piece_id=43D7BZZXISPRFCKYXSTCZONLHLQ2Y6IRLFD7PDEUAJWYP6ID7DMQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.420875018 (556h22m33.291958063s) [noise=true , address=145.79.227.41:4087 , piece_id=SILOBZTKKG4QOW6F2XRCEXG64GCTMC2ULONWTO2D3QFFL74FP5BQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.421544332 (556h22m33.292627377s) [noise=true , address=145.79.227.2:4113 , piece_id=QJTHNBJ6SMI7GHV5634NHXQUD2P4TXERE77FO4ECEVYHQ6GUQ3FQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.427849308 (556h22m33.298932353s) [noise=true , address=145.79.227.9:4366 , piece_id=PC3TE34FAHG6J6TWTEU7GKWFGP4DDUELNIT2RIAEKGTM6XDRWIZA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.649006914 (556h22m33.520089949s) [noise=true , address=45.91.116.9:4261 , piece_id=BQE24O2CURUFHQS4WSIG722BGBPCPOY32W6KAOECGV7JPCKWGOKA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.688630355 (556h22m33.55971339s) [noise=true , address=77.90.31.14:4661 , piece_id=XLUH6WZPNM2W67MLQ7KFKGM3Y4WL4TFJ6GSYKPX325FSDPKZCUUA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.694890445 (556h22m33.56597348s) [noise=true , address=45.91.116.5:4300 , piece_id=NYOXGWDJWHYHW6BY3DQHFGISYSYQV4JT2H2XDJ4WLMUNE2IHN6LA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.738432388 (556h22m33.609515433s) [noise=true , address=45.91.116.7:4011 , piece_id=NQDPXLIESHRAWWBPVYOGPARTQ6PDN3TO7YCWTLXI4KBFD7JZFXXQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:34.314051275 (556h22m34.18513431s) [noise=true , address=77.90.31.10:4519 , piece_id=VL3CQZWQBCPGDPMEAGIQCCHVRPMV36BXHPHJ2LPJLIC6LDGWR5YA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:34.314075711 (556h22m34.185158746s) [seg_pos={0 0}, upload=487778]: Segment reached optimal threshold of 13 pieces. Dec 28 19:51:34.314354963 (556h22m34.185437998s) [noise=true , address=45.91.116.6:4974 , piece_id=FSPEWZOEC4SHIHA6IJSXCHUCOL6GPFRDFUFXJPCQR7263Y2YFIMQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Dec 28 19:51:34.314365733 (556h22m34.185448778s) [noise=true , address=45.91.116.6:4974 , piece_id=FSPEWZOEC4SHIHA6IJSXCHUCOL6GPFRDFUFXJPCQR7263Y2YFIMQ, seg_pos={0 0}, upload=487778]: Long tail cancellation due to optimal threshold Dec 28 19:51:34.314429585 (556h22m34.18551263s) [noise=true , address=145.79.227.5:4221 , piece_id=O73W2GAUHPHLTWZTO7IJT4OBBXAX22MAABQ2MQSTZPNLPIQXHKRQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR): protocol: expected piece hash; context canceled; EOF Dec 28 19:51:34.314451267 (556h22m34.185534302s) [noise=true , address=145.79.227.5:4221 , piece_id=O73W2GAUHPHLTWZTO7IJT4OBBXAX22MAABQ2MQSTZPNLPIQXHKRQ, seg_pos={0 0}, upload=487778]: Long tail cancellation due to optimal threshold Dec 28 19:51:34.314521831 (556h22m34.185604866s) [noise=true , address=77.90.31.16:4735 , piece_id=EBCUZV26NTWYQFRJEZQE3N5EU2FPNPTQRCEPR6FNK5YTRZRORP2Q, seg_pos={0 0}, upload=487778]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 28 19:51:34.314530057 (556h22m34.185613092s) [noise=true , address=77.90.31.16:4735 , piece_id=EBCUZV26NTWYQFRJEZQE3N5EU2FPNPTQRCEPR6FNK5YTRZRORP2Q, seg_pos={0 0}, upload=487778]: Long tail cancellation due to optimal threshold Dec 28 19:51:34.314593758 (556h22m34.185676793s) [seg_pos={0 0}, upload=487778]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 19:51:34.314673079 (556h22m34.185756124s) [upload=487778]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 19:51:34.314678910 (556h22m34.185761965s) [upload=487778]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 19:51:34.609917501 (556h22m34.481000566s) [upload=487778]: Done uploading segments... Dec 28 19:51:34.609974519 (556h22m34.481057594s) [upload=487778]: Upload finished. err: <nil> Duration: 6.042709633s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)