Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 12.721528288s trace: https://jaeger.dev.storj.io/trace/499b96822b8dda80 FAIL (2025-12-28 20:56:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12S7WBBQ1Nq8jbeGrZiRKTk23CdJC31hPW2vU1aaM5SsHeVyeRH, Piece ID: V4P3KCUDLI543TSROBYUMVTQKKSQDDVSKSJSXKEZTQAHTXUJ2G4Q) context deadline exceeded; piecestore: (Node ID: 1kjCEPtjvcJ3D7BBKRSRirPybBwkRZ1dJwyAQCadzCvaLYvWhz, Piece ID: 4OULHJ6Z3QJ3BOPDY74TJSKVJG23TOEUCJCUEGY24I5C3MUTAVLQ) context deadline exceeded; piecestore: (Node ID: 1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c, Piece ID: LIYXCPAZOUPKXLKRH3EOWREXUY2LAJXSRZFQILM2QIU7HTW7SY2Q) context deadline exceeded ======LOGS====== ue , address=145.79.227.23:4580 , piece_id=OHYI2SFM6TKJE4LHVNGO2PVGZDNFIQDAIQGKG4FOEHTTASBWA5ZQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.496428825 (557h3m1.36751187s) [noise=true , address=77.90.31.19:4867 , piece_id=LDAWD2ZFG3BZATDRY7MMNWYIX6TOHQ2OIUPCAWAYN266GNUUKZ5A, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.496861509 (557h3m1.367944544s) [noise=true , address=145.79.227.19:4105 , piece_id=RY35FSM6NTWPNF3JE43FZRSNXXYZPUN2B6LRNNUOG6QQR6XSU36Q, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.496899933 (557h3m1.367982978s) [noise=true , address=77.90.31.19:4858 , piece_id=LIYXCPAZOUPKXLKRH3EOWREXUY2LAJXSRZFQILM2QIU7HTW7SY2Q, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497229089 (557h3m1.368312134s) [noise=true , address=77.90.31.17:4784 , piece_id=JV2ROD6MN65B6LZKOL3S7ATJWXAY6FBQSZWITA6IHAXELY7R42KA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497326194 (557h3m1.368409239s) [noise=true , address=145.79.227.33:4904 , piece_id=QCMN4OSBWIWS4JRLIGQIGDSHZFUF2JWEPVXQ4FQ4G3LG5FBCBAWQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497664128 (557h3m1.368747173s) [noise=true , address=145.79.227.34:4918 , piece_id=SPDT2KI2774UJMQIUSVTHXCDF7PD2FBG64BXUYW67UMLPE6H7SBQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497731405 (557h3m1.36881445s) [noise=true , address=77.90.31.12:4602 , piece_id=2YPYKKTQ6ID6VTI4AUZQEG2SY7JYPRSAPU4JLF7DWSOSBBO74LWQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.498205939 (557h3m1.369288984s) [noise=true , address=77.90.31.16:4745 , piece_id=CZBFU33BHMP2BEZIDJKW3J526537HR6HP7OCT2KMAQVY7LGZLPZQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.498424295 (557h3m1.36950734s) [noise=true , address=77.90.31.21:4895 , piece_id=V4P3KCUDLI543TSROBYUMVTQKKSQDDVSKSJSXKEZTQAHTXUJ2G4Q, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.498685422 (557h3m1.369768457s) [noise=true , address=145.79.227.35:4496 , piece_id=JH4XQBT7XYEJXWBGPPF6SJWCPLV77GBOLGOA53C3WUQMWHZXVZQA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499066348 (557h3m1.370149393s) [noise=true , address=77.90.31.11:4563 , piece_id=DIFJPHSKHROTLKCHP6HQBWT7XGE77HVRZTH2354MV2UEWZLHVGCQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499185725 (557h3m1.37026877s) [noise=true , address=145.79.227.30:4768 , piece_id=KK4LEV6RG3IGLYXID5MJF3KMNSJ7WS5JS45PNOOJ26LXLYAUCWOA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499468122 (557h3m1.370551157s) [noise=true , address=77.90.31.14:4688 , piece_id=4OULHJ6Z3QJ3BOPDY74TJSKVJG23TOEUCJCUEGY24I5C3MUTAVLQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499901047 (557h3m1.370984102s) [noise=true , address=45.91.116.5:4330 , piece_id=EVHLSVVR5JGY43TLSKVEQLQ6KDBEZENVAFL6YT7ZQ77Y7X2TXIKA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.500546175 (557h3m1.37162921s) [noise=true , address=145.79.227.43:4171 , piece_id=YF5DRDPGRRVYFL5SWMH2PKTSV3C72BIWKUDSAIZTXOIVOTYKRMNA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:02.979572381 (557h3m2.850655436s) [upload=488362]: Next returned nil segment Dec 28 20:32:02.979609411 (557h3m2.850692456s) [upload=488362]: Waiting for error group managing segments... Dec 28 20:32:03.416896733 (557h3m3.287979778s) [noise=true , address=145.79.227.23:4580 , piece_id=OHYI2SFM6TKJE4LHVNGO2PVGZDNFIQDAIQGKG4FOEHTTASBWA5ZQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.552263655 (557h3m3.4233467s) [noise=true , address=77.90.31.17:4784 , piece_id=JV2ROD6MN65B6LZKOL3S7ATJWXAY6FBQSZWITA6IHAXELY7R42KA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.573564856 (557h3m3.444647902s) [noise=true , address=77.90.31.12:4602 , piece_id=2YPYKKTQ6ID6VTI4AUZQEG2SY7JYPRSAPU4JLF7DWSOSBBO74LWQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.598560923 (557h3m3.469643968s) [noise=true , address=77.90.31.19:4867 , piece_id=LDAWD2ZFG3BZATDRY7MMNWYIX6TOHQ2OIUPCAWAYN266GNUUKZ5A, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.626925925 (557h3m3.49800897s) [noise=true , address=77.90.31.16:4745 , piece_id=CZBFU33BHMP2BEZIDJKW3J526537HR6HP7OCT2KMAQVY7LGZLPZQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.709061947 (557h3m3.580144992s) [noise=true , address=77.90.31.14:4688 , piece_id=4OULHJ6Z3QJ3BOPDY74TJSKVJG23TOEUCJCUEGY24I5C3MUTAVLQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.788879373 (557h3m3.659962418s) [noise=true , address=77.90.31.21:4895 , piece_id=V4P3KCUDLI543TSROBYUMVTQKKSQDDVSKSJSXKEZTQAHTXUJ2G4Q, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.217616523 (557h3m4.088699569s) [noise=true , address=77.90.31.19:4858 , piece_id=LIYXCPAZOUPKXLKRH3EOWREXUY2LAJXSRZFQILM2QIU7HTW7SY2Q, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.220337625 (557h3m4.09142067s) [noise=true , address=77.90.31.11:4563 , piece_id=DIFJPHSKHROTLKCHP6HQBWT7XGE77HVRZTH2354MV2UEWZLHVGCQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.303958744 (557h3m4.175041789s) [noise=true , address=145.79.227.44:4213 , piece_id=UEAQDBBCP7DIG6ZD3EEM2CKAL4W5QKW23LWFUAMSVUW5ZS2PJDRQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.312794398 (557h3m4.183877444s) [noise=true , address=145.79.227.34:4918 , piece_id=SPDT2KI2774UJMQIUSVTHXCDF7PD2FBG64BXUYW67UMLPE6H7SBQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.349308798 (557h3m4.220391833s) [noise=true , address=145.79.227.40:4058 , piece_id=EQ4LA7T2BCGWY76K3NJI4X77OQJFJED6H4VNGXDQMILU6IVQWOMQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.371518379 (557h3m4.242601424s) [noise=true , address=145.79.227.43:4171 , piece_id=YF5DRDPGRRVYFL5SWMH2PKTSV3C72BIWKUDSAIZTXOIVOTYKRMNA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.376232316 (557h3m4.247315361s) [noise=true , address=145.79.227.30:4768 , piece_id=KK4LEV6RG3IGLYXID5MJF3KMNSJ7WS5JS45PNOOJ26LXLYAUCWOA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.376257844 (557h3m4.247340879s) [seg_pos={0 0}, upload=488362]: Segment reached optimal threshold of 14 pieces. Dec 28 20:32:04.376575799 (557h3m4.247658834s) [noise=true , address=145.79.227.33:4904 , piece_id=QCMN4OSBWIWS4JRLIGQIGDSHZFUF2JWEPVXQ4FQ4G3LG5FBCBAWQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1dC7mC2Cbxmys7ukEkELyF9Us7F1qtptSP6yDJh1EfjVJ6uqsC): protocol: expected piece hash; context canceled; EOF Dec 28 20:32:04.376585127 (557h3m4.247668163s) [noise=true , address=145.79.227.33:4904 , piece_id=QCMN4OSBWIWS4JRLIGQIGDSHZFUF2JWEPVXQ4FQ4G3LG5FBCBAWQ, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.376764359 (557h3m4.247847404s) [noise=true , address=145.79.227.19:4105 , piece_id=RY35FSM6NTWPNF3JE43FZRSNXXYZPUN2B6LRNNUOG6QQR6XSU36Q, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GhDrKMcDEg2RxamqKtiHphpgNdvtZYdHXrjzGsNJFjBFYeEen): protocol: write tcp 188.166.225.170:38838->145.79.227.19:4105: use of closed network connection; write tcp 188.166.225.170:38838->145.79.227.19:4105: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38838->145.79.227.19:4105: use of closed network connection Dec 28 20:32:04.376772865 (557h3m4.2478559s) [noise=true , address=145.79.227.19:4105 , piece_id=RY35FSM6NTWPNF3JE43FZRSNXXYZPUN2B6LRNNUOG6QQR6XSU36Q, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.376926658 (557h3m4.248009703s) [noise=true , address=145.79.227.35:4496 , piece_id=JH4XQBT7XYEJXWBGPPF6SJWCPLV77GBOLGOA53C3WUQMWHZXVZQA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V8XhSWkyGUVYFR8MykifFAP1YAfGCwzqucDcMMfzw3xsVW77Z): protocol: expected piece hash; context canceled; EOF Dec 28 20:32:04.377165332 (557h3m4.248248377s) [noise=true , address=145.79.227.35:4496 , piece_id=JH4XQBT7XYEJXWBGPPF6SJWCPLV77GBOLGOA53C3WUQMWHZXVZQA, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.377212682 (557h3m4.248295848s) [noise=true , address=45.91.116.5:4330 , piece_id=EVHLSVVR5JGY43TLSKVEQLQ6KDBEZENVAFL6YT7ZQ77Y7X2TXIKA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12MK9aVDRwfWPfckFR6RfPUL1Nrm9bwfsgV7TXCKGbtadHebWvW): protocol: read tcp 188.166.225.170:55416->45.91.116.5:4330: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 20:32:04.377289859 (557h3m4.248372924s) [noise=true , address=45.91.116.5:4330 , piece_id=EVHLSVVR5JGY43TLSKVEQLQ6KDBEZENVAFL6YT7ZQ77Y7X2TXIKA, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.377387736 (557h3m4.248470781s) [seg_pos={0 0}, upload=488362]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 20:32:04.377599729 (557h3m4.248682774s) [upload=488362]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 20:32:04.377613535 (557h3m4.24869662s) [upload=488362]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 20:32:04.675090855 (557h3m4.5461739s) [upload=488362]: Done uploading segments... Dec 28 20:32:04.675121233 (557h3m4.546204268s) [upload=488362]: Upload finished. err: <nil> Duration: 12.721558206s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..X...........X..................X.......X......XX..........] duration: 17.599069228s trace: https://jaeger.dev.storj.io/trace/3ce821e76930523a FAIL (2025-12-28 20:56:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 20:49:03.805105648 (557h20m3.676188693s) [upload=488615]: Starting upload Dec 28 20:49:03.805166324 (557h20m3.676249369s) [upload=488615]: Done starting upload Dec 28 20:49:03.805293475 (557h20m3.67637651s) [upload=488615]: Uploading segments... Dec 28 20:49:03.805403545 (557h20m3.67648658s) [upload=488615]: Got next segment. Inline: false Dec 28 20:49:03.805460854 (557h20m3.676543889s) [upload=488615]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 20:49:03.805467066 (557h20m3.676550111s) [upload=488615]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 20:49:04.052770455 (557h20m3.9238535s) [seg_pos={0 0}, upload=488615]: Begin upload segment... Dec 28 20:49:04.052835940 (557h20m3.923918985s) [seg_pos={0 0}, upload=488615]: Done begin upload segment. Dec 28 20:49:04.052874203 (557h20m3.923957248s) [noise=true , address=145.79.227.44:4194 , piece_id=WYODZVE7TGDTRNVGZ5HAD2JJT3TSOB7KGNGWH5VCQMLCCICMORRA, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.052970917 (557h20m3.924053962s) [noise=true , address=145.79.227.39:4015 , piece_id=UNPAANTMSZ3O356MXFAX357E3L4X5OLMJBXTDQEVGXUH5B5KDYIQ, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.053474816 (557h20m3.924557861s) [noise=true , address=45.91.116.8:4046 , piece_id=2DXIM4FJEZZQ3LRR4GFUEJ6XD6IGDET3JVKWBZJQGKMICWUITTTA, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.053938549 (557h20m3.925021584s) [noise=true , address=145.79.227.37:4565 , piece_id=6EAD5WOUCVJXUOKHGTSMMHDYPKHHJFZZFIXH2FRUNNZTZNQNK5IA, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.054376714 (557h20m3.925459759s) [noise=true , address=145.79.227.19:4078 , piece_id=DCMFZ42KOAHHMA3YOVW7655PLKCMJDHTLETUU4NK2D6EE27TEKWQ, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.055140448 (557h20m3.926223493s) [noise=true , address=145.79.227.36:4519 , piece_id=GQENJD2GLGJSL7H7VWUN6JXBZTW6ZHFLMMYTKYF3TRBGNNBQC3YQ, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.055719992 (557h20m3.926803027s) [noise=true , address=145.79.227.51:4411 , piece_id=IMS72BZZCJ3WMXU3FTBKM3F4MSXTL4VCVCPOCNLQI6WHKJGIOXQQ, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.056924334 (557h20m3.92800737s) [noise=true , address=45.91.116.6:4973 , piece_id=HMTEGMPYPTWCLRSAH45LLVZTTG2TRVUS44MTKBPZT6SIMRFYYNGQ, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.058635102 (557h20m3.929718147s) [noise=true , address=145.79.227.13:4804 , piece_id=UYO4W6FOSDQSVVEO6XWVTGXMFLXYWNYA3DMICFUABVSFFDKPFRXA, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.058693383 (557h20m3.929776418s) [noise=true , address=45.91.116.5:4299 , piece_id=HI6B35X52HVYVDIFVSJQT6TXYBBDA5VB5COKH53CPKDTO3UHKWJA, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.061111298 (557h20m3.932194343s) [noise=true , address=77.90.31.14:4661 , piece_id=OP5ES2G4MLZBJWJDYYPO7KLHUC6NDX5FUF6ZO6PCYMCFSFTKPVDA, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.061856947 (557h20m3.932939992s) [noise=true , address=145.79.227.42:4123 , piece_id=6YEKCEU5A5MPTRVGRFH2GWQJVSWMBXHCIZSAJKPUY2M2I3P63SFQ, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.064452048 (557h20m3.935535093s) [noise=true , address=45.91.116.10:4082 , piece_id=WGN35FKEXP3UEANOQA4NSJUQLDKIZ36RYD6OFP2FHEBNJKLI2TTA, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.066246636 (557h20m3.937329681s) [noise=true , address=145.79.227.5:4221 , piece_id=GVQO2XLQ2D6XVAHLGZVB6SHA476DL3TUHAXLFKZU5H2BN3LEB5XQ, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.069098717 (557h20m3.940181762s) [noise=true , address=145.79.227.38:4977 , piece_id=23SZDYDT2OWMLO7EGT6OT5L4S4WYR5LY2SWCCZGSCZMKQ5RK6XPQ, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.070882644 (557h20m3.941965689s) [noise=true , address=145.79.227.32:4834 , piece_id=52YM22AS3BG3OGILTWR36HSHD5SKNRKKUR5TIE7HLWOSK3L6KQOQ, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:05.778149774 (557h20m5.649232819s) [upload=488615]: Next returned nil segment Dec 28 20:49:05.778171825 (557h20m5.64925487s) [upload=488615]: Waiting for error group managing segments... Dec 28 20:49:06.067849953 (557h20m5.938932998s) [noise=true , address=145.79.227.37:4565 , piece_id=6EAD5WOUCVJXUOKHGTSMMHDYPKHHJFZZFIXH2FRUNNZTZNQNK5IA, seg_pos={0 0}, upload=488615]: Done uploading piece. err: <nil> Dec 28 20:49:06.073409289 (557h20m5.944492334s) [noise=true , address=145.79.227.39:4015 , piece_id=UNPAANTMSZ3O356MXFAX357E3L4X5OLMJBXTDQEVGXUH5B5KDYIQ, seg_pos={0 0}, upload=488615]: Done uploading piece. err: <nil> Dec 28 20:49:06.078797668 (557h20m5.949880714s) [noise=true , address=145.79.227.13:4804 , piece_id=UYO4W6FOSDQSVVEO6XWVTGXMFLXYWNYA3DMICFUABVSFFDKPFRXA, seg_pos={0 0}, upload=488615]: Done uploading piece. err: <nil> Dec 28 20:49:06.084142657 (557h20m5.955225692s) [noise=true , address=145.79.227.32:4834 , piece_id=52YM22AS3BG3OGILTWR36HSHD5SKNRKKUR5TIE7HLWOSK3L6KQOQ, seg_pos={0 0}, upload=488615]: Done uploading piece. err: <nil> Dec 28 20:49:06.084205807 (557h20m5.955288842s) [noise=true , address=145.79.227.19:4078 , piece_id=DCMFZ42KOAHHMA3YOVW7655PLKCMJDHTLETUU4NK2D6EE27TEKWQ, seg_pos={0 0}, upload=488615]: Done uploading piece. err: <nil> Dec 28 20:49:06.085908499 (557h20m5.956991534s) [noise=true , address=145.79.227.51:4411 , piece_id=IMS72BZZCJ3WMXU3FTBKM3F4MSXTL4VCVCPOCNLQI6WHKJGIOXQQ, seg_pos={0 0}, upload=488615]: Done uploading piece. err: <nil> Dec 28 20:49:06.089752589 (557h20m5.960835624s) [noise=true , address=145.79.227.5:4221 , piece_id=GVQO2XLQ2D6XVAHLGZVB6SHA476DL3TUHAXLFKZU5H2BN3LEB5XQ, seg_pos={0 0}, upload=488615]: Done uploading piece. err: <nil> Dec 28 20:49:06.096096639 (557h20m5.967179674s) [noise=true , address=145.79.227.36:4519 , piece_id=GQENJD2GLGJSL7H7VWUN6JXBZTW6ZHFLMMYTKYF3TRBGNNBQC3YQ, seg_pos={0 0}, upload=488615]: Done uploading piece. err: <nil> Dec 28 20:49:06.097668182 (557h20m5.968751227s) [noise=true , address=145.79.227.42:4123 , piece_id=6YEKCEU5A5MPTRVGRFH2GWQJVSWMBXHCIZSAJKPUY2M2I3P63SFQ, seg_pos={0 0}, upload=488615]: Done uploading piece. err: <nil> Dec 28 20:49:06.107376347 (557h20m5.978459392s) [noise=true , address=145.79.227.38:4977 , piece_id=23SZDYDT2OWMLO7EGT6OT5L4S4WYR5LY2SWCCZGSCZMKQ5RK6XPQ, seg_pos={0 0}, upload=488615]: Done uploading piece. err: <nil> Dec 28 20:49:06.255907707 (557h20m6.126990752s) [noise=true , address=77.90.31.14:4661 , piece_id=OP5ES2G4MLZBJWJDYYPO7KLHUC6NDX5FUF6ZO6PCYMCFSFTKPVDA, seg_pos={0 0}, upload=488615]: Done uploading piece. err: <nil> Dec 28 20:49:06.306350829 (557h20m6.177433874s) [noise=true , address=45.91.116.5:4299 , piece_id=HI6B35X52HVYVDIFVSJQT6TXYBBDA5VB5COKH53CPKDTO3UHKWJA, seg_pos={0 0}, upload=488615]: Done uploading piece. err: <nil> Dec 28 20:49:06.312533491 (557h20m6.183616536s) [noise=true , address=45.91.116.8:4046 , piece_id=2DXIM4FJEZZQ3LRR4GFUEJ6XD6IGDET3JVKWBZJQGKMICWUITTTA, seg_pos={0 0}, upload=488615]: Done uploading piece. err: <nil> Dec 28 20:49:06.312558800 (557h20m6.183641845s) [seg_pos={0 0}, upload=488615]: Segment reached optimal threshold of 13 pieces. Dec 28 20:49:06.313074843 (557h20m6.184157888s) [noise=true , address=145.79.227.44:4194 , piece_id=WYODZVE7TGDTRNVGZ5HAD2JJT3TSOB7KGNGWH5VCQMLCCICMORRA, seg_pos={0 0}, upload=488615]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 28 20:49:06.313086825 (557h20m6.18416987s) [noise=true , address=145.79.227.44:4194 , piece_id=WYODZVE7TGDTRNVGZ5HAD2JJT3TSOB7KGNGWH5VCQMLCCICMORRA, seg_pos={0 0}, upload=488615]: Long tail cancellation due to optimal threshold Dec 28 20:49:06.313142812 (557h20m6.184225847s) [noise=true , address=45.91.116.6:4973 , piece_id=HMTEGMPYPTWCLRSAH45LLVZTTG2TRVUS44MTKBPZT6SIMRFYYNGQ, seg_pos={0 0}, upload=488615]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF Dec 28 20:49:06.313177759 (557h20m6.184260804s) [noise=true , address=45.91.116.6:4973 , piece_id=HMTEGMPYPTWCLRSAH45LLVZTTG2TRVUS44MTKBPZT6SIMRFYYNGQ, seg_pos={0 0}, upload=488615]: Long tail cancellation due to optimal threshold Dec 28 20:49:06.313256448 (557h20m6.184339503s) [noise=true , address=45.91.116.10:4082 , piece_id=WGN35FKEXP3UEANOQA4NSJUQLDKIZ36RYD6OFP2FHEBNJKLI2TTA, seg_pos={0 0}, upload=488615]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 28 20:49:06.313271217 (557h20m6.184354262s) [noise=true , address=45.91.116.10:4082 , piece_id=WGN35FKEXP3UEANOQA4NSJUQLDKIZ36RYD6OFP2FHEBNJKLI2TTA, seg_pos={0 0}, upload=488615]: Long tail cancellation due to optimal threshold Dec 28 20:49:06.313296314 (557h20m6.184379359s) [seg_pos={0 0}, upload=488615]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 20:49:06.313381627 (557h20m6.184464672s) [upload=488615]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 20:49:06.313389141 (557h20m6.184472186s) [upload=488615]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 20:49:06.604901881 (557h20m6.475984916s) [upload=488615]: Done uploading segments... Dec 28 20:49:06.604926539 (557h20m6.476009584s) [upload=488615]: Upload finished. err: <nil> Duration: 17.59912794s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 14.896720688s trace: https://jaeger.dev.storj.io/trace/499b96822b8dda80 FAIL (2025-12-28 20:55:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12S7WBBQ1Nq8jbeGrZiRKTk23CdJC31hPW2vU1aaM5SsHeVyeRH, Piece ID: V4P3KCUDLI543TSROBYUMVTQKKSQDDVSKSJSXKEZTQAHTXUJ2G4Q) context deadline exceeded; piecestore: (Node ID: 1kjCEPtjvcJ3D7BBKRSRirPybBwkRZ1dJwyAQCadzCvaLYvWhz, Piece ID: 4OULHJ6Z3QJ3BOPDY74TJSKVJG23TOEUCJCUEGY24I5C3MUTAVLQ) context deadline exceeded; piecestore: (Node ID: 1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c, Piece ID: LIYXCPAZOUPKXLKRH3EOWREXUY2LAJXSRZFQILM2QIU7HTW7SY2Q) context deadline exceeded ======LOGS====== ue , address=145.79.227.23:4580 , piece_id=OHYI2SFM6TKJE4LHVNGO2PVGZDNFIQDAIQGKG4FOEHTTASBWA5ZQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.496428825 (557h3m1.36751187s) [noise=true , address=77.90.31.19:4867 , piece_id=LDAWD2ZFG3BZATDRY7MMNWYIX6TOHQ2OIUPCAWAYN266GNUUKZ5A, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.496861509 (557h3m1.367944544s) [noise=true , address=145.79.227.19:4105 , piece_id=RY35FSM6NTWPNF3JE43FZRSNXXYZPUN2B6LRNNUOG6QQR6XSU36Q, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.496899933 (557h3m1.367982978s) [noise=true , address=77.90.31.19:4858 , piece_id=LIYXCPAZOUPKXLKRH3EOWREXUY2LAJXSRZFQILM2QIU7HTW7SY2Q, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497229089 (557h3m1.368312134s) [noise=true , address=77.90.31.17:4784 , piece_id=JV2ROD6MN65B6LZKOL3S7ATJWXAY6FBQSZWITA6IHAXELY7R42KA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497326194 (557h3m1.368409239s) [noise=true , address=145.79.227.33:4904 , piece_id=QCMN4OSBWIWS4JRLIGQIGDSHZFUF2JWEPVXQ4FQ4G3LG5FBCBAWQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497664128 (557h3m1.368747173s) [noise=true , address=145.79.227.34:4918 , piece_id=SPDT2KI2774UJMQIUSVTHXCDF7PD2FBG64BXUYW67UMLPE6H7SBQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497731405 (557h3m1.36881445s) [noise=true , address=77.90.31.12:4602 , piece_id=2YPYKKTQ6ID6VTI4AUZQEG2SY7JYPRSAPU4JLF7DWSOSBBO74LWQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.498205939 (557h3m1.369288984s) [noise=true , address=77.90.31.16:4745 , piece_id=CZBFU33BHMP2BEZIDJKW3J526537HR6HP7OCT2KMAQVY7LGZLPZQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.498424295 (557h3m1.36950734s) [noise=true , address=77.90.31.21:4895 , piece_id=V4P3KCUDLI543TSROBYUMVTQKKSQDDVSKSJSXKEZTQAHTXUJ2G4Q, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.498685422 (557h3m1.369768457s) [noise=true , address=145.79.227.35:4496 , piece_id=JH4XQBT7XYEJXWBGPPF6SJWCPLV77GBOLGOA53C3WUQMWHZXVZQA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499066348 (557h3m1.370149393s) [noise=true , address=77.90.31.11:4563 , piece_id=DIFJPHSKHROTLKCHP6HQBWT7XGE77HVRZTH2354MV2UEWZLHVGCQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499185725 (557h3m1.37026877s) [noise=true , address=145.79.227.30:4768 , piece_id=KK4LEV6RG3IGLYXID5MJF3KMNSJ7WS5JS45PNOOJ26LXLYAUCWOA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499468122 (557h3m1.370551157s) [noise=true , address=77.90.31.14:4688 , piece_id=4OULHJ6Z3QJ3BOPDY74TJSKVJG23TOEUCJCUEGY24I5C3MUTAVLQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499901047 (557h3m1.370984102s) [noise=true , address=45.91.116.5:4330 , piece_id=EVHLSVVR5JGY43TLSKVEQLQ6KDBEZENVAFL6YT7ZQ77Y7X2TXIKA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.500546175 (557h3m1.37162921s) [noise=true , address=145.79.227.43:4171 , piece_id=YF5DRDPGRRVYFL5SWMH2PKTSV3C72BIWKUDSAIZTXOIVOTYKRMNA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:02.979572381 (557h3m2.850655436s) [upload=488362]: Next returned nil segment Dec 28 20:32:02.979609411 (557h3m2.850692456s) [upload=488362]: Waiting for error group managing segments... Dec 28 20:32:03.416896733 (557h3m3.287979778s) [noise=true , address=145.79.227.23:4580 , piece_id=OHYI2SFM6TKJE4LHVNGO2PVGZDNFIQDAIQGKG4FOEHTTASBWA5ZQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.552263655 (557h3m3.4233467s) [noise=true , address=77.90.31.17:4784 , piece_id=JV2ROD6MN65B6LZKOL3S7ATJWXAY6FBQSZWITA6IHAXELY7R42KA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.573564856 (557h3m3.444647902s) [noise=true , address=77.90.31.12:4602 , piece_id=2YPYKKTQ6ID6VTI4AUZQEG2SY7JYPRSAPU4JLF7DWSOSBBO74LWQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.598560923 (557h3m3.469643968s) [noise=true , address=77.90.31.19:4867 , piece_id=LDAWD2ZFG3BZATDRY7MMNWYIX6TOHQ2OIUPCAWAYN266GNUUKZ5A, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.626925925 (557h3m3.49800897s) [noise=true , address=77.90.31.16:4745 , piece_id=CZBFU33BHMP2BEZIDJKW3J526537HR6HP7OCT2KMAQVY7LGZLPZQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.709061947 (557h3m3.580144992s) [noise=true , address=77.90.31.14:4688 , piece_id=4OULHJ6Z3QJ3BOPDY74TJSKVJG23TOEUCJCUEGY24I5C3MUTAVLQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.788879373 (557h3m3.659962418s) [noise=true , address=77.90.31.21:4895 , piece_id=V4P3KCUDLI543TSROBYUMVTQKKSQDDVSKSJSXKEZTQAHTXUJ2G4Q, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.217616523 (557h3m4.088699569s) [noise=true , address=77.90.31.19:4858 , piece_id=LIYXCPAZOUPKXLKRH3EOWREXUY2LAJXSRZFQILM2QIU7HTW7SY2Q, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.220337625 (557h3m4.09142067s) [noise=true , address=77.90.31.11:4563 , piece_id=DIFJPHSKHROTLKCHP6HQBWT7XGE77HVRZTH2354MV2UEWZLHVGCQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.303958744 (557h3m4.175041789s) [noise=true , address=145.79.227.44:4213 , piece_id=UEAQDBBCP7DIG6ZD3EEM2CKAL4W5QKW23LWFUAMSVUW5ZS2PJDRQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.312794398 (557h3m4.183877444s) [noise=true , address=145.79.227.34:4918 , piece_id=SPDT2KI2774UJMQIUSVTHXCDF7PD2FBG64BXUYW67UMLPE6H7SBQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.349308798 (557h3m4.220391833s) [noise=true , address=145.79.227.40:4058 , piece_id=EQ4LA7T2BCGWY76K3NJI4X77OQJFJED6H4VNGXDQMILU6IVQWOMQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.371518379 (557h3m4.242601424s) [noise=true , address=145.79.227.43:4171 , piece_id=YF5DRDPGRRVYFL5SWMH2PKTSV3C72BIWKUDSAIZTXOIVOTYKRMNA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.376232316 (557h3m4.247315361s) [noise=true , address=145.79.227.30:4768 , piece_id=KK4LEV6RG3IGLYXID5MJF3KMNSJ7WS5JS45PNOOJ26LXLYAUCWOA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.376257844 (557h3m4.247340879s) [seg_pos={0 0}, upload=488362]: Segment reached optimal threshold of 14 pieces. Dec 28 20:32:04.376575799 (557h3m4.247658834s) [noise=true , address=145.79.227.33:4904 , piece_id=QCMN4OSBWIWS4JRLIGQIGDSHZFUF2JWEPVXQ4FQ4G3LG5FBCBAWQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1dC7mC2Cbxmys7ukEkELyF9Us7F1qtptSP6yDJh1EfjVJ6uqsC): protocol: expected piece hash; context canceled; EOF Dec 28 20:32:04.376585127 (557h3m4.247668163s) [noise=true , address=145.79.227.33:4904 , piece_id=QCMN4OSBWIWS4JRLIGQIGDSHZFUF2JWEPVXQ4FQ4G3LG5FBCBAWQ, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.376764359 (557h3m4.247847404s) [noise=true , address=145.79.227.19:4105 , piece_id=RY35FSM6NTWPNF3JE43FZRSNXXYZPUN2B6LRNNUOG6QQR6XSU36Q, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GhDrKMcDEg2RxamqKtiHphpgNdvtZYdHXrjzGsNJFjBFYeEen): protocol: write tcp 188.166.225.170:38838->145.79.227.19:4105: use of closed network connection; write tcp 188.166.225.170:38838->145.79.227.19:4105: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38838->145.79.227.19:4105: use of closed network connection Dec 28 20:32:04.376772865 (557h3m4.2478559s) [noise=true , address=145.79.227.19:4105 , piece_id=RY35FSM6NTWPNF3JE43FZRSNXXYZPUN2B6LRNNUOG6QQR6XSU36Q, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.376926658 (557h3m4.248009703s) [noise=true , address=145.79.227.35:4496 , piece_id=JH4XQBT7XYEJXWBGPPF6SJWCPLV77GBOLGOA53C3WUQMWHZXVZQA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V8XhSWkyGUVYFR8MykifFAP1YAfGCwzqucDcMMfzw3xsVW77Z): protocol: expected piece hash; context canceled; EOF Dec 28 20:32:04.377165332 (557h3m4.248248377s) [noise=true , address=145.79.227.35:4496 , piece_id=JH4XQBT7XYEJXWBGPPF6SJWCPLV77GBOLGOA53C3WUQMWHZXVZQA, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.377212682 (557h3m4.248295848s) [noise=true , address=45.91.116.5:4330 , piece_id=EVHLSVVR5JGY43TLSKVEQLQ6KDBEZENVAFL6YT7ZQ77Y7X2TXIKA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12MK9aVDRwfWPfckFR6RfPUL1Nrm9bwfsgV7TXCKGbtadHebWvW): protocol: read tcp 188.166.225.170:55416->45.91.116.5:4330: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 20:32:04.377289859 (557h3m4.248372924s) [noise=true , address=45.91.116.5:4330 , piece_id=EVHLSVVR5JGY43TLSKVEQLQ6KDBEZENVAFL6YT7ZQ77Y7X2TXIKA, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.377387736 (557h3m4.248470781s) [seg_pos={0 0}, upload=488362]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 20:32:04.377599729 (557h3m4.248682774s) [upload=488362]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 20:32:04.377613535 (557h3m4.24869662s) [upload=488362]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 20:32:04.675090855 (557h3m4.5461739s) [upload=488362]: Done uploading segments... Dec 28 20:32:04.675121233 (557h3m4.546204268s) [upload=488362]: Upload finished. err: <nil> Duration: 14.896788888s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [...X...........X..................X.......X......XX.........] duration: 26.697513101s trace: https://jaeger.dev.storj.io/trace/3ce821e76930523a FAIL (2025-12-28 20:55:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 20:49:03.805105648 (557h20m3.676188693s) [upload=488615]: Starting upload Dec 28 20:49:03.805166324 (557h20m3.676249369s) [upload=488615]: Done starting upload Dec 28 20:49:03.805293475 (557h20m3.67637651s) [upload=488615]: Uploading segments... Dec 28 20:49:03.805403545 (557h20m3.67648658s) [upload=488615]: Got next segment. Inline: false Dec 28 20:49:03.805460854 (557h20m3.676543889s) [upload=488615]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 20:49:03.805467066 (557h20m3.676550111s) [upload=488615]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 20:49:04.052770455 (557h20m3.9238535s) [seg_pos={0 0}, upload=488615]: Begin upload segment... Dec 28 20:49:04.052835940 (557h20m3.923918985s) [seg_pos={0 0}, upload=488615]: Done begin upload segment. Dec 28 20:49:04.052874203 (557h20m3.923957248s) [noise=true , address=145.79.227.44:4194 , piece_id=WYODZVE7TGDTRNVGZ5HAD2JJT3TSOB7KGNGWH5VCQMLCCICMORRA, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.052970917 (557h20m3.924053962s) [noise=true , address=145.79.227.39:4015 , piece_id=UNPAANTMSZ3O356MXFAX357E3L4X5OLMJBXTDQEVGXUH5B5KDYIQ, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.053474816 (557h20m3.924557861s) [noise=true , address=45.91.116.8:4046 , piece_id=2DXIM4FJEZZQ3LRR4GFUEJ6XD6IGDET3JVKWBZJQGKMICWUITTTA, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.053938549 (557h20m3.925021584s) [noise=true , address=145.79.227.37:4565 , piece_id=6EAD5WOUCVJXUOKHGTSMMHDYPKHHJFZZFIXH2FRUNNZTZNQNK5IA, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.054376714 (557h20m3.925459759s) [noise=true , address=145.79.227.19:4078 , piece_id=DCMFZ42KOAHHMA3YOVW7655PLKCMJDHTLETUU4NK2D6EE27TEKWQ, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.055140448 (557h20m3.926223493s) [noise=true , address=145.79.227.36:4519 , piece_id=GQENJD2GLGJSL7H7VWUN6JXBZTW6ZHFLMMYTKYF3TRBGNNBQC3YQ, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.055719992 (557h20m3.926803027s) [noise=true , address=145.79.227.51:4411 , piece_id=IMS72BZZCJ3WMXU3FTBKM3F4MSXTL4VCVCPOCNLQI6WHKJGIOXQQ, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.056924334 (557h20m3.92800737s) [noise=true , address=45.91.116.6:4973 , piece_id=HMTEGMPYPTWCLRSAH45LLVZTTG2TRVUS44MTKBPZT6SIMRFYYNGQ, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.058635102 (557h20m3.929718147s) [noise=true , address=145.79.227.13:4804 , piece_id=UYO4W6FOSDQSVVEO6XWVTGXMFLXYWNYA3DMICFUABVSFFDKPFRXA, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.058693383 (557h20m3.929776418s) [noise=true , address=45.91.116.5:4299 , piece_id=HI6B35X52HVYVDIFVSJQT6TXYBBDA5VB5COKH53CPKDTO3UHKWJA, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.061111298 (557h20m3.932194343s) [noise=true , address=77.90.31.14:4661 , piece_id=OP5ES2G4MLZBJWJDYYPO7KLHUC6NDX5FUF6ZO6PCYMCFSFTKPVDA, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.061856947 (557h20m3.932939992s) [noise=true , address=145.79.227.42:4123 , piece_id=6YEKCEU5A5MPTRVGRFH2GWQJVSWMBXHCIZSAJKPUY2M2I3P63SFQ, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.064452048 (557h20m3.935535093s) [noise=true , address=45.91.116.10:4082 , piece_id=WGN35FKEXP3UEANOQA4NSJUQLDKIZ36RYD6OFP2FHEBNJKLI2TTA, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.066246636 (557h20m3.937329681s) [noise=true , address=145.79.227.5:4221 , piece_id=GVQO2XLQ2D6XVAHLGZVB6SHA476DL3TUHAXLFKZU5H2BN3LEB5XQ, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.069098717 (557h20m3.940181762s) [noise=true , address=145.79.227.38:4977 , piece_id=23SZDYDT2OWMLO7EGT6OT5L4S4WYR5LY2SWCCZGSCZMKQ5RK6XPQ, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.070882644 (557h20m3.941965689s) [noise=true , address=145.79.227.32:4834 , piece_id=52YM22AS3BG3OGILTWR36HSHD5SKNRKKUR5TIE7HLWOSK3L6KQOQ, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:05.778149774 (557h20m5.649232819s) [upload=488615]: Next returned nil segment Dec 28 20:49:05.778171825 (557h20m5.64925487s) [upload=488615]: Waiting for error group managing segments... Dec 28 20:49:06.067849953 (557h20m5.938932998s) [noise=true , address=145.79.227.37:4565 , piece_id=6EAD5WOUCVJXUOKHGTSMMHDYPKHHJFZZFIXH2FRUNNZTZNQNK5IA, seg_pos={0 0}, upload=488615]: Done uploading piece. err: <nil> Dec 28 20:49:06.073409289 (557h20m5.944492334s) [noise=true , address=145.79.227.39:4015 , piece_id=UNPAANTMSZ3O356MXFAX357E3L4X5OLMJBXTDQEVGXUH5B5KDYIQ, seg_pos={0 0}, upload=488615]: Done uploading piece. err: <nil> Dec 28 20:49:06.078797668 (557h20m5.949880714s) [noise=true , address=145.79.227.13:4804 , piece_id=UYO4W6FOSDQSVVEO6XWVTGXMFLXYWNYA3DMICFUABVSFFDKPFRXA, seg_pos={0 0}, upload=488615]: Done uploading piece. err: <nil> Dec 28 20:49:06.084142657 (557h20m5.955225692s) [noise=true , address=145.79.227.32:4834 , piece_id=52YM22AS3BG3OGILTWR36HSHD5SKNRKKUR5TIE7HLWOSK3L6KQOQ, seg_pos={0 0}, upload=488615]: Done uploading piece. err: <nil> Dec 28 20:49:06.084205807 (557h20m5.955288842s) [noise=true , address=145.79.227.19:4078 , piece_id=DCMFZ42KOAHHMA3YOVW7655PLKCMJDHTLETUU4NK2D6EE27TEKWQ, seg_pos={0 0}, upload=488615]: Done uploading piece. err: <nil> Dec 28 20:49:06.085908499 (557h20m5.956991534s) [noise=true , address=145.79.227.51:4411 , piece_id=IMS72BZZCJ3WMXU3FTBKM3F4MSXTL4VCVCPOCNLQI6WHKJGIOXQQ, seg_pos={0 0}, upload=488615]: Done uploading piece. err: <nil> Dec 28 20:49:06.089752589 (557h20m5.960835624s) [noise=true , address=145.79.227.5:4221 , piece_id=GVQO2XLQ2D6XVAHLGZVB6SHA476DL3TUHAXLFKZU5H2BN3LEB5XQ, seg_pos={0 0}, upload=488615]: Done uploading piece. err: <nil> Dec 28 20:49:06.096096639 (557h20m5.967179674s) [noise=true , address=145.79.227.36:4519 , piece_id=GQENJD2GLGJSL7H7VWUN6JXBZTW6ZHFLMMYTKYF3TRBGNNBQC3YQ, seg_pos={0 0}, upload=488615]: Done uploading piece. err: <nil> Dec 28 20:49:06.097668182 (557h20m5.968751227s) [noise=true , address=145.79.227.42:4123 , piece_id=6YEKCEU5A5MPTRVGRFH2GWQJVSWMBXHCIZSAJKPUY2M2I3P63SFQ, seg_pos={0 0}, upload=488615]: Done uploading piece. err: <nil> Dec 28 20:49:06.107376347 (557h20m5.978459392s) [noise=true , address=145.79.227.38:4977 , piece_id=23SZDYDT2OWMLO7EGT6OT5L4S4WYR5LY2SWCCZGSCZMKQ5RK6XPQ, seg_pos={0 0}, upload=488615]: Done uploading piece. err: <nil> Dec 28 20:49:06.255907707 (557h20m6.126990752s) [noise=true , address=77.90.31.14:4661 , piece_id=OP5ES2G4MLZBJWJDYYPO7KLHUC6NDX5FUF6ZO6PCYMCFSFTKPVDA, seg_pos={0 0}, upload=488615]: Done uploading piece. err: <nil> Dec 28 20:49:06.306350829 (557h20m6.177433874s) [noise=true , address=45.91.116.5:4299 , piece_id=HI6B35X52HVYVDIFVSJQT6TXYBBDA5VB5COKH53CPKDTO3UHKWJA, seg_pos={0 0}, upload=488615]: Done uploading piece. err: <nil> Dec 28 20:49:06.312533491 (557h20m6.183616536s) [noise=true , address=45.91.116.8:4046 , piece_id=2DXIM4FJEZZQ3LRR4GFUEJ6XD6IGDET3JVKWBZJQGKMICWUITTTA, seg_pos={0 0}, upload=488615]: Done uploading piece. err: <nil> Dec 28 20:49:06.312558800 (557h20m6.183641845s) [seg_pos={0 0}, upload=488615]: Segment reached optimal threshold of 13 pieces. Dec 28 20:49:06.313074843 (557h20m6.184157888s) [noise=true , address=145.79.227.44:4194 , piece_id=WYODZVE7TGDTRNVGZ5HAD2JJT3TSOB7KGNGWH5VCQMLCCICMORRA, seg_pos={0 0}, upload=488615]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 28 20:49:06.313086825 (557h20m6.18416987s) [noise=true , address=145.79.227.44:4194 , piece_id=WYODZVE7TGDTRNVGZ5HAD2JJT3TSOB7KGNGWH5VCQMLCCICMORRA, seg_pos={0 0}, upload=488615]: Long tail cancellation due to optimal threshold Dec 28 20:49:06.313142812 (557h20m6.184225847s) [noise=true , address=45.91.116.6:4973 , piece_id=HMTEGMPYPTWCLRSAH45LLVZTTG2TRVUS44MTKBPZT6SIMRFYYNGQ, seg_pos={0 0}, upload=488615]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF Dec 28 20:49:06.313177759 (557h20m6.184260804s) [noise=true , address=45.91.116.6:4973 , piece_id=HMTEGMPYPTWCLRSAH45LLVZTTG2TRVUS44MTKBPZT6SIMRFYYNGQ, seg_pos={0 0}, upload=488615]: Long tail cancellation due to optimal threshold Dec 28 20:49:06.313256448 (557h20m6.184339503s) [noise=true , address=45.91.116.10:4082 , piece_id=WGN35FKEXP3UEANOQA4NSJUQLDKIZ36RYD6OFP2FHEBNJKLI2TTA, seg_pos={0 0}, upload=488615]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 28 20:49:06.313271217 (557h20m6.184354262s) [noise=true , address=45.91.116.10:4082 , piece_id=WGN35FKEXP3UEANOQA4NSJUQLDKIZ36RYD6OFP2FHEBNJKLI2TTA, seg_pos={0 0}, upload=488615]: Long tail cancellation due to optimal threshold Dec 28 20:49:06.313296314 (557h20m6.184379359s) [seg_pos={0 0}, upload=488615]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 20:49:06.313381627 (557h20m6.184464672s) [upload=488615]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 20:49:06.313389141 (557h20m6.184472186s) [upload=488615]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 20:49:06.604901881 (557h20m6.475984916s) [upload=488615]: Done uploading segments... Dec 28 20:49:06.604926539 (557h20m6.476009584s) [upload=488615]: Upload finished. err: <nil> Duration: 26.697542297s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [....X...........X..................X.......X......XX........] duration: 5.858867882s trace: https://jaeger.dev.storj.io/trace/3ce821e76930523a FAIL (2025-12-28 20:54:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 20:49:03.805105648 (557h20m3.676188693s) [upload=488615]: Starting upload Dec 28 20:49:03.805166324 (557h20m3.676249369s) [upload=488615]: Done starting upload Dec 28 20:49:03.805293475 (557h20m3.67637651s) [upload=488615]: Uploading segments... Dec 28 20:49:03.805403545 (557h20m3.67648658s) [upload=488615]: Got next segment. Inline: false Dec 28 20:49:03.805460854 (557h20m3.676543889s) [upload=488615]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 20:49:03.805467066 (557h20m3.676550111s) [upload=488615]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 20:49:04.052770455 (557h20m3.9238535s) [seg_pos={0 0}, upload=488615]: Begin upload segment... Dec 28 20:49:04.052835940 (557h20m3.923918985s) [seg_pos={0 0}, upload=488615]: Done begin upload segment. Dec 28 20:49:04.052874203 (557h20m3.923957248s) [noise=true , address=145.79.227.44:4194 , piece_id=WYODZVE7TGDTRNVGZ5HAD2JJT3TSOB7KGNGWH5VCQMLCCICMORRA, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.052970917 (557h20m3.924053962s) [noise=true , address=145.79.227.39:4015 , piece_id=UNPAANTMSZ3O356MXFAX357E3L4X5OLMJBXTDQEVGXUH5B5KDYIQ, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.053474816 (557h20m3.924557861s) [noise=true , address=45.91.116.8:4046 , piece_id=2DXIM4FJEZZQ3LRR4GFUEJ6XD6IGDET3JVKWBZJQGKMICWUITTTA, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.053938549 (557h20m3.925021584s) [noise=true , address=145.79.227.37:4565 , piece_id=6EAD5WOUCVJXUOKHGTSMMHDYPKHHJFZZFIXH2FRUNNZTZNQNK5IA, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.054376714 (557h20m3.925459759s) [noise=true , address=145.79.227.19:4078 , piece_id=DCMFZ42KOAHHMA3YOVW7655PLKCMJDHTLETUU4NK2D6EE27TEKWQ, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.055140448 (557h20m3.926223493s) [noise=true , address=145.79.227.36:4519 , piece_id=GQENJD2GLGJSL7H7VWUN6JXBZTW6ZHFLMMYTKYF3TRBGNNBQC3YQ, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.055719992 (557h20m3.926803027s) [noise=true , address=145.79.227.51:4411 , piece_id=IMS72BZZCJ3WMXU3FTBKM3F4MSXTL4VCVCPOCNLQI6WHKJGIOXQQ, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.056924334 (557h20m3.92800737s) [noise=true , address=45.91.116.6:4973 , piece_id=HMTEGMPYPTWCLRSAH45LLVZTTG2TRVUS44MTKBPZT6SIMRFYYNGQ, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.058635102 (557h20m3.929718147s) [noise=true , address=145.79.227.13:4804 , piece_id=UYO4W6FOSDQSVVEO6XWVTGXMFLXYWNYA3DMICFUABVSFFDKPFRXA, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.058693383 (557h20m3.929776418s) [noise=true , address=45.91.116.5:4299 , piece_id=HI6B35X52HVYVDIFVSJQT6TXYBBDA5VB5COKH53CPKDTO3UHKWJA, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.061111298 (557h20m3.932194343s) [noise=true , address=77.90.31.14:4661 , piece_id=OP5ES2G4MLZBJWJDYYPO7KLHUC6NDX5FUF6ZO6PCYMCFSFTKPVDA, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.061856947 (557h20m3.932939992s) [noise=true , address=145.79.227.42:4123 , piece_id=6YEKCEU5A5MPTRVGRFH2GWQJVSWMBXHCIZSAJKPUY2M2I3P63SFQ, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.064452048 (557h20m3.935535093s) [noise=true , address=45.91.116.10:4082 , piece_id=WGN35FKEXP3UEANOQA4NSJUQLDKIZ36RYD6OFP2FHEBNJKLI2TTA, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.066246636 (557h20m3.937329681s) [noise=true , address=145.79.227.5:4221 , piece_id=GVQO2XLQ2D6XVAHLGZVB6SHA476DL3TUHAXLFKZU5H2BN3LEB5XQ, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.069098717 (557h20m3.940181762s) [noise=true , address=145.79.227.38:4977 , piece_id=23SZDYDT2OWMLO7EGT6OT5L4S4WYR5LY2SWCCZGSCZMKQ5RK6XPQ, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.070882644 (557h20m3.941965689s) [noise=true , address=145.79.227.32:4834 , piece_id=52YM22AS3BG3OGILTWR36HSHD5SKNRKKUR5TIE7HLWOSK3L6KQOQ, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:05.778149774 (557h20m5.649232819s) [upload=488615]: Next returned nil segment Dec 28 20:49:05.778171825 (557h20m5.64925487s) [upload=488615]: Waiting for error group managing segments... Dec 28 20:49:06.067849953 (557h20m5.938932998s) [noise=true , address=145.79.227.37:4565 , piece_id=6EAD5WOUCVJXUOKHGTSMMHDYPKHHJFZZFIXH2FRUNNZTZNQNK5IA, seg_pos={0 0}, upload=488615]: Done uploading piece. err: <nil> Dec 28 20:49:06.073409289 (557h20m5.944492334s) [noise=true , address=145.79.227.39:4015 , piece_id=UNPAANTMSZ3O356MXFAX357E3L4X5OLMJBXTDQEVGXUH5B5KDYIQ, seg_pos={0 0}, upload=488615]: Done uploading piece. err: <nil> Dec 28 20:49:06.078797668 (557h20m5.949880714s) [noise=true , address=145.79.227.13:4804 , piece_id=UYO4W6FOSDQSVVEO6XWVTGXMFLXYWNYA3DMICFUABVSFFDKPFRXA, seg_pos={0 0}, upload=488615]: Done uploading piece. err: <nil> Dec 28 20:49:06.084142657 (557h20m5.955225692s) [noise=true , address=145.79.227.32:4834 , piece_id=52YM22AS3BG3OGILTWR36HSHD5SKNRKKUR5TIE7HLWOSK3L6KQOQ, seg_pos={0 0}, upload=488615]: Done uploading piece. err: <nil> Dec 28 20:49:06.084205807 (557h20m5.955288842s) [noise=true , address=145.79.227.19:4078 , piece_id=DCMFZ42KOAHHMA3YOVW7655PLKCMJDHTLETUU4NK2D6EE27TEKWQ, seg_pos={0 0}, upload=488615]: Done uploading piece. err: <nil> Dec 28 20:49:06.085908499 (557h20m5.956991534s) [noise=true , address=145.79.227.51:4411 , piece_id=IMS72BZZCJ3WMXU3FTBKM3F4MSXTL4VCVCPOCNLQI6WHKJGIOXQQ, seg_pos={0 0}, upload=488615]: Done uploading piece. err: <nil> Dec 28 20:49:06.089752589 (557h20m5.960835624s) [noise=true , address=145.79.227.5:4221 , piece_id=GVQO2XLQ2D6XVAHLGZVB6SHA476DL3TUHAXLFKZU5H2BN3LEB5XQ, seg_pos={0 0}, upload=488615]: Done uploading piece. err: <nil> Dec 28 20:49:06.096096639 (557h20m5.967179674s) [noise=true , address=145.79.227.36:4519 , piece_id=GQENJD2GLGJSL7H7VWUN6JXBZTW6ZHFLMMYTKYF3TRBGNNBQC3YQ, seg_pos={0 0}, upload=488615]: Done uploading piece. err: <nil> Dec 28 20:49:06.097668182 (557h20m5.968751227s) [noise=true , address=145.79.227.42:4123 , piece_id=6YEKCEU5A5MPTRVGRFH2GWQJVSWMBXHCIZSAJKPUY2M2I3P63SFQ, seg_pos={0 0}, upload=488615]: Done uploading piece. err: <nil> Dec 28 20:49:06.107376347 (557h20m5.978459392s) [noise=true , address=145.79.227.38:4977 , piece_id=23SZDYDT2OWMLO7EGT6OT5L4S4WYR5LY2SWCCZGSCZMKQ5RK6XPQ, seg_pos={0 0}, upload=488615]: Done uploading piece. err: <nil> Dec 28 20:49:06.255907707 (557h20m6.126990752s) [noise=true , address=77.90.31.14:4661 , piece_id=OP5ES2G4MLZBJWJDYYPO7KLHUC6NDX5FUF6ZO6PCYMCFSFTKPVDA, seg_pos={0 0}, upload=488615]: Done uploading piece. err: <nil> Dec 28 20:49:06.306350829 (557h20m6.177433874s) [noise=true , address=45.91.116.5:4299 , piece_id=HI6B35X52HVYVDIFVSJQT6TXYBBDA5VB5COKH53CPKDTO3UHKWJA, seg_pos={0 0}, upload=488615]: Done uploading piece. err: <nil> Dec 28 20:49:06.312533491 (557h20m6.183616536s) [noise=true , address=45.91.116.8:4046 , piece_id=2DXIM4FJEZZQ3LRR4GFUEJ6XD6IGDET3JVKWBZJQGKMICWUITTTA, seg_pos={0 0}, upload=488615]: Done uploading piece. err: <nil> Dec 28 20:49:06.312558800 (557h20m6.183641845s) [seg_pos={0 0}, upload=488615]: Segment reached optimal threshold of 13 pieces. Dec 28 20:49:06.313074843 (557h20m6.184157888s) [noise=true , address=145.79.227.44:4194 , piece_id=WYODZVE7TGDTRNVGZ5HAD2JJT3TSOB7KGNGWH5VCQMLCCICMORRA, seg_pos={0 0}, upload=488615]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 28 20:49:06.313086825 (557h20m6.18416987s) [noise=true , address=145.79.227.44:4194 , piece_id=WYODZVE7TGDTRNVGZ5HAD2JJT3TSOB7KGNGWH5VCQMLCCICMORRA, seg_pos={0 0}, upload=488615]: Long tail cancellation due to optimal threshold Dec 28 20:49:06.313142812 (557h20m6.184225847s) [noise=true , address=45.91.116.6:4973 , piece_id=HMTEGMPYPTWCLRSAH45LLVZTTG2TRVUS44MTKBPZT6SIMRFYYNGQ, seg_pos={0 0}, upload=488615]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF Dec 28 20:49:06.313177759 (557h20m6.184260804s) [noise=true , address=45.91.116.6:4973 , piece_id=HMTEGMPYPTWCLRSAH45LLVZTTG2TRVUS44MTKBPZT6SIMRFYYNGQ, seg_pos={0 0}, upload=488615]: Long tail cancellation due to optimal threshold Dec 28 20:49:06.313256448 (557h20m6.184339503s) [noise=true , address=45.91.116.10:4082 , piece_id=WGN35FKEXP3UEANOQA4NSJUQLDKIZ36RYD6OFP2FHEBNJKLI2TTA, seg_pos={0 0}, upload=488615]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 28 20:49:06.313271217 (557h20m6.184354262s) [noise=true , address=45.91.116.10:4082 , piece_id=WGN35FKEXP3UEANOQA4NSJUQLDKIZ36RYD6OFP2FHEBNJKLI2TTA, seg_pos={0 0}, upload=488615]: Long tail cancellation due to optimal threshold Dec 28 20:49:06.313296314 (557h20m6.184379359s) [seg_pos={0 0}, upload=488615]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 20:49:06.313381627 (557h20m6.184464672s) [upload=488615]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 20:49:06.313389141 (557h20m6.184472186s) [upload=488615]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 20:49:06.604901881 (557h20m6.475984916s) [upload=488615]: Done uploading segments... Dec 28 20:49:06.604926539 (557h20m6.476009584s) [upload=488615]: Upload finished. err: <nil> Duration: 5.858894802s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 7.059729047s trace: https://jaeger.dev.storj.io/trace/499b96822b8dda80 FAIL (2025-12-28 20:54:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12S7WBBQ1Nq8jbeGrZiRKTk23CdJC31hPW2vU1aaM5SsHeVyeRH, Piece ID: V4P3KCUDLI543TSROBYUMVTQKKSQDDVSKSJSXKEZTQAHTXUJ2G4Q) context deadline exceeded; piecestore: (Node ID: 1kjCEPtjvcJ3D7BBKRSRirPybBwkRZ1dJwyAQCadzCvaLYvWhz, Piece ID: 4OULHJ6Z3QJ3BOPDY74TJSKVJG23TOEUCJCUEGY24I5C3MUTAVLQ) context deadline exceeded; piecestore: (Node ID: 1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c, Piece ID: LIYXCPAZOUPKXLKRH3EOWREXUY2LAJXSRZFQILM2QIU7HTW7SY2Q) context deadline exceeded ======LOGS====== ue , address=145.79.227.23:4580 , piece_id=OHYI2SFM6TKJE4LHVNGO2PVGZDNFIQDAIQGKG4FOEHTTASBWA5ZQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.496428825 (557h3m1.36751187s) [noise=true , address=77.90.31.19:4867 , piece_id=LDAWD2ZFG3BZATDRY7MMNWYIX6TOHQ2OIUPCAWAYN266GNUUKZ5A, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.496861509 (557h3m1.367944544s) [noise=true , address=145.79.227.19:4105 , piece_id=RY35FSM6NTWPNF3JE43FZRSNXXYZPUN2B6LRNNUOG6QQR6XSU36Q, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.496899933 (557h3m1.367982978s) [noise=true , address=77.90.31.19:4858 , piece_id=LIYXCPAZOUPKXLKRH3EOWREXUY2LAJXSRZFQILM2QIU7HTW7SY2Q, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497229089 (557h3m1.368312134s) [noise=true , address=77.90.31.17:4784 , piece_id=JV2ROD6MN65B6LZKOL3S7ATJWXAY6FBQSZWITA6IHAXELY7R42KA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497326194 (557h3m1.368409239s) [noise=true , address=145.79.227.33:4904 , piece_id=QCMN4OSBWIWS4JRLIGQIGDSHZFUF2JWEPVXQ4FQ4G3LG5FBCBAWQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497664128 (557h3m1.368747173s) [noise=true , address=145.79.227.34:4918 , piece_id=SPDT2KI2774UJMQIUSVTHXCDF7PD2FBG64BXUYW67UMLPE6H7SBQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497731405 (557h3m1.36881445s) [noise=true , address=77.90.31.12:4602 , piece_id=2YPYKKTQ6ID6VTI4AUZQEG2SY7JYPRSAPU4JLF7DWSOSBBO74LWQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.498205939 (557h3m1.369288984s) [noise=true , address=77.90.31.16:4745 , piece_id=CZBFU33BHMP2BEZIDJKW3J526537HR6HP7OCT2KMAQVY7LGZLPZQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.498424295 (557h3m1.36950734s) [noise=true , address=77.90.31.21:4895 , piece_id=V4P3KCUDLI543TSROBYUMVTQKKSQDDVSKSJSXKEZTQAHTXUJ2G4Q, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.498685422 (557h3m1.369768457s) [noise=true , address=145.79.227.35:4496 , piece_id=JH4XQBT7XYEJXWBGPPF6SJWCPLV77GBOLGOA53C3WUQMWHZXVZQA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499066348 (557h3m1.370149393s) [noise=true , address=77.90.31.11:4563 , piece_id=DIFJPHSKHROTLKCHP6HQBWT7XGE77HVRZTH2354MV2UEWZLHVGCQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499185725 (557h3m1.37026877s) [noise=true , address=145.79.227.30:4768 , piece_id=KK4LEV6RG3IGLYXID5MJF3KMNSJ7WS5JS45PNOOJ26LXLYAUCWOA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499468122 (557h3m1.370551157s) [noise=true , address=77.90.31.14:4688 , piece_id=4OULHJ6Z3QJ3BOPDY74TJSKVJG23TOEUCJCUEGY24I5C3MUTAVLQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499901047 (557h3m1.370984102s) [noise=true , address=45.91.116.5:4330 , piece_id=EVHLSVVR5JGY43TLSKVEQLQ6KDBEZENVAFL6YT7ZQ77Y7X2TXIKA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.500546175 (557h3m1.37162921s) [noise=true , address=145.79.227.43:4171 , piece_id=YF5DRDPGRRVYFL5SWMH2PKTSV3C72BIWKUDSAIZTXOIVOTYKRMNA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:02.979572381 (557h3m2.850655436s) [upload=488362]: Next returned nil segment Dec 28 20:32:02.979609411 (557h3m2.850692456s) [upload=488362]: Waiting for error group managing segments... Dec 28 20:32:03.416896733 (557h3m3.287979778s) [noise=true , address=145.79.227.23:4580 , piece_id=OHYI2SFM6TKJE4LHVNGO2PVGZDNFIQDAIQGKG4FOEHTTASBWA5ZQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.552263655 (557h3m3.4233467s) [noise=true , address=77.90.31.17:4784 , piece_id=JV2ROD6MN65B6LZKOL3S7ATJWXAY6FBQSZWITA6IHAXELY7R42KA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.573564856 (557h3m3.444647902s) [noise=true , address=77.90.31.12:4602 , piece_id=2YPYKKTQ6ID6VTI4AUZQEG2SY7JYPRSAPU4JLF7DWSOSBBO74LWQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.598560923 (557h3m3.469643968s) [noise=true , address=77.90.31.19:4867 , piece_id=LDAWD2ZFG3BZATDRY7MMNWYIX6TOHQ2OIUPCAWAYN266GNUUKZ5A, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.626925925 (557h3m3.49800897s) [noise=true , address=77.90.31.16:4745 , piece_id=CZBFU33BHMP2BEZIDJKW3J526537HR6HP7OCT2KMAQVY7LGZLPZQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.709061947 (557h3m3.580144992s) [noise=true , address=77.90.31.14:4688 , piece_id=4OULHJ6Z3QJ3BOPDY74TJSKVJG23TOEUCJCUEGY24I5C3MUTAVLQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.788879373 (557h3m3.659962418s) [noise=true , address=77.90.31.21:4895 , piece_id=V4P3KCUDLI543TSROBYUMVTQKKSQDDVSKSJSXKEZTQAHTXUJ2G4Q, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.217616523 (557h3m4.088699569s) [noise=true , address=77.90.31.19:4858 , piece_id=LIYXCPAZOUPKXLKRH3EOWREXUY2LAJXSRZFQILM2QIU7HTW7SY2Q, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.220337625 (557h3m4.09142067s) [noise=true , address=77.90.31.11:4563 , piece_id=DIFJPHSKHROTLKCHP6HQBWT7XGE77HVRZTH2354MV2UEWZLHVGCQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.303958744 (557h3m4.175041789s) [noise=true , address=145.79.227.44:4213 , piece_id=UEAQDBBCP7DIG6ZD3EEM2CKAL4W5QKW23LWFUAMSVUW5ZS2PJDRQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.312794398 (557h3m4.183877444s) [noise=true , address=145.79.227.34:4918 , piece_id=SPDT2KI2774UJMQIUSVTHXCDF7PD2FBG64BXUYW67UMLPE6H7SBQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.349308798 (557h3m4.220391833s) [noise=true , address=145.79.227.40:4058 , piece_id=EQ4LA7T2BCGWY76K3NJI4X77OQJFJED6H4VNGXDQMILU6IVQWOMQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.371518379 (557h3m4.242601424s) [noise=true , address=145.79.227.43:4171 , piece_id=YF5DRDPGRRVYFL5SWMH2PKTSV3C72BIWKUDSAIZTXOIVOTYKRMNA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.376232316 (557h3m4.247315361s) [noise=true , address=145.79.227.30:4768 , piece_id=KK4LEV6RG3IGLYXID5MJF3KMNSJ7WS5JS45PNOOJ26LXLYAUCWOA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.376257844 (557h3m4.247340879s) [seg_pos={0 0}, upload=488362]: Segment reached optimal threshold of 14 pieces. Dec 28 20:32:04.376575799 (557h3m4.247658834s) [noise=true , address=145.79.227.33:4904 , piece_id=QCMN4OSBWIWS4JRLIGQIGDSHZFUF2JWEPVXQ4FQ4G3LG5FBCBAWQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1dC7mC2Cbxmys7ukEkELyF9Us7F1qtptSP6yDJh1EfjVJ6uqsC): protocol: expected piece hash; context canceled; EOF Dec 28 20:32:04.376585127 (557h3m4.247668163s) [noise=true , address=145.79.227.33:4904 , piece_id=QCMN4OSBWIWS4JRLIGQIGDSHZFUF2JWEPVXQ4FQ4G3LG5FBCBAWQ, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.376764359 (557h3m4.247847404s) [noise=true , address=145.79.227.19:4105 , piece_id=RY35FSM6NTWPNF3JE43FZRSNXXYZPUN2B6LRNNUOG6QQR6XSU36Q, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GhDrKMcDEg2RxamqKtiHphpgNdvtZYdHXrjzGsNJFjBFYeEen): protocol: write tcp 188.166.225.170:38838->145.79.227.19:4105: use of closed network connection; write tcp 188.166.225.170:38838->145.79.227.19:4105: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38838->145.79.227.19:4105: use of closed network connection Dec 28 20:32:04.376772865 (557h3m4.2478559s) [noise=true , address=145.79.227.19:4105 , piece_id=RY35FSM6NTWPNF3JE43FZRSNXXYZPUN2B6LRNNUOG6QQR6XSU36Q, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.376926658 (557h3m4.248009703s) [noise=true , address=145.79.227.35:4496 , piece_id=JH4XQBT7XYEJXWBGPPF6SJWCPLV77GBOLGOA53C3WUQMWHZXVZQA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V8XhSWkyGUVYFR8MykifFAP1YAfGCwzqucDcMMfzw3xsVW77Z): protocol: expected piece hash; context canceled; EOF Dec 28 20:32:04.377165332 (557h3m4.248248377s) [noise=true , address=145.79.227.35:4496 , piece_id=JH4XQBT7XYEJXWBGPPF6SJWCPLV77GBOLGOA53C3WUQMWHZXVZQA, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.377212682 (557h3m4.248295848s) [noise=true , address=45.91.116.5:4330 , piece_id=EVHLSVVR5JGY43TLSKVEQLQ6KDBEZENVAFL6YT7ZQ77Y7X2TXIKA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12MK9aVDRwfWPfckFR6RfPUL1Nrm9bwfsgV7TXCKGbtadHebWvW): protocol: read tcp 188.166.225.170:55416->45.91.116.5:4330: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 20:32:04.377289859 (557h3m4.248372924s) [noise=true , address=45.91.116.5:4330 , piece_id=EVHLSVVR5JGY43TLSKVEQLQ6KDBEZENVAFL6YT7ZQ77Y7X2TXIKA, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.377387736 (557h3m4.248470781s) [seg_pos={0 0}, upload=488362]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 20:32:04.377599729 (557h3m4.248682774s) [upload=488362]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 20:32:04.377613535 (557h3m4.24869662s) [upload=488362]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 20:32:04.675090855 (557h3m4.5461739s) [upload=488362]: Done uploading segments... Dec 28 20:32:04.675121233 (557h3m4.546204268s) [upload=488362]: Upload finished. err: <nil> Duration: 7.059757602s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 17.460093078s trace: https://jaeger.dev.storj.io/trace/499b96822b8dda80 FAIL (2025-12-28 20:53:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12S7WBBQ1Nq8jbeGrZiRKTk23CdJC31hPW2vU1aaM5SsHeVyeRH, Piece ID: V4P3KCUDLI543TSROBYUMVTQKKSQDDVSKSJSXKEZTQAHTXUJ2G4Q) context deadline exceeded; piecestore: (Node ID: 1kjCEPtjvcJ3D7BBKRSRirPybBwkRZ1dJwyAQCadzCvaLYvWhz, Piece ID: 4OULHJ6Z3QJ3BOPDY74TJSKVJG23TOEUCJCUEGY24I5C3MUTAVLQ) context deadline exceeded; piecestore: (Node ID: 1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c, Piece ID: LIYXCPAZOUPKXLKRH3EOWREXUY2LAJXSRZFQILM2QIU7HTW7SY2Q) context deadline exceeded ======LOGS====== ue , address=145.79.227.23:4580 , piece_id=OHYI2SFM6TKJE4LHVNGO2PVGZDNFIQDAIQGKG4FOEHTTASBWA5ZQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.496428825 (557h3m1.36751187s) [noise=true , address=77.90.31.19:4867 , piece_id=LDAWD2ZFG3BZATDRY7MMNWYIX6TOHQ2OIUPCAWAYN266GNUUKZ5A, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.496861509 (557h3m1.367944544s) [noise=true , address=145.79.227.19:4105 , piece_id=RY35FSM6NTWPNF3JE43FZRSNXXYZPUN2B6LRNNUOG6QQR6XSU36Q, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.496899933 (557h3m1.367982978s) [noise=true , address=77.90.31.19:4858 , piece_id=LIYXCPAZOUPKXLKRH3EOWREXUY2LAJXSRZFQILM2QIU7HTW7SY2Q, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497229089 (557h3m1.368312134s) [noise=true , address=77.90.31.17:4784 , piece_id=JV2ROD6MN65B6LZKOL3S7ATJWXAY6FBQSZWITA6IHAXELY7R42KA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497326194 (557h3m1.368409239s) [noise=true , address=145.79.227.33:4904 , piece_id=QCMN4OSBWIWS4JRLIGQIGDSHZFUF2JWEPVXQ4FQ4G3LG5FBCBAWQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497664128 (557h3m1.368747173s) [noise=true , address=145.79.227.34:4918 , piece_id=SPDT2KI2774UJMQIUSVTHXCDF7PD2FBG64BXUYW67UMLPE6H7SBQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497731405 (557h3m1.36881445s) [noise=true , address=77.90.31.12:4602 , piece_id=2YPYKKTQ6ID6VTI4AUZQEG2SY7JYPRSAPU4JLF7DWSOSBBO74LWQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.498205939 (557h3m1.369288984s) [noise=true , address=77.90.31.16:4745 , piece_id=CZBFU33BHMP2BEZIDJKW3J526537HR6HP7OCT2KMAQVY7LGZLPZQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.498424295 (557h3m1.36950734s) [noise=true , address=77.90.31.21:4895 , piece_id=V4P3KCUDLI543TSROBYUMVTQKKSQDDVSKSJSXKEZTQAHTXUJ2G4Q, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.498685422 (557h3m1.369768457s) [noise=true , address=145.79.227.35:4496 , piece_id=JH4XQBT7XYEJXWBGPPF6SJWCPLV77GBOLGOA53C3WUQMWHZXVZQA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499066348 (557h3m1.370149393s) [noise=true , address=77.90.31.11:4563 , piece_id=DIFJPHSKHROTLKCHP6HQBWT7XGE77HVRZTH2354MV2UEWZLHVGCQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499185725 (557h3m1.37026877s) [noise=true , address=145.79.227.30:4768 , piece_id=KK4LEV6RG3IGLYXID5MJF3KMNSJ7WS5JS45PNOOJ26LXLYAUCWOA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499468122 (557h3m1.370551157s) [noise=true , address=77.90.31.14:4688 , piece_id=4OULHJ6Z3QJ3BOPDY74TJSKVJG23TOEUCJCUEGY24I5C3MUTAVLQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499901047 (557h3m1.370984102s) [noise=true , address=45.91.116.5:4330 , piece_id=EVHLSVVR5JGY43TLSKVEQLQ6KDBEZENVAFL6YT7ZQ77Y7X2TXIKA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.500546175 (557h3m1.37162921s) [noise=true , address=145.79.227.43:4171 , piece_id=YF5DRDPGRRVYFL5SWMH2PKTSV3C72BIWKUDSAIZTXOIVOTYKRMNA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:02.979572381 (557h3m2.850655436s) [upload=488362]: Next returned nil segment Dec 28 20:32:02.979609411 (557h3m2.850692456s) [upload=488362]: Waiting for error group managing segments... Dec 28 20:32:03.416896733 (557h3m3.287979778s) [noise=true , address=145.79.227.23:4580 , piece_id=OHYI2SFM6TKJE4LHVNGO2PVGZDNFIQDAIQGKG4FOEHTTASBWA5ZQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.552263655 (557h3m3.4233467s) [noise=true , address=77.90.31.17:4784 , piece_id=JV2ROD6MN65B6LZKOL3S7ATJWXAY6FBQSZWITA6IHAXELY7R42KA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.573564856 (557h3m3.444647902s) [noise=true , address=77.90.31.12:4602 , piece_id=2YPYKKTQ6ID6VTI4AUZQEG2SY7JYPRSAPU4JLF7DWSOSBBO74LWQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.598560923 (557h3m3.469643968s) [noise=true , address=77.90.31.19:4867 , piece_id=LDAWD2ZFG3BZATDRY7MMNWYIX6TOHQ2OIUPCAWAYN266GNUUKZ5A, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.626925925 (557h3m3.49800897s) [noise=true , address=77.90.31.16:4745 , piece_id=CZBFU33BHMP2BEZIDJKW3J526537HR6HP7OCT2KMAQVY7LGZLPZQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.709061947 (557h3m3.580144992s) [noise=true , address=77.90.31.14:4688 , piece_id=4OULHJ6Z3QJ3BOPDY74TJSKVJG23TOEUCJCUEGY24I5C3MUTAVLQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.788879373 (557h3m3.659962418s) [noise=true , address=77.90.31.21:4895 , piece_id=V4P3KCUDLI543TSROBYUMVTQKKSQDDVSKSJSXKEZTQAHTXUJ2G4Q, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.217616523 (557h3m4.088699569s) [noise=true , address=77.90.31.19:4858 , piece_id=LIYXCPAZOUPKXLKRH3EOWREXUY2LAJXSRZFQILM2QIU7HTW7SY2Q, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.220337625 (557h3m4.09142067s) [noise=true , address=77.90.31.11:4563 , piece_id=DIFJPHSKHROTLKCHP6HQBWT7XGE77HVRZTH2354MV2UEWZLHVGCQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.303958744 (557h3m4.175041789s) [noise=true , address=145.79.227.44:4213 , piece_id=UEAQDBBCP7DIG6ZD3EEM2CKAL4W5QKW23LWFUAMSVUW5ZS2PJDRQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.312794398 (557h3m4.183877444s) [noise=true , address=145.79.227.34:4918 , piece_id=SPDT2KI2774UJMQIUSVTHXCDF7PD2FBG64BXUYW67UMLPE6H7SBQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.349308798 (557h3m4.220391833s) [noise=true , address=145.79.227.40:4058 , piece_id=EQ4LA7T2BCGWY76K3NJI4X77OQJFJED6H4VNGXDQMILU6IVQWOMQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.371518379 (557h3m4.242601424s) [noise=true , address=145.79.227.43:4171 , piece_id=YF5DRDPGRRVYFL5SWMH2PKTSV3C72BIWKUDSAIZTXOIVOTYKRMNA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.376232316 (557h3m4.247315361s) [noise=true , address=145.79.227.30:4768 , piece_id=KK4LEV6RG3IGLYXID5MJF3KMNSJ7WS5JS45PNOOJ26LXLYAUCWOA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.376257844 (557h3m4.247340879s) [seg_pos={0 0}, upload=488362]: Segment reached optimal threshold of 14 pieces. Dec 28 20:32:04.376575799 (557h3m4.247658834s) [noise=true , address=145.79.227.33:4904 , piece_id=QCMN4OSBWIWS4JRLIGQIGDSHZFUF2JWEPVXQ4FQ4G3LG5FBCBAWQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1dC7mC2Cbxmys7ukEkELyF9Us7F1qtptSP6yDJh1EfjVJ6uqsC): protocol: expected piece hash; context canceled; EOF Dec 28 20:32:04.376585127 (557h3m4.247668163s) [noise=true , address=145.79.227.33:4904 , piece_id=QCMN4OSBWIWS4JRLIGQIGDSHZFUF2JWEPVXQ4FQ4G3LG5FBCBAWQ, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.376764359 (557h3m4.247847404s) [noise=true , address=145.79.227.19:4105 , piece_id=RY35FSM6NTWPNF3JE43FZRSNXXYZPUN2B6LRNNUOG6QQR6XSU36Q, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GhDrKMcDEg2RxamqKtiHphpgNdvtZYdHXrjzGsNJFjBFYeEen): protocol: write tcp 188.166.225.170:38838->145.79.227.19:4105: use of closed network connection; write tcp 188.166.225.170:38838->145.79.227.19:4105: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38838->145.79.227.19:4105: use of closed network connection Dec 28 20:32:04.376772865 (557h3m4.2478559s) [noise=true , address=145.79.227.19:4105 , piece_id=RY35FSM6NTWPNF3JE43FZRSNXXYZPUN2B6LRNNUOG6QQR6XSU36Q, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.376926658 (557h3m4.248009703s) [noise=true , address=145.79.227.35:4496 , piece_id=JH4XQBT7XYEJXWBGPPF6SJWCPLV77GBOLGOA53C3WUQMWHZXVZQA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V8XhSWkyGUVYFR8MykifFAP1YAfGCwzqucDcMMfzw3xsVW77Z): protocol: expected piece hash; context canceled; EOF Dec 28 20:32:04.377165332 (557h3m4.248248377s) [noise=true , address=145.79.227.35:4496 , piece_id=JH4XQBT7XYEJXWBGPPF6SJWCPLV77GBOLGOA53C3WUQMWHZXVZQA, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.377212682 (557h3m4.248295848s) [noise=true , address=45.91.116.5:4330 , piece_id=EVHLSVVR5JGY43TLSKVEQLQ6KDBEZENVAFL6YT7ZQ77Y7X2TXIKA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12MK9aVDRwfWPfckFR6RfPUL1Nrm9bwfsgV7TXCKGbtadHebWvW): protocol: read tcp 188.166.225.170:55416->45.91.116.5:4330: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 20:32:04.377289859 (557h3m4.248372924s) [noise=true , address=45.91.116.5:4330 , piece_id=EVHLSVVR5JGY43TLSKVEQLQ6KDBEZENVAFL6YT7ZQ77Y7X2TXIKA, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.377387736 (557h3m4.248470781s) [seg_pos={0 0}, upload=488362]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 20:32:04.377599729 (557h3m4.248682774s) [upload=488362]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 20:32:04.377613535 (557h3m4.24869662s) [upload=488362]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 20:32:04.675090855 (557h3m4.5461739s) [upload=488362]: Done uploading segments... Dec 28 20:32:04.675121233 (557h3m4.546204268s) [upload=488362]: Upload finished. err: <nil> Duration: 17.460121522s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.....X...........X..................X.......X......XX.......] duration: 6.783748503s trace: https://jaeger.dev.storj.io/trace/3ce821e76930523a FAIL (2025-12-28 20:53:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 20:49:03.805105648 (557h20m3.676188693s) [upload=488615]: Starting upload Dec 28 20:49:03.805166324 (557h20m3.676249369s) [upload=488615]: Done starting upload Dec 28 20:49:03.805293475 (557h20m3.67637651s) [upload=488615]: Uploading segments... Dec 28 20:49:03.805403545 (557h20m3.67648658s) [upload=488615]: Got next segment. Inline: false Dec 28 20:49:03.805460854 (557h20m3.676543889s) [upload=488615]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 20:49:03.805467066 (557h20m3.676550111s) [upload=488615]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 20:49:04.052770455 (557h20m3.9238535s) [seg_pos={0 0}, upload=488615]: Begin upload segment... Dec 28 20:49:04.052835940 (557h20m3.923918985s) [seg_pos={0 0}, upload=488615]: Done begin upload segment. Dec 28 20:49:04.052874203 (557h20m3.923957248s) [noise=true , address=145.79.227.44:4194 , piece_id=WYODZVE7TGDTRNVGZ5HAD2JJT3TSOB7KGNGWH5VCQMLCCICMORRA, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.052970917 (557h20m3.924053962s) [noise=true , address=145.79.227.39:4015 , piece_id=UNPAANTMSZ3O356MXFAX357E3L4X5OLMJBXTDQEVGXUH5B5KDYIQ, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.053474816 (557h20m3.924557861s) [noise=true , address=45.91.116.8:4046 , piece_id=2DXIM4FJEZZQ3LRR4GFUEJ6XD6IGDET3JVKWBZJQGKMICWUITTTA, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.053938549 (557h20m3.925021584s) [noise=true , address=145.79.227.37:4565 , piece_id=6EAD5WOUCVJXUOKHGTSMMHDYPKHHJFZZFIXH2FRUNNZTZNQNK5IA, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.054376714 (557h20m3.925459759s) [noise=true , address=145.79.227.19:4078 , piece_id=DCMFZ42KOAHHMA3YOVW7655PLKCMJDHTLETUU4NK2D6EE27TEKWQ, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.055140448 (557h20m3.926223493s) [noise=true , address=145.79.227.36:4519 , piece_id=GQENJD2GLGJSL7H7VWUN6JXBZTW6ZHFLMMYTKYF3TRBGNNBQC3YQ, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.055719992 (557h20m3.926803027s) [noise=true , address=145.79.227.51:4411 , piece_id=IMS72BZZCJ3WMXU3FTBKM3F4MSXTL4VCVCPOCNLQI6WHKJGIOXQQ, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.056924334 (557h20m3.92800737s) [noise=true , address=45.91.116.6:4973 , piece_id=HMTEGMPYPTWCLRSAH45LLVZTTG2TRVUS44MTKBPZT6SIMRFYYNGQ, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.058635102 (557h20m3.929718147s) [noise=true , address=145.79.227.13:4804 , piece_id=UYO4W6FOSDQSVVEO6XWVTGXMFLXYWNYA3DMICFUABVSFFDKPFRXA, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.058693383 (557h20m3.929776418s) [noise=true , address=45.91.116.5:4299 , piece_id=HI6B35X52HVYVDIFVSJQT6TXYBBDA5VB5COKH53CPKDTO3UHKWJA, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.061111298 (557h20m3.932194343s) [noise=true , address=77.90.31.14:4661 , piece_id=OP5ES2G4MLZBJWJDYYPO7KLHUC6NDX5FUF6ZO6PCYMCFSFTKPVDA, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.061856947 (557h20m3.932939992s) [noise=true , address=145.79.227.42:4123 , piece_id=6YEKCEU5A5MPTRVGRFH2GWQJVSWMBXHCIZSAJKPUY2M2I3P63SFQ, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.064452048 (557h20m3.935535093s) [noise=true , address=45.91.116.10:4082 , piece_id=WGN35FKEXP3UEANOQA4NSJUQLDKIZ36RYD6OFP2FHEBNJKLI2TTA, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.066246636 (557h20m3.937329681s) [noise=true , address=145.79.227.5:4221 , piece_id=GVQO2XLQ2D6XVAHLGZVB6SHA476DL3TUHAXLFKZU5H2BN3LEB5XQ, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.069098717 (557h20m3.940181762s) [noise=true , address=145.79.227.38:4977 , piece_id=23SZDYDT2OWMLO7EGT6OT5L4S4WYR5LY2SWCCZGSCZMKQ5RK6XPQ, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.070882644 (557h20m3.941965689s) [noise=true , address=145.79.227.32:4834 , piece_id=52YM22AS3BG3OGILTWR36HSHD5SKNRKKUR5TIE7HLWOSK3L6KQOQ, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:05.778149774 (557h20m5.649232819s) [upload=488615]: Next returned nil segment Dec 28 20:49:05.778171825 (557h20m5.64925487s) [upload=488615]: Waiting for error group managing segments... Dec 28 20:49:06.067849953 (557h20m5.938932998s) [noise=true , address=145.79.227.37:4565 , piece_id=6EAD5WOUCVJXUOKHGTSMMHDYPKHHJFZZFIXH2FRUNNZTZNQNK5IA, seg_pos={0 0}, upload=488615]: Done uploading piece. err: <nil> Dec 28 20:49:06.073409289 (557h20m5.944492334s) [noise=true , address=145.79.227.39:4015 , piece_id=UNPAANTMSZ3O356MXFAX357E3L4X5OLMJBXTDQEVGXUH5B5KDYIQ, seg_pos={0 0}, upload=488615]: Done uploading piece. err: <nil> Dec 28 20:49:06.078797668 (557h20m5.949880714s) [noise=true , address=145.79.227.13:4804 , piece_id=UYO4W6FOSDQSVVEO6XWVTGXMFLXYWNYA3DMICFUABVSFFDKPFRXA, seg_pos={0 0}, upload=488615]: Done uploading piece. err: <nil> Dec 28 20:49:06.084142657 (557h20m5.955225692s) [noise=true , address=145.79.227.32:4834 , piece_id=52YM22AS3BG3OGILTWR36HSHD5SKNRKKUR5TIE7HLWOSK3L6KQOQ, seg_pos={0 0}, upload=488615]: Done uploading piece. err: <nil> Dec 28 20:49:06.084205807 (557h20m5.955288842s) [noise=true , address=145.79.227.19:4078 , piece_id=DCMFZ42KOAHHMA3YOVW7655PLKCMJDHTLETUU4NK2D6EE27TEKWQ, seg_pos={0 0}, upload=488615]: Done uploading piece. err: <nil> Dec 28 20:49:06.085908499 (557h20m5.956991534s) [noise=true , address=145.79.227.51:4411 , piece_id=IMS72BZZCJ3WMXU3FTBKM3F4MSXTL4VCVCPOCNLQI6WHKJGIOXQQ, seg_pos={0 0}, upload=488615]: Done uploading piece. err: <nil> Dec 28 20:49:06.089752589 (557h20m5.960835624s) [noise=true , address=145.79.227.5:4221 , piece_id=GVQO2XLQ2D6XVAHLGZVB6SHA476DL3TUHAXLFKZU5H2BN3LEB5XQ, seg_pos={0 0}, upload=488615]: Done uploading piece. err: <nil> Dec 28 20:49:06.096096639 (557h20m5.967179674s) [noise=true , address=145.79.227.36:4519 , piece_id=GQENJD2GLGJSL7H7VWUN6JXBZTW6ZHFLMMYTKYF3TRBGNNBQC3YQ, seg_pos={0 0}, upload=488615]: Done uploading piece. err: <nil> Dec 28 20:49:06.097668182 (557h20m5.968751227s) [noise=true , address=145.79.227.42:4123 , piece_id=6YEKCEU5A5MPTRVGRFH2GWQJVSWMBXHCIZSAJKPUY2M2I3P63SFQ, seg_pos={0 0}, upload=488615]: Done uploading piece. err: <nil> Dec 28 20:49:06.107376347 (557h20m5.978459392s) [noise=true , address=145.79.227.38:4977 , piece_id=23SZDYDT2OWMLO7EGT6OT5L4S4WYR5LY2SWCCZGSCZMKQ5RK6XPQ, seg_pos={0 0}, upload=488615]: Done uploading piece. err: <nil> Dec 28 20:49:06.255907707 (557h20m6.126990752s) [noise=true , address=77.90.31.14:4661 , piece_id=OP5ES2G4MLZBJWJDYYPO7KLHUC6NDX5FUF6ZO6PCYMCFSFTKPVDA, seg_pos={0 0}, upload=488615]: Done uploading piece. err: <nil> Dec 28 20:49:06.306350829 (557h20m6.177433874s) [noise=true , address=45.91.116.5:4299 , piece_id=HI6B35X52HVYVDIFVSJQT6TXYBBDA5VB5COKH53CPKDTO3UHKWJA, seg_pos={0 0}, upload=488615]: Done uploading piece. err: <nil> Dec 28 20:49:06.312533491 (557h20m6.183616536s) [noise=true , address=45.91.116.8:4046 , piece_id=2DXIM4FJEZZQ3LRR4GFUEJ6XD6IGDET3JVKWBZJQGKMICWUITTTA, seg_pos={0 0}, upload=488615]: Done uploading piece. err: <nil> Dec 28 20:49:06.312558800 (557h20m6.183641845s) [seg_pos={0 0}, upload=488615]: Segment reached optimal threshold of 13 pieces. Dec 28 20:49:06.313074843 (557h20m6.184157888s) [noise=true , address=145.79.227.44:4194 , piece_id=WYODZVE7TGDTRNVGZ5HAD2JJT3TSOB7KGNGWH5VCQMLCCICMORRA, seg_pos={0 0}, upload=488615]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 28 20:49:06.313086825 (557h20m6.18416987s) [noise=true , address=145.79.227.44:4194 , piece_id=WYODZVE7TGDTRNVGZ5HAD2JJT3TSOB7KGNGWH5VCQMLCCICMORRA, seg_pos={0 0}, upload=488615]: Long tail cancellation due to optimal threshold Dec 28 20:49:06.313142812 (557h20m6.184225847s) [noise=true , address=45.91.116.6:4973 , piece_id=HMTEGMPYPTWCLRSAH45LLVZTTG2TRVUS44MTKBPZT6SIMRFYYNGQ, seg_pos={0 0}, upload=488615]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF Dec 28 20:49:06.313177759 (557h20m6.184260804s) [noise=true , address=45.91.116.6:4973 , piece_id=HMTEGMPYPTWCLRSAH45LLVZTTG2TRVUS44MTKBPZT6SIMRFYYNGQ, seg_pos={0 0}, upload=488615]: Long tail cancellation due to optimal threshold Dec 28 20:49:06.313256448 (557h20m6.184339503s) [noise=true , address=45.91.116.10:4082 , piece_id=WGN35FKEXP3UEANOQA4NSJUQLDKIZ36RYD6OFP2FHEBNJKLI2TTA, seg_pos={0 0}, upload=488615]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 28 20:49:06.313271217 (557h20m6.184354262s) [noise=true , address=45.91.116.10:4082 , piece_id=WGN35FKEXP3UEANOQA4NSJUQLDKIZ36RYD6OFP2FHEBNJKLI2TTA, seg_pos={0 0}, upload=488615]: Long tail cancellation due to optimal threshold Dec 28 20:49:06.313296314 (557h20m6.184379359s) [seg_pos={0 0}, upload=488615]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 20:49:06.313381627 (557h20m6.184464672s) [upload=488615]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 20:49:06.313389141 (557h20m6.184472186s) [upload=488615]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 20:49:06.604901881 (557h20m6.475984916s) [upload=488615]: Done uploading segments... Dec 28 20:49:06.604926539 (557h20m6.476009584s) [upload=488615]: Upload finished. err: <nil> Duration: 6.783868832s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [......X...........X..................X.......X......XX......] duration: 5.858878621s trace: https://jaeger.dev.storj.io/trace/3ce821e76930523a FAIL (2025-12-28 20:53:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 20:49:03.805105648 (557h20m3.676188693s) [upload=488615]: Starting upload Dec 28 20:49:03.805166324 (557h20m3.676249369s) [upload=488615]: Done starting upload Dec 28 20:49:03.805293475 (557h20m3.67637651s) [upload=488615]: Uploading segments... Dec 28 20:49:03.805403545 (557h20m3.67648658s) [upload=488615]: Got next segment. Inline: false Dec 28 20:49:03.805460854 (557h20m3.676543889s) [upload=488615]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 20:49:03.805467066 (557h20m3.676550111s) [upload=488615]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 20:49:04.052770455 (557h20m3.9238535s) [seg_pos={0 0}, upload=488615]: Begin upload segment... Dec 28 20:49:04.052835940 (557h20m3.923918985s) [seg_pos={0 0}, upload=488615]: Done begin upload segment. Dec 28 20:49:04.052874203 (557h20m3.923957248s) [noise=true , address=145.79.227.44:4194 , piece_id=WYODZVE7TGDTRNVGZ5HAD2JJT3TSOB7KGNGWH5VCQMLCCICMORRA, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.052970917 (557h20m3.924053962s) [noise=true , address=145.79.227.39:4015 , piece_id=UNPAANTMSZ3O356MXFAX357E3L4X5OLMJBXTDQEVGXUH5B5KDYIQ, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.053474816 (557h20m3.924557861s) [noise=true , address=45.91.116.8:4046 , piece_id=2DXIM4FJEZZQ3LRR4GFUEJ6XD6IGDET3JVKWBZJQGKMICWUITTTA, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.053938549 (557h20m3.925021584s) [noise=true , address=145.79.227.37:4565 , piece_id=6EAD5WOUCVJXUOKHGTSMMHDYPKHHJFZZFIXH2FRUNNZTZNQNK5IA, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.054376714 (557h20m3.925459759s) [noise=true , address=145.79.227.19:4078 , piece_id=DCMFZ42KOAHHMA3YOVW7655PLKCMJDHTLETUU4NK2D6EE27TEKWQ, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.055140448 (557h20m3.926223493s) [noise=true , address=145.79.227.36:4519 , piece_id=GQENJD2GLGJSL7H7VWUN6JXBZTW6ZHFLMMYTKYF3TRBGNNBQC3YQ, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.055719992 (557h20m3.926803027s) [noise=true , address=145.79.227.51:4411 , piece_id=IMS72BZZCJ3WMXU3FTBKM3F4MSXTL4VCVCPOCNLQI6WHKJGIOXQQ, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.056924334 (557h20m3.92800737s) [noise=true , address=45.91.116.6:4973 , piece_id=HMTEGMPYPTWCLRSAH45LLVZTTG2TRVUS44MTKBPZT6SIMRFYYNGQ, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.058635102 (557h20m3.929718147s) [noise=true , address=145.79.227.13:4804 , piece_id=UYO4W6FOSDQSVVEO6XWVTGXMFLXYWNYA3DMICFUABVSFFDKPFRXA, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.058693383 (557h20m3.929776418s) [noise=true , address=45.91.116.5:4299 , piece_id=HI6B35X52HVYVDIFVSJQT6TXYBBDA5VB5COKH53CPKDTO3UHKWJA, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.061111298 (557h20m3.932194343s) [noise=true , address=77.90.31.14:4661 , piece_id=OP5ES2G4MLZBJWJDYYPO7KLHUC6NDX5FUF6ZO6PCYMCFSFTKPVDA, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.061856947 (557h20m3.932939992s) [noise=true , address=145.79.227.42:4123 , piece_id=6YEKCEU5A5MPTRVGRFH2GWQJVSWMBXHCIZSAJKPUY2M2I3P63SFQ, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.064452048 (557h20m3.935535093s) [noise=true , address=45.91.116.10:4082 , piece_id=WGN35FKEXP3UEANOQA4NSJUQLDKIZ36RYD6OFP2FHEBNJKLI2TTA, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.066246636 (557h20m3.937329681s) [noise=true , address=145.79.227.5:4221 , piece_id=GVQO2XLQ2D6XVAHLGZVB6SHA476DL3TUHAXLFKZU5H2BN3LEB5XQ, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.069098717 (557h20m3.940181762s) [noise=true , address=145.79.227.38:4977 , piece_id=23SZDYDT2OWMLO7EGT6OT5L4S4WYR5LY2SWCCZGSCZMKQ5RK6XPQ, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.070882644 (557h20m3.941965689s) [noise=true , address=145.79.227.32:4834 , piece_id=52YM22AS3BG3OGILTWR36HSHD5SKNRKKUR5TIE7HLWOSK3L6KQOQ, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:05.778149774 (557h20m5.649232819s) [upload=488615]: Next returned nil segment Dec 28 20:49:05.778171825 (557h20m5.64925487s) [upload=488615]: Waiting for error group managing segments... Dec 28 20:49:06.067849953 (557h20m5.938932998s) [noise=true , address=145.79.227.37:4565 , piece_id=6EAD5WOUCVJXUOKHGTSMMHDYPKHHJFZZFIXH2FRUNNZTZNQNK5IA, seg_pos={0 0}, upload=488615]: Done uploading piece. err: <nil> Dec 28 20:49:06.073409289 (557h20m5.944492334s) [noise=true , address=145.79.227.39:4015 , piece_id=UNPAANTMSZ3O356MXFAX357E3L4X5OLMJBXTDQEVGXUH5B5KDYIQ, seg_pos={0 0}, upload=488615]: Done uploading piece. err: <nil> Dec 28 20:49:06.078797668 (557h20m5.949880714s) [noise=true , address=145.79.227.13:4804 , piece_id=UYO4W6FOSDQSVVEO6XWVTGXMFLXYWNYA3DMICFUABVSFFDKPFRXA, seg_pos={0 0}, upload=488615]: Done uploading piece. err: <nil> Dec 28 20:49:06.084142657 (557h20m5.955225692s) [noise=true , address=145.79.227.32:4834 , piece_id=52YM22AS3BG3OGILTWR36HSHD5SKNRKKUR5TIE7HLWOSK3L6KQOQ, seg_pos={0 0}, upload=488615]: Done uploading piece. err: <nil> Dec 28 20:49:06.084205807 (557h20m5.955288842s) [noise=true , address=145.79.227.19:4078 , piece_id=DCMFZ42KOAHHMA3YOVW7655PLKCMJDHTLETUU4NK2D6EE27TEKWQ, seg_pos={0 0}, upload=488615]: Done uploading piece. err: <nil> Dec 28 20:49:06.085908499 (557h20m5.956991534s) [noise=true , address=145.79.227.51:4411 , piece_id=IMS72BZZCJ3WMXU3FTBKM3F4MSXTL4VCVCPOCNLQI6WHKJGIOXQQ, seg_pos={0 0}, upload=488615]: Done uploading piece. err: <nil> Dec 28 20:49:06.089752589 (557h20m5.960835624s) [noise=true , address=145.79.227.5:4221 , piece_id=GVQO2XLQ2D6XVAHLGZVB6SHA476DL3TUHAXLFKZU5H2BN3LEB5XQ, seg_pos={0 0}, upload=488615]: Done uploading piece. err: <nil> Dec 28 20:49:06.096096639 (557h20m5.967179674s) [noise=true , address=145.79.227.36:4519 , piece_id=GQENJD2GLGJSL7H7VWUN6JXBZTW6ZHFLMMYTKYF3TRBGNNBQC3YQ, seg_pos={0 0}, upload=488615]: Done uploading piece. err: <nil> Dec 28 20:49:06.097668182 (557h20m5.968751227s) [noise=true , address=145.79.227.42:4123 , piece_id=6YEKCEU5A5MPTRVGRFH2GWQJVSWMBXHCIZSAJKPUY2M2I3P63SFQ, seg_pos={0 0}, upload=488615]: Done uploading piece. err: <nil> Dec 28 20:49:06.107376347 (557h20m5.978459392s) [noise=true , address=145.79.227.38:4977 , piece_id=23SZDYDT2OWMLO7EGT6OT5L4S4WYR5LY2SWCCZGSCZMKQ5RK6XPQ, seg_pos={0 0}, upload=488615]: Done uploading piece. err: <nil> Dec 28 20:49:06.255907707 (557h20m6.126990752s) [noise=true , address=77.90.31.14:4661 , piece_id=OP5ES2G4MLZBJWJDYYPO7KLHUC6NDX5FUF6ZO6PCYMCFSFTKPVDA, seg_pos={0 0}, upload=488615]: Done uploading piece. err: <nil> Dec 28 20:49:06.306350829 (557h20m6.177433874s) [noise=true , address=45.91.116.5:4299 , piece_id=HI6B35X52HVYVDIFVSJQT6TXYBBDA5VB5COKH53CPKDTO3UHKWJA, seg_pos={0 0}, upload=488615]: Done uploading piece. err: <nil> Dec 28 20:49:06.312533491 (557h20m6.183616536s) [noise=true , address=45.91.116.8:4046 , piece_id=2DXIM4FJEZZQ3LRR4GFUEJ6XD6IGDET3JVKWBZJQGKMICWUITTTA, seg_pos={0 0}, upload=488615]: Done uploading piece. err: <nil> Dec 28 20:49:06.312558800 (557h20m6.183641845s) [seg_pos={0 0}, upload=488615]: Segment reached optimal threshold of 13 pieces. Dec 28 20:49:06.313074843 (557h20m6.184157888s) [noise=true , address=145.79.227.44:4194 , piece_id=WYODZVE7TGDTRNVGZ5HAD2JJT3TSOB7KGNGWH5VCQMLCCICMORRA, seg_pos={0 0}, upload=488615]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 28 20:49:06.313086825 (557h20m6.18416987s) [noise=true , address=145.79.227.44:4194 , piece_id=WYODZVE7TGDTRNVGZ5HAD2JJT3TSOB7KGNGWH5VCQMLCCICMORRA, seg_pos={0 0}, upload=488615]: Long tail cancellation due to optimal threshold Dec 28 20:49:06.313142812 (557h20m6.184225847s) [noise=true , address=45.91.116.6:4973 , piece_id=HMTEGMPYPTWCLRSAH45LLVZTTG2TRVUS44MTKBPZT6SIMRFYYNGQ, seg_pos={0 0}, upload=488615]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF Dec 28 20:49:06.313177759 (557h20m6.184260804s) [noise=true , address=45.91.116.6:4973 , piece_id=HMTEGMPYPTWCLRSAH45LLVZTTG2TRVUS44MTKBPZT6SIMRFYYNGQ, seg_pos={0 0}, upload=488615]: Long tail cancellation due to optimal threshold Dec 28 20:49:06.313256448 (557h20m6.184339503s) [noise=true , address=45.91.116.10:4082 , piece_id=WGN35FKEXP3UEANOQA4NSJUQLDKIZ36RYD6OFP2FHEBNJKLI2TTA, seg_pos={0 0}, upload=488615]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 28 20:49:06.313271217 (557h20m6.184354262s) [noise=true , address=45.91.116.10:4082 , piece_id=WGN35FKEXP3UEANOQA4NSJUQLDKIZ36RYD6OFP2FHEBNJKLI2TTA, seg_pos={0 0}, upload=488615]: Long tail cancellation due to optimal threshold Dec 28 20:49:06.313296314 (557h20m6.184379359s) [seg_pos={0 0}, upload=488615]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 20:49:06.313381627 (557h20m6.184464672s) [upload=488615]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 20:49:06.313389141 (557h20m6.184472186s) [upload=488615]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 20:49:06.604901881 (557h20m6.475984916s) [upload=488615]: Done uploading segments... Dec 28 20:49:06.604926539 (557h20m6.476009584s) [upload=488615]: Upload finished. err: <nil> Duration: 5.858908078s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 10.647240375s trace: https://jaeger.dev.storj.io/trace/499b96822b8dda80 FAIL (2025-12-28 20:53:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12S7WBBQ1Nq8jbeGrZiRKTk23CdJC31hPW2vU1aaM5SsHeVyeRH, Piece ID: V4P3KCUDLI543TSROBYUMVTQKKSQDDVSKSJSXKEZTQAHTXUJ2G4Q) context deadline exceeded; piecestore: (Node ID: 1kjCEPtjvcJ3D7BBKRSRirPybBwkRZ1dJwyAQCadzCvaLYvWhz, Piece ID: 4OULHJ6Z3QJ3BOPDY74TJSKVJG23TOEUCJCUEGY24I5C3MUTAVLQ) context deadline exceeded; piecestore: (Node ID: 1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c, Piece ID: LIYXCPAZOUPKXLKRH3EOWREXUY2LAJXSRZFQILM2QIU7HTW7SY2Q) context deadline exceeded ======LOGS====== ue , address=145.79.227.23:4580 , piece_id=OHYI2SFM6TKJE4LHVNGO2PVGZDNFIQDAIQGKG4FOEHTTASBWA5ZQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.496428825 (557h3m1.36751187s) [noise=true , address=77.90.31.19:4867 , piece_id=LDAWD2ZFG3BZATDRY7MMNWYIX6TOHQ2OIUPCAWAYN266GNUUKZ5A, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.496861509 (557h3m1.367944544s) [noise=true , address=145.79.227.19:4105 , piece_id=RY35FSM6NTWPNF3JE43FZRSNXXYZPUN2B6LRNNUOG6QQR6XSU36Q, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.496899933 (557h3m1.367982978s) [noise=true , address=77.90.31.19:4858 , piece_id=LIYXCPAZOUPKXLKRH3EOWREXUY2LAJXSRZFQILM2QIU7HTW7SY2Q, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497229089 (557h3m1.368312134s) [noise=true , address=77.90.31.17:4784 , piece_id=JV2ROD6MN65B6LZKOL3S7ATJWXAY6FBQSZWITA6IHAXELY7R42KA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497326194 (557h3m1.368409239s) [noise=true , address=145.79.227.33:4904 , piece_id=QCMN4OSBWIWS4JRLIGQIGDSHZFUF2JWEPVXQ4FQ4G3LG5FBCBAWQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497664128 (557h3m1.368747173s) [noise=true , address=145.79.227.34:4918 , piece_id=SPDT2KI2774UJMQIUSVTHXCDF7PD2FBG64BXUYW67UMLPE6H7SBQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497731405 (557h3m1.36881445s) [noise=true , address=77.90.31.12:4602 , piece_id=2YPYKKTQ6ID6VTI4AUZQEG2SY7JYPRSAPU4JLF7DWSOSBBO74LWQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.498205939 (557h3m1.369288984s) [noise=true , address=77.90.31.16:4745 , piece_id=CZBFU33BHMP2BEZIDJKW3J526537HR6HP7OCT2KMAQVY7LGZLPZQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.498424295 (557h3m1.36950734s) [noise=true , address=77.90.31.21:4895 , piece_id=V4P3KCUDLI543TSROBYUMVTQKKSQDDVSKSJSXKEZTQAHTXUJ2G4Q, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.498685422 (557h3m1.369768457s) [noise=true , address=145.79.227.35:4496 , piece_id=JH4XQBT7XYEJXWBGPPF6SJWCPLV77GBOLGOA53C3WUQMWHZXVZQA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499066348 (557h3m1.370149393s) [noise=true , address=77.90.31.11:4563 , piece_id=DIFJPHSKHROTLKCHP6HQBWT7XGE77HVRZTH2354MV2UEWZLHVGCQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499185725 (557h3m1.37026877s) [noise=true , address=145.79.227.30:4768 , piece_id=KK4LEV6RG3IGLYXID5MJF3KMNSJ7WS5JS45PNOOJ26LXLYAUCWOA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499468122 (557h3m1.370551157s) [noise=true , address=77.90.31.14:4688 , piece_id=4OULHJ6Z3QJ3BOPDY74TJSKVJG23TOEUCJCUEGY24I5C3MUTAVLQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499901047 (557h3m1.370984102s) [noise=true , address=45.91.116.5:4330 , piece_id=EVHLSVVR5JGY43TLSKVEQLQ6KDBEZENVAFL6YT7ZQ77Y7X2TXIKA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.500546175 (557h3m1.37162921s) [noise=true , address=145.79.227.43:4171 , piece_id=YF5DRDPGRRVYFL5SWMH2PKTSV3C72BIWKUDSAIZTXOIVOTYKRMNA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:02.979572381 (557h3m2.850655436s) [upload=488362]: Next returned nil segment Dec 28 20:32:02.979609411 (557h3m2.850692456s) [upload=488362]: Waiting for error group managing segments... Dec 28 20:32:03.416896733 (557h3m3.287979778s) [noise=true , address=145.79.227.23:4580 , piece_id=OHYI2SFM6TKJE4LHVNGO2PVGZDNFIQDAIQGKG4FOEHTTASBWA5ZQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.552263655 (557h3m3.4233467s) [noise=true , address=77.90.31.17:4784 , piece_id=JV2ROD6MN65B6LZKOL3S7ATJWXAY6FBQSZWITA6IHAXELY7R42KA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.573564856 (557h3m3.444647902s) [noise=true , address=77.90.31.12:4602 , piece_id=2YPYKKTQ6ID6VTI4AUZQEG2SY7JYPRSAPU4JLF7DWSOSBBO74LWQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.598560923 (557h3m3.469643968s) [noise=true , address=77.90.31.19:4867 , piece_id=LDAWD2ZFG3BZATDRY7MMNWYIX6TOHQ2OIUPCAWAYN266GNUUKZ5A, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.626925925 (557h3m3.49800897s) [noise=true , address=77.90.31.16:4745 , piece_id=CZBFU33BHMP2BEZIDJKW3J526537HR6HP7OCT2KMAQVY7LGZLPZQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.709061947 (557h3m3.580144992s) [noise=true , address=77.90.31.14:4688 , piece_id=4OULHJ6Z3QJ3BOPDY74TJSKVJG23TOEUCJCUEGY24I5C3MUTAVLQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.788879373 (557h3m3.659962418s) [noise=true , address=77.90.31.21:4895 , piece_id=V4P3KCUDLI543TSROBYUMVTQKKSQDDVSKSJSXKEZTQAHTXUJ2G4Q, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.217616523 (557h3m4.088699569s) [noise=true , address=77.90.31.19:4858 , piece_id=LIYXCPAZOUPKXLKRH3EOWREXUY2LAJXSRZFQILM2QIU7HTW7SY2Q, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.220337625 (557h3m4.09142067s) [noise=true , address=77.90.31.11:4563 , piece_id=DIFJPHSKHROTLKCHP6HQBWT7XGE77HVRZTH2354MV2UEWZLHVGCQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.303958744 (557h3m4.175041789s) [noise=true , address=145.79.227.44:4213 , piece_id=UEAQDBBCP7DIG6ZD3EEM2CKAL4W5QKW23LWFUAMSVUW5ZS2PJDRQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.312794398 (557h3m4.183877444s) [noise=true , address=145.79.227.34:4918 , piece_id=SPDT2KI2774UJMQIUSVTHXCDF7PD2FBG64BXUYW67UMLPE6H7SBQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.349308798 (557h3m4.220391833s) [noise=true , address=145.79.227.40:4058 , piece_id=EQ4LA7T2BCGWY76K3NJI4X77OQJFJED6H4VNGXDQMILU6IVQWOMQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.371518379 (557h3m4.242601424s) [noise=true , address=145.79.227.43:4171 , piece_id=YF5DRDPGRRVYFL5SWMH2PKTSV3C72BIWKUDSAIZTXOIVOTYKRMNA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.376232316 (557h3m4.247315361s) [noise=true , address=145.79.227.30:4768 , piece_id=KK4LEV6RG3IGLYXID5MJF3KMNSJ7WS5JS45PNOOJ26LXLYAUCWOA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.376257844 (557h3m4.247340879s) [seg_pos={0 0}, upload=488362]: Segment reached optimal threshold of 14 pieces. Dec 28 20:32:04.376575799 (557h3m4.247658834s) [noise=true , address=145.79.227.33:4904 , piece_id=QCMN4OSBWIWS4JRLIGQIGDSHZFUF2JWEPVXQ4FQ4G3LG5FBCBAWQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1dC7mC2Cbxmys7ukEkELyF9Us7F1qtptSP6yDJh1EfjVJ6uqsC): protocol: expected piece hash; context canceled; EOF Dec 28 20:32:04.376585127 (557h3m4.247668163s) [noise=true , address=145.79.227.33:4904 , piece_id=QCMN4OSBWIWS4JRLIGQIGDSHZFUF2JWEPVXQ4FQ4G3LG5FBCBAWQ, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.376764359 (557h3m4.247847404s) [noise=true , address=145.79.227.19:4105 , piece_id=RY35FSM6NTWPNF3JE43FZRSNXXYZPUN2B6LRNNUOG6QQR6XSU36Q, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GhDrKMcDEg2RxamqKtiHphpgNdvtZYdHXrjzGsNJFjBFYeEen): protocol: write tcp 188.166.225.170:38838->145.79.227.19:4105: use of closed network connection; write tcp 188.166.225.170:38838->145.79.227.19:4105: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38838->145.79.227.19:4105: use of closed network connection Dec 28 20:32:04.376772865 (557h3m4.2478559s) [noise=true , address=145.79.227.19:4105 , piece_id=RY35FSM6NTWPNF3JE43FZRSNXXYZPUN2B6LRNNUOG6QQR6XSU36Q, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.376926658 (557h3m4.248009703s) [noise=true , address=145.79.227.35:4496 , piece_id=JH4XQBT7XYEJXWBGPPF6SJWCPLV77GBOLGOA53C3WUQMWHZXVZQA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V8XhSWkyGUVYFR8MykifFAP1YAfGCwzqucDcMMfzw3xsVW77Z): protocol: expected piece hash; context canceled; EOF Dec 28 20:32:04.377165332 (557h3m4.248248377s) [noise=true , address=145.79.227.35:4496 , piece_id=JH4XQBT7XYEJXWBGPPF6SJWCPLV77GBOLGOA53C3WUQMWHZXVZQA, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.377212682 (557h3m4.248295848s) [noise=true , address=45.91.116.5:4330 , piece_id=EVHLSVVR5JGY43TLSKVEQLQ6KDBEZENVAFL6YT7ZQ77Y7X2TXIKA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12MK9aVDRwfWPfckFR6RfPUL1Nrm9bwfsgV7TXCKGbtadHebWvW): protocol: read tcp 188.166.225.170:55416->45.91.116.5:4330: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 20:32:04.377289859 (557h3m4.248372924s) [noise=true , address=45.91.116.5:4330 , piece_id=EVHLSVVR5JGY43TLSKVEQLQ6KDBEZENVAFL6YT7ZQ77Y7X2TXIKA, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.377387736 (557h3m4.248470781s) [seg_pos={0 0}, upload=488362]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 20:32:04.377599729 (557h3m4.248682774s) [upload=488362]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 20:32:04.377613535 (557h3m4.24869662s) [upload=488362]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 20:32:04.675090855 (557h3m4.5461739s) [upload=488362]: Done uploading segments... Dec 28 20:32:04.675121233 (557h3m4.546204268s) [upload=488362]: Upload finished. err: <nil> Duration: 10.647287614s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 15.474982747s trace: https://jaeger.dev.storj.io/trace/499b96822b8dda80 FAIL (2025-12-28 20:52:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12S7WBBQ1Nq8jbeGrZiRKTk23CdJC31hPW2vU1aaM5SsHeVyeRH, Piece ID: V4P3KCUDLI543TSROBYUMVTQKKSQDDVSKSJSXKEZTQAHTXUJ2G4Q) context deadline exceeded; piecestore: (Node ID: 1kjCEPtjvcJ3D7BBKRSRirPybBwkRZ1dJwyAQCadzCvaLYvWhz, Piece ID: 4OULHJ6Z3QJ3BOPDY74TJSKVJG23TOEUCJCUEGY24I5C3MUTAVLQ) context deadline exceeded; piecestore: (Node ID: 1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c, Piece ID: LIYXCPAZOUPKXLKRH3EOWREXUY2LAJXSRZFQILM2QIU7HTW7SY2Q) context deadline exceeded ======LOGS====== ue , address=145.79.227.23:4580 , piece_id=OHYI2SFM6TKJE4LHVNGO2PVGZDNFIQDAIQGKG4FOEHTTASBWA5ZQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.496428825 (557h3m1.36751187s) [noise=true , address=77.90.31.19:4867 , piece_id=LDAWD2ZFG3BZATDRY7MMNWYIX6TOHQ2OIUPCAWAYN266GNUUKZ5A, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.496861509 (557h3m1.367944544s) [noise=true , address=145.79.227.19:4105 , piece_id=RY35FSM6NTWPNF3JE43FZRSNXXYZPUN2B6LRNNUOG6QQR6XSU36Q, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.496899933 (557h3m1.367982978s) [noise=true , address=77.90.31.19:4858 , piece_id=LIYXCPAZOUPKXLKRH3EOWREXUY2LAJXSRZFQILM2QIU7HTW7SY2Q, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497229089 (557h3m1.368312134s) [noise=true , address=77.90.31.17:4784 , piece_id=JV2ROD6MN65B6LZKOL3S7ATJWXAY6FBQSZWITA6IHAXELY7R42KA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497326194 (557h3m1.368409239s) [noise=true , address=145.79.227.33:4904 , piece_id=QCMN4OSBWIWS4JRLIGQIGDSHZFUF2JWEPVXQ4FQ4G3LG5FBCBAWQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497664128 (557h3m1.368747173s) [noise=true , address=145.79.227.34:4918 , piece_id=SPDT2KI2774UJMQIUSVTHXCDF7PD2FBG64BXUYW67UMLPE6H7SBQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497731405 (557h3m1.36881445s) [noise=true , address=77.90.31.12:4602 , piece_id=2YPYKKTQ6ID6VTI4AUZQEG2SY7JYPRSAPU4JLF7DWSOSBBO74LWQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.498205939 (557h3m1.369288984s) [noise=true , address=77.90.31.16:4745 , piece_id=CZBFU33BHMP2BEZIDJKW3J526537HR6HP7OCT2KMAQVY7LGZLPZQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.498424295 (557h3m1.36950734s) [noise=true , address=77.90.31.21:4895 , piece_id=V4P3KCUDLI543TSROBYUMVTQKKSQDDVSKSJSXKEZTQAHTXUJ2G4Q, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.498685422 (557h3m1.369768457s) [noise=true , address=145.79.227.35:4496 , piece_id=JH4XQBT7XYEJXWBGPPF6SJWCPLV77GBOLGOA53C3WUQMWHZXVZQA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499066348 (557h3m1.370149393s) [noise=true , address=77.90.31.11:4563 , piece_id=DIFJPHSKHROTLKCHP6HQBWT7XGE77HVRZTH2354MV2UEWZLHVGCQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499185725 (557h3m1.37026877s) [noise=true , address=145.79.227.30:4768 , piece_id=KK4LEV6RG3IGLYXID5MJF3KMNSJ7WS5JS45PNOOJ26LXLYAUCWOA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499468122 (557h3m1.370551157s) [noise=true , address=77.90.31.14:4688 , piece_id=4OULHJ6Z3QJ3BOPDY74TJSKVJG23TOEUCJCUEGY24I5C3MUTAVLQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499901047 (557h3m1.370984102s) [noise=true , address=45.91.116.5:4330 , piece_id=EVHLSVVR5JGY43TLSKVEQLQ6KDBEZENVAFL6YT7ZQ77Y7X2TXIKA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.500546175 (557h3m1.37162921s) [noise=true , address=145.79.227.43:4171 , piece_id=YF5DRDPGRRVYFL5SWMH2PKTSV3C72BIWKUDSAIZTXOIVOTYKRMNA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:02.979572381 (557h3m2.850655436s) [upload=488362]: Next returned nil segment Dec 28 20:32:02.979609411 (557h3m2.850692456s) [upload=488362]: Waiting for error group managing segments... Dec 28 20:32:03.416896733 (557h3m3.287979778s) [noise=true , address=145.79.227.23:4580 , piece_id=OHYI2SFM6TKJE4LHVNGO2PVGZDNFIQDAIQGKG4FOEHTTASBWA5ZQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.552263655 (557h3m3.4233467s) [noise=true , address=77.90.31.17:4784 , piece_id=JV2ROD6MN65B6LZKOL3S7ATJWXAY6FBQSZWITA6IHAXELY7R42KA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.573564856 (557h3m3.444647902s) [noise=true , address=77.90.31.12:4602 , piece_id=2YPYKKTQ6ID6VTI4AUZQEG2SY7JYPRSAPU4JLF7DWSOSBBO74LWQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.598560923 (557h3m3.469643968s) [noise=true , address=77.90.31.19:4867 , piece_id=LDAWD2ZFG3BZATDRY7MMNWYIX6TOHQ2OIUPCAWAYN266GNUUKZ5A, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.626925925 (557h3m3.49800897s) [noise=true , address=77.90.31.16:4745 , piece_id=CZBFU33BHMP2BEZIDJKW3J526537HR6HP7OCT2KMAQVY7LGZLPZQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.709061947 (557h3m3.580144992s) [noise=true , address=77.90.31.14:4688 , piece_id=4OULHJ6Z3QJ3BOPDY74TJSKVJG23TOEUCJCUEGY24I5C3MUTAVLQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.788879373 (557h3m3.659962418s) [noise=true , address=77.90.31.21:4895 , piece_id=V4P3KCUDLI543TSROBYUMVTQKKSQDDVSKSJSXKEZTQAHTXUJ2G4Q, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.217616523 (557h3m4.088699569s) [noise=true , address=77.90.31.19:4858 , piece_id=LIYXCPAZOUPKXLKRH3EOWREXUY2LAJXSRZFQILM2QIU7HTW7SY2Q, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.220337625 (557h3m4.09142067s) [noise=true , address=77.90.31.11:4563 , piece_id=DIFJPHSKHROTLKCHP6HQBWT7XGE77HVRZTH2354MV2UEWZLHVGCQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.303958744 (557h3m4.175041789s) [noise=true , address=145.79.227.44:4213 , piece_id=UEAQDBBCP7DIG6ZD3EEM2CKAL4W5QKW23LWFUAMSVUW5ZS2PJDRQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.312794398 (557h3m4.183877444s) [noise=true , address=145.79.227.34:4918 , piece_id=SPDT2KI2774UJMQIUSVTHXCDF7PD2FBG64BXUYW67UMLPE6H7SBQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.349308798 (557h3m4.220391833s) [noise=true , address=145.79.227.40:4058 , piece_id=EQ4LA7T2BCGWY76K3NJI4X77OQJFJED6H4VNGXDQMILU6IVQWOMQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.371518379 (557h3m4.242601424s) [noise=true , address=145.79.227.43:4171 , piece_id=YF5DRDPGRRVYFL5SWMH2PKTSV3C72BIWKUDSAIZTXOIVOTYKRMNA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.376232316 (557h3m4.247315361s) [noise=true , address=145.79.227.30:4768 , piece_id=KK4LEV6RG3IGLYXID5MJF3KMNSJ7WS5JS45PNOOJ26LXLYAUCWOA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.376257844 (557h3m4.247340879s) [seg_pos={0 0}, upload=488362]: Segment reached optimal threshold of 14 pieces. Dec 28 20:32:04.376575799 (557h3m4.247658834s) [noise=true , address=145.79.227.33:4904 , piece_id=QCMN4OSBWIWS4JRLIGQIGDSHZFUF2JWEPVXQ4FQ4G3LG5FBCBAWQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1dC7mC2Cbxmys7ukEkELyF9Us7F1qtptSP6yDJh1EfjVJ6uqsC): protocol: expected piece hash; context canceled; EOF Dec 28 20:32:04.376585127 (557h3m4.247668163s) [noise=true , address=145.79.227.33:4904 , piece_id=QCMN4OSBWIWS4JRLIGQIGDSHZFUF2JWEPVXQ4FQ4G3LG5FBCBAWQ, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.376764359 (557h3m4.247847404s) [noise=true , address=145.79.227.19:4105 , piece_id=RY35FSM6NTWPNF3JE43FZRSNXXYZPUN2B6LRNNUOG6QQR6XSU36Q, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GhDrKMcDEg2RxamqKtiHphpgNdvtZYdHXrjzGsNJFjBFYeEen): protocol: write tcp 188.166.225.170:38838->145.79.227.19:4105: use of closed network connection; write tcp 188.166.225.170:38838->145.79.227.19:4105: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38838->145.79.227.19:4105: use of closed network connection Dec 28 20:32:04.376772865 (557h3m4.2478559s) [noise=true , address=145.79.227.19:4105 , piece_id=RY35FSM6NTWPNF3JE43FZRSNXXYZPUN2B6LRNNUOG6QQR6XSU36Q, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.376926658 (557h3m4.248009703s) [noise=true , address=145.79.227.35:4496 , piece_id=JH4XQBT7XYEJXWBGPPF6SJWCPLV77GBOLGOA53C3WUQMWHZXVZQA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V8XhSWkyGUVYFR8MykifFAP1YAfGCwzqucDcMMfzw3xsVW77Z): protocol: expected piece hash; context canceled; EOF Dec 28 20:32:04.377165332 (557h3m4.248248377s) [noise=true , address=145.79.227.35:4496 , piece_id=JH4XQBT7XYEJXWBGPPF6SJWCPLV77GBOLGOA53C3WUQMWHZXVZQA, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.377212682 (557h3m4.248295848s) [noise=true , address=45.91.116.5:4330 , piece_id=EVHLSVVR5JGY43TLSKVEQLQ6KDBEZENVAFL6YT7ZQ77Y7X2TXIKA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12MK9aVDRwfWPfckFR6RfPUL1Nrm9bwfsgV7TXCKGbtadHebWvW): protocol: read tcp 188.166.225.170:55416->45.91.116.5:4330: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 20:32:04.377289859 (557h3m4.248372924s) [noise=true , address=45.91.116.5:4330 , piece_id=EVHLSVVR5JGY43TLSKVEQLQ6KDBEZENVAFL6YT7ZQ77Y7X2TXIKA, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.377387736 (557h3m4.248470781s) [seg_pos={0 0}, upload=488362]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 20:32:04.377599729 (557h3m4.248682774s) [upload=488362]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 20:32:04.377613535 (557h3m4.24869662s) [upload=488362]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 20:32:04.675090855 (557h3m4.5461739s) [upload=488362]: Done uploading segments... Dec 28 20:32:04.675121233 (557h3m4.546204268s) [upload=488362]: Upload finished. err: <nil> Duration: 15.475006573s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.......X...........X..................X.......X......XX.....] duration: 5.974371791s trace: https://jaeger.dev.storj.io/trace/3ce821e76930523a FAIL (2025-12-28 20:52:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 20:49:03.805105648 (557h20m3.676188693s) [upload=488615]: Starting upload Dec 28 20:49:03.805166324 (557h20m3.676249369s) [upload=488615]: Done starting upload Dec 28 20:49:03.805293475 (557h20m3.67637651s) [upload=488615]: Uploading segments... Dec 28 20:49:03.805403545 (557h20m3.67648658s) [upload=488615]: Got next segment. Inline: false Dec 28 20:49:03.805460854 (557h20m3.676543889s) [upload=488615]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 20:49:03.805467066 (557h20m3.676550111s) [upload=488615]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 20:49:04.052770455 (557h20m3.9238535s) [seg_pos={0 0}, upload=488615]: Begin upload segment... Dec 28 20:49:04.052835940 (557h20m3.923918985s) [seg_pos={0 0}, upload=488615]: Done begin upload segment. Dec 28 20:49:04.052874203 (557h20m3.923957248s) [noise=true , address=145.79.227.44:4194 , piece_id=WYODZVE7TGDTRNVGZ5HAD2JJT3TSOB7KGNGWH5VCQMLCCICMORRA, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.052970917 (557h20m3.924053962s) [noise=true , address=145.79.227.39:4015 , piece_id=UNPAANTMSZ3O356MXFAX357E3L4X5OLMJBXTDQEVGXUH5B5KDYIQ, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.053474816 (557h20m3.924557861s) [noise=true , address=45.91.116.8:4046 , piece_id=2DXIM4FJEZZQ3LRR4GFUEJ6XD6IGDET3JVKWBZJQGKMICWUITTTA, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.053938549 (557h20m3.925021584s) [noise=true , address=145.79.227.37:4565 , piece_id=6EAD5WOUCVJXUOKHGTSMMHDYPKHHJFZZFIXH2FRUNNZTZNQNK5IA, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.054376714 (557h20m3.925459759s) [noise=true , address=145.79.227.19:4078 , piece_id=DCMFZ42KOAHHMA3YOVW7655PLKCMJDHTLETUU4NK2D6EE27TEKWQ, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.055140448 (557h20m3.926223493s) [noise=true , address=145.79.227.36:4519 , piece_id=GQENJD2GLGJSL7H7VWUN6JXBZTW6ZHFLMMYTKYF3TRBGNNBQC3YQ, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.055719992 (557h20m3.926803027s) [noise=true , address=145.79.227.51:4411 , piece_id=IMS72BZZCJ3WMXU3FTBKM3F4MSXTL4VCVCPOCNLQI6WHKJGIOXQQ, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.056924334 (557h20m3.92800737s) [noise=true , address=45.91.116.6:4973 , piece_id=HMTEGMPYPTWCLRSAH45LLVZTTG2TRVUS44MTKBPZT6SIMRFYYNGQ, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.058635102 (557h20m3.929718147s) [noise=true , address=145.79.227.13:4804 , piece_id=UYO4W6FOSDQSVVEO6XWVTGXMFLXYWNYA3DMICFUABVSFFDKPFRXA, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.058693383 (557h20m3.929776418s) [noise=true , address=45.91.116.5:4299 , piece_id=HI6B35X52HVYVDIFVSJQT6TXYBBDA5VB5COKH53CPKDTO3UHKWJA, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.061111298 (557h20m3.932194343s) [noise=true , address=77.90.31.14:4661 , piece_id=OP5ES2G4MLZBJWJDYYPO7KLHUC6NDX5FUF6ZO6PCYMCFSFTKPVDA, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.061856947 (557h20m3.932939992s) [noise=true , address=145.79.227.42:4123 , piece_id=6YEKCEU5A5MPTRVGRFH2GWQJVSWMBXHCIZSAJKPUY2M2I3P63SFQ, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.064452048 (557h20m3.935535093s) [noise=true , address=45.91.116.10:4082 , piece_id=WGN35FKEXP3UEANOQA4NSJUQLDKIZ36RYD6OFP2FHEBNJKLI2TTA, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.066246636 (557h20m3.937329681s) [noise=true , address=145.79.227.5:4221 , piece_id=GVQO2XLQ2D6XVAHLGZVB6SHA476DL3TUHAXLFKZU5H2BN3LEB5XQ, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.069098717 (557h20m3.940181762s) [noise=true , address=145.79.227.38:4977 , piece_id=23SZDYDT2OWMLO7EGT6OT5L4S4WYR5LY2SWCCZGSCZMKQ5RK6XPQ, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.070882644 (557h20m3.941965689s) [noise=true , address=145.79.227.32:4834 , piece_id=52YM22AS3BG3OGILTWR36HSHD5SKNRKKUR5TIE7HLWOSK3L6KQOQ, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:05.778149774 (557h20m5.649232819s) [upload=488615]: Next returned nil segment Dec 28 20:49:05.778171825 (557h20m5.64925487s) [upload=488615]: Waiting for error group managing segments... Dec 28 20:49:06.067849953 (557h20m5.938932998s) [noise=true , address=145.79.227.37:4565 , piece_id=6EAD5WOUCVJXUOKHGTSMMHDYPKHHJFZZFIXH2FRUNNZTZNQNK5IA, seg_pos={0 0}, upload=488615]: Done uploading piece. err: <nil> Dec 28 20:49:06.073409289 (557h20m5.944492334s) [noise=true , address=145.79.227.39:4015 , piece_id=UNPAANTMSZ3O356MXFAX357E3L4X5OLMJBXTDQEVGXUH5B5KDYIQ, seg_pos={0 0}, upload=488615]: Done uploading piece. err: <nil> Dec 28 20:49:06.078797668 (557h20m5.949880714s) [noise=true , address=145.79.227.13:4804 , piece_id=UYO4W6FOSDQSVVEO6XWVTGXMFLXYWNYA3DMICFUABVSFFDKPFRXA, seg_pos={0 0}, upload=488615]: Done uploading piece. err: <nil> Dec 28 20:49:06.084142657 (557h20m5.955225692s) [noise=true , address=145.79.227.32:4834 , piece_id=52YM22AS3BG3OGILTWR36HSHD5SKNRKKUR5TIE7HLWOSK3L6KQOQ, seg_pos={0 0}, upload=488615]: Done uploading piece. err: <nil> Dec 28 20:49:06.084205807 (557h20m5.955288842s) [noise=true , address=145.79.227.19:4078 , piece_id=DCMFZ42KOAHHMA3YOVW7655PLKCMJDHTLETUU4NK2D6EE27TEKWQ, seg_pos={0 0}, upload=488615]: Done uploading piece. err: <nil> Dec 28 20:49:06.085908499 (557h20m5.956991534s) [noise=true , address=145.79.227.51:4411 , piece_id=IMS72BZZCJ3WMXU3FTBKM3F4MSXTL4VCVCPOCNLQI6WHKJGIOXQQ, seg_pos={0 0}, upload=488615]: Done uploading piece. err: <nil> Dec 28 20:49:06.089752589 (557h20m5.960835624s) [noise=true , address=145.79.227.5:4221 , piece_id=GVQO2XLQ2D6XVAHLGZVB6SHA476DL3TUHAXLFKZU5H2BN3LEB5XQ, seg_pos={0 0}, upload=488615]: Done uploading piece. err: <nil> Dec 28 20:49:06.096096639 (557h20m5.967179674s) [noise=true , address=145.79.227.36:4519 , piece_id=GQENJD2GLGJSL7H7VWUN6JXBZTW6ZHFLMMYTKYF3TRBGNNBQC3YQ, seg_pos={0 0}, upload=488615]: Done uploading piece. err: <nil> Dec 28 20:49:06.097668182 (557h20m5.968751227s) [noise=true , address=145.79.227.42:4123 , piece_id=6YEKCEU5A5MPTRVGRFH2GWQJVSWMBXHCIZSAJKPUY2M2I3P63SFQ, seg_pos={0 0}, upload=488615]: Done uploading piece. err: <nil> Dec 28 20:49:06.107376347 (557h20m5.978459392s) [noise=true , address=145.79.227.38:4977 , piece_id=23SZDYDT2OWMLO7EGT6OT5L4S4WYR5LY2SWCCZGSCZMKQ5RK6XPQ, seg_pos={0 0}, upload=488615]: Done uploading piece. err: <nil> Dec 28 20:49:06.255907707 (557h20m6.126990752s) [noise=true , address=77.90.31.14:4661 , piece_id=OP5ES2G4MLZBJWJDYYPO7KLHUC6NDX5FUF6ZO6PCYMCFSFTKPVDA, seg_pos={0 0}, upload=488615]: Done uploading piece. err: <nil> Dec 28 20:49:06.306350829 (557h20m6.177433874s) [noise=true , address=45.91.116.5:4299 , piece_id=HI6B35X52HVYVDIFVSJQT6TXYBBDA5VB5COKH53CPKDTO3UHKWJA, seg_pos={0 0}, upload=488615]: Done uploading piece. err: <nil> Dec 28 20:49:06.312533491 (557h20m6.183616536s) [noise=true , address=45.91.116.8:4046 , piece_id=2DXIM4FJEZZQ3LRR4GFUEJ6XD6IGDET3JVKWBZJQGKMICWUITTTA, seg_pos={0 0}, upload=488615]: Done uploading piece. err: <nil> Dec 28 20:49:06.312558800 (557h20m6.183641845s) [seg_pos={0 0}, upload=488615]: Segment reached optimal threshold of 13 pieces. Dec 28 20:49:06.313074843 (557h20m6.184157888s) [noise=true , address=145.79.227.44:4194 , piece_id=WYODZVE7TGDTRNVGZ5HAD2JJT3TSOB7KGNGWH5VCQMLCCICMORRA, seg_pos={0 0}, upload=488615]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 28 20:49:06.313086825 (557h20m6.18416987s) [noise=true , address=145.79.227.44:4194 , piece_id=WYODZVE7TGDTRNVGZ5HAD2JJT3TSOB7KGNGWH5VCQMLCCICMORRA, seg_pos={0 0}, upload=488615]: Long tail cancellation due to optimal threshold Dec 28 20:49:06.313142812 (557h20m6.184225847s) [noise=true , address=45.91.116.6:4973 , piece_id=HMTEGMPYPTWCLRSAH45LLVZTTG2TRVUS44MTKBPZT6SIMRFYYNGQ, seg_pos={0 0}, upload=488615]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF Dec 28 20:49:06.313177759 (557h20m6.184260804s) [noise=true , address=45.91.116.6:4973 , piece_id=HMTEGMPYPTWCLRSAH45LLVZTTG2TRVUS44MTKBPZT6SIMRFYYNGQ, seg_pos={0 0}, upload=488615]: Long tail cancellation due to optimal threshold Dec 28 20:49:06.313256448 (557h20m6.184339503s) [noise=true , address=45.91.116.10:4082 , piece_id=WGN35FKEXP3UEANOQA4NSJUQLDKIZ36RYD6OFP2FHEBNJKLI2TTA, seg_pos={0 0}, upload=488615]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 28 20:49:06.313271217 (557h20m6.184354262s) [noise=true , address=45.91.116.10:4082 , piece_id=WGN35FKEXP3UEANOQA4NSJUQLDKIZ36RYD6OFP2FHEBNJKLI2TTA, seg_pos={0 0}, upload=488615]: Long tail cancellation due to optimal threshold Dec 28 20:49:06.313296314 (557h20m6.184379359s) [seg_pos={0 0}, upload=488615]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 20:49:06.313381627 (557h20m6.184464672s) [upload=488615]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 20:49:06.313389141 (557h20m6.184472186s) [upload=488615]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 20:49:06.604901881 (557h20m6.475984916s) [upload=488615]: Done uploading segments... Dec 28 20:49:06.604926539 (557h20m6.476009584s) [upload=488615]: Upload finished. err: <nil> Duration: 5.974400434s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [........X...........X..................X.......X......XX....] duration: 5.96611113s trace: https://jaeger.dev.storj.io/trace/3ce821e76930523a FAIL (2025-12-28 20:51:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 20:49:03.805105648 (557h20m3.676188693s) [upload=488615]: Starting upload Dec 28 20:49:03.805166324 (557h20m3.676249369s) [upload=488615]: Done starting upload Dec 28 20:49:03.805293475 (557h20m3.67637651s) [upload=488615]: Uploading segments... Dec 28 20:49:03.805403545 (557h20m3.67648658s) [upload=488615]: Got next segment. Inline: false Dec 28 20:49:03.805460854 (557h20m3.676543889s) [upload=488615]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 20:49:03.805467066 (557h20m3.676550111s) [upload=488615]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 20:49:04.052770455 (557h20m3.9238535s) [seg_pos={0 0}, upload=488615]: Begin upload segment... Dec 28 20:49:04.052835940 (557h20m3.923918985s) [seg_pos={0 0}, upload=488615]: Done begin upload segment. Dec 28 20:49:04.052874203 (557h20m3.923957248s) [noise=true , address=145.79.227.44:4194 , piece_id=WYODZVE7TGDTRNVGZ5HAD2JJT3TSOB7KGNGWH5VCQMLCCICMORRA, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.052970917 (557h20m3.924053962s) [noise=true , address=145.79.227.39:4015 , piece_id=UNPAANTMSZ3O356MXFAX357E3L4X5OLMJBXTDQEVGXUH5B5KDYIQ, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.053474816 (557h20m3.924557861s) [noise=true , address=45.91.116.8:4046 , piece_id=2DXIM4FJEZZQ3LRR4GFUEJ6XD6IGDET3JVKWBZJQGKMICWUITTTA, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.053938549 (557h20m3.925021584s) [noise=true , address=145.79.227.37:4565 , piece_id=6EAD5WOUCVJXUOKHGTSMMHDYPKHHJFZZFIXH2FRUNNZTZNQNK5IA, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.054376714 (557h20m3.925459759s) [noise=true , address=145.79.227.19:4078 , piece_id=DCMFZ42KOAHHMA3YOVW7655PLKCMJDHTLETUU4NK2D6EE27TEKWQ, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.055140448 (557h20m3.926223493s) [noise=true , address=145.79.227.36:4519 , piece_id=GQENJD2GLGJSL7H7VWUN6JXBZTW6ZHFLMMYTKYF3TRBGNNBQC3YQ, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.055719992 (557h20m3.926803027s) [noise=true , address=145.79.227.51:4411 , piece_id=IMS72BZZCJ3WMXU3FTBKM3F4MSXTL4VCVCPOCNLQI6WHKJGIOXQQ, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.056924334 (557h20m3.92800737s) [noise=true , address=45.91.116.6:4973 , piece_id=HMTEGMPYPTWCLRSAH45LLVZTTG2TRVUS44MTKBPZT6SIMRFYYNGQ, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.058635102 (557h20m3.929718147s) [noise=true , address=145.79.227.13:4804 , piece_id=UYO4W6FOSDQSVVEO6XWVTGXMFLXYWNYA3DMICFUABVSFFDKPFRXA, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.058693383 (557h20m3.929776418s) [noise=true , address=45.91.116.5:4299 , piece_id=HI6B35X52HVYVDIFVSJQT6TXYBBDA5VB5COKH53CPKDTO3UHKWJA, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.061111298 (557h20m3.932194343s) [noise=true , address=77.90.31.14:4661 , piece_id=OP5ES2G4MLZBJWJDYYPO7KLHUC6NDX5FUF6ZO6PCYMCFSFTKPVDA, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.061856947 (557h20m3.932939992s) [noise=true , address=145.79.227.42:4123 , piece_id=6YEKCEU5A5MPTRVGRFH2GWQJVSWMBXHCIZSAJKPUY2M2I3P63SFQ, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.064452048 (557h20m3.935535093s) [noise=true , address=45.91.116.10:4082 , piece_id=WGN35FKEXP3UEANOQA4NSJUQLDKIZ36RYD6OFP2FHEBNJKLI2TTA, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.066246636 (557h20m3.937329681s) [noise=true , address=145.79.227.5:4221 , piece_id=GVQO2XLQ2D6XVAHLGZVB6SHA476DL3TUHAXLFKZU5H2BN3LEB5XQ, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.069098717 (557h20m3.940181762s) [noise=true , address=145.79.227.38:4977 , piece_id=23SZDYDT2OWMLO7EGT6OT5L4S4WYR5LY2SWCCZGSCZMKQ5RK6XPQ, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.070882644 (557h20m3.941965689s) [noise=true , address=145.79.227.32:4834 , piece_id=52YM22AS3BG3OGILTWR36HSHD5SKNRKKUR5TIE7HLWOSK3L6KQOQ, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:05.778149774 (557h20m5.649232819s) [upload=488615]: Next returned nil segment Dec 28 20:49:05.778171825 (557h20m5.64925487s) [upload=488615]: Waiting for error group managing segments... Dec 28 20:49:06.067849953 (557h20m5.938932998s) [noise=true , address=145.79.227.37:4565 , piece_id=6EAD5WOUCVJXUOKHGTSMMHDYPKHHJFZZFIXH2FRUNNZTZNQNK5IA, seg_pos={0 0}, upload=488615]: Done uploading piece. err: <nil> Dec 28 20:49:06.073409289 (557h20m5.944492334s) [noise=true , address=145.79.227.39:4015 , piece_id=UNPAANTMSZ3O356MXFAX357E3L4X5OLMJBXTDQEVGXUH5B5KDYIQ, seg_pos={0 0}, upload=488615]: Done uploading piece. err: <nil> Dec 28 20:49:06.078797668 (557h20m5.949880714s) [noise=true , address=145.79.227.13:4804 , piece_id=UYO4W6FOSDQSVVEO6XWVTGXMFLXYWNYA3DMICFUABVSFFDKPFRXA, seg_pos={0 0}, upload=488615]: Done uploading piece. err: <nil> Dec 28 20:49:06.084142657 (557h20m5.955225692s) [noise=true , address=145.79.227.32:4834 , piece_id=52YM22AS3BG3OGILTWR36HSHD5SKNRKKUR5TIE7HLWOSK3L6KQOQ, seg_pos={0 0}, upload=488615]: Done uploading piece. err: <nil> Dec 28 20:49:06.084205807 (557h20m5.955288842s) [noise=true , address=145.79.227.19:4078 , piece_id=DCMFZ42KOAHHMA3YOVW7655PLKCMJDHTLETUU4NK2D6EE27TEKWQ, seg_pos={0 0}, upload=488615]: Done uploading piece. err: <nil> Dec 28 20:49:06.085908499 (557h20m5.956991534s) [noise=true , address=145.79.227.51:4411 , piece_id=IMS72BZZCJ3WMXU3FTBKM3F4MSXTL4VCVCPOCNLQI6WHKJGIOXQQ, seg_pos={0 0}, upload=488615]: Done uploading piece. err: <nil> Dec 28 20:49:06.089752589 (557h20m5.960835624s) [noise=true , address=145.79.227.5:4221 , piece_id=GVQO2XLQ2D6XVAHLGZVB6SHA476DL3TUHAXLFKZU5H2BN3LEB5XQ, seg_pos={0 0}, upload=488615]: Done uploading piece. err: <nil> Dec 28 20:49:06.096096639 (557h20m5.967179674s) [noise=true , address=145.79.227.36:4519 , piece_id=GQENJD2GLGJSL7H7VWUN6JXBZTW6ZHFLMMYTKYF3TRBGNNBQC3YQ, seg_pos={0 0}, upload=488615]: Done uploading piece. err: <nil> Dec 28 20:49:06.097668182 (557h20m5.968751227s) [noise=true , address=145.79.227.42:4123 , piece_id=6YEKCEU5A5MPTRVGRFH2GWQJVSWMBXHCIZSAJKPUY2M2I3P63SFQ, seg_pos={0 0}, upload=488615]: Done uploading piece. err: <nil> Dec 28 20:49:06.107376347 (557h20m5.978459392s) [noise=true , address=145.79.227.38:4977 , piece_id=23SZDYDT2OWMLO7EGT6OT5L4S4WYR5LY2SWCCZGSCZMKQ5RK6XPQ, seg_pos={0 0}, upload=488615]: Done uploading piece. err: <nil> Dec 28 20:49:06.255907707 (557h20m6.126990752s) [noise=true , address=77.90.31.14:4661 , piece_id=OP5ES2G4MLZBJWJDYYPO7KLHUC6NDX5FUF6ZO6PCYMCFSFTKPVDA, seg_pos={0 0}, upload=488615]: Done uploading piece. err: <nil> Dec 28 20:49:06.306350829 (557h20m6.177433874s) [noise=true , address=45.91.116.5:4299 , piece_id=HI6B35X52HVYVDIFVSJQT6TXYBBDA5VB5COKH53CPKDTO3UHKWJA, seg_pos={0 0}, upload=488615]: Done uploading piece. err: <nil> Dec 28 20:49:06.312533491 (557h20m6.183616536s) [noise=true , address=45.91.116.8:4046 , piece_id=2DXIM4FJEZZQ3LRR4GFUEJ6XD6IGDET3JVKWBZJQGKMICWUITTTA, seg_pos={0 0}, upload=488615]: Done uploading piece. err: <nil> Dec 28 20:49:06.312558800 (557h20m6.183641845s) [seg_pos={0 0}, upload=488615]: Segment reached optimal threshold of 13 pieces. Dec 28 20:49:06.313074843 (557h20m6.184157888s) [noise=true , address=145.79.227.44:4194 , piece_id=WYODZVE7TGDTRNVGZ5HAD2JJT3TSOB7KGNGWH5VCQMLCCICMORRA, seg_pos={0 0}, upload=488615]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 28 20:49:06.313086825 (557h20m6.18416987s) [noise=true , address=145.79.227.44:4194 , piece_id=WYODZVE7TGDTRNVGZ5HAD2JJT3TSOB7KGNGWH5VCQMLCCICMORRA, seg_pos={0 0}, upload=488615]: Long tail cancellation due to optimal threshold Dec 28 20:49:06.313142812 (557h20m6.184225847s) [noise=true , address=45.91.116.6:4973 , piece_id=HMTEGMPYPTWCLRSAH45LLVZTTG2TRVUS44MTKBPZT6SIMRFYYNGQ, seg_pos={0 0}, upload=488615]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF Dec 28 20:49:06.313177759 (557h20m6.184260804s) [noise=true , address=45.91.116.6:4973 , piece_id=HMTEGMPYPTWCLRSAH45LLVZTTG2TRVUS44MTKBPZT6SIMRFYYNGQ, seg_pos={0 0}, upload=488615]: Long tail cancellation due to optimal threshold Dec 28 20:49:06.313256448 (557h20m6.184339503s) [noise=true , address=45.91.116.10:4082 , piece_id=WGN35FKEXP3UEANOQA4NSJUQLDKIZ36RYD6OFP2FHEBNJKLI2TTA, seg_pos={0 0}, upload=488615]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 28 20:49:06.313271217 (557h20m6.184354262s) [noise=true , address=45.91.116.10:4082 , piece_id=WGN35FKEXP3UEANOQA4NSJUQLDKIZ36RYD6OFP2FHEBNJKLI2TTA, seg_pos={0 0}, upload=488615]: Long tail cancellation due to optimal threshold Dec 28 20:49:06.313296314 (557h20m6.184379359s) [seg_pos={0 0}, upload=488615]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 20:49:06.313381627 (557h20m6.184464672s) [upload=488615]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 20:49:06.313389141 (557h20m6.184472186s) [upload=488615]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 20:49:06.604901881 (557h20m6.475984916s) [upload=488615]: Done uploading segments... Dec 28 20:49:06.604926539 (557h20m6.476009584s) [upload=488615]: Upload finished. err: <nil> Duration: 5.966139385s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 6.997744327s trace: https://jaeger.dev.storj.io/trace/499b96822b8dda80 FAIL (2025-12-28 20:51:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12S7WBBQ1Nq8jbeGrZiRKTk23CdJC31hPW2vU1aaM5SsHeVyeRH, Piece ID: V4P3KCUDLI543TSROBYUMVTQKKSQDDVSKSJSXKEZTQAHTXUJ2G4Q) context deadline exceeded; piecestore: (Node ID: 1kjCEPtjvcJ3D7BBKRSRirPybBwkRZ1dJwyAQCadzCvaLYvWhz, Piece ID: 4OULHJ6Z3QJ3BOPDY74TJSKVJG23TOEUCJCUEGY24I5C3MUTAVLQ) context deadline exceeded; piecestore: (Node ID: 1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c, Piece ID: LIYXCPAZOUPKXLKRH3EOWREXUY2LAJXSRZFQILM2QIU7HTW7SY2Q) context deadline exceeded ======LOGS====== ue , address=145.79.227.23:4580 , piece_id=OHYI2SFM6TKJE4LHVNGO2PVGZDNFIQDAIQGKG4FOEHTTASBWA5ZQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.496428825 (557h3m1.36751187s) [noise=true , address=77.90.31.19:4867 , piece_id=LDAWD2ZFG3BZATDRY7MMNWYIX6TOHQ2OIUPCAWAYN266GNUUKZ5A, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.496861509 (557h3m1.367944544s) [noise=true , address=145.79.227.19:4105 , piece_id=RY35FSM6NTWPNF3JE43FZRSNXXYZPUN2B6LRNNUOG6QQR6XSU36Q, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.496899933 (557h3m1.367982978s) [noise=true , address=77.90.31.19:4858 , piece_id=LIYXCPAZOUPKXLKRH3EOWREXUY2LAJXSRZFQILM2QIU7HTW7SY2Q, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497229089 (557h3m1.368312134s) [noise=true , address=77.90.31.17:4784 , piece_id=JV2ROD6MN65B6LZKOL3S7ATJWXAY6FBQSZWITA6IHAXELY7R42KA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497326194 (557h3m1.368409239s) [noise=true , address=145.79.227.33:4904 , piece_id=QCMN4OSBWIWS4JRLIGQIGDSHZFUF2JWEPVXQ4FQ4G3LG5FBCBAWQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497664128 (557h3m1.368747173s) [noise=true , address=145.79.227.34:4918 , piece_id=SPDT2KI2774UJMQIUSVTHXCDF7PD2FBG64BXUYW67UMLPE6H7SBQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497731405 (557h3m1.36881445s) [noise=true , address=77.90.31.12:4602 , piece_id=2YPYKKTQ6ID6VTI4AUZQEG2SY7JYPRSAPU4JLF7DWSOSBBO74LWQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.498205939 (557h3m1.369288984s) [noise=true , address=77.90.31.16:4745 , piece_id=CZBFU33BHMP2BEZIDJKW3J526537HR6HP7OCT2KMAQVY7LGZLPZQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.498424295 (557h3m1.36950734s) [noise=true , address=77.90.31.21:4895 , piece_id=V4P3KCUDLI543TSROBYUMVTQKKSQDDVSKSJSXKEZTQAHTXUJ2G4Q, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.498685422 (557h3m1.369768457s) [noise=true , address=145.79.227.35:4496 , piece_id=JH4XQBT7XYEJXWBGPPF6SJWCPLV77GBOLGOA53C3WUQMWHZXVZQA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499066348 (557h3m1.370149393s) [noise=true , address=77.90.31.11:4563 , piece_id=DIFJPHSKHROTLKCHP6HQBWT7XGE77HVRZTH2354MV2UEWZLHVGCQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499185725 (557h3m1.37026877s) [noise=true , address=145.79.227.30:4768 , piece_id=KK4LEV6RG3IGLYXID5MJF3KMNSJ7WS5JS45PNOOJ26LXLYAUCWOA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499468122 (557h3m1.370551157s) [noise=true , address=77.90.31.14:4688 , piece_id=4OULHJ6Z3QJ3BOPDY74TJSKVJG23TOEUCJCUEGY24I5C3MUTAVLQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499901047 (557h3m1.370984102s) [noise=true , address=45.91.116.5:4330 , piece_id=EVHLSVVR5JGY43TLSKVEQLQ6KDBEZENVAFL6YT7ZQ77Y7X2TXIKA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.500546175 (557h3m1.37162921s) [noise=true , address=145.79.227.43:4171 , piece_id=YF5DRDPGRRVYFL5SWMH2PKTSV3C72BIWKUDSAIZTXOIVOTYKRMNA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:02.979572381 (557h3m2.850655436s) [upload=488362]: Next returned nil segment Dec 28 20:32:02.979609411 (557h3m2.850692456s) [upload=488362]: Waiting for error group managing segments... Dec 28 20:32:03.416896733 (557h3m3.287979778s) [noise=true , address=145.79.227.23:4580 , piece_id=OHYI2SFM6TKJE4LHVNGO2PVGZDNFIQDAIQGKG4FOEHTTASBWA5ZQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.552263655 (557h3m3.4233467s) [noise=true , address=77.90.31.17:4784 , piece_id=JV2ROD6MN65B6LZKOL3S7ATJWXAY6FBQSZWITA6IHAXELY7R42KA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.573564856 (557h3m3.444647902s) [noise=true , address=77.90.31.12:4602 , piece_id=2YPYKKTQ6ID6VTI4AUZQEG2SY7JYPRSAPU4JLF7DWSOSBBO74LWQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.598560923 (557h3m3.469643968s) [noise=true , address=77.90.31.19:4867 , piece_id=LDAWD2ZFG3BZATDRY7MMNWYIX6TOHQ2OIUPCAWAYN266GNUUKZ5A, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.626925925 (557h3m3.49800897s) [noise=true , address=77.90.31.16:4745 , piece_id=CZBFU33BHMP2BEZIDJKW3J526537HR6HP7OCT2KMAQVY7LGZLPZQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.709061947 (557h3m3.580144992s) [noise=true , address=77.90.31.14:4688 , piece_id=4OULHJ6Z3QJ3BOPDY74TJSKVJG23TOEUCJCUEGY24I5C3MUTAVLQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.788879373 (557h3m3.659962418s) [noise=true , address=77.90.31.21:4895 , piece_id=V4P3KCUDLI543TSROBYUMVTQKKSQDDVSKSJSXKEZTQAHTXUJ2G4Q, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.217616523 (557h3m4.088699569s) [noise=true , address=77.90.31.19:4858 , piece_id=LIYXCPAZOUPKXLKRH3EOWREXUY2LAJXSRZFQILM2QIU7HTW7SY2Q, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.220337625 (557h3m4.09142067s) [noise=true , address=77.90.31.11:4563 , piece_id=DIFJPHSKHROTLKCHP6HQBWT7XGE77HVRZTH2354MV2UEWZLHVGCQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.303958744 (557h3m4.175041789s) [noise=true , address=145.79.227.44:4213 , piece_id=UEAQDBBCP7DIG6ZD3EEM2CKAL4W5QKW23LWFUAMSVUW5ZS2PJDRQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.312794398 (557h3m4.183877444s) [noise=true , address=145.79.227.34:4918 , piece_id=SPDT2KI2774UJMQIUSVTHXCDF7PD2FBG64BXUYW67UMLPE6H7SBQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.349308798 (557h3m4.220391833s) [noise=true , address=145.79.227.40:4058 , piece_id=EQ4LA7T2BCGWY76K3NJI4X77OQJFJED6H4VNGXDQMILU6IVQWOMQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.371518379 (557h3m4.242601424s) [noise=true , address=145.79.227.43:4171 , piece_id=YF5DRDPGRRVYFL5SWMH2PKTSV3C72BIWKUDSAIZTXOIVOTYKRMNA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.376232316 (557h3m4.247315361s) [noise=true , address=145.79.227.30:4768 , piece_id=KK4LEV6RG3IGLYXID5MJF3KMNSJ7WS5JS45PNOOJ26LXLYAUCWOA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.376257844 (557h3m4.247340879s) [seg_pos={0 0}, upload=488362]: Segment reached optimal threshold of 14 pieces. Dec 28 20:32:04.376575799 (557h3m4.247658834s) [noise=true , address=145.79.227.33:4904 , piece_id=QCMN4OSBWIWS4JRLIGQIGDSHZFUF2JWEPVXQ4FQ4G3LG5FBCBAWQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1dC7mC2Cbxmys7ukEkELyF9Us7F1qtptSP6yDJh1EfjVJ6uqsC): protocol: expected piece hash; context canceled; EOF Dec 28 20:32:04.376585127 (557h3m4.247668163s) [noise=true , address=145.79.227.33:4904 , piece_id=QCMN4OSBWIWS4JRLIGQIGDSHZFUF2JWEPVXQ4FQ4G3LG5FBCBAWQ, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.376764359 (557h3m4.247847404s) [noise=true , address=145.79.227.19:4105 , piece_id=RY35FSM6NTWPNF3JE43FZRSNXXYZPUN2B6LRNNUOG6QQR6XSU36Q, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GhDrKMcDEg2RxamqKtiHphpgNdvtZYdHXrjzGsNJFjBFYeEen): protocol: write tcp 188.166.225.170:38838->145.79.227.19:4105: use of closed network connection; write tcp 188.166.225.170:38838->145.79.227.19:4105: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38838->145.79.227.19:4105: use of closed network connection Dec 28 20:32:04.376772865 (557h3m4.2478559s) [noise=true , address=145.79.227.19:4105 , piece_id=RY35FSM6NTWPNF3JE43FZRSNXXYZPUN2B6LRNNUOG6QQR6XSU36Q, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.376926658 (557h3m4.248009703s) [noise=true , address=145.79.227.35:4496 , piece_id=JH4XQBT7XYEJXWBGPPF6SJWCPLV77GBOLGOA53C3WUQMWHZXVZQA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V8XhSWkyGUVYFR8MykifFAP1YAfGCwzqucDcMMfzw3xsVW77Z): protocol: expected piece hash; context canceled; EOF Dec 28 20:32:04.377165332 (557h3m4.248248377s) [noise=true , address=145.79.227.35:4496 , piece_id=JH4XQBT7XYEJXWBGPPF6SJWCPLV77GBOLGOA53C3WUQMWHZXVZQA, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.377212682 (557h3m4.248295848s) [noise=true , address=45.91.116.5:4330 , piece_id=EVHLSVVR5JGY43TLSKVEQLQ6KDBEZENVAFL6YT7ZQ77Y7X2TXIKA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12MK9aVDRwfWPfckFR6RfPUL1Nrm9bwfsgV7TXCKGbtadHebWvW): protocol: read tcp 188.166.225.170:55416->45.91.116.5:4330: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 20:32:04.377289859 (557h3m4.248372924s) [noise=true , address=45.91.116.5:4330 , piece_id=EVHLSVVR5JGY43TLSKVEQLQ6KDBEZENVAFL6YT7ZQ77Y7X2TXIKA, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.377387736 (557h3m4.248470781s) [seg_pos={0 0}, upload=488362]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 20:32:04.377599729 (557h3m4.248682774s) [upload=488362]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 20:32:04.377613535 (557h3m4.24869662s) [upload=488362]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 20:32:04.675090855 (557h3m4.5461739s) [upload=488362]: Done uploading segments... Dec 28 20:32:04.675121233 (557h3m4.546204268s) [upload=488362]: Upload finished. err: <nil> Duration: 6.997772983s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.........X...........X..................X.......X......XX...] duration: 8.499064147s trace: https://jaeger.dev.storj.io/trace/3ce821e76930523a FAIL (2025-12-28 20:50:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 20:49:03.805105648 (557h20m3.676188693s) [upload=488615]: Starting upload Dec 28 20:49:03.805166324 (557h20m3.676249369s) [upload=488615]: Done starting upload Dec 28 20:49:03.805293475 (557h20m3.67637651s) [upload=488615]: Uploading segments... Dec 28 20:49:03.805403545 (557h20m3.67648658s) [upload=488615]: Got next segment. Inline: false Dec 28 20:49:03.805460854 (557h20m3.676543889s) [upload=488615]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 20:49:03.805467066 (557h20m3.676550111s) [upload=488615]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 20:49:04.052770455 (557h20m3.9238535s) [seg_pos={0 0}, upload=488615]: Begin upload segment... Dec 28 20:49:04.052835940 (557h20m3.923918985s) [seg_pos={0 0}, upload=488615]: Done begin upload segment. Dec 28 20:49:04.052874203 (557h20m3.923957248s) [noise=true , address=145.79.227.44:4194 , piece_id=WYODZVE7TGDTRNVGZ5HAD2JJT3TSOB7KGNGWH5VCQMLCCICMORRA, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.052970917 (557h20m3.924053962s) [noise=true , address=145.79.227.39:4015 , piece_id=UNPAANTMSZ3O356MXFAX357E3L4X5OLMJBXTDQEVGXUH5B5KDYIQ, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.053474816 (557h20m3.924557861s) [noise=true , address=45.91.116.8:4046 , piece_id=2DXIM4FJEZZQ3LRR4GFUEJ6XD6IGDET3JVKWBZJQGKMICWUITTTA, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.053938549 (557h20m3.925021584s) [noise=true , address=145.79.227.37:4565 , piece_id=6EAD5WOUCVJXUOKHGTSMMHDYPKHHJFZZFIXH2FRUNNZTZNQNK5IA, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.054376714 (557h20m3.925459759s) [noise=true , address=145.79.227.19:4078 , piece_id=DCMFZ42KOAHHMA3YOVW7655PLKCMJDHTLETUU4NK2D6EE27TEKWQ, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.055140448 (557h20m3.926223493s) [noise=true , address=145.79.227.36:4519 , piece_id=GQENJD2GLGJSL7H7VWUN6JXBZTW6ZHFLMMYTKYF3TRBGNNBQC3YQ, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.055719992 (557h20m3.926803027s) [noise=true , address=145.79.227.51:4411 , piece_id=IMS72BZZCJ3WMXU3FTBKM3F4MSXTL4VCVCPOCNLQI6WHKJGIOXQQ, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.056924334 (557h20m3.92800737s) [noise=true , address=45.91.116.6:4973 , piece_id=HMTEGMPYPTWCLRSAH45LLVZTTG2TRVUS44MTKBPZT6SIMRFYYNGQ, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.058635102 (557h20m3.929718147s) [noise=true , address=145.79.227.13:4804 , piece_id=UYO4W6FOSDQSVVEO6XWVTGXMFLXYWNYA3DMICFUABVSFFDKPFRXA, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.058693383 (557h20m3.929776418s) [noise=true , address=45.91.116.5:4299 , piece_id=HI6B35X52HVYVDIFVSJQT6TXYBBDA5VB5COKH53CPKDTO3UHKWJA, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.061111298 (557h20m3.932194343s) [noise=true , address=77.90.31.14:4661 , piece_id=OP5ES2G4MLZBJWJDYYPO7KLHUC6NDX5FUF6ZO6PCYMCFSFTKPVDA, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.061856947 (557h20m3.932939992s) [noise=true , address=145.79.227.42:4123 , piece_id=6YEKCEU5A5MPTRVGRFH2GWQJVSWMBXHCIZSAJKPUY2M2I3P63SFQ, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.064452048 (557h20m3.935535093s) [noise=true , address=45.91.116.10:4082 , piece_id=WGN35FKEXP3UEANOQA4NSJUQLDKIZ36RYD6OFP2FHEBNJKLI2TTA, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.066246636 (557h20m3.937329681s) [noise=true , address=145.79.227.5:4221 , piece_id=GVQO2XLQ2D6XVAHLGZVB6SHA476DL3TUHAXLFKZU5H2BN3LEB5XQ, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.069098717 (557h20m3.940181762s) [noise=true , address=145.79.227.38:4977 , piece_id=23SZDYDT2OWMLO7EGT6OT5L4S4WYR5LY2SWCCZGSCZMKQ5RK6XPQ, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.070882644 (557h20m3.941965689s) [noise=true , address=145.79.227.32:4834 , piece_id=52YM22AS3BG3OGILTWR36HSHD5SKNRKKUR5TIE7HLWOSK3L6KQOQ, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:05.778149774 (557h20m5.649232819s) [upload=488615]: Next returned nil segment Dec 28 20:49:05.778171825 (557h20m5.64925487s) [upload=488615]: Waiting for error group managing segments... Dec 28 20:49:06.067849953 (557h20m5.938932998s) [noise=true , address=145.79.227.37:4565 , piece_id=6EAD5WOUCVJXUOKHGTSMMHDYPKHHJFZZFIXH2FRUNNZTZNQNK5IA, seg_pos={0 0}, upload=488615]: Done uploading piece. err: <nil> Dec 28 20:49:06.073409289 (557h20m5.944492334s) [noise=true , address=145.79.227.39:4015 , piece_id=UNPAANTMSZ3O356MXFAX357E3L4X5OLMJBXTDQEVGXUH5B5KDYIQ, seg_pos={0 0}, upload=488615]: Done uploading piece. err: <nil> Dec 28 20:49:06.078797668 (557h20m5.949880714s) [noise=true , address=145.79.227.13:4804 , piece_id=UYO4W6FOSDQSVVEO6XWVTGXMFLXYWNYA3DMICFUABVSFFDKPFRXA, seg_pos={0 0}, upload=488615]: Done uploading piece. err: <nil> Dec 28 20:49:06.084142657 (557h20m5.955225692s) [noise=true , address=145.79.227.32:4834 , piece_id=52YM22AS3BG3OGILTWR36HSHD5SKNRKKUR5TIE7HLWOSK3L6KQOQ, seg_pos={0 0}, upload=488615]: Done uploading piece. err: <nil> Dec 28 20:49:06.084205807 (557h20m5.955288842s) [noise=true , address=145.79.227.19:4078 , piece_id=DCMFZ42KOAHHMA3YOVW7655PLKCMJDHTLETUU4NK2D6EE27TEKWQ, seg_pos={0 0}, upload=488615]: Done uploading piece. err: <nil> Dec 28 20:49:06.085908499 (557h20m5.956991534s) [noise=true , address=145.79.227.51:4411 , piece_id=IMS72BZZCJ3WMXU3FTBKM3F4MSXTL4VCVCPOCNLQI6WHKJGIOXQQ, seg_pos={0 0}, upload=488615]: Done uploading piece. err: <nil> Dec 28 20:49:06.089752589 (557h20m5.960835624s) [noise=true , address=145.79.227.5:4221 , piece_id=GVQO2XLQ2D6XVAHLGZVB6SHA476DL3TUHAXLFKZU5H2BN3LEB5XQ, seg_pos={0 0}, upload=488615]: Done uploading piece. err: <nil> Dec 28 20:49:06.096096639 (557h20m5.967179674s) [noise=true , address=145.79.227.36:4519 , piece_id=GQENJD2GLGJSL7H7VWUN6JXBZTW6ZHFLMMYTKYF3TRBGNNBQC3YQ, seg_pos={0 0}, upload=488615]: Done uploading piece. err: <nil> Dec 28 20:49:06.097668182 (557h20m5.968751227s) [noise=true , address=145.79.227.42:4123 , piece_id=6YEKCEU5A5MPTRVGRFH2GWQJVSWMBXHCIZSAJKPUY2M2I3P63SFQ, seg_pos={0 0}, upload=488615]: Done uploading piece. err: <nil> Dec 28 20:49:06.107376347 (557h20m5.978459392s) [noise=true , address=145.79.227.38:4977 , piece_id=23SZDYDT2OWMLO7EGT6OT5L4S4WYR5LY2SWCCZGSCZMKQ5RK6XPQ, seg_pos={0 0}, upload=488615]: Done uploading piece. err: <nil> Dec 28 20:49:06.255907707 (557h20m6.126990752s) [noise=true , address=77.90.31.14:4661 , piece_id=OP5ES2G4MLZBJWJDYYPO7KLHUC6NDX5FUF6ZO6PCYMCFSFTKPVDA, seg_pos={0 0}, upload=488615]: Done uploading piece. err: <nil> Dec 28 20:49:06.306350829 (557h20m6.177433874s) [noise=true , address=45.91.116.5:4299 , piece_id=HI6B35X52HVYVDIFVSJQT6TXYBBDA5VB5COKH53CPKDTO3UHKWJA, seg_pos={0 0}, upload=488615]: Done uploading piece. err: <nil> Dec 28 20:49:06.312533491 (557h20m6.183616536s) [noise=true , address=45.91.116.8:4046 , piece_id=2DXIM4FJEZZQ3LRR4GFUEJ6XD6IGDET3JVKWBZJQGKMICWUITTTA, seg_pos={0 0}, upload=488615]: Done uploading piece. err: <nil> Dec 28 20:49:06.312558800 (557h20m6.183641845s) [seg_pos={0 0}, upload=488615]: Segment reached optimal threshold of 13 pieces. Dec 28 20:49:06.313074843 (557h20m6.184157888s) [noise=true , address=145.79.227.44:4194 , piece_id=WYODZVE7TGDTRNVGZ5HAD2JJT3TSOB7KGNGWH5VCQMLCCICMORRA, seg_pos={0 0}, upload=488615]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 28 20:49:06.313086825 (557h20m6.18416987s) [noise=true , address=145.79.227.44:4194 , piece_id=WYODZVE7TGDTRNVGZ5HAD2JJT3TSOB7KGNGWH5VCQMLCCICMORRA, seg_pos={0 0}, upload=488615]: Long tail cancellation due to optimal threshold Dec 28 20:49:06.313142812 (557h20m6.184225847s) [noise=true , address=45.91.116.6:4973 , piece_id=HMTEGMPYPTWCLRSAH45LLVZTTG2TRVUS44MTKBPZT6SIMRFYYNGQ, seg_pos={0 0}, upload=488615]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF Dec 28 20:49:06.313177759 (557h20m6.184260804s) [noise=true , address=45.91.116.6:4973 , piece_id=HMTEGMPYPTWCLRSAH45LLVZTTG2TRVUS44MTKBPZT6SIMRFYYNGQ, seg_pos={0 0}, upload=488615]: Long tail cancellation due to optimal threshold Dec 28 20:49:06.313256448 (557h20m6.184339503s) [noise=true , address=45.91.116.10:4082 , piece_id=WGN35FKEXP3UEANOQA4NSJUQLDKIZ36RYD6OFP2FHEBNJKLI2TTA, seg_pos={0 0}, upload=488615]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 28 20:49:06.313271217 (557h20m6.184354262s) [noise=true , address=45.91.116.10:4082 , piece_id=WGN35FKEXP3UEANOQA4NSJUQLDKIZ36RYD6OFP2FHEBNJKLI2TTA, seg_pos={0 0}, upload=488615]: Long tail cancellation due to optimal threshold Dec 28 20:49:06.313296314 (557h20m6.184379359s) [seg_pos={0 0}, upload=488615]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 20:49:06.313381627 (557h20m6.184464672s) [upload=488615]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 20:49:06.313389141 (557h20m6.184472186s) [upload=488615]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 20:49:06.604901881 (557h20m6.475984916s) [upload=488615]: Done uploading segments... Dec 28 20:49:06.604926539 (557h20m6.476009584s) [upload=488615]: Upload finished. err: <nil> Duration: 8.49909641s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..........X...........X..................X.......X......XX..] duration: 11.790801533s trace: https://jaeger.dev.storj.io/trace/3ce821e76930523a FAIL (2025-12-28 20:50:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 20:49:03.805105648 (557h20m3.676188693s) [upload=488615]: Starting upload Dec 28 20:49:03.805166324 (557h20m3.676249369s) [upload=488615]: Done starting upload Dec 28 20:49:03.805293475 (557h20m3.67637651s) [upload=488615]: Uploading segments... Dec 28 20:49:03.805403545 (557h20m3.67648658s) [upload=488615]: Got next segment. Inline: false Dec 28 20:49:03.805460854 (557h20m3.676543889s) [upload=488615]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 20:49:03.805467066 (557h20m3.676550111s) [upload=488615]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 20:49:04.052770455 (557h20m3.9238535s) [seg_pos={0 0}, upload=488615]: Begin upload segment... Dec 28 20:49:04.052835940 (557h20m3.923918985s) [seg_pos={0 0}, upload=488615]: Done begin upload segment. Dec 28 20:49:04.052874203 (557h20m3.923957248s) [noise=true , address=145.79.227.44:4194 , piece_id=WYODZVE7TGDTRNVGZ5HAD2JJT3TSOB7KGNGWH5VCQMLCCICMORRA, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.052970917 (557h20m3.924053962s) [noise=true , address=145.79.227.39:4015 , piece_id=UNPAANTMSZ3O356MXFAX357E3L4X5OLMJBXTDQEVGXUH5B5KDYIQ, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.053474816 (557h20m3.924557861s) [noise=true , address=45.91.116.8:4046 , piece_id=2DXIM4FJEZZQ3LRR4GFUEJ6XD6IGDET3JVKWBZJQGKMICWUITTTA, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.053938549 (557h20m3.925021584s) [noise=true , address=145.79.227.37:4565 , piece_id=6EAD5WOUCVJXUOKHGTSMMHDYPKHHJFZZFIXH2FRUNNZTZNQNK5IA, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.054376714 (557h20m3.925459759s) [noise=true , address=145.79.227.19:4078 , piece_id=DCMFZ42KOAHHMA3YOVW7655PLKCMJDHTLETUU4NK2D6EE27TEKWQ, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.055140448 (557h20m3.926223493s) [noise=true , address=145.79.227.36:4519 , piece_id=GQENJD2GLGJSL7H7VWUN6JXBZTW6ZHFLMMYTKYF3TRBGNNBQC3YQ, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.055719992 (557h20m3.926803027s) [noise=true , address=145.79.227.51:4411 , piece_id=IMS72BZZCJ3WMXU3FTBKM3F4MSXTL4VCVCPOCNLQI6WHKJGIOXQQ, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.056924334 (557h20m3.92800737s) [noise=true , address=45.91.116.6:4973 , piece_id=HMTEGMPYPTWCLRSAH45LLVZTTG2TRVUS44MTKBPZT6SIMRFYYNGQ, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.058635102 (557h20m3.929718147s) [noise=true , address=145.79.227.13:4804 , piece_id=UYO4W6FOSDQSVVEO6XWVTGXMFLXYWNYA3DMICFUABVSFFDKPFRXA, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.058693383 (557h20m3.929776418s) [noise=true , address=45.91.116.5:4299 , piece_id=HI6B35X52HVYVDIFVSJQT6TXYBBDA5VB5COKH53CPKDTO3UHKWJA, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.061111298 (557h20m3.932194343s) [noise=true , address=77.90.31.14:4661 , piece_id=OP5ES2G4MLZBJWJDYYPO7KLHUC6NDX5FUF6ZO6PCYMCFSFTKPVDA, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.061856947 (557h20m3.932939992s) [noise=true , address=145.79.227.42:4123 , piece_id=6YEKCEU5A5MPTRVGRFH2GWQJVSWMBXHCIZSAJKPUY2M2I3P63SFQ, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.064452048 (557h20m3.935535093s) [noise=true , address=45.91.116.10:4082 , piece_id=WGN35FKEXP3UEANOQA4NSJUQLDKIZ36RYD6OFP2FHEBNJKLI2TTA, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.066246636 (557h20m3.937329681s) [noise=true , address=145.79.227.5:4221 , piece_id=GVQO2XLQ2D6XVAHLGZVB6SHA476DL3TUHAXLFKZU5H2BN3LEB5XQ, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.069098717 (557h20m3.940181762s) [noise=true , address=145.79.227.38:4977 , piece_id=23SZDYDT2OWMLO7EGT6OT5L4S4WYR5LY2SWCCZGSCZMKQ5RK6XPQ, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.070882644 (557h20m3.941965689s) [noise=true , address=145.79.227.32:4834 , piece_id=52YM22AS3BG3OGILTWR36HSHD5SKNRKKUR5TIE7HLWOSK3L6KQOQ, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:05.778149774 (557h20m5.649232819s) [upload=488615]: Next returned nil segment Dec 28 20:49:05.778171825 (557h20m5.64925487s) [upload=488615]: Waiting for error group managing segments... Dec 28 20:49:06.067849953 (557h20m5.938932998s) [noise=true , address=145.79.227.37:4565 , piece_id=6EAD5WOUCVJXUOKHGTSMMHDYPKHHJFZZFIXH2FRUNNZTZNQNK5IA, seg_pos={0 0}, upload=488615]: Done uploading piece. err: <nil> Dec 28 20:49:06.073409289 (557h20m5.944492334s) [noise=true , address=145.79.227.39:4015 , piece_id=UNPAANTMSZ3O356MXFAX357E3L4X5OLMJBXTDQEVGXUH5B5KDYIQ, seg_pos={0 0}, upload=488615]: Done uploading piece. err: <nil> Dec 28 20:49:06.078797668 (557h20m5.949880714s) [noise=true , address=145.79.227.13:4804 , piece_id=UYO4W6FOSDQSVVEO6XWVTGXMFLXYWNYA3DMICFUABVSFFDKPFRXA, seg_pos={0 0}, upload=488615]: Done uploading piece. err: <nil> Dec 28 20:49:06.084142657 (557h20m5.955225692s) [noise=true , address=145.79.227.32:4834 , piece_id=52YM22AS3BG3OGILTWR36HSHD5SKNRKKUR5TIE7HLWOSK3L6KQOQ, seg_pos={0 0}, upload=488615]: Done uploading piece. err: <nil> Dec 28 20:49:06.084205807 (557h20m5.955288842s) [noise=true , address=145.79.227.19:4078 , piece_id=DCMFZ42KOAHHMA3YOVW7655PLKCMJDHTLETUU4NK2D6EE27TEKWQ, seg_pos={0 0}, upload=488615]: Done uploading piece. err: <nil> Dec 28 20:49:06.085908499 (557h20m5.956991534s) [noise=true , address=145.79.227.51:4411 , piece_id=IMS72BZZCJ3WMXU3FTBKM3F4MSXTL4VCVCPOCNLQI6WHKJGIOXQQ, seg_pos={0 0}, upload=488615]: Done uploading piece. err: <nil> Dec 28 20:49:06.089752589 (557h20m5.960835624s) [noise=true , address=145.79.227.5:4221 , piece_id=GVQO2XLQ2D6XVAHLGZVB6SHA476DL3TUHAXLFKZU5H2BN3LEB5XQ, seg_pos={0 0}, upload=488615]: Done uploading piece. err: <nil> Dec 28 20:49:06.096096639 (557h20m5.967179674s) [noise=true , address=145.79.227.36:4519 , piece_id=GQENJD2GLGJSL7H7VWUN6JXBZTW6ZHFLMMYTKYF3TRBGNNBQC3YQ, seg_pos={0 0}, upload=488615]: Done uploading piece. err: <nil> Dec 28 20:49:06.097668182 (557h20m5.968751227s) [noise=true , address=145.79.227.42:4123 , piece_id=6YEKCEU5A5MPTRVGRFH2GWQJVSWMBXHCIZSAJKPUY2M2I3P63SFQ, seg_pos={0 0}, upload=488615]: Done uploading piece. err: <nil> Dec 28 20:49:06.107376347 (557h20m5.978459392s) [noise=true , address=145.79.227.38:4977 , piece_id=23SZDYDT2OWMLO7EGT6OT5L4S4WYR5LY2SWCCZGSCZMKQ5RK6XPQ, seg_pos={0 0}, upload=488615]: Done uploading piece. err: <nil> Dec 28 20:49:06.255907707 (557h20m6.126990752s) [noise=true , address=77.90.31.14:4661 , piece_id=OP5ES2G4MLZBJWJDYYPO7KLHUC6NDX5FUF6ZO6PCYMCFSFTKPVDA, seg_pos={0 0}, upload=488615]: Done uploading piece. err: <nil> Dec 28 20:49:06.306350829 (557h20m6.177433874s) [noise=true , address=45.91.116.5:4299 , piece_id=HI6B35X52HVYVDIFVSJQT6TXYBBDA5VB5COKH53CPKDTO3UHKWJA, seg_pos={0 0}, upload=488615]: Done uploading piece. err: <nil> Dec 28 20:49:06.312533491 (557h20m6.183616536s) [noise=true , address=45.91.116.8:4046 , piece_id=2DXIM4FJEZZQ3LRR4GFUEJ6XD6IGDET3JVKWBZJQGKMICWUITTTA, seg_pos={0 0}, upload=488615]: Done uploading piece. err: <nil> Dec 28 20:49:06.312558800 (557h20m6.183641845s) [seg_pos={0 0}, upload=488615]: Segment reached optimal threshold of 13 pieces. Dec 28 20:49:06.313074843 (557h20m6.184157888s) [noise=true , address=145.79.227.44:4194 , piece_id=WYODZVE7TGDTRNVGZ5HAD2JJT3TSOB7KGNGWH5VCQMLCCICMORRA, seg_pos={0 0}, upload=488615]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 28 20:49:06.313086825 (557h20m6.18416987s) [noise=true , address=145.79.227.44:4194 , piece_id=WYODZVE7TGDTRNVGZ5HAD2JJT3TSOB7KGNGWH5VCQMLCCICMORRA, seg_pos={0 0}, upload=488615]: Long tail cancellation due to optimal threshold Dec 28 20:49:06.313142812 (557h20m6.184225847s) [noise=true , address=45.91.116.6:4973 , piece_id=HMTEGMPYPTWCLRSAH45LLVZTTG2TRVUS44MTKBPZT6SIMRFYYNGQ, seg_pos={0 0}, upload=488615]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF Dec 28 20:49:06.313177759 (557h20m6.184260804s) [noise=true , address=45.91.116.6:4973 , piece_id=HMTEGMPYPTWCLRSAH45LLVZTTG2TRVUS44MTKBPZT6SIMRFYYNGQ, seg_pos={0 0}, upload=488615]: Long tail cancellation due to optimal threshold Dec 28 20:49:06.313256448 (557h20m6.184339503s) [noise=true , address=45.91.116.10:4082 , piece_id=WGN35FKEXP3UEANOQA4NSJUQLDKIZ36RYD6OFP2FHEBNJKLI2TTA, seg_pos={0 0}, upload=488615]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 28 20:49:06.313271217 (557h20m6.184354262s) [noise=true , address=45.91.116.10:4082 , piece_id=WGN35FKEXP3UEANOQA4NSJUQLDKIZ36RYD6OFP2FHEBNJKLI2TTA, seg_pos={0 0}, upload=488615]: Long tail cancellation due to optimal threshold Dec 28 20:49:06.313296314 (557h20m6.184379359s) [seg_pos={0 0}, upload=488615]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 20:49:06.313381627 (557h20m6.184464672s) [upload=488615]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 20:49:06.313389141 (557h20m6.184472186s) [upload=488615]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 20:49:06.604901881 (557h20m6.475984916s) [upload=488615]: Done uploading segments... Dec 28 20:49:06.604926539 (557h20m6.476009584s) [upload=488615]: Upload finished. err: <nil> Duration: 11.79083107s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 15.380183402s trace: https://jaeger.dev.storj.io/trace/499b96822b8dda80 FAIL (2025-12-28 20:50:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12S7WBBQ1Nq8jbeGrZiRKTk23CdJC31hPW2vU1aaM5SsHeVyeRH, Piece ID: V4P3KCUDLI543TSROBYUMVTQKKSQDDVSKSJSXKEZTQAHTXUJ2G4Q) context deadline exceeded; piecestore: (Node ID: 1kjCEPtjvcJ3D7BBKRSRirPybBwkRZ1dJwyAQCadzCvaLYvWhz, Piece ID: 4OULHJ6Z3QJ3BOPDY74TJSKVJG23TOEUCJCUEGY24I5C3MUTAVLQ) context deadline exceeded; piecestore: (Node ID: 1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c, Piece ID: LIYXCPAZOUPKXLKRH3EOWREXUY2LAJXSRZFQILM2QIU7HTW7SY2Q) context deadline exceeded ======LOGS====== ue , address=145.79.227.23:4580 , piece_id=OHYI2SFM6TKJE4LHVNGO2PVGZDNFIQDAIQGKG4FOEHTTASBWA5ZQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.496428825 (557h3m1.36751187s) [noise=true , address=77.90.31.19:4867 , piece_id=LDAWD2ZFG3BZATDRY7MMNWYIX6TOHQ2OIUPCAWAYN266GNUUKZ5A, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.496861509 (557h3m1.367944544s) [noise=true , address=145.79.227.19:4105 , piece_id=RY35FSM6NTWPNF3JE43FZRSNXXYZPUN2B6LRNNUOG6QQR6XSU36Q, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.496899933 (557h3m1.367982978s) [noise=true , address=77.90.31.19:4858 , piece_id=LIYXCPAZOUPKXLKRH3EOWREXUY2LAJXSRZFQILM2QIU7HTW7SY2Q, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497229089 (557h3m1.368312134s) [noise=true , address=77.90.31.17:4784 , piece_id=JV2ROD6MN65B6LZKOL3S7ATJWXAY6FBQSZWITA6IHAXELY7R42KA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497326194 (557h3m1.368409239s) [noise=true , address=145.79.227.33:4904 , piece_id=QCMN4OSBWIWS4JRLIGQIGDSHZFUF2JWEPVXQ4FQ4G3LG5FBCBAWQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497664128 (557h3m1.368747173s) [noise=true , address=145.79.227.34:4918 , piece_id=SPDT2KI2774UJMQIUSVTHXCDF7PD2FBG64BXUYW67UMLPE6H7SBQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497731405 (557h3m1.36881445s) [noise=true , address=77.90.31.12:4602 , piece_id=2YPYKKTQ6ID6VTI4AUZQEG2SY7JYPRSAPU4JLF7DWSOSBBO74LWQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.498205939 (557h3m1.369288984s) [noise=true , address=77.90.31.16:4745 , piece_id=CZBFU33BHMP2BEZIDJKW3J526537HR6HP7OCT2KMAQVY7LGZLPZQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.498424295 (557h3m1.36950734s) [noise=true , address=77.90.31.21:4895 , piece_id=V4P3KCUDLI543TSROBYUMVTQKKSQDDVSKSJSXKEZTQAHTXUJ2G4Q, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.498685422 (557h3m1.369768457s) [noise=true , address=145.79.227.35:4496 , piece_id=JH4XQBT7XYEJXWBGPPF6SJWCPLV77GBOLGOA53C3WUQMWHZXVZQA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499066348 (557h3m1.370149393s) [noise=true , address=77.90.31.11:4563 , piece_id=DIFJPHSKHROTLKCHP6HQBWT7XGE77HVRZTH2354MV2UEWZLHVGCQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499185725 (557h3m1.37026877s) [noise=true , address=145.79.227.30:4768 , piece_id=KK4LEV6RG3IGLYXID5MJF3KMNSJ7WS5JS45PNOOJ26LXLYAUCWOA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499468122 (557h3m1.370551157s) [noise=true , address=77.90.31.14:4688 , piece_id=4OULHJ6Z3QJ3BOPDY74TJSKVJG23TOEUCJCUEGY24I5C3MUTAVLQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499901047 (557h3m1.370984102s) [noise=true , address=45.91.116.5:4330 , piece_id=EVHLSVVR5JGY43TLSKVEQLQ6KDBEZENVAFL6YT7ZQ77Y7X2TXIKA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.500546175 (557h3m1.37162921s) [noise=true , address=145.79.227.43:4171 , piece_id=YF5DRDPGRRVYFL5SWMH2PKTSV3C72BIWKUDSAIZTXOIVOTYKRMNA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:02.979572381 (557h3m2.850655436s) [upload=488362]: Next returned nil segment Dec 28 20:32:02.979609411 (557h3m2.850692456s) [upload=488362]: Waiting for error group managing segments... Dec 28 20:32:03.416896733 (557h3m3.287979778s) [noise=true , address=145.79.227.23:4580 , piece_id=OHYI2SFM6TKJE4LHVNGO2PVGZDNFIQDAIQGKG4FOEHTTASBWA5ZQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.552263655 (557h3m3.4233467s) [noise=true , address=77.90.31.17:4784 , piece_id=JV2ROD6MN65B6LZKOL3S7ATJWXAY6FBQSZWITA6IHAXELY7R42KA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.573564856 (557h3m3.444647902s) [noise=true , address=77.90.31.12:4602 , piece_id=2YPYKKTQ6ID6VTI4AUZQEG2SY7JYPRSAPU4JLF7DWSOSBBO74LWQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.598560923 (557h3m3.469643968s) [noise=true , address=77.90.31.19:4867 , piece_id=LDAWD2ZFG3BZATDRY7MMNWYIX6TOHQ2OIUPCAWAYN266GNUUKZ5A, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.626925925 (557h3m3.49800897s) [noise=true , address=77.90.31.16:4745 , piece_id=CZBFU33BHMP2BEZIDJKW3J526537HR6HP7OCT2KMAQVY7LGZLPZQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.709061947 (557h3m3.580144992s) [noise=true , address=77.90.31.14:4688 , piece_id=4OULHJ6Z3QJ3BOPDY74TJSKVJG23TOEUCJCUEGY24I5C3MUTAVLQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.788879373 (557h3m3.659962418s) [noise=true , address=77.90.31.21:4895 , piece_id=V4P3KCUDLI543TSROBYUMVTQKKSQDDVSKSJSXKEZTQAHTXUJ2G4Q, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.217616523 (557h3m4.088699569s) [noise=true , address=77.90.31.19:4858 , piece_id=LIYXCPAZOUPKXLKRH3EOWREXUY2LAJXSRZFQILM2QIU7HTW7SY2Q, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.220337625 (557h3m4.09142067s) [noise=true , address=77.90.31.11:4563 , piece_id=DIFJPHSKHROTLKCHP6HQBWT7XGE77HVRZTH2354MV2UEWZLHVGCQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.303958744 (557h3m4.175041789s) [noise=true , address=145.79.227.44:4213 , piece_id=UEAQDBBCP7DIG6ZD3EEM2CKAL4W5QKW23LWFUAMSVUW5ZS2PJDRQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.312794398 (557h3m4.183877444s) [noise=true , address=145.79.227.34:4918 , piece_id=SPDT2KI2774UJMQIUSVTHXCDF7PD2FBG64BXUYW67UMLPE6H7SBQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.349308798 (557h3m4.220391833s) [noise=true , address=145.79.227.40:4058 , piece_id=EQ4LA7T2BCGWY76K3NJI4X77OQJFJED6H4VNGXDQMILU6IVQWOMQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.371518379 (557h3m4.242601424s) [noise=true , address=145.79.227.43:4171 , piece_id=YF5DRDPGRRVYFL5SWMH2PKTSV3C72BIWKUDSAIZTXOIVOTYKRMNA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.376232316 (557h3m4.247315361s) [noise=true , address=145.79.227.30:4768 , piece_id=KK4LEV6RG3IGLYXID5MJF3KMNSJ7WS5JS45PNOOJ26LXLYAUCWOA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.376257844 (557h3m4.247340879s) [seg_pos={0 0}, upload=488362]: Segment reached optimal threshold of 14 pieces. Dec 28 20:32:04.376575799 (557h3m4.247658834s) [noise=true , address=145.79.227.33:4904 , piece_id=QCMN4OSBWIWS4JRLIGQIGDSHZFUF2JWEPVXQ4FQ4G3LG5FBCBAWQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1dC7mC2Cbxmys7ukEkELyF9Us7F1qtptSP6yDJh1EfjVJ6uqsC): protocol: expected piece hash; context canceled; EOF Dec 28 20:32:04.376585127 (557h3m4.247668163s) [noise=true , address=145.79.227.33:4904 , piece_id=QCMN4OSBWIWS4JRLIGQIGDSHZFUF2JWEPVXQ4FQ4G3LG5FBCBAWQ, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.376764359 (557h3m4.247847404s) [noise=true , address=145.79.227.19:4105 , piece_id=RY35FSM6NTWPNF3JE43FZRSNXXYZPUN2B6LRNNUOG6QQR6XSU36Q, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GhDrKMcDEg2RxamqKtiHphpgNdvtZYdHXrjzGsNJFjBFYeEen): protocol: write tcp 188.166.225.170:38838->145.79.227.19:4105: use of closed network connection; write tcp 188.166.225.170:38838->145.79.227.19:4105: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38838->145.79.227.19:4105: use of closed network connection Dec 28 20:32:04.376772865 (557h3m4.2478559s) [noise=true , address=145.79.227.19:4105 , piece_id=RY35FSM6NTWPNF3JE43FZRSNXXYZPUN2B6LRNNUOG6QQR6XSU36Q, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.376926658 (557h3m4.248009703s) [noise=true , address=145.79.227.35:4496 , piece_id=JH4XQBT7XYEJXWBGPPF6SJWCPLV77GBOLGOA53C3WUQMWHZXVZQA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V8XhSWkyGUVYFR8MykifFAP1YAfGCwzqucDcMMfzw3xsVW77Z): protocol: expected piece hash; context canceled; EOF Dec 28 20:32:04.377165332 (557h3m4.248248377s) [noise=true , address=145.79.227.35:4496 , piece_id=JH4XQBT7XYEJXWBGPPF6SJWCPLV77GBOLGOA53C3WUQMWHZXVZQA, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.377212682 (557h3m4.248295848s) [noise=true , address=45.91.116.5:4330 , piece_id=EVHLSVVR5JGY43TLSKVEQLQ6KDBEZENVAFL6YT7ZQ77Y7X2TXIKA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12MK9aVDRwfWPfckFR6RfPUL1Nrm9bwfsgV7TXCKGbtadHebWvW): protocol: read tcp 188.166.225.170:55416->45.91.116.5:4330: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 20:32:04.377289859 (557h3m4.248372924s) [noise=true , address=45.91.116.5:4330 , piece_id=EVHLSVVR5JGY43TLSKVEQLQ6KDBEZENVAFL6YT7ZQ77Y7X2TXIKA, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.377387736 (557h3m4.248470781s) [seg_pos={0 0}, upload=488362]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 20:32:04.377599729 (557h3m4.248682774s) [upload=488362]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 20:32:04.377613535 (557h3m4.24869662s) [upload=488362]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 20:32:04.675090855 (557h3m4.5461739s) [upload=488362]: Done uploading segments... Dec 28 20:32:04.675121233 (557h3m4.546204268s) [upload=488362]: Upload finished. err: <nil> Duration: 15.380230131s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [...........X...........X..................X.......X......XX.] duration: 11.714249736s trace: https://jaeger.dev.storj.io/trace/3ce821e76930523a FAIL (2025-12-28 20:50:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 20:49:03.805105648 (557h20m3.676188693s) [upload=488615]: Starting upload Dec 28 20:49:03.805166324 (557h20m3.676249369s) [upload=488615]: Done starting upload Dec 28 20:49:03.805293475 (557h20m3.67637651s) [upload=488615]: Uploading segments... Dec 28 20:49:03.805403545 (557h20m3.67648658s) [upload=488615]: Got next segment. Inline: false Dec 28 20:49:03.805460854 (557h20m3.676543889s) [upload=488615]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 20:49:03.805467066 (557h20m3.676550111s) [upload=488615]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 20:49:04.052770455 (557h20m3.9238535s) [seg_pos={0 0}, upload=488615]: Begin upload segment... Dec 28 20:49:04.052835940 (557h20m3.923918985s) [seg_pos={0 0}, upload=488615]: Done begin upload segment. Dec 28 20:49:04.052874203 (557h20m3.923957248s) [noise=true , address=145.79.227.44:4194 , piece_id=WYODZVE7TGDTRNVGZ5HAD2JJT3TSOB7KGNGWH5VCQMLCCICMORRA, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.052970917 (557h20m3.924053962s) [noise=true , address=145.79.227.39:4015 , piece_id=UNPAANTMSZ3O356MXFAX357E3L4X5OLMJBXTDQEVGXUH5B5KDYIQ, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.053474816 (557h20m3.924557861s) [noise=true , address=45.91.116.8:4046 , piece_id=2DXIM4FJEZZQ3LRR4GFUEJ6XD6IGDET3JVKWBZJQGKMICWUITTTA, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.053938549 (557h20m3.925021584s) [noise=true , address=145.79.227.37:4565 , piece_id=6EAD5WOUCVJXUOKHGTSMMHDYPKHHJFZZFIXH2FRUNNZTZNQNK5IA, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.054376714 (557h20m3.925459759s) [noise=true , address=145.79.227.19:4078 , piece_id=DCMFZ42KOAHHMA3YOVW7655PLKCMJDHTLETUU4NK2D6EE27TEKWQ, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.055140448 (557h20m3.926223493s) [noise=true , address=145.79.227.36:4519 , piece_id=GQENJD2GLGJSL7H7VWUN6JXBZTW6ZHFLMMYTKYF3TRBGNNBQC3YQ, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.055719992 (557h20m3.926803027s) [noise=true , address=145.79.227.51:4411 , piece_id=IMS72BZZCJ3WMXU3FTBKM3F4MSXTL4VCVCPOCNLQI6WHKJGIOXQQ, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.056924334 (557h20m3.92800737s) [noise=true , address=45.91.116.6:4973 , piece_id=HMTEGMPYPTWCLRSAH45LLVZTTG2TRVUS44MTKBPZT6SIMRFYYNGQ, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.058635102 (557h20m3.929718147s) [noise=true , address=145.79.227.13:4804 , piece_id=UYO4W6FOSDQSVVEO6XWVTGXMFLXYWNYA3DMICFUABVSFFDKPFRXA, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.058693383 (557h20m3.929776418s) [noise=true , address=45.91.116.5:4299 , piece_id=HI6B35X52HVYVDIFVSJQT6TXYBBDA5VB5COKH53CPKDTO3UHKWJA, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.061111298 (557h20m3.932194343s) [noise=true , address=77.90.31.14:4661 , piece_id=OP5ES2G4MLZBJWJDYYPO7KLHUC6NDX5FUF6ZO6PCYMCFSFTKPVDA, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.061856947 (557h20m3.932939992s) [noise=true , address=145.79.227.42:4123 , piece_id=6YEKCEU5A5MPTRVGRFH2GWQJVSWMBXHCIZSAJKPUY2M2I3P63SFQ, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.064452048 (557h20m3.935535093s) [noise=true , address=45.91.116.10:4082 , piece_id=WGN35FKEXP3UEANOQA4NSJUQLDKIZ36RYD6OFP2FHEBNJKLI2TTA, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.066246636 (557h20m3.937329681s) [noise=true , address=145.79.227.5:4221 , piece_id=GVQO2XLQ2D6XVAHLGZVB6SHA476DL3TUHAXLFKZU5H2BN3LEB5XQ, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.069098717 (557h20m3.940181762s) [noise=true , address=145.79.227.38:4977 , piece_id=23SZDYDT2OWMLO7EGT6OT5L4S4WYR5LY2SWCCZGSCZMKQ5RK6XPQ, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.070882644 (557h20m3.941965689s) [noise=true , address=145.79.227.32:4834 , piece_id=52YM22AS3BG3OGILTWR36HSHD5SKNRKKUR5TIE7HLWOSK3L6KQOQ, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:05.778149774 (557h20m5.649232819s) [upload=488615]: Next returned nil segment Dec 28 20:49:05.778171825 (557h20m5.64925487s) [upload=488615]: Waiting for error group managing segments... Dec 28 20:49:06.067849953 (557h20m5.938932998s) [noise=true , address=145.79.227.37:4565 , piece_id=6EAD5WOUCVJXUOKHGTSMMHDYPKHHJFZZFIXH2FRUNNZTZNQNK5IA, seg_pos={0 0}, upload=488615]: Done uploading piece. err: <nil> Dec 28 20:49:06.073409289 (557h20m5.944492334s) [noise=true , address=145.79.227.39:4015 , piece_id=UNPAANTMSZ3O356MXFAX357E3L4X5OLMJBXTDQEVGXUH5B5KDYIQ, seg_pos={0 0}, upload=488615]: Done uploading piece. err: <nil> Dec 28 20:49:06.078797668 (557h20m5.949880714s) [noise=true , address=145.79.227.13:4804 , piece_id=UYO4W6FOSDQSVVEO6XWVTGXMFLXYWNYA3DMICFUABVSFFDKPFRXA, seg_pos={0 0}, upload=488615]: Done uploading piece. err: <nil> Dec 28 20:49:06.084142657 (557h20m5.955225692s) [noise=true , address=145.79.227.32:4834 , piece_id=52YM22AS3BG3OGILTWR36HSHD5SKNRKKUR5TIE7HLWOSK3L6KQOQ, seg_pos={0 0}, upload=488615]: Done uploading piece. err: <nil> Dec 28 20:49:06.084205807 (557h20m5.955288842s) [noise=true , address=145.79.227.19:4078 , piece_id=DCMFZ42KOAHHMA3YOVW7655PLKCMJDHTLETUU4NK2D6EE27TEKWQ, seg_pos={0 0}, upload=488615]: Done uploading piece. err: <nil> Dec 28 20:49:06.085908499 (557h20m5.956991534s) [noise=true , address=145.79.227.51:4411 , piece_id=IMS72BZZCJ3WMXU3FTBKM3F4MSXTL4VCVCPOCNLQI6WHKJGIOXQQ, seg_pos={0 0}, upload=488615]: Done uploading piece. err: <nil> Dec 28 20:49:06.089752589 (557h20m5.960835624s) [noise=true , address=145.79.227.5:4221 , piece_id=GVQO2XLQ2D6XVAHLGZVB6SHA476DL3TUHAXLFKZU5H2BN3LEB5XQ, seg_pos={0 0}, upload=488615]: Done uploading piece. err: <nil> Dec 28 20:49:06.096096639 (557h20m5.967179674s) [noise=true , address=145.79.227.36:4519 , piece_id=GQENJD2GLGJSL7H7VWUN6JXBZTW6ZHFLMMYTKYF3TRBGNNBQC3YQ, seg_pos={0 0}, upload=488615]: Done uploading piece. err: <nil> Dec 28 20:49:06.097668182 (557h20m5.968751227s) [noise=true , address=145.79.227.42:4123 , piece_id=6YEKCEU5A5MPTRVGRFH2GWQJVSWMBXHCIZSAJKPUY2M2I3P63SFQ, seg_pos={0 0}, upload=488615]: Done uploading piece. err: <nil> Dec 28 20:49:06.107376347 (557h20m5.978459392s) [noise=true , address=145.79.227.38:4977 , piece_id=23SZDYDT2OWMLO7EGT6OT5L4S4WYR5LY2SWCCZGSCZMKQ5RK6XPQ, seg_pos={0 0}, upload=488615]: Done uploading piece. err: <nil> Dec 28 20:49:06.255907707 (557h20m6.126990752s) [noise=true , address=77.90.31.14:4661 , piece_id=OP5ES2G4MLZBJWJDYYPO7KLHUC6NDX5FUF6ZO6PCYMCFSFTKPVDA, seg_pos={0 0}, upload=488615]: Done uploading piece. err: <nil> Dec 28 20:49:06.306350829 (557h20m6.177433874s) [noise=true , address=45.91.116.5:4299 , piece_id=HI6B35X52HVYVDIFVSJQT6TXYBBDA5VB5COKH53CPKDTO3UHKWJA, seg_pos={0 0}, upload=488615]: Done uploading piece. err: <nil> Dec 28 20:49:06.312533491 (557h20m6.183616536s) [noise=true , address=45.91.116.8:4046 , piece_id=2DXIM4FJEZZQ3LRR4GFUEJ6XD6IGDET3JVKWBZJQGKMICWUITTTA, seg_pos={0 0}, upload=488615]: Done uploading piece. err: <nil> Dec 28 20:49:06.312558800 (557h20m6.183641845s) [seg_pos={0 0}, upload=488615]: Segment reached optimal threshold of 13 pieces. Dec 28 20:49:06.313074843 (557h20m6.184157888s) [noise=true , address=145.79.227.44:4194 , piece_id=WYODZVE7TGDTRNVGZ5HAD2JJT3TSOB7KGNGWH5VCQMLCCICMORRA, seg_pos={0 0}, upload=488615]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 28 20:49:06.313086825 (557h20m6.18416987s) [noise=true , address=145.79.227.44:4194 , piece_id=WYODZVE7TGDTRNVGZ5HAD2JJT3TSOB7KGNGWH5VCQMLCCICMORRA, seg_pos={0 0}, upload=488615]: Long tail cancellation due to optimal threshold Dec 28 20:49:06.313142812 (557h20m6.184225847s) [noise=true , address=45.91.116.6:4973 , piece_id=HMTEGMPYPTWCLRSAH45LLVZTTG2TRVUS44MTKBPZT6SIMRFYYNGQ, seg_pos={0 0}, upload=488615]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF Dec 28 20:49:06.313177759 (557h20m6.184260804s) [noise=true , address=45.91.116.6:4973 , piece_id=HMTEGMPYPTWCLRSAH45LLVZTTG2TRVUS44MTKBPZT6SIMRFYYNGQ, seg_pos={0 0}, upload=488615]: Long tail cancellation due to optimal threshold Dec 28 20:49:06.313256448 (557h20m6.184339503s) [noise=true , address=45.91.116.10:4082 , piece_id=WGN35FKEXP3UEANOQA4NSJUQLDKIZ36RYD6OFP2FHEBNJKLI2TTA, seg_pos={0 0}, upload=488615]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 28 20:49:06.313271217 (557h20m6.184354262s) [noise=true , address=45.91.116.10:4082 , piece_id=WGN35FKEXP3UEANOQA4NSJUQLDKIZ36RYD6OFP2FHEBNJKLI2TTA, seg_pos={0 0}, upload=488615]: Long tail cancellation due to optimal threshold Dec 28 20:49:06.313296314 (557h20m6.184379359s) [seg_pos={0 0}, upload=488615]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 20:49:06.313381627 (557h20m6.184464672s) [upload=488615]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 20:49:06.313389141 (557h20m6.184472186s) [upload=488615]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 20:49:06.604901881 (557h20m6.475984916s) [upload=488615]: Done uploading segments... Dec 28 20:49:06.604926539 (557h20m6.476009584s) [upload=488615]: Upload finished. err: <nil> Duration: 11.714301114s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [............X...........X..................X.......X......XX] duration: 30.003111148s trace: https://jaeger.dev.storj.io/trace/3ce821e76930523a FAIL (2025-12-28 20:49:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 20:49:03.805105648 (557h20m3.676188693s) [upload=488615]: Starting upload Dec 28 20:49:03.805166324 (557h20m3.676249369s) [upload=488615]: Done starting upload Dec 28 20:49:03.805293475 (557h20m3.67637651s) [upload=488615]: Uploading segments... Dec 28 20:49:03.805403545 (557h20m3.67648658s) [upload=488615]: Got next segment. Inline: false Dec 28 20:49:03.805460854 (557h20m3.676543889s) [upload=488615]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 20:49:03.805467066 (557h20m3.676550111s) [upload=488615]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 20:49:04.052770455 (557h20m3.9238535s) [seg_pos={0 0}, upload=488615]: Begin upload segment... Dec 28 20:49:04.052835940 (557h20m3.923918985s) [seg_pos={0 0}, upload=488615]: Done begin upload segment. Dec 28 20:49:04.052874203 (557h20m3.923957248s) [noise=true , address=145.79.227.44:4194 , piece_id=WYODZVE7TGDTRNVGZ5HAD2JJT3TSOB7KGNGWH5VCQMLCCICMORRA, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.052970917 (557h20m3.924053962s) [noise=true , address=145.79.227.39:4015 , piece_id=UNPAANTMSZ3O356MXFAX357E3L4X5OLMJBXTDQEVGXUH5B5KDYIQ, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.053474816 (557h20m3.924557861s) [noise=true , address=45.91.116.8:4046 , piece_id=2DXIM4FJEZZQ3LRR4GFUEJ6XD6IGDET3JVKWBZJQGKMICWUITTTA, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.053938549 (557h20m3.925021584s) [noise=true , address=145.79.227.37:4565 , piece_id=6EAD5WOUCVJXUOKHGTSMMHDYPKHHJFZZFIXH2FRUNNZTZNQNK5IA, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.054376714 (557h20m3.925459759s) [noise=true , address=145.79.227.19:4078 , piece_id=DCMFZ42KOAHHMA3YOVW7655PLKCMJDHTLETUU4NK2D6EE27TEKWQ, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.055140448 (557h20m3.926223493s) [noise=true , address=145.79.227.36:4519 , piece_id=GQENJD2GLGJSL7H7VWUN6JXBZTW6ZHFLMMYTKYF3TRBGNNBQC3YQ, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.055719992 (557h20m3.926803027s) [noise=true , address=145.79.227.51:4411 , piece_id=IMS72BZZCJ3WMXU3FTBKM3F4MSXTL4VCVCPOCNLQI6WHKJGIOXQQ, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.056924334 (557h20m3.92800737s) [noise=true , address=45.91.116.6:4973 , piece_id=HMTEGMPYPTWCLRSAH45LLVZTTG2TRVUS44MTKBPZT6SIMRFYYNGQ, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.058635102 (557h20m3.929718147s) [noise=true , address=145.79.227.13:4804 , piece_id=UYO4W6FOSDQSVVEO6XWVTGXMFLXYWNYA3DMICFUABVSFFDKPFRXA, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.058693383 (557h20m3.929776418s) [noise=true , address=45.91.116.5:4299 , piece_id=HI6B35X52HVYVDIFVSJQT6TXYBBDA5VB5COKH53CPKDTO3UHKWJA, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.061111298 (557h20m3.932194343s) [noise=true , address=77.90.31.14:4661 , piece_id=OP5ES2G4MLZBJWJDYYPO7KLHUC6NDX5FUF6ZO6PCYMCFSFTKPVDA, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.061856947 (557h20m3.932939992s) [noise=true , address=145.79.227.42:4123 , piece_id=6YEKCEU5A5MPTRVGRFH2GWQJVSWMBXHCIZSAJKPUY2M2I3P63SFQ, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.064452048 (557h20m3.935535093s) [noise=true , address=45.91.116.10:4082 , piece_id=WGN35FKEXP3UEANOQA4NSJUQLDKIZ36RYD6OFP2FHEBNJKLI2TTA, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.066246636 (557h20m3.937329681s) [noise=true , address=145.79.227.5:4221 , piece_id=GVQO2XLQ2D6XVAHLGZVB6SHA476DL3TUHAXLFKZU5H2BN3LEB5XQ, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.069098717 (557h20m3.940181762s) [noise=true , address=145.79.227.38:4977 , piece_id=23SZDYDT2OWMLO7EGT6OT5L4S4WYR5LY2SWCCZGSCZMKQ5RK6XPQ, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:04.070882644 (557h20m3.941965689s) [noise=true , address=145.79.227.32:4834 , piece_id=52YM22AS3BG3OGILTWR36HSHD5SKNRKKUR5TIE7HLWOSK3L6KQOQ, seg_pos={0 0}, upload=488615]: Uploading piece... Dec 28 20:49:05.778149774 (557h20m5.649232819s) [upload=488615]: Next returned nil segment Dec 28 20:49:05.778171825 (557h20m5.64925487s) [upload=488615]: Waiting for error group managing segments... Dec 28 20:49:06.067849953 (557h20m5.938932998s) [noise=true , address=145.79.227.37:4565 , piece_id=6EAD5WOUCVJXUOKHGTSMMHDYPKHHJFZZFIXH2FRUNNZTZNQNK5IA, seg_pos={0 0}, upload=488615]: Done uploading piece. err: <nil> Dec 28 20:49:06.073409289 (557h20m5.944492334s) [noise=true , address=145.79.227.39:4015 , piece_id=UNPAANTMSZ3O356MXFAX357E3L4X5OLMJBXTDQEVGXUH5B5KDYIQ, seg_pos={0 0}, upload=488615]: Done uploading piece. err: <nil> Dec 28 20:49:06.078797668 (557h20m5.949880714s) [noise=true , address=145.79.227.13:4804 , piece_id=UYO4W6FOSDQSVVEO6XWVTGXMFLXYWNYA3DMICFUABVSFFDKPFRXA, seg_pos={0 0}, upload=488615]: Done uploading piece. err: <nil> Dec 28 20:49:06.084142657 (557h20m5.955225692s) [noise=true , address=145.79.227.32:4834 , piece_id=52YM22AS3BG3OGILTWR36HSHD5SKNRKKUR5TIE7HLWOSK3L6KQOQ, seg_pos={0 0}, upload=488615]: Done uploading piece. err: <nil> Dec 28 20:49:06.084205807 (557h20m5.955288842s) [noise=true , address=145.79.227.19:4078 , piece_id=DCMFZ42KOAHHMA3YOVW7655PLKCMJDHTLETUU4NK2D6EE27TEKWQ, seg_pos={0 0}, upload=488615]: Done uploading piece. err: <nil> Dec 28 20:49:06.085908499 (557h20m5.956991534s) [noise=true , address=145.79.227.51:4411 , piece_id=IMS72BZZCJ3WMXU3FTBKM3F4MSXTL4VCVCPOCNLQI6WHKJGIOXQQ, seg_pos={0 0}, upload=488615]: Done uploading piece. err: <nil> Dec 28 20:49:06.089752589 (557h20m5.960835624s) [noise=true , address=145.79.227.5:4221 , piece_id=GVQO2XLQ2D6XVAHLGZVB6SHA476DL3TUHAXLFKZU5H2BN3LEB5XQ, seg_pos={0 0}, upload=488615]: Done uploading piece. err: <nil> Dec 28 20:49:06.096096639 (557h20m5.967179674s) [noise=true , address=145.79.227.36:4519 , piece_id=GQENJD2GLGJSL7H7VWUN6JXBZTW6ZHFLMMYTKYF3TRBGNNBQC3YQ, seg_pos={0 0}, upload=488615]: Done uploading piece. err: <nil> Dec 28 20:49:06.097668182 (557h20m5.968751227s) [noise=true , address=145.79.227.42:4123 , piece_id=6YEKCEU5A5MPTRVGRFH2GWQJVSWMBXHCIZSAJKPUY2M2I3P63SFQ, seg_pos={0 0}, upload=488615]: Done uploading piece. err: <nil> Dec 28 20:49:06.107376347 (557h20m5.978459392s) [noise=true , address=145.79.227.38:4977 , piece_id=23SZDYDT2OWMLO7EGT6OT5L4S4WYR5LY2SWCCZGSCZMKQ5RK6XPQ, seg_pos={0 0}, upload=488615]: Done uploading piece. err: <nil> Dec 28 20:49:06.255907707 (557h20m6.126990752s) [noise=true , address=77.90.31.14:4661 , piece_id=OP5ES2G4MLZBJWJDYYPO7KLHUC6NDX5FUF6ZO6PCYMCFSFTKPVDA, seg_pos={0 0}, upload=488615]: Done uploading piece. err: <nil> Dec 28 20:49:06.306350829 (557h20m6.177433874s) [noise=true , address=45.91.116.5:4299 , piece_id=HI6B35X52HVYVDIFVSJQT6TXYBBDA5VB5COKH53CPKDTO3UHKWJA, seg_pos={0 0}, upload=488615]: Done uploading piece. err: <nil> Dec 28 20:49:06.312533491 (557h20m6.183616536s) [noise=true , address=45.91.116.8:4046 , piece_id=2DXIM4FJEZZQ3LRR4GFUEJ6XD6IGDET3JVKWBZJQGKMICWUITTTA, seg_pos={0 0}, upload=488615]: Done uploading piece. err: <nil> Dec 28 20:49:06.312558800 (557h20m6.183641845s) [seg_pos={0 0}, upload=488615]: Segment reached optimal threshold of 13 pieces. Dec 28 20:49:06.313074843 (557h20m6.184157888s) [noise=true , address=145.79.227.44:4194 , piece_id=WYODZVE7TGDTRNVGZ5HAD2JJT3TSOB7KGNGWH5VCQMLCCICMORRA, seg_pos={0 0}, upload=488615]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 28 20:49:06.313086825 (557h20m6.18416987s) [noise=true , address=145.79.227.44:4194 , piece_id=WYODZVE7TGDTRNVGZ5HAD2JJT3TSOB7KGNGWH5VCQMLCCICMORRA, seg_pos={0 0}, upload=488615]: Long tail cancellation due to optimal threshold Dec 28 20:49:06.313142812 (557h20m6.184225847s) [noise=true , address=45.91.116.6:4973 , piece_id=HMTEGMPYPTWCLRSAH45LLVZTTG2TRVUS44MTKBPZT6SIMRFYYNGQ, seg_pos={0 0}, upload=488615]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF Dec 28 20:49:06.313177759 (557h20m6.184260804s) [noise=true , address=45.91.116.6:4973 , piece_id=HMTEGMPYPTWCLRSAH45LLVZTTG2TRVUS44MTKBPZT6SIMRFYYNGQ, seg_pos={0 0}, upload=488615]: Long tail cancellation due to optimal threshold Dec 28 20:49:06.313256448 (557h20m6.184339503s) [noise=true , address=45.91.116.10:4082 , piece_id=WGN35FKEXP3UEANOQA4NSJUQLDKIZ36RYD6OFP2FHEBNJKLI2TTA, seg_pos={0 0}, upload=488615]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 28 20:49:06.313271217 (557h20m6.184354262s) [noise=true , address=45.91.116.10:4082 , piece_id=WGN35FKEXP3UEANOQA4NSJUQLDKIZ36RYD6OFP2FHEBNJKLI2TTA, seg_pos={0 0}, upload=488615]: Long tail cancellation due to optimal threshold Dec 28 20:49:06.313296314 (557h20m6.184379359s) [seg_pos={0 0}, upload=488615]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 20:49:06.313381627 (557h20m6.184464672s) [upload=488615]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 20:49:06.313389141 (557h20m6.184472186s) [upload=488615]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 20:49:06.604901881 (557h20m6.475984916s) [upload=488615]: Done uploading segments... Dec 28 20:49:06.604926539 (557h20m6.476009584s) [upload=488615]: Upload finished. err: <nil> Duration: 30.00314797s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 15.429248851s trace: https://jaeger.dev.storj.io/trace/499b96822b8dda80 FAIL (2025-12-28 20:49:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12S7WBBQ1Nq8jbeGrZiRKTk23CdJC31hPW2vU1aaM5SsHeVyeRH, Piece ID: V4P3KCUDLI543TSROBYUMVTQKKSQDDVSKSJSXKEZTQAHTXUJ2G4Q) context deadline exceeded; piecestore: (Node ID: 1kjCEPtjvcJ3D7BBKRSRirPybBwkRZ1dJwyAQCadzCvaLYvWhz, Piece ID: 4OULHJ6Z3QJ3BOPDY74TJSKVJG23TOEUCJCUEGY24I5C3MUTAVLQ) context deadline exceeded; piecestore: (Node ID: 1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c, Piece ID: LIYXCPAZOUPKXLKRH3EOWREXUY2LAJXSRZFQILM2QIU7HTW7SY2Q) context deadline exceeded ======LOGS====== ue , address=145.79.227.23:4580 , piece_id=OHYI2SFM6TKJE4LHVNGO2PVGZDNFIQDAIQGKG4FOEHTTASBWA5ZQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.496428825 (557h3m1.36751187s) [noise=true , address=77.90.31.19:4867 , piece_id=LDAWD2ZFG3BZATDRY7MMNWYIX6TOHQ2OIUPCAWAYN266GNUUKZ5A, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.496861509 (557h3m1.367944544s) [noise=true , address=145.79.227.19:4105 , piece_id=RY35FSM6NTWPNF3JE43FZRSNXXYZPUN2B6LRNNUOG6QQR6XSU36Q, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.496899933 (557h3m1.367982978s) [noise=true , address=77.90.31.19:4858 , piece_id=LIYXCPAZOUPKXLKRH3EOWREXUY2LAJXSRZFQILM2QIU7HTW7SY2Q, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497229089 (557h3m1.368312134s) [noise=true , address=77.90.31.17:4784 , piece_id=JV2ROD6MN65B6LZKOL3S7ATJWXAY6FBQSZWITA6IHAXELY7R42KA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497326194 (557h3m1.368409239s) [noise=true , address=145.79.227.33:4904 , piece_id=QCMN4OSBWIWS4JRLIGQIGDSHZFUF2JWEPVXQ4FQ4G3LG5FBCBAWQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497664128 (557h3m1.368747173s) [noise=true , address=145.79.227.34:4918 , piece_id=SPDT2KI2774UJMQIUSVTHXCDF7PD2FBG64BXUYW67UMLPE6H7SBQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497731405 (557h3m1.36881445s) [noise=true , address=77.90.31.12:4602 , piece_id=2YPYKKTQ6ID6VTI4AUZQEG2SY7JYPRSAPU4JLF7DWSOSBBO74LWQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.498205939 (557h3m1.369288984s) [noise=true , address=77.90.31.16:4745 , piece_id=CZBFU33BHMP2BEZIDJKW3J526537HR6HP7OCT2KMAQVY7LGZLPZQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.498424295 (557h3m1.36950734s) [noise=true , address=77.90.31.21:4895 , piece_id=V4P3KCUDLI543TSROBYUMVTQKKSQDDVSKSJSXKEZTQAHTXUJ2G4Q, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.498685422 (557h3m1.369768457s) [noise=true , address=145.79.227.35:4496 , piece_id=JH4XQBT7XYEJXWBGPPF6SJWCPLV77GBOLGOA53C3WUQMWHZXVZQA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499066348 (557h3m1.370149393s) [noise=true , address=77.90.31.11:4563 , piece_id=DIFJPHSKHROTLKCHP6HQBWT7XGE77HVRZTH2354MV2UEWZLHVGCQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499185725 (557h3m1.37026877s) [noise=true , address=145.79.227.30:4768 , piece_id=KK4LEV6RG3IGLYXID5MJF3KMNSJ7WS5JS45PNOOJ26LXLYAUCWOA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499468122 (557h3m1.370551157s) [noise=true , address=77.90.31.14:4688 , piece_id=4OULHJ6Z3QJ3BOPDY74TJSKVJG23TOEUCJCUEGY24I5C3MUTAVLQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499901047 (557h3m1.370984102s) [noise=true , address=45.91.116.5:4330 , piece_id=EVHLSVVR5JGY43TLSKVEQLQ6KDBEZENVAFL6YT7ZQ77Y7X2TXIKA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.500546175 (557h3m1.37162921s) [noise=true , address=145.79.227.43:4171 , piece_id=YF5DRDPGRRVYFL5SWMH2PKTSV3C72BIWKUDSAIZTXOIVOTYKRMNA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:02.979572381 (557h3m2.850655436s) [upload=488362]: Next returned nil segment Dec 28 20:32:02.979609411 (557h3m2.850692456s) [upload=488362]: Waiting for error group managing segments... Dec 28 20:32:03.416896733 (557h3m3.287979778s) [noise=true , address=145.79.227.23:4580 , piece_id=OHYI2SFM6TKJE4LHVNGO2PVGZDNFIQDAIQGKG4FOEHTTASBWA5ZQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.552263655 (557h3m3.4233467s) [noise=true , address=77.90.31.17:4784 , piece_id=JV2ROD6MN65B6LZKOL3S7ATJWXAY6FBQSZWITA6IHAXELY7R42KA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.573564856 (557h3m3.444647902s) [noise=true , address=77.90.31.12:4602 , piece_id=2YPYKKTQ6ID6VTI4AUZQEG2SY7JYPRSAPU4JLF7DWSOSBBO74LWQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.598560923 (557h3m3.469643968s) [noise=true , address=77.90.31.19:4867 , piece_id=LDAWD2ZFG3BZATDRY7MMNWYIX6TOHQ2OIUPCAWAYN266GNUUKZ5A, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.626925925 (557h3m3.49800897s) [noise=true , address=77.90.31.16:4745 , piece_id=CZBFU33BHMP2BEZIDJKW3J526537HR6HP7OCT2KMAQVY7LGZLPZQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.709061947 (557h3m3.580144992s) [noise=true , address=77.90.31.14:4688 , piece_id=4OULHJ6Z3QJ3BOPDY74TJSKVJG23TOEUCJCUEGY24I5C3MUTAVLQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.788879373 (557h3m3.659962418s) [noise=true , address=77.90.31.21:4895 , piece_id=V4P3KCUDLI543TSROBYUMVTQKKSQDDVSKSJSXKEZTQAHTXUJ2G4Q, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.217616523 (557h3m4.088699569s) [noise=true , address=77.90.31.19:4858 , piece_id=LIYXCPAZOUPKXLKRH3EOWREXUY2LAJXSRZFQILM2QIU7HTW7SY2Q, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.220337625 (557h3m4.09142067s) [noise=true , address=77.90.31.11:4563 , piece_id=DIFJPHSKHROTLKCHP6HQBWT7XGE77HVRZTH2354MV2UEWZLHVGCQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.303958744 (557h3m4.175041789s) [noise=true , address=145.79.227.44:4213 , piece_id=UEAQDBBCP7DIG6ZD3EEM2CKAL4W5QKW23LWFUAMSVUW5ZS2PJDRQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.312794398 (557h3m4.183877444s) [noise=true , address=145.79.227.34:4918 , piece_id=SPDT2KI2774UJMQIUSVTHXCDF7PD2FBG64BXUYW67UMLPE6H7SBQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.349308798 (557h3m4.220391833s) [noise=true , address=145.79.227.40:4058 , piece_id=EQ4LA7T2BCGWY76K3NJI4X77OQJFJED6H4VNGXDQMILU6IVQWOMQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.371518379 (557h3m4.242601424s) [noise=true , address=145.79.227.43:4171 , piece_id=YF5DRDPGRRVYFL5SWMH2PKTSV3C72BIWKUDSAIZTXOIVOTYKRMNA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.376232316 (557h3m4.247315361s) [noise=true , address=145.79.227.30:4768 , piece_id=KK4LEV6RG3IGLYXID5MJF3KMNSJ7WS5JS45PNOOJ26LXLYAUCWOA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.376257844 (557h3m4.247340879s) [seg_pos={0 0}, upload=488362]: Segment reached optimal threshold of 14 pieces. Dec 28 20:32:04.376575799 (557h3m4.247658834s) [noise=true , address=145.79.227.33:4904 , piece_id=QCMN4OSBWIWS4JRLIGQIGDSHZFUF2JWEPVXQ4FQ4G3LG5FBCBAWQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1dC7mC2Cbxmys7ukEkELyF9Us7F1qtptSP6yDJh1EfjVJ6uqsC): protocol: expected piece hash; context canceled; EOF Dec 28 20:32:04.376585127 (557h3m4.247668163s) [noise=true , address=145.79.227.33:4904 , piece_id=QCMN4OSBWIWS4JRLIGQIGDSHZFUF2JWEPVXQ4FQ4G3LG5FBCBAWQ, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.376764359 (557h3m4.247847404s) [noise=true , address=145.79.227.19:4105 , piece_id=RY35FSM6NTWPNF3JE43FZRSNXXYZPUN2B6LRNNUOG6QQR6XSU36Q, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GhDrKMcDEg2RxamqKtiHphpgNdvtZYdHXrjzGsNJFjBFYeEen): protocol: write tcp 188.166.225.170:38838->145.79.227.19:4105: use of closed network connection; write tcp 188.166.225.170:38838->145.79.227.19:4105: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38838->145.79.227.19:4105: use of closed network connection Dec 28 20:32:04.376772865 (557h3m4.2478559s) [noise=true , address=145.79.227.19:4105 , piece_id=RY35FSM6NTWPNF3JE43FZRSNXXYZPUN2B6LRNNUOG6QQR6XSU36Q, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.376926658 (557h3m4.248009703s) [noise=true , address=145.79.227.35:4496 , piece_id=JH4XQBT7XYEJXWBGPPF6SJWCPLV77GBOLGOA53C3WUQMWHZXVZQA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V8XhSWkyGUVYFR8MykifFAP1YAfGCwzqucDcMMfzw3xsVW77Z): protocol: expected piece hash; context canceled; EOF Dec 28 20:32:04.377165332 (557h3m4.248248377s) [noise=true , address=145.79.227.35:4496 , piece_id=JH4XQBT7XYEJXWBGPPF6SJWCPLV77GBOLGOA53C3WUQMWHZXVZQA, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.377212682 (557h3m4.248295848s) [noise=true , address=45.91.116.5:4330 , piece_id=EVHLSVVR5JGY43TLSKVEQLQ6KDBEZENVAFL6YT7ZQ77Y7X2TXIKA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12MK9aVDRwfWPfckFR6RfPUL1Nrm9bwfsgV7TXCKGbtadHebWvW): protocol: read tcp 188.166.225.170:55416->45.91.116.5:4330: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 20:32:04.377289859 (557h3m4.248372924s) [noise=true , address=45.91.116.5:4330 , piece_id=EVHLSVVR5JGY43TLSKVEQLQ6KDBEZENVAFL6YT7ZQ77Y7X2TXIKA, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.377387736 (557h3m4.248470781s) [seg_pos={0 0}, upload=488362]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 20:32:04.377599729 (557h3m4.248682774s) [upload=488362]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 20:32:04.377613535 (557h3m4.24869662s) [upload=488362]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 20:32:04.675090855 (557h3m4.5461739s) [upload=488362]: Done uploading segments... Dec 28 20:32:04.675121233 (557h3m4.546204268s) [upload=488362]: Upload finished. err: <nil> Duration: 15.429280741s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.............X...........X..................X.......X......X] duration: 30.003353652s trace: https://jaeger.dev.storj.io/trace/3cdef822c7925df2 FAIL (2025-12-28 20:48:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8, Piece ID: AS5VOTV4OWXSVCS4DTVX73RFBQVA4KV6HHMYJIT6NOW57IYYAK2A) context deadline exceeded; piecestore: (Node ID: 12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, Piece ID: NLAW5SITQAJWHA4Z665426EH7447KJ34JFXWKEX6NJ2NTNXAE3FA) context deadline exceeded; piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: MHD4KAJFFFC2QPNLKODGVWDP6NZF464JRYQW7YCT5H6Y3EKDAESA) context deadline exceeded ======LOGS====== Dec 28 20:48:09.895562971 (557h19m9.766646006s) [upload=488598]: Starting upload Dec 28 20:48:09.895617085 (557h19m9.76670012s) [upload=488598]: Done starting upload Dec 28 20:48:09.895678551 (557h19m9.766761596s) [upload=488598]: Uploading segments... Dec 28 20:48:09.895915883 (557h19m9.766998918s) [upload=488598]: Got next segment. Inline: false Dec 28 20:48:09.895966861 (557h19m9.767049896s) [upload=488598]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 20:48:09.895974355 (557h19m9.7670574s) [upload=488598]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 20:48:10.149008545 (557h19m10.020091591s) [seg_pos={0 0}, upload=488598]: Begin upload segment... Dec 28 20:48:10.149062078 (557h19m10.020145113s) [seg_pos={0 0}, upload=488598]: Done begin upload segment. Dec 28 20:48:10.149099138 (557h19m10.020182173s) [noise=true , address=145.79.227.15:4876 , piece_id=53ZNYCUA2MT47AIY3VSAPH5TAA7KZIOARUQDUU7YFYP3T4UUUK5A, seg_pos={0 0}, upload=488598]: Uploading piece... Dec 28 20:48:10.149126841 (557h19m10.020209886s) [noise=true , address=45.91.116.11:4120 , piece_id=AS5VOTV4OWXSVCS4DTVX73RFBQVA4KV6HHMYJIT6NOW57IYYAK2A, seg_pos={0 0}, upload=488598]: Uploading piece... Dec 28 20:48:10.149746871 (557h19m10.020829906s) [noise=true , address=145.79.227.37:4565 , piece_id=FK6AS5NIDZBDNVVIEQQ7T6BY4Y6XXV3M4QY75VP33YBYG3GZDJ6Q, seg_pos={0 0}, upload=488598]: Uploading piece... Dec 28 20:48:10.149844076 (557h19m10.020927112s) [noise=true , address=77.90.31.14:4661 , piece_id=NLAW5SITQAJWHA4Z665426EH7447KJ34JFXWKEX6NJ2NTNXAE3FA, seg_pos={0 0}, upload=488598]: Uploading piece... Dec 28 20:48:10.150226184 (557h19m10.021309229s) [noise=true , address=145.79.227.50:4373 , piece_id=VCR2ZTWCLJJC52Q4XRP6EA5B6PRSWWIXSH2YNO7UW6OUHYL4E6EQ, seg_pos={0 0}, upload=488598]: Uploading piece... Dec 28 20:48:10.150263425 (557h19m10.02134646s) [noise=true , address=145.79.227.45:4230 , piece_id=FM4EPI7UA3S43QTOWXIJETED3QNPUAJ7R4W7DTHWWZA665KW6W2Q, seg_pos={0 0}, upload=488598]: Uploading piece... Dec 28 20:48:10.150560120 (557h19m10.021643155s) [noise=true , address=45.91.116.10:4083 , piece_id=XGZF3ADEUF3CBYXA3DXTX66CQWSKEPBZCIA6J6KMPM75SWZS4ZBQ, seg_pos={0 0}, upload=488598]: Uploading piece... Dec 28 20:48:10.150658929 (557h19m10.021741964s) [noise=true , address=145.79.227.48:4302 , piece_id=I6GKPVBJM3OISD4BQHAHWWOJKIM7EYNPUK45NPW4W6AT5ZES6KQA, seg_pos={0 0}, upload=488598]: Uploading piece... Dec 28 20:48:10.150889197 (557h19m10.021972232s) [noise=true , address=45.91.116.12:4225 , piece_id=MHD4KAJFFFC2QPNLKODGVWDP6NZF464JRYQW7YCT5H6Y3EKDAESA, seg_pos={0 0}, upload=488598]: Uploading piece... Dec 28 20:48:10.151305369 (557h19m10.022388414s) [noise=true , address=45.91.116.6:4973 , piece_id=SW43IVEWB3CUPOBAG4LAD2WEOO2A2XDSPOUZ73LQ2LBPI2WFD6MA, seg_pos={0 0}, upload=488598]: Uploading piece... Dec 28 20:48:10.151373860 (557h19m10.022456895s) [noise=true , address=145.79.227.6:4259 , piece_id=VEAAX5MWB7YPEU4NQVBIIXFCMOUGKSILMXC2JD76Y4ZD2K46GCWA, seg_pos={0 0}, upload=488598]: Uploading piece... Dec 28 20:48:10.151790703 (557h19m10.022873738s) [noise=true , address=45.91.116.14:4192 , piece_id=4OQK3XVUATBNVSHPFAI6HW6LO33YRDQ2ZCPNF3OE7UJQFTALGFKQ, seg_pos={0 0}, upload=488598]: Uploading piece... Dec 28 20:48:10.151884101 (557h19m10.022967136s) [noise=true , address=145.79.227.13:4803 , piece_id=XGIKPPCX6BKI5PIV3LXJBGVOGUW5UNEDHL2N3BA4D6OHEWILD7OA, seg_pos={0 0}, upload=488598]: Uploading piece... Dec 28 20:48:10.152162451 (557h19m10.023245487s) [noise=true , address=145.79.227.35:4483 , piece_id=YG43SVXZKWRQ2JKJY7N2EO6O2IKIDXEGACDJFKSMPYFODNO3V4JA, seg_pos={0 0}, upload=488598]: Uploading piece... Dec 28 20:48:10.152418990 (557h19m10.023502025s) [noise=true , address=145.79.227.24:4582 , piece_id=YV25ZFJCQ4YU5OWGBIO5CKG2G6ZNLZ2AZBWN2XJZFHDJ5IFPRPIQ, seg_pos={0 0}, upload=488598]: Uploading piece... Dec 28 20:48:10.153010546 (557h19m10.024093591s) [noise=true , address=145.79.227.26:4654 , piece_id=M5O3MROPVXWTRDYS7RLFHWW63O7N76IHNDRDNVABCLNUEI63M3RA, seg_pos={0 0}, upload=488598]: Uploading piece... Dec 28 20:48:11.857501022 (557h19m11.728584057s) [upload=488598]: Next returned nil segment Dec 28 20:48:11.857516461 (557h19m11.728599506s) [upload=488598]: Waiting for error group managing segments... Dec 28 20:48:12.140525278 (557h19m12.011608323s) [noise=true , address=145.79.227.50:4373 , piece_id=VCR2ZTWCLJJC52Q4XRP6EA5B6PRSWWIXSH2YNO7UW6OUHYL4E6EQ, seg_pos={0 0}, upload=488598]: Done uploading piece. err: <nil> Dec 28 20:48:12.154493266 (557h19m12.025576301s) [noise=true , address=145.79.227.6:4259 , piece_id=VEAAX5MWB7YPEU4NQVBIIXFCMOUGKSILMXC2JD76Y4ZD2K46GCWA, seg_pos={0 0}, upload=488598]: Done uploading piece. err: <nil> Dec 28 20:48:12.158157262 (557h19m12.029240297s) [noise=true , address=145.79.227.48:4302 , piece_id=I6GKPVBJM3OISD4BQHAHWWOJKIM7EYNPUK45NPW4W6AT5ZES6KQA, seg_pos={0 0}, upload=488598]: Done uploading piece. err: <nil> Dec 28 20:48:12.162252751 (557h19m12.033335786s) [noise=true , address=145.79.227.24:4582 , piece_id=YV25ZFJCQ4YU5OWGBIO5CKG2G6ZNLZ2AZBWN2XJZFHDJ5IFPRPIQ, seg_pos={0 0}, upload=488598]: Done uploading piece. err: <nil> Dec 28 20:48:12.167407226 (557h19m12.038490261s) [noise=true , address=145.79.227.37:4565 , piece_id=FK6AS5NIDZBDNVVIEQQ7T6BY4Y6XXV3M4QY75VP33YBYG3GZDJ6Q, seg_pos={0 0}, upload=488598]: Done uploading piece. err: <nil> Dec 28 20:48:12.171496353 (557h19m12.042579388s) [noise=true , address=145.79.227.13:4803 , piece_id=XGIKPPCX6BKI5PIV3LXJBGVOGUW5UNEDHL2N3BA4D6OHEWILD7OA, seg_pos={0 0}, upload=488598]: Done uploading piece. err: <nil> Dec 28 20:48:12.183891686 (557h19m12.054974721s) [noise=true , address=145.79.227.26:4654 , piece_id=M5O3MROPVXWTRDYS7RLFHWW63O7N76IHNDRDNVABCLNUEI63M3RA, seg_pos={0 0}, upload=488598]: Done uploading piece. err: <nil> Dec 28 20:48:12.191525542 (557h19m12.062608577s) [noise=true , address=145.79.227.45:4230 , piece_id=FM4EPI7UA3S43QTOWXIJETED3QNPUAJ7R4W7DTHWWZA665KW6W2Q, seg_pos={0 0}, upload=488598]: Done uploading piece. err: <nil> Dec 28 20:48:12.409151199 (557h19m12.280234244s) [noise=true , address=45.91.116.10:4083 , piece_id=XGZF3ADEUF3CBYXA3DXTX66CQWSKEPBZCIA6J6KMPM75SWZS4ZBQ, seg_pos={0 0}, upload=488598]: Done uploading piece. err: <nil> Dec 28 20:48:12.416435419 (557h19m12.287518464s) [noise=true , address=45.91.116.6:4973 , piece_id=SW43IVEWB3CUPOBAG4LAD2WEOO2A2XDSPOUZ73LQ2LBPI2WFD6MA, seg_pos={0 0}, upload=488598]: Done uploading piece. err: <nil> Dec 28 20:48:12.419178122 (557h19m12.290261157s) [noise=true , address=45.91.116.12:4225 , piece_id=MHD4KAJFFFC2QPNLKODGVWDP6NZF464JRYQW7YCT5H6Y3EKDAESA, seg_pos={0 0}, upload=488598]: Done uploading piece. err: <nil> Dec 28 20:48:12.429211857 (557h19m12.300294892s) [noise=true , address=77.90.31.14:4661 , piece_id=NLAW5SITQAJWHA4Z665426EH7447KJ34JFXWKEX6NJ2NTNXAE3FA, seg_pos={0 0}, upload=488598]: Done uploading piece. err: <nil> Dec 28 20:48:12.432588968 (557h19m12.303672013s) [noise=true , address=45.91.116.11:4120 , piece_id=AS5VOTV4OWXSVCS4DTVX73RFBQVA4KV6HHMYJIT6NOW57IYYAK2A, seg_pos={0 0}, upload=488598]: Done uploading piece. err: <nil> Dec 28 20:48:12.432603265 (557h19m12.3036863s) [seg_pos={0 0}, upload=488598]: Segment reached optimal threshold of 13 pieces. Dec 28 20:48:12.432769441 (557h19m12.303852476s) [noise=true , address=45.91.116.14:4192 , piece_id=4OQK3XVUATBNVSHPFAI6HW6LO33YRDQ2ZCPNF3OE7UJQFTALGFKQ, seg_pos={0 0}, upload=488598]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: expected piece hash; context canceled; EOF Dec 28 20:48:12.432775954 (557h19m12.303858989s) [noise=true , address=45.91.116.14:4192 , piece_id=4OQK3XVUATBNVSHPFAI6HW6LO33YRDQ2ZCPNF3OE7UJQFTALGFKQ, seg_pos={0 0}, upload=488598]: Long tail cancellation due to optimal threshold Dec 28 20:48:12.432894570 (557h19m12.303977605s) [noise=true , address=145.79.227.35:4483 , piece_id=YG43SVXZKWRQ2JKJY7N2EO6O2IKIDXEGACDJFKSMPYFODNO3V4JA, seg_pos={0 0}, upload=488598]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY): protocol: expected piece hash; context canceled; EOF Dec 28 20:48:12.432900952 (557h19m12.303983987s) [noise=true , address=145.79.227.35:4483 , piece_id=YG43SVXZKWRQ2JKJY7N2EO6O2IKIDXEGACDJFKSMPYFODNO3V4JA, seg_pos={0 0}, upload=488598]: Long tail cancellation due to optimal threshold Dec 28 20:48:12.432919938 (557h19m12.304002973s) [noise=true , address=145.79.227.15:4876 , piece_id=53ZNYCUA2MT47AIY3VSAPH5TAA7KZIOARUQDUU7YFYP3T4UUUK5A, seg_pos={0 0}, upload=488598]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP): protocol: expected piece hash; context canceled; EOF Dec 28 20:48:12.432935638 (557h19m12.304018673s) [noise=true , address=145.79.227.15:4876 , piece_id=53ZNYCUA2MT47AIY3VSAPH5TAA7KZIOARUQDUU7YFYP3T4UUUK5A, seg_pos={0 0}, upload=488598]: Long tail cancellation due to optimal threshold Dec 28 20:48:12.432973019 (557h19m12.304056054s) [seg_pos={0 0}, upload=488598]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 20:48:12.433093799 (557h19m12.304176844s) [upload=488598]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 20:48:12.433099110 (557h19m12.304182145s) [upload=488598]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 20:48:12.729042787 (557h19m12.600125822s) [upload=488598]: Done uploading segments... Dec 28 20:48:12.729071291 (557h19m12.600154367s) [upload=488598]: Upload finished. err: <nil> Duration: 30.003390953s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 15.362466639s trace: https://jaeger.dev.storj.io/trace/499b96822b8dda80 FAIL (2025-12-28 20:48:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12S7WBBQ1Nq8jbeGrZiRKTk23CdJC31hPW2vU1aaM5SsHeVyeRH, Piece ID: V4P3KCUDLI543TSROBYUMVTQKKSQDDVSKSJSXKEZTQAHTXUJ2G4Q) context deadline exceeded; piecestore: (Node ID: 1kjCEPtjvcJ3D7BBKRSRirPybBwkRZ1dJwyAQCadzCvaLYvWhz, Piece ID: 4OULHJ6Z3QJ3BOPDY74TJSKVJG23TOEUCJCUEGY24I5C3MUTAVLQ) context deadline exceeded; piecestore: (Node ID: 1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c, Piece ID: LIYXCPAZOUPKXLKRH3EOWREXUY2LAJXSRZFQILM2QIU7HTW7SY2Q) context deadline exceeded ======LOGS====== ue , address=145.79.227.23:4580 , piece_id=OHYI2SFM6TKJE4LHVNGO2PVGZDNFIQDAIQGKG4FOEHTTASBWA5ZQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.496428825 (557h3m1.36751187s) [noise=true , address=77.90.31.19:4867 , piece_id=LDAWD2ZFG3BZATDRY7MMNWYIX6TOHQ2OIUPCAWAYN266GNUUKZ5A, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.496861509 (557h3m1.367944544s) [noise=true , address=145.79.227.19:4105 , piece_id=RY35FSM6NTWPNF3JE43FZRSNXXYZPUN2B6LRNNUOG6QQR6XSU36Q, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.496899933 (557h3m1.367982978s) [noise=true , address=77.90.31.19:4858 , piece_id=LIYXCPAZOUPKXLKRH3EOWREXUY2LAJXSRZFQILM2QIU7HTW7SY2Q, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497229089 (557h3m1.368312134s) [noise=true , address=77.90.31.17:4784 , piece_id=JV2ROD6MN65B6LZKOL3S7ATJWXAY6FBQSZWITA6IHAXELY7R42KA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497326194 (557h3m1.368409239s) [noise=true , address=145.79.227.33:4904 , piece_id=QCMN4OSBWIWS4JRLIGQIGDSHZFUF2JWEPVXQ4FQ4G3LG5FBCBAWQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497664128 (557h3m1.368747173s) [noise=true , address=145.79.227.34:4918 , piece_id=SPDT2KI2774UJMQIUSVTHXCDF7PD2FBG64BXUYW67UMLPE6H7SBQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497731405 (557h3m1.36881445s) [noise=true , address=77.90.31.12:4602 , piece_id=2YPYKKTQ6ID6VTI4AUZQEG2SY7JYPRSAPU4JLF7DWSOSBBO74LWQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.498205939 (557h3m1.369288984s) [noise=true , address=77.90.31.16:4745 , piece_id=CZBFU33BHMP2BEZIDJKW3J526537HR6HP7OCT2KMAQVY7LGZLPZQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.498424295 (557h3m1.36950734s) [noise=true , address=77.90.31.21:4895 , piece_id=V4P3KCUDLI543TSROBYUMVTQKKSQDDVSKSJSXKEZTQAHTXUJ2G4Q, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.498685422 (557h3m1.369768457s) [noise=true , address=145.79.227.35:4496 , piece_id=JH4XQBT7XYEJXWBGPPF6SJWCPLV77GBOLGOA53C3WUQMWHZXVZQA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499066348 (557h3m1.370149393s) [noise=true , address=77.90.31.11:4563 , piece_id=DIFJPHSKHROTLKCHP6HQBWT7XGE77HVRZTH2354MV2UEWZLHVGCQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499185725 (557h3m1.37026877s) [noise=true , address=145.79.227.30:4768 , piece_id=KK4LEV6RG3IGLYXID5MJF3KMNSJ7WS5JS45PNOOJ26LXLYAUCWOA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499468122 (557h3m1.370551157s) [noise=true , address=77.90.31.14:4688 , piece_id=4OULHJ6Z3QJ3BOPDY74TJSKVJG23TOEUCJCUEGY24I5C3MUTAVLQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499901047 (557h3m1.370984102s) [noise=true , address=45.91.116.5:4330 , piece_id=EVHLSVVR5JGY43TLSKVEQLQ6KDBEZENVAFL6YT7ZQ77Y7X2TXIKA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.500546175 (557h3m1.37162921s) [noise=true , address=145.79.227.43:4171 , piece_id=YF5DRDPGRRVYFL5SWMH2PKTSV3C72BIWKUDSAIZTXOIVOTYKRMNA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:02.979572381 (557h3m2.850655436s) [upload=488362]: Next returned nil segment Dec 28 20:32:02.979609411 (557h3m2.850692456s) [upload=488362]: Waiting for error group managing segments... Dec 28 20:32:03.416896733 (557h3m3.287979778s) [noise=true , address=145.79.227.23:4580 , piece_id=OHYI2SFM6TKJE4LHVNGO2PVGZDNFIQDAIQGKG4FOEHTTASBWA5ZQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.552263655 (557h3m3.4233467s) [noise=true , address=77.90.31.17:4784 , piece_id=JV2ROD6MN65B6LZKOL3S7ATJWXAY6FBQSZWITA6IHAXELY7R42KA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.573564856 (557h3m3.444647902s) [noise=true , address=77.90.31.12:4602 , piece_id=2YPYKKTQ6ID6VTI4AUZQEG2SY7JYPRSAPU4JLF7DWSOSBBO74LWQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.598560923 (557h3m3.469643968s) [noise=true , address=77.90.31.19:4867 , piece_id=LDAWD2ZFG3BZATDRY7MMNWYIX6TOHQ2OIUPCAWAYN266GNUUKZ5A, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.626925925 (557h3m3.49800897s) [noise=true , address=77.90.31.16:4745 , piece_id=CZBFU33BHMP2BEZIDJKW3J526537HR6HP7OCT2KMAQVY7LGZLPZQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.709061947 (557h3m3.580144992s) [noise=true , address=77.90.31.14:4688 , piece_id=4OULHJ6Z3QJ3BOPDY74TJSKVJG23TOEUCJCUEGY24I5C3MUTAVLQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.788879373 (557h3m3.659962418s) [noise=true , address=77.90.31.21:4895 , piece_id=V4P3KCUDLI543TSROBYUMVTQKKSQDDVSKSJSXKEZTQAHTXUJ2G4Q, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.217616523 (557h3m4.088699569s) [noise=true , address=77.90.31.19:4858 , piece_id=LIYXCPAZOUPKXLKRH3EOWREXUY2LAJXSRZFQILM2QIU7HTW7SY2Q, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.220337625 (557h3m4.09142067s) [noise=true , address=77.90.31.11:4563 , piece_id=DIFJPHSKHROTLKCHP6HQBWT7XGE77HVRZTH2354MV2UEWZLHVGCQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.303958744 (557h3m4.175041789s) [noise=true , address=145.79.227.44:4213 , piece_id=UEAQDBBCP7DIG6ZD3EEM2CKAL4W5QKW23LWFUAMSVUW5ZS2PJDRQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.312794398 (557h3m4.183877444s) [noise=true , address=145.79.227.34:4918 , piece_id=SPDT2KI2774UJMQIUSVTHXCDF7PD2FBG64BXUYW67UMLPE6H7SBQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.349308798 (557h3m4.220391833s) [noise=true , address=145.79.227.40:4058 , piece_id=EQ4LA7T2BCGWY76K3NJI4X77OQJFJED6H4VNGXDQMILU6IVQWOMQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.371518379 (557h3m4.242601424s) [noise=true , address=145.79.227.43:4171 , piece_id=YF5DRDPGRRVYFL5SWMH2PKTSV3C72BIWKUDSAIZTXOIVOTYKRMNA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.376232316 (557h3m4.247315361s) [noise=true , address=145.79.227.30:4768 , piece_id=KK4LEV6RG3IGLYXID5MJF3KMNSJ7WS5JS45PNOOJ26LXLYAUCWOA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.376257844 (557h3m4.247340879s) [seg_pos={0 0}, upload=488362]: Segment reached optimal threshold of 14 pieces. Dec 28 20:32:04.376575799 (557h3m4.247658834s) [noise=true , address=145.79.227.33:4904 , piece_id=QCMN4OSBWIWS4JRLIGQIGDSHZFUF2JWEPVXQ4FQ4G3LG5FBCBAWQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1dC7mC2Cbxmys7ukEkELyF9Us7F1qtptSP6yDJh1EfjVJ6uqsC): protocol: expected piece hash; context canceled; EOF Dec 28 20:32:04.376585127 (557h3m4.247668163s) [noise=true , address=145.79.227.33:4904 , piece_id=QCMN4OSBWIWS4JRLIGQIGDSHZFUF2JWEPVXQ4FQ4G3LG5FBCBAWQ, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.376764359 (557h3m4.247847404s) [noise=true , address=145.79.227.19:4105 , piece_id=RY35FSM6NTWPNF3JE43FZRSNXXYZPUN2B6LRNNUOG6QQR6XSU36Q, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GhDrKMcDEg2RxamqKtiHphpgNdvtZYdHXrjzGsNJFjBFYeEen): protocol: write tcp 188.166.225.170:38838->145.79.227.19:4105: use of closed network connection; write tcp 188.166.225.170:38838->145.79.227.19:4105: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38838->145.79.227.19:4105: use of closed network connection Dec 28 20:32:04.376772865 (557h3m4.2478559s) [noise=true , address=145.79.227.19:4105 , piece_id=RY35FSM6NTWPNF3JE43FZRSNXXYZPUN2B6LRNNUOG6QQR6XSU36Q, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.376926658 (557h3m4.248009703s) [noise=true , address=145.79.227.35:4496 , piece_id=JH4XQBT7XYEJXWBGPPF6SJWCPLV77GBOLGOA53C3WUQMWHZXVZQA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V8XhSWkyGUVYFR8MykifFAP1YAfGCwzqucDcMMfzw3xsVW77Z): protocol: expected piece hash; context canceled; EOF Dec 28 20:32:04.377165332 (557h3m4.248248377s) [noise=true , address=145.79.227.35:4496 , piece_id=JH4XQBT7XYEJXWBGPPF6SJWCPLV77GBOLGOA53C3WUQMWHZXVZQA, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.377212682 (557h3m4.248295848s) [noise=true , address=45.91.116.5:4330 , piece_id=EVHLSVVR5JGY43TLSKVEQLQ6KDBEZENVAFL6YT7ZQ77Y7X2TXIKA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12MK9aVDRwfWPfckFR6RfPUL1Nrm9bwfsgV7TXCKGbtadHebWvW): protocol: read tcp 188.166.225.170:55416->45.91.116.5:4330: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 20:32:04.377289859 (557h3m4.248372924s) [noise=true , address=45.91.116.5:4330 , piece_id=EVHLSVVR5JGY43TLSKVEQLQ6KDBEZENVAFL6YT7ZQ77Y7X2TXIKA, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.377387736 (557h3m4.248470781s) [seg_pos={0 0}, upload=488362]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 20:32:04.377599729 (557h3m4.248682774s) [upload=488362]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 20:32:04.377613535 (557h3m4.24869662s) [upload=488362]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 20:32:04.675090855 (557h3m4.5461739s) [upload=488362]: Done uploading segments... Dec 28 20:32:04.675121233 (557h3m4.546204268s) [upload=488362]: Upload finished. err: <nil> Duration: 15.362495093s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..............X...........X..................X.......X......] duration: 6.036961874s trace: https://jaeger.dev.storj.io/trace/27073096d2c66e66 FAIL (2025-12-28 20:47:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: VE6AZM6VXOSYVYC2Q3TWSR74QOV43BHZ2CQQO6RSM34RVAD5MKJQ) context deadline exceeded; piecestore: (Node ID: 12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, Piece ID: CNBN2IQKKV3BN5SPAFLZ6ZB4NLL6SVVQDISYAQPDYWWVNKRSU45A) context deadline exceeded; piecestore: (Node ID: 13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, Piece ID: VADBIEKNEJKZ6E22T2JM6O7QLHWHDCCFKZAUC3YKURVQBBAIDWKQ) context deadline exceeded ======LOGS====== g segments... Dec 28 20:42:45.662246218 (557h13m45.533329253s) [upload=488520]: Got next segment. Inline: false Dec 28 20:42:45.662270836 (557h13m45.533353881s) [upload=488520]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 20:42:45.662274963 (557h13m45.533357998s) [upload=488520]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 20:42:45.907695937 (557h13m45.778778992s) [seg_pos={0 0}, upload=488520]: Begin upload segment... Dec 28 20:42:45.907765980 (557h13m45.778849015s) [seg_pos={0 0}, upload=488520]: Done begin upload segment. Dec 28 20:42:45.907818851 (557h13m45.778901886s) [noise=true , address=77.90.31.21:4878 , piece_id=HMF5EGQKGM2YEBY3OGIOXMVDMCUSGZHFMISWLUFETMPX6TXQGNUQ, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.907835683 (557h13m45.778918728s) [noise=true , address=145.79.227.16:4911 , piece_id=VADBIEKNEJKZ6E22T2JM6O7QLHWHDCCFKZAUC3YKURVQBBAIDWKQ, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.908356295 (557h13m45.77943934s) [noise=true , address=77.90.31.14:4661 , piece_id=HLBOPWGXKQBLUCMFUZVHGOR3GRF5PQXTDEQZYXR7DOINQQWOKFPQ, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.908784009 (557h13m45.779867054s) [noise=true , address=145.79.227.35:4482 , piece_id=D4TRJTUIJG763XYZ7EGTICLOMJZWC4XCCAJWQ63FDXBK3VUHQ6BQ, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.909197486 (557h13m45.780280531s) [noise=true , address=145.79.227.45:4231 , piece_id=RA5ZQVPR42YPOQG6S6SG6FFWRIM4STOF2L6IUWSD5YIBDPNVU3SQ, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.909211162 (557h13m45.780294207s) [noise=true , address=145.79.227.15:4875 , piece_id=WP727FDSSU2NI5T2L7VYLJOSMJ4VOQN5PYFJIZRVWZ7JX5N6MKWA, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.909627475 (557h13m45.78071052s) [noise=true , address=145.79.227.46:4602 , piece_id=5NNVLQOPKQSKXMJPA2VM5HFE5RFV5BBQLEFUUI23QVHVGNBQSV6Q, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.909651611 (557h13m45.780734656s) [noise=true , address=145.79.227.37:4565 , piece_id=6GSLWIFYPMOS3DIYSJHBIGD6OMBOFNUMLLST7AUXICEDCZBDJ5LA, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.909972622 (557h13m45.781055657s) [noise=true , address=77.90.31.15:4697 , piece_id=CNBN2IQKKV3BN5SPAFLZ6ZB4NLL6SVVQDISYAQPDYWWVNKRSU45A, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.910169056 (557h13m45.781252101s) [noise=true , address=45.91.116.5:4300 , piece_id=CRR2UNPJ5AL263FKPAG553XNG52M4OLO4KAGGSCIQPASTGJFLC5A, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.910608624 (557h13m45.781691659s) [noise=true , address=77.90.31.12:4590 , piece_id=V7WS7Y6UCOCREAIBLK5V5CAF4UHF6LQ36V5BOPHSZJZBC5MRSM4Q, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.910880450 (557h13m45.781963485s) [noise=true , address=77.90.31.19:4842 , piece_id=DJIYFXSNF3LPOCFXPBBAPHD5MV4X5RKFCC66AEQEVTEA6NZR2MZA, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.911138993 (557h13m45.782222038s) [noise=true , address=145.79.227.19:4077 , piece_id=74G3RRZP4KLQIU6SUZ3OPH7JYMFYLWVIO3YDO7MVMCT7JAOXT7IQ, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.911273580 (557h13m45.782356615s) [noise=true , address=145.79.227.39:4013 , piece_id=JVIOUEJBGKXM33NHZEL4O3TA3UAHA2LCV27AYFTCSEKF7I4OJZXA, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.911547812 (557h13m45.782630857s) [noise=true , address=45.91.116.4:4937 , piece_id=K3WLF64M4S24U4AEPEFBJKIJXQ47AVTM4D3GARCOFHCXSGLILOKQ, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.911971338 (557h13m45.783054383s) [noise=true , address=45.91.116.12:4227 , piece_id=VE6AZM6VXOSYVYC2Q3TWSR74QOV43BHZ2CQQO6RSM34RVAD5MKJQ, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:47.615828034 (557h13m47.486911079s) [upload=488520]: Next returned nil segment Dec 28 20:42:47.615867419 (557h13m47.486950454s) [upload=488520]: Waiting for error group managing segments... Dec 28 20:42:47.902889789 (557h13m47.773972834s) [noise=true , address=145.79.227.37:4565 , piece_id=6GSLWIFYPMOS3DIYSJHBIGD6OMBOFNUMLLST7AUXICEDCZBDJ5LA, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:47.911211395 (557h13m47.78229443s) [noise=true , address=145.79.227.15:4875 , piece_id=WP727FDSSU2NI5T2L7VYLJOSMJ4VOQN5PYFJIZRVWZ7JX5N6MKWA, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:47.915667801 (557h13m47.786750836s) [noise=true , address=145.79.227.39:4013 , piece_id=JVIOUEJBGKXM33NHZEL4O3TA3UAHA2LCV27AYFTCSEKF7I4OJZXA, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:47.915898529 (557h13m47.786981564s) [noise=true , address=145.79.227.16:4911 , piece_id=VADBIEKNEJKZ6E22T2JM6O7QLHWHDCCFKZAUC3YKURVQBBAIDWKQ, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:47.916613942 (557h13m47.787696987s) [noise=true , address=145.79.227.45:4231 , piece_id=RA5ZQVPR42YPOQG6S6SG6FFWRIM4STOF2L6IUWSD5YIBDPNVU3SQ, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:47.919951998 (557h13m47.791035033s) [noise=true , address=145.79.227.19:4077 , piece_id=74G3RRZP4KLQIU6SUZ3OPH7JYMFYLWVIO3YDO7MVMCT7JAOXT7IQ, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:47.926441575 (557h13m47.79752462s) [noise=true , address=145.79.227.46:4602 , piece_id=5NNVLQOPKQSKXMJPA2VM5HFE5RFV5BBQLEFUUI23QVHVGNBQSV6Q, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:48.144918112 (557h13m48.016001157s) [noise=true , address=45.91.116.5:4300 , piece_id=CRR2UNPJ5AL263FKPAG553XNG52M4OLO4KAGGSCIQPASTGJFLC5A, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:48.149275801 (557h13m48.020358836s) [noise=true , address=77.90.31.21:4878 , piece_id=HMF5EGQKGM2YEBY3OGIOXMVDMCUSGZHFMISWLUFETMPX6TXQGNUQ, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:48.205776737 (557h13m48.076859782s) [noise=true , address=77.90.31.14:4661 , piece_id=HLBOPWGXKQBLUCMFUZVHGOR3GRF5PQXTDEQZYXR7DOINQQWOKFPQ, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:48.219929196 (557h13m48.091012241s) [noise=true , address=77.90.31.15:4697 , piece_id=CNBN2IQKKV3BN5SPAFLZ6ZB4NLL6SVVQDISYAQPDYWWVNKRSU45A, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:48.227276968 (557h13m48.098360013s) [noise=true , address=45.91.116.12:4227 , piece_id=VE6AZM6VXOSYVYC2Q3TWSR74QOV43BHZ2CQQO6RSM34RVAD5MKJQ, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:48.235297289 (557h13m48.106380334s) [noise=true , address=45.91.116.4:4937 , piece_id=K3WLF64M4S24U4AEPEFBJKIJXQ47AVTM4D3GARCOFHCXSGLILOKQ, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:48.235330643 (557h13m48.106413678s) [seg_pos={0 0}, upload=488520]: Segment reached optimal threshold of 13 pieces. Dec 28 20:42:48.235563236 (557h13m48.106646271s) [noise=true , address=77.90.31.12:4590 , piece_id=V7WS7Y6UCOCREAIBLK5V5CAF4UHF6LQ36V5BOPHSZJZBC5MRSM4Q, seg_pos={0 0}, upload=488520]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: write tcp 188.166.225.170:59310->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:59310->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59310->77.90.31.12:4590: use of closed network connection Dec 28 20:42:48.235571932 (557h13m48.106654978s) [noise=true , address=77.90.31.12:4590 , piece_id=V7WS7Y6UCOCREAIBLK5V5CAF4UHF6LQ36V5BOPHSZJZBC5MRSM4Q, seg_pos={0 0}, upload=488520]: Long tail cancellation due to optimal threshold Dec 28 20:42:48.235663838 (557h13m48.106746873s) [noise=true , address=145.79.227.35:4482 , piece_id=D4TRJTUIJG763XYZ7EGTICLOMJZWC4XCCAJWQ63FDXBK3VUHQ6BQ, seg_pos={0 0}, upload=488520]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV): protocol: expected piece hash; context canceled; EOF Dec 28 20:42:48.235688946 (557h13m48.106771991s) [noise=true , address=145.79.227.35:4482 , piece_id=D4TRJTUIJG763XYZ7EGTICLOMJZWC4XCCAJWQ63FDXBK3VUHQ6BQ, seg_pos={0 0}, upload=488520]: Long tail cancellation due to optimal threshold Dec 28 20:42:48.235931368 (557h13m48.107014403s) [noise=true , address=77.90.31.19:4842 , piece_id=DJIYFXSNF3LPOCFXPBBAPHD5MV4X5RKFCC66AEQEVTEA6NZR2MZA, seg_pos={0 0}, upload=488520]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: write tcp 188.166.225.170:36760->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:36760->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36760->77.90.31.19:4842: use of closed network connection Dec 28 20:42:48.235940565 (557h13m48.10702361s) [noise=true , address=77.90.31.19:4842 , piece_id=DJIYFXSNF3LPOCFXPBBAPHD5MV4X5RKFCC66AEQEVTEA6NZR2MZA, seg_pos={0 0}, upload=488520]: Long tail cancellation due to optimal threshold Dec 28 20:42:48.235960813 (557h13m48.107043848s) [seg_pos={0 0}, upload=488520]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 20:42:48.236068709 (557h13m48.107151754s) [upload=488520]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 20:42:48.236076794 (557h13m48.107159829s) [upload=488520]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 20:42:48.536044414 (557h13m48.407127459s) [upload=488520]: Done uploading segments... Dec 28 20:42:48.536067258 (557h13m48.407150293s) [upload=488520]: Upload finished. err: <nil> Duration: 6.037019092s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 15.435929472s trace: https://jaeger.dev.storj.io/trace/499b96822b8dda80 FAIL (2025-12-28 20:47:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12S7WBBQ1Nq8jbeGrZiRKTk23CdJC31hPW2vU1aaM5SsHeVyeRH, Piece ID: V4P3KCUDLI543TSROBYUMVTQKKSQDDVSKSJSXKEZTQAHTXUJ2G4Q) context deadline exceeded; piecestore: (Node ID: 1kjCEPtjvcJ3D7BBKRSRirPybBwkRZ1dJwyAQCadzCvaLYvWhz, Piece ID: 4OULHJ6Z3QJ3BOPDY74TJSKVJG23TOEUCJCUEGY24I5C3MUTAVLQ) context deadline exceeded; piecestore: (Node ID: 1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c, Piece ID: LIYXCPAZOUPKXLKRH3EOWREXUY2LAJXSRZFQILM2QIU7HTW7SY2Q) context deadline exceeded ======LOGS====== ue , address=145.79.227.23:4580 , piece_id=OHYI2SFM6TKJE4LHVNGO2PVGZDNFIQDAIQGKG4FOEHTTASBWA5ZQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.496428825 (557h3m1.36751187s) [noise=true , address=77.90.31.19:4867 , piece_id=LDAWD2ZFG3BZATDRY7MMNWYIX6TOHQ2OIUPCAWAYN266GNUUKZ5A, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.496861509 (557h3m1.367944544s) [noise=true , address=145.79.227.19:4105 , piece_id=RY35FSM6NTWPNF3JE43FZRSNXXYZPUN2B6LRNNUOG6QQR6XSU36Q, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.496899933 (557h3m1.367982978s) [noise=true , address=77.90.31.19:4858 , piece_id=LIYXCPAZOUPKXLKRH3EOWREXUY2LAJXSRZFQILM2QIU7HTW7SY2Q, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497229089 (557h3m1.368312134s) [noise=true , address=77.90.31.17:4784 , piece_id=JV2ROD6MN65B6LZKOL3S7ATJWXAY6FBQSZWITA6IHAXELY7R42KA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497326194 (557h3m1.368409239s) [noise=true , address=145.79.227.33:4904 , piece_id=QCMN4OSBWIWS4JRLIGQIGDSHZFUF2JWEPVXQ4FQ4G3LG5FBCBAWQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497664128 (557h3m1.368747173s) [noise=true , address=145.79.227.34:4918 , piece_id=SPDT2KI2774UJMQIUSVTHXCDF7PD2FBG64BXUYW67UMLPE6H7SBQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497731405 (557h3m1.36881445s) [noise=true , address=77.90.31.12:4602 , piece_id=2YPYKKTQ6ID6VTI4AUZQEG2SY7JYPRSAPU4JLF7DWSOSBBO74LWQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.498205939 (557h3m1.369288984s) [noise=true , address=77.90.31.16:4745 , piece_id=CZBFU33BHMP2BEZIDJKW3J526537HR6HP7OCT2KMAQVY7LGZLPZQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.498424295 (557h3m1.36950734s) [noise=true , address=77.90.31.21:4895 , piece_id=V4P3KCUDLI543TSROBYUMVTQKKSQDDVSKSJSXKEZTQAHTXUJ2G4Q, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.498685422 (557h3m1.369768457s) [noise=true , address=145.79.227.35:4496 , piece_id=JH4XQBT7XYEJXWBGPPF6SJWCPLV77GBOLGOA53C3WUQMWHZXVZQA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499066348 (557h3m1.370149393s) [noise=true , address=77.90.31.11:4563 , piece_id=DIFJPHSKHROTLKCHP6HQBWT7XGE77HVRZTH2354MV2UEWZLHVGCQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499185725 (557h3m1.37026877s) [noise=true , address=145.79.227.30:4768 , piece_id=KK4LEV6RG3IGLYXID5MJF3KMNSJ7WS5JS45PNOOJ26LXLYAUCWOA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499468122 (557h3m1.370551157s) [noise=true , address=77.90.31.14:4688 , piece_id=4OULHJ6Z3QJ3BOPDY74TJSKVJG23TOEUCJCUEGY24I5C3MUTAVLQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499901047 (557h3m1.370984102s) [noise=true , address=45.91.116.5:4330 , piece_id=EVHLSVVR5JGY43TLSKVEQLQ6KDBEZENVAFL6YT7ZQ77Y7X2TXIKA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.500546175 (557h3m1.37162921s) [noise=true , address=145.79.227.43:4171 , piece_id=YF5DRDPGRRVYFL5SWMH2PKTSV3C72BIWKUDSAIZTXOIVOTYKRMNA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:02.979572381 (557h3m2.850655436s) [upload=488362]: Next returned nil segment Dec 28 20:32:02.979609411 (557h3m2.850692456s) [upload=488362]: Waiting for error group managing segments... Dec 28 20:32:03.416896733 (557h3m3.287979778s) [noise=true , address=145.79.227.23:4580 , piece_id=OHYI2SFM6TKJE4LHVNGO2PVGZDNFIQDAIQGKG4FOEHTTASBWA5ZQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.552263655 (557h3m3.4233467s) [noise=true , address=77.90.31.17:4784 , piece_id=JV2ROD6MN65B6LZKOL3S7ATJWXAY6FBQSZWITA6IHAXELY7R42KA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.573564856 (557h3m3.444647902s) [noise=true , address=77.90.31.12:4602 , piece_id=2YPYKKTQ6ID6VTI4AUZQEG2SY7JYPRSAPU4JLF7DWSOSBBO74LWQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.598560923 (557h3m3.469643968s) [noise=true , address=77.90.31.19:4867 , piece_id=LDAWD2ZFG3BZATDRY7MMNWYIX6TOHQ2OIUPCAWAYN266GNUUKZ5A, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.626925925 (557h3m3.49800897s) [noise=true , address=77.90.31.16:4745 , piece_id=CZBFU33BHMP2BEZIDJKW3J526537HR6HP7OCT2KMAQVY7LGZLPZQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.709061947 (557h3m3.580144992s) [noise=true , address=77.90.31.14:4688 , piece_id=4OULHJ6Z3QJ3BOPDY74TJSKVJG23TOEUCJCUEGY24I5C3MUTAVLQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.788879373 (557h3m3.659962418s) [noise=true , address=77.90.31.21:4895 , piece_id=V4P3KCUDLI543TSROBYUMVTQKKSQDDVSKSJSXKEZTQAHTXUJ2G4Q, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.217616523 (557h3m4.088699569s) [noise=true , address=77.90.31.19:4858 , piece_id=LIYXCPAZOUPKXLKRH3EOWREXUY2LAJXSRZFQILM2QIU7HTW7SY2Q, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.220337625 (557h3m4.09142067s) [noise=true , address=77.90.31.11:4563 , piece_id=DIFJPHSKHROTLKCHP6HQBWT7XGE77HVRZTH2354MV2UEWZLHVGCQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.303958744 (557h3m4.175041789s) [noise=true , address=145.79.227.44:4213 , piece_id=UEAQDBBCP7DIG6ZD3EEM2CKAL4W5QKW23LWFUAMSVUW5ZS2PJDRQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.312794398 (557h3m4.183877444s) [noise=true , address=145.79.227.34:4918 , piece_id=SPDT2KI2774UJMQIUSVTHXCDF7PD2FBG64BXUYW67UMLPE6H7SBQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.349308798 (557h3m4.220391833s) [noise=true , address=145.79.227.40:4058 , piece_id=EQ4LA7T2BCGWY76K3NJI4X77OQJFJED6H4VNGXDQMILU6IVQWOMQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.371518379 (557h3m4.242601424s) [noise=true , address=145.79.227.43:4171 , piece_id=YF5DRDPGRRVYFL5SWMH2PKTSV3C72BIWKUDSAIZTXOIVOTYKRMNA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.376232316 (557h3m4.247315361s) [noise=true , address=145.79.227.30:4768 , piece_id=KK4LEV6RG3IGLYXID5MJF3KMNSJ7WS5JS45PNOOJ26LXLYAUCWOA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.376257844 (557h3m4.247340879s) [seg_pos={0 0}, upload=488362]: Segment reached optimal threshold of 14 pieces. Dec 28 20:32:04.376575799 (557h3m4.247658834s) [noise=true , address=145.79.227.33:4904 , piece_id=QCMN4OSBWIWS4JRLIGQIGDSHZFUF2JWEPVXQ4FQ4G3LG5FBCBAWQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1dC7mC2Cbxmys7ukEkELyF9Us7F1qtptSP6yDJh1EfjVJ6uqsC): protocol: expected piece hash; context canceled; EOF Dec 28 20:32:04.376585127 (557h3m4.247668163s) [noise=true , address=145.79.227.33:4904 , piece_id=QCMN4OSBWIWS4JRLIGQIGDSHZFUF2JWEPVXQ4FQ4G3LG5FBCBAWQ, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.376764359 (557h3m4.247847404s) [noise=true , address=145.79.227.19:4105 , piece_id=RY35FSM6NTWPNF3JE43FZRSNXXYZPUN2B6LRNNUOG6QQR6XSU36Q, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GhDrKMcDEg2RxamqKtiHphpgNdvtZYdHXrjzGsNJFjBFYeEen): protocol: write tcp 188.166.225.170:38838->145.79.227.19:4105: use of closed network connection; write tcp 188.166.225.170:38838->145.79.227.19:4105: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38838->145.79.227.19:4105: use of closed network connection Dec 28 20:32:04.376772865 (557h3m4.2478559s) [noise=true , address=145.79.227.19:4105 , piece_id=RY35FSM6NTWPNF3JE43FZRSNXXYZPUN2B6LRNNUOG6QQR6XSU36Q, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.376926658 (557h3m4.248009703s) [noise=true , address=145.79.227.35:4496 , piece_id=JH4XQBT7XYEJXWBGPPF6SJWCPLV77GBOLGOA53C3WUQMWHZXVZQA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V8XhSWkyGUVYFR8MykifFAP1YAfGCwzqucDcMMfzw3xsVW77Z): protocol: expected piece hash; context canceled; EOF Dec 28 20:32:04.377165332 (557h3m4.248248377s) [noise=true , address=145.79.227.35:4496 , piece_id=JH4XQBT7XYEJXWBGPPF6SJWCPLV77GBOLGOA53C3WUQMWHZXVZQA, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.377212682 (557h3m4.248295848s) [noise=true , address=45.91.116.5:4330 , piece_id=EVHLSVVR5JGY43TLSKVEQLQ6KDBEZENVAFL6YT7ZQ77Y7X2TXIKA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12MK9aVDRwfWPfckFR6RfPUL1Nrm9bwfsgV7TXCKGbtadHebWvW): protocol: read tcp 188.166.225.170:55416->45.91.116.5:4330: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 20:32:04.377289859 (557h3m4.248372924s) [noise=true , address=45.91.116.5:4330 , piece_id=EVHLSVVR5JGY43TLSKVEQLQ6KDBEZENVAFL6YT7ZQ77Y7X2TXIKA, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.377387736 (557h3m4.248470781s) [seg_pos={0 0}, upload=488362]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 20:32:04.377599729 (557h3m4.248682774s) [upload=488362]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 20:32:04.377613535 (557h3m4.24869662s) [upload=488362]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 20:32:04.675090855 (557h3m4.5461739s) [upload=488362]: Done uploading segments... Dec 28 20:32:04.675121233 (557h3m4.546204268s) [upload=488362]: Upload finished. err: <nil> Duration: 15.435957466s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 15.172699726s trace: https://jaeger.dev.storj.io/trace/499b96822b8dda80 FAIL (2025-12-28 20:46:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12S7WBBQ1Nq8jbeGrZiRKTk23CdJC31hPW2vU1aaM5SsHeVyeRH, Piece ID: V4P3KCUDLI543TSROBYUMVTQKKSQDDVSKSJSXKEZTQAHTXUJ2G4Q) context deadline exceeded; piecestore: (Node ID: 1kjCEPtjvcJ3D7BBKRSRirPybBwkRZ1dJwyAQCadzCvaLYvWhz, Piece ID: 4OULHJ6Z3QJ3BOPDY74TJSKVJG23TOEUCJCUEGY24I5C3MUTAVLQ) context deadline exceeded; piecestore: (Node ID: 1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c, Piece ID: LIYXCPAZOUPKXLKRH3EOWREXUY2LAJXSRZFQILM2QIU7HTW7SY2Q) context deadline exceeded ======LOGS====== ue , address=145.79.227.23:4580 , piece_id=OHYI2SFM6TKJE4LHVNGO2PVGZDNFIQDAIQGKG4FOEHTTASBWA5ZQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.496428825 (557h3m1.36751187s) [noise=true , address=77.90.31.19:4867 , piece_id=LDAWD2ZFG3BZATDRY7MMNWYIX6TOHQ2OIUPCAWAYN266GNUUKZ5A, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.496861509 (557h3m1.367944544s) [noise=true , address=145.79.227.19:4105 , piece_id=RY35FSM6NTWPNF3JE43FZRSNXXYZPUN2B6LRNNUOG6QQR6XSU36Q, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.496899933 (557h3m1.367982978s) [noise=true , address=77.90.31.19:4858 , piece_id=LIYXCPAZOUPKXLKRH3EOWREXUY2LAJXSRZFQILM2QIU7HTW7SY2Q, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497229089 (557h3m1.368312134s) [noise=true , address=77.90.31.17:4784 , piece_id=JV2ROD6MN65B6LZKOL3S7ATJWXAY6FBQSZWITA6IHAXELY7R42KA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497326194 (557h3m1.368409239s) [noise=true , address=145.79.227.33:4904 , piece_id=QCMN4OSBWIWS4JRLIGQIGDSHZFUF2JWEPVXQ4FQ4G3LG5FBCBAWQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497664128 (557h3m1.368747173s) [noise=true , address=145.79.227.34:4918 , piece_id=SPDT2KI2774UJMQIUSVTHXCDF7PD2FBG64BXUYW67UMLPE6H7SBQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497731405 (557h3m1.36881445s) [noise=true , address=77.90.31.12:4602 , piece_id=2YPYKKTQ6ID6VTI4AUZQEG2SY7JYPRSAPU4JLF7DWSOSBBO74LWQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.498205939 (557h3m1.369288984s) [noise=true , address=77.90.31.16:4745 , piece_id=CZBFU33BHMP2BEZIDJKW3J526537HR6HP7OCT2KMAQVY7LGZLPZQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.498424295 (557h3m1.36950734s) [noise=true , address=77.90.31.21:4895 , piece_id=V4P3KCUDLI543TSROBYUMVTQKKSQDDVSKSJSXKEZTQAHTXUJ2G4Q, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.498685422 (557h3m1.369768457s) [noise=true , address=145.79.227.35:4496 , piece_id=JH4XQBT7XYEJXWBGPPF6SJWCPLV77GBOLGOA53C3WUQMWHZXVZQA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499066348 (557h3m1.370149393s) [noise=true , address=77.90.31.11:4563 , piece_id=DIFJPHSKHROTLKCHP6HQBWT7XGE77HVRZTH2354MV2UEWZLHVGCQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499185725 (557h3m1.37026877s) [noise=true , address=145.79.227.30:4768 , piece_id=KK4LEV6RG3IGLYXID5MJF3KMNSJ7WS5JS45PNOOJ26LXLYAUCWOA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499468122 (557h3m1.370551157s) [noise=true , address=77.90.31.14:4688 , piece_id=4OULHJ6Z3QJ3BOPDY74TJSKVJG23TOEUCJCUEGY24I5C3MUTAVLQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499901047 (557h3m1.370984102s) [noise=true , address=45.91.116.5:4330 , piece_id=EVHLSVVR5JGY43TLSKVEQLQ6KDBEZENVAFL6YT7ZQ77Y7X2TXIKA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.500546175 (557h3m1.37162921s) [noise=true , address=145.79.227.43:4171 , piece_id=YF5DRDPGRRVYFL5SWMH2PKTSV3C72BIWKUDSAIZTXOIVOTYKRMNA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:02.979572381 (557h3m2.850655436s) [upload=488362]: Next returned nil segment Dec 28 20:32:02.979609411 (557h3m2.850692456s) [upload=488362]: Waiting for error group managing segments... Dec 28 20:32:03.416896733 (557h3m3.287979778s) [noise=true , address=145.79.227.23:4580 , piece_id=OHYI2SFM6TKJE4LHVNGO2PVGZDNFIQDAIQGKG4FOEHTTASBWA5ZQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.552263655 (557h3m3.4233467s) [noise=true , address=77.90.31.17:4784 , piece_id=JV2ROD6MN65B6LZKOL3S7ATJWXAY6FBQSZWITA6IHAXELY7R42KA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.573564856 (557h3m3.444647902s) [noise=true , address=77.90.31.12:4602 , piece_id=2YPYKKTQ6ID6VTI4AUZQEG2SY7JYPRSAPU4JLF7DWSOSBBO74LWQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.598560923 (557h3m3.469643968s) [noise=true , address=77.90.31.19:4867 , piece_id=LDAWD2ZFG3BZATDRY7MMNWYIX6TOHQ2OIUPCAWAYN266GNUUKZ5A, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.626925925 (557h3m3.49800897s) [noise=true , address=77.90.31.16:4745 , piece_id=CZBFU33BHMP2BEZIDJKW3J526537HR6HP7OCT2KMAQVY7LGZLPZQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.709061947 (557h3m3.580144992s) [noise=true , address=77.90.31.14:4688 , piece_id=4OULHJ6Z3QJ3BOPDY74TJSKVJG23TOEUCJCUEGY24I5C3MUTAVLQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.788879373 (557h3m3.659962418s) [noise=true , address=77.90.31.21:4895 , piece_id=V4P3KCUDLI543TSROBYUMVTQKKSQDDVSKSJSXKEZTQAHTXUJ2G4Q, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.217616523 (557h3m4.088699569s) [noise=true , address=77.90.31.19:4858 , piece_id=LIYXCPAZOUPKXLKRH3EOWREXUY2LAJXSRZFQILM2QIU7HTW7SY2Q, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.220337625 (557h3m4.09142067s) [noise=true , address=77.90.31.11:4563 , piece_id=DIFJPHSKHROTLKCHP6HQBWT7XGE77HVRZTH2354MV2UEWZLHVGCQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.303958744 (557h3m4.175041789s) [noise=true , address=145.79.227.44:4213 , piece_id=UEAQDBBCP7DIG6ZD3EEM2CKAL4W5QKW23LWFUAMSVUW5ZS2PJDRQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.312794398 (557h3m4.183877444s) [noise=true , address=145.79.227.34:4918 , piece_id=SPDT2KI2774UJMQIUSVTHXCDF7PD2FBG64BXUYW67UMLPE6H7SBQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.349308798 (557h3m4.220391833s) [noise=true , address=145.79.227.40:4058 , piece_id=EQ4LA7T2BCGWY76K3NJI4X77OQJFJED6H4VNGXDQMILU6IVQWOMQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.371518379 (557h3m4.242601424s) [noise=true , address=145.79.227.43:4171 , piece_id=YF5DRDPGRRVYFL5SWMH2PKTSV3C72BIWKUDSAIZTXOIVOTYKRMNA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.376232316 (557h3m4.247315361s) [noise=true , address=145.79.227.30:4768 , piece_id=KK4LEV6RG3IGLYXID5MJF3KMNSJ7WS5JS45PNOOJ26LXLYAUCWOA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.376257844 (557h3m4.247340879s) [seg_pos={0 0}, upload=488362]: Segment reached optimal threshold of 14 pieces. Dec 28 20:32:04.376575799 (557h3m4.247658834s) [noise=true , address=145.79.227.33:4904 , piece_id=QCMN4OSBWIWS4JRLIGQIGDSHZFUF2JWEPVXQ4FQ4G3LG5FBCBAWQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1dC7mC2Cbxmys7ukEkELyF9Us7F1qtptSP6yDJh1EfjVJ6uqsC): protocol: expected piece hash; context canceled; EOF Dec 28 20:32:04.376585127 (557h3m4.247668163s) [noise=true , address=145.79.227.33:4904 , piece_id=QCMN4OSBWIWS4JRLIGQIGDSHZFUF2JWEPVXQ4FQ4G3LG5FBCBAWQ, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.376764359 (557h3m4.247847404s) [noise=true , address=145.79.227.19:4105 , piece_id=RY35FSM6NTWPNF3JE43FZRSNXXYZPUN2B6LRNNUOG6QQR6XSU36Q, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GhDrKMcDEg2RxamqKtiHphpgNdvtZYdHXrjzGsNJFjBFYeEen): protocol: write tcp 188.166.225.170:38838->145.79.227.19:4105: use of closed network connection; write tcp 188.166.225.170:38838->145.79.227.19:4105: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38838->145.79.227.19:4105: use of closed network connection Dec 28 20:32:04.376772865 (557h3m4.2478559s) [noise=true , address=145.79.227.19:4105 , piece_id=RY35FSM6NTWPNF3JE43FZRSNXXYZPUN2B6LRNNUOG6QQR6XSU36Q, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.376926658 (557h3m4.248009703s) [noise=true , address=145.79.227.35:4496 , piece_id=JH4XQBT7XYEJXWBGPPF6SJWCPLV77GBOLGOA53C3WUQMWHZXVZQA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V8XhSWkyGUVYFR8MykifFAP1YAfGCwzqucDcMMfzw3xsVW77Z): protocol: expected piece hash; context canceled; EOF Dec 28 20:32:04.377165332 (557h3m4.248248377s) [noise=true , address=145.79.227.35:4496 , piece_id=JH4XQBT7XYEJXWBGPPF6SJWCPLV77GBOLGOA53C3WUQMWHZXVZQA, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.377212682 (557h3m4.248295848s) [noise=true , address=45.91.116.5:4330 , piece_id=EVHLSVVR5JGY43TLSKVEQLQ6KDBEZENVAFL6YT7ZQ77Y7X2TXIKA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12MK9aVDRwfWPfckFR6RfPUL1Nrm9bwfsgV7TXCKGbtadHebWvW): protocol: read tcp 188.166.225.170:55416->45.91.116.5:4330: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 20:32:04.377289859 (557h3m4.248372924s) [noise=true , address=45.91.116.5:4330 , piece_id=EVHLSVVR5JGY43TLSKVEQLQ6KDBEZENVAFL6YT7ZQ77Y7X2TXIKA, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.377387736 (557h3m4.248470781s) [seg_pos={0 0}, upload=488362]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 20:32:04.377599729 (557h3m4.248682774s) [upload=488362]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 20:32:04.377613535 (557h3m4.24869662s) [upload=488362]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 20:32:04.675090855 (557h3m4.5461739s) [upload=488362]: Done uploading segments... Dec 28 20:32:04.675121233 (557h3m4.546204268s) [upload=488362]: Upload finished. err: <nil> Duration: 15.172716798s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...............X...........X..................X.......X.....] duration: 13.079990831s trace: https://jaeger.dev.storj.io/trace/27073096d2c66e66 FAIL (2025-12-28 20:46:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: VE6AZM6VXOSYVYC2Q3TWSR74QOV43BHZ2CQQO6RSM34RVAD5MKJQ) context deadline exceeded; piecestore: (Node ID: 12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, Piece ID: CNBN2IQKKV3BN5SPAFLZ6ZB4NLL6SVVQDISYAQPDYWWVNKRSU45A) context deadline exceeded; piecestore: (Node ID: 13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, Piece ID: VADBIEKNEJKZ6E22T2JM6O7QLHWHDCCFKZAUC3YKURVQBBAIDWKQ) context deadline exceeded ======LOGS====== g segments... Dec 28 20:42:45.662246218 (557h13m45.533329253s) [upload=488520]: Got next segment. Inline: false Dec 28 20:42:45.662270836 (557h13m45.533353881s) [upload=488520]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 20:42:45.662274963 (557h13m45.533357998s) [upload=488520]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 20:42:45.907695937 (557h13m45.778778992s) [seg_pos={0 0}, upload=488520]: Begin upload segment... Dec 28 20:42:45.907765980 (557h13m45.778849015s) [seg_pos={0 0}, upload=488520]: Done begin upload segment. Dec 28 20:42:45.907818851 (557h13m45.778901886s) [noise=true , address=77.90.31.21:4878 , piece_id=HMF5EGQKGM2YEBY3OGIOXMVDMCUSGZHFMISWLUFETMPX6TXQGNUQ, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.907835683 (557h13m45.778918728s) [noise=true , address=145.79.227.16:4911 , piece_id=VADBIEKNEJKZ6E22T2JM6O7QLHWHDCCFKZAUC3YKURVQBBAIDWKQ, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.908356295 (557h13m45.77943934s) [noise=true , address=77.90.31.14:4661 , piece_id=HLBOPWGXKQBLUCMFUZVHGOR3GRF5PQXTDEQZYXR7DOINQQWOKFPQ, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.908784009 (557h13m45.779867054s) [noise=true , address=145.79.227.35:4482 , piece_id=D4TRJTUIJG763XYZ7EGTICLOMJZWC4XCCAJWQ63FDXBK3VUHQ6BQ, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.909197486 (557h13m45.780280531s) [noise=true , address=145.79.227.45:4231 , piece_id=RA5ZQVPR42YPOQG6S6SG6FFWRIM4STOF2L6IUWSD5YIBDPNVU3SQ, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.909211162 (557h13m45.780294207s) [noise=true , address=145.79.227.15:4875 , piece_id=WP727FDSSU2NI5T2L7VYLJOSMJ4VOQN5PYFJIZRVWZ7JX5N6MKWA, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.909627475 (557h13m45.78071052s) [noise=true , address=145.79.227.46:4602 , piece_id=5NNVLQOPKQSKXMJPA2VM5HFE5RFV5BBQLEFUUI23QVHVGNBQSV6Q, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.909651611 (557h13m45.780734656s) [noise=true , address=145.79.227.37:4565 , piece_id=6GSLWIFYPMOS3DIYSJHBIGD6OMBOFNUMLLST7AUXICEDCZBDJ5LA, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.909972622 (557h13m45.781055657s) [noise=true , address=77.90.31.15:4697 , piece_id=CNBN2IQKKV3BN5SPAFLZ6ZB4NLL6SVVQDISYAQPDYWWVNKRSU45A, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.910169056 (557h13m45.781252101s) [noise=true , address=45.91.116.5:4300 , piece_id=CRR2UNPJ5AL263FKPAG553XNG52M4OLO4KAGGSCIQPASTGJFLC5A, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.910608624 (557h13m45.781691659s) [noise=true , address=77.90.31.12:4590 , piece_id=V7WS7Y6UCOCREAIBLK5V5CAF4UHF6LQ36V5BOPHSZJZBC5MRSM4Q, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.910880450 (557h13m45.781963485s) [noise=true , address=77.90.31.19:4842 , piece_id=DJIYFXSNF3LPOCFXPBBAPHD5MV4X5RKFCC66AEQEVTEA6NZR2MZA, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.911138993 (557h13m45.782222038s) [noise=true , address=145.79.227.19:4077 , piece_id=74G3RRZP4KLQIU6SUZ3OPH7JYMFYLWVIO3YDO7MVMCT7JAOXT7IQ, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.911273580 (557h13m45.782356615s) [noise=true , address=145.79.227.39:4013 , piece_id=JVIOUEJBGKXM33NHZEL4O3TA3UAHA2LCV27AYFTCSEKF7I4OJZXA, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.911547812 (557h13m45.782630857s) [noise=true , address=45.91.116.4:4937 , piece_id=K3WLF64M4S24U4AEPEFBJKIJXQ47AVTM4D3GARCOFHCXSGLILOKQ, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.911971338 (557h13m45.783054383s) [noise=true , address=45.91.116.12:4227 , piece_id=VE6AZM6VXOSYVYC2Q3TWSR74QOV43BHZ2CQQO6RSM34RVAD5MKJQ, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:47.615828034 (557h13m47.486911079s) [upload=488520]: Next returned nil segment Dec 28 20:42:47.615867419 (557h13m47.486950454s) [upload=488520]: Waiting for error group managing segments... Dec 28 20:42:47.902889789 (557h13m47.773972834s) [noise=true , address=145.79.227.37:4565 , piece_id=6GSLWIFYPMOS3DIYSJHBIGD6OMBOFNUMLLST7AUXICEDCZBDJ5LA, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:47.911211395 (557h13m47.78229443s) [noise=true , address=145.79.227.15:4875 , piece_id=WP727FDSSU2NI5T2L7VYLJOSMJ4VOQN5PYFJIZRVWZ7JX5N6MKWA, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:47.915667801 (557h13m47.786750836s) [noise=true , address=145.79.227.39:4013 , piece_id=JVIOUEJBGKXM33NHZEL4O3TA3UAHA2LCV27AYFTCSEKF7I4OJZXA, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:47.915898529 (557h13m47.786981564s) [noise=true , address=145.79.227.16:4911 , piece_id=VADBIEKNEJKZ6E22T2JM6O7QLHWHDCCFKZAUC3YKURVQBBAIDWKQ, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:47.916613942 (557h13m47.787696987s) [noise=true , address=145.79.227.45:4231 , piece_id=RA5ZQVPR42YPOQG6S6SG6FFWRIM4STOF2L6IUWSD5YIBDPNVU3SQ, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:47.919951998 (557h13m47.791035033s) [noise=true , address=145.79.227.19:4077 , piece_id=74G3RRZP4KLQIU6SUZ3OPH7JYMFYLWVIO3YDO7MVMCT7JAOXT7IQ, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:47.926441575 (557h13m47.79752462s) [noise=true , address=145.79.227.46:4602 , piece_id=5NNVLQOPKQSKXMJPA2VM5HFE5RFV5BBQLEFUUI23QVHVGNBQSV6Q, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:48.144918112 (557h13m48.016001157s) [noise=true , address=45.91.116.5:4300 , piece_id=CRR2UNPJ5AL263FKPAG553XNG52M4OLO4KAGGSCIQPASTGJFLC5A, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:48.149275801 (557h13m48.020358836s) [noise=true , address=77.90.31.21:4878 , piece_id=HMF5EGQKGM2YEBY3OGIOXMVDMCUSGZHFMISWLUFETMPX6TXQGNUQ, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:48.205776737 (557h13m48.076859782s) [noise=true , address=77.90.31.14:4661 , piece_id=HLBOPWGXKQBLUCMFUZVHGOR3GRF5PQXTDEQZYXR7DOINQQWOKFPQ, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:48.219929196 (557h13m48.091012241s) [noise=true , address=77.90.31.15:4697 , piece_id=CNBN2IQKKV3BN5SPAFLZ6ZB4NLL6SVVQDISYAQPDYWWVNKRSU45A, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:48.227276968 (557h13m48.098360013s) [noise=true , address=45.91.116.12:4227 , piece_id=VE6AZM6VXOSYVYC2Q3TWSR74QOV43BHZ2CQQO6RSM34RVAD5MKJQ, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:48.235297289 (557h13m48.106380334s) [noise=true , address=45.91.116.4:4937 , piece_id=K3WLF64M4S24U4AEPEFBJKIJXQ47AVTM4D3GARCOFHCXSGLILOKQ, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:48.235330643 (557h13m48.106413678s) [seg_pos={0 0}, upload=488520]: Segment reached optimal threshold of 13 pieces. Dec 28 20:42:48.235563236 (557h13m48.106646271s) [noise=true , address=77.90.31.12:4590 , piece_id=V7WS7Y6UCOCREAIBLK5V5CAF4UHF6LQ36V5BOPHSZJZBC5MRSM4Q, seg_pos={0 0}, upload=488520]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: write tcp 188.166.225.170:59310->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:59310->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59310->77.90.31.12:4590: use of closed network connection Dec 28 20:42:48.235571932 (557h13m48.106654978s) [noise=true , address=77.90.31.12:4590 , piece_id=V7WS7Y6UCOCREAIBLK5V5CAF4UHF6LQ36V5BOPHSZJZBC5MRSM4Q, seg_pos={0 0}, upload=488520]: Long tail cancellation due to optimal threshold Dec 28 20:42:48.235663838 (557h13m48.106746873s) [noise=true , address=145.79.227.35:4482 , piece_id=D4TRJTUIJG763XYZ7EGTICLOMJZWC4XCCAJWQ63FDXBK3VUHQ6BQ, seg_pos={0 0}, upload=488520]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV): protocol: expected piece hash; context canceled; EOF Dec 28 20:42:48.235688946 (557h13m48.106771991s) [noise=true , address=145.79.227.35:4482 , piece_id=D4TRJTUIJG763XYZ7EGTICLOMJZWC4XCCAJWQ63FDXBK3VUHQ6BQ, seg_pos={0 0}, upload=488520]: Long tail cancellation due to optimal threshold Dec 28 20:42:48.235931368 (557h13m48.107014403s) [noise=true , address=77.90.31.19:4842 , piece_id=DJIYFXSNF3LPOCFXPBBAPHD5MV4X5RKFCC66AEQEVTEA6NZR2MZA, seg_pos={0 0}, upload=488520]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: write tcp 188.166.225.170:36760->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:36760->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36760->77.90.31.19:4842: use of closed network connection Dec 28 20:42:48.235940565 (557h13m48.10702361s) [noise=true , address=77.90.31.19:4842 , piece_id=DJIYFXSNF3LPOCFXPBBAPHD5MV4X5RKFCC66AEQEVTEA6NZR2MZA, seg_pos={0 0}, upload=488520]: Long tail cancellation due to optimal threshold Dec 28 20:42:48.235960813 (557h13m48.107043848s) [seg_pos={0 0}, upload=488520]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 20:42:48.236068709 (557h13m48.107151754s) [upload=488520]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 20:42:48.236076794 (557h13m48.107159829s) [upload=488520]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 20:42:48.536044414 (557h13m48.407127459s) [upload=488520]: Done uploading segments... Dec 28 20:42:48.536067258 (557h13m48.407150293s) [upload=488520]: Upload finished. err: <nil> Duration: 13.080027009s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 26.705226549s trace: https://jaeger.dev.storj.io/trace/499b96822b8dda80 FAIL (2025-12-28 20:46:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12S7WBBQ1Nq8jbeGrZiRKTk23CdJC31hPW2vU1aaM5SsHeVyeRH, Piece ID: V4P3KCUDLI543TSROBYUMVTQKKSQDDVSKSJSXKEZTQAHTXUJ2G4Q) context deadline exceeded; piecestore: (Node ID: 1kjCEPtjvcJ3D7BBKRSRirPybBwkRZ1dJwyAQCadzCvaLYvWhz, Piece ID: 4OULHJ6Z3QJ3BOPDY74TJSKVJG23TOEUCJCUEGY24I5C3MUTAVLQ) context deadline exceeded; piecestore: (Node ID: 1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c, Piece ID: LIYXCPAZOUPKXLKRH3EOWREXUY2LAJXSRZFQILM2QIU7HTW7SY2Q) context deadline exceeded ======LOGS====== ue , address=145.79.227.23:4580 , piece_id=OHYI2SFM6TKJE4LHVNGO2PVGZDNFIQDAIQGKG4FOEHTTASBWA5ZQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.496428825 (557h3m1.36751187s) [noise=true , address=77.90.31.19:4867 , piece_id=LDAWD2ZFG3BZATDRY7MMNWYIX6TOHQ2OIUPCAWAYN266GNUUKZ5A, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.496861509 (557h3m1.367944544s) [noise=true , address=145.79.227.19:4105 , piece_id=RY35FSM6NTWPNF3JE43FZRSNXXYZPUN2B6LRNNUOG6QQR6XSU36Q, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.496899933 (557h3m1.367982978s) [noise=true , address=77.90.31.19:4858 , piece_id=LIYXCPAZOUPKXLKRH3EOWREXUY2LAJXSRZFQILM2QIU7HTW7SY2Q, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497229089 (557h3m1.368312134s) [noise=true , address=77.90.31.17:4784 , piece_id=JV2ROD6MN65B6LZKOL3S7ATJWXAY6FBQSZWITA6IHAXELY7R42KA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497326194 (557h3m1.368409239s) [noise=true , address=145.79.227.33:4904 , piece_id=QCMN4OSBWIWS4JRLIGQIGDSHZFUF2JWEPVXQ4FQ4G3LG5FBCBAWQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497664128 (557h3m1.368747173s) [noise=true , address=145.79.227.34:4918 , piece_id=SPDT2KI2774UJMQIUSVTHXCDF7PD2FBG64BXUYW67UMLPE6H7SBQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497731405 (557h3m1.36881445s) [noise=true , address=77.90.31.12:4602 , piece_id=2YPYKKTQ6ID6VTI4AUZQEG2SY7JYPRSAPU4JLF7DWSOSBBO74LWQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.498205939 (557h3m1.369288984s) [noise=true , address=77.90.31.16:4745 , piece_id=CZBFU33BHMP2BEZIDJKW3J526537HR6HP7OCT2KMAQVY7LGZLPZQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.498424295 (557h3m1.36950734s) [noise=true , address=77.90.31.21:4895 , piece_id=V4P3KCUDLI543TSROBYUMVTQKKSQDDVSKSJSXKEZTQAHTXUJ2G4Q, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.498685422 (557h3m1.369768457s) [noise=true , address=145.79.227.35:4496 , piece_id=JH4XQBT7XYEJXWBGPPF6SJWCPLV77GBOLGOA53C3WUQMWHZXVZQA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499066348 (557h3m1.370149393s) [noise=true , address=77.90.31.11:4563 , piece_id=DIFJPHSKHROTLKCHP6HQBWT7XGE77HVRZTH2354MV2UEWZLHVGCQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499185725 (557h3m1.37026877s) [noise=true , address=145.79.227.30:4768 , piece_id=KK4LEV6RG3IGLYXID5MJF3KMNSJ7WS5JS45PNOOJ26LXLYAUCWOA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499468122 (557h3m1.370551157s) [noise=true , address=77.90.31.14:4688 , piece_id=4OULHJ6Z3QJ3BOPDY74TJSKVJG23TOEUCJCUEGY24I5C3MUTAVLQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499901047 (557h3m1.370984102s) [noise=true , address=45.91.116.5:4330 , piece_id=EVHLSVVR5JGY43TLSKVEQLQ6KDBEZENVAFL6YT7ZQ77Y7X2TXIKA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.500546175 (557h3m1.37162921s) [noise=true , address=145.79.227.43:4171 , piece_id=YF5DRDPGRRVYFL5SWMH2PKTSV3C72BIWKUDSAIZTXOIVOTYKRMNA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:02.979572381 (557h3m2.850655436s) [upload=488362]: Next returned nil segment Dec 28 20:32:02.979609411 (557h3m2.850692456s) [upload=488362]: Waiting for error group managing segments... Dec 28 20:32:03.416896733 (557h3m3.287979778s) [noise=true , address=145.79.227.23:4580 , piece_id=OHYI2SFM6TKJE4LHVNGO2PVGZDNFIQDAIQGKG4FOEHTTASBWA5ZQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.552263655 (557h3m3.4233467s) [noise=true , address=77.90.31.17:4784 , piece_id=JV2ROD6MN65B6LZKOL3S7ATJWXAY6FBQSZWITA6IHAXELY7R42KA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.573564856 (557h3m3.444647902s) [noise=true , address=77.90.31.12:4602 , piece_id=2YPYKKTQ6ID6VTI4AUZQEG2SY7JYPRSAPU4JLF7DWSOSBBO74LWQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.598560923 (557h3m3.469643968s) [noise=true , address=77.90.31.19:4867 , piece_id=LDAWD2ZFG3BZATDRY7MMNWYIX6TOHQ2OIUPCAWAYN266GNUUKZ5A, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.626925925 (557h3m3.49800897s) [noise=true , address=77.90.31.16:4745 , piece_id=CZBFU33BHMP2BEZIDJKW3J526537HR6HP7OCT2KMAQVY7LGZLPZQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.709061947 (557h3m3.580144992s) [noise=true , address=77.90.31.14:4688 , piece_id=4OULHJ6Z3QJ3BOPDY74TJSKVJG23TOEUCJCUEGY24I5C3MUTAVLQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.788879373 (557h3m3.659962418s) [noise=true , address=77.90.31.21:4895 , piece_id=V4P3KCUDLI543TSROBYUMVTQKKSQDDVSKSJSXKEZTQAHTXUJ2G4Q, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.217616523 (557h3m4.088699569s) [noise=true , address=77.90.31.19:4858 , piece_id=LIYXCPAZOUPKXLKRH3EOWREXUY2LAJXSRZFQILM2QIU7HTW7SY2Q, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.220337625 (557h3m4.09142067s) [noise=true , address=77.90.31.11:4563 , piece_id=DIFJPHSKHROTLKCHP6HQBWT7XGE77HVRZTH2354MV2UEWZLHVGCQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.303958744 (557h3m4.175041789s) [noise=true , address=145.79.227.44:4213 , piece_id=UEAQDBBCP7DIG6ZD3EEM2CKAL4W5QKW23LWFUAMSVUW5ZS2PJDRQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.312794398 (557h3m4.183877444s) [noise=true , address=145.79.227.34:4918 , piece_id=SPDT2KI2774UJMQIUSVTHXCDF7PD2FBG64BXUYW67UMLPE6H7SBQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.349308798 (557h3m4.220391833s) [noise=true , address=145.79.227.40:4058 , piece_id=EQ4LA7T2BCGWY76K3NJI4X77OQJFJED6H4VNGXDQMILU6IVQWOMQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.371518379 (557h3m4.242601424s) [noise=true , address=145.79.227.43:4171 , piece_id=YF5DRDPGRRVYFL5SWMH2PKTSV3C72BIWKUDSAIZTXOIVOTYKRMNA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.376232316 (557h3m4.247315361s) [noise=true , address=145.79.227.30:4768 , piece_id=KK4LEV6RG3IGLYXID5MJF3KMNSJ7WS5JS45PNOOJ26LXLYAUCWOA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.376257844 (557h3m4.247340879s) [seg_pos={0 0}, upload=488362]: Segment reached optimal threshold of 14 pieces. Dec 28 20:32:04.376575799 (557h3m4.247658834s) [noise=true , address=145.79.227.33:4904 , piece_id=QCMN4OSBWIWS4JRLIGQIGDSHZFUF2JWEPVXQ4FQ4G3LG5FBCBAWQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1dC7mC2Cbxmys7ukEkELyF9Us7F1qtptSP6yDJh1EfjVJ6uqsC): protocol: expected piece hash; context canceled; EOF Dec 28 20:32:04.376585127 (557h3m4.247668163s) [noise=true , address=145.79.227.33:4904 , piece_id=QCMN4OSBWIWS4JRLIGQIGDSHZFUF2JWEPVXQ4FQ4G3LG5FBCBAWQ, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.376764359 (557h3m4.247847404s) [noise=true , address=145.79.227.19:4105 , piece_id=RY35FSM6NTWPNF3JE43FZRSNXXYZPUN2B6LRNNUOG6QQR6XSU36Q, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GhDrKMcDEg2RxamqKtiHphpgNdvtZYdHXrjzGsNJFjBFYeEen): protocol: write tcp 188.166.225.170:38838->145.79.227.19:4105: use of closed network connection; write tcp 188.166.225.170:38838->145.79.227.19:4105: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38838->145.79.227.19:4105: use of closed network connection Dec 28 20:32:04.376772865 (557h3m4.2478559s) [noise=true , address=145.79.227.19:4105 , piece_id=RY35FSM6NTWPNF3JE43FZRSNXXYZPUN2B6LRNNUOG6QQR6XSU36Q, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.376926658 (557h3m4.248009703s) [noise=true , address=145.79.227.35:4496 , piece_id=JH4XQBT7XYEJXWBGPPF6SJWCPLV77GBOLGOA53C3WUQMWHZXVZQA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V8XhSWkyGUVYFR8MykifFAP1YAfGCwzqucDcMMfzw3xsVW77Z): protocol: expected piece hash; context canceled; EOF Dec 28 20:32:04.377165332 (557h3m4.248248377s) [noise=true , address=145.79.227.35:4496 , piece_id=JH4XQBT7XYEJXWBGPPF6SJWCPLV77GBOLGOA53C3WUQMWHZXVZQA, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.377212682 (557h3m4.248295848s) [noise=true , address=45.91.116.5:4330 , piece_id=EVHLSVVR5JGY43TLSKVEQLQ6KDBEZENVAFL6YT7ZQ77Y7X2TXIKA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12MK9aVDRwfWPfckFR6RfPUL1Nrm9bwfsgV7TXCKGbtadHebWvW): protocol: read tcp 188.166.225.170:55416->45.91.116.5:4330: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 20:32:04.377289859 (557h3m4.248372924s) [noise=true , address=45.91.116.5:4330 , piece_id=EVHLSVVR5JGY43TLSKVEQLQ6KDBEZENVAFL6YT7ZQ77Y7X2TXIKA, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.377387736 (557h3m4.248470781s) [seg_pos={0 0}, upload=488362]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 20:32:04.377599729 (557h3m4.248682774s) [upload=488362]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 20:32:04.377613535 (557h3m4.24869662s) [upload=488362]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 20:32:04.675090855 (557h3m4.5461739s) [upload=488362]: Done uploading segments... Dec 28 20:32:04.675121233 (557h3m4.546204268s) [upload=488362]: Upload finished. err: <nil> Duration: 26.705254462s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [................X...........X..................X.......X....] duration: 5.927284218s trace: https://jaeger.dev.storj.io/trace/27073096d2c66e66 FAIL (2025-12-28 20:45:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: VE6AZM6VXOSYVYC2Q3TWSR74QOV43BHZ2CQQO6RSM34RVAD5MKJQ) context deadline exceeded; piecestore: (Node ID: 12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, Piece ID: CNBN2IQKKV3BN5SPAFLZ6ZB4NLL6SVVQDISYAQPDYWWVNKRSU45A) context deadline exceeded; piecestore: (Node ID: 13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, Piece ID: VADBIEKNEJKZ6E22T2JM6O7QLHWHDCCFKZAUC3YKURVQBBAIDWKQ) context deadline exceeded ======LOGS====== g segments... Dec 28 20:42:45.662246218 (557h13m45.533329253s) [upload=488520]: Got next segment. Inline: false Dec 28 20:42:45.662270836 (557h13m45.533353881s) [upload=488520]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 20:42:45.662274963 (557h13m45.533357998s) [upload=488520]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 20:42:45.907695937 (557h13m45.778778992s) [seg_pos={0 0}, upload=488520]: Begin upload segment... Dec 28 20:42:45.907765980 (557h13m45.778849015s) [seg_pos={0 0}, upload=488520]: Done begin upload segment. Dec 28 20:42:45.907818851 (557h13m45.778901886s) [noise=true , address=77.90.31.21:4878 , piece_id=HMF5EGQKGM2YEBY3OGIOXMVDMCUSGZHFMISWLUFETMPX6TXQGNUQ, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.907835683 (557h13m45.778918728s) [noise=true , address=145.79.227.16:4911 , piece_id=VADBIEKNEJKZ6E22T2JM6O7QLHWHDCCFKZAUC3YKURVQBBAIDWKQ, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.908356295 (557h13m45.77943934s) [noise=true , address=77.90.31.14:4661 , piece_id=HLBOPWGXKQBLUCMFUZVHGOR3GRF5PQXTDEQZYXR7DOINQQWOKFPQ, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.908784009 (557h13m45.779867054s) [noise=true , address=145.79.227.35:4482 , piece_id=D4TRJTUIJG763XYZ7EGTICLOMJZWC4XCCAJWQ63FDXBK3VUHQ6BQ, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.909197486 (557h13m45.780280531s) [noise=true , address=145.79.227.45:4231 , piece_id=RA5ZQVPR42YPOQG6S6SG6FFWRIM4STOF2L6IUWSD5YIBDPNVU3SQ, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.909211162 (557h13m45.780294207s) [noise=true , address=145.79.227.15:4875 , piece_id=WP727FDSSU2NI5T2L7VYLJOSMJ4VOQN5PYFJIZRVWZ7JX5N6MKWA, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.909627475 (557h13m45.78071052s) [noise=true , address=145.79.227.46:4602 , piece_id=5NNVLQOPKQSKXMJPA2VM5HFE5RFV5BBQLEFUUI23QVHVGNBQSV6Q, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.909651611 (557h13m45.780734656s) [noise=true , address=145.79.227.37:4565 , piece_id=6GSLWIFYPMOS3DIYSJHBIGD6OMBOFNUMLLST7AUXICEDCZBDJ5LA, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.909972622 (557h13m45.781055657s) [noise=true , address=77.90.31.15:4697 , piece_id=CNBN2IQKKV3BN5SPAFLZ6ZB4NLL6SVVQDISYAQPDYWWVNKRSU45A, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.910169056 (557h13m45.781252101s) [noise=true , address=45.91.116.5:4300 , piece_id=CRR2UNPJ5AL263FKPAG553XNG52M4OLO4KAGGSCIQPASTGJFLC5A, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.910608624 (557h13m45.781691659s) [noise=true , address=77.90.31.12:4590 , piece_id=V7WS7Y6UCOCREAIBLK5V5CAF4UHF6LQ36V5BOPHSZJZBC5MRSM4Q, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.910880450 (557h13m45.781963485s) [noise=true , address=77.90.31.19:4842 , piece_id=DJIYFXSNF3LPOCFXPBBAPHD5MV4X5RKFCC66AEQEVTEA6NZR2MZA, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.911138993 (557h13m45.782222038s) [noise=true , address=145.79.227.19:4077 , piece_id=74G3RRZP4KLQIU6SUZ3OPH7JYMFYLWVIO3YDO7MVMCT7JAOXT7IQ, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.911273580 (557h13m45.782356615s) [noise=true , address=145.79.227.39:4013 , piece_id=JVIOUEJBGKXM33NHZEL4O3TA3UAHA2LCV27AYFTCSEKF7I4OJZXA, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.911547812 (557h13m45.782630857s) [noise=true , address=45.91.116.4:4937 , piece_id=K3WLF64M4S24U4AEPEFBJKIJXQ47AVTM4D3GARCOFHCXSGLILOKQ, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.911971338 (557h13m45.783054383s) [noise=true , address=45.91.116.12:4227 , piece_id=VE6AZM6VXOSYVYC2Q3TWSR74QOV43BHZ2CQQO6RSM34RVAD5MKJQ, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:47.615828034 (557h13m47.486911079s) [upload=488520]: Next returned nil segment Dec 28 20:42:47.615867419 (557h13m47.486950454s) [upload=488520]: Waiting for error group managing segments... Dec 28 20:42:47.902889789 (557h13m47.773972834s) [noise=true , address=145.79.227.37:4565 , piece_id=6GSLWIFYPMOS3DIYSJHBIGD6OMBOFNUMLLST7AUXICEDCZBDJ5LA, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:47.911211395 (557h13m47.78229443s) [noise=true , address=145.79.227.15:4875 , piece_id=WP727FDSSU2NI5T2L7VYLJOSMJ4VOQN5PYFJIZRVWZ7JX5N6MKWA, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:47.915667801 (557h13m47.786750836s) [noise=true , address=145.79.227.39:4013 , piece_id=JVIOUEJBGKXM33NHZEL4O3TA3UAHA2LCV27AYFTCSEKF7I4OJZXA, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:47.915898529 (557h13m47.786981564s) [noise=true , address=145.79.227.16:4911 , piece_id=VADBIEKNEJKZ6E22T2JM6O7QLHWHDCCFKZAUC3YKURVQBBAIDWKQ, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:47.916613942 (557h13m47.787696987s) [noise=true , address=145.79.227.45:4231 , piece_id=RA5ZQVPR42YPOQG6S6SG6FFWRIM4STOF2L6IUWSD5YIBDPNVU3SQ, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:47.919951998 (557h13m47.791035033s) [noise=true , address=145.79.227.19:4077 , piece_id=74G3RRZP4KLQIU6SUZ3OPH7JYMFYLWVIO3YDO7MVMCT7JAOXT7IQ, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:47.926441575 (557h13m47.79752462s) [noise=true , address=145.79.227.46:4602 , piece_id=5NNVLQOPKQSKXMJPA2VM5HFE5RFV5BBQLEFUUI23QVHVGNBQSV6Q, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:48.144918112 (557h13m48.016001157s) [noise=true , address=45.91.116.5:4300 , piece_id=CRR2UNPJ5AL263FKPAG553XNG52M4OLO4KAGGSCIQPASTGJFLC5A, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:48.149275801 (557h13m48.020358836s) [noise=true , address=77.90.31.21:4878 , piece_id=HMF5EGQKGM2YEBY3OGIOXMVDMCUSGZHFMISWLUFETMPX6TXQGNUQ, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:48.205776737 (557h13m48.076859782s) [noise=true , address=77.90.31.14:4661 , piece_id=HLBOPWGXKQBLUCMFUZVHGOR3GRF5PQXTDEQZYXR7DOINQQWOKFPQ, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:48.219929196 (557h13m48.091012241s) [noise=true , address=77.90.31.15:4697 , piece_id=CNBN2IQKKV3BN5SPAFLZ6ZB4NLL6SVVQDISYAQPDYWWVNKRSU45A, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:48.227276968 (557h13m48.098360013s) [noise=true , address=45.91.116.12:4227 , piece_id=VE6AZM6VXOSYVYC2Q3TWSR74QOV43BHZ2CQQO6RSM34RVAD5MKJQ, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:48.235297289 (557h13m48.106380334s) [noise=true , address=45.91.116.4:4937 , piece_id=K3WLF64M4S24U4AEPEFBJKIJXQ47AVTM4D3GARCOFHCXSGLILOKQ, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:48.235330643 (557h13m48.106413678s) [seg_pos={0 0}, upload=488520]: Segment reached optimal threshold of 13 pieces. Dec 28 20:42:48.235563236 (557h13m48.106646271s) [noise=true , address=77.90.31.12:4590 , piece_id=V7WS7Y6UCOCREAIBLK5V5CAF4UHF6LQ36V5BOPHSZJZBC5MRSM4Q, seg_pos={0 0}, upload=488520]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: write tcp 188.166.225.170:59310->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:59310->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59310->77.90.31.12:4590: use of closed network connection Dec 28 20:42:48.235571932 (557h13m48.106654978s) [noise=true , address=77.90.31.12:4590 , piece_id=V7WS7Y6UCOCREAIBLK5V5CAF4UHF6LQ36V5BOPHSZJZBC5MRSM4Q, seg_pos={0 0}, upload=488520]: Long tail cancellation due to optimal threshold Dec 28 20:42:48.235663838 (557h13m48.106746873s) [noise=true , address=145.79.227.35:4482 , piece_id=D4TRJTUIJG763XYZ7EGTICLOMJZWC4XCCAJWQ63FDXBK3VUHQ6BQ, seg_pos={0 0}, upload=488520]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV): protocol: expected piece hash; context canceled; EOF Dec 28 20:42:48.235688946 (557h13m48.106771991s) [noise=true , address=145.79.227.35:4482 , piece_id=D4TRJTUIJG763XYZ7EGTICLOMJZWC4XCCAJWQ63FDXBK3VUHQ6BQ, seg_pos={0 0}, upload=488520]: Long tail cancellation due to optimal threshold Dec 28 20:42:48.235931368 (557h13m48.107014403s) [noise=true , address=77.90.31.19:4842 , piece_id=DJIYFXSNF3LPOCFXPBBAPHD5MV4X5RKFCC66AEQEVTEA6NZR2MZA, seg_pos={0 0}, upload=488520]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: write tcp 188.166.225.170:36760->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:36760->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36760->77.90.31.19:4842: use of closed network connection Dec 28 20:42:48.235940565 (557h13m48.10702361s) [noise=true , address=77.90.31.19:4842 , piece_id=DJIYFXSNF3LPOCFXPBBAPHD5MV4X5RKFCC66AEQEVTEA6NZR2MZA, seg_pos={0 0}, upload=488520]: Long tail cancellation due to optimal threshold Dec 28 20:42:48.235960813 (557h13m48.107043848s) [seg_pos={0 0}, upload=488520]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 20:42:48.236068709 (557h13m48.107151754s) [upload=488520]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 20:42:48.236076794 (557h13m48.107159829s) [upload=488520]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 20:42:48.536044414 (557h13m48.407127459s) [upload=488520]: Done uploading segments... Dec 28 20:42:48.536067258 (557h13m48.407150293s) [upload=488520]: Upload finished. err: <nil> Duration: 5.927326538s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 15.387015112s trace: https://jaeger.dev.storj.io/trace/499b96822b8dda80 FAIL (2025-12-28 20:45:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12S7WBBQ1Nq8jbeGrZiRKTk23CdJC31hPW2vU1aaM5SsHeVyeRH, Piece ID: V4P3KCUDLI543TSROBYUMVTQKKSQDDVSKSJSXKEZTQAHTXUJ2G4Q) context deadline exceeded; piecestore: (Node ID: 1kjCEPtjvcJ3D7BBKRSRirPybBwkRZ1dJwyAQCadzCvaLYvWhz, Piece ID: 4OULHJ6Z3QJ3BOPDY74TJSKVJG23TOEUCJCUEGY24I5C3MUTAVLQ) context deadline exceeded; piecestore: (Node ID: 1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c, Piece ID: LIYXCPAZOUPKXLKRH3EOWREXUY2LAJXSRZFQILM2QIU7HTW7SY2Q) context deadline exceeded ======LOGS====== ue , address=145.79.227.23:4580 , piece_id=OHYI2SFM6TKJE4LHVNGO2PVGZDNFIQDAIQGKG4FOEHTTASBWA5ZQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.496428825 (557h3m1.36751187s) [noise=true , address=77.90.31.19:4867 , piece_id=LDAWD2ZFG3BZATDRY7MMNWYIX6TOHQ2OIUPCAWAYN266GNUUKZ5A, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.496861509 (557h3m1.367944544s) [noise=true , address=145.79.227.19:4105 , piece_id=RY35FSM6NTWPNF3JE43FZRSNXXYZPUN2B6LRNNUOG6QQR6XSU36Q, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.496899933 (557h3m1.367982978s) [noise=true , address=77.90.31.19:4858 , piece_id=LIYXCPAZOUPKXLKRH3EOWREXUY2LAJXSRZFQILM2QIU7HTW7SY2Q, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497229089 (557h3m1.368312134s) [noise=true , address=77.90.31.17:4784 , piece_id=JV2ROD6MN65B6LZKOL3S7ATJWXAY6FBQSZWITA6IHAXELY7R42KA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497326194 (557h3m1.368409239s) [noise=true , address=145.79.227.33:4904 , piece_id=QCMN4OSBWIWS4JRLIGQIGDSHZFUF2JWEPVXQ4FQ4G3LG5FBCBAWQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497664128 (557h3m1.368747173s) [noise=true , address=145.79.227.34:4918 , piece_id=SPDT2KI2774UJMQIUSVTHXCDF7PD2FBG64BXUYW67UMLPE6H7SBQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497731405 (557h3m1.36881445s) [noise=true , address=77.90.31.12:4602 , piece_id=2YPYKKTQ6ID6VTI4AUZQEG2SY7JYPRSAPU4JLF7DWSOSBBO74LWQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.498205939 (557h3m1.369288984s) [noise=true , address=77.90.31.16:4745 , piece_id=CZBFU33BHMP2BEZIDJKW3J526537HR6HP7OCT2KMAQVY7LGZLPZQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.498424295 (557h3m1.36950734s) [noise=true , address=77.90.31.21:4895 , piece_id=V4P3KCUDLI543TSROBYUMVTQKKSQDDVSKSJSXKEZTQAHTXUJ2G4Q, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.498685422 (557h3m1.369768457s) [noise=true , address=145.79.227.35:4496 , piece_id=JH4XQBT7XYEJXWBGPPF6SJWCPLV77GBOLGOA53C3WUQMWHZXVZQA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499066348 (557h3m1.370149393s) [noise=true , address=77.90.31.11:4563 , piece_id=DIFJPHSKHROTLKCHP6HQBWT7XGE77HVRZTH2354MV2UEWZLHVGCQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499185725 (557h3m1.37026877s) [noise=true , address=145.79.227.30:4768 , piece_id=KK4LEV6RG3IGLYXID5MJF3KMNSJ7WS5JS45PNOOJ26LXLYAUCWOA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499468122 (557h3m1.370551157s) [noise=true , address=77.90.31.14:4688 , piece_id=4OULHJ6Z3QJ3BOPDY74TJSKVJG23TOEUCJCUEGY24I5C3MUTAVLQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499901047 (557h3m1.370984102s) [noise=true , address=45.91.116.5:4330 , piece_id=EVHLSVVR5JGY43TLSKVEQLQ6KDBEZENVAFL6YT7ZQ77Y7X2TXIKA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.500546175 (557h3m1.37162921s) [noise=true , address=145.79.227.43:4171 , piece_id=YF5DRDPGRRVYFL5SWMH2PKTSV3C72BIWKUDSAIZTXOIVOTYKRMNA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:02.979572381 (557h3m2.850655436s) [upload=488362]: Next returned nil segment Dec 28 20:32:02.979609411 (557h3m2.850692456s) [upload=488362]: Waiting for error group managing segments... Dec 28 20:32:03.416896733 (557h3m3.287979778s) [noise=true , address=145.79.227.23:4580 , piece_id=OHYI2SFM6TKJE4LHVNGO2PVGZDNFIQDAIQGKG4FOEHTTASBWA5ZQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.552263655 (557h3m3.4233467s) [noise=true , address=77.90.31.17:4784 , piece_id=JV2ROD6MN65B6LZKOL3S7ATJWXAY6FBQSZWITA6IHAXELY7R42KA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.573564856 (557h3m3.444647902s) [noise=true , address=77.90.31.12:4602 , piece_id=2YPYKKTQ6ID6VTI4AUZQEG2SY7JYPRSAPU4JLF7DWSOSBBO74LWQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.598560923 (557h3m3.469643968s) [noise=true , address=77.90.31.19:4867 , piece_id=LDAWD2ZFG3BZATDRY7MMNWYIX6TOHQ2OIUPCAWAYN266GNUUKZ5A, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.626925925 (557h3m3.49800897s) [noise=true , address=77.90.31.16:4745 , piece_id=CZBFU33BHMP2BEZIDJKW3J526537HR6HP7OCT2KMAQVY7LGZLPZQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.709061947 (557h3m3.580144992s) [noise=true , address=77.90.31.14:4688 , piece_id=4OULHJ6Z3QJ3BOPDY74TJSKVJG23TOEUCJCUEGY24I5C3MUTAVLQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.788879373 (557h3m3.659962418s) [noise=true , address=77.90.31.21:4895 , piece_id=V4P3KCUDLI543TSROBYUMVTQKKSQDDVSKSJSXKEZTQAHTXUJ2G4Q, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.217616523 (557h3m4.088699569s) [noise=true , address=77.90.31.19:4858 , piece_id=LIYXCPAZOUPKXLKRH3EOWREXUY2LAJXSRZFQILM2QIU7HTW7SY2Q, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.220337625 (557h3m4.09142067s) [noise=true , address=77.90.31.11:4563 , piece_id=DIFJPHSKHROTLKCHP6HQBWT7XGE77HVRZTH2354MV2UEWZLHVGCQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.303958744 (557h3m4.175041789s) [noise=true , address=145.79.227.44:4213 , piece_id=UEAQDBBCP7DIG6ZD3EEM2CKAL4W5QKW23LWFUAMSVUW5ZS2PJDRQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.312794398 (557h3m4.183877444s) [noise=true , address=145.79.227.34:4918 , piece_id=SPDT2KI2774UJMQIUSVTHXCDF7PD2FBG64BXUYW67UMLPE6H7SBQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.349308798 (557h3m4.220391833s) [noise=true , address=145.79.227.40:4058 , piece_id=EQ4LA7T2BCGWY76K3NJI4X77OQJFJED6H4VNGXDQMILU6IVQWOMQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.371518379 (557h3m4.242601424s) [noise=true , address=145.79.227.43:4171 , piece_id=YF5DRDPGRRVYFL5SWMH2PKTSV3C72BIWKUDSAIZTXOIVOTYKRMNA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.376232316 (557h3m4.247315361s) [noise=true , address=145.79.227.30:4768 , piece_id=KK4LEV6RG3IGLYXID5MJF3KMNSJ7WS5JS45PNOOJ26LXLYAUCWOA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.376257844 (557h3m4.247340879s) [seg_pos={0 0}, upload=488362]: Segment reached optimal threshold of 14 pieces. Dec 28 20:32:04.376575799 (557h3m4.247658834s) [noise=true , address=145.79.227.33:4904 , piece_id=QCMN4OSBWIWS4JRLIGQIGDSHZFUF2JWEPVXQ4FQ4G3LG5FBCBAWQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1dC7mC2Cbxmys7ukEkELyF9Us7F1qtptSP6yDJh1EfjVJ6uqsC): protocol: expected piece hash; context canceled; EOF Dec 28 20:32:04.376585127 (557h3m4.247668163s) [noise=true , address=145.79.227.33:4904 , piece_id=QCMN4OSBWIWS4JRLIGQIGDSHZFUF2JWEPVXQ4FQ4G3LG5FBCBAWQ, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.376764359 (557h3m4.247847404s) [noise=true , address=145.79.227.19:4105 , piece_id=RY35FSM6NTWPNF3JE43FZRSNXXYZPUN2B6LRNNUOG6QQR6XSU36Q, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GhDrKMcDEg2RxamqKtiHphpgNdvtZYdHXrjzGsNJFjBFYeEen): protocol: write tcp 188.166.225.170:38838->145.79.227.19:4105: use of closed network connection; write tcp 188.166.225.170:38838->145.79.227.19:4105: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38838->145.79.227.19:4105: use of closed network connection Dec 28 20:32:04.376772865 (557h3m4.2478559s) [noise=true , address=145.79.227.19:4105 , piece_id=RY35FSM6NTWPNF3JE43FZRSNXXYZPUN2B6LRNNUOG6QQR6XSU36Q, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.376926658 (557h3m4.248009703s) [noise=true , address=145.79.227.35:4496 , piece_id=JH4XQBT7XYEJXWBGPPF6SJWCPLV77GBOLGOA53C3WUQMWHZXVZQA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V8XhSWkyGUVYFR8MykifFAP1YAfGCwzqucDcMMfzw3xsVW77Z): protocol: expected piece hash; context canceled; EOF Dec 28 20:32:04.377165332 (557h3m4.248248377s) [noise=true , address=145.79.227.35:4496 , piece_id=JH4XQBT7XYEJXWBGPPF6SJWCPLV77GBOLGOA53C3WUQMWHZXVZQA, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.377212682 (557h3m4.248295848s) [noise=true , address=45.91.116.5:4330 , piece_id=EVHLSVVR5JGY43TLSKVEQLQ6KDBEZENVAFL6YT7ZQ77Y7X2TXIKA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12MK9aVDRwfWPfckFR6RfPUL1Nrm9bwfsgV7TXCKGbtadHebWvW): protocol: read tcp 188.166.225.170:55416->45.91.116.5:4330: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 20:32:04.377289859 (557h3m4.248372924s) [noise=true , address=45.91.116.5:4330 , piece_id=EVHLSVVR5JGY43TLSKVEQLQ6KDBEZENVAFL6YT7ZQ77Y7X2TXIKA, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.377387736 (557h3m4.248470781s) [seg_pos={0 0}, upload=488362]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 20:32:04.377599729 (557h3m4.248682774s) [upload=488362]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 20:32:04.377613535 (557h3m4.24869662s) [upload=488362]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 20:32:04.675090855 (557h3m4.5461739s) [upload=488362]: Done uploading segments... Dec 28 20:32:04.675121233 (557h3m4.546204268s) [upload=488362]: Upload finished. err: <nil> Duration: 15.38704532s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.................X...........X..................X.......X...] duration: 15.882161917s trace: https://jaeger.dev.storj.io/trace/27073096d2c66e66 FAIL (2025-12-28 20:45:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: VE6AZM6VXOSYVYC2Q3TWSR74QOV43BHZ2CQQO6RSM34RVAD5MKJQ) context deadline exceeded; piecestore: (Node ID: 12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, Piece ID: CNBN2IQKKV3BN5SPAFLZ6ZB4NLL6SVVQDISYAQPDYWWVNKRSU45A) context deadline exceeded; piecestore: (Node ID: 13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, Piece ID: VADBIEKNEJKZ6E22T2JM6O7QLHWHDCCFKZAUC3YKURVQBBAIDWKQ) context deadline exceeded ======LOGS====== g segments... Dec 28 20:42:45.662246218 (557h13m45.533329253s) [upload=488520]: Got next segment. Inline: false Dec 28 20:42:45.662270836 (557h13m45.533353881s) [upload=488520]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 20:42:45.662274963 (557h13m45.533357998s) [upload=488520]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 20:42:45.907695937 (557h13m45.778778992s) [seg_pos={0 0}, upload=488520]: Begin upload segment... Dec 28 20:42:45.907765980 (557h13m45.778849015s) [seg_pos={0 0}, upload=488520]: Done begin upload segment. Dec 28 20:42:45.907818851 (557h13m45.778901886s) [noise=true , address=77.90.31.21:4878 , piece_id=HMF5EGQKGM2YEBY3OGIOXMVDMCUSGZHFMISWLUFETMPX6TXQGNUQ, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.907835683 (557h13m45.778918728s) [noise=true , address=145.79.227.16:4911 , piece_id=VADBIEKNEJKZ6E22T2JM6O7QLHWHDCCFKZAUC3YKURVQBBAIDWKQ, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.908356295 (557h13m45.77943934s) [noise=true , address=77.90.31.14:4661 , piece_id=HLBOPWGXKQBLUCMFUZVHGOR3GRF5PQXTDEQZYXR7DOINQQWOKFPQ, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.908784009 (557h13m45.779867054s) [noise=true , address=145.79.227.35:4482 , piece_id=D4TRJTUIJG763XYZ7EGTICLOMJZWC4XCCAJWQ63FDXBK3VUHQ6BQ, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.909197486 (557h13m45.780280531s) [noise=true , address=145.79.227.45:4231 , piece_id=RA5ZQVPR42YPOQG6S6SG6FFWRIM4STOF2L6IUWSD5YIBDPNVU3SQ, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.909211162 (557h13m45.780294207s) [noise=true , address=145.79.227.15:4875 , piece_id=WP727FDSSU2NI5T2L7VYLJOSMJ4VOQN5PYFJIZRVWZ7JX5N6MKWA, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.909627475 (557h13m45.78071052s) [noise=true , address=145.79.227.46:4602 , piece_id=5NNVLQOPKQSKXMJPA2VM5HFE5RFV5BBQLEFUUI23QVHVGNBQSV6Q, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.909651611 (557h13m45.780734656s) [noise=true , address=145.79.227.37:4565 , piece_id=6GSLWIFYPMOS3DIYSJHBIGD6OMBOFNUMLLST7AUXICEDCZBDJ5LA, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.909972622 (557h13m45.781055657s) [noise=true , address=77.90.31.15:4697 , piece_id=CNBN2IQKKV3BN5SPAFLZ6ZB4NLL6SVVQDISYAQPDYWWVNKRSU45A, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.910169056 (557h13m45.781252101s) [noise=true , address=45.91.116.5:4300 , piece_id=CRR2UNPJ5AL263FKPAG553XNG52M4OLO4KAGGSCIQPASTGJFLC5A, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.910608624 (557h13m45.781691659s) [noise=true , address=77.90.31.12:4590 , piece_id=V7WS7Y6UCOCREAIBLK5V5CAF4UHF6LQ36V5BOPHSZJZBC5MRSM4Q, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.910880450 (557h13m45.781963485s) [noise=true , address=77.90.31.19:4842 , piece_id=DJIYFXSNF3LPOCFXPBBAPHD5MV4X5RKFCC66AEQEVTEA6NZR2MZA, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.911138993 (557h13m45.782222038s) [noise=true , address=145.79.227.19:4077 , piece_id=74G3RRZP4KLQIU6SUZ3OPH7JYMFYLWVIO3YDO7MVMCT7JAOXT7IQ, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.911273580 (557h13m45.782356615s) [noise=true , address=145.79.227.39:4013 , piece_id=JVIOUEJBGKXM33NHZEL4O3TA3UAHA2LCV27AYFTCSEKF7I4OJZXA, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.911547812 (557h13m45.782630857s) [noise=true , address=45.91.116.4:4937 , piece_id=K3WLF64M4S24U4AEPEFBJKIJXQ47AVTM4D3GARCOFHCXSGLILOKQ, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.911971338 (557h13m45.783054383s) [noise=true , address=45.91.116.12:4227 , piece_id=VE6AZM6VXOSYVYC2Q3TWSR74QOV43BHZ2CQQO6RSM34RVAD5MKJQ, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:47.615828034 (557h13m47.486911079s) [upload=488520]: Next returned nil segment Dec 28 20:42:47.615867419 (557h13m47.486950454s) [upload=488520]: Waiting for error group managing segments... Dec 28 20:42:47.902889789 (557h13m47.773972834s) [noise=true , address=145.79.227.37:4565 , piece_id=6GSLWIFYPMOS3DIYSJHBIGD6OMBOFNUMLLST7AUXICEDCZBDJ5LA, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:47.911211395 (557h13m47.78229443s) [noise=true , address=145.79.227.15:4875 , piece_id=WP727FDSSU2NI5T2L7VYLJOSMJ4VOQN5PYFJIZRVWZ7JX5N6MKWA, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:47.915667801 (557h13m47.786750836s) [noise=true , address=145.79.227.39:4013 , piece_id=JVIOUEJBGKXM33NHZEL4O3TA3UAHA2LCV27AYFTCSEKF7I4OJZXA, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:47.915898529 (557h13m47.786981564s) [noise=true , address=145.79.227.16:4911 , piece_id=VADBIEKNEJKZ6E22T2JM6O7QLHWHDCCFKZAUC3YKURVQBBAIDWKQ, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:47.916613942 (557h13m47.787696987s) [noise=true , address=145.79.227.45:4231 , piece_id=RA5ZQVPR42YPOQG6S6SG6FFWRIM4STOF2L6IUWSD5YIBDPNVU3SQ, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:47.919951998 (557h13m47.791035033s) [noise=true , address=145.79.227.19:4077 , piece_id=74G3RRZP4KLQIU6SUZ3OPH7JYMFYLWVIO3YDO7MVMCT7JAOXT7IQ, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:47.926441575 (557h13m47.79752462s) [noise=true , address=145.79.227.46:4602 , piece_id=5NNVLQOPKQSKXMJPA2VM5HFE5RFV5BBQLEFUUI23QVHVGNBQSV6Q, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:48.144918112 (557h13m48.016001157s) [noise=true , address=45.91.116.5:4300 , piece_id=CRR2UNPJ5AL263FKPAG553XNG52M4OLO4KAGGSCIQPASTGJFLC5A, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:48.149275801 (557h13m48.020358836s) [noise=true , address=77.90.31.21:4878 , piece_id=HMF5EGQKGM2YEBY3OGIOXMVDMCUSGZHFMISWLUFETMPX6TXQGNUQ, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:48.205776737 (557h13m48.076859782s) [noise=true , address=77.90.31.14:4661 , piece_id=HLBOPWGXKQBLUCMFUZVHGOR3GRF5PQXTDEQZYXR7DOINQQWOKFPQ, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:48.219929196 (557h13m48.091012241s) [noise=true , address=77.90.31.15:4697 , piece_id=CNBN2IQKKV3BN5SPAFLZ6ZB4NLL6SVVQDISYAQPDYWWVNKRSU45A, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:48.227276968 (557h13m48.098360013s) [noise=true , address=45.91.116.12:4227 , piece_id=VE6AZM6VXOSYVYC2Q3TWSR74QOV43BHZ2CQQO6RSM34RVAD5MKJQ, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:48.235297289 (557h13m48.106380334s) [noise=true , address=45.91.116.4:4937 , piece_id=K3WLF64M4S24U4AEPEFBJKIJXQ47AVTM4D3GARCOFHCXSGLILOKQ, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:48.235330643 (557h13m48.106413678s) [seg_pos={0 0}, upload=488520]: Segment reached optimal threshold of 13 pieces. Dec 28 20:42:48.235563236 (557h13m48.106646271s) [noise=true , address=77.90.31.12:4590 , piece_id=V7WS7Y6UCOCREAIBLK5V5CAF4UHF6LQ36V5BOPHSZJZBC5MRSM4Q, seg_pos={0 0}, upload=488520]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: write tcp 188.166.225.170:59310->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:59310->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59310->77.90.31.12:4590: use of closed network connection Dec 28 20:42:48.235571932 (557h13m48.106654978s) [noise=true , address=77.90.31.12:4590 , piece_id=V7WS7Y6UCOCREAIBLK5V5CAF4UHF6LQ36V5BOPHSZJZBC5MRSM4Q, seg_pos={0 0}, upload=488520]: Long tail cancellation due to optimal threshold Dec 28 20:42:48.235663838 (557h13m48.106746873s) [noise=true , address=145.79.227.35:4482 , piece_id=D4TRJTUIJG763XYZ7EGTICLOMJZWC4XCCAJWQ63FDXBK3VUHQ6BQ, seg_pos={0 0}, upload=488520]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV): protocol: expected piece hash; context canceled; EOF Dec 28 20:42:48.235688946 (557h13m48.106771991s) [noise=true , address=145.79.227.35:4482 , piece_id=D4TRJTUIJG763XYZ7EGTICLOMJZWC4XCCAJWQ63FDXBK3VUHQ6BQ, seg_pos={0 0}, upload=488520]: Long tail cancellation due to optimal threshold Dec 28 20:42:48.235931368 (557h13m48.107014403s) [noise=true , address=77.90.31.19:4842 , piece_id=DJIYFXSNF3LPOCFXPBBAPHD5MV4X5RKFCC66AEQEVTEA6NZR2MZA, seg_pos={0 0}, upload=488520]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: write tcp 188.166.225.170:36760->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:36760->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36760->77.90.31.19:4842: use of closed network connection Dec 28 20:42:48.235940565 (557h13m48.10702361s) [noise=true , address=77.90.31.19:4842 , piece_id=DJIYFXSNF3LPOCFXPBBAPHD5MV4X5RKFCC66AEQEVTEA6NZR2MZA, seg_pos={0 0}, upload=488520]: Long tail cancellation due to optimal threshold Dec 28 20:42:48.235960813 (557h13m48.107043848s) [seg_pos={0 0}, upload=488520]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 20:42:48.236068709 (557h13m48.107151754s) [upload=488520]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 20:42:48.236076794 (557h13m48.107159829s) [upload=488520]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 20:42:48.536044414 (557h13m48.407127459s) [upload=488520]: Done uploading segments... Dec 28 20:42:48.536067258 (557h13m48.407150293s) [upload=488520]: Upload finished. err: <nil> Duration: 15.882192206s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 33.701487384s trace: https://jaeger.dev.storj.io/trace/499b96822b8dda80 FAIL (2025-12-28 20:44:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12S7WBBQ1Nq8jbeGrZiRKTk23CdJC31hPW2vU1aaM5SsHeVyeRH, Piece ID: V4P3KCUDLI543TSROBYUMVTQKKSQDDVSKSJSXKEZTQAHTXUJ2G4Q) context deadline exceeded; piecestore: (Node ID: 1kjCEPtjvcJ3D7BBKRSRirPybBwkRZ1dJwyAQCadzCvaLYvWhz, Piece ID: 4OULHJ6Z3QJ3BOPDY74TJSKVJG23TOEUCJCUEGY24I5C3MUTAVLQ) context deadline exceeded; piecestore: (Node ID: 1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c, Piece ID: LIYXCPAZOUPKXLKRH3EOWREXUY2LAJXSRZFQILM2QIU7HTW7SY2Q) context deadline exceeded ======LOGS====== ue , address=145.79.227.23:4580 , piece_id=OHYI2SFM6TKJE4LHVNGO2PVGZDNFIQDAIQGKG4FOEHTTASBWA5ZQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.496428825 (557h3m1.36751187s) [noise=true , address=77.90.31.19:4867 , piece_id=LDAWD2ZFG3BZATDRY7MMNWYIX6TOHQ2OIUPCAWAYN266GNUUKZ5A, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.496861509 (557h3m1.367944544s) [noise=true , address=145.79.227.19:4105 , piece_id=RY35FSM6NTWPNF3JE43FZRSNXXYZPUN2B6LRNNUOG6QQR6XSU36Q, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.496899933 (557h3m1.367982978s) [noise=true , address=77.90.31.19:4858 , piece_id=LIYXCPAZOUPKXLKRH3EOWREXUY2LAJXSRZFQILM2QIU7HTW7SY2Q, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497229089 (557h3m1.368312134s) [noise=true , address=77.90.31.17:4784 , piece_id=JV2ROD6MN65B6LZKOL3S7ATJWXAY6FBQSZWITA6IHAXELY7R42KA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497326194 (557h3m1.368409239s) [noise=true , address=145.79.227.33:4904 , piece_id=QCMN4OSBWIWS4JRLIGQIGDSHZFUF2JWEPVXQ4FQ4G3LG5FBCBAWQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497664128 (557h3m1.368747173s) [noise=true , address=145.79.227.34:4918 , piece_id=SPDT2KI2774UJMQIUSVTHXCDF7PD2FBG64BXUYW67UMLPE6H7SBQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497731405 (557h3m1.36881445s) [noise=true , address=77.90.31.12:4602 , piece_id=2YPYKKTQ6ID6VTI4AUZQEG2SY7JYPRSAPU4JLF7DWSOSBBO74LWQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.498205939 (557h3m1.369288984s) [noise=true , address=77.90.31.16:4745 , piece_id=CZBFU33BHMP2BEZIDJKW3J526537HR6HP7OCT2KMAQVY7LGZLPZQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.498424295 (557h3m1.36950734s) [noise=true , address=77.90.31.21:4895 , piece_id=V4P3KCUDLI543TSROBYUMVTQKKSQDDVSKSJSXKEZTQAHTXUJ2G4Q, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.498685422 (557h3m1.369768457s) [noise=true , address=145.79.227.35:4496 , piece_id=JH4XQBT7XYEJXWBGPPF6SJWCPLV77GBOLGOA53C3WUQMWHZXVZQA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499066348 (557h3m1.370149393s) [noise=true , address=77.90.31.11:4563 , piece_id=DIFJPHSKHROTLKCHP6HQBWT7XGE77HVRZTH2354MV2UEWZLHVGCQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499185725 (557h3m1.37026877s) [noise=true , address=145.79.227.30:4768 , piece_id=KK4LEV6RG3IGLYXID5MJF3KMNSJ7WS5JS45PNOOJ26LXLYAUCWOA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499468122 (557h3m1.370551157s) [noise=true , address=77.90.31.14:4688 , piece_id=4OULHJ6Z3QJ3BOPDY74TJSKVJG23TOEUCJCUEGY24I5C3MUTAVLQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499901047 (557h3m1.370984102s) [noise=true , address=45.91.116.5:4330 , piece_id=EVHLSVVR5JGY43TLSKVEQLQ6KDBEZENVAFL6YT7ZQ77Y7X2TXIKA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.500546175 (557h3m1.37162921s) [noise=true , address=145.79.227.43:4171 , piece_id=YF5DRDPGRRVYFL5SWMH2PKTSV3C72BIWKUDSAIZTXOIVOTYKRMNA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:02.979572381 (557h3m2.850655436s) [upload=488362]: Next returned nil segment Dec 28 20:32:02.979609411 (557h3m2.850692456s) [upload=488362]: Waiting for error group managing segments... Dec 28 20:32:03.416896733 (557h3m3.287979778s) [noise=true , address=145.79.227.23:4580 , piece_id=OHYI2SFM6TKJE4LHVNGO2PVGZDNFIQDAIQGKG4FOEHTTASBWA5ZQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.552263655 (557h3m3.4233467s) [noise=true , address=77.90.31.17:4784 , piece_id=JV2ROD6MN65B6LZKOL3S7ATJWXAY6FBQSZWITA6IHAXELY7R42KA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.573564856 (557h3m3.444647902s) [noise=true , address=77.90.31.12:4602 , piece_id=2YPYKKTQ6ID6VTI4AUZQEG2SY7JYPRSAPU4JLF7DWSOSBBO74LWQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.598560923 (557h3m3.469643968s) [noise=true , address=77.90.31.19:4867 , piece_id=LDAWD2ZFG3BZATDRY7MMNWYIX6TOHQ2OIUPCAWAYN266GNUUKZ5A, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.626925925 (557h3m3.49800897s) [noise=true , address=77.90.31.16:4745 , piece_id=CZBFU33BHMP2BEZIDJKW3J526537HR6HP7OCT2KMAQVY7LGZLPZQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.709061947 (557h3m3.580144992s) [noise=true , address=77.90.31.14:4688 , piece_id=4OULHJ6Z3QJ3BOPDY74TJSKVJG23TOEUCJCUEGY24I5C3MUTAVLQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.788879373 (557h3m3.659962418s) [noise=true , address=77.90.31.21:4895 , piece_id=V4P3KCUDLI543TSROBYUMVTQKKSQDDVSKSJSXKEZTQAHTXUJ2G4Q, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.217616523 (557h3m4.088699569s) [noise=true , address=77.90.31.19:4858 , piece_id=LIYXCPAZOUPKXLKRH3EOWREXUY2LAJXSRZFQILM2QIU7HTW7SY2Q, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.220337625 (557h3m4.09142067s) [noise=true , address=77.90.31.11:4563 , piece_id=DIFJPHSKHROTLKCHP6HQBWT7XGE77HVRZTH2354MV2UEWZLHVGCQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.303958744 (557h3m4.175041789s) [noise=true , address=145.79.227.44:4213 , piece_id=UEAQDBBCP7DIG6ZD3EEM2CKAL4W5QKW23LWFUAMSVUW5ZS2PJDRQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.312794398 (557h3m4.183877444s) [noise=true , address=145.79.227.34:4918 , piece_id=SPDT2KI2774UJMQIUSVTHXCDF7PD2FBG64BXUYW67UMLPE6H7SBQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.349308798 (557h3m4.220391833s) [noise=true , address=145.79.227.40:4058 , piece_id=EQ4LA7T2BCGWY76K3NJI4X77OQJFJED6H4VNGXDQMILU6IVQWOMQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.371518379 (557h3m4.242601424s) [noise=true , address=145.79.227.43:4171 , piece_id=YF5DRDPGRRVYFL5SWMH2PKTSV3C72BIWKUDSAIZTXOIVOTYKRMNA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.376232316 (557h3m4.247315361s) [noise=true , address=145.79.227.30:4768 , piece_id=KK4LEV6RG3IGLYXID5MJF3KMNSJ7WS5JS45PNOOJ26LXLYAUCWOA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.376257844 (557h3m4.247340879s) [seg_pos={0 0}, upload=488362]: Segment reached optimal threshold of 14 pieces. Dec 28 20:32:04.376575799 (557h3m4.247658834s) [noise=true , address=145.79.227.33:4904 , piece_id=QCMN4OSBWIWS4JRLIGQIGDSHZFUF2JWEPVXQ4FQ4G3LG5FBCBAWQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1dC7mC2Cbxmys7ukEkELyF9Us7F1qtptSP6yDJh1EfjVJ6uqsC): protocol: expected piece hash; context canceled; EOF Dec 28 20:32:04.376585127 (557h3m4.247668163s) [noise=true , address=145.79.227.33:4904 , piece_id=QCMN4OSBWIWS4JRLIGQIGDSHZFUF2JWEPVXQ4FQ4G3LG5FBCBAWQ, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.376764359 (557h3m4.247847404s) [noise=true , address=145.79.227.19:4105 , piece_id=RY35FSM6NTWPNF3JE43FZRSNXXYZPUN2B6LRNNUOG6QQR6XSU36Q, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GhDrKMcDEg2RxamqKtiHphpgNdvtZYdHXrjzGsNJFjBFYeEen): protocol: write tcp 188.166.225.170:38838->145.79.227.19:4105: use of closed network connection; write tcp 188.166.225.170:38838->145.79.227.19:4105: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38838->145.79.227.19:4105: use of closed network connection Dec 28 20:32:04.376772865 (557h3m4.2478559s) [noise=true , address=145.79.227.19:4105 , piece_id=RY35FSM6NTWPNF3JE43FZRSNXXYZPUN2B6LRNNUOG6QQR6XSU36Q, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.376926658 (557h3m4.248009703s) [noise=true , address=145.79.227.35:4496 , piece_id=JH4XQBT7XYEJXWBGPPF6SJWCPLV77GBOLGOA53C3WUQMWHZXVZQA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V8XhSWkyGUVYFR8MykifFAP1YAfGCwzqucDcMMfzw3xsVW77Z): protocol: expected piece hash; context canceled; EOF Dec 28 20:32:04.377165332 (557h3m4.248248377s) [noise=true , address=145.79.227.35:4496 , piece_id=JH4XQBT7XYEJXWBGPPF6SJWCPLV77GBOLGOA53C3WUQMWHZXVZQA, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.377212682 (557h3m4.248295848s) [noise=true , address=45.91.116.5:4330 , piece_id=EVHLSVVR5JGY43TLSKVEQLQ6KDBEZENVAFL6YT7ZQ77Y7X2TXIKA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12MK9aVDRwfWPfckFR6RfPUL1Nrm9bwfsgV7TXCKGbtadHebWvW): protocol: read tcp 188.166.225.170:55416->45.91.116.5:4330: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 20:32:04.377289859 (557h3m4.248372924s) [noise=true , address=45.91.116.5:4330 , piece_id=EVHLSVVR5JGY43TLSKVEQLQ6KDBEZENVAFL6YT7ZQ77Y7X2TXIKA, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.377387736 (557h3m4.248470781s) [seg_pos={0 0}, upload=488362]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 20:32:04.377599729 (557h3m4.248682774s) [upload=488362]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 20:32:04.377613535 (557h3m4.24869662s) [upload=488362]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 20:32:04.675090855 (557h3m4.5461739s) [upload=488362]: Done uploading segments... Dec 28 20:32:04.675121233 (557h3m4.546204268s) [upload=488362]: Upload finished. err: <nil> Duration: 33.701521469s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..................X...........X..................X.......X..] duration: 13.497161673s trace: https://jaeger.dev.storj.io/trace/27073096d2c66e66 FAIL (2025-12-28 20:44:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: VE6AZM6VXOSYVYC2Q3TWSR74QOV43BHZ2CQQO6RSM34RVAD5MKJQ) context deadline exceeded; piecestore: (Node ID: 12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, Piece ID: CNBN2IQKKV3BN5SPAFLZ6ZB4NLL6SVVQDISYAQPDYWWVNKRSU45A) context deadline exceeded; piecestore: (Node ID: 13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, Piece ID: VADBIEKNEJKZ6E22T2JM6O7QLHWHDCCFKZAUC3YKURVQBBAIDWKQ) context deadline exceeded ======LOGS====== g segments... Dec 28 20:42:45.662246218 (557h13m45.533329253s) [upload=488520]: Got next segment. Inline: false Dec 28 20:42:45.662270836 (557h13m45.533353881s) [upload=488520]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 20:42:45.662274963 (557h13m45.533357998s) [upload=488520]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 20:42:45.907695937 (557h13m45.778778992s) [seg_pos={0 0}, upload=488520]: Begin upload segment... Dec 28 20:42:45.907765980 (557h13m45.778849015s) [seg_pos={0 0}, upload=488520]: Done begin upload segment. Dec 28 20:42:45.907818851 (557h13m45.778901886s) [noise=true , address=77.90.31.21:4878 , piece_id=HMF5EGQKGM2YEBY3OGIOXMVDMCUSGZHFMISWLUFETMPX6TXQGNUQ, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.907835683 (557h13m45.778918728s) [noise=true , address=145.79.227.16:4911 , piece_id=VADBIEKNEJKZ6E22T2JM6O7QLHWHDCCFKZAUC3YKURVQBBAIDWKQ, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.908356295 (557h13m45.77943934s) [noise=true , address=77.90.31.14:4661 , piece_id=HLBOPWGXKQBLUCMFUZVHGOR3GRF5PQXTDEQZYXR7DOINQQWOKFPQ, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.908784009 (557h13m45.779867054s) [noise=true , address=145.79.227.35:4482 , piece_id=D4TRJTUIJG763XYZ7EGTICLOMJZWC4XCCAJWQ63FDXBK3VUHQ6BQ, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.909197486 (557h13m45.780280531s) [noise=true , address=145.79.227.45:4231 , piece_id=RA5ZQVPR42YPOQG6S6SG6FFWRIM4STOF2L6IUWSD5YIBDPNVU3SQ, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.909211162 (557h13m45.780294207s) [noise=true , address=145.79.227.15:4875 , piece_id=WP727FDSSU2NI5T2L7VYLJOSMJ4VOQN5PYFJIZRVWZ7JX5N6MKWA, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.909627475 (557h13m45.78071052s) [noise=true , address=145.79.227.46:4602 , piece_id=5NNVLQOPKQSKXMJPA2VM5HFE5RFV5BBQLEFUUI23QVHVGNBQSV6Q, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.909651611 (557h13m45.780734656s) [noise=true , address=145.79.227.37:4565 , piece_id=6GSLWIFYPMOS3DIYSJHBIGD6OMBOFNUMLLST7AUXICEDCZBDJ5LA, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.909972622 (557h13m45.781055657s) [noise=true , address=77.90.31.15:4697 , piece_id=CNBN2IQKKV3BN5SPAFLZ6ZB4NLL6SVVQDISYAQPDYWWVNKRSU45A, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.910169056 (557h13m45.781252101s) [noise=true , address=45.91.116.5:4300 , piece_id=CRR2UNPJ5AL263FKPAG553XNG52M4OLO4KAGGSCIQPASTGJFLC5A, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.910608624 (557h13m45.781691659s) [noise=true , address=77.90.31.12:4590 , piece_id=V7WS7Y6UCOCREAIBLK5V5CAF4UHF6LQ36V5BOPHSZJZBC5MRSM4Q, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.910880450 (557h13m45.781963485s) [noise=true , address=77.90.31.19:4842 , piece_id=DJIYFXSNF3LPOCFXPBBAPHD5MV4X5RKFCC66AEQEVTEA6NZR2MZA, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.911138993 (557h13m45.782222038s) [noise=true , address=145.79.227.19:4077 , piece_id=74G3RRZP4KLQIU6SUZ3OPH7JYMFYLWVIO3YDO7MVMCT7JAOXT7IQ, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.911273580 (557h13m45.782356615s) [noise=true , address=145.79.227.39:4013 , piece_id=JVIOUEJBGKXM33NHZEL4O3TA3UAHA2LCV27AYFTCSEKF7I4OJZXA, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.911547812 (557h13m45.782630857s) [noise=true , address=45.91.116.4:4937 , piece_id=K3WLF64M4S24U4AEPEFBJKIJXQ47AVTM4D3GARCOFHCXSGLILOKQ, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.911971338 (557h13m45.783054383s) [noise=true , address=45.91.116.12:4227 , piece_id=VE6AZM6VXOSYVYC2Q3TWSR74QOV43BHZ2CQQO6RSM34RVAD5MKJQ, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:47.615828034 (557h13m47.486911079s) [upload=488520]: Next returned nil segment Dec 28 20:42:47.615867419 (557h13m47.486950454s) [upload=488520]: Waiting for error group managing segments... Dec 28 20:42:47.902889789 (557h13m47.773972834s) [noise=true , address=145.79.227.37:4565 , piece_id=6GSLWIFYPMOS3DIYSJHBIGD6OMBOFNUMLLST7AUXICEDCZBDJ5LA, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:47.911211395 (557h13m47.78229443s) [noise=true , address=145.79.227.15:4875 , piece_id=WP727FDSSU2NI5T2L7VYLJOSMJ4VOQN5PYFJIZRVWZ7JX5N6MKWA, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:47.915667801 (557h13m47.786750836s) [noise=true , address=145.79.227.39:4013 , piece_id=JVIOUEJBGKXM33NHZEL4O3TA3UAHA2LCV27AYFTCSEKF7I4OJZXA, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:47.915898529 (557h13m47.786981564s) [noise=true , address=145.79.227.16:4911 , piece_id=VADBIEKNEJKZ6E22T2JM6O7QLHWHDCCFKZAUC3YKURVQBBAIDWKQ, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:47.916613942 (557h13m47.787696987s) [noise=true , address=145.79.227.45:4231 , piece_id=RA5ZQVPR42YPOQG6S6SG6FFWRIM4STOF2L6IUWSD5YIBDPNVU3SQ, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:47.919951998 (557h13m47.791035033s) [noise=true , address=145.79.227.19:4077 , piece_id=74G3RRZP4KLQIU6SUZ3OPH7JYMFYLWVIO3YDO7MVMCT7JAOXT7IQ, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:47.926441575 (557h13m47.79752462s) [noise=true , address=145.79.227.46:4602 , piece_id=5NNVLQOPKQSKXMJPA2VM5HFE5RFV5BBQLEFUUI23QVHVGNBQSV6Q, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:48.144918112 (557h13m48.016001157s) [noise=true , address=45.91.116.5:4300 , piece_id=CRR2UNPJ5AL263FKPAG553XNG52M4OLO4KAGGSCIQPASTGJFLC5A, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:48.149275801 (557h13m48.020358836s) [noise=true , address=77.90.31.21:4878 , piece_id=HMF5EGQKGM2YEBY3OGIOXMVDMCUSGZHFMISWLUFETMPX6TXQGNUQ, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:48.205776737 (557h13m48.076859782s) [noise=true , address=77.90.31.14:4661 , piece_id=HLBOPWGXKQBLUCMFUZVHGOR3GRF5PQXTDEQZYXR7DOINQQWOKFPQ, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:48.219929196 (557h13m48.091012241s) [noise=true , address=77.90.31.15:4697 , piece_id=CNBN2IQKKV3BN5SPAFLZ6ZB4NLL6SVVQDISYAQPDYWWVNKRSU45A, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:48.227276968 (557h13m48.098360013s) [noise=true , address=45.91.116.12:4227 , piece_id=VE6AZM6VXOSYVYC2Q3TWSR74QOV43BHZ2CQQO6RSM34RVAD5MKJQ, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:48.235297289 (557h13m48.106380334s) [noise=true , address=45.91.116.4:4937 , piece_id=K3WLF64M4S24U4AEPEFBJKIJXQ47AVTM4D3GARCOFHCXSGLILOKQ, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:48.235330643 (557h13m48.106413678s) [seg_pos={0 0}, upload=488520]: Segment reached optimal threshold of 13 pieces. Dec 28 20:42:48.235563236 (557h13m48.106646271s) [noise=true , address=77.90.31.12:4590 , piece_id=V7WS7Y6UCOCREAIBLK5V5CAF4UHF6LQ36V5BOPHSZJZBC5MRSM4Q, seg_pos={0 0}, upload=488520]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: write tcp 188.166.225.170:59310->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:59310->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59310->77.90.31.12:4590: use of closed network connection Dec 28 20:42:48.235571932 (557h13m48.106654978s) [noise=true , address=77.90.31.12:4590 , piece_id=V7WS7Y6UCOCREAIBLK5V5CAF4UHF6LQ36V5BOPHSZJZBC5MRSM4Q, seg_pos={0 0}, upload=488520]: Long tail cancellation due to optimal threshold Dec 28 20:42:48.235663838 (557h13m48.106746873s) [noise=true , address=145.79.227.35:4482 , piece_id=D4TRJTUIJG763XYZ7EGTICLOMJZWC4XCCAJWQ63FDXBK3VUHQ6BQ, seg_pos={0 0}, upload=488520]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV): protocol: expected piece hash; context canceled; EOF Dec 28 20:42:48.235688946 (557h13m48.106771991s) [noise=true , address=145.79.227.35:4482 , piece_id=D4TRJTUIJG763XYZ7EGTICLOMJZWC4XCCAJWQ63FDXBK3VUHQ6BQ, seg_pos={0 0}, upload=488520]: Long tail cancellation due to optimal threshold Dec 28 20:42:48.235931368 (557h13m48.107014403s) [noise=true , address=77.90.31.19:4842 , piece_id=DJIYFXSNF3LPOCFXPBBAPHD5MV4X5RKFCC66AEQEVTEA6NZR2MZA, seg_pos={0 0}, upload=488520]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: write tcp 188.166.225.170:36760->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:36760->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36760->77.90.31.19:4842: use of closed network connection Dec 28 20:42:48.235940565 (557h13m48.10702361s) [noise=true , address=77.90.31.19:4842 , piece_id=DJIYFXSNF3LPOCFXPBBAPHD5MV4X5RKFCC66AEQEVTEA6NZR2MZA, seg_pos={0 0}, upload=488520]: Long tail cancellation due to optimal threshold Dec 28 20:42:48.235960813 (557h13m48.107043848s) [seg_pos={0 0}, upload=488520]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 20:42:48.236068709 (557h13m48.107151754s) [upload=488520]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 20:42:48.236076794 (557h13m48.107159829s) [upload=488520]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 20:42:48.536044414 (557h13m48.407127459s) [upload=488520]: Done uploading segments... Dec 28 20:42:48.536067258 (557h13m48.407150293s) [upload=488520]: Upload finished. err: <nil> Duration: 13.497188145s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 14.874941604s trace: https://jaeger.dev.storj.io/trace/499b96822b8dda80 FAIL (2025-12-28 20:44:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12S7WBBQ1Nq8jbeGrZiRKTk23CdJC31hPW2vU1aaM5SsHeVyeRH, Piece ID: V4P3KCUDLI543TSROBYUMVTQKKSQDDVSKSJSXKEZTQAHTXUJ2G4Q) context deadline exceeded; piecestore: (Node ID: 1kjCEPtjvcJ3D7BBKRSRirPybBwkRZ1dJwyAQCadzCvaLYvWhz, Piece ID: 4OULHJ6Z3QJ3BOPDY74TJSKVJG23TOEUCJCUEGY24I5C3MUTAVLQ) context deadline exceeded; piecestore: (Node ID: 1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c, Piece ID: LIYXCPAZOUPKXLKRH3EOWREXUY2LAJXSRZFQILM2QIU7HTW7SY2Q) context deadline exceeded ======LOGS====== ue , address=145.79.227.23:4580 , piece_id=OHYI2SFM6TKJE4LHVNGO2PVGZDNFIQDAIQGKG4FOEHTTASBWA5ZQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.496428825 (557h3m1.36751187s) [noise=true , address=77.90.31.19:4867 , piece_id=LDAWD2ZFG3BZATDRY7MMNWYIX6TOHQ2OIUPCAWAYN266GNUUKZ5A, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.496861509 (557h3m1.367944544s) [noise=true , address=145.79.227.19:4105 , piece_id=RY35FSM6NTWPNF3JE43FZRSNXXYZPUN2B6LRNNUOG6QQR6XSU36Q, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.496899933 (557h3m1.367982978s) [noise=true , address=77.90.31.19:4858 , piece_id=LIYXCPAZOUPKXLKRH3EOWREXUY2LAJXSRZFQILM2QIU7HTW7SY2Q, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497229089 (557h3m1.368312134s) [noise=true , address=77.90.31.17:4784 , piece_id=JV2ROD6MN65B6LZKOL3S7ATJWXAY6FBQSZWITA6IHAXELY7R42KA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497326194 (557h3m1.368409239s) [noise=true , address=145.79.227.33:4904 , piece_id=QCMN4OSBWIWS4JRLIGQIGDSHZFUF2JWEPVXQ4FQ4G3LG5FBCBAWQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497664128 (557h3m1.368747173s) [noise=true , address=145.79.227.34:4918 , piece_id=SPDT2KI2774UJMQIUSVTHXCDF7PD2FBG64BXUYW67UMLPE6H7SBQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497731405 (557h3m1.36881445s) [noise=true , address=77.90.31.12:4602 , piece_id=2YPYKKTQ6ID6VTI4AUZQEG2SY7JYPRSAPU4JLF7DWSOSBBO74LWQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.498205939 (557h3m1.369288984s) [noise=true , address=77.90.31.16:4745 , piece_id=CZBFU33BHMP2BEZIDJKW3J526537HR6HP7OCT2KMAQVY7LGZLPZQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.498424295 (557h3m1.36950734s) [noise=true , address=77.90.31.21:4895 , piece_id=V4P3KCUDLI543TSROBYUMVTQKKSQDDVSKSJSXKEZTQAHTXUJ2G4Q, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.498685422 (557h3m1.369768457s) [noise=true , address=145.79.227.35:4496 , piece_id=JH4XQBT7XYEJXWBGPPF6SJWCPLV77GBOLGOA53C3WUQMWHZXVZQA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499066348 (557h3m1.370149393s) [noise=true , address=77.90.31.11:4563 , piece_id=DIFJPHSKHROTLKCHP6HQBWT7XGE77HVRZTH2354MV2UEWZLHVGCQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499185725 (557h3m1.37026877s) [noise=true , address=145.79.227.30:4768 , piece_id=KK4LEV6RG3IGLYXID5MJF3KMNSJ7WS5JS45PNOOJ26LXLYAUCWOA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499468122 (557h3m1.370551157s) [noise=true , address=77.90.31.14:4688 , piece_id=4OULHJ6Z3QJ3BOPDY74TJSKVJG23TOEUCJCUEGY24I5C3MUTAVLQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499901047 (557h3m1.370984102s) [noise=true , address=45.91.116.5:4330 , piece_id=EVHLSVVR5JGY43TLSKVEQLQ6KDBEZENVAFL6YT7ZQ77Y7X2TXIKA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.500546175 (557h3m1.37162921s) [noise=true , address=145.79.227.43:4171 , piece_id=YF5DRDPGRRVYFL5SWMH2PKTSV3C72BIWKUDSAIZTXOIVOTYKRMNA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:02.979572381 (557h3m2.850655436s) [upload=488362]: Next returned nil segment Dec 28 20:32:02.979609411 (557h3m2.850692456s) [upload=488362]: Waiting for error group managing segments... Dec 28 20:32:03.416896733 (557h3m3.287979778s) [noise=true , address=145.79.227.23:4580 , piece_id=OHYI2SFM6TKJE4LHVNGO2PVGZDNFIQDAIQGKG4FOEHTTASBWA5ZQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.552263655 (557h3m3.4233467s) [noise=true , address=77.90.31.17:4784 , piece_id=JV2ROD6MN65B6LZKOL3S7ATJWXAY6FBQSZWITA6IHAXELY7R42KA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.573564856 (557h3m3.444647902s) [noise=true , address=77.90.31.12:4602 , piece_id=2YPYKKTQ6ID6VTI4AUZQEG2SY7JYPRSAPU4JLF7DWSOSBBO74LWQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.598560923 (557h3m3.469643968s) [noise=true , address=77.90.31.19:4867 , piece_id=LDAWD2ZFG3BZATDRY7MMNWYIX6TOHQ2OIUPCAWAYN266GNUUKZ5A, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.626925925 (557h3m3.49800897s) [noise=true , address=77.90.31.16:4745 , piece_id=CZBFU33BHMP2BEZIDJKW3J526537HR6HP7OCT2KMAQVY7LGZLPZQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.709061947 (557h3m3.580144992s) [noise=true , address=77.90.31.14:4688 , piece_id=4OULHJ6Z3QJ3BOPDY74TJSKVJG23TOEUCJCUEGY24I5C3MUTAVLQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.788879373 (557h3m3.659962418s) [noise=true , address=77.90.31.21:4895 , piece_id=V4P3KCUDLI543TSROBYUMVTQKKSQDDVSKSJSXKEZTQAHTXUJ2G4Q, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.217616523 (557h3m4.088699569s) [noise=true , address=77.90.31.19:4858 , piece_id=LIYXCPAZOUPKXLKRH3EOWREXUY2LAJXSRZFQILM2QIU7HTW7SY2Q, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.220337625 (557h3m4.09142067s) [noise=true , address=77.90.31.11:4563 , piece_id=DIFJPHSKHROTLKCHP6HQBWT7XGE77HVRZTH2354MV2UEWZLHVGCQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.303958744 (557h3m4.175041789s) [noise=true , address=145.79.227.44:4213 , piece_id=UEAQDBBCP7DIG6ZD3EEM2CKAL4W5QKW23LWFUAMSVUW5ZS2PJDRQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.312794398 (557h3m4.183877444s) [noise=true , address=145.79.227.34:4918 , piece_id=SPDT2KI2774UJMQIUSVTHXCDF7PD2FBG64BXUYW67UMLPE6H7SBQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.349308798 (557h3m4.220391833s) [noise=true , address=145.79.227.40:4058 , piece_id=EQ4LA7T2BCGWY76K3NJI4X77OQJFJED6H4VNGXDQMILU6IVQWOMQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.371518379 (557h3m4.242601424s) [noise=true , address=145.79.227.43:4171 , piece_id=YF5DRDPGRRVYFL5SWMH2PKTSV3C72BIWKUDSAIZTXOIVOTYKRMNA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.376232316 (557h3m4.247315361s) [noise=true , address=145.79.227.30:4768 , piece_id=KK4LEV6RG3IGLYXID5MJF3KMNSJ7WS5JS45PNOOJ26LXLYAUCWOA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.376257844 (557h3m4.247340879s) [seg_pos={0 0}, upload=488362]: Segment reached optimal threshold of 14 pieces. Dec 28 20:32:04.376575799 (557h3m4.247658834s) [noise=true , address=145.79.227.33:4904 , piece_id=QCMN4OSBWIWS4JRLIGQIGDSHZFUF2JWEPVXQ4FQ4G3LG5FBCBAWQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1dC7mC2Cbxmys7ukEkELyF9Us7F1qtptSP6yDJh1EfjVJ6uqsC): protocol: expected piece hash; context canceled; EOF Dec 28 20:32:04.376585127 (557h3m4.247668163s) [noise=true , address=145.79.227.33:4904 , piece_id=QCMN4OSBWIWS4JRLIGQIGDSHZFUF2JWEPVXQ4FQ4G3LG5FBCBAWQ, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.376764359 (557h3m4.247847404s) [noise=true , address=145.79.227.19:4105 , piece_id=RY35FSM6NTWPNF3JE43FZRSNXXYZPUN2B6LRNNUOG6QQR6XSU36Q, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GhDrKMcDEg2RxamqKtiHphpgNdvtZYdHXrjzGsNJFjBFYeEen): protocol: write tcp 188.166.225.170:38838->145.79.227.19:4105: use of closed network connection; write tcp 188.166.225.170:38838->145.79.227.19:4105: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38838->145.79.227.19:4105: use of closed network connection Dec 28 20:32:04.376772865 (557h3m4.2478559s) [noise=true , address=145.79.227.19:4105 , piece_id=RY35FSM6NTWPNF3JE43FZRSNXXYZPUN2B6LRNNUOG6QQR6XSU36Q, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.376926658 (557h3m4.248009703s) [noise=true , address=145.79.227.35:4496 , piece_id=JH4XQBT7XYEJXWBGPPF6SJWCPLV77GBOLGOA53C3WUQMWHZXVZQA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V8XhSWkyGUVYFR8MykifFAP1YAfGCwzqucDcMMfzw3xsVW77Z): protocol: expected piece hash; context canceled; EOF Dec 28 20:32:04.377165332 (557h3m4.248248377s) [noise=true , address=145.79.227.35:4496 , piece_id=JH4XQBT7XYEJXWBGPPF6SJWCPLV77GBOLGOA53C3WUQMWHZXVZQA, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.377212682 (557h3m4.248295848s) [noise=true , address=45.91.116.5:4330 , piece_id=EVHLSVVR5JGY43TLSKVEQLQ6KDBEZENVAFL6YT7ZQ77Y7X2TXIKA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12MK9aVDRwfWPfckFR6RfPUL1Nrm9bwfsgV7TXCKGbtadHebWvW): protocol: read tcp 188.166.225.170:55416->45.91.116.5:4330: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 20:32:04.377289859 (557h3m4.248372924s) [noise=true , address=45.91.116.5:4330 , piece_id=EVHLSVVR5JGY43TLSKVEQLQ6KDBEZENVAFL6YT7ZQ77Y7X2TXIKA, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.377387736 (557h3m4.248470781s) [seg_pos={0 0}, upload=488362]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 20:32:04.377599729 (557h3m4.248682774s) [upload=488362]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 20:32:04.377613535 (557h3m4.24869662s) [upload=488362]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 20:32:04.675090855 (557h3m4.5461739s) [upload=488362]: Done uploading segments... Dec 28 20:32:04.675121233 (557h3m4.546204268s) [upload=488362]: Upload finished. err: <nil> Duration: 14.874960389s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...................X...........X..................X.......X.] duration: 6.16747382s trace: https://jaeger.dev.storj.io/trace/27073096d2c66e66 FAIL (2025-12-28 20:43:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: VE6AZM6VXOSYVYC2Q3TWSR74QOV43BHZ2CQQO6RSM34RVAD5MKJQ) context deadline exceeded; piecestore: (Node ID: 12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, Piece ID: CNBN2IQKKV3BN5SPAFLZ6ZB4NLL6SVVQDISYAQPDYWWVNKRSU45A) context deadline exceeded; piecestore: (Node ID: 13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, Piece ID: VADBIEKNEJKZ6E22T2JM6O7QLHWHDCCFKZAUC3YKURVQBBAIDWKQ) context deadline exceeded ======LOGS====== g segments... Dec 28 20:42:45.662246218 (557h13m45.533329253s) [upload=488520]: Got next segment. Inline: false Dec 28 20:42:45.662270836 (557h13m45.533353881s) [upload=488520]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 20:42:45.662274963 (557h13m45.533357998s) [upload=488520]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 20:42:45.907695937 (557h13m45.778778992s) [seg_pos={0 0}, upload=488520]: Begin upload segment... Dec 28 20:42:45.907765980 (557h13m45.778849015s) [seg_pos={0 0}, upload=488520]: Done begin upload segment. Dec 28 20:42:45.907818851 (557h13m45.778901886s) [noise=true , address=77.90.31.21:4878 , piece_id=HMF5EGQKGM2YEBY3OGIOXMVDMCUSGZHFMISWLUFETMPX6TXQGNUQ, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.907835683 (557h13m45.778918728s) [noise=true , address=145.79.227.16:4911 , piece_id=VADBIEKNEJKZ6E22T2JM6O7QLHWHDCCFKZAUC3YKURVQBBAIDWKQ, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.908356295 (557h13m45.77943934s) [noise=true , address=77.90.31.14:4661 , piece_id=HLBOPWGXKQBLUCMFUZVHGOR3GRF5PQXTDEQZYXR7DOINQQWOKFPQ, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.908784009 (557h13m45.779867054s) [noise=true , address=145.79.227.35:4482 , piece_id=D4TRJTUIJG763XYZ7EGTICLOMJZWC4XCCAJWQ63FDXBK3VUHQ6BQ, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.909197486 (557h13m45.780280531s) [noise=true , address=145.79.227.45:4231 , piece_id=RA5ZQVPR42YPOQG6S6SG6FFWRIM4STOF2L6IUWSD5YIBDPNVU3SQ, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.909211162 (557h13m45.780294207s) [noise=true , address=145.79.227.15:4875 , piece_id=WP727FDSSU2NI5T2L7VYLJOSMJ4VOQN5PYFJIZRVWZ7JX5N6MKWA, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.909627475 (557h13m45.78071052s) [noise=true , address=145.79.227.46:4602 , piece_id=5NNVLQOPKQSKXMJPA2VM5HFE5RFV5BBQLEFUUI23QVHVGNBQSV6Q, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.909651611 (557h13m45.780734656s) [noise=true , address=145.79.227.37:4565 , piece_id=6GSLWIFYPMOS3DIYSJHBIGD6OMBOFNUMLLST7AUXICEDCZBDJ5LA, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.909972622 (557h13m45.781055657s) [noise=true , address=77.90.31.15:4697 , piece_id=CNBN2IQKKV3BN5SPAFLZ6ZB4NLL6SVVQDISYAQPDYWWVNKRSU45A, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.910169056 (557h13m45.781252101s) [noise=true , address=45.91.116.5:4300 , piece_id=CRR2UNPJ5AL263FKPAG553XNG52M4OLO4KAGGSCIQPASTGJFLC5A, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.910608624 (557h13m45.781691659s) [noise=true , address=77.90.31.12:4590 , piece_id=V7WS7Y6UCOCREAIBLK5V5CAF4UHF6LQ36V5BOPHSZJZBC5MRSM4Q, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.910880450 (557h13m45.781963485s) [noise=true , address=77.90.31.19:4842 , piece_id=DJIYFXSNF3LPOCFXPBBAPHD5MV4X5RKFCC66AEQEVTEA6NZR2MZA, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.911138993 (557h13m45.782222038s) [noise=true , address=145.79.227.19:4077 , piece_id=74G3RRZP4KLQIU6SUZ3OPH7JYMFYLWVIO3YDO7MVMCT7JAOXT7IQ, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.911273580 (557h13m45.782356615s) [noise=true , address=145.79.227.39:4013 , piece_id=JVIOUEJBGKXM33NHZEL4O3TA3UAHA2LCV27AYFTCSEKF7I4OJZXA, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.911547812 (557h13m45.782630857s) [noise=true , address=45.91.116.4:4937 , piece_id=K3WLF64M4S24U4AEPEFBJKIJXQ47AVTM4D3GARCOFHCXSGLILOKQ, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.911971338 (557h13m45.783054383s) [noise=true , address=45.91.116.12:4227 , piece_id=VE6AZM6VXOSYVYC2Q3TWSR74QOV43BHZ2CQQO6RSM34RVAD5MKJQ, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:47.615828034 (557h13m47.486911079s) [upload=488520]: Next returned nil segment Dec 28 20:42:47.615867419 (557h13m47.486950454s) [upload=488520]: Waiting for error group managing segments... Dec 28 20:42:47.902889789 (557h13m47.773972834s) [noise=true , address=145.79.227.37:4565 , piece_id=6GSLWIFYPMOS3DIYSJHBIGD6OMBOFNUMLLST7AUXICEDCZBDJ5LA, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:47.911211395 (557h13m47.78229443s) [noise=true , address=145.79.227.15:4875 , piece_id=WP727FDSSU2NI5T2L7VYLJOSMJ4VOQN5PYFJIZRVWZ7JX5N6MKWA, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:47.915667801 (557h13m47.786750836s) [noise=true , address=145.79.227.39:4013 , piece_id=JVIOUEJBGKXM33NHZEL4O3TA3UAHA2LCV27AYFTCSEKF7I4OJZXA, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:47.915898529 (557h13m47.786981564s) [noise=true , address=145.79.227.16:4911 , piece_id=VADBIEKNEJKZ6E22T2JM6O7QLHWHDCCFKZAUC3YKURVQBBAIDWKQ, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:47.916613942 (557h13m47.787696987s) [noise=true , address=145.79.227.45:4231 , piece_id=RA5ZQVPR42YPOQG6S6SG6FFWRIM4STOF2L6IUWSD5YIBDPNVU3SQ, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:47.919951998 (557h13m47.791035033s) [noise=true , address=145.79.227.19:4077 , piece_id=74G3RRZP4KLQIU6SUZ3OPH7JYMFYLWVIO3YDO7MVMCT7JAOXT7IQ, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:47.926441575 (557h13m47.79752462s) [noise=true , address=145.79.227.46:4602 , piece_id=5NNVLQOPKQSKXMJPA2VM5HFE5RFV5BBQLEFUUI23QVHVGNBQSV6Q, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:48.144918112 (557h13m48.016001157s) [noise=true , address=45.91.116.5:4300 , piece_id=CRR2UNPJ5AL263FKPAG553XNG52M4OLO4KAGGSCIQPASTGJFLC5A, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:48.149275801 (557h13m48.020358836s) [noise=true , address=77.90.31.21:4878 , piece_id=HMF5EGQKGM2YEBY3OGIOXMVDMCUSGZHFMISWLUFETMPX6TXQGNUQ, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:48.205776737 (557h13m48.076859782s) [noise=true , address=77.90.31.14:4661 , piece_id=HLBOPWGXKQBLUCMFUZVHGOR3GRF5PQXTDEQZYXR7DOINQQWOKFPQ, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:48.219929196 (557h13m48.091012241s) [noise=true , address=77.90.31.15:4697 , piece_id=CNBN2IQKKV3BN5SPAFLZ6ZB4NLL6SVVQDISYAQPDYWWVNKRSU45A, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:48.227276968 (557h13m48.098360013s) [noise=true , address=45.91.116.12:4227 , piece_id=VE6AZM6VXOSYVYC2Q3TWSR74QOV43BHZ2CQQO6RSM34RVAD5MKJQ, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:48.235297289 (557h13m48.106380334s) [noise=true , address=45.91.116.4:4937 , piece_id=K3WLF64M4S24U4AEPEFBJKIJXQ47AVTM4D3GARCOFHCXSGLILOKQ, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:48.235330643 (557h13m48.106413678s) [seg_pos={0 0}, upload=488520]: Segment reached optimal threshold of 13 pieces. Dec 28 20:42:48.235563236 (557h13m48.106646271s) [noise=true , address=77.90.31.12:4590 , piece_id=V7WS7Y6UCOCREAIBLK5V5CAF4UHF6LQ36V5BOPHSZJZBC5MRSM4Q, seg_pos={0 0}, upload=488520]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: write tcp 188.166.225.170:59310->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:59310->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59310->77.90.31.12:4590: use of closed network connection Dec 28 20:42:48.235571932 (557h13m48.106654978s) [noise=true , address=77.90.31.12:4590 , piece_id=V7WS7Y6UCOCREAIBLK5V5CAF4UHF6LQ36V5BOPHSZJZBC5MRSM4Q, seg_pos={0 0}, upload=488520]: Long tail cancellation due to optimal threshold Dec 28 20:42:48.235663838 (557h13m48.106746873s) [noise=true , address=145.79.227.35:4482 , piece_id=D4TRJTUIJG763XYZ7EGTICLOMJZWC4XCCAJWQ63FDXBK3VUHQ6BQ, seg_pos={0 0}, upload=488520]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV): protocol: expected piece hash; context canceled; EOF Dec 28 20:42:48.235688946 (557h13m48.106771991s) [noise=true , address=145.79.227.35:4482 , piece_id=D4TRJTUIJG763XYZ7EGTICLOMJZWC4XCCAJWQ63FDXBK3VUHQ6BQ, seg_pos={0 0}, upload=488520]: Long tail cancellation due to optimal threshold Dec 28 20:42:48.235931368 (557h13m48.107014403s) [noise=true , address=77.90.31.19:4842 , piece_id=DJIYFXSNF3LPOCFXPBBAPHD5MV4X5RKFCC66AEQEVTEA6NZR2MZA, seg_pos={0 0}, upload=488520]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: write tcp 188.166.225.170:36760->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:36760->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36760->77.90.31.19:4842: use of closed network connection Dec 28 20:42:48.235940565 (557h13m48.10702361s) [noise=true , address=77.90.31.19:4842 , piece_id=DJIYFXSNF3LPOCFXPBBAPHD5MV4X5RKFCC66AEQEVTEA6NZR2MZA, seg_pos={0 0}, upload=488520]: Long tail cancellation due to optimal threshold Dec 28 20:42:48.235960813 (557h13m48.107043848s) [seg_pos={0 0}, upload=488520]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 20:42:48.236068709 (557h13m48.107151754s) [upload=488520]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 20:42:48.236076794 (557h13m48.107159829s) [upload=488520]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 20:42:48.536044414 (557h13m48.407127459s) [upload=488520]: Done uploading segments... Dec 28 20:42:48.536067258 (557h13m48.407150293s) [upload=488520]: Upload finished. err: <nil> Duration: 6.167502605s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 10.007722622s trace: https://jaeger.dev.storj.io/trace/499b96822b8dda80 FAIL (2025-12-28 20:43:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12S7WBBQ1Nq8jbeGrZiRKTk23CdJC31hPW2vU1aaM5SsHeVyeRH, Piece ID: V4P3KCUDLI543TSROBYUMVTQKKSQDDVSKSJSXKEZTQAHTXUJ2G4Q) context deadline exceeded; piecestore: (Node ID: 1kjCEPtjvcJ3D7BBKRSRirPybBwkRZ1dJwyAQCadzCvaLYvWhz, Piece ID: 4OULHJ6Z3QJ3BOPDY74TJSKVJG23TOEUCJCUEGY24I5C3MUTAVLQ) context deadline exceeded; piecestore: (Node ID: 1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c, Piece ID: LIYXCPAZOUPKXLKRH3EOWREXUY2LAJXSRZFQILM2QIU7HTW7SY2Q) context deadline exceeded ======LOGS====== ue , address=145.79.227.23:4580 , piece_id=OHYI2SFM6TKJE4LHVNGO2PVGZDNFIQDAIQGKG4FOEHTTASBWA5ZQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.496428825 (557h3m1.36751187s) [noise=true , address=77.90.31.19:4867 , piece_id=LDAWD2ZFG3BZATDRY7MMNWYIX6TOHQ2OIUPCAWAYN266GNUUKZ5A, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.496861509 (557h3m1.367944544s) [noise=true , address=145.79.227.19:4105 , piece_id=RY35FSM6NTWPNF3JE43FZRSNXXYZPUN2B6LRNNUOG6QQR6XSU36Q, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.496899933 (557h3m1.367982978s) [noise=true , address=77.90.31.19:4858 , piece_id=LIYXCPAZOUPKXLKRH3EOWREXUY2LAJXSRZFQILM2QIU7HTW7SY2Q, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497229089 (557h3m1.368312134s) [noise=true , address=77.90.31.17:4784 , piece_id=JV2ROD6MN65B6LZKOL3S7ATJWXAY6FBQSZWITA6IHAXELY7R42KA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497326194 (557h3m1.368409239s) [noise=true , address=145.79.227.33:4904 , piece_id=QCMN4OSBWIWS4JRLIGQIGDSHZFUF2JWEPVXQ4FQ4G3LG5FBCBAWQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497664128 (557h3m1.368747173s) [noise=true , address=145.79.227.34:4918 , piece_id=SPDT2KI2774UJMQIUSVTHXCDF7PD2FBG64BXUYW67UMLPE6H7SBQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497731405 (557h3m1.36881445s) [noise=true , address=77.90.31.12:4602 , piece_id=2YPYKKTQ6ID6VTI4AUZQEG2SY7JYPRSAPU4JLF7DWSOSBBO74LWQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.498205939 (557h3m1.369288984s) [noise=true , address=77.90.31.16:4745 , piece_id=CZBFU33BHMP2BEZIDJKW3J526537HR6HP7OCT2KMAQVY7LGZLPZQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.498424295 (557h3m1.36950734s) [noise=true , address=77.90.31.21:4895 , piece_id=V4P3KCUDLI543TSROBYUMVTQKKSQDDVSKSJSXKEZTQAHTXUJ2G4Q, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.498685422 (557h3m1.369768457s) [noise=true , address=145.79.227.35:4496 , piece_id=JH4XQBT7XYEJXWBGPPF6SJWCPLV77GBOLGOA53C3WUQMWHZXVZQA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499066348 (557h3m1.370149393s) [noise=true , address=77.90.31.11:4563 , piece_id=DIFJPHSKHROTLKCHP6HQBWT7XGE77HVRZTH2354MV2UEWZLHVGCQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499185725 (557h3m1.37026877s) [noise=true , address=145.79.227.30:4768 , piece_id=KK4LEV6RG3IGLYXID5MJF3KMNSJ7WS5JS45PNOOJ26LXLYAUCWOA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499468122 (557h3m1.370551157s) [noise=true , address=77.90.31.14:4688 , piece_id=4OULHJ6Z3QJ3BOPDY74TJSKVJG23TOEUCJCUEGY24I5C3MUTAVLQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499901047 (557h3m1.370984102s) [noise=true , address=45.91.116.5:4330 , piece_id=EVHLSVVR5JGY43TLSKVEQLQ6KDBEZENVAFL6YT7ZQ77Y7X2TXIKA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.500546175 (557h3m1.37162921s) [noise=true , address=145.79.227.43:4171 , piece_id=YF5DRDPGRRVYFL5SWMH2PKTSV3C72BIWKUDSAIZTXOIVOTYKRMNA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:02.979572381 (557h3m2.850655436s) [upload=488362]: Next returned nil segment Dec 28 20:32:02.979609411 (557h3m2.850692456s) [upload=488362]: Waiting for error group managing segments... Dec 28 20:32:03.416896733 (557h3m3.287979778s) [noise=true , address=145.79.227.23:4580 , piece_id=OHYI2SFM6TKJE4LHVNGO2PVGZDNFIQDAIQGKG4FOEHTTASBWA5ZQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.552263655 (557h3m3.4233467s) [noise=true , address=77.90.31.17:4784 , piece_id=JV2ROD6MN65B6LZKOL3S7ATJWXAY6FBQSZWITA6IHAXELY7R42KA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.573564856 (557h3m3.444647902s) [noise=true , address=77.90.31.12:4602 , piece_id=2YPYKKTQ6ID6VTI4AUZQEG2SY7JYPRSAPU4JLF7DWSOSBBO74LWQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.598560923 (557h3m3.469643968s) [noise=true , address=77.90.31.19:4867 , piece_id=LDAWD2ZFG3BZATDRY7MMNWYIX6TOHQ2OIUPCAWAYN266GNUUKZ5A, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.626925925 (557h3m3.49800897s) [noise=true , address=77.90.31.16:4745 , piece_id=CZBFU33BHMP2BEZIDJKW3J526537HR6HP7OCT2KMAQVY7LGZLPZQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.709061947 (557h3m3.580144992s) [noise=true , address=77.90.31.14:4688 , piece_id=4OULHJ6Z3QJ3BOPDY74TJSKVJG23TOEUCJCUEGY24I5C3MUTAVLQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.788879373 (557h3m3.659962418s) [noise=true , address=77.90.31.21:4895 , piece_id=V4P3KCUDLI543TSROBYUMVTQKKSQDDVSKSJSXKEZTQAHTXUJ2G4Q, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.217616523 (557h3m4.088699569s) [noise=true , address=77.90.31.19:4858 , piece_id=LIYXCPAZOUPKXLKRH3EOWREXUY2LAJXSRZFQILM2QIU7HTW7SY2Q, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.220337625 (557h3m4.09142067s) [noise=true , address=77.90.31.11:4563 , piece_id=DIFJPHSKHROTLKCHP6HQBWT7XGE77HVRZTH2354MV2UEWZLHVGCQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.303958744 (557h3m4.175041789s) [noise=true , address=145.79.227.44:4213 , piece_id=UEAQDBBCP7DIG6ZD3EEM2CKAL4W5QKW23LWFUAMSVUW5ZS2PJDRQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.312794398 (557h3m4.183877444s) [noise=true , address=145.79.227.34:4918 , piece_id=SPDT2KI2774UJMQIUSVTHXCDF7PD2FBG64BXUYW67UMLPE6H7SBQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.349308798 (557h3m4.220391833s) [noise=true , address=145.79.227.40:4058 , piece_id=EQ4LA7T2BCGWY76K3NJI4X77OQJFJED6H4VNGXDQMILU6IVQWOMQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.371518379 (557h3m4.242601424s) [noise=true , address=145.79.227.43:4171 , piece_id=YF5DRDPGRRVYFL5SWMH2PKTSV3C72BIWKUDSAIZTXOIVOTYKRMNA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.376232316 (557h3m4.247315361s) [noise=true , address=145.79.227.30:4768 , piece_id=KK4LEV6RG3IGLYXID5MJF3KMNSJ7WS5JS45PNOOJ26LXLYAUCWOA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.376257844 (557h3m4.247340879s) [seg_pos={0 0}, upload=488362]: Segment reached optimal threshold of 14 pieces. Dec 28 20:32:04.376575799 (557h3m4.247658834s) [noise=true , address=145.79.227.33:4904 , piece_id=QCMN4OSBWIWS4JRLIGQIGDSHZFUF2JWEPVXQ4FQ4G3LG5FBCBAWQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1dC7mC2Cbxmys7ukEkELyF9Us7F1qtptSP6yDJh1EfjVJ6uqsC): protocol: expected piece hash; context canceled; EOF Dec 28 20:32:04.376585127 (557h3m4.247668163s) [noise=true , address=145.79.227.33:4904 , piece_id=QCMN4OSBWIWS4JRLIGQIGDSHZFUF2JWEPVXQ4FQ4G3LG5FBCBAWQ, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.376764359 (557h3m4.247847404s) [noise=true , address=145.79.227.19:4105 , piece_id=RY35FSM6NTWPNF3JE43FZRSNXXYZPUN2B6LRNNUOG6QQR6XSU36Q, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GhDrKMcDEg2RxamqKtiHphpgNdvtZYdHXrjzGsNJFjBFYeEen): protocol: write tcp 188.166.225.170:38838->145.79.227.19:4105: use of closed network connection; write tcp 188.166.225.170:38838->145.79.227.19:4105: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38838->145.79.227.19:4105: use of closed network connection Dec 28 20:32:04.376772865 (557h3m4.2478559s) [noise=true , address=145.79.227.19:4105 , piece_id=RY35FSM6NTWPNF3JE43FZRSNXXYZPUN2B6LRNNUOG6QQR6XSU36Q, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.376926658 (557h3m4.248009703s) [noise=true , address=145.79.227.35:4496 , piece_id=JH4XQBT7XYEJXWBGPPF6SJWCPLV77GBOLGOA53C3WUQMWHZXVZQA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V8XhSWkyGUVYFR8MykifFAP1YAfGCwzqucDcMMfzw3xsVW77Z): protocol: expected piece hash; context canceled; EOF Dec 28 20:32:04.377165332 (557h3m4.248248377s) [noise=true , address=145.79.227.35:4496 , piece_id=JH4XQBT7XYEJXWBGPPF6SJWCPLV77GBOLGOA53C3WUQMWHZXVZQA, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.377212682 (557h3m4.248295848s) [noise=true , address=45.91.116.5:4330 , piece_id=EVHLSVVR5JGY43TLSKVEQLQ6KDBEZENVAFL6YT7ZQ77Y7X2TXIKA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12MK9aVDRwfWPfckFR6RfPUL1Nrm9bwfsgV7TXCKGbtadHebWvW): protocol: read tcp 188.166.225.170:55416->45.91.116.5:4330: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 20:32:04.377289859 (557h3m4.248372924s) [noise=true , address=45.91.116.5:4330 , piece_id=EVHLSVVR5JGY43TLSKVEQLQ6KDBEZENVAFL6YT7ZQ77Y7X2TXIKA, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.377387736 (557h3m4.248470781s) [seg_pos={0 0}, upload=488362]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 20:32:04.377599729 (557h3m4.248682774s) [upload=488362]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 20:32:04.377613535 (557h3m4.24869662s) [upload=488362]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 20:32:04.675090855 (557h3m4.5461739s) [upload=488362]: Done uploading segments... Dec 28 20:32:04.675121233 (557h3m4.546204268s) [upload=488362]: Upload finished. err: <nil> Duration: 10.007792014s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [....................X...........X..................X.......X] duration: 30.003100549s trace: https://jaeger.dev.storj.io/trace/27073096d2c66e66 FAIL (2025-12-28 20:43:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: VE6AZM6VXOSYVYC2Q3TWSR74QOV43BHZ2CQQO6RSM34RVAD5MKJQ) context deadline exceeded; piecestore: (Node ID: 12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, Piece ID: CNBN2IQKKV3BN5SPAFLZ6ZB4NLL6SVVQDISYAQPDYWWVNKRSU45A) context deadline exceeded; piecestore: (Node ID: 13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, Piece ID: VADBIEKNEJKZ6E22T2JM6O7QLHWHDCCFKZAUC3YKURVQBBAIDWKQ) context deadline exceeded ======LOGS====== g segments... Dec 28 20:42:45.662246218 (557h13m45.533329253s) [upload=488520]: Got next segment. Inline: false Dec 28 20:42:45.662270836 (557h13m45.533353881s) [upload=488520]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 20:42:45.662274963 (557h13m45.533357998s) [upload=488520]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 20:42:45.907695937 (557h13m45.778778992s) [seg_pos={0 0}, upload=488520]: Begin upload segment... Dec 28 20:42:45.907765980 (557h13m45.778849015s) [seg_pos={0 0}, upload=488520]: Done begin upload segment. Dec 28 20:42:45.907818851 (557h13m45.778901886s) [noise=true , address=77.90.31.21:4878 , piece_id=HMF5EGQKGM2YEBY3OGIOXMVDMCUSGZHFMISWLUFETMPX6TXQGNUQ, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.907835683 (557h13m45.778918728s) [noise=true , address=145.79.227.16:4911 , piece_id=VADBIEKNEJKZ6E22T2JM6O7QLHWHDCCFKZAUC3YKURVQBBAIDWKQ, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.908356295 (557h13m45.77943934s) [noise=true , address=77.90.31.14:4661 , piece_id=HLBOPWGXKQBLUCMFUZVHGOR3GRF5PQXTDEQZYXR7DOINQQWOKFPQ, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.908784009 (557h13m45.779867054s) [noise=true , address=145.79.227.35:4482 , piece_id=D4TRJTUIJG763XYZ7EGTICLOMJZWC4XCCAJWQ63FDXBK3VUHQ6BQ, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.909197486 (557h13m45.780280531s) [noise=true , address=145.79.227.45:4231 , piece_id=RA5ZQVPR42YPOQG6S6SG6FFWRIM4STOF2L6IUWSD5YIBDPNVU3SQ, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.909211162 (557h13m45.780294207s) [noise=true , address=145.79.227.15:4875 , piece_id=WP727FDSSU2NI5T2L7VYLJOSMJ4VOQN5PYFJIZRVWZ7JX5N6MKWA, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.909627475 (557h13m45.78071052s) [noise=true , address=145.79.227.46:4602 , piece_id=5NNVLQOPKQSKXMJPA2VM5HFE5RFV5BBQLEFUUI23QVHVGNBQSV6Q, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.909651611 (557h13m45.780734656s) [noise=true , address=145.79.227.37:4565 , piece_id=6GSLWIFYPMOS3DIYSJHBIGD6OMBOFNUMLLST7AUXICEDCZBDJ5LA, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.909972622 (557h13m45.781055657s) [noise=true , address=77.90.31.15:4697 , piece_id=CNBN2IQKKV3BN5SPAFLZ6ZB4NLL6SVVQDISYAQPDYWWVNKRSU45A, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.910169056 (557h13m45.781252101s) [noise=true , address=45.91.116.5:4300 , piece_id=CRR2UNPJ5AL263FKPAG553XNG52M4OLO4KAGGSCIQPASTGJFLC5A, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.910608624 (557h13m45.781691659s) [noise=true , address=77.90.31.12:4590 , piece_id=V7WS7Y6UCOCREAIBLK5V5CAF4UHF6LQ36V5BOPHSZJZBC5MRSM4Q, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.910880450 (557h13m45.781963485s) [noise=true , address=77.90.31.19:4842 , piece_id=DJIYFXSNF3LPOCFXPBBAPHD5MV4X5RKFCC66AEQEVTEA6NZR2MZA, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.911138993 (557h13m45.782222038s) [noise=true , address=145.79.227.19:4077 , piece_id=74G3RRZP4KLQIU6SUZ3OPH7JYMFYLWVIO3YDO7MVMCT7JAOXT7IQ, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.911273580 (557h13m45.782356615s) [noise=true , address=145.79.227.39:4013 , piece_id=JVIOUEJBGKXM33NHZEL4O3TA3UAHA2LCV27AYFTCSEKF7I4OJZXA, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.911547812 (557h13m45.782630857s) [noise=true , address=45.91.116.4:4937 , piece_id=K3WLF64M4S24U4AEPEFBJKIJXQ47AVTM4D3GARCOFHCXSGLILOKQ, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.911971338 (557h13m45.783054383s) [noise=true , address=45.91.116.12:4227 , piece_id=VE6AZM6VXOSYVYC2Q3TWSR74QOV43BHZ2CQQO6RSM34RVAD5MKJQ, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:47.615828034 (557h13m47.486911079s) [upload=488520]: Next returned nil segment Dec 28 20:42:47.615867419 (557h13m47.486950454s) [upload=488520]: Waiting for error group managing segments... Dec 28 20:42:47.902889789 (557h13m47.773972834s) [noise=true , address=145.79.227.37:4565 , piece_id=6GSLWIFYPMOS3DIYSJHBIGD6OMBOFNUMLLST7AUXICEDCZBDJ5LA, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:47.911211395 (557h13m47.78229443s) [noise=true , address=145.79.227.15:4875 , piece_id=WP727FDSSU2NI5T2L7VYLJOSMJ4VOQN5PYFJIZRVWZ7JX5N6MKWA, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:47.915667801 (557h13m47.786750836s) [noise=true , address=145.79.227.39:4013 , piece_id=JVIOUEJBGKXM33NHZEL4O3TA3UAHA2LCV27AYFTCSEKF7I4OJZXA, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:47.915898529 (557h13m47.786981564s) [noise=true , address=145.79.227.16:4911 , piece_id=VADBIEKNEJKZ6E22T2JM6O7QLHWHDCCFKZAUC3YKURVQBBAIDWKQ, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:47.916613942 (557h13m47.787696987s) [noise=true , address=145.79.227.45:4231 , piece_id=RA5ZQVPR42YPOQG6S6SG6FFWRIM4STOF2L6IUWSD5YIBDPNVU3SQ, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:47.919951998 (557h13m47.791035033s) [noise=true , address=145.79.227.19:4077 , piece_id=74G3RRZP4KLQIU6SUZ3OPH7JYMFYLWVIO3YDO7MVMCT7JAOXT7IQ, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:47.926441575 (557h13m47.79752462s) [noise=true , address=145.79.227.46:4602 , piece_id=5NNVLQOPKQSKXMJPA2VM5HFE5RFV5BBQLEFUUI23QVHVGNBQSV6Q, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:48.144918112 (557h13m48.016001157s) [noise=true , address=45.91.116.5:4300 , piece_id=CRR2UNPJ5AL263FKPAG553XNG52M4OLO4KAGGSCIQPASTGJFLC5A, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:48.149275801 (557h13m48.020358836s) [noise=true , address=77.90.31.21:4878 , piece_id=HMF5EGQKGM2YEBY3OGIOXMVDMCUSGZHFMISWLUFETMPX6TXQGNUQ, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:48.205776737 (557h13m48.076859782s) [noise=true , address=77.90.31.14:4661 , piece_id=HLBOPWGXKQBLUCMFUZVHGOR3GRF5PQXTDEQZYXR7DOINQQWOKFPQ, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:48.219929196 (557h13m48.091012241s) [noise=true , address=77.90.31.15:4697 , piece_id=CNBN2IQKKV3BN5SPAFLZ6ZB4NLL6SVVQDISYAQPDYWWVNKRSU45A, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:48.227276968 (557h13m48.098360013s) [noise=true , address=45.91.116.12:4227 , piece_id=VE6AZM6VXOSYVYC2Q3TWSR74QOV43BHZ2CQQO6RSM34RVAD5MKJQ, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:48.235297289 (557h13m48.106380334s) [noise=true , address=45.91.116.4:4937 , piece_id=K3WLF64M4S24U4AEPEFBJKIJXQ47AVTM4D3GARCOFHCXSGLILOKQ, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:48.235330643 (557h13m48.106413678s) [seg_pos={0 0}, upload=488520]: Segment reached optimal threshold of 13 pieces. Dec 28 20:42:48.235563236 (557h13m48.106646271s) [noise=true , address=77.90.31.12:4590 , piece_id=V7WS7Y6UCOCREAIBLK5V5CAF4UHF6LQ36V5BOPHSZJZBC5MRSM4Q, seg_pos={0 0}, upload=488520]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: write tcp 188.166.225.170:59310->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:59310->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59310->77.90.31.12:4590: use of closed network connection Dec 28 20:42:48.235571932 (557h13m48.106654978s) [noise=true , address=77.90.31.12:4590 , piece_id=V7WS7Y6UCOCREAIBLK5V5CAF4UHF6LQ36V5BOPHSZJZBC5MRSM4Q, seg_pos={0 0}, upload=488520]: Long tail cancellation due to optimal threshold Dec 28 20:42:48.235663838 (557h13m48.106746873s) [noise=true , address=145.79.227.35:4482 , piece_id=D4TRJTUIJG763XYZ7EGTICLOMJZWC4XCCAJWQ63FDXBK3VUHQ6BQ, seg_pos={0 0}, upload=488520]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV): protocol: expected piece hash; context canceled; EOF Dec 28 20:42:48.235688946 (557h13m48.106771991s) [noise=true , address=145.79.227.35:4482 , piece_id=D4TRJTUIJG763XYZ7EGTICLOMJZWC4XCCAJWQ63FDXBK3VUHQ6BQ, seg_pos={0 0}, upload=488520]: Long tail cancellation due to optimal threshold Dec 28 20:42:48.235931368 (557h13m48.107014403s) [noise=true , address=77.90.31.19:4842 , piece_id=DJIYFXSNF3LPOCFXPBBAPHD5MV4X5RKFCC66AEQEVTEA6NZR2MZA, seg_pos={0 0}, upload=488520]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: write tcp 188.166.225.170:36760->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:36760->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36760->77.90.31.19:4842: use of closed network connection Dec 28 20:42:48.235940565 (557h13m48.10702361s) [noise=true , address=77.90.31.19:4842 , piece_id=DJIYFXSNF3LPOCFXPBBAPHD5MV4X5RKFCC66AEQEVTEA6NZR2MZA, seg_pos={0 0}, upload=488520]: Long tail cancellation due to optimal threshold Dec 28 20:42:48.235960813 (557h13m48.107043848s) [seg_pos={0 0}, upload=488520]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 20:42:48.236068709 (557h13m48.107151754s) [upload=488520]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 20:42:48.236076794 (557h13m48.107159829s) [upload=488520]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 20:42:48.536044414 (557h13m48.407127459s) [upload=488520]: Done uploading segments... Dec 28 20:42:48.536067258 (557h13m48.407150293s) [upload=488520]: Upload finished. err: <nil> Duration: 30.003121779s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 33.785015085s trace: https://jaeger.dev.storj.io/trace/499b96822b8dda80 FAIL (2025-12-28 20:43:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12S7WBBQ1Nq8jbeGrZiRKTk23CdJC31hPW2vU1aaM5SsHeVyeRH, Piece ID: V4P3KCUDLI543TSROBYUMVTQKKSQDDVSKSJSXKEZTQAHTXUJ2G4Q) context deadline exceeded; piecestore: (Node ID: 1kjCEPtjvcJ3D7BBKRSRirPybBwkRZ1dJwyAQCadzCvaLYvWhz, Piece ID: 4OULHJ6Z3QJ3BOPDY74TJSKVJG23TOEUCJCUEGY24I5C3MUTAVLQ) context deadline exceeded; piecestore: (Node ID: 1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c, Piece ID: LIYXCPAZOUPKXLKRH3EOWREXUY2LAJXSRZFQILM2QIU7HTW7SY2Q) context deadline exceeded ======LOGS====== ue , address=145.79.227.23:4580 , piece_id=OHYI2SFM6TKJE4LHVNGO2PVGZDNFIQDAIQGKG4FOEHTTASBWA5ZQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.496428825 (557h3m1.36751187s) [noise=true , address=77.90.31.19:4867 , piece_id=LDAWD2ZFG3BZATDRY7MMNWYIX6TOHQ2OIUPCAWAYN266GNUUKZ5A, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.496861509 (557h3m1.367944544s) [noise=true , address=145.79.227.19:4105 , piece_id=RY35FSM6NTWPNF3JE43FZRSNXXYZPUN2B6LRNNUOG6QQR6XSU36Q, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.496899933 (557h3m1.367982978s) [noise=true , address=77.90.31.19:4858 , piece_id=LIYXCPAZOUPKXLKRH3EOWREXUY2LAJXSRZFQILM2QIU7HTW7SY2Q, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497229089 (557h3m1.368312134s) [noise=true , address=77.90.31.17:4784 , piece_id=JV2ROD6MN65B6LZKOL3S7ATJWXAY6FBQSZWITA6IHAXELY7R42KA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497326194 (557h3m1.368409239s) [noise=true , address=145.79.227.33:4904 , piece_id=QCMN4OSBWIWS4JRLIGQIGDSHZFUF2JWEPVXQ4FQ4G3LG5FBCBAWQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497664128 (557h3m1.368747173s) [noise=true , address=145.79.227.34:4918 , piece_id=SPDT2KI2774UJMQIUSVTHXCDF7PD2FBG64BXUYW67UMLPE6H7SBQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497731405 (557h3m1.36881445s) [noise=true , address=77.90.31.12:4602 , piece_id=2YPYKKTQ6ID6VTI4AUZQEG2SY7JYPRSAPU4JLF7DWSOSBBO74LWQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.498205939 (557h3m1.369288984s) [noise=true , address=77.90.31.16:4745 , piece_id=CZBFU33BHMP2BEZIDJKW3J526537HR6HP7OCT2KMAQVY7LGZLPZQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.498424295 (557h3m1.36950734s) [noise=true , address=77.90.31.21:4895 , piece_id=V4P3KCUDLI543TSROBYUMVTQKKSQDDVSKSJSXKEZTQAHTXUJ2G4Q, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.498685422 (557h3m1.369768457s) [noise=true , address=145.79.227.35:4496 , piece_id=JH4XQBT7XYEJXWBGPPF6SJWCPLV77GBOLGOA53C3WUQMWHZXVZQA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499066348 (557h3m1.370149393s) [noise=true , address=77.90.31.11:4563 , piece_id=DIFJPHSKHROTLKCHP6HQBWT7XGE77HVRZTH2354MV2UEWZLHVGCQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499185725 (557h3m1.37026877s) [noise=true , address=145.79.227.30:4768 , piece_id=KK4LEV6RG3IGLYXID5MJF3KMNSJ7WS5JS45PNOOJ26LXLYAUCWOA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499468122 (557h3m1.370551157s) [noise=true , address=77.90.31.14:4688 , piece_id=4OULHJ6Z3QJ3BOPDY74TJSKVJG23TOEUCJCUEGY24I5C3MUTAVLQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499901047 (557h3m1.370984102s) [noise=true , address=45.91.116.5:4330 , piece_id=EVHLSVVR5JGY43TLSKVEQLQ6KDBEZENVAFL6YT7ZQ77Y7X2TXIKA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.500546175 (557h3m1.37162921s) [noise=true , address=145.79.227.43:4171 , piece_id=YF5DRDPGRRVYFL5SWMH2PKTSV3C72BIWKUDSAIZTXOIVOTYKRMNA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:02.979572381 (557h3m2.850655436s) [upload=488362]: Next returned nil segment Dec 28 20:32:02.979609411 (557h3m2.850692456s) [upload=488362]: Waiting for error group managing segments... Dec 28 20:32:03.416896733 (557h3m3.287979778s) [noise=true , address=145.79.227.23:4580 , piece_id=OHYI2SFM6TKJE4LHVNGO2PVGZDNFIQDAIQGKG4FOEHTTASBWA5ZQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.552263655 (557h3m3.4233467s) [noise=true , address=77.90.31.17:4784 , piece_id=JV2ROD6MN65B6LZKOL3S7ATJWXAY6FBQSZWITA6IHAXELY7R42KA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.573564856 (557h3m3.444647902s) [noise=true , address=77.90.31.12:4602 , piece_id=2YPYKKTQ6ID6VTI4AUZQEG2SY7JYPRSAPU4JLF7DWSOSBBO74LWQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.598560923 (557h3m3.469643968s) [noise=true , address=77.90.31.19:4867 , piece_id=LDAWD2ZFG3BZATDRY7MMNWYIX6TOHQ2OIUPCAWAYN266GNUUKZ5A, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.626925925 (557h3m3.49800897s) [noise=true , address=77.90.31.16:4745 , piece_id=CZBFU33BHMP2BEZIDJKW3J526537HR6HP7OCT2KMAQVY7LGZLPZQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.709061947 (557h3m3.580144992s) [noise=true , address=77.90.31.14:4688 , piece_id=4OULHJ6Z3QJ3BOPDY74TJSKVJG23TOEUCJCUEGY24I5C3MUTAVLQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.788879373 (557h3m3.659962418s) [noise=true , address=77.90.31.21:4895 , piece_id=V4P3KCUDLI543TSROBYUMVTQKKSQDDVSKSJSXKEZTQAHTXUJ2G4Q, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.217616523 (557h3m4.088699569s) [noise=true , address=77.90.31.19:4858 , piece_id=LIYXCPAZOUPKXLKRH3EOWREXUY2LAJXSRZFQILM2QIU7HTW7SY2Q, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.220337625 (557h3m4.09142067s) [noise=true , address=77.90.31.11:4563 , piece_id=DIFJPHSKHROTLKCHP6HQBWT7XGE77HVRZTH2354MV2UEWZLHVGCQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.303958744 (557h3m4.175041789s) [noise=true , address=145.79.227.44:4213 , piece_id=UEAQDBBCP7DIG6ZD3EEM2CKAL4W5QKW23LWFUAMSVUW5ZS2PJDRQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.312794398 (557h3m4.183877444s) [noise=true , address=145.79.227.34:4918 , piece_id=SPDT2KI2774UJMQIUSVTHXCDF7PD2FBG64BXUYW67UMLPE6H7SBQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.349308798 (557h3m4.220391833s) [noise=true , address=145.79.227.40:4058 , piece_id=EQ4LA7T2BCGWY76K3NJI4X77OQJFJED6H4VNGXDQMILU6IVQWOMQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.371518379 (557h3m4.242601424s) [noise=true , address=145.79.227.43:4171 , piece_id=YF5DRDPGRRVYFL5SWMH2PKTSV3C72BIWKUDSAIZTXOIVOTYKRMNA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.376232316 (557h3m4.247315361s) [noise=true , address=145.79.227.30:4768 , piece_id=KK4LEV6RG3IGLYXID5MJF3KMNSJ7WS5JS45PNOOJ26LXLYAUCWOA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.376257844 (557h3m4.247340879s) [seg_pos={0 0}, upload=488362]: Segment reached optimal threshold of 14 pieces. Dec 28 20:32:04.376575799 (557h3m4.247658834s) [noise=true , address=145.79.227.33:4904 , piece_id=QCMN4OSBWIWS4JRLIGQIGDSHZFUF2JWEPVXQ4FQ4G3LG5FBCBAWQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1dC7mC2Cbxmys7ukEkELyF9Us7F1qtptSP6yDJh1EfjVJ6uqsC): protocol: expected piece hash; context canceled; EOF Dec 28 20:32:04.376585127 (557h3m4.247668163s) [noise=true , address=145.79.227.33:4904 , piece_id=QCMN4OSBWIWS4JRLIGQIGDSHZFUF2JWEPVXQ4FQ4G3LG5FBCBAWQ, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.376764359 (557h3m4.247847404s) [noise=true , address=145.79.227.19:4105 , piece_id=RY35FSM6NTWPNF3JE43FZRSNXXYZPUN2B6LRNNUOG6QQR6XSU36Q, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GhDrKMcDEg2RxamqKtiHphpgNdvtZYdHXrjzGsNJFjBFYeEen): protocol: write tcp 188.166.225.170:38838->145.79.227.19:4105: use of closed network connection; write tcp 188.166.225.170:38838->145.79.227.19:4105: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38838->145.79.227.19:4105: use of closed network connection Dec 28 20:32:04.376772865 (557h3m4.2478559s) [noise=true , address=145.79.227.19:4105 , piece_id=RY35FSM6NTWPNF3JE43FZRSNXXYZPUN2B6LRNNUOG6QQR6XSU36Q, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.376926658 (557h3m4.248009703s) [noise=true , address=145.79.227.35:4496 , piece_id=JH4XQBT7XYEJXWBGPPF6SJWCPLV77GBOLGOA53C3WUQMWHZXVZQA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V8XhSWkyGUVYFR8MykifFAP1YAfGCwzqucDcMMfzw3xsVW77Z): protocol: expected piece hash; context canceled; EOF Dec 28 20:32:04.377165332 (557h3m4.248248377s) [noise=true , address=145.79.227.35:4496 , piece_id=JH4XQBT7XYEJXWBGPPF6SJWCPLV77GBOLGOA53C3WUQMWHZXVZQA, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.377212682 (557h3m4.248295848s) [noise=true , address=45.91.116.5:4330 , piece_id=EVHLSVVR5JGY43TLSKVEQLQ6KDBEZENVAFL6YT7ZQ77Y7X2TXIKA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12MK9aVDRwfWPfckFR6RfPUL1Nrm9bwfsgV7TXCKGbtadHebWvW): protocol: read tcp 188.166.225.170:55416->45.91.116.5:4330: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 20:32:04.377289859 (557h3m4.248372924s) [noise=true , address=45.91.116.5:4330 , piece_id=EVHLSVVR5JGY43TLSKVEQLQ6KDBEZENVAFL6YT7ZQ77Y7X2TXIKA, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.377387736 (557h3m4.248470781s) [seg_pos={0 0}, upload=488362]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 20:32:04.377599729 (557h3m4.248682774s) [upload=488362]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 20:32:04.377613535 (557h3m4.24869662s) [upload=488362]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 20:32:04.675090855 (557h3m4.5461739s) [upload=488362]: Done uploading segments... Dec 28 20:32:04.675121233 (557h3m4.546204268s) [upload=488362]: Upload finished. err: <nil> Duration: 33.78504433s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....................X...........X..................X.......] duration: 6.17911129s trace: https://jaeger.dev.storj.io/trace/60ddd33e73f0de61 FAIL (2025-12-28 20:42:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 065810223s) [upload=488441]: Uploading segments... Dec 28 20:37:27.194835273 (557h8m27.065918308s) [upload=488441]: Got next segment. Inline: false Dec 28 20:37:27.194877263 (557h8m27.065960308s) [upload=488441]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 20:37:27.194884157 (557h8m27.065967212s) [upload=488441]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 20:37:27.450350479 (557h8m27.321433524s) [seg_pos={0 0}, upload=488441]: Begin upload segment... Dec 28 20:37:27.450441583 (557h8m27.321524628s) [seg_pos={0 0}, upload=488441]: Done begin upload segment. Dec 28 20:37:27.450581259 (557h8m27.321664304s) [noise=true , address=145.79.227.2:4114 , piece_id=MTU5VYRZVWYYP44THTSGDURQ3WHCCNH2C3XJJYQLUD7JBXUU5NZQ, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.450619542 (557h8m27.321702597s) [noise=true , address=45.91.116.6:4975 , piece_id=DCEIFZV6754QAYC5N2SXZ5DIYC7NR6TMHCXD66T4A5JRL7E4LPHA, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.451283105 (557h8m27.32236617s) [noise=true , address=145.79.227.51:4410 , piece_id=HGHWLMHV7LMCQGNQGSFRXUGJUVOIPYBPT3XATCP4EBZ2N24YW5LA, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.451709617 (557h8m27.322792662s) [noise=true , address=145.79.227.14:4840 , piece_id=VVOFKZVGGDZG6E75IVAUFUGSFNAMN5STZIPSURJ64JPGOWY6BJZQ, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.452173731 (557h8m27.323256766s) [noise=true , address=145.79.227.6:4257 , piece_id=DSINVYJEDHKBN7OGTQZM34CT6MBFUZMALJZQFOGTS5CLTSZLD4CA, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.452596105 (557h8m27.32367914s) [noise=true , address=145.79.227.32:4835 , piece_id=NIDZUDIX6K564WEVU7QDG2DH76GODGMLOJVUVLH6GS7GMUV45D7A, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.453086699 (557h8m27.324169734s) [noise=true , address=145.79.227.11:4731 , piece_id=R3RGSPD3B7K72ZH4AWXSR6K6D5KYZYLXIOM6QB346S4XZOGKR24A, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.453622139 (557h8m27.324705184s) [noise=true , address=145.79.227.15:4873 , piece_id=2MYCI55SJDFGKGRPUC2BLKB4XXLTFHCNH65JEV6KADXDBJUNBH3Q, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.455648808 (557h8m27.326731853s) [noise=true , address=145.79.227.5:4222 , piece_id=IBMJHO2KIDNN5AIIEGX5G5QVDJXXGLOTAMOQC4LZDUBBGY5E6GLQ, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.456606212 (557h8m27.327689247s) [noise=true , address=45.91.116.7:4012 , piece_id=C4J7O54YJEGULQT2ZWHPSWTZUU2LU2MN2ND2HSTJI2E6F7OHJSRQ, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.457955700 (557h8m27.329038735s) [noise=true , address=145.79.227.37:4565 , piece_id=3EJIDJFRGBG6272N24VFWNAXKUW7DHHM563DGJTMRURHA27CEREQ, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.459219468 (557h8m27.330302513s) [noise=true , address=145.79.227.27:4690 , piece_id=SZGKSH5YEIQBDSIMVFXRL2X6PXVXGYZ4DOFXWWE6UDQ4CEFTQHJA, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.460412208 (557h8m27.331495254s) [noise=true , address=77.90.31.16:4736 , piece_id=6NKLMJS4Z4DRFNCOYKO36LAI4EUWFB7LVXEYFDUYUMKSX6R235TQ, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.461658171 (557h8m27.332741206s) [noise=true , address=145.79.227.26:4654 , piece_id=XZZEQ7Y3SLK6WJJ22HH5RUTABJIYTONKTITYKENJXREOXSNSDHIA, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.462997892 (557h8m27.334080927s) [noise=true , address=45.91.116.5:4299 , piece_id=6UHFIECEOY2EGUKVFY2T5STZWL4DXXFP3NY4WVZKEEIREZ7ZVX2A, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.465119382 (557h8m27.336202427s) [noise=true , address=45.91.116.4:4939 , piece_id=XQHY6GBPJ6XEOFL2FTOFNBRLD3GF6VLGUBWY2D7TUOPQSFERNQTA, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:29.157770565 (557h8m29.02885361s) [upload=488441]: Next returned nil segment Dec 28 20:37:29.157787036 (557h8m29.028870091s) [upload=488441]: Waiting for error group managing segments... Dec 28 20:37:29.455180586 (557h8m29.326263631s) [noise=true , address=145.79.227.26:4654 , piece_id=XZZEQ7Y3SLK6WJJ22HH5RUTABJIYTONKTITYKENJXREOXSNSDHIA, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.457221773 (557h8m29.328304808s) [noise=true , address=145.79.227.6:4257 , piece_id=DSINVYJEDHKBN7OGTQZM34CT6MBFUZMALJZQFOGTS5CLTSZLD4CA, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.458700027 (557h8m29.329783062s) [noise=true , address=145.79.227.11:4731 , piece_id=R3RGSPD3B7K72ZH4AWXSR6K6D5KYZYLXIOM6QB346S4XZOGKR24A, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.459240897 (557h8m29.330323933s) [noise=true , address=145.79.227.51:4410 , piece_id=HGHWLMHV7LMCQGNQGSFRXUGJUVOIPYBPT3XATCP4EBZ2N24YW5LA, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.462155848 (557h8m29.333238883s) [noise=true , address=145.79.227.15:4873 , piece_id=2MYCI55SJDFGKGRPUC2BLKB4XXLTFHCNH65JEV6KADXDBJUNBH3Q, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.465572805 (557h8m29.33665585s) [noise=true , address=145.79.227.32:4835 , piece_id=NIDZUDIX6K564WEVU7QDG2DH76GODGMLOJVUVLH6GS7GMUV45D7A, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.470903064 (557h8m29.341986109s) [noise=true , address=145.79.227.2:4114 , piece_id=MTU5VYRZVWYYP44THTSGDURQ3WHCCNH2C3XJJYQLUD7JBXUU5NZQ, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.485295701 (557h8m29.356378736s) [noise=true , address=145.79.227.37:4565 , piece_id=3EJIDJFRGBG6272N24VFWNAXKUW7DHHM563DGJTMRURHA27CEREQ, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.497253901 (557h8m29.368336936s) [noise=true , address=145.79.227.14:4840 , piece_id=VVOFKZVGGDZG6E75IVAUFUGSFNAMN5STZIPSURJ64JPGOWY6BJZQ, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.641989392 (557h8m29.513072437s) [noise=true , address=77.90.31.16:4736 , piece_id=6NKLMJS4Z4DRFNCOYKO36LAI4EUWFB7LVXEYFDUYUMKSX6R235TQ, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.701668272 (557h8m29.572751307s) [noise=true , address=45.91.116.5:4299 , piece_id=6UHFIECEOY2EGUKVFY2T5STZWL4DXXFP3NY4WVZKEEIREZ7ZVX2A, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.702897923 (557h8m29.573980958s) [noise=true , address=45.91.116.6:4975 , piece_id=DCEIFZV6754QAYC5N2SXZ5DIYC7NR6TMHCXD66T4A5JRL7E4LPHA, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.762609574 (557h8m29.633692619s) [noise=true , address=45.91.116.7:4012 , piece_id=C4J7O54YJEGULQT2ZWHPSWTZUU2LU2MN2ND2HSTJI2E6F7OHJSRQ, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.762632388 (557h8m29.633715433s) [seg_pos={0 0}, upload=488441]: Segment reached optimal threshold of 13 pieces. Dec 28 20:37:29.762867626 (557h8m29.633950671s) [noise=true , address=145.79.227.5:4222 , piece_id=IBMJHO2KIDNN5AIIEGX5G5QVDJXXGLOTAMOQC4LZDUBBGY5E6GLQ, seg_pos={0 0}, upload=488441]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1): protocol: write tcp 188.166.225.170:44926->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:44926->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44926->145.79.227.5:4222: use of closed network connection Dec 28 20:37:29.762890609 (557h8m29.633973644s) [noise=true , address=145.79.227.5:4222 , piece_id=IBMJHO2KIDNN5AIIEGX5G5QVDJXXGLOTAMOQC4LZDUBBGY5E6GLQ, seg_pos={0 0}, upload=488441]: Long tail cancellation due to optimal threshold Dec 28 20:37:29.762944091 (557h8m29.634027126s) [noise=true , address=45.91.116.4:4939 , piece_id=XQHY6GBPJ6XEOFL2FTOFNBRLD3GF6VLGUBWY2D7TUOPQSFERNQTA, seg_pos={0 0}, upload=488441]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 28 20:37:29.762949461 (557h8m29.634032496s) [noise=true , address=45.91.116.4:4939 , piece_id=XQHY6GBPJ6XEOFL2FTOFNBRLD3GF6VLGUBWY2D7TUOPQSFERNQTA, seg_pos={0 0}, upload=488441]: Long tail cancellation due to optimal threshold Dec 28 20:37:29.763074540 (557h8m29.634157585s) [noise=true , address=145.79.227.27:4690 , piece_id=SZGKSH5YEIQBDSIMVFXRL2X6PXVXGYZ4DOFXWWE6UDQ4CEFTQHJA, seg_pos={0 0}, upload=488441]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H): protocol: write tcp 188.166.225.170:54746->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:54746->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54746->145.79.227.27:4690: use of closed network connection Dec 28 20:37:29.763081002 (557h8m29.634164037s) [noise=true , address=145.79.227.27:4690 , piece_id=SZGKSH5YEIQBDSIMVFXRL2X6PXVXGYZ4DOFXWWE6UDQ4CEFTQHJA, seg_pos={0 0}, upload=488441]: Long tail cancellation due to optimal threshold Dec 28 20:37:29.763098996 (557h8m29.634182031s) [seg_pos={0 0}, upload=488441]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 20:37:29.763231739 (557h8m29.634314774s) [upload=488441]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 20:37:29.763236939 (557h8m29.634319984s) [upload=488441]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 20:37:30.055521760 (557h8m29.926604805s) [upload=488441]: Done uploading segments... Dec 28 20:37:30.055544604 (557h8m29.926627639s) [upload=488441]: Upload finished. err: <nil> Duration: 6.179497444s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [......................X...........X..................X......] duration: 24.396574929s trace: https://jaeger.dev.storj.io/trace/60ddd33e73f0de61 FAIL (2025-12-28 20:41:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 065810223s) [upload=488441]: Uploading segments... Dec 28 20:37:27.194835273 (557h8m27.065918308s) [upload=488441]: Got next segment. Inline: false Dec 28 20:37:27.194877263 (557h8m27.065960308s) [upload=488441]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 20:37:27.194884157 (557h8m27.065967212s) [upload=488441]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 20:37:27.450350479 (557h8m27.321433524s) [seg_pos={0 0}, upload=488441]: Begin upload segment... Dec 28 20:37:27.450441583 (557h8m27.321524628s) [seg_pos={0 0}, upload=488441]: Done begin upload segment. Dec 28 20:37:27.450581259 (557h8m27.321664304s) [noise=true , address=145.79.227.2:4114 , piece_id=MTU5VYRZVWYYP44THTSGDURQ3WHCCNH2C3XJJYQLUD7JBXUU5NZQ, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.450619542 (557h8m27.321702597s) [noise=true , address=45.91.116.6:4975 , piece_id=DCEIFZV6754QAYC5N2SXZ5DIYC7NR6TMHCXD66T4A5JRL7E4LPHA, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.451283105 (557h8m27.32236617s) [noise=true , address=145.79.227.51:4410 , piece_id=HGHWLMHV7LMCQGNQGSFRXUGJUVOIPYBPT3XATCP4EBZ2N24YW5LA, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.451709617 (557h8m27.322792662s) [noise=true , address=145.79.227.14:4840 , piece_id=VVOFKZVGGDZG6E75IVAUFUGSFNAMN5STZIPSURJ64JPGOWY6BJZQ, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.452173731 (557h8m27.323256766s) [noise=true , address=145.79.227.6:4257 , piece_id=DSINVYJEDHKBN7OGTQZM34CT6MBFUZMALJZQFOGTS5CLTSZLD4CA, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.452596105 (557h8m27.32367914s) [noise=true , address=145.79.227.32:4835 , piece_id=NIDZUDIX6K564WEVU7QDG2DH76GODGMLOJVUVLH6GS7GMUV45D7A, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.453086699 (557h8m27.324169734s) [noise=true , address=145.79.227.11:4731 , piece_id=R3RGSPD3B7K72ZH4AWXSR6K6D5KYZYLXIOM6QB346S4XZOGKR24A, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.453622139 (557h8m27.324705184s) [noise=true , address=145.79.227.15:4873 , piece_id=2MYCI55SJDFGKGRPUC2BLKB4XXLTFHCNH65JEV6KADXDBJUNBH3Q, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.455648808 (557h8m27.326731853s) [noise=true , address=145.79.227.5:4222 , piece_id=IBMJHO2KIDNN5AIIEGX5G5QVDJXXGLOTAMOQC4LZDUBBGY5E6GLQ, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.456606212 (557h8m27.327689247s) [noise=true , address=45.91.116.7:4012 , piece_id=C4J7O54YJEGULQT2ZWHPSWTZUU2LU2MN2ND2HSTJI2E6F7OHJSRQ, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.457955700 (557h8m27.329038735s) [noise=true , address=145.79.227.37:4565 , piece_id=3EJIDJFRGBG6272N24VFWNAXKUW7DHHM563DGJTMRURHA27CEREQ, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.459219468 (557h8m27.330302513s) [noise=true , address=145.79.227.27:4690 , piece_id=SZGKSH5YEIQBDSIMVFXRL2X6PXVXGYZ4DOFXWWE6UDQ4CEFTQHJA, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.460412208 (557h8m27.331495254s) [noise=true , address=77.90.31.16:4736 , piece_id=6NKLMJS4Z4DRFNCOYKO36LAI4EUWFB7LVXEYFDUYUMKSX6R235TQ, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.461658171 (557h8m27.332741206s) [noise=true , address=145.79.227.26:4654 , piece_id=XZZEQ7Y3SLK6WJJ22HH5RUTABJIYTONKTITYKENJXREOXSNSDHIA, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.462997892 (557h8m27.334080927s) [noise=true , address=45.91.116.5:4299 , piece_id=6UHFIECEOY2EGUKVFY2T5STZWL4DXXFP3NY4WVZKEEIREZ7ZVX2A, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.465119382 (557h8m27.336202427s) [noise=true , address=45.91.116.4:4939 , piece_id=XQHY6GBPJ6XEOFL2FTOFNBRLD3GF6VLGUBWY2D7TUOPQSFERNQTA, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:29.157770565 (557h8m29.02885361s) [upload=488441]: Next returned nil segment Dec 28 20:37:29.157787036 (557h8m29.028870091s) [upload=488441]: Waiting for error group managing segments... Dec 28 20:37:29.455180586 (557h8m29.326263631s) [noise=true , address=145.79.227.26:4654 , piece_id=XZZEQ7Y3SLK6WJJ22HH5RUTABJIYTONKTITYKENJXREOXSNSDHIA, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.457221773 (557h8m29.328304808s) [noise=true , address=145.79.227.6:4257 , piece_id=DSINVYJEDHKBN7OGTQZM34CT6MBFUZMALJZQFOGTS5CLTSZLD4CA, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.458700027 (557h8m29.329783062s) [noise=true , address=145.79.227.11:4731 , piece_id=R3RGSPD3B7K72ZH4AWXSR6K6D5KYZYLXIOM6QB346S4XZOGKR24A, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.459240897 (557h8m29.330323933s) [noise=true , address=145.79.227.51:4410 , piece_id=HGHWLMHV7LMCQGNQGSFRXUGJUVOIPYBPT3XATCP4EBZ2N24YW5LA, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.462155848 (557h8m29.333238883s) [noise=true , address=145.79.227.15:4873 , piece_id=2MYCI55SJDFGKGRPUC2BLKB4XXLTFHCNH65JEV6KADXDBJUNBH3Q, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.465572805 (557h8m29.33665585s) [noise=true , address=145.79.227.32:4835 , piece_id=NIDZUDIX6K564WEVU7QDG2DH76GODGMLOJVUVLH6GS7GMUV45D7A, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.470903064 (557h8m29.341986109s) [noise=true , address=145.79.227.2:4114 , piece_id=MTU5VYRZVWYYP44THTSGDURQ3WHCCNH2C3XJJYQLUD7JBXUU5NZQ, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.485295701 (557h8m29.356378736s) [noise=true , address=145.79.227.37:4565 , piece_id=3EJIDJFRGBG6272N24VFWNAXKUW7DHHM563DGJTMRURHA27CEREQ, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.497253901 (557h8m29.368336936s) [noise=true , address=145.79.227.14:4840 , piece_id=VVOFKZVGGDZG6E75IVAUFUGSFNAMN5STZIPSURJ64JPGOWY6BJZQ, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.641989392 (557h8m29.513072437s) [noise=true , address=77.90.31.16:4736 , piece_id=6NKLMJS4Z4DRFNCOYKO36LAI4EUWFB7LVXEYFDUYUMKSX6R235TQ, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.701668272 (557h8m29.572751307s) [noise=true , address=45.91.116.5:4299 , piece_id=6UHFIECEOY2EGUKVFY2T5STZWL4DXXFP3NY4WVZKEEIREZ7ZVX2A, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.702897923 (557h8m29.573980958s) [noise=true , address=45.91.116.6:4975 , piece_id=DCEIFZV6754QAYC5N2SXZ5DIYC7NR6TMHCXD66T4A5JRL7E4LPHA, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.762609574 (557h8m29.633692619s) [noise=true , address=45.91.116.7:4012 , piece_id=C4J7O54YJEGULQT2ZWHPSWTZUU2LU2MN2ND2HSTJI2E6F7OHJSRQ, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.762632388 (557h8m29.633715433s) [seg_pos={0 0}, upload=488441]: Segment reached optimal threshold of 13 pieces. Dec 28 20:37:29.762867626 (557h8m29.633950671s) [noise=true , address=145.79.227.5:4222 , piece_id=IBMJHO2KIDNN5AIIEGX5G5QVDJXXGLOTAMOQC4LZDUBBGY5E6GLQ, seg_pos={0 0}, upload=488441]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1): protocol: write tcp 188.166.225.170:44926->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:44926->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44926->145.79.227.5:4222: use of closed network connection Dec 28 20:37:29.762890609 (557h8m29.633973644s) [noise=true , address=145.79.227.5:4222 , piece_id=IBMJHO2KIDNN5AIIEGX5G5QVDJXXGLOTAMOQC4LZDUBBGY5E6GLQ, seg_pos={0 0}, upload=488441]: Long tail cancellation due to optimal threshold Dec 28 20:37:29.762944091 (557h8m29.634027126s) [noise=true , address=45.91.116.4:4939 , piece_id=XQHY6GBPJ6XEOFL2FTOFNBRLD3GF6VLGUBWY2D7TUOPQSFERNQTA, seg_pos={0 0}, upload=488441]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 28 20:37:29.762949461 (557h8m29.634032496s) [noise=true , address=45.91.116.4:4939 , piece_id=XQHY6GBPJ6XEOFL2FTOFNBRLD3GF6VLGUBWY2D7TUOPQSFERNQTA, seg_pos={0 0}, upload=488441]: Long tail cancellation due to optimal threshold Dec 28 20:37:29.763074540 (557h8m29.634157585s) [noise=true , address=145.79.227.27:4690 , piece_id=SZGKSH5YEIQBDSIMVFXRL2X6PXVXGYZ4DOFXWWE6UDQ4CEFTQHJA, seg_pos={0 0}, upload=488441]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H): protocol: write tcp 188.166.225.170:54746->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:54746->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54746->145.79.227.27:4690: use of closed network connection Dec 28 20:37:29.763081002 (557h8m29.634164037s) [noise=true , address=145.79.227.27:4690 , piece_id=SZGKSH5YEIQBDSIMVFXRL2X6PXVXGYZ4DOFXWWE6UDQ4CEFTQHJA, seg_pos={0 0}, upload=488441]: Long tail cancellation due to optimal threshold Dec 28 20:37:29.763098996 (557h8m29.634182031s) [seg_pos={0 0}, upload=488441]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 20:37:29.763231739 (557h8m29.634314774s) [upload=488441]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 20:37:29.763236939 (557h8m29.634319984s) [upload=488441]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 20:37:30.055521760 (557h8m29.926604805s) [upload=488441]: Done uploading segments... Dec 28 20:37:30.055544604 (557h8m29.926627639s) [upload=488441]: Upload finished. err: <nil> Duration: 24.396594566s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 25.359651363s trace: https://jaeger.dev.storj.io/trace/499b96822b8dda80 FAIL (2025-12-28 20:41:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12S7WBBQ1Nq8jbeGrZiRKTk23CdJC31hPW2vU1aaM5SsHeVyeRH, Piece ID: V4P3KCUDLI543TSROBYUMVTQKKSQDDVSKSJSXKEZTQAHTXUJ2G4Q) context deadline exceeded; piecestore: (Node ID: 1kjCEPtjvcJ3D7BBKRSRirPybBwkRZ1dJwyAQCadzCvaLYvWhz, Piece ID: 4OULHJ6Z3QJ3BOPDY74TJSKVJG23TOEUCJCUEGY24I5C3MUTAVLQ) context deadline exceeded; piecestore: (Node ID: 1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c, Piece ID: LIYXCPAZOUPKXLKRH3EOWREXUY2LAJXSRZFQILM2QIU7HTW7SY2Q) context deadline exceeded ======LOGS====== ue , address=145.79.227.23:4580 , piece_id=OHYI2SFM6TKJE4LHVNGO2PVGZDNFIQDAIQGKG4FOEHTTASBWA5ZQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.496428825 (557h3m1.36751187s) [noise=true , address=77.90.31.19:4867 , piece_id=LDAWD2ZFG3BZATDRY7MMNWYIX6TOHQ2OIUPCAWAYN266GNUUKZ5A, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.496861509 (557h3m1.367944544s) [noise=true , address=145.79.227.19:4105 , piece_id=RY35FSM6NTWPNF3JE43FZRSNXXYZPUN2B6LRNNUOG6QQR6XSU36Q, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.496899933 (557h3m1.367982978s) [noise=true , address=77.90.31.19:4858 , piece_id=LIYXCPAZOUPKXLKRH3EOWREXUY2LAJXSRZFQILM2QIU7HTW7SY2Q, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497229089 (557h3m1.368312134s) [noise=true , address=77.90.31.17:4784 , piece_id=JV2ROD6MN65B6LZKOL3S7ATJWXAY6FBQSZWITA6IHAXELY7R42KA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497326194 (557h3m1.368409239s) [noise=true , address=145.79.227.33:4904 , piece_id=QCMN4OSBWIWS4JRLIGQIGDSHZFUF2JWEPVXQ4FQ4G3LG5FBCBAWQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497664128 (557h3m1.368747173s) [noise=true , address=145.79.227.34:4918 , piece_id=SPDT2KI2774UJMQIUSVTHXCDF7PD2FBG64BXUYW67UMLPE6H7SBQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497731405 (557h3m1.36881445s) [noise=true , address=77.90.31.12:4602 , piece_id=2YPYKKTQ6ID6VTI4AUZQEG2SY7JYPRSAPU4JLF7DWSOSBBO74LWQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.498205939 (557h3m1.369288984s) [noise=true , address=77.90.31.16:4745 , piece_id=CZBFU33BHMP2BEZIDJKW3J526537HR6HP7OCT2KMAQVY7LGZLPZQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.498424295 (557h3m1.36950734s) [noise=true , address=77.90.31.21:4895 , piece_id=V4P3KCUDLI543TSROBYUMVTQKKSQDDVSKSJSXKEZTQAHTXUJ2G4Q, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.498685422 (557h3m1.369768457s) [noise=true , address=145.79.227.35:4496 , piece_id=JH4XQBT7XYEJXWBGPPF6SJWCPLV77GBOLGOA53C3WUQMWHZXVZQA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499066348 (557h3m1.370149393s) [noise=true , address=77.90.31.11:4563 , piece_id=DIFJPHSKHROTLKCHP6HQBWT7XGE77HVRZTH2354MV2UEWZLHVGCQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499185725 (557h3m1.37026877s) [noise=true , address=145.79.227.30:4768 , piece_id=KK4LEV6RG3IGLYXID5MJF3KMNSJ7WS5JS45PNOOJ26LXLYAUCWOA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499468122 (557h3m1.370551157s) [noise=true , address=77.90.31.14:4688 , piece_id=4OULHJ6Z3QJ3BOPDY74TJSKVJG23TOEUCJCUEGY24I5C3MUTAVLQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499901047 (557h3m1.370984102s) [noise=true , address=45.91.116.5:4330 , piece_id=EVHLSVVR5JGY43TLSKVEQLQ6KDBEZENVAFL6YT7ZQ77Y7X2TXIKA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.500546175 (557h3m1.37162921s) [noise=true , address=145.79.227.43:4171 , piece_id=YF5DRDPGRRVYFL5SWMH2PKTSV3C72BIWKUDSAIZTXOIVOTYKRMNA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:02.979572381 (557h3m2.850655436s) [upload=488362]: Next returned nil segment Dec 28 20:32:02.979609411 (557h3m2.850692456s) [upload=488362]: Waiting for error group managing segments... Dec 28 20:32:03.416896733 (557h3m3.287979778s) [noise=true , address=145.79.227.23:4580 , piece_id=OHYI2SFM6TKJE4LHVNGO2PVGZDNFIQDAIQGKG4FOEHTTASBWA5ZQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.552263655 (557h3m3.4233467s) [noise=true , address=77.90.31.17:4784 , piece_id=JV2ROD6MN65B6LZKOL3S7ATJWXAY6FBQSZWITA6IHAXELY7R42KA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.573564856 (557h3m3.444647902s) [noise=true , address=77.90.31.12:4602 , piece_id=2YPYKKTQ6ID6VTI4AUZQEG2SY7JYPRSAPU4JLF7DWSOSBBO74LWQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.598560923 (557h3m3.469643968s) [noise=true , address=77.90.31.19:4867 , piece_id=LDAWD2ZFG3BZATDRY7MMNWYIX6TOHQ2OIUPCAWAYN266GNUUKZ5A, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.626925925 (557h3m3.49800897s) [noise=true , address=77.90.31.16:4745 , piece_id=CZBFU33BHMP2BEZIDJKW3J526537HR6HP7OCT2KMAQVY7LGZLPZQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.709061947 (557h3m3.580144992s) [noise=true , address=77.90.31.14:4688 , piece_id=4OULHJ6Z3QJ3BOPDY74TJSKVJG23TOEUCJCUEGY24I5C3MUTAVLQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.788879373 (557h3m3.659962418s) [noise=true , address=77.90.31.21:4895 , piece_id=V4P3KCUDLI543TSROBYUMVTQKKSQDDVSKSJSXKEZTQAHTXUJ2G4Q, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.217616523 (557h3m4.088699569s) [noise=true , address=77.90.31.19:4858 , piece_id=LIYXCPAZOUPKXLKRH3EOWREXUY2LAJXSRZFQILM2QIU7HTW7SY2Q, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.220337625 (557h3m4.09142067s) [noise=true , address=77.90.31.11:4563 , piece_id=DIFJPHSKHROTLKCHP6HQBWT7XGE77HVRZTH2354MV2UEWZLHVGCQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.303958744 (557h3m4.175041789s) [noise=true , address=145.79.227.44:4213 , piece_id=UEAQDBBCP7DIG6ZD3EEM2CKAL4W5QKW23LWFUAMSVUW5ZS2PJDRQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.312794398 (557h3m4.183877444s) [noise=true , address=145.79.227.34:4918 , piece_id=SPDT2KI2774UJMQIUSVTHXCDF7PD2FBG64BXUYW67UMLPE6H7SBQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.349308798 (557h3m4.220391833s) [noise=true , address=145.79.227.40:4058 , piece_id=EQ4LA7T2BCGWY76K3NJI4X77OQJFJED6H4VNGXDQMILU6IVQWOMQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.371518379 (557h3m4.242601424s) [noise=true , address=145.79.227.43:4171 , piece_id=YF5DRDPGRRVYFL5SWMH2PKTSV3C72BIWKUDSAIZTXOIVOTYKRMNA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.376232316 (557h3m4.247315361s) [noise=true , address=145.79.227.30:4768 , piece_id=KK4LEV6RG3IGLYXID5MJF3KMNSJ7WS5JS45PNOOJ26LXLYAUCWOA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.376257844 (557h3m4.247340879s) [seg_pos={0 0}, upload=488362]: Segment reached optimal threshold of 14 pieces. Dec 28 20:32:04.376575799 (557h3m4.247658834s) [noise=true , address=145.79.227.33:4904 , piece_id=QCMN4OSBWIWS4JRLIGQIGDSHZFUF2JWEPVXQ4FQ4G3LG5FBCBAWQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1dC7mC2Cbxmys7ukEkELyF9Us7F1qtptSP6yDJh1EfjVJ6uqsC): protocol: expected piece hash; context canceled; EOF Dec 28 20:32:04.376585127 (557h3m4.247668163s) [noise=true , address=145.79.227.33:4904 , piece_id=QCMN4OSBWIWS4JRLIGQIGDSHZFUF2JWEPVXQ4FQ4G3LG5FBCBAWQ, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.376764359 (557h3m4.247847404s) [noise=true , address=145.79.227.19:4105 , piece_id=RY35FSM6NTWPNF3JE43FZRSNXXYZPUN2B6LRNNUOG6QQR6XSU36Q, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GhDrKMcDEg2RxamqKtiHphpgNdvtZYdHXrjzGsNJFjBFYeEen): protocol: write tcp 188.166.225.170:38838->145.79.227.19:4105: use of closed network connection; write tcp 188.166.225.170:38838->145.79.227.19:4105: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38838->145.79.227.19:4105: use of closed network connection Dec 28 20:32:04.376772865 (557h3m4.2478559s) [noise=true , address=145.79.227.19:4105 , piece_id=RY35FSM6NTWPNF3JE43FZRSNXXYZPUN2B6LRNNUOG6QQR6XSU36Q, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.376926658 (557h3m4.248009703s) [noise=true , address=145.79.227.35:4496 , piece_id=JH4XQBT7XYEJXWBGPPF6SJWCPLV77GBOLGOA53C3WUQMWHZXVZQA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V8XhSWkyGUVYFR8MykifFAP1YAfGCwzqucDcMMfzw3xsVW77Z): protocol: expected piece hash; context canceled; EOF Dec 28 20:32:04.377165332 (557h3m4.248248377s) [noise=true , address=145.79.227.35:4496 , piece_id=JH4XQBT7XYEJXWBGPPF6SJWCPLV77GBOLGOA53C3WUQMWHZXVZQA, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.377212682 (557h3m4.248295848s) [noise=true , address=45.91.116.5:4330 , piece_id=EVHLSVVR5JGY43TLSKVEQLQ6KDBEZENVAFL6YT7ZQ77Y7X2TXIKA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12MK9aVDRwfWPfckFR6RfPUL1Nrm9bwfsgV7TXCKGbtadHebWvW): protocol: read tcp 188.166.225.170:55416->45.91.116.5:4330: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 20:32:04.377289859 (557h3m4.248372924s) [noise=true , address=45.91.116.5:4330 , piece_id=EVHLSVVR5JGY43TLSKVEQLQ6KDBEZENVAFL6YT7ZQ77Y7X2TXIKA, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.377387736 (557h3m4.248470781s) [seg_pos={0 0}, upload=488362]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 20:32:04.377599729 (557h3m4.248682774s) [upload=488362]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 20:32:04.377613535 (557h3m4.24869662s) [upload=488362]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 20:32:04.675090855 (557h3m4.5461739s) [upload=488362]: Done uploading segments... Dec 28 20:32:04.675121233 (557h3m4.546204268s) [upload=488362]: Upload finished. err: <nil> Duration: 25.359679516s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 36.926128315s trace: https://jaeger.dev.storj.io/trace/499b96822b8dda80 FAIL (2025-12-28 20:41:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12S7WBBQ1Nq8jbeGrZiRKTk23CdJC31hPW2vU1aaM5SsHeVyeRH, Piece ID: V4P3KCUDLI543TSROBYUMVTQKKSQDDVSKSJSXKEZTQAHTXUJ2G4Q) context deadline exceeded; piecestore: (Node ID: 1kjCEPtjvcJ3D7BBKRSRirPybBwkRZ1dJwyAQCadzCvaLYvWhz, Piece ID: 4OULHJ6Z3QJ3BOPDY74TJSKVJG23TOEUCJCUEGY24I5C3MUTAVLQ) context deadline exceeded; piecestore: (Node ID: 1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c, Piece ID: LIYXCPAZOUPKXLKRH3EOWREXUY2LAJXSRZFQILM2QIU7HTW7SY2Q) context deadline exceeded ======LOGS====== ue , address=145.79.227.23:4580 , piece_id=OHYI2SFM6TKJE4LHVNGO2PVGZDNFIQDAIQGKG4FOEHTTASBWA5ZQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.496428825 (557h3m1.36751187s) [noise=true , address=77.90.31.19:4867 , piece_id=LDAWD2ZFG3BZATDRY7MMNWYIX6TOHQ2OIUPCAWAYN266GNUUKZ5A, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.496861509 (557h3m1.367944544s) [noise=true , address=145.79.227.19:4105 , piece_id=RY35FSM6NTWPNF3JE43FZRSNXXYZPUN2B6LRNNUOG6QQR6XSU36Q, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.496899933 (557h3m1.367982978s) [noise=true , address=77.90.31.19:4858 , piece_id=LIYXCPAZOUPKXLKRH3EOWREXUY2LAJXSRZFQILM2QIU7HTW7SY2Q, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497229089 (557h3m1.368312134s) [noise=true , address=77.90.31.17:4784 , piece_id=JV2ROD6MN65B6LZKOL3S7ATJWXAY6FBQSZWITA6IHAXELY7R42KA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497326194 (557h3m1.368409239s) [noise=true , address=145.79.227.33:4904 , piece_id=QCMN4OSBWIWS4JRLIGQIGDSHZFUF2JWEPVXQ4FQ4G3LG5FBCBAWQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497664128 (557h3m1.368747173s) [noise=true , address=145.79.227.34:4918 , piece_id=SPDT2KI2774UJMQIUSVTHXCDF7PD2FBG64BXUYW67UMLPE6H7SBQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497731405 (557h3m1.36881445s) [noise=true , address=77.90.31.12:4602 , piece_id=2YPYKKTQ6ID6VTI4AUZQEG2SY7JYPRSAPU4JLF7DWSOSBBO74LWQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.498205939 (557h3m1.369288984s) [noise=true , address=77.90.31.16:4745 , piece_id=CZBFU33BHMP2BEZIDJKW3J526537HR6HP7OCT2KMAQVY7LGZLPZQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.498424295 (557h3m1.36950734s) [noise=true , address=77.90.31.21:4895 , piece_id=V4P3KCUDLI543TSROBYUMVTQKKSQDDVSKSJSXKEZTQAHTXUJ2G4Q, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.498685422 (557h3m1.369768457s) [noise=true , address=145.79.227.35:4496 , piece_id=JH4XQBT7XYEJXWBGPPF6SJWCPLV77GBOLGOA53C3WUQMWHZXVZQA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499066348 (557h3m1.370149393s) [noise=true , address=77.90.31.11:4563 , piece_id=DIFJPHSKHROTLKCHP6HQBWT7XGE77HVRZTH2354MV2UEWZLHVGCQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499185725 (557h3m1.37026877s) [noise=true , address=145.79.227.30:4768 , piece_id=KK4LEV6RG3IGLYXID5MJF3KMNSJ7WS5JS45PNOOJ26LXLYAUCWOA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499468122 (557h3m1.370551157s) [noise=true , address=77.90.31.14:4688 , piece_id=4OULHJ6Z3QJ3BOPDY74TJSKVJG23TOEUCJCUEGY24I5C3MUTAVLQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499901047 (557h3m1.370984102s) [noise=true , address=45.91.116.5:4330 , piece_id=EVHLSVVR5JGY43TLSKVEQLQ6KDBEZENVAFL6YT7ZQ77Y7X2TXIKA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.500546175 (557h3m1.37162921s) [noise=true , address=145.79.227.43:4171 , piece_id=YF5DRDPGRRVYFL5SWMH2PKTSV3C72BIWKUDSAIZTXOIVOTYKRMNA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:02.979572381 (557h3m2.850655436s) [upload=488362]: Next returned nil segment Dec 28 20:32:02.979609411 (557h3m2.850692456s) [upload=488362]: Waiting for error group managing segments... Dec 28 20:32:03.416896733 (557h3m3.287979778s) [noise=true , address=145.79.227.23:4580 , piece_id=OHYI2SFM6TKJE4LHVNGO2PVGZDNFIQDAIQGKG4FOEHTTASBWA5ZQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.552263655 (557h3m3.4233467s) [noise=true , address=77.90.31.17:4784 , piece_id=JV2ROD6MN65B6LZKOL3S7ATJWXAY6FBQSZWITA6IHAXELY7R42KA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.573564856 (557h3m3.444647902s) [noise=true , address=77.90.31.12:4602 , piece_id=2YPYKKTQ6ID6VTI4AUZQEG2SY7JYPRSAPU4JLF7DWSOSBBO74LWQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.598560923 (557h3m3.469643968s) [noise=true , address=77.90.31.19:4867 , piece_id=LDAWD2ZFG3BZATDRY7MMNWYIX6TOHQ2OIUPCAWAYN266GNUUKZ5A, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.626925925 (557h3m3.49800897s) [noise=true , address=77.90.31.16:4745 , piece_id=CZBFU33BHMP2BEZIDJKW3J526537HR6HP7OCT2KMAQVY7LGZLPZQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.709061947 (557h3m3.580144992s) [noise=true , address=77.90.31.14:4688 , piece_id=4OULHJ6Z3QJ3BOPDY74TJSKVJG23TOEUCJCUEGY24I5C3MUTAVLQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.788879373 (557h3m3.659962418s) [noise=true , address=77.90.31.21:4895 , piece_id=V4P3KCUDLI543TSROBYUMVTQKKSQDDVSKSJSXKEZTQAHTXUJ2G4Q, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.217616523 (557h3m4.088699569s) [noise=true , address=77.90.31.19:4858 , piece_id=LIYXCPAZOUPKXLKRH3EOWREXUY2LAJXSRZFQILM2QIU7HTW7SY2Q, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.220337625 (557h3m4.09142067s) [noise=true , address=77.90.31.11:4563 , piece_id=DIFJPHSKHROTLKCHP6HQBWT7XGE77HVRZTH2354MV2UEWZLHVGCQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.303958744 (557h3m4.175041789s) [noise=true , address=145.79.227.44:4213 , piece_id=UEAQDBBCP7DIG6ZD3EEM2CKAL4W5QKW23LWFUAMSVUW5ZS2PJDRQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.312794398 (557h3m4.183877444s) [noise=true , address=145.79.227.34:4918 , piece_id=SPDT2KI2774UJMQIUSVTHXCDF7PD2FBG64BXUYW67UMLPE6H7SBQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.349308798 (557h3m4.220391833s) [noise=true , address=145.79.227.40:4058 , piece_id=EQ4LA7T2BCGWY76K3NJI4X77OQJFJED6H4VNGXDQMILU6IVQWOMQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.371518379 (557h3m4.242601424s) [noise=true , address=145.79.227.43:4171 , piece_id=YF5DRDPGRRVYFL5SWMH2PKTSV3C72BIWKUDSAIZTXOIVOTYKRMNA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.376232316 (557h3m4.247315361s) [noise=true , address=145.79.227.30:4768 , piece_id=KK4LEV6RG3IGLYXID5MJF3KMNSJ7WS5JS45PNOOJ26LXLYAUCWOA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.376257844 (557h3m4.247340879s) [seg_pos={0 0}, upload=488362]: Segment reached optimal threshold of 14 pieces. Dec 28 20:32:04.376575799 (557h3m4.247658834s) [noise=true , address=145.79.227.33:4904 , piece_id=QCMN4OSBWIWS4JRLIGQIGDSHZFUF2JWEPVXQ4FQ4G3LG5FBCBAWQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1dC7mC2Cbxmys7ukEkELyF9Us7F1qtptSP6yDJh1EfjVJ6uqsC): protocol: expected piece hash; context canceled; EOF Dec 28 20:32:04.376585127 (557h3m4.247668163s) [noise=true , address=145.79.227.33:4904 , piece_id=QCMN4OSBWIWS4JRLIGQIGDSHZFUF2JWEPVXQ4FQ4G3LG5FBCBAWQ, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.376764359 (557h3m4.247847404s) [noise=true , address=145.79.227.19:4105 , piece_id=RY35FSM6NTWPNF3JE43FZRSNXXYZPUN2B6LRNNUOG6QQR6XSU36Q, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GhDrKMcDEg2RxamqKtiHphpgNdvtZYdHXrjzGsNJFjBFYeEen): protocol: write tcp 188.166.225.170:38838->145.79.227.19:4105: use of closed network connection; write tcp 188.166.225.170:38838->145.79.227.19:4105: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38838->145.79.227.19:4105: use of closed network connection Dec 28 20:32:04.376772865 (557h3m4.2478559s) [noise=true , address=145.79.227.19:4105 , piece_id=RY35FSM6NTWPNF3JE43FZRSNXXYZPUN2B6LRNNUOG6QQR6XSU36Q, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.376926658 (557h3m4.248009703s) [noise=true , address=145.79.227.35:4496 , piece_id=JH4XQBT7XYEJXWBGPPF6SJWCPLV77GBOLGOA53C3WUQMWHZXVZQA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V8XhSWkyGUVYFR8MykifFAP1YAfGCwzqucDcMMfzw3xsVW77Z): protocol: expected piece hash; context canceled; EOF Dec 28 20:32:04.377165332 (557h3m4.248248377s) [noise=true , address=145.79.227.35:4496 , piece_id=JH4XQBT7XYEJXWBGPPF6SJWCPLV77GBOLGOA53C3WUQMWHZXVZQA, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.377212682 (557h3m4.248295848s) [noise=true , address=45.91.116.5:4330 , piece_id=EVHLSVVR5JGY43TLSKVEQLQ6KDBEZENVAFL6YT7ZQ77Y7X2TXIKA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12MK9aVDRwfWPfckFR6RfPUL1Nrm9bwfsgV7TXCKGbtadHebWvW): protocol: read tcp 188.166.225.170:55416->45.91.116.5:4330: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 20:32:04.377289859 (557h3m4.248372924s) [noise=true , address=45.91.116.5:4330 , piece_id=EVHLSVVR5JGY43TLSKVEQLQ6KDBEZENVAFL6YT7ZQ77Y7X2TXIKA, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.377387736 (557h3m4.248470781s) [seg_pos={0 0}, upload=488362]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 20:32:04.377599729 (557h3m4.248682774s) [upload=488362]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 20:32:04.377613535 (557h3m4.24869662s) [upload=488362]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 20:32:04.675090855 (557h3m4.5461739s) [upload=488362]: Done uploading segments... Dec 28 20:32:04.675121233 (557h3m4.546204268s) [upload=488362]: Upload finished. err: <nil> Duration: 36.926158573s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......................X...........X..................X.....] duration: 7.059598399s trace: https://jaeger.dev.storj.io/trace/60ddd33e73f0de61 FAIL (2025-12-28 20:40:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 065810223s) [upload=488441]: Uploading segments... Dec 28 20:37:27.194835273 (557h8m27.065918308s) [upload=488441]: Got next segment. Inline: false Dec 28 20:37:27.194877263 (557h8m27.065960308s) [upload=488441]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 20:37:27.194884157 (557h8m27.065967212s) [upload=488441]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 20:37:27.450350479 (557h8m27.321433524s) [seg_pos={0 0}, upload=488441]: Begin upload segment... Dec 28 20:37:27.450441583 (557h8m27.321524628s) [seg_pos={0 0}, upload=488441]: Done begin upload segment. Dec 28 20:37:27.450581259 (557h8m27.321664304s) [noise=true , address=145.79.227.2:4114 , piece_id=MTU5VYRZVWYYP44THTSGDURQ3WHCCNH2C3XJJYQLUD7JBXUU5NZQ, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.450619542 (557h8m27.321702597s) [noise=true , address=45.91.116.6:4975 , piece_id=DCEIFZV6754QAYC5N2SXZ5DIYC7NR6TMHCXD66T4A5JRL7E4LPHA, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.451283105 (557h8m27.32236617s) [noise=true , address=145.79.227.51:4410 , piece_id=HGHWLMHV7LMCQGNQGSFRXUGJUVOIPYBPT3XATCP4EBZ2N24YW5LA, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.451709617 (557h8m27.322792662s) [noise=true , address=145.79.227.14:4840 , piece_id=VVOFKZVGGDZG6E75IVAUFUGSFNAMN5STZIPSURJ64JPGOWY6BJZQ, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.452173731 (557h8m27.323256766s) [noise=true , address=145.79.227.6:4257 , piece_id=DSINVYJEDHKBN7OGTQZM34CT6MBFUZMALJZQFOGTS5CLTSZLD4CA, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.452596105 (557h8m27.32367914s) [noise=true , address=145.79.227.32:4835 , piece_id=NIDZUDIX6K564WEVU7QDG2DH76GODGMLOJVUVLH6GS7GMUV45D7A, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.453086699 (557h8m27.324169734s) [noise=true , address=145.79.227.11:4731 , piece_id=R3RGSPD3B7K72ZH4AWXSR6K6D5KYZYLXIOM6QB346S4XZOGKR24A, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.453622139 (557h8m27.324705184s) [noise=true , address=145.79.227.15:4873 , piece_id=2MYCI55SJDFGKGRPUC2BLKB4XXLTFHCNH65JEV6KADXDBJUNBH3Q, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.455648808 (557h8m27.326731853s) [noise=true , address=145.79.227.5:4222 , piece_id=IBMJHO2KIDNN5AIIEGX5G5QVDJXXGLOTAMOQC4LZDUBBGY5E6GLQ, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.456606212 (557h8m27.327689247s) [noise=true , address=45.91.116.7:4012 , piece_id=C4J7O54YJEGULQT2ZWHPSWTZUU2LU2MN2ND2HSTJI2E6F7OHJSRQ, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.457955700 (557h8m27.329038735s) [noise=true , address=145.79.227.37:4565 , piece_id=3EJIDJFRGBG6272N24VFWNAXKUW7DHHM563DGJTMRURHA27CEREQ, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.459219468 (557h8m27.330302513s) [noise=true , address=145.79.227.27:4690 , piece_id=SZGKSH5YEIQBDSIMVFXRL2X6PXVXGYZ4DOFXWWE6UDQ4CEFTQHJA, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.460412208 (557h8m27.331495254s) [noise=true , address=77.90.31.16:4736 , piece_id=6NKLMJS4Z4DRFNCOYKO36LAI4EUWFB7LVXEYFDUYUMKSX6R235TQ, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.461658171 (557h8m27.332741206s) [noise=true , address=145.79.227.26:4654 , piece_id=XZZEQ7Y3SLK6WJJ22HH5RUTABJIYTONKTITYKENJXREOXSNSDHIA, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.462997892 (557h8m27.334080927s) [noise=true , address=45.91.116.5:4299 , piece_id=6UHFIECEOY2EGUKVFY2T5STZWL4DXXFP3NY4WVZKEEIREZ7ZVX2A, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.465119382 (557h8m27.336202427s) [noise=true , address=45.91.116.4:4939 , piece_id=XQHY6GBPJ6XEOFL2FTOFNBRLD3GF6VLGUBWY2D7TUOPQSFERNQTA, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:29.157770565 (557h8m29.02885361s) [upload=488441]: Next returned nil segment Dec 28 20:37:29.157787036 (557h8m29.028870091s) [upload=488441]: Waiting for error group managing segments... Dec 28 20:37:29.455180586 (557h8m29.326263631s) [noise=true , address=145.79.227.26:4654 , piece_id=XZZEQ7Y3SLK6WJJ22HH5RUTABJIYTONKTITYKENJXREOXSNSDHIA, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.457221773 (557h8m29.328304808s) [noise=true , address=145.79.227.6:4257 , piece_id=DSINVYJEDHKBN7OGTQZM34CT6MBFUZMALJZQFOGTS5CLTSZLD4CA, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.458700027 (557h8m29.329783062s) [noise=true , address=145.79.227.11:4731 , piece_id=R3RGSPD3B7K72ZH4AWXSR6K6D5KYZYLXIOM6QB346S4XZOGKR24A, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.459240897 (557h8m29.330323933s) [noise=true , address=145.79.227.51:4410 , piece_id=HGHWLMHV7LMCQGNQGSFRXUGJUVOIPYBPT3XATCP4EBZ2N24YW5LA, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.462155848 (557h8m29.333238883s) [noise=true , address=145.79.227.15:4873 , piece_id=2MYCI55SJDFGKGRPUC2BLKB4XXLTFHCNH65JEV6KADXDBJUNBH3Q, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.465572805 (557h8m29.33665585s) [noise=true , address=145.79.227.32:4835 , piece_id=NIDZUDIX6K564WEVU7QDG2DH76GODGMLOJVUVLH6GS7GMUV45D7A, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.470903064 (557h8m29.341986109s) [noise=true , address=145.79.227.2:4114 , piece_id=MTU5VYRZVWYYP44THTSGDURQ3WHCCNH2C3XJJYQLUD7JBXUU5NZQ, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.485295701 (557h8m29.356378736s) [noise=true , address=145.79.227.37:4565 , piece_id=3EJIDJFRGBG6272N24VFWNAXKUW7DHHM563DGJTMRURHA27CEREQ, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.497253901 (557h8m29.368336936s) [noise=true , address=145.79.227.14:4840 , piece_id=VVOFKZVGGDZG6E75IVAUFUGSFNAMN5STZIPSURJ64JPGOWY6BJZQ, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.641989392 (557h8m29.513072437s) [noise=true , address=77.90.31.16:4736 , piece_id=6NKLMJS4Z4DRFNCOYKO36LAI4EUWFB7LVXEYFDUYUMKSX6R235TQ, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.701668272 (557h8m29.572751307s) [noise=true , address=45.91.116.5:4299 , piece_id=6UHFIECEOY2EGUKVFY2T5STZWL4DXXFP3NY4WVZKEEIREZ7ZVX2A, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.702897923 (557h8m29.573980958s) [noise=true , address=45.91.116.6:4975 , piece_id=DCEIFZV6754QAYC5N2SXZ5DIYC7NR6TMHCXD66T4A5JRL7E4LPHA, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.762609574 (557h8m29.633692619s) [noise=true , address=45.91.116.7:4012 , piece_id=C4J7O54YJEGULQT2ZWHPSWTZUU2LU2MN2ND2HSTJI2E6F7OHJSRQ, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.762632388 (557h8m29.633715433s) [seg_pos={0 0}, upload=488441]: Segment reached optimal threshold of 13 pieces. Dec 28 20:37:29.762867626 (557h8m29.633950671s) [noise=true , address=145.79.227.5:4222 , piece_id=IBMJHO2KIDNN5AIIEGX5G5QVDJXXGLOTAMOQC4LZDUBBGY5E6GLQ, seg_pos={0 0}, upload=488441]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1): protocol: write tcp 188.166.225.170:44926->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:44926->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44926->145.79.227.5:4222: use of closed network connection Dec 28 20:37:29.762890609 (557h8m29.633973644s) [noise=true , address=145.79.227.5:4222 , piece_id=IBMJHO2KIDNN5AIIEGX5G5QVDJXXGLOTAMOQC4LZDUBBGY5E6GLQ, seg_pos={0 0}, upload=488441]: Long tail cancellation due to optimal threshold Dec 28 20:37:29.762944091 (557h8m29.634027126s) [noise=true , address=45.91.116.4:4939 , piece_id=XQHY6GBPJ6XEOFL2FTOFNBRLD3GF6VLGUBWY2D7TUOPQSFERNQTA, seg_pos={0 0}, upload=488441]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 28 20:37:29.762949461 (557h8m29.634032496s) [noise=true , address=45.91.116.4:4939 , piece_id=XQHY6GBPJ6XEOFL2FTOFNBRLD3GF6VLGUBWY2D7TUOPQSFERNQTA, seg_pos={0 0}, upload=488441]: Long tail cancellation due to optimal threshold Dec 28 20:37:29.763074540 (557h8m29.634157585s) [noise=true , address=145.79.227.27:4690 , piece_id=SZGKSH5YEIQBDSIMVFXRL2X6PXVXGYZ4DOFXWWE6UDQ4CEFTQHJA, seg_pos={0 0}, upload=488441]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H): protocol: write tcp 188.166.225.170:54746->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:54746->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54746->145.79.227.27:4690: use of closed network connection Dec 28 20:37:29.763081002 (557h8m29.634164037s) [noise=true , address=145.79.227.27:4690 , piece_id=SZGKSH5YEIQBDSIMVFXRL2X6PXVXGYZ4DOFXWWE6UDQ4CEFTQHJA, seg_pos={0 0}, upload=488441]: Long tail cancellation due to optimal threshold Dec 28 20:37:29.763098996 (557h8m29.634182031s) [seg_pos={0 0}, upload=488441]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 20:37:29.763231739 (557h8m29.634314774s) [upload=488441]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 20:37:29.763236939 (557h8m29.634319984s) [upload=488441]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 20:37:30.055521760 (557h8m29.926604805s) [upload=488441]: Done uploading segments... Dec 28 20:37:30.055544604 (557h8m29.926627639s) [upload=488441]: Upload finished. err: <nil> Duration: 7.059646561s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [........................X...........X..................X....] duration: 20.172161812s trace: https://jaeger.dev.storj.io/trace/60ddd33e73f0de61 FAIL (2025-12-28 20:40:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 065810223s) [upload=488441]: Uploading segments... Dec 28 20:37:27.194835273 (557h8m27.065918308s) [upload=488441]: Got next segment. Inline: false Dec 28 20:37:27.194877263 (557h8m27.065960308s) [upload=488441]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 20:37:27.194884157 (557h8m27.065967212s) [upload=488441]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 20:37:27.450350479 (557h8m27.321433524s) [seg_pos={0 0}, upload=488441]: Begin upload segment... Dec 28 20:37:27.450441583 (557h8m27.321524628s) [seg_pos={0 0}, upload=488441]: Done begin upload segment. Dec 28 20:37:27.450581259 (557h8m27.321664304s) [noise=true , address=145.79.227.2:4114 , piece_id=MTU5VYRZVWYYP44THTSGDURQ3WHCCNH2C3XJJYQLUD7JBXUU5NZQ, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.450619542 (557h8m27.321702597s) [noise=true , address=45.91.116.6:4975 , piece_id=DCEIFZV6754QAYC5N2SXZ5DIYC7NR6TMHCXD66T4A5JRL7E4LPHA, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.451283105 (557h8m27.32236617s) [noise=true , address=145.79.227.51:4410 , piece_id=HGHWLMHV7LMCQGNQGSFRXUGJUVOIPYBPT3XATCP4EBZ2N24YW5LA, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.451709617 (557h8m27.322792662s) [noise=true , address=145.79.227.14:4840 , piece_id=VVOFKZVGGDZG6E75IVAUFUGSFNAMN5STZIPSURJ64JPGOWY6BJZQ, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.452173731 (557h8m27.323256766s) [noise=true , address=145.79.227.6:4257 , piece_id=DSINVYJEDHKBN7OGTQZM34CT6MBFUZMALJZQFOGTS5CLTSZLD4CA, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.452596105 (557h8m27.32367914s) [noise=true , address=145.79.227.32:4835 , piece_id=NIDZUDIX6K564WEVU7QDG2DH76GODGMLOJVUVLH6GS7GMUV45D7A, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.453086699 (557h8m27.324169734s) [noise=true , address=145.79.227.11:4731 , piece_id=R3RGSPD3B7K72ZH4AWXSR6K6D5KYZYLXIOM6QB346S4XZOGKR24A, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.453622139 (557h8m27.324705184s) [noise=true , address=145.79.227.15:4873 , piece_id=2MYCI55SJDFGKGRPUC2BLKB4XXLTFHCNH65JEV6KADXDBJUNBH3Q, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.455648808 (557h8m27.326731853s) [noise=true , address=145.79.227.5:4222 , piece_id=IBMJHO2KIDNN5AIIEGX5G5QVDJXXGLOTAMOQC4LZDUBBGY5E6GLQ, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.456606212 (557h8m27.327689247s) [noise=true , address=45.91.116.7:4012 , piece_id=C4J7O54YJEGULQT2ZWHPSWTZUU2LU2MN2ND2HSTJI2E6F7OHJSRQ, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.457955700 (557h8m27.329038735s) [noise=true , address=145.79.227.37:4565 , piece_id=3EJIDJFRGBG6272N24VFWNAXKUW7DHHM563DGJTMRURHA27CEREQ, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.459219468 (557h8m27.330302513s) [noise=true , address=145.79.227.27:4690 , piece_id=SZGKSH5YEIQBDSIMVFXRL2X6PXVXGYZ4DOFXWWE6UDQ4CEFTQHJA, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.460412208 (557h8m27.331495254s) [noise=true , address=77.90.31.16:4736 , piece_id=6NKLMJS4Z4DRFNCOYKO36LAI4EUWFB7LVXEYFDUYUMKSX6R235TQ, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.461658171 (557h8m27.332741206s) [noise=true , address=145.79.227.26:4654 , piece_id=XZZEQ7Y3SLK6WJJ22HH5RUTABJIYTONKTITYKENJXREOXSNSDHIA, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.462997892 (557h8m27.334080927s) [noise=true , address=45.91.116.5:4299 , piece_id=6UHFIECEOY2EGUKVFY2T5STZWL4DXXFP3NY4WVZKEEIREZ7ZVX2A, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.465119382 (557h8m27.336202427s) [noise=true , address=45.91.116.4:4939 , piece_id=XQHY6GBPJ6XEOFL2FTOFNBRLD3GF6VLGUBWY2D7TUOPQSFERNQTA, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:29.157770565 (557h8m29.02885361s) [upload=488441]: Next returned nil segment Dec 28 20:37:29.157787036 (557h8m29.028870091s) [upload=488441]: Waiting for error group managing segments... Dec 28 20:37:29.455180586 (557h8m29.326263631s) [noise=true , address=145.79.227.26:4654 , piece_id=XZZEQ7Y3SLK6WJJ22HH5RUTABJIYTONKTITYKENJXREOXSNSDHIA, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.457221773 (557h8m29.328304808s) [noise=true , address=145.79.227.6:4257 , piece_id=DSINVYJEDHKBN7OGTQZM34CT6MBFUZMALJZQFOGTS5CLTSZLD4CA, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.458700027 (557h8m29.329783062s) [noise=true , address=145.79.227.11:4731 , piece_id=R3RGSPD3B7K72ZH4AWXSR6K6D5KYZYLXIOM6QB346S4XZOGKR24A, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.459240897 (557h8m29.330323933s) [noise=true , address=145.79.227.51:4410 , piece_id=HGHWLMHV7LMCQGNQGSFRXUGJUVOIPYBPT3XATCP4EBZ2N24YW5LA, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.462155848 (557h8m29.333238883s) [noise=true , address=145.79.227.15:4873 , piece_id=2MYCI55SJDFGKGRPUC2BLKB4XXLTFHCNH65JEV6KADXDBJUNBH3Q, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.465572805 (557h8m29.33665585s) [noise=true , address=145.79.227.32:4835 , piece_id=NIDZUDIX6K564WEVU7QDG2DH76GODGMLOJVUVLH6GS7GMUV45D7A, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.470903064 (557h8m29.341986109s) [noise=true , address=145.79.227.2:4114 , piece_id=MTU5VYRZVWYYP44THTSGDURQ3WHCCNH2C3XJJYQLUD7JBXUU5NZQ, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.485295701 (557h8m29.356378736s) [noise=true , address=145.79.227.37:4565 , piece_id=3EJIDJFRGBG6272N24VFWNAXKUW7DHHM563DGJTMRURHA27CEREQ, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.497253901 (557h8m29.368336936s) [noise=true , address=145.79.227.14:4840 , piece_id=VVOFKZVGGDZG6E75IVAUFUGSFNAMN5STZIPSURJ64JPGOWY6BJZQ, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.641989392 (557h8m29.513072437s) [noise=true , address=77.90.31.16:4736 , piece_id=6NKLMJS4Z4DRFNCOYKO36LAI4EUWFB7LVXEYFDUYUMKSX6R235TQ, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.701668272 (557h8m29.572751307s) [noise=true , address=45.91.116.5:4299 , piece_id=6UHFIECEOY2EGUKVFY2T5STZWL4DXXFP3NY4WVZKEEIREZ7ZVX2A, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.702897923 (557h8m29.573980958s) [noise=true , address=45.91.116.6:4975 , piece_id=DCEIFZV6754QAYC5N2SXZ5DIYC7NR6TMHCXD66T4A5JRL7E4LPHA, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.762609574 (557h8m29.633692619s) [noise=true , address=45.91.116.7:4012 , piece_id=C4J7O54YJEGULQT2ZWHPSWTZUU2LU2MN2ND2HSTJI2E6F7OHJSRQ, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.762632388 (557h8m29.633715433s) [seg_pos={0 0}, upload=488441]: Segment reached optimal threshold of 13 pieces. Dec 28 20:37:29.762867626 (557h8m29.633950671s) [noise=true , address=145.79.227.5:4222 , piece_id=IBMJHO2KIDNN5AIIEGX5G5QVDJXXGLOTAMOQC4LZDUBBGY5E6GLQ, seg_pos={0 0}, upload=488441]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1): protocol: write tcp 188.166.225.170:44926->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:44926->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44926->145.79.227.5:4222: use of closed network connection Dec 28 20:37:29.762890609 (557h8m29.633973644s) [noise=true , address=145.79.227.5:4222 , piece_id=IBMJHO2KIDNN5AIIEGX5G5QVDJXXGLOTAMOQC4LZDUBBGY5E6GLQ, seg_pos={0 0}, upload=488441]: Long tail cancellation due to optimal threshold Dec 28 20:37:29.762944091 (557h8m29.634027126s) [noise=true , address=45.91.116.4:4939 , piece_id=XQHY6GBPJ6XEOFL2FTOFNBRLD3GF6VLGUBWY2D7TUOPQSFERNQTA, seg_pos={0 0}, upload=488441]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 28 20:37:29.762949461 (557h8m29.634032496s) [noise=true , address=45.91.116.4:4939 , piece_id=XQHY6GBPJ6XEOFL2FTOFNBRLD3GF6VLGUBWY2D7TUOPQSFERNQTA, seg_pos={0 0}, upload=488441]: Long tail cancellation due to optimal threshold Dec 28 20:37:29.763074540 (557h8m29.634157585s) [noise=true , address=145.79.227.27:4690 , piece_id=SZGKSH5YEIQBDSIMVFXRL2X6PXVXGYZ4DOFXWWE6UDQ4CEFTQHJA, seg_pos={0 0}, upload=488441]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H): protocol: write tcp 188.166.225.170:54746->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:54746->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54746->145.79.227.27:4690: use of closed network connection Dec 28 20:37:29.763081002 (557h8m29.634164037s) [noise=true , address=145.79.227.27:4690 , piece_id=SZGKSH5YEIQBDSIMVFXRL2X6PXVXGYZ4DOFXWWE6UDQ4CEFTQHJA, seg_pos={0 0}, upload=488441]: Long tail cancellation due to optimal threshold Dec 28 20:37:29.763098996 (557h8m29.634182031s) [seg_pos={0 0}, upload=488441]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 20:37:29.763231739 (557h8m29.634314774s) [upload=488441]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 20:37:29.763236939 (557h8m29.634319984s) [upload=488441]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 20:37:30.055521760 (557h8m29.926604805s) [upload=488441]: Done uploading segments... Dec 28 20:37:30.055544604 (557h8m29.926627639s) [upload=488441]: Upload finished. err: <nil> Duration: 20.172191941s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 25.331743784s trace: https://jaeger.dev.storj.io/trace/499b96822b8dda80 FAIL (2025-12-28 20:39:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12S7WBBQ1Nq8jbeGrZiRKTk23CdJC31hPW2vU1aaM5SsHeVyeRH, Piece ID: V4P3KCUDLI543TSROBYUMVTQKKSQDDVSKSJSXKEZTQAHTXUJ2G4Q) context deadline exceeded; piecestore: (Node ID: 1kjCEPtjvcJ3D7BBKRSRirPybBwkRZ1dJwyAQCadzCvaLYvWhz, Piece ID: 4OULHJ6Z3QJ3BOPDY74TJSKVJG23TOEUCJCUEGY24I5C3MUTAVLQ) context deadline exceeded; piecestore: (Node ID: 1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c, Piece ID: LIYXCPAZOUPKXLKRH3EOWREXUY2LAJXSRZFQILM2QIU7HTW7SY2Q) context deadline exceeded ======LOGS====== ue , address=145.79.227.23:4580 , piece_id=OHYI2SFM6TKJE4LHVNGO2PVGZDNFIQDAIQGKG4FOEHTTASBWA5ZQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.496428825 (557h3m1.36751187s) [noise=true , address=77.90.31.19:4867 , piece_id=LDAWD2ZFG3BZATDRY7MMNWYIX6TOHQ2OIUPCAWAYN266GNUUKZ5A, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.496861509 (557h3m1.367944544s) [noise=true , address=145.79.227.19:4105 , piece_id=RY35FSM6NTWPNF3JE43FZRSNXXYZPUN2B6LRNNUOG6QQR6XSU36Q, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.496899933 (557h3m1.367982978s) [noise=true , address=77.90.31.19:4858 , piece_id=LIYXCPAZOUPKXLKRH3EOWREXUY2LAJXSRZFQILM2QIU7HTW7SY2Q, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497229089 (557h3m1.368312134s) [noise=true , address=77.90.31.17:4784 , piece_id=JV2ROD6MN65B6LZKOL3S7ATJWXAY6FBQSZWITA6IHAXELY7R42KA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497326194 (557h3m1.368409239s) [noise=true , address=145.79.227.33:4904 , piece_id=QCMN4OSBWIWS4JRLIGQIGDSHZFUF2JWEPVXQ4FQ4G3LG5FBCBAWQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497664128 (557h3m1.368747173s) [noise=true , address=145.79.227.34:4918 , piece_id=SPDT2KI2774UJMQIUSVTHXCDF7PD2FBG64BXUYW67UMLPE6H7SBQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497731405 (557h3m1.36881445s) [noise=true , address=77.90.31.12:4602 , piece_id=2YPYKKTQ6ID6VTI4AUZQEG2SY7JYPRSAPU4JLF7DWSOSBBO74LWQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.498205939 (557h3m1.369288984s) [noise=true , address=77.90.31.16:4745 , piece_id=CZBFU33BHMP2BEZIDJKW3J526537HR6HP7OCT2KMAQVY7LGZLPZQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.498424295 (557h3m1.36950734s) [noise=true , address=77.90.31.21:4895 , piece_id=V4P3KCUDLI543TSROBYUMVTQKKSQDDVSKSJSXKEZTQAHTXUJ2G4Q, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.498685422 (557h3m1.369768457s) [noise=true , address=145.79.227.35:4496 , piece_id=JH4XQBT7XYEJXWBGPPF6SJWCPLV77GBOLGOA53C3WUQMWHZXVZQA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499066348 (557h3m1.370149393s) [noise=true , address=77.90.31.11:4563 , piece_id=DIFJPHSKHROTLKCHP6HQBWT7XGE77HVRZTH2354MV2UEWZLHVGCQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499185725 (557h3m1.37026877s) [noise=true , address=145.79.227.30:4768 , piece_id=KK4LEV6RG3IGLYXID5MJF3KMNSJ7WS5JS45PNOOJ26LXLYAUCWOA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499468122 (557h3m1.370551157s) [noise=true , address=77.90.31.14:4688 , piece_id=4OULHJ6Z3QJ3BOPDY74TJSKVJG23TOEUCJCUEGY24I5C3MUTAVLQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499901047 (557h3m1.370984102s) [noise=true , address=45.91.116.5:4330 , piece_id=EVHLSVVR5JGY43TLSKVEQLQ6KDBEZENVAFL6YT7ZQ77Y7X2TXIKA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.500546175 (557h3m1.37162921s) [noise=true , address=145.79.227.43:4171 , piece_id=YF5DRDPGRRVYFL5SWMH2PKTSV3C72BIWKUDSAIZTXOIVOTYKRMNA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:02.979572381 (557h3m2.850655436s) [upload=488362]: Next returned nil segment Dec 28 20:32:02.979609411 (557h3m2.850692456s) [upload=488362]: Waiting for error group managing segments... Dec 28 20:32:03.416896733 (557h3m3.287979778s) [noise=true , address=145.79.227.23:4580 , piece_id=OHYI2SFM6TKJE4LHVNGO2PVGZDNFIQDAIQGKG4FOEHTTASBWA5ZQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.552263655 (557h3m3.4233467s) [noise=true , address=77.90.31.17:4784 , piece_id=JV2ROD6MN65B6LZKOL3S7ATJWXAY6FBQSZWITA6IHAXELY7R42KA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.573564856 (557h3m3.444647902s) [noise=true , address=77.90.31.12:4602 , piece_id=2YPYKKTQ6ID6VTI4AUZQEG2SY7JYPRSAPU4JLF7DWSOSBBO74LWQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.598560923 (557h3m3.469643968s) [noise=true , address=77.90.31.19:4867 , piece_id=LDAWD2ZFG3BZATDRY7MMNWYIX6TOHQ2OIUPCAWAYN266GNUUKZ5A, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.626925925 (557h3m3.49800897s) [noise=true , address=77.90.31.16:4745 , piece_id=CZBFU33BHMP2BEZIDJKW3J526537HR6HP7OCT2KMAQVY7LGZLPZQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.709061947 (557h3m3.580144992s) [noise=true , address=77.90.31.14:4688 , piece_id=4OULHJ6Z3QJ3BOPDY74TJSKVJG23TOEUCJCUEGY24I5C3MUTAVLQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.788879373 (557h3m3.659962418s) [noise=true , address=77.90.31.21:4895 , piece_id=V4P3KCUDLI543TSROBYUMVTQKKSQDDVSKSJSXKEZTQAHTXUJ2G4Q, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.217616523 (557h3m4.088699569s) [noise=true , address=77.90.31.19:4858 , piece_id=LIYXCPAZOUPKXLKRH3EOWREXUY2LAJXSRZFQILM2QIU7HTW7SY2Q, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.220337625 (557h3m4.09142067s) [noise=true , address=77.90.31.11:4563 , piece_id=DIFJPHSKHROTLKCHP6HQBWT7XGE77HVRZTH2354MV2UEWZLHVGCQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.303958744 (557h3m4.175041789s) [noise=true , address=145.79.227.44:4213 , piece_id=UEAQDBBCP7DIG6ZD3EEM2CKAL4W5QKW23LWFUAMSVUW5ZS2PJDRQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.312794398 (557h3m4.183877444s) [noise=true , address=145.79.227.34:4918 , piece_id=SPDT2KI2774UJMQIUSVTHXCDF7PD2FBG64BXUYW67UMLPE6H7SBQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.349308798 (557h3m4.220391833s) [noise=true , address=145.79.227.40:4058 , piece_id=EQ4LA7T2BCGWY76K3NJI4X77OQJFJED6H4VNGXDQMILU6IVQWOMQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.371518379 (557h3m4.242601424s) [noise=true , address=145.79.227.43:4171 , piece_id=YF5DRDPGRRVYFL5SWMH2PKTSV3C72BIWKUDSAIZTXOIVOTYKRMNA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.376232316 (557h3m4.247315361s) [noise=true , address=145.79.227.30:4768 , piece_id=KK4LEV6RG3IGLYXID5MJF3KMNSJ7WS5JS45PNOOJ26LXLYAUCWOA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.376257844 (557h3m4.247340879s) [seg_pos={0 0}, upload=488362]: Segment reached optimal threshold of 14 pieces. Dec 28 20:32:04.376575799 (557h3m4.247658834s) [noise=true , address=145.79.227.33:4904 , piece_id=QCMN4OSBWIWS4JRLIGQIGDSHZFUF2JWEPVXQ4FQ4G3LG5FBCBAWQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1dC7mC2Cbxmys7ukEkELyF9Us7F1qtptSP6yDJh1EfjVJ6uqsC): protocol: expected piece hash; context canceled; EOF Dec 28 20:32:04.376585127 (557h3m4.247668163s) [noise=true , address=145.79.227.33:4904 , piece_id=QCMN4OSBWIWS4JRLIGQIGDSHZFUF2JWEPVXQ4FQ4G3LG5FBCBAWQ, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.376764359 (557h3m4.247847404s) [noise=true , address=145.79.227.19:4105 , piece_id=RY35FSM6NTWPNF3JE43FZRSNXXYZPUN2B6LRNNUOG6QQR6XSU36Q, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GhDrKMcDEg2RxamqKtiHphpgNdvtZYdHXrjzGsNJFjBFYeEen): protocol: write tcp 188.166.225.170:38838->145.79.227.19:4105: use of closed network connection; write tcp 188.166.225.170:38838->145.79.227.19:4105: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38838->145.79.227.19:4105: use of closed network connection Dec 28 20:32:04.376772865 (557h3m4.2478559s) [noise=true , address=145.79.227.19:4105 , piece_id=RY35FSM6NTWPNF3JE43FZRSNXXYZPUN2B6LRNNUOG6QQR6XSU36Q, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.376926658 (557h3m4.248009703s) [noise=true , address=145.79.227.35:4496 , piece_id=JH4XQBT7XYEJXWBGPPF6SJWCPLV77GBOLGOA53C3WUQMWHZXVZQA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V8XhSWkyGUVYFR8MykifFAP1YAfGCwzqucDcMMfzw3xsVW77Z): protocol: expected piece hash; context canceled; EOF Dec 28 20:32:04.377165332 (557h3m4.248248377s) [noise=true , address=145.79.227.35:4496 , piece_id=JH4XQBT7XYEJXWBGPPF6SJWCPLV77GBOLGOA53C3WUQMWHZXVZQA, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.377212682 (557h3m4.248295848s) [noise=true , address=45.91.116.5:4330 , piece_id=EVHLSVVR5JGY43TLSKVEQLQ6KDBEZENVAFL6YT7ZQ77Y7X2TXIKA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12MK9aVDRwfWPfckFR6RfPUL1Nrm9bwfsgV7TXCKGbtadHebWvW): protocol: read tcp 188.166.225.170:55416->45.91.116.5:4330: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 20:32:04.377289859 (557h3m4.248372924s) [noise=true , address=45.91.116.5:4330 , piece_id=EVHLSVVR5JGY43TLSKVEQLQ6KDBEZENVAFL6YT7ZQ77Y7X2TXIKA, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.377387736 (557h3m4.248470781s) [seg_pos={0 0}, upload=488362]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 20:32:04.377599729 (557h3m4.248682774s) [upload=488362]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 20:32:04.377613535 (557h3m4.24869662s) [upload=488362]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 20:32:04.675090855 (557h3m4.5461739s) [upload=488362]: Done uploading segments... Dec 28 20:32:04.675121233 (557h3m4.546204268s) [upload=488362]: Upload finished. err: <nil> Duration: 25.331771536s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........................X...........X..................X...] duration: 6.416913838s trace: https://jaeger.dev.storj.io/trace/60ddd33e73f0de61 FAIL (2025-12-28 20:39:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 065810223s) [upload=488441]: Uploading segments... Dec 28 20:37:27.194835273 (557h8m27.065918308s) [upload=488441]: Got next segment. Inline: false Dec 28 20:37:27.194877263 (557h8m27.065960308s) [upload=488441]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 20:37:27.194884157 (557h8m27.065967212s) [upload=488441]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 20:37:27.450350479 (557h8m27.321433524s) [seg_pos={0 0}, upload=488441]: Begin upload segment... Dec 28 20:37:27.450441583 (557h8m27.321524628s) [seg_pos={0 0}, upload=488441]: Done begin upload segment. Dec 28 20:37:27.450581259 (557h8m27.321664304s) [noise=true , address=145.79.227.2:4114 , piece_id=MTU5VYRZVWYYP44THTSGDURQ3WHCCNH2C3XJJYQLUD7JBXUU5NZQ, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.450619542 (557h8m27.321702597s) [noise=true , address=45.91.116.6:4975 , piece_id=DCEIFZV6754QAYC5N2SXZ5DIYC7NR6TMHCXD66T4A5JRL7E4LPHA, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.451283105 (557h8m27.32236617s) [noise=true , address=145.79.227.51:4410 , piece_id=HGHWLMHV7LMCQGNQGSFRXUGJUVOIPYBPT3XATCP4EBZ2N24YW5LA, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.451709617 (557h8m27.322792662s) [noise=true , address=145.79.227.14:4840 , piece_id=VVOFKZVGGDZG6E75IVAUFUGSFNAMN5STZIPSURJ64JPGOWY6BJZQ, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.452173731 (557h8m27.323256766s) [noise=true , address=145.79.227.6:4257 , piece_id=DSINVYJEDHKBN7OGTQZM34CT6MBFUZMALJZQFOGTS5CLTSZLD4CA, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.452596105 (557h8m27.32367914s) [noise=true , address=145.79.227.32:4835 , piece_id=NIDZUDIX6K564WEVU7QDG2DH76GODGMLOJVUVLH6GS7GMUV45D7A, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.453086699 (557h8m27.324169734s) [noise=true , address=145.79.227.11:4731 , piece_id=R3RGSPD3B7K72ZH4AWXSR6K6D5KYZYLXIOM6QB346S4XZOGKR24A, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.453622139 (557h8m27.324705184s) [noise=true , address=145.79.227.15:4873 , piece_id=2MYCI55SJDFGKGRPUC2BLKB4XXLTFHCNH65JEV6KADXDBJUNBH3Q, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.455648808 (557h8m27.326731853s) [noise=true , address=145.79.227.5:4222 , piece_id=IBMJHO2KIDNN5AIIEGX5G5QVDJXXGLOTAMOQC4LZDUBBGY5E6GLQ, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.456606212 (557h8m27.327689247s) [noise=true , address=45.91.116.7:4012 , piece_id=C4J7O54YJEGULQT2ZWHPSWTZUU2LU2MN2ND2HSTJI2E6F7OHJSRQ, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.457955700 (557h8m27.329038735s) [noise=true , address=145.79.227.37:4565 , piece_id=3EJIDJFRGBG6272N24VFWNAXKUW7DHHM563DGJTMRURHA27CEREQ, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.459219468 (557h8m27.330302513s) [noise=true , address=145.79.227.27:4690 , piece_id=SZGKSH5YEIQBDSIMVFXRL2X6PXVXGYZ4DOFXWWE6UDQ4CEFTQHJA, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.460412208 (557h8m27.331495254s) [noise=true , address=77.90.31.16:4736 , piece_id=6NKLMJS4Z4DRFNCOYKO36LAI4EUWFB7LVXEYFDUYUMKSX6R235TQ, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.461658171 (557h8m27.332741206s) [noise=true , address=145.79.227.26:4654 , piece_id=XZZEQ7Y3SLK6WJJ22HH5RUTABJIYTONKTITYKENJXREOXSNSDHIA, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.462997892 (557h8m27.334080927s) [noise=true , address=45.91.116.5:4299 , piece_id=6UHFIECEOY2EGUKVFY2T5STZWL4DXXFP3NY4WVZKEEIREZ7ZVX2A, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.465119382 (557h8m27.336202427s) [noise=true , address=45.91.116.4:4939 , piece_id=XQHY6GBPJ6XEOFL2FTOFNBRLD3GF6VLGUBWY2D7TUOPQSFERNQTA, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:29.157770565 (557h8m29.02885361s) [upload=488441]: Next returned nil segment Dec 28 20:37:29.157787036 (557h8m29.028870091s) [upload=488441]: Waiting for error group managing segments... Dec 28 20:37:29.455180586 (557h8m29.326263631s) [noise=true , address=145.79.227.26:4654 , piece_id=XZZEQ7Y3SLK6WJJ22HH5RUTABJIYTONKTITYKENJXREOXSNSDHIA, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.457221773 (557h8m29.328304808s) [noise=true , address=145.79.227.6:4257 , piece_id=DSINVYJEDHKBN7OGTQZM34CT6MBFUZMALJZQFOGTS5CLTSZLD4CA, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.458700027 (557h8m29.329783062s) [noise=true , address=145.79.227.11:4731 , piece_id=R3RGSPD3B7K72ZH4AWXSR6K6D5KYZYLXIOM6QB346S4XZOGKR24A, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.459240897 (557h8m29.330323933s) [noise=true , address=145.79.227.51:4410 , piece_id=HGHWLMHV7LMCQGNQGSFRXUGJUVOIPYBPT3XATCP4EBZ2N24YW5LA, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.462155848 (557h8m29.333238883s) [noise=true , address=145.79.227.15:4873 , piece_id=2MYCI55SJDFGKGRPUC2BLKB4XXLTFHCNH65JEV6KADXDBJUNBH3Q, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.465572805 (557h8m29.33665585s) [noise=true , address=145.79.227.32:4835 , piece_id=NIDZUDIX6K564WEVU7QDG2DH76GODGMLOJVUVLH6GS7GMUV45D7A, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.470903064 (557h8m29.341986109s) [noise=true , address=145.79.227.2:4114 , piece_id=MTU5VYRZVWYYP44THTSGDURQ3WHCCNH2C3XJJYQLUD7JBXUU5NZQ, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.485295701 (557h8m29.356378736s) [noise=true , address=145.79.227.37:4565 , piece_id=3EJIDJFRGBG6272N24VFWNAXKUW7DHHM563DGJTMRURHA27CEREQ, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.497253901 (557h8m29.368336936s) [noise=true , address=145.79.227.14:4840 , piece_id=VVOFKZVGGDZG6E75IVAUFUGSFNAMN5STZIPSURJ64JPGOWY6BJZQ, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.641989392 (557h8m29.513072437s) [noise=true , address=77.90.31.16:4736 , piece_id=6NKLMJS4Z4DRFNCOYKO36LAI4EUWFB7LVXEYFDUYUMKSX6R235TQ, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.701668272 (557h8m29.572751307s) [noise=true , address=45.91.116.5:4299 , piece_id=6UHFIECEOY2EGUKVFY2T5STZWL4DXXFP3NY4WVZKEEIREZ7ZVX2A, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.702897923 (557h8m29.573980958s) [noise=true , address=45.91.116.6:4975 , piece_id=DCEIFZV6754QAYC5N2SXZ5DIYC7NR6TMHCXD66T4A5JRL7E4LPHA, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.762609574 (557h8m29.633692619s) [noise=true , address=45.91.116.7:4012 , piece_id=C4J7O54YJEGULQT2ZWHPSWTZUU2LU2MN2ND2HSTJI2E6F7OHJSRQ, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.762632388 (557h8m29.633715433s) [seg_pos={0 0}, upload=488441]: Segment reached optimal threshold of 13 pieces. Dec 28 20:37:29.762867626 (557h8m29.633950671s) [noise=true , address=145.79.227.5:4222 , piece_id=IBMJHO2KIDNN5AIIEGX5G5QVDJXXGLOTAMOQC4LZDUBBGY5E6GLQ, seg_pos={0 0}, upload=488441]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1): protocol: write tcp 188.166.225.170:44926->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:44926->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44926->145.79.227.5:4222: use of closed network connection Dec 28 20:37:29.762890609 (557h8m29.633973644s) [noise=true , address=145.79.227.5:4222 , piece_id=IBMJHO2KIDNN5AIIEGX5G5QVDJXXGLOTAMOQC4LZDUBBGY5E6GLQ, seg_pos={0 0}, upload=488441]: Long tail cancellation due to optimal threshold Dec 28 20:37:29.762944091 (557h8m29.634027126s) [noise=true , address=45.91.116.4:4939 , piece_id=XQHY6GBPJ6XEOFL2FTOFNBRLD3GF6VLGUBWY2D7TUOPQSFERNQTA, seg_pos={0 0}, upload=488441]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 28 20:37:29.762949461 (557h8m29.634032496s) [noise=true , address=45.91.116.4:4939 , piece_id=XQHY6GBPJ6XEOFL2FTOFNBRLD3GF6VLGUBWY2D7TUOPQSFERNQTA, seg_pos={0 0}, upload=488441]: Long tail cancellation due to optimal threshold Dec 28 20:37:29.763074540 (557h8m29.634157585s) [noise=true , address=145.79.227.27:4690 , piece_id=SZGKSH5YEIQBDSIMVFXRL2X6PXVXGYZ4DOFXWWE6UDQ4CEFTQHJA, seg_pos={0 0}, upload=488441]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H): protocol: write tcp 188.166.225.170:54746->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:54746->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54746->145.79.227.27:4690: use of closed network connection Dec 28 20:37:29.763081002 (557h8m29.634164037s) [noise=true , address=145.79.227.27:4690 , piece_id=SZGKSH5YEIQBDSIMVFXRL2X6PXVXGYZ4DOFXWWE6UDQ4CEFTQHJA, seg_pos={0 0}, upload=488441]: Long tail cancellation due to optimal threshold Dec 28 20:37:29.763098996 (557h8m29.634182031s) [seg_pos={0 0}, upload=488441]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 20:37:29.763231739 (557h8m29.634314774s) [upload=488441]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 20:37:29.763236939 (557h8m29.634319984s) [upload=488441]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 20:37:30.055521760 (557h8m29.926604805s) [upload=488441]: Done uploading segments... Dec 28 20:37:30.055544604 (557h8m29.926627639s) [upload=488441]: Upload finished. err: <nil> Duration: 6.416931712s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........................X...........X..................X..] duration: 28.074753124s trace: https://jaeger.dev.storj.io/trace/60ddd33e73f0de61 FAIL (2025-12-28 20:39:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 065810223s) [upload=488441]: Uploading segments... Dec 28 20:37:27.194835273 (557h8m27.065918308s) [upload=488441]: Got next segment. Inline: false Dec 28 20:37:27.194877263 (557h8m27.065960308s) [upload=488441]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 20:37:27.194884157 (557h8m27.065967212s) [upload=488441]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 20:37:27.450350479 (557h8m27.321433524s) [seg_pos={0 0}, upload=488441]: Begin upload segment... Dec 28 20:37:27.450441583 (557h8m27.321524628s) [seg_pos={0 0}, upload=488441]: Done begin upload segment. Dec 28 20:37:27.450581259 (557h8m27.321664304s) [noise=true , address=145.79.227.2:4114 , piece_id=MTU5VYRZVWYYP44THTSGDURQ3WHCCNH2C3XJJYQLUD7JBXUU5NZQ, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.450619542 (557h8m27.321702597s) [noise=true , address=45.91.116.6:4975 , piece_id=DCEIFZV6754QAYC5N2SXZ5DIYC7NR6TMHCXD66T4A5JRL7E4LPHA, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.451283105 (557h8m27.32236617s) [noise=true , address=145.79.227.51:4410 , piece_id=HGHWLMHV7LMCQGNQGSFRXUGJUVOIPYBPT3XATCP4EBZ2N24YW5LA, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.451709617 (557h8m27.322792662s) [noise=true , address=145.79.227.14:4840 , piece_id=VVOFKZVGGDZG6E75IVAUFUGSFNAMN5STZIPSURJ64JPGOWY6BJZQ, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.452173731 (557h8m27.323256766s) [noise=true , address=145.79.227.6:4257 , piece_id=DSINVYJEDHKBN7OGTQZM34CT6MBFUZMALJZQFOGTS5CLTSZLD4CA, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.452596105 (557h8m27.32367914s) [noise=true , address=145.79.227.32:4835 , piece_id=NIDZUDIX6K564WEVU7QDG2DH76GODGMLOJVUVLH6GS7GMUV45D7A, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.453086699 (557h8m27.324169734s) [noise=true , address=145.79.227.11:4731 , piece_id=R3RGSPD3B7K72ZH4AWXSR6K6D5KYZYLXIOM6QB346S4XZOGKR24A, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.453622139 (557h8m27.324705184s) [noise=true , address=145.79.227.15:4873 , piece_id=2MYCI55SJDFGKGRPUC2BLKB4XXLTFHCNH65JEV6KADXDBJUNBH3Q, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.455648808 (557h8m27.326731853s) [noise=true , address=145.79.227.5:4222 , piece_id=IBMJHO2KIDNN5AIIEGX5G5QVDJXXGLOTAMOQC4LZDUBBGY5E6GLQ, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.456606212 (557h8m27.327689247s) [noise=true , address=45.91.116.7:4012 , piece_id=C4J7O54YJEGULQT2ZWHPSWTZUU2LU2MN2ND2HSTJI2E6F7OHJSRQ, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.457955700 (557h8m27.329038735s) [noise=true , address=145.79.227.37:4565 , piece_id=3EJIDJFRGBG6272N24VFWNAXKUW7DHHM563DGJTMRURHA27CEREQ, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.459219468 (557h8m27.330302513s) [noise=true , address=145.79.227.27:4690 , piece_id=SZGKSH5YEIQBDSIMVFXRL2X6PXVXGYZ4DOFXWWE6UDQ4CEFTQHJA, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.460412208 (557h8m27.331495254s) [noise=true , address=77.90.31.16:4736 , piece_id=6NKLMJS4Z4DRFNCOYKO36LAI4EUWFB7LVXEYFDUYUMKSX6R235TQ, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.461658171 (557h8m27.332741206s) [noise=true , address=145.79.227.26:4654 , piece_id=XZZEQ7Y3SLK6WJJ22HH5RUTABJIYTONKTITYKENJXREOXSNSDHIA, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.462997892 (557h8m27.334080927s) [noise=true , address=45.91.116.5:4299 , piece_id=6UHFIECEOY2EGUKVFY2T5STZWL4DXXFP3NY4WVZKEEIREZ7ZVX2A, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.465119382 (557h8m27.336202427s) [noise=true , address=45.91.116.4:4939 , piece_id=XQHY6GBPJ6XEOFL2FTOFNBRLD3GF6VLGUBWY2D7TUOPQSFERNQTA, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:29.157770565 (557h8m29.02885361s) [upload=488441]: Next returned nil segment Dec 28 20:37:29.157787036 (557h8m29.028870091s) [upload=488441]: Waiting for error group managing segments... Dec 28 20:37:29.455180586 (557h8m29.326263631s) [noise=true , address=145.79.227.26:4654 , piece_id=XZZEQ7Y3SLK6WJJ22HH5RUTABJIYTONKTITYKENJXREOXSNSDHIA, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.457221773 (557h8m29.328304808s) [noise=true , address=145.79.227.6:4257 , piece_id=DSINVYJEDHKBN7OGTQZM34CT6MBFUZMALJZQFOGTS5CLTSZLD4CA, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.458700027 (557h8m29.329783062s) [noise=true , address=145.79.227.11:4731 , piece_id=R3RGSPD3B7K72ZH4AWXSR6K6D5KYZYLXIOM6QB346S4XZOGKR24A, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.459240897 (557h8m29.330323933s) [noise=true , address=145.79.227.51:4410 , piece_id=HGHWLMHV7LMCQGNQGSFRXUGJUVOIPYBPT3XATCP4EBZ2N24YW5LA, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.462155848 (557h8m29.333238883s) [noise=true , address=145.79.227.15:4873 , piece_id=2MYCI55SJDFGKGRPUC2BLKB4XXLTFHCNH65JEV6KADXDBJUNBH3Q, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.465572805 (557h8m29.33665585s) [noise=true , address=145.79.227.32:4835 , piece_id=NIDZUDIX6K564WEVU7QDG2DH76GODGMLOJVUVLH6GS7GMUV45D7A, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.470903064 (557h8m29.341986109s) [noise=true , address=145.79.227.2:4114 , piece_id=MTU5VYRZVWYYP44THTSGDURQ3WHCCNH2C3XJJYQLUD7JBXUU5NZQ, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.485295701 (557h8m29.356378736s) [noise=true , address=145.79.227.37:4565 , piece_id=3EJIDJFRGBG6272N24VFWNAXKUW7DHHM563DGJTMRURHA27CEREQ, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.497253901 (557h8m29.368336936s) [noise=true , address=145.79.227.14:4840 , piece_id=VVOFKZVGGDZG6E75IVAUFUGSFNAMN5STZIPSURJ64JPGOWY6BJZQ, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.641989392 (557h8m29.513072437s) [noise=true , address=77.90.31.16:4736 , piece_id=6NKLMJS4Z4DRFNCOYKO36LAI4EUWFB7LVXEYFDUYUMKSX6R235TQ, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.701668272 (557h8m29.572751307s) [noise=true , address=45.91.116.5:4299 , piece_id=6UHFIECEOY2EGUKVFY2T5STZWL4DXXFP3NY4WVZKEEIREZ7ZVX2A, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.702897923 (557h8m29.573980958s) [noise=true , address=45.91.116.6:4975 , piece_id=DCEIFZV6754QAYC5N2SXZ5DIYC7NR6TMHCXD66T4A5JRL7E4LPHA, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.762609574 (557h8m29.633692619s) [noise=true , address=45.91.116.7:4012 , piece_id=C4J7O54YJEGULQT2ZWHPSWTZUU2LU2MN2ND2HSTJI2E6F7OHJSRQ, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.762632388 (557h8m29.633715433s) [seg_pos={0 0}, upload=488441]: Segment reached optimal threshold of 13 pieces. Dec 28 20:37:29.762867626 (557h8m29.633950671s) [noise=true , address=145.79.227.5:4222 , piece_id=IBMJHO2KIDNN5AIIEGX5G5QVDJXXGLOTAMOQC4LZDUBBGY5E6GLQ, seg_pos={0 0}, upload=488441]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1): protocol: write tcp 188.166.225.170:44926->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:44926->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44926->145.79.227.5:4222: use of closed network connection Dec 28 20:37:29.762890609 (557h8m29.633973644s) [noise=true , address=145.79.227.5:4222 , piece_id=IBMJHO2KIDNN5AIIEGX5G5QVDJXXGLOTAMOQC4LZDUBBGY5E6GLQ, seg_pos={0 0}, upload=488441]: Long tail cancellation due to optimal threshold Dec 28 20:37:29.762944091 (557h8m29.634027126s) [noise=true , address=45.91.116.4:4939 , piece_id=XQHY6GBPJ6XEOFL2FTOFNBRLD3GF6VLGUBWY2D7TUOPQSFERNQTA, seg_pos={0 0}, upload=488441]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 28 20:37:29.762949461 (557h8m29.634032496s) [noise=true , address=45.91.116.4:4939 , piece_id=XQHY6GBPJ6XEOFL2FTOFNBRLD3GF6VLGUBWY2D7TUOPQSFERNQTA, seg_pos={0 0}, upload=488441]: Long tail cancellation due to optimal threshold Dec 28 20:37:29.763074540 (557h8m29.634157585s) [noise=true , address=145.79.227.27:4690 , piece_id=SZGKSH5YEIQBDSIMVFXRL2X6PXVXGYZ4DOFXWWE6UDQ4CEFTQHJA, seg_pos={0 0}, upload=488441]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H): protocol: write tcp 188.166.225.170:54746->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:54746->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54746->145.79.227.27:4690: use of closed network connection Dec 28 20:37:29.763081002 (557h8m29.634164037s) [noise=true , address=145.79.227.27:4690 , piece_id=SZGKSH5YEIQBDSIMVFXRL2X6PXVXGYZ4DOFXWWE6UDQ4CEFTQHJA, seg_pos={0 0}, upload=488441]: Long tail cancellation due to optimal threshold Dec 28 20:37:29.763098996 (557h8m29.634182031s) [seg_pos={0 0}, upload=488441]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 20:37:29.763231739 (557h8m29.634314774s) [upload=488441]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 20:37:29.763236939 (557h8m29.634319984s) [upload=488441]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 20:37:30.055521760 (557h8m29.926604805s) [upload=488441]: Done uploading segments... Dec 28 20:37:30.055544604 (557h8m29.926627639s) [upload=488441]: Upload finished. err: <nil> Duration: 28.074792137s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 41.02160755s trace: https://jaeger.dev.storj.io/trace/499b96822b8dda80 FAIL (2025-12-28 20:39:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12S7WBBQ1Nq8jbeGrZiRKTk23CdJC31hPW2vU1aaM5SsHeVyeRH, Piece ID: V4P3KCUDLI543TSROBYUMVTQKKSQDDVSKSJSXKEZTQAHTXUJ2G4Q) context deadline exceeded; piecestore: (Node ID: 1kjCEPtjvcJ3D7BBKRSRirPybBwkRZ1dJwyAQCadzCvaLYvWhz, Piece ID: 4OULHJ6Z3QJ3BOPDY74TJSKVJG23TOEUCJCUEGY24I5C3MUTAVLQ) context deadline exceeded; piecestore: (Node ID: 1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c, Piece ID: LIYXCPAZOUPKXLKRH3EOWREXUY2LAJXSRZFQILM2QIU7HTW7SY2Q) context deadline exceeded ======LOGS====== ue , address=145.79.227.23:4580 , piece_id=OHYI2SFM6TKJE4LHVNGO2PVGZDNFIQDAIQGKG4FOEHTTASBWA5ZQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.496428825 (557h3m1.36751187s) [noise=true , address=77.90.31.19:4867 , piece_id=LDAWD2ZFG3BZATDRY7MMNWYIX6TOHQ2OIUPCAWAYN266GNUUKZ5A, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.496861509 (557h3m1.367944544s) [noise=true , address=145.79.227.19:4105 , piece_id=RY35FSM6NTWPNF3JE43FZRSNXXYZPUN2B6LRNNUOG6QQR6XSU36Q, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.496899933 (557h3m1.367982978s) [noise=true , address=77.90.31.19:4858 , piece_id=LIYXCPAZOUPKXLKRH3EOWREXUY2LAJXSRZFQILM2QIU7HTW7SY2Q, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497229089 (557h3m1.368312134s) [noise=true , address=77.90.31.17:4784 , piece_id=JV2ROD6MN65B6LZKOL3S7ATJWXAY6FBQSZWITA6IHAXELY7R42KA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497326194 (557h3m1.368409239s) [noise=true , address=145.79.227.33:4904 , piece_id=QCMN4OSBWIWS4JRLIGQIGDSHZFUF2JWEPVXQ4FQ4G3LG5FBCBAWQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497664128 (557h3m1.368747173s) [noise=true , address=145.79.227.34:4918 , piece_id=SPDT2KI2774UJMQIUSVTHXCDF7PD2FBG64BXUYW67UMLPE6H7SBQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497731405 (557h3m1.36881445s) [noise=true , address=77.90.31.12:4602 , piece_id=2YPYKKTQ6ID6VTI4AUZQEG2SY7JYPRSAPU4JLF7DWSOSBBO74LWQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.498205939 (557h3m1.369288984s) [noise=true , address=77.90.31.16:4745 , piece_id=CZBFU33BHMP2BEZIDJKW3J526537HR6HP7OCT2KMAQVY7LGZLPZQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.498424295 (557h3m1.36950734s) [noise=true , address=77.90.31.21:4895 , piece_id=V4P3KCUDLI543TSROBYUMVTQKKSQDDVSKSJSXKEZTQAHTXUJ2G4Q, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.498685422 (557h3m1.369768457s) [noise=true , address=145.79.227.35:4496 , piece_id=JH4XQBT7XYEJXWBGPPF6SJWCPLV77GBOLGOA53C3WUQMWHZXVZQA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499066348 (557h3m1.370149393s) [noise=true , address=77.90.31.11:4563 , piece_id=DIFJPHSKHROTLKCHP6HQBWT7XGE77HVRZTH2354MV2UEWZLHVGCQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499185725 (557h3m1.37026877s) [noise=true , address=145.79.227.30:4768 , piece_id=KK4LEV6RG3IGLYXID5MJF3KMNSJ7WS5JS45PNOOJ26LXLYAUCWOA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499468122 (557h3m1.370551157s) [noise=true , address=77.90.31.14:4688 , piece_id=4OULHJ6Z3QJ3BOPDY74TJSKVJG23TOEUCJCUEGY24I5C3MUTAVLQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499901047 (557h3m1.370984102s) [noise=true , address=45.91.116.5:4330 , piece_id=EVHLSVVR5JGY43TLSKVEQLQ6KDBEZENVAFL6YT7ZQ77Y7X2TXIKA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.500546175 (557h3m1.37162921s) [noise=true , address=145.79.227.43:4171 , piece_id=YF5DRDPGRRVYFL5SWMH2PKTSV3C72BIWKUDSAIZTXOIVOTYKRMNA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:02.979572381 (557h3m2.850655436s) [upload=488362]: Next returned nil segment Dec 28 20:32:02.979609411 (557h3m2.850692456s) [upload=488362]: Waiting for error group managing segments... Dec 28 20:32:03.416896733 (557h3m3.287979778s) [noise=true , address=145.79.227.23:4580 , piece_id=OHYI2SFM6TKJE4LHVNGO2PVGZDNFIQDAIQGKG4FOEHTTASBWA5ZQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.552263655 (557h3m3.4233467s) [noise=true , address=77.90.31.17:4784 , piece_id=JV2ROD6MN65B6LZKOL3S7ATJWXAY6FBQSZWITA6IHAXELY7R42KA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.573564856 (557h3m3.444647902s) [noise=true , address=77.90.31.12:4602 , piece_id=2YPYKKTQ6ID6VTI4AUZQEG2SY7JYPRSAPU4JLF7DWSOSBBO74LWQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.598560923 (557h3m3.469643968s) [noise=true , address=77.90.31.19:4867 , piece_id=LDAWD2ZFG3BZATDRY7MMNWYIX6TOHQ2OIUPCAWAYN266GNUUKZ5A, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.626925925 (557h3m3.49800897s) [noise=true , address=77.90.31.16:4745 , piece_id=CZBFU33BHMP2BEZIDJKW3J526537HR6HP7OCT2KMAQVY7LGZLPZQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.709061947 (557h3m3.580144992s) [noise=true , address=77.90.31.14:4688 , piece_id=4OULHJ6Z3QJ3BOPDY74TJSKVJG23TOEUCJCUEGY24I5C3MUTAVLQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.788879373 (557h3m3.659962418s) [noise=true , address=77.90.31.21:4895 , piece_id=V4P3KCUDLI543TSROBYUMVTQKKSQDDVSKSJSXKEZTQAHTXUJ2G4Q, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.217616523 (557h3m4.088699569s) [noise=true , address=77.90.31.19:4858 , piece_id=LIYXCPAZOUPKXLKRH3EOWREXUY2LAJXSRZFQILM2QIU7HTW7SY2Q, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.220337625 (557h3m4.09142067s) [noise=true , address=77.90.31.11:4563 , piece_id=DIFJPHSKHROTLKCHP6HQBWT7XGE77HVRZTH2354MV2UEWZLHVGCQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.303958744 (557h3m4.175041789s) [noise=true , address=145.79.227.44:4213 , piece_id=UEAQDBBCP7DIG6ZD3EEM2CKAL4W5QKW23LWFUAMSVUW5ZS2PJDRQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.312794398 (557h3m4.183877444s) [noise=true , address=145.79.227.34:4918 , piece_id=SPDT2KI2774UJMQIUSVTHXCDF7PD2FBG64BXUYW67UMLPE6H7SBQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.349308798 (557h3m4.220391833s) [noise=true , address=145.79.227.40:4058 , piece_id=EQ4LA7T2BCGWY76K3NJI4X77OQJFJED6H4VNGXDQMILU6IVQWOMQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.371518379 (557h3m4.242601424s) [noise=true , address=145.79.227.43:4171 , piece_id=YF5DRDPGRRVYFL5SWMH2PKTSV3C72BIWKUDSAIZTXOIVOTYKRMNA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.376232316 (557h3m4.247315361s) [noise=true , address=145.79.227.30:4768 , piece_id=KK4LEV6RG3IGLYXID5MJF3KMNSJ7WS5JS45PNOOJ26LXLYAUCWOA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.376257844 (557h3m4.247340879s) [seg_pos={0 0}, upload=488362]: Segment reached optimal threshold of 14 pieces. Dec 28 20:32:04.376575799 (557h3m4.247658834s) [noise=true , address=145.79.227.33:4904 , piece_id=QCMN4OSBWIWS4JRLIGQIGDSHZFUF2JWEPVXQ4FQ4G3LG5FBCBAWQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1dC7mC2Cbxmys7ukEkELyF9Us7F1qtptSP6yDJh1EfjVJ6uqsC): protocol: expected piece hash; context canceled; EOF Dec 28 20:32:04.376585127 (557h3m4.247668163s) [noise=true , address=145.79.227.33:4904 , piece_id=QCMN4OSBWIWS4JRLIGQIGDSHZFUF2JWEPVXQ4FQ4G3LG5FBCBAWQ, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.376764359 (557h3m4.247847404s) [noise=true , address=145.79.227.19:4105 , piece_id=RY35FSM6NTWPNF3JE43FZRSNXXYZPUN2B6LRNNUOG6QQR6XSU36Q, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GhDrKMcDEg2RxamqKtiHphpgNdvtZYdHXrjzGsNJFjBFYeEen): protocol: write tcp 188.166.225.170:38838->145.79.227.19:4105: use of closed network connection; write tcp 188.166.225.170:38838->145.79.227.19:4105: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38838->145.79.227.19:4105: use of closed network connection Dec 28 20:32:04.376772865 (557h3m4.2478559s) [noise=true , address=145.79.227.19:4105 , piece_id=RY35FSM6NTWPNF3JE43FZRSNXXYZPUN2B6LRNNUOG6QQR6XSU36Q, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.376926658 (557h3m4.248009703s) [noise=true , address=145.79.227.35:4496 , piece_id=JH4XQBT7XYEJXWBGPPF6SJWCPLV77GBOLGOA53C3WUQMWHZXVZQA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V8XhSWkyGUVYFR8MykifFAP1YAfGCwzqucDcMMfzw3xsVW77Z): protocol: expected piece hash; context canceled; EOF Dec 28 20:32:04.377165332 (557h3m4.248248377s) [noise=true , address=145.79.227.35:4496 , piece_id=JH4XQBT7XYEJXWBGPPF6SJWCPLV77GBOLGOA53C3WUQMWHZXVZQA, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.377212682 (557h3m4.248295848s) [noise=true , address=45.91.116.5:4330 , piece_id=EVHLSVVR5JGY43TLSKVEQLQ6KDBEZENVAFL6YT7ZQ77Y7X2TXIKA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12MK9aVDRwfWPfckFR6RfPUL1Nrm9bwfsgV7TXCKGbtadHebWvW): protocol: read tcp 188.166.225.170:55416->45.91.116.5:4330: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 20:32:04.377289859 (557h3m4.248372924s) [noise=true , address=45.91.116.5:4330 , piece_id=EVHLSVVR5JGY43TLSKVEQLQ6KDBEZENVAFL6YT7ZQ77Y7X2TXIKA, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.377387736 (557h3m4.248470781s) [seg_pos={0 0}, upload=488362]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 20:32:04.377599729 (557h3m4.248682774s) [upload=488362]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 20:32:04.377613535 (557h3m4.24869662s) [upload=488362]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 20:32:04.675090855 (557h3m4.5461739s) [upload=488362]: Done uploading segments... Dec 28 20:32:04.675121233 (557h3m4.546204268s) [upload=488362]: Upload finished. err: <nil> Duration: 41.021637247s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [X..........................X...........X..................X.] duration: 5.83937331s trace: https://jaeger.dev.storj.io/trace/60ddd33e73f0de61 FAIL (2025-12-28 20:38:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 065810223s) [upload=488441]: Uploading segments... Dec 28 20:37:27.194835273 (557h8m27.065918308s) [upload=488441]: Got next segment. Inline: false Dec 28 20:37:27.194877263 (557h8m27.065960308s) [upload=488441]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 20:37:27.194884157 (557h8m27.065967212s) [upload=488441]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 20:37:27.450350479 (557h8m27.321433524s) [seg_pos={0 0}, upload=488441]: Begin upload segment... Dec 28 20:37:27.450441583 (557h8m27.321524628s) [seg_pos={0 0}, upload=488441]: Done begin upload segment. Dec 28 20:37:27.450581259 (557h8m27.321664304s) [noise=true , address=145.79.227.2:4114 , piece_id=MTU5VYRZVWYYP44THTSGDURQ3WHCCNH2C3XJJYQLUD7JBXUU5NZQ, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.450619542 (557h8m27.321702597s) [noise=true , address=45.91.116.6:4975 , piece_id=DCEIFZV6754QAYC5N2SXZ5DIYC7NR6TMHCXD66T4A5JRL7E4LPHA, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.451283105 (557h8m27.32236617s) [noise=true , address=145.79.227.51:4410 , piece_id=HGHWLMHV7LMCQGNQGSFRXUGJUVOIPYBPT3XATCP4EBZ2N24YW5LA, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.451709617 (557h8m27.322792662s) [noise=true , address=145.79.227.14:4840 , piece_id=VVOFKZVGGDZG6E75IVAUFUGSFNAMN5STZIPSURJ64JPGOWY6BJZQ, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.452173731 (557h8m27.323256766s) [noise=true , address=145.79.227.6:4257 , piece_id=DSINVYJEDHKBN7OGTQZM34CT6MBFUZMALJZQFOGTS5CLTSZLD4CA, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.452596105 (557h8m27.32367914s) [noise=true , address=145.79.227.32:4835 , piece_id=NIDZUDIX6K564WEVU7QDG2DH76GODGMLOJVUVLH6GS7GMUV45D7A, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.453086699 (557h8m27.324169734s) [noise=true , address=145.79.227.11:4731 , piece_id=R3RGSPD3B7K72ZH4AWXSR6K6D5KYZYLXIOM6QB346S4XZOGKR24A, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.453622139 (557h8m27.324705184s) [noise=true , address=145.79.227.15:4873 , piece_id=2MYCI55SJDFGKGRPUC2BLKB4XXLTFHCNH65JEV6KADXDBJUNBH3Q, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.455648808 (557h8m27.326731853s) [noise=true , address=145.79.227.5:4222 , piece_id=IBMJHO2KIDNN5AIIEGX5G5QVDJXXGLOTAMOQC4LZDUBBGY5E6GLQ, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.456606212 (557h8m27.327689247s) [noise=true , address=45.91.116.7:4012 , piece_id=C4J7O54YJEGULQT2ZWHPSWTZUU2LU2MN2ND2HSTJI2E6F7OHJSRQ, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.457955700 (557h8m27.329038735s) [noise=true , address=145.79.227.37:4565 , piece_id=3EJIDJFRGBG6272N24VFWNAXKUW7DHHM563DGJTMRURHA27CEREQ, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.459219468 (557h8m27.330302513s) [noise=true , address=145.79.227.27:4690 , piece_id=SZGKSH5YEIQBDSIMVFXRL2X6PXVXGYZ4DOFXWWE6UDQ4CEFTQHJA, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.460412208 (557h8m27.331495254s) [noise=true , address=77.90.31.16:4736 , piece_id=6NKLMJS4Z4DRFNCOYKO36LAI4EUWFB7LVXEYFDUYUMKSX6R235TQ, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.461658171 (557h8m27.332741206s) [noise=true , address=145.79.227.26:4654 , piece_id=XZZEQ7Y3SLK6WJJ22HH5RUTABJIYTONKTITYKENJXREOXSNSDHIA, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.462997892 (557h8m27.334080927s) [noise=true , address=45.91.116.5:4299 , piece_id=6UHFIECEOY2EGUKVFY2T5STZWL4DXXFP3NY4WVZKEEIREZ7ZVX2A, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.465119382 (557h8m27.336202427s) [noise=true , address=45.91.116.4:4939 , piece_id=XQHY6GBPJ6XEOFL2FTOFNBRLD3GF6VLGUBWY2D7TUOPQSFERNQTA, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:29.157770565 (557h8m29.02885361s) [upload=488441]: Next returned nil segment Dec 28 20:37:29.157787036 (557h8m29.028870091s) [upload=488441]: Waiting for error group managing segments... Dec 28 20:37:29.455180586 (557h8m29.326263631s) [noise=true , address=145.79.227.26:4654 , piece_id=XZZEQ7Y3SLK6WJJ22HH5RUTABJIYTONKTITYKENJXREOXSNSDHIA, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.457221773 (557h8m29.328304808s) [noise=true , address=145.79.227.6:4257 , piece_id=DSINVYJEDHKBN7OGTQZM34CT6MBFUZMALJZQFOGTS5CLTSZLD4CA, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.458700027 (557h8m29.329783062s) [noise=true , address=145.79.227.11:4731 , piece_id=R3RGSPD3B7K72ZH4AWXSR6K6D5KYZYLXIOM6QB346S4XZOGKR24A, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.459240897 (557h8m29.330323933s) [noise=true , address=145.79.227.51:4410 , piece_id=HGHWLMHV7LMCQGNQGSFRXUGJUVOIPYBPT3XATCP4EBZ2N24YW5LA, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.462155848 (557h8m29.333238883s) [noise=true , address=145.79.227.15:4873 , piece_id=2MYCI55SJDFGKGRPUC2BLKB4XXLTFHCNH65JEV6KADXDBJUNBH3Q, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.465572805 (557h8m29.33665585s) [noise=true , address=145.79.227.32:4835 , piece_id=NIDZUDIX6K564WEVU7QDG2DH76GODGMLOJVUVLH6GS7GMUV45D7A, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.470903064 (557h8m29.341986109s) [noise=true , address=145.79.227.2:4114 , piece_id=MTU5VYRZVWYYP44THTSGDURQ3WHCCNH2C3XJJYQLUD7JBXUU5NZQ, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.485295701 (557h8m29.356378736s) [noise=true , address=145.79.227.37:4565 , piece_id=3EJIDJFRGBG6272N24VFWNAXKUW7DHHM563DGJTMRURHA27CEREQ, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.497253901 (557h8m29.368336936s) [noise=true , address=145.79.227.14:4840 , piece_id=VVOFKZVGGDZG6E75IVAUFUGSFNAMN5STZIPSURJ64JPGOWY6BJZQ, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.641989392 (557h8m29.513072437s) [noise=true , address=77.90.31.16:4736 , piece_id=6NKLMJS4Z4DRFNCOYKO36LAI4EUWFB7LVXEYFDUYUMKSX6R235TQ, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.701668272 (557h8m29.572751307s) [noise=true , address=45.91.116.5:4299 , piece_id=6UHFIECEOY2EGUKVFY2T5STZWL4DXXFP3NY4WVZKEEIREZ7ZVX2A, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.702897923 (557h8m29.573980958s) [noise=true , address=45.91.116.6:4975 , piece_id=DCEIFZV6754QAYC5N2SXZ5DIYC7NR6TMHCXD66T4A5JRL7E4LPHA, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.762609574 (557h8m29.633692619s) [noise=true , address=45.91.116.7:4012 , piece_id=C4J7O54YJEGULQT2ZWHPSWTZUU2LU2MN2ND2HSTJI2E6F7OHJSRQ, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.762632388 (557h8m29.633715433s) [seg_pos={0 0}, upload=488441]: Segment reached optimal threshold of 13 pieces. Dec 28 20:37:29.762867626 (557h8m29.633950671s) [noise=true , address=145.79.227.5:4222 , piece_id=IBMJHO2KIDNN5AIIEGX5G5QVDJXXGLOTAMOQC4LZDUBBGY5E6GLQ, seg_pos={0 0}, upload=488441]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1): protocol: write tcp 188.166.225.170:44926->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:44926->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44926->145.79.227.5:4222: use of closed network connection Dec 28 20:37:29.762890609 (557h8m29.633973644s) [noise=true , address=145.79.227.5:4222 , piece_id=IBMJHO2KIDNN5AIIEGX5G5QVDJXXGLOTAMOQC4LZDUBBGY5E6GLQ, seg_pos={0 0}, upload=488441]: Long tail cancellation due to optimal threshold Dec 28 20:37:29.762944091 (557h8m29.634027126s) [noise=true , address=45.91.116.4:4939 , piece_id=XQHY6GBPJ6XEOFL2FTOFNBRLD3GF6VLGUBWY2D7TUOPQSFERNQTA, seg_pos={0 0}, upload=488441]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 28 20:37:29.762949461 (557h8m29.634032496s) [noise=true , address=45.91.116.4:4939 , piece_id=XQHY6GBPJ6XEOFL2FTOFNBRLD3GF6VLGUBWY2D7TUOPQSFERNQTA, seg_pos={0 0}, upload=488441]: Long tail cancellation due to optimal threshold Dec 28 20:37:29.763074540 (557h8m29.634157585s) [noise=true , address=145.79.227.27:4690 , piece_id=SZGKSH5YEIQBDSIMVFXRL2X6PXVXGYZ4DOFXWWE6UDQ4CEFTQHJA, seg_pos={0 0}, upload=488441]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H): protocol: write tcp 188.166.225.170:54746->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:54746->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54746->145.79.227.27:4690: use of closed network connection Dec 28 20:37:29.763081002 (557h8m29.634164037s) [noise=true , address=145.79.227.27:4690 , piece_id=SZGKSH5YEIQBDSIMVFXRL2X6PXVXGYZ4DOFXWWE6UDQ4CEFTQHJA, seg_pos={0 0}, upload=488441]: Long tail cancellation due to optimal threshold Dec 28 20:37:29.763098996 (557h8m29.634182031s) [seg_pos={0 0}, upload=488441]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 20:37:29.763231739 (557h8m29.634314774s) [upload=488441]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 20:37:29.763236939 (557h8m29.634319984s) [upload=488441]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 20:37:30.055521760 (557h8m29.926604805s) [upload=488441]: Done uploading segments... Dec 28 20:37:30.055544604 (557h8m29.926627639s) [upload=488441]: Upload finished. err: <nil> Duration: 5.839402254s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.X..........................X...........X..................X] duration: 30.002949498s trace: https://jaeger.dev.storj.io/trace/60ddd33e73f0de61 FAIL (2025-12-28 20:37:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 065810223s) [upload=488441]: Uploading segments... Dec 28 20:37:27.194835273 (557h8m27.065918308s) [upload=488441]: Got next segment. Inline: false Dec 28 20:37:27.194877263 (557h8m27.065960308s) [upload=488441]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 20:37:27.194884157 (557h8m27.065967212s) [upload=488441]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 20:37:27.450350479 (557h8m27.321433524s) [seg_pos={0 0}, upload=488441]: Begin upload segment... Dec 28 20:37:27.450441583 (557h8m27.321524628s) [seg_pos={0 0}, upload=488441]: Done begin upload segment. Dec 28 20:37:27.450581259 (557h8m27.321664304s) [noise=true , address=145.79.227.2:4114 , piece_id=MTU5VYRZVWYYP44THTSGDURQ3WHCCNH2C3XJJYQLUD7JBXUU5NZQ, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.450619542 (557h8m27.321702597s) [noise=true , address=45.91.116.6:4975 , piece_id=DCEIFZV6754QAYC5N2SXZ5DIYC7NR6TMHCXD66T4A5JRL7E4LPHA, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.451283105 (557h8m27.32236617s) [noise=true , address=145.79.227.51:4410 , piece_id=HGHWLMHV7LMCQGNQGSFRXUGJUVOIPYBPT3XATCP4EBZ2N24YW5LA, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.451709617 (557h8m27.322792662s) [noise=true , address=145.79.227.14:4840 , piece_id=VVOFKZVGGDZG6E75IVAUFUGSFNAMN5STZIPSURJ64JPGOWY6BJZQ, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.452173731 (557h8m27.323256766s) [noise=true , address=145.79.227.6:4257 , piece_id=DSINVYJEDHKBN7OGTQZM34CT6MBFUZMALJZQFOGTS5CLTSZLD4CA, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.452596105 (557h8m27.32367914s) [noise=true , address=145.79.227.32:4835 , piece_id=NIDZUDIX6K564WEVU7QDG2DH76GODGMLOJVUVLH6GS7GMUV45D7A, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.453086699 (557h8m27.324169734s) [noise=true , address=145.79.227.11:4731 , piece_id=R3RGSPD3B7K72ZH4AWXSR6K6D5KYZYLXIOM6QB346S4XZOGKR24A, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.453622139 (557h8m27.324705184s) [noise=true , address=145.79.227.15:4873 , piece_id=2MYCI55SJDFGKGRPUC2BLKB4XXLTFHCNH65JEV6KADXDBJUNBH3Q, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.455648808 (557h8m27.326731853s) [noise=true , address=145.79.227.5:4222 , piece_id=IBMJHO2KIDNN5AIIEGX5G5QVDJXXGLOTAMOQC4LZDUBBGY5E6GLQ, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.456606212 (557h8m27.327689247s) [noise=true , address=45.91.116.7:4012 , piece_id=C4J7O54YJEGULQT2ZWHPSWTZUU2LU2MN2ND2HSTJI2E6F7OHJSRQ, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.457955700 (557h8m27.329038735s) [noise=true , address=145.79.227.37:4565 , piece_id=3EJIDJFRGBG6272N24VFWNAXKUW7DHHM563DGJTMRURHA27CEREQ, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.459219468 (557h8m27.330302513s) [noise=true , address=145.79.227.27:4690 , piece_id=SZGKSH5YEIQBDSIMVFXRL2X6PXVXGYZ4DOFXWWE6UDQ4CEFTQHJA, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.460412208 (557h8m27.331495254s) [noise=true , address=77.90.31.16:4736 , piece_id=6NKLMJS4Z4DRFNCOYKO36LAI4EUWFB7LVXEYFDUYUMKSX6R235TQ, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.461658171 (557h8m27.332741206s) [noise=true , address=145.79.227.26:4654 , piece_id=XZZEQ7Y3SLK6WJJ22HH5RUTABJIYTONKTITYKENJXREOXSNSDHIA, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.462997892 (557h8m27.334080927s) [noise=true , address=45.91.116.5:4299 , piece_id=6UHFIECEOY2EGUKVFY2T5STZWL4DXXFP3NY4WVZKEEIREZ7ZVX2A, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.465119382 (557h8m27.336202427s) [noise=true , address=45.91.116.4:4939 , piece_id=XQHY6GBPJ6XEOFL2FTOFNBRLD3GF6VLGUBWY2D7TUOPQSFERNQTA, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:29.157770565 (557h8m29.02885361s) [upload=488441]: Next returned nil segment Dec 28 20:37:29.157787036 (557h8m29.028870091s) [upload=488441]: Waiting for error group managing segments... Dec 28 20:37:29.455180586 (557h8m29.326263631s) [noise=true , address=145.79.227.26:4654 , piece_id=XZZEQ7Y3SLK6WJJ22HH5RUTABJIYTONKTITYKENJXREOXSNSDHIA, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.457221773 (557h8m29.328304808s) [noise=true , address=145.79.227.6:4257 , piece_id=DSINVYJEDHKBN7OGTQZM34CT6MBFUZMALJZQFOGTS5CLTSZLD4CA, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.458700027 (557h8m29.329783062s) [noise=true , address=145.79.227.11:4731 , piece_id=R3RGSPD3B7K72ZH4AWXSR6K6D5KYZYLXIOM6QB346S4XZOGKR24A, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.459240897 (557h8m29.330323933s) [noise=true , address=145.79.227.51:4410 , piece_id=HGHWLMHV7LMCQGNQGSFRXUGJUVOIPYBPT3XATCP4EBZ2N24YW5LA, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.462155848 (557h8m29.333238883s) [noise=true , address=145.79.227.15:4873 , piece_id=2MYCI55SJDFGKGRPUC2BLKB4XXLTFHCNH65JEV6KADXDBJUNBH3Q, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.465572805 (557h8m29.33665585s) [noise=true , address=145.79.227.32:4835 , piece_id=NIDZUDIX6K564WEVU7QDG2DH76GODGMLOJVUVLH6GS7GMUV45D7A, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.470903064 (557h8m29.341986109s) [noise=true , address=145.79.227.2:4114 , piece_id=MTU5VYRZVWYYP44THTSGDURQ3WHCCNH2C3XJJYQLUD7JBXUU5NZQ, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.485295701 (557h8m29.356378736s) [noise=true , address=145.79.227.37:4565 , piece_id=3EJIDJFRGBG6272N24VFWNAXKUW7DHHM563DGJTMRURHA27CEREQ, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.497253901 (557h8m29.368336936s) [noise=true , address=145.79.227.14:4840 , piece_id=VVOFKZVGGDZG6E75IVAUFUGSFNAMN5STZIPSURJ64JPGOWY6BJZQ, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.641989392 (557h8m29.513072437s) [noise=true , address=77.90.31.16:4736 , piece_id=6NKLMJS4Z4DRFNCOYKO36LAI4EUWFB7LVXEYFDUYUMKSX6R235TQ, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.701668272 (557h8m29.572751307s) [noise=true , address=45.91.116.5:4299 , piece_id=6UHFIECEOY2EGUKVFY2T5STZWL4DXXFP3NY4WVZKEEIREZ7ZVX2A, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.702897923 (557h8m29.573980958s) [noise=true , address=45.91.116.6:4975 , piece_id=DCEIFZV6754QAYC5N2SXZ5DIYC7NR6TMHCXD66T4A5JRL7E4LPHA, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.762609574 (557h8m29.633692619s) [noise=true , address=45.91.116.7:4012 , piece_id=C4J7O54YJEGULQT2ZWHPSWTZUU2LU2MN2ND2HSTJI2E6F7OHJSRQ, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.762632388 (557h8m29.633715433s) [seg_pos={0 0}, upload=488441]: Segment reached optimal threshold of 13 pieces. Dec 28 20:37:29.762867626 (557h8m29.633950671s) [noise=true , address=145.79.227.5:4222 , piece_id=IBMJHO2KIDNN5AIIEGX5G5QVDJXXGLOTAMOQC4LZDUBBGY5E6GLQ, seg_pos={0 0}, upload=488441]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1): protocol: write tcp 188.166.225.170:44926->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:44926->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44926->145.79.227.5:4222: use of closed network connection Dec 28 20:37:29.762890609 (557h8m29.633973644s) [noise=true , address=145.79.227.5:4222 , piece_id=IBMJHO2KIDNN5AIIEGX5G5QVDJXXGLOTAMOQC4LZDUBBGY5E6GLQ, seg_pos={0 0}, upload=488441]: Long tail cancellation due to optimal threshold Dec 28 20:37:29.762944091 (557h8m29.634027126s) [noise=true , address=45.91.116.4:4939 , piece_id=XQHY6GBPJ6XEOFL2FTOFNBRLD3GF6VLGUBWY2D7TUOPQSFERNQTA, seg_pos={0 0}, upload=488441]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 28 20:37:29.762949461 (557h8m29.634032496s) [noise=true , address=45.91.116.4:4939 , piece_id=XQHY6GBPJ6XEOFL2FTOFNBRLD3GF6VLGUBWY2D7TUOPQSFERNQTA, seg_pos={0 0}, upload=488441]: Long tail cancellation due to optimal threshold Dec 28 20:37:29.763074540 (557h8m29.634157585s) [noise=true , address=145.79.227.27:4690 , piece_id=SZGKSH5YEIQBDSIMVFXRL2X6PXVXGYZ4DOFXWWE6UDQ4CEFTQHJA, seg_pos={0 0}, upload=488441]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H): protocol: write tcp 188.166.225.170:54746->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:54746->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54746->145.79.227.27:4690: use of closed network connection Dec 28 20:37:29.763081002 (557h8m29.634164037s) [noise=true , address=145.79.227.27:4690 , piece_id=SZGKSH5YEIQBDSIMVFXRL2X6PXVXGYZ4DOFXWWE6UDQ4CEFTQHJA, seg_pos={0 0}, upload=488441]: Long tail cancellation due to optimal threshold Dec 28 20:37:29.763098996 (557h8m29.634182031s) [seg_pos={0 0}, upload=488441]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 20:37:29.763231739 (557h8m29.634314774s) [upload=488441]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 20:37:29.763236939 (557h8m29.634319984s) [upload=488441]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 20:37:30.055521760 (557h8m29.926604805s) [upload=488441]: Done uploading segments... Dec 28 20:37:30.055544604 (557h8m29.926627639s) [upload=488441]: Upload finished. err: <nil> Duration: 30.002976549s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 17.679037293s trace: https://jaeger.dev.storj.io/trace/499b96822b8dda80 FAIL (2025-12-28 20:37:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12S7WBBQ1Nq8jbeGrZiRKTk23CdJC31hPW2vU1aaM5SsHeVyeRH, Piece ID: V4P3KCUDLI543TSROBYUMVTQKKSQDDVSKSJSXKEZTQAHTXUJ2G4Q) context deadline exceeded; piecestore: (Node ID: 1kjCEPtjvcJ3D7BBKRSRirPybBwkRZ1dJwyAQCadzCvaLYvWhz, Piece ID: 4OULHJ6Z3QJ3BOPDY74TJSKVJG23TOEUCJCUEGY24I5C3MUTAVLQ) context deadline exceeded; piecestore: (Node ID: 1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c, Piece ID: LIYXCPAZOUPKXLKRH3EOWREXUY2LAJXSRZFQILM2QIU7HTW7SY2Q) context deadline exceeded ======LOGS====== ue , address=145.79.227.23:4580 , piece_id=OHYI2SFM6TKJE4LHVNGO2PVGZDNFIQDAIQGKG4FOEHTTASBWA5ZQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.496428825 (557h3m1.36751187s) [noise=true , address=77.90.31.19:4867 , piece_id=LDAWD2ZFG3BZATDRY7MMNWYIX6TOHQ2OIUPCAWAYN266GNUUKZ5A, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.496861509 (557h3m1.367944544s) [noise=true , address=145.79.227.19:4105 , piece_id=RY35FSM6NTWPNF3JE43FZRSNXXYZPUN2B6LRNNUOG6QQR6XSU36Q, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.496899933 (557h3m1.367982978s) [noise=true , address=77.90.31.19:4858 , piece_id=LIYXCPAZOUPKXLKRH3EOWREXUY2LAJXSRZFQILM2QIU7HTW7SY2Q, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497229089 (557h3m1.368312134s) [noise=true , address=77.90.31.17:4784 , piece_id=JV2ROD6MN65B6LZKOL3S7ATJWXAY6FBQSZWITA6IHAXELY7R42KA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497326194 (557h3m1.368409239s) [noise=true , address=145.79.227.33:4904 , piece_id=QCMN4OSBWIWS4JRLIGQIGDSHZFUF2JWEPVXQ4FQ4G3LG5FBCBAWQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497664128 (557h3m1.368747173s) [noise=true , address=145.79.227.34:4918 , piece_id=SPDT2KI2774UJMQIUSVTHXCDF7PD2FBG64BXUYW67UMLPE6H7SBQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497731405 (557h3m1.36881445s) [noise=true , address=77.90.31.12:4602 , piece_id=2YPYKKTQ6ID6VTI4AUZQEG2SY7JYPRSAPU4JLF7DWSOSBBO74LWQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.498205939 (557h3m1.369288984s) [noise=true , address=77.90.31.16:4745 , piece_id=CZBFU33BHMP2BEZIDJKW3J526537HR6HP7OCT2KMAQVY7LGZLPZQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.498424295 (557h3m1.36950734s) [noise=true , address=77.90.31.21:4895 , piece_id=V4P3KCUDLI543TSROBYUMVTQKKSQDDVSKSJSXKEZTQAHTXUJ2G4Q, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.498685422 (557h3m1.369768457s) [noise=true , address=145.79.227.35:4496 , piece_id=JH4XQBT7XYEJXWBGPPF6SJWCPLV77GBOLGOA53C3WUQMWHZXVZQA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499066348 (557h3m1.370149393s) [noise=true , address=77.90.31.11:4563 , piece_id=DIFJPHSKHROTLKCHP6HQBWT7XGE77HVRZTH2354MV2UEWZLHVGCQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499185725 (557h3m1.37026877s) [noise=true , address=145.79.227.30:4768 , piece_id=KK4LEV6RG3IGLYXID5MJF3KMNSJ7WS5JS45PNOOJ26LXLYAUCWOA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499468122 (557h3m1.370551157s) [noise=true , address=77.90.31.14:4688 , piece_id=4OULHJ6Z3QJ3BOPDY74TJSKVJG23TOEUCJCUEGY24I5C3MUTAVLQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499901047 (557h3m1.370984102s) [noise=true , address=45.91.116.5:4330 , piece_id=EVHLSVVR5JGY43TLSKVEQLQ6KDBEZENVAFL6YT7ZQ77Y7X2TXIKA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.500546175 (557h3m1.37162921s) [noise=true , address=145.79.227.43:4171 , piece_id=YF5DRDPGRRVYFL5SWMH2PKTSV3C72BIWKUDSAIZTXOIVOTYKRMNA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:02.979572381 (557h3m2.850655436s) [upload=488362]: Next returned nil segment Dec 28 20:32:02.979609411 (557h3m2.850692456s) [upload=488362]: Waiting for error group managing segments... Dec 28 20:32:03.416896733 (557h3m3.287979778s) [noise=true , address=145.79.227.23:4580 , piece_id=OHYI2SFM6TKJE4LHVNGO2PVGZDNFIQDAIQGKG4FOEHTTASBWA5ZQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.552263655 (557h3m3.4233467s) [noise=true , address=77.90.31.17:4784 , piece_id=JV2ROD6MN65B6LZKOL3S7ATJWXAY6FBQSZWITA6IHAXELY7R42KA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.573564856 (557h3m3.444647902s) [noise=true , address=77.90.31.12:4602 , piece_id=2YPYKKTQ6ID6VTI4AUZQEG2SY7JYPRSAPU4JLF7DWSOSBBO74LWQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.598560923 (557h3m3.469643968s) [noise=true , address=77.90.31.19:4867 , piece_id=LDAWD2ZFG3BZATDRY7MMNWYIX6TOHQ2OIUPCAWAYN266GNUUKZ5A, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.626925925 (557h3m3.49800897s) [noise=true , address=77.90.31.16:4745 , piece_id=CZBFU33BHMP2BEZIDJKW3J526537HR6HP7OCT2KMAQVY7LGZLPZQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.709061947 (557h3m3.580144992s) [noise=true , address=77.90.31.14:4688 , piece_id=4OULHJ6Z3QJ3BOPDY74TJSKVJG23TOEUCJCUEGY24I5C3MUTAVLQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.788879373 (557h3m3.659962418s) [noise=true , address=77.90.31.21:4895 , piece_id=V4P3KCUDLI543TSROBYUMVTQKKSQDDVSKSJSXKEZTQAHTXUJ2G4Q, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.217616523 (557h3m4.088699569s) [noise=true , address=77.90.31.19:4858 , piece_id=LIYXCPAZOUPKXLKRH3EOWREXUY2LAJXSRZFQILM2QIU7HTW7SY2Q, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.220337625 (557h3m4.09142067s) [noise=true , address=77.90.31.11:4563 , piece_id=DIFJPHSKHROTLKCHP6HQBWT7XGE77HVRZTH2354MV2UEWZLHVGCQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.303958744 (557h3m4.175041789s) [noise=true , address=145.79.227.44:4213 , piece_id=UEAQDBBCP7DIG6ZD3EEM2CKAL4W5QKW23LWFUAMSVUW5ZS2PJDRQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.312794398 (557h3m4.183877444s) [noise=true , address=145.79.227.34:4918 , piece_id=SPDT2KI2774UJMQIUSVTHXCDF7PD2FBG64BXUYW67UMLPE6H7SBQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.349308798 (557h3m4.220391833s) [noise=true , address=145.79.227.40:4058 , piece_id=EQ4LA7T2BCGWY76K3NJI4X77OQJFJED6H4VNGXDQMILU6IVQWOMQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.371518379 (557h3m4.242601424s) [noise=true , address=145.79.227.43:4171 , piece_id=YF5DRDPGRRVYFL5SWMH2PKTSV3C72BIWKUDSAIZTXOIVOTYKRMNA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.376232316 (557h3m4.247315361s) [noise=true , address=145.79.227.30:4768 , piece_id=KK4LEV6RG3IGLYXID5MJF3KMNSJ7WS5JS45PNOOJ26LXLYAUCWOA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.376257844 (557h3m4.247340879s) [seg_pos={0 0}, upload=488362]: Segment reached optimal threshold of 14 pieces. Dec 28 20:32:04.376575799 (557h3m4.247658834s) [noise=true , address=145.79.227.33:4904 , piece_id=QCMN4OSBWIWS4JRLIGQIGDSHZFUF2JWEPVXQ4FQ4G3LG5FBCBAWQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1dC7mC2Cbxmys7ukEkELyF9Us7F1qtptSP6yDJh1EfjVJ6uqsC): protocol: expected piece hash; context canceled; EOF Dec 28 20:32:04.376585127 (557h3m4.247668163s) [noise=true , address=145.79.227.33:4904 , piece_id=QCMN4OSBWIWS4JRLIGQIGDSHZFUF2JWEPVXQ4FQ4G3LG5FBCBAWQ, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.376764359 (557h3m4.247847404s) [noise=true , address=145.79.227.19:4105 , piece_id=RY35FSM6NTWPNF3JE43FZRSNXXYZPUN2B6LRNNUOG6QQR6XSU36Q, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GhDrKMcDEg2RxamqKtiHphpgNdvtZYdHXrjzGsNJFjBFYeEen): protocol: write tcp 188.166.225.170:38838->145.79.227.19:4105: use of closed network connection; write tcp 188.166.225.170:38838->145.79.227.19:4105: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38838->145.79.227.19:4105: use of closed network connection Dec 28 20:32:04.376772865 (557h3m4.2478559s) [noise=true , address=145.79.227.19:4105 , piece_id=RY35FSM6NTWPNF3JE43FZRSNXXYZPUN2B6LRNNUOG6QQR6XSU36Q, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.376926658 (557h3m4.248009703s) [noise=true , address=145.79.227.35:4496 , piece_id=JH4XQBT7XYEJXWBGPPF6SJWCPLV77GBOLGOA53C3WUQMWHZXVZQA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V8XhSWkyGUVYFR8MykifFAP1YAfGCwzqucDcMMfzw3xsVW77Z): protocol: expected piece hash; context canceled; EOF Dec 28 20:32:04.377165332 (557h3m4.248248377s) [noise=true , address=145.79.227.35:4496 , piece_id=JH4XQBT7XYEJXWBGPPF6SJWCPLV77GBOLGOA53C3WUQMWHZXVZQA, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.377212682 (557h3m4.248295848s) [noise=true , address=45.91.116.5:4330 , piece_id=EVHLSVVR5JGY43TLSKVEQLQ6KDBEZENVAFL6YT7ZQ77Y7X2TXIKA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12MK9aVDRwfWPfckFR6RfPUL1Nrm9bwfsgV7TXCKGbtadHebWvW): protocol: read tcp 188.166.225.170:55416->45.91.116.5:4330: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 20:32:04.377289859 (557h3m4.248372924s) [noise=true , address=45.91.116.5:4330 , piece_id=EVHLSVVR5JGY43TLSKVEQLQ6KDBEZENVAFL6YT7ZQ77Y7X2TXIKA, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.377387736 (557h3m4.248470781s) [seg_pos={0 0}, upload=488362]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 20:32:04.377599729 (557h3m4.248682774s) [upload=488362]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 20:32:04.377613535 (557h3m4.24869662s) [upload=488362]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 20:32:04.675090855 (557h3m4.5461739s) [upload=488362]: Done uploading segments... Dec 28 20:32:04.675121233 (557h3m4.546204268s) [upload=488362]: Upload finished. err: <nil> Duration: 17.679063934s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)